[neek]

This is weird, I used to add "site:nwhikers.net" to my google searches all the time to search this site exclusively, but it no longer returns results (or sometimes just one). Did we get blacklisted? I'll try again later, maybe google is just having a moment.

[dave allyn]

Try it with The Angry Hiker

[neek]

Try it with whatever you like. Are you getting different results? For me even w/o cookies it's just crickets.

[seawallrunner]

Same. A few weeks ago I tried searching for previous posts with REI in the title or body, and the search returned empty.

[Tom]

This summer MCaver needed to add some stuff to block a rogue bot that was hammering the site. I'm guessing it might have led to unintended side effects. Nothing else has changed.

[huron]

Works from duckduckgo

[zimmertr]

Probably this: https://developers.google.com/search/docs/crawling-indexing/block-indexing

[Tom]

Seems as if MCaver unintentionally blocked all bots, I've tweaked the robots.txt to allow all bots and block rogue bots via .htaccess instead. Hopefully the google indexing comes back. FWIW, this is how bad that rogue bot was hammering the site this summer, caused several hundred dollars of bandwidth overage charges. If you google Bytespider you'll see it continues to hammer other sites with webmasters trying to figure out how to block it. Apparently it doesn't honor robots.txt so maybe we just got lucky that it stopped hammering us, at the cost of putting in something that only served to block well behaved bots.

[zimmertr]

IIRC, ByteSpider advertises itself by user agent. I don't really know what the CDN/LB looks like for this website, but you might be able to drop ingress traffic when the user agent matches Bytespider. For example, using CloudFlare WAF:

[zimmertr]

PS, ByteSpider is owned/operated by ByteDance (TikTok) and used to train AI models. Maybe soon we'll be able to ask AI where we should go hiking :P

[Tom]

I don't think it's necessarily TikTok, per this topic. https://www.webmasterworld.com/search_engine_spiders/5088284.htm In any case I blocked the user agent via .htaccess similar to what you suggested.

[zimmertr]

It works!

[Tom]

Seems like google is crawling NWH again. Not sure how extensive it will be but at least it's a fresh index. Plenty of hits for "The Angry Hiker" which is a good sign.