Forum Index > Support & Feedback > can't search nwhikers with google anymore
 Reply to topic
Previous :: Next Topic
Author Message
neek
Member
Member


Joined: 12 Sep 2011
Posts: 2317 | TRs | Pics
Location: Seattle, WA
neek
Member
PostSun Jan 07, 2024 9:28 am 
This is weird, I used to add "site:nwhikers.net" to my google searches all the time to search this site exclusively, but it no longer returns results (or sometimes just one). Did we get blacklisted? I'll try again later, maybe google is just having a moment.

Back to top Reply to topic Reply with quote Send private message
dave allyn
Member
Member


Joined: 05 Apr 2011
Posts: 420 | TRs | Pics
dave allyn
Member
PostSun Jan 07, 2024 9:39 am 
Try it with The Angry Hiker

Back to top Reply to topic Reply with quote Send private message
neek
Member
Member


Joined: 12 Sep 2011
Posts: 2317 | TRs | Pics
Location: Seattle, WA
neek
Member
PostSun Jan 07, 2024 10:29 am 
Try it with whatever you like. Are you getting different results? For me even w/o cookies it's just crickets.

Back to top Reply to topic Reply with quote Send private message
seawallrunner
dilettante



Joined: 27 Apr 2005
Posts: 3303 | TRs | Pics
Location: Lotusland
seawallrunner
dilettante
PostSun Jan 07, 2024 10:49 am 
Same. A few weeks ago I tried searching for previous posts with REI in the title or body, and the search returned empty.

Back to top Reply to topic Reply with quote Send private message
Tom
Admin



Joined: 15 Dec 2001
Posts: 17821 | TRs | Pics
Tom
Admin
PostSun Jan 07, 2024 1:30 pm 
This summer MCaver needed to add some stuff to block a rogue bot that was hammering the site. I'm guessing it might have led to unintended side effects. Nothing else has changed.

Back to top Reply to topic Reply with quote Send private message
huron
Member
Member


Joined: 13 Sep 2004
Posts: 1022 | TRs | Pics
huron
Member
PostSun Jan 07, 2024 4:05 pm 
Works from duckduckgo

Back to top Reply to topic Reply with quote Send private message
zimmertr
TJ Zimmerman



Joined: 24 Jun 2018
Posts: 1193 | TRs | Pics
Location: Issaquah
zimmertr
TJ Zimmerman
PostSun Jan 07, 2024 5:51 pm 
Back to top Reply to topic Reply with quote Send private message
Tom
Admin



Joined: 15 Dec 2001
Posts: 17821 | TRs | Pics
Tom
Admin
PostMon Jan 08, 2024 1:45 pm 
Seems as if MCaver unintentionally blocked all bots, I've tweaked the robots.txt to allow all bots and block rogue bots via .htaccess instead. Hopefully the google indexing comes back. FWIW, this is how bad that rogue bot was hammering the site this summer, caused several hundred dollars of bandwidth overage charges. If you google Bytespider you'll see it continues to hammer other sites with webmasters trying to figure out how to block it. Apparently it doesn't honor robots.txt so maybe we just got lucky that it stopped hammering us, at the cost of putting in something that only served to block well behaved bots. doh.gif

Back to top Reply to topic Reply with quote Send private message
zimmertr
TJ Zimmerman



Joined: 24 Jun 2018
Posts: 1193 | TRs | Pics
Location: Issaquah
zimmertr
TJ Zimmerman
PostMon Jan 08, 2024 2:00 pm 
IIRC, ByteSpider advertises itself by user agent. I don't really know what the CDN/LB looks like for this website, but you might be able to drop ingress traffic when the user agent matches Bytespider. For example, using CloudFlare WAF:

Back to top Reply to topic Reply with quote Send private message
zimmertr
TJ Zimmerman



Joined: 24 Jun 2018
Posts: 1193 | TRs | Pics
Location: Issaquah
zimmertr
TJ Zimmerman
PostMon Jan 08, 2024 2:02 pm 
PS, ByteSpider is owned/operated by ByteDance (TikTok) and used to train AI models. Maybe soon we'll be able to ask AI where we should go hiking :P

Back to top Reply to topic Reply with quote Send private message
Tom
Admin



Joined: 15 Dec 2001
Posts: 17821 | TRs | Pics
Tom
Admin
PostMon Jan 08, 2024 2:14 pm 
I don't think it's necessarily TikTok, per this topic. https://www.webmasterworld.com/search_engine_spiders/5088284.htm In any case I blocked the user agent via .htaccess similar to what you suggested.

zimmertr
Back to top Reply to topic Reply with quote Send private message
zimmertr
TJ Zimmerman



Joined: 24 Jun 2018
Posts: 1193 | TRs | Pics
Location: Issaquah
zimmertr
TJ Zimmerman
PostMon Jan 08, 2024 2:36 pm 
It works! smile.gif

Flickr | Strava

neek
Back to top Reply to topic Reply with quote Send private message
Tom
Admin



Joined: 15 Dec 2001
Posts: 17821 | TRs | Pics
Tom
Admin
PostMon Jan 29, 2024 9:52 pm 
Seems like google is crawling NWH again. Not sure how extensive it will be but at least it's a fresh index. Plenty of hits for "The Angry Hiker" which is a good sign.

Back to top Reply to topic Reply with quote Send private message
   All times are GMT - 8 Hours
 Reply to topic
Forum Index > Support & Feedback > can't search nwhikers with google anymore
  Happy Birthday Captain Trips, Backcountry Blake, Flower Sniffer, nwcjc, backpacker92out!
Jump to:   
Search this topic:

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum