Bing Bot Server Abuse - Problem

S

sermonindex Aug 20, 2013

Saints,

We believe we have found a possible reason for the site loading slow a for the last few days at points. There seems to be over 14,000 requests from bing.com crawling bot which is unhealthy and is hard on the server to accept all of these requests so fast ontop of all the other traffic. Here is an example of this:

msnbot-65-55-24-217.search.msn.com - - [20/Aug/2013:00:20:38 -0700] "GET /modules/newbb/viewtopic.php?topic_id=47359&forum=21 HTTP/1.1" 200 4448 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
crawl-66-249-72-205.googlebot.com - - [20/Aug/2013:00:19:37 -0700] "GET /modules/myalbum/photo.php?lid=1230 HTTP/1.1" 200 3451 "-" "Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)"
msnbot-65-55-24-217.search.msn.com - - [20/Aug/2013:00:20:38 -0700] "GET /modules/myvideo/viewcat.php?cid=12 HTTP/1.1" 200 5076 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
d154-5-233-80.bchsia.telus.net - - [20/Aug/2013:00:20:38 -0700] "GET /modules/myvideo/viewcat.php?cid=72&min=30&orderby=titleA&show=15 HTTP/1.1" 200 4675 "https://www.sermonindex.net/modules/myvideo/viewcat.php?cid=72&min=15&orderby=titleA&show=15" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:23.0) Gecko/20100101 Firefox/23.0"
msnbot-157-55-33-79.search.msn.com - - [20/Aug/2013:00:20:38 -0700] "GET /modules/articles/index.php?view=article&aid=27618 HTTP/1.1" 200 4204 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"

Thank you for praying for the site to be preserved from evil or disruptions that as many sermons as possible can go out to the glory of God!

S

sermonindex Aug 22, 2013

We decided to block the Bing.com search engine from accessing the website. They accounted for 4% of the traffic but their Spider Bot that got links from SermonIndex would crash the site because of over-crawling.

Bing is owned by Microsoft and after talking with their technical support they were not willing to work with us to help remedy the situation so we have to manually block the IP addresses of the MSNbot and Bing bot. It is a shame companies like microsoft to try and make their search better do not care if they damage or hurt other smaller websites in the process.

C

ccchhhrrriiisss Aug 22, 2013

Thanks for the update and all of the work, Greg.

I am surprised that the Bing spiders were causing such interruption. I've worked on some websites where we had spider bots from Bing but didn't really have any issues.

The 4% figure may not seem like a big deal, but it can mean quite a bit to one of those individuals trying to access the website or hear the messages included here.

There is a method to report "overcrawling" from Bing bots:

http://www.bing.com/webmaster/help/how-to-report-an-issue-with-bingbot-25c19802

In addition, there are some helpful tips here (including an email address that works with this specific problem):

http://www.bing.com/blogs/webmaster/f/12248/p/653349/9572519.aspx

I will continue to remember this in my prayers.