« previous: How To Fix a Broken Mac | next: Mac OS X Network Problem: Slow Remote Login (SSH): Fixed »

Live Search Referrer Spam

# Filed on Dec 3, 2007 by AnthonyDiSante / 2 replies

Microsoft’s Live Search is really starting to irritate me.

As this log snippet from VisitorLog shows, I get about 30 separate hits per day from hosts named livebot-65-55-*-*.search.live.com.  The vast majority of them are bots, not real humans, as evidenced by the fact that they have no screen resolution (and therefore no screen), which while not a guarantee of botness, is a pretty strong sign of it, especially when combined with other bot-like characteristics such as having "livebot" in the hostname.

So far this is all OK.  However, the bot’s USER_AGENT string is set to IE7/Win2003, which is bogus [the full string is: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.2; .NET CLR 1.1.4322)].  It’s clearly a bot, and possibly a spider, so it’s not a real IE7 browser; it should identify itself with an accurate user-agent string like a responsible internet citizen.

And what’s worse is that, unlike most bots/spiders, this one actually has a non-null HTTP_REFERER string: it claims to have come from Microsoft’s Live Search engine, searching for extremely generic single terms like "files" for which my site isn’t even in the first 10 pages of search results.  The only logical conclusion I see here is that Microsoft is doing some serious referer spamming to get hits back to its Live Search pages.

This has been going on since November 21st, but for months before that, the exact same thing had been happening except that the hostname was bl2sch1082113.phx.gbl (or similar) instead of livebot-65-55-*-*.search.live.com.

Now don’t get me wrong: spiders are good.  I like spiders crawling my sites, and I’d really like for Google to have some strong competition in the search space.  But faking your user-agent and spoofing the referer field with bogus data aren’t good practices for a search engine.  Someone please tell me there is a valid explanation for this.

Comments:

01. Jun 26, 2008 at 01:50am by AnthonyDiSante:

Six months later and Microsoft is still using this scummy technique.  Now they’re also using hostnames of the form msnbot-65-55-211-113.search.msn.com.

02. Jul 7, 2008 at 02:13pm by Malik:

Yeah from 2 months they are doing this in every day...
They are using lot of generic keywords about your site.

Reply to this message here:

Your name
Email (why?)
Website (if you have one)
Subject
search posts:

[ home | archives ]

Client Quotes

I just installed the demo of your product and got it up and running in no time.  I searched high and low for a decent login script and thank God I found yours.
– Adrian F.
I spent ages trying to find a way of making my own log in page for my website - if you're thinking of doing that forget it - don't waste your time!  UserBase is a 1st class product at a very reasonable price.  The software works faultlessly and can be adapted to any situation.  The service that I have received from Encodable is terrific!  I am very very impressed.  Nothing was too much trouble and I am most grateful to Anthony DiSante in particular for all his help and patience.
– Paul S.
Worked like a charm... man, this piece of software is a dream and I really appreciate all your customer service help getting this taken care of.
– Kyle M.
I just want to say you guys really stand alone in that you have a quality product and you provide genuine customer service.  It's sad but those qualities are seldom found separately, much less together.  Thanks again for your time and help.
– Alex S.
Also, I wanted to tell you that I was very skeptical about buying this script.  I've spent a lot of time and money over the past 3 months trying to find a solution that works, but I ended up having problems with so many of the scripts I tried that I was almost to the point of giving up.  But then I came across your script, and it actually does what it's supposed to.  An absolute wow.  A very impressive and powerful script indeed!  Many, many thanks!
– Mike E.
I can't thank you enough, I was up against a deadline that required me to get this up and running in 48 hours and you have probably the best customer service I've ever seen.
– Dan T.
Your scripts/software are the greatest, I mean I really love how customizable they are, how intuitive they are, and so on.  Thanks again, I love this stuff!
– Tucker O.
We searched for a long time for an application to password protect directories and allow file uploads.  Userbase & Filechucker are far superior to anything out there.  Simple yet powerful programming, extremely flexible in configuration, and great customer service.  Thanks for a superb product.
– Kat G.
Thank you VERY much for all of your help.  You've really impressed me.  We have support agreements for other software that costs thousands of dollars / year (just for the support), and most of them aren't as helpful as you have been.
– Keith Y.
There are a lot of these scripts out there, but I think they all pale in comparison to yours.
– Peter W.
The software has some great features, is well presented, runs where others are problematic and will make a good impression on our clients.  We look forward to reaping its benefits!
– Alex H.