Blog
By Jon Henshaw
Fabrice Canel, part of the Live Search Crawling Team, announced on Tuesday (February 12, 2008) significant updates to the Live Search Engine crawler, MSNBot. The updates significantly improve the efficiency with which they crawl and index websites. The two biggest updates were improvements with HTTP Compression and Conditional Get.
We support conditional get as defined by RFC 2616 (Section 14.25), generally we will not download the page unless it has changed since the last time we crawled it. As per the standard, our crawler will include the "If-Modified-Since" header & time of last download in the GET request and when available, our crawler will include the "If-None-Match" header and the ETag value in the GET request. If the content hasn't changed the web server will respond with a 304 HTTP response.
They also updated their user agent.
In addition to these two features there are many more improvements in performance that should help further optimize our crawling. As a result, we've also upgraded our user agent to reflect the changes, it is now "msnbot/1.1". If you think you are experiencing any issues with MSNbot, or have any questions about the updates, please use our Crawler Feedback & Discussion form.
When Nathan Buggia, Lead Program Manager at Live Search Webmaster Center, first brought this news to my attention, one of the first questions I asked him was regarding the respect (or disrespect) of robots.txt. Some people have the impression that MSNBot doesn't respect robots.txt, because they often see their content in their index when they've specifically requested that it not be crawled. Nathan replied with:
We do read and respect the robots.txt file, however, if there is a link on a 3rd party site, that points to a page blocked by the REP on your site, we may still put that link (& associated anchor text) into our index. And we may surface that link (and anchor text) in our search results if it appears to be relevant, but we still won't go and crawl/index the actual page.
This is something that we spend a lot of time debating about internally, I would love to hear your thoughts on this.
I thought that was interesting, and the fact that they occasionally include links in their SERPs to pages they don't actually crawl, might be the reason for all of the confusion. For me, it's a foreign concept that a search engine result would contain links to websites that they haven't even crawled. However, I can see the case for that if many trusted websites keep referencing a resource and it's determined that the destination URL is an appropriate search result — regardless of whether or not the destination content has been crawled or analyzed.
By Jon Henshaw
Microsoft has a history of either arriving too early or too late to the party. In the case of search engines, they were too late. However, Microsoft has a not-so-secret weapon — money! Regardless of when they get to the party, they often times simply end up buying it, including the crappy band playing at the party.
Such is the case with Microsoft's new unsolicited bid to buy Yahoo!. Microsoft is offering $44.6 billion in cash and stock for search engine operator Yahoo Inc. in a move to boost its competitive edge in the online services market. This offer comes on the coattails of Yahoo!'s declining stock and recent announcement of 1,000 layoffs. Danny Sullivan has more details about the Microsoft offer on Search Engine Land.
It would seem that for many people, Microsoft's offer to buy Yahoo! is not a welcome thing. They see it as a threat to something good, namely Yahoo!'s services and brand. However, I see things in a very different light. I see it as a chance for Microsoft — regardless of how much I don't care for them as a company — to get things right. If they can acquire Yahoo! and take a similar approach that Yahoo! has taken with their own acquisitions — like maintaining the branding of MyBlogLog, Flickr and del.icio.us — they have an opportunity to redefine themselves and their own image.
An infusion of cash from Microsoft and new synergies between each company's Web properties would also provide the best challenge to Google's domination. Microsoft and Yahoo! are two very smart and strong companies, so if they can integrate their resources well and fill in each other's gaps, then they may actually have a fighting chance against Google's lead in the market.
Oh, but what will happen to Live? I think "Live" will live on. It may become it's own marginalized service or you might see a hybridization of Yahoo!'s and Microsoft's Web applications into something called "Yahoo! Live." Regardless, I doubt that we've seen the last of of Live, and it may be that infusing it with Yahoo!'s brand, instead of Microsoft's, might be the kick-start it so desperately needs.
[1]
By Jon Henshaw
Many people tend to give a negative stigma to anything that Microsoft works on. Because of Microsoft's PC dominance and business practices, just about everything that comes out of Redmond is reviewed with a fine tooth comb and equally criticized. In fact, the general consensus seems to be that anything Microsoft creates — especially products and services for the Internet — have an uphill battle to fight. Live Search is no exception.
The overall perception among the SEO community has been that Microsoft hasn't taken their search engine strategy seriously enough and that it's traditionally been a search engine that's easily influenced by search engine optimizers. However, that sentiment appears to be changing. It appears that Microsoft is taking a much more serious and agile approach towards Live Search.
I had the privilege of hanging out with Derrick Wheeler while waiting in line to get my conference badge at PubCon Las Vegas 2007. Derrick was recently hired on by Microsoft as their Senior SEO Architect for their Central Marketing Group. He's been tasked with standardizing and optimizing all of Microsoft's Web properties — no easy task mind you. I also met Jeremiah Andrick at the Live Search vendor booth. Jeremiah is a Program Manager that works closely with the Live Search team. In my brief conversation with him, he described the Live Search team as a semi-autonomous small group of intelligent engineers. Of particular interest was his description of the team being somewhat cut-off from the rest of Redmond, allowing them to be more innovative, reactionary and agile.
I have no reason to doubt Jeremiah, especially after we saw how quickly they were able to respond to the Google AdSense bug we first reported on. Nathan Buggia, the Lead Program Manager for the Live Search Webmaster Center, quickly jumped on the problem and fixed it very quickly. He also did a good job of transparently communicating about the problem and what they did about it.
With all of that being said, I think Live Search deserves more attention than it's previously received — especially attention towards the innovations that they're currently developing right now. A few weeks ago, buried in the comments of an SEOmoz blog entry, Nathan highlighted four of his favorite new features of Live Search that I think are worth taking a closer look at.
Traffic Info, Directions, Maps, Gas Prices, Movie times and Local Search. Access Live's local search and mapping features from a native app on your smartphone. And if you have excess cash, you can connect it to a GPS receiver and it just works. They also added voice recognition and while it isn't perfect yet, it has saved Nathan from crashing his car while trying to look up directions. He's only tried this on his WM Dash, but there's also has a native client for the blackberry.
This technology scans videos and automatically builds a "highlights" reel for the video. Hovering over the video in the results plays the highlights, and then you can click on it to see the full video. They crawl YouTube and thousands of other sites.
If you host an event and want to include general directions for everyone, you can grab one of these maps which provide auto-gens directions to your location from major arterial routes north, south, east and west.
If traffic is no good where you live, then you may also find this useful. (Notice that the map is interactive)
[1]
By Jon Henshaw
Update: Nathan Buggia, Lead Program Manager at Live Search, contacted me to let me know they've fixed the problem. Here's what happened, in their own words.
Live Search is offering a new way for AdWords customers to get click-throughs — their own natural SERPs. They appear to be indexing AdWords ads from Google's own SERPs. Doing a search on Live for colorations paint (at the time of this writing) will give you AdWords ads in the third and fourth natural result. I can only assume this is unintentional and is simply a bug in their crawler. Regardless, it's a little embarrassing.
[7]
By Jon Henshaw
What's the least likely place you would go to find something? Your old girlfriend's apartment? A landfill? Utah? Actually, it's not any of those. The least likely place you would go is obviously page 87 of Live Search. Which of course is why you should go there.
So let's get right to it. I will choose a random keyword phrase, and by random, I mean the first thing that pops into my head. pink earmuffs. Oh wow, that's embarrassing, but search I must.
Huh, I found a website called Feministing. So what's this feministing all about?
Young women are rarely given the opportunity to speak on their own behalf on issues that affect their lives and futures. Feministing provides a platform for us to comment, analyze and influence.
The site is also about this very attractive woman selling this t-shirt — a t-shirt that appears to be giving me the middle finger.
Okay, what's next? Thinking, thinking, thinking... How about purple dinosaur poop! This should be good.
First off, I can't believe there are 87+ search results for purple dinosaur poop, but I digress and must reveal my findings. I found Library Thing Beta. what is Library Thing you ask? Well, let me tell you:
LibraryThing is an online service to help people catalog their books easily. You can access your catalog from anywhere — even on your mobile phone. Because everyone catalogs together, LibraryThing also connects people with the same books, comes up with suggestions for what to read next, and so forth.
Apparently, Live found this without the use of poop, because there was no poop to be found on this page.
One more and then we'll call it an entry. I'm going to think back to a distant memory from when I was a kid. Lizard head in ice cream. Don't ask...
An interesting result — Gumby and Pokey in the 80s Cartoons. I guess that result makes about enough sense to me as it did having a lizard head my ice cream when I was in fifth grade. The page is a giant collection of Gumby stories, I think.
Decades after the original shorts were created, Art Clokey started this updated take on Gumby. Gumby, Pokey, Prickle, and Goo get a farm and their own band. Gumby gets a little sister and a grandmother. More and more of the important characters actually look like humans. The Blockheads appear more frequently, giving them official recurring villain status. They don't quite have the outrageously bizzarre appeal of the earlier cartoons, but it's still Gumby at heart. Let's watch.
And by "Let's watch," the author must mean let's read hundreds of short story plots of Gumby. If you're brave enough to read through all of them, your prize is the link at the bottom of the page to buy Gumby swag.
Features 
