With all the search engine giants like Google and Yahoo updating their content crawling method, MSN also joins the party. Live search has made many updates in the crawler to improve the efficiency of its ability to crawl and index your websites. The two main updates include ‘HTTP compression’ and ‘Conditional get’HTTP compression:According to Fabrice Canel, of live search crawling team, “HTTP compression allows faster transmission time by compressing static files and application responses, reducing network load between your servers and our crawler”. The HTTP compression is now supported by most of the browsers and popular search engines like Google.Conditional Get:As stated in official Live Search blog, “Generally we will not download the page unless it has changed since the last time we crawled it. As per the standard, our crawler will include the "If-Modified-Since" header & time of last download in the GET request and when available, our crawler will include the "If-None-Match" header and the ETag value in the GET request. If the content hasn't changed the web server will respond with a 304 HTTP response.” Configuring the Conditional get in your website is simple and is well explained by Fabrice. Checking whether your website already supports the "If-Modified-Since" is made simple with the help of online tools. You can check it using Fiddler for Internet Explorer, or Live Headers for Firefox. Each of these tools allow creating a custom GET request and send it to your server. You'll want to make sure that your request includes the "If-Modified-Since" header. Fabrice has given a sample “If modified-Since” header: “GET /sa/3_12_0_163076/webmaster/webmaster_layout.css HTTP/1.1 Host: webmaster.live.com If-Modified-Since: Tue, 22 Jan 2008 01:28:49 GMTYou should receive a server response similar to the following simplified sample: "HTTP/1.x 304 Not Modified”“If you have not yet configured conditional get on your site, we would strongly encourage you to do so, as it can significantly help reduce server load as most browsers and crawlers already support this feature” says Fabrice Canel.Live search has also upgraded the user agent to "msnbot/1.1" to optimize the crawling. According to official MSN space, there are many more updates that are on the way.
The Author is a Search Engine Optimization researcher with more than 5 years of expertise in the ....