Google Sitemap Standard Adopted by Leading International Search Engines

Each of the three major search engines (Google, Yahoo! and Microsoft’s Live Search, now Bing) have announced joint support of Google’s sitemaps protocol.

2007-04-11: Ask announces support of the sitemap standard. It is not yet clear if they actually use sitemaps. While Google and Yahoo do process sitemaps, Microsoft does not yet use them.

2008-06-02: Yandex supports xml sitemaps. China’s Baidu also supports sitemaps through their Baidu webmaster tools which is currently (2011) invite only. For those interenested in the Czech Republic, Seznam supports sitemaps.

A new site, www.sitemaps.org, has been created to support the sitemaps protocol. While the Yahoo blog indicates Yahoo is apparently already accepting submissions, there is not mention of this on their Site Explorer submission form. Microsoft is committed to supporting sitemaps after finishing internal testing which is currently underway.

We heartily commend each of the search engines for this constructive collaboration. Large companies often resist industry collaboration – the “it wasn’t invented here” syndrome. Collaboration on sitemaps is the second joint effort this year – each of the major engines now supports the noodp meta tag introduced by Microsoft’s Live Search, now Bing, to exclude open directory / dmoz titles from search results. Google signed-up in July, Yahoo! committed in October. Unfortunately, Ask hasn’t yet pronounced on either initiative.

Keep in mind that while a sitemap can help search engines find otherwise hidden content, they shouldn’t really be used for this purpose – the underlying site architecture issues should be addressed directly.

The true value of a sitemap is to pro-actively notify search engines of new and updated content. This is a win-win for both parties – search engines can better focus their crawling resources and site owners will (hopefully) see their content updated in the engines sooner. Google’s original sitemap format has many advantages over other attempts, such as Yahoo!’s, which we discussed in our article Yahoo! Sitemap Feed Submission… worth the Effort?.

2007-04-11: The sitemap protocol has been extended to support automatic detection by search engines via a robots.txt entry:

Sitemap: <sitemap_location>

We suggest that you avoid using this approach as it facilitates content theft.

Ping search engine crawlers with an updated Sitemap

You should notify each search engine by using a URL encoded ping syntax:

• Ping Google: http://www.google.com/webmasters/sitemaps/ping?sitemap=http%3A%2F%2Fwww.yoursite.com%2Fsitemap.gz
• Ping Yahoo: http://search.yahooapis.com/SiteExplorerService/V1/ping?sitemap=http%3A%2F%2Fwww.yoursite.com%2Fsitemap.gz (Will be shutdown 12/2010 as Yahoo has outsourced Search to Microsoft, the so-called search alliance)
• Ping Bing: http://www.bing.com/webmaster/ping.aspx?siteMap=http%3A%2F%2Fwww.yoursite.com%2Fsitemap.gz
• Ping Ask.com: http://submissions.ask.com/ping?sitemap=http%3A%2F%2Fwww.yoursite.com%2Fsitemap.gz (As of Nov 2010, Ask has outsourced most of its search crawling, most likely to Google.)
• Baidu sitemap ping – NA
• Seznam sitemap ping – NA
• Yandex sitemap ping – NA

Similar Posts:

• 6 methods to control what and how your content appears in search engines
• The Google Webmaster Dashboard, a.k.a. Google Sitemaps
• Now there are 6 ways to keep website content out of search engines
• SEO Session: Up Close With Google Blog Search
• Google Zeitgeist has a Sibling, Google Trends

Registration is now open for the next SEO Course (May 14 and 15) and Google Analytics Course (May 9 and 10) in Milan. Don’t miss the opportunity!