Yahoo Search Marketing Tools: What’s at Risk & How to Avoid Surprises

When Yahoo and Microsoft announced their Search Alliance in July 2009, only the high level agreement details were available:

• Microsoft will provide the development and management of search engine results technology (bing)
• Microsoft will provide the search and content network ad platform (adCenter)
• Microsoft will manage the relationship with self-service advertisers
• Yahoo will manage the relationship with large accounts
• Yahoo will provide their own user interface on top of the Bing results which will appear on Yahoo properties

Now that US and EU regulators have approved the deal, search marketers need to assess which Yahoo tools they rely on – and need to be prepared with alternatives should these tools be discontinued.

During the SMX West 2010 session Microsoft + Yahoo: What’s It All Mean?, I looked at the agreement’s implications for three Yahoo tools search marketing professionals have come to know and love:

• Yahoo Site Explorer, used by many SEO professionals as a source of competitive backlink data
• The Yahoo Search Marketing keyword research tool, which provides free data from a trusted source
• Yahoo Web Analytics, a well regarded alternative to Google Analytics.

In each of the cases above, Microsoft does not currently provide an adequate equivalent tool (Microsoft’s keyword research tool is off-limits to non-customers and in any case only supports English and French). While you’ll get a lot out of the following write-up, you’d be wise to consider attending a future SMX conference in person. Write-ups like these are a poor substitution for the valuable information that often emerges during the question and answers sessions, not to mention the excellent networking and, of course, food!

Competitive Backlink Analysis with Yahoo Site Explorer

Inbound links are one of the strongest ranking factors used by search engines today. Yet there aren’t many reliable sources of this data. The task of crawling all 300+ billion web pages on a regular basis is immense and very few companies are up to it. While Google provides the link: operator, seasoned SEO practitioners know it only shows a subset of the links Google knows of. Both Yahoo and Microsoft have provided competitive link data, but Microsoft discontinued this service in March 2007, leaving Yahoo as the primary source.

Figure 1: Backlink data from Yahoo’s Site Explorer

Link data from Yahoo Site Explorer – Today

Advantages

• Yahoo Site Explorer provides up to 1000 inbound links to any page indexed by Yahoo
• The underlying data is from from the Yahoo! crawling infrastructure (presumably fresh & extensive)
• Data can be exported to spreadsheet for further analysis
• A programmatic interface (API) is available (Yahoo Search Boss)
• Data can be augmented with information from other sources, e.g. Alexa (no, not the Alexa rank! See below)

Disadvantages

• 1000 links can be very limiting at times
• No link attributes are supplied, such as anchor text or nofollow; no authority information on the linking site

Scenarios for Yahoo Site Explorer after the Microsoft – Yahoo Search Alliance takes effect

There are three clear scenarios for Yahoo’s Site Explorer:

1. Yahoo could keep Site Explorer to demonstrate their continued commitment to user search, but I find this unlikely given they’re not keeping their Panama ad platform – potentially as important from an image perspective.
2. Microsoft’s Bing takes over Site Explorer. Such a move would engender much goodwill from search professionals, many of whom want to embrace Microsoft as a significant Google competitor but need concrete reasons to do so. Yet Microsoft’s track record with the SEO community is mixed, they removed support for link operators in March 2007.
3. Yahoo shuts Site Explorer down. Given Yahoo’s drive to reduce costs and Microsoft’s apparent focus on advertising, I suspect this is the most likely scenario, but I’d love to be proven wrong. During the Ask the Search Engines session at SMX West, Yahoo’s Arnab Bhattacharjee said the companies are discussing Site Explorer’s future, not really too revealing in of itself. Bing Program Manager Sasi Parthasarathy added that the partners would put users first, but it seemed like a quick rote answer (not that I blame him) and tellingly he didn’t specify which users he was referring to – web marketing professionals are a very different breed from Bing’s general user community.

Alternative sources of link intelligence data

Who’s Crawling the Web

When looking for alternative sources of link data to Yahoo Site Explorer, the first place to look is at the companies crawling the web. Consider their challenges:

• The task is immense – very few companies are up to it
  • There are somewhere between 300 bln and 1 trillion web pages to crawl
• There are multiple crawling dimensions
  • breadth – the number of sites crawled and their geographic distribution
  • depth – the percent of pages crawled per site
  • freshness – web pages come and go, sites come and go. Crawling needs to be a frequent, continuous process.
  • history – search engines will look at the history of linking to a site, in part to detect paid linking schemes. SEO professionals will also want to track link changes over time.

The quality of any link intelligence tool will be conditioned by the quality of the data source. As the old IT adage goes, garbage in, garbage out.

What the Search Engines Offer Today

• Yahoo’s Site Explorer is the most generous source of data today
• Bing & Google offer data for sites we control. Enable their respective webmaster dashboards.
• Google offers the link: operator, however, it only shows a subset of known links, misleading lots of folks not in the know
• Ask does not offer backlink data

None of the other major international search engines seem to provide backlink data.

Figure 2: Yandex (Russia)


Figure 3: Rambler (Russia)


Figure 4: Baidu (China)


Figure 5: Naver (Korea)

Majestic SEO

Majestic SEO is probably the most promising commercial source of competitive link data available today. This isn’t too surprising as Majestic SEO was started in February 2008 as an offshoot of a search engine project which began in 2004.

As of February 2010, Majestic SEO counts:

• 170 bln unique crawled pages
• 1,474 bln unique URLs (in pages crawled)
• 6.5 trillion (10^12) mapping relationships (URL pointing to URL)

Majestic notes that “most of index was crawled in last year” and that they intend to provide more information on freshness data in the future. Majestic SEO provides free basic data and advanced reporting for a company’s own domains (using an authentication procedure similar to that used by Google and Bing Webmaster Tools), allowing for a no strings attached evaluation. An API is available for programmers who want to augment link data with information from other sources. Majestic is also providing their data to various partners such as SearchDNA, so you might actually come across Majestic data in someone else’s tool. Reporting on competitive data is available by subscription.

Linkscape / Open Site Explorer

2008 also saw the start of another promising commercial source of competitive link data, Linkscape.

• Linkscape was launched 10/2008 by SEOmoz
• The current DB is smaller than MajesticSEO
  • ~43+ bln pages
  • ~440+ bln links
• The underlying tool data source, thus reliability, is not clear. SEOmoz provides a long list of potential data sources, but that doesn’t really help us in understanding the actual data quality and limitations. I believe there needs to be transparency from SEOmoz on this front.
• Open Site Explorer is a limited free interface to Linkscape data. Only 20 links can be saved without a subscription, so the use of the word “Open” appears to be a rather ill-advised marketing gimmick. A more honorable approach would be to allow free trials of Linkscape using sites a company can authenticate – just as Majestic SEO does. Convinced the tool works, SEO professionals will still need to subscribe if they want to perform competitive analysis for their customers or employer.
• A free API is available, albeit with restrictions

Alexa

No, not the Alexa rank you have in mind! Alexa is an Amazon.com division, offering internet data & statistics including competitive link intelligence information.

• Alexa’s crawling powers the Internet Archive Wayback Machine and at one point, Amazon’s own A9 search engine.
• A web interface offers simple (read: sample) reporting, 20 results per page, 5 pages maximum.
  • http://www.alexa.com/site/linksin/<domain> (Replace <domain> with the domain root, e.g. example.com)
• More useful is Alexa’s API, available for 15¢ per 1000 requests

I’ve actually included Alexa with Majestic SEO and Linkscape to highlight the excellent information Alexa provides to augment link data. The following is the XML data feed used in the Alexa Toolbar, in this case, for the New York Times. Note, for example, the domain whois record, the average page load speed (new Google ranking factor) and the DMOZ categories:

Figure 6: Alexa XML data feed for the New York Times

I trust, dear reader, that you’ll find this digression useful!

Keyword Research using Yahoo’s Search Marketing Tool

Figure 7: The Overture Keyword Tool For years savvy search marketers went to Overture‘s bare bones keyword research tool when looking for keyword data. While it didn’t offer much, it did tap real data from a significant search engine. Other famous tools get their data from dubious sources such as Infospace’s dogpile. That’s great if your target customers are dogpile users, rather unlikely. When Yahoo launched their Panama platform to replace the Overture system, they didn’t promote the fact that anyone could test Yahoo’s keyword tool just by simulating the first step of the Yahoo Search Marketing sign-up process. International SEO practitioners appreciate the keyword tool’s coverage of more than 30 countries and almost as many languages. While users of Google’s Keyword Tool or Search based Keyword Tool won’t be overwhelmed, at least the Yahoo Keyword Tool offers a sanity check on Google’s data, should you need one.

Microsoft does offer a few very interesting keyword tools in their laboratory but it isn’t clear if the tools will make it into production or be discontinued. I also fear that Microsoft will cripple the tools by using Silverlight, software limited to Windows and Mac.

Figure 8: Yahoo Search Marketing Keyword Tool, free to all by starting the Yahoo Search Marketing sign-up process

Yahoo Web Analytics: Will it suffer the same fate as Microsoft’s adCenter Analytics?

Following Google’s acquisition of Urchin (3/2005), and Microsoft’s acquisition of DeepMetrix (5/2006), Yahoo joined the Web Analytics club by purchasing IndexTools (4/2008). Yet Eric Peterson notes that Web Analytics is hard and you can be certain that running and developing a Web Analytics platform is hard too. As Yahoo works to drive cost out of its operations, it only seems logical that Yahoo Web Analytics, not part of the search alliance agreement, will come under significant internal scrutiny.

In the plus column:

• Yahoo Web Analytics (YWA) is free for Yahoo Search Marketing customers
• For Yahoo customers Yahoo Web Analytics offers an attractive alternative to the Google hegemony

But:

• Yahoo is divesting itself of costly technology and the associated infrastructure:
  • search engine technology
  • search and contextual ad platform
• What’s the business case to keep Yahoo! Web Analytics?

In response to a longer article on the future of Yahoo Web Analytics, Dennis R. Mortensen, Director of Data Insights at Yahoo! (YWA), commented

…don’t worry about YWA, what you (the industry) should worry about is: what is MS to do when not having a serious external pre- and post-click analysis and reporting tool they can give to their advertisers?

The underlying problem is that future funding for Yahoo Web Analytics will be decided at a higher corporate level, and so far, cutting seems to be the current trend. In the event Yahoo does decide to exit the business of being a Web Analytics supplier, Microsoft is obvious acquisition candidate for Yahoo! Web Analytics:

• Microsoft doesn’t currently offer a Web Analytics platform
  • many advertisers now turn to Google for a Web Analytics tool – offering a complete 360º solution would provide Microsoft great credibility and mind-share
• A generally available free Web Analytics tool would be excellent as
  • a source of web intelligence for Microsoft to improve Bing’s organic search algorithms. If Google has a data driven culture, you can be sure they look at Google Analytics data, at least when developing algorithms, if not in determining the rankings themselves (Google’s Matt Cutts has said (at 27:40) they don’t currently use Google Analytics as a ranking signal; this is of course subject to change).
  • PR. While many are wishing Microsoft well, Microsoft does have a legacy reputation akin to a school yard bully due to their history of targeting market leaders with clone like applications. Just ask Netscape about Internet Explorer, Quicken about MS Money (now abandoned), Adobe’s PDF about XPS, not to mention the would be Flash killer, Silverlight. Rather than innovation, there seems to be too much flattery going on. Hopefully Microsoft will use search to turn over a new leaf and strut their innovation stuff. Taking over Yahoo Web Analytics and allowing anyone on the web to use it would be a way for Microsoft to send out a clear signal that it is learning to out-Google Google in the soft PR department. If Microsoft adds competitive link data and a set of free keyword research tools to the mix, many hearts and minds might be won over to the idea that Microsoft is finally getting the web. I’m not holding my breath, but I’d love to be surprised.

Speaking against a Microsoft acquisition of Yahoo Web Analytics:

• Microsoft has its prior adCenter Analytics (LiveMetrics) experience. adCenter Analytics was shut down in December 2009. Presumably the underlying code base was too old to justify the investments needed to make it competitive with similar tools.
• Microsoft suffers from a well documented Internet attention deficit disorder. Bill Gates acknowledged missing the boat in the mid nineties yet once the browser wars were won, Microsoft waited 5 years before releasing IE 7. Even now, when Microsoft has to choose between web standards and its proprietary world, it chooses to break the web, as seen in CSS support regression in MS Outlook. Microsoft is currently pushing its Flash clone, Silverlight, which unfortunately doesn’t work on Linux despite the vaunted partnership with Novell. Want to see the next version of Bing Maps? Forget it, you’re out of luck.

Figure 9: Can’t install Silverlight, can’t view new Bing maps. Microsoft shoots itself in the foot.

Users of Yahoo Web Analytics might want to prepare for the worst by:

• installing additional tracking code for an alternative Web Analytics system
• backing up historic data
• saving reports / exporting report data

Microsoft – Yahoo Search Alliance Timeline

The search alliance partners have said they will begin implementing the agreement in the US at the end of 2010 or the beginning of 2011. Implementation in the other 30 or so markets covered by Yahoo Search Marketing should happen by the beginning of 2012.

View my SMX West 2010 presentation. If you like it, you might find my SEO course right for you. Feel free to contact me for more information.

Similar Posts:

• And then there were two: Microsoft – Yahoo! Search Deal at SMX West 2010
• Yahoo Web Analytics (ex IndexTools) soon in no man’s land?
• Eying Search Engine Market Share in the era of Bing
• Microsoft / Yahoo Search Alliance: a closer look at Bing Webmaster Tools & a free data export tool
• 7 sources of link intelligence data and key link analysis considerations

Registration is now open for the next SEO Course (March 22 and 23) and Google Analytics Course (March 14 and 15) in Milan. Don’t miss the opportunity!