Google AJAX Search results, tracking in Google Analytics and, um, an API rant

As many may know by now, Google has been experimenting for a few months with Ajax (JavaScript) based search results. One problem with the initial trial was that no referrer information was passed when a user clicked on a search result, “breaking” the historic ability of Web Analytics systems to track search traffic from Google. Google has more than one service on each of it’s domains which may send traffic to a website, such as the Google Reader, so just knowing traffic is from Google isn’t so informative.

Keyword information from search referrers is in particular very important as we want to know not only where our visitors came from, but what was their intent, intent indicated though the keywords they use to express their need or desire while searching.

Keyword click position tracking in Google Analytics

After a a bit of outcry, Google found a work-around, which results in a new URL referrer from the AJAX search results. One really nice enhancement (bravo Google!) is that Google is also passing the position of the search result the user clicked on, through the use of a URL parameter cd=. Previously it was only possible to track the page number of the search result. Several people have already documented methods to use Google Analytics to track result position in Google search results for a given keyword. I would strongly suggest making your tests in a copy of your main Google Analytics profile to which you add new filters. I also suggest implementing the two filter approach which only considers organic traffic.

Even if you apply new logic to Google Analytics, going forward not all keywords will have position information. Not all searches from Google use the new referrer syntax. Nor have any of the other main search engines yet announced support for this tracking parameter, though I hope they will. Also note that each keyword or keyword phrase will be repeated for every position it occupies in search results, e.g.

awstats
awstats (4)
awstats (7)

Google Analytics has a nice on the fly filtering ability. Once you’ve set up the filter logic to add Google’s ranking information to your keywords and keyword phrases, you can enter the two characters \( to view just keywords with ranking information. The ( is the character we want to search for, the backslash is needed to “escape” the parenthesis which, by itself, would be interpreted differently.

I suspect that Google may offer a more elegant integration of this tracking in GA in the future, perhaps by presenting the average search position for a keyword along with range extremes for the reporting period. But maybe not, we’ll see. It will also be interesting to see how the big boys, Omniture, Coremetrics, and the direct competitors like Yahoo Web Analytics (ex IndexTools) respond.

Why the new AJAX Search Results?

Google Engineer Matt Cutts says the move from basic html SERPs (search engine result pages) to JavaScript / Ajax search results is to improve the speed of search results rendering. I must admit, I’m not too excited by AJAX and suspect there are other issues involved as well.

Google search URL parameters, going, going, gone?

First, I fear that URL parameters which allow power users to easily refine their queries, e.g. pws=0, gl=US, etc, will disappear. This seems to be already underway as evidenced with Google’s Adwords preview tool. When originally launched, Google documented URL parameters which allowed a power user to simulate their geographic location. But currently there’s no mention of these parameters in the relevant documentation. If you’re not in the know, you need to do your best with the Ajax interface. Worse, Google could decide to deprecate or obsolete the URL parameters. I do hope I’m very wrong on these points….

Crawling, or scraping, if you will, Google search results, going, going, gone?

I imagine that another goal, unstated, is to complicate the current crawling / scraping of Google SERPs for SEO purposes – many programming tools which retrieve web pages are not JavaScript capable. This would be fine if Google allowed reasonable API access to search results for standalone programs.

A Google search API rant

Despite being one of the world’s largest consumers of other people’s content, Google does not seem to currently sanction programmatic access to their/our data by stand-alone programs. There is the old Google Soap API but Google inexplicably closed access to new accounts in 2006. Google made a new API available, implying it was the Soap API replacement, but it appears that its terms of service exclude use by standalone programs (ironically, you need to send a referring URL, which of course doesn’t really exist in a standalone application), at least if the API usage is SEO oriented. As I’m not a lawyer, I hope I’ve got it all wrong and someone authoritative will set me straight.

Unfortunately, this seems to be one instance in which an otherwise great company is caught in a bit of hypocrisy.

Why is it that Google is so magnanimous in allowing users to export data from Gmail, Google Apps, AdWords, but not from Google Web Search? The official answer would be of the type “automated programs put too much demand on our servers“. While this may even be true, there are simple ways to solve this problem. Place reasonable daily or weekly usage limits. Charge reasonable fees for “excessive” usage. Yahoo! is able to manage the problem and so is Microsoft. If they can solve the problem, I would think the search leader could too, but I digress….

Similar Posts:

• Enhance your blog or website with Google’s Site Search. Measure the results in Google Analytics.
• Digital media measurement: campaign tracking made easy + free tools
• Decrypting Ask’s Web Search API
• Google Analytics Event Tracking – New Option to Avoid Impacting Bounce Rate
• Tracking Search Engine Cache Page Views with Web Analytics

Registration is now open for the next SEO Course (March 22 and 23) and Google Analytics Course (March 14 and 15) in Milan. Don’t miss the opportunity!