Improve Search Engine Recognition in Google Universal Analytics


Google Analytics LogoSocial media may be all the rage, but search engines are arguably still among the best ways to attract highly qualified visitors to a website, visitors who are explicitly looking for what we’ve got to offer.

For organizations using Google’s Universal Analytics, Analytics automatically reports on merit-based™ organic search traffic from over 40 leading international and regional search engines, including Google, Bing, Yandex, Baidu and Naver. Visits from search engines recognized by Google Analytics appear in the Organic Search Traffic report (Standard Reports → Traffic Sources → Sources → Search → Organic).

Why customize Universal Analytics search engine recognition?

A few search engines are missing from the default set, particularly those that primarily serve a national market. Local ISPs which provide someone else’s re-branded search are also a source of search based traffic. Users are often involuntarily diverted to ISP branded search engines when they try to navigate to an nonexistent domain. As a side note, this can be avoided by using Google’s public DNS. Browser toolbars, often installed without a user’s knowledge, usually provide a search function. Users tend to use the toolbars, given their prominence in a browser, at least until they figure out how to remove them. A few people use meta search engines, like dogpile, which Google Analytics does not recognize by default.

Clean up the inbound link referral traffic report

links
By Stomchak (Own work) [Public domain], via Wikimedia Commons

Search traffic from ISPs and toolbars will end up in the Referral Traffic report (Standard Reports → Traffic Sources → Sources → Referrals) if Universal Analytics is unable to detect them. Ideally the referral report should just report on inbound links, an essential element of SEO success. By moving search driven visits to the Organic Search Traffic report where they belong, a cleaner picture of inbound link traffic ensues.

Get valuable keyword data!

Searcher keywords are a marketing treasure trove: they tell us explicitly what a person was thinking when they arrived on a site. Search keywords are marketing gold, pure gold! Search engines have traditionally supplied the search query phrase as a parameter in the referral URL which is passed in http headers to the site the user visits. Like many things in life, this golden age wasn’t to last.

words, words
By Carol Hartmann (Wordle Tag Cloud) [CC0], via Wikimedia Commons

In a move characterized as making search more secure, Google decided to stop supplying searcher keywords to the target sites. Such traffic still appears in the organic search report, but the search keyword phrase which triggered the visit appears with the “keywords” (not provided). Initially Google limited the number of searches impacted, but over time has applied the change to most organic search traffic from Google.

The most plausible reason for the change was a lawsuit – which also presented Google with an opportunity. In a bout of pure hypocrisy, Google continues to provide search keywords if the searcher clicks on an ad – so clearly the change had nothing to do with “security”, which was a linguistic abuse in the first place, the change is really about privacy, a very different concept. AdWords now provides something organic search doesn’t: user intent expressed by the search phrase. There have been numerous blog posts which tried to minimize the impact of the missing keywords or purported to offer alternative solutions. Balderdash. You cannot invent data which isn’t there. This change is a massive loss to web site owners. We also know the damage extends beyond the marketing community.

What organizations can do is capture keywords from searches performed using ISP branded search and browser toolbars. In most cases this traffic will be minimal, but the keyword data supplied would otherwise be lost and thus is still better than nothing.

Universal Analytics users should also enable Google’s Webmaster Tools data sharing in Analytics. It will then be possible to see the number of clicks on a selection of keyword phrases for a limited time period, currently the last 3 months (Standard Reports → Traffic Sources → Search Engine Optimization → Queries). To be clear, Google Webmaster Tools reporting is only a partial solution, the data is not really integrated into Universal Analytics; this is clear if you try to customize the report:

Universal Analytics error message
Figure 1: Universal Analytics error message appears when customizing SEO report

Break out SEO traffic by market for international search engines

globe
By Flappiefh [Public domain], via Wikimedia Commons

A website targeting German speakers may want to better understand how it is performing in national variants of Google and Yahoo for each of the countries where German is spoken, including Germany, Austria, Switzerland, Luxembourg, Liechtenstein, Belgium and Italy. The site add the following search engines to the Google Universal Analytics settings:

Search Engine Domain MatchParameter
google.atq
google.beq
google.chq
google.deq
google.itq
google.liq
google.luq
at.search.yahoo.comp
be.search.yahoo.comp
ch.search.yahoo.comp
de.search.yahoo.comp
it.search.yahoo.comp

We’ve documented search domains for google, bing, yahoo, yandex and ask which can be used to improve Digital Analytics reporting and analysis. There is one caveat to breaking out Google search traffic by domain. Google became even more opaque in reporting search traffic from browsers which recognize the new referrer meta tag:

<meta content="origin" id="mref" name="referrer">

A few lines of JavaScript added to the Google Universal Analytics Tracking Code can restore the missing organic search parameter which is required in order to break out search traffic by google domain:

ga('create', 'UA-XXXXXX-1', 'mydomain.com');
var gsref = document.referrer ;
if (~gsref.indexOf('://www.google.')) {
	if (gsref.indexOf('?') ==-1) {
		gsref = document.referrer + '?q=';
		ga('set', 'referrer', gsref);
	}
}
ga('send', 'pageview');

The intent is to look for www.google. in the document referer. Currently other Google services use their own sub-domains, so we’re making the assumption that www.google is a search referrer. There are a few exceptions, e.g. some Google corporate information pages are on the domain www.google., but it is unlikely they link to our site. To capture traffic from all Google markets, e.g. google,com, google.co.uk, google.de, google.com.mx the full domain isn’t specified. There is the risk we could capture traffic from a completely unrelated domain which contains www.google. as sub-domains. The code could be enhanced to check that the missing domain characters are of 2, 3, 5 or 6 in length and the referer contains just domain information.

It isn’t actually possible to change the html document referrer value directly, the ga(‘set’, ‘referrer’, gsref); piece accomplishes this. This block of code needs to be inserted after ga(‘create’…) and before ga(‘send’….). Use at your own risk. Code improvement submissions are welcome.

The above logic can also be modified to track search traffic from blekko:

ga('create', 'UA-XXXXXX-1', 'mydomain.com');
var gsref = document.referrer ;
if (~gsref.indexOf('://www.google.') || ~gsref.indexOf('://blekko.com/ ')) {
	if (gsref.indexOf('?') ==-1) {
		gsref = document.referrer + '?q=';
		ga('set', 'referrer', gsref);
	}
}
ga('send', 'pageview');

Once the new code is inserted in web pages, add blekko as a search engine with the parameter q using the administration interface.

How to add a search engine to Google Universal Analytics

Adding search engine recognition to Universal Analytics is deceptively simple. Google says,

  1. Navigate to a property. If you’re not in the settings menu, click Admin. Select the account and property you want to edit.
  2. From the property column, click Tracking Info then Organic Search Sources.
  3. Click +Add Search Engine.
  4. In the form, specify what the Domain Name Contains and the Query Parameter. You can also specify a Search Engine Name and what the Path Contains.
  5. Click Create.
Adding a search engine definition to Google Universal Analytics
Figure 2: Adding a search engine definition to Google Universal Analytics

As is often the case, the devil is hidden in the details. The following considerations apply:

  1. Universal Analytics performs an include match on a domain. google.com will match google.com, images,google.com, google.com.au, google.com.mx and many others! When specifying match criteria, the goal is to be specific as possible to avoid false matches while avoiding to be so specific that future changes by the search engine won’t automatically be detected. Currently web.de uses suche.web.de for their search engine. I chose to specify .web.de so as to avoid matching anything which contains web.de, including other domains. I avoided specifying suche as web.de could decide to use www or something else for their search results in the future.
  2. The custom search engine domain list is evaluated in order. As soon as Universal Analytics finds a suitable match, it stops evaluating the list. Thus if we want to track traffic from google.com, we need to explicitly specify all other possible matches first: images,google.com, google.com.au, google.com.mx etc.
  3. Universal Analytics default search engine definitions are evaluated after custom search engine definitions. Note that Google has not always been very good at keeping the documented list of recognized search engines up to date, a point I cover in Antezeta’s Google Analytics course. The default search engines can no longer be removed; however the name which appears in reports can be overridden by specifying by using the new Search Engine Name field, a really nice improvement over the previous _addOrganic function. Some may want to capitalize the search engine name or use non-Latin characters to make reports more understandable.
  4. The Search Engine Name is case sensitive. Google.com will appear separately from google.com if at some point in time the case is changed for a definition.
  5. The new path parameter can be used should search and non-search services available on a domain make use of the same URL parameter, another nice improvement over the _addOrganic function.
  6. Only one search keyword phrase parameter may be specified. If a search engine uses multiple possible parameter names for search keywords, create a separate entry for each, using the same exact search engine name for each occurrence. Universal Analytics will automatically aggregate the results. For example create one entry for yahoo with the query parameter p and a separate entry for yahoo with the query parameter q (This is just an example, Google already does this for yahoo).
  7. It isn’t currently possible to copy custom search engine definitions from one account property to another nor to share search engine lists via a bulk import / export function. Those managing multiple sites should brace themselves for tedious and repetitious data entry.
  8. There isn’t any error checking during data entry. You can create a duplicate entry without any warning. This isn’t as harmless as one might think. In a long list it could become a debugging nightmare.
  9. Search engine definitions are not retroactive. They should be configured as soon as possible. A good “best practice” is to use an annotation in reports to note the Universal Analytics configuration change.
  10. Search engine definitions need to be maintained. If a search engine begins to appear in the referral traffic report, it may be that the keyword query parameter has changed. Keep in mind some “search engine” domains also host other services which may provide web traffic. A link in a Google doc will correctly show up as a referrer, not as search traffic. Some “search engines” will undergo name changes, such as when tiscali sold much of its operations outside Italy. New search engines will need to be added from time to time.
  11. It isn’t possible to document tracking issues. E.g. it is possible to measure search traffic from DuckDuckGo and internet.tre.it but the associated URL parameters do not actually provide the search keyword phrase.
  12. View filters (formerly profile filters) are evaluated after custom search engines have been applied to the collected data. This allows use of a work-around to add search engines such as DuckDuckGo and tre.it which don’t provide searcher keywords.
  13. It isn’t currently possible to use regular expressions to better specify domain match conditions.

Phew, that was a long list! Never worry, help and training are available if you’d like.

Suggested general search engines to add

Search Engine Domain Match StringURL Query ParameterPathNotes
alexa.comq
attrakt.comq
blekko.comqCurrently requires extra JavaScript in tracking code to add a dummy parameter q. See example in this document.
dogpileq
duckduckgo.comuddgKeyword phrase not available. However this should at least place Duckduckgo in the search report instead of the referrers report. The landing page will appear as a keyword. This can easily be fixed.
excite.q
teoma.comq

Suggested regional search engines to add to Universal Analytics

Search Engine Domain Match StringURL Query ParameterMarketNotes
1und1.deqGermany
benefind.deqGermany
freenet.dequeryGermany
metager.deeingabeGermany
suche.t-online.deqGermany
suche.web.deqGermany
fastweb.itqItaly
.tre.itqItalyKeyword not tracked; will appear as “Cerca nel web”. Consider adding a profile/view filter to change this to (not provided) or similar.
istella.itkeyItaly
libero.itqueryItaly
qrobe.itqItaly
zapmeta.itqItaly
.bt.compUK
sky.comtermUK
talktalk.co.ukqueryUK
virginmedia.comqUK
earthlink.netqUSA
netzero.netqueryUSA
verizon.netqUSA

Not a new feature, but adding search engines was hard to do

Previous versions of Google Analytics already allowed organizations to customize the default list of recognized search engines. This was done by advanced configuration of the tracking code placed in web pages to executed by a web browser. Universal Analytics has moved some tracking code configuration from the client (the browser) to Google’s servers. Google’s goals are undoubtedly to improve page load times, a.k.a. the user experience, and to reduce Marketing’s dependency on IT for Universal Analytics configuration support. Server based configuration also facilitates measurement beyond the browser using Universal Analytics’ Measurement Protocol. In my experience very few companies actually improve their Google Analytics search engine recognition, due to a lack of awareness of the need to do so and to the need to work with IT to fulfill the configuration requirements.

Enhancements I’d like to see

For those adding one or two search engines, the Universal Analytics administration interface works well. There are several improvements which would be wonderful in my view.

  1. Ability to share search engine definitions between properties. Ideally they would be defined at the account level and applied to properties so they would only need to be maintained once at the account level, similar to view filters (formerly profile filters)
  2. Ability to import and export definitions to a text file. This would facilitate sharing between Universal Analytics accounts.
  3. Add a note field to the search engine definition fields. It would be nice to specify that the order of a particular entry is important. Or that the entry has a view filter associated, as some may want to do with a DuckDuckGo custom search configuration.
  4. Ability to add a new search engine at any position in a list. Should the list becomes long, as it will if you specify each local version of google, yahoo, yandex and bing, it becomes very tedious to order new search engines which are placed at the bottom of the list.
  5. It is currently possible to add the same search engine match string multiple times. Error checking would be nice.

Looking to get more out of Google’s Universal Analytics? Sign-up today for Antezeta’s intensive Google Analytics training course!

This article refers to tracking search engines in Google’s Universal Analytics. Sites using the asynchronous version of ga.js should consult this article.

Similar Posts:

Registration is now open for the next SEO Course and Google Analytics Course in Milan. Don’t miss the opportunity!


About Sean Carlos

Sean Carlos is a digital marketing consultant & teacher, assisting companies with their Search (SEO + SEA = SEM), Social Media & Digital Media Analytics strategies. Sean first worked with text indexing in 1990 in a project for the Los Angeles County Museum of Art. Since then he worked for Hewlett-Packard Consulting and later as IT Manager of a real estate website before founding Antezeta in 2006. Sean is an official instructor of the Digital Analytics Association and collaborates with the Bocconi University. He is Chairman of the SMX Search and Social Media Conference, 13 & 14 November in Milan. He is also a co-author of the Treccani encyclopedic dictionary of computer science, ICT & digital media. Born in Providence, RI, USA, Sean received Honors in Physics from Bates College, Maine. He speaks English, Italian and German.

2 Responses to "Improve Search Engine Recognition in Google Universal Analytics"