Google Crawling and Execution of JavaScript: where are we at today?


For a long time, Google’s advice to website developers was to keep things simple to ensure search engine spiders could successfully crawl and process website content:

Use a text browser such as Lynx to examine your site, because most search engine spiders see your site much as Lynx would. If fancy features such as JavaScript, cookies, session IDs, frames, DHTML, or Flash keep you from seeing all of your site in a text browser, then search engine spiders may have trouble crawling your site.1

In reality, Google often found links in Flash objects, significantly improving this ability as announced last June (creating much confusion by misrepresenting this as a new feature rather than an improvement). And despite the hoopla, there are still many good reasons to avoid Flash.

JavaScript has been another can of worms. Early search engine crawlers and indexers were not sophisticated enough to process JavaScript. Yet astute observers will note that Google has downloaded external JavaScript files linked in html <script> tags for awhile now (and css for that matter); others noted Google has extracted and followed easily identifiable links in JavaScript code for several years now. Clearly Google hasn’t been completely JavaScript blind.

Yet actual interpretation of JavaScript code presents many problems, as evidenced by the difficult deployment several years ago of the much-maligned Google Web Accelerator.

Yesterday I had two people contact me about Google crawling and actually executing JavaScript (rather than just extracting obvious links). Due to the previous experience with the Google Web Accelerator, I admit I was a bit skeptical. I hadn’t seen an announcement by Google or Google representatives. Yet today, while investigating this hypothesis of Google spidering and following JavaScript, I saw this post on Search Engine Land about Google successfully interpreting a JavaScript based menu system. This particular case may be no more than what Google was already doing 5 years ago.

To what extent does Google use JavaScript today, and what are the implications – both pro and con?

I don’t yet have an up-to-date answer. But while working to figure this out, I’d love to know your experience.


1 http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=35769

Similar Posts:

Registration is now open for the next SEO Course and Google Analytics Course in Milan. Don’t miss the opportunity!


About Sean Carlos

Sean Carlos is a digital marketing consultant & teacher, assisting companies with their Search (SEO + SEA = SEM), Social Media & Digital Media Measurement strategies. Sean first worked with text indexing in 1990 in a project for the Los Angeles County Museum of Art. Since then he worked for Hewlett-Packard Consulting and later as IT Manager of a real estate website before founding Antezeta in 2006. Sean is an official instructor of the Digital Analytics Association and collaborates with the Bocconi University. He is Chairman of the SMX Search and Social Media Conference, 13 & 14 November in Milan. He is also a co-author of the Treccani encyclopedic dictionary of computer science, ICT & digital media. Born in Providence, RI, USA, Sean received Honors in Physics from Bates College, Maine. He speaks English, Italian and German.

5 Responses to "Google Crawling and Execution of JavaScript: where are we at today?"

Leave a reply

Warning: Comments are very welcome insofar as they add something to the discussion. Spam and/or polemical comments without a rational justification of the author's position risk being mercilessly deleted at the sole discretion of the administrator. Yes, life is hard :-).