Keep out: an often overlooked part of Search Engine Optimization


Rare is the web professional who doesn’t know that building a great website isn’t usually enough to guarantee its success. Sites have to be visible in search engines for the keywords and phrases web navigators are most likely to associate with the site’s content. An entire industry has grown up around SEO, search engine optimization. Yawn, you say.

What about the reverse side of the coin, keeping content out of search engines? Should be easy, no? Maybe not. In February, we looked at 5 ways to stop Google and the other search engines from downloading and indexing a website’s pages.

Unfortunately, it appears that the folks behind the personal lubricant astroglide didn’t understand the implications of leaving sensitive customer data on a public web server. They, and their customers, found out how search engines can be all too effective in finding content – as long as it is in a public area and there is a public link to it! Too bad astroglide blamed Google rather than admitting the error of their ways.

The astroglide case isn’t an isolated incident. A consortium of French and German Press in Belgium, Copiepresse, has been battling the search engines to keep Belgium news out of search results – all while blissfully ignoring the circa 1996 robots.txt protocol. It does appear that Copiepresse is finally making progress in learning how to manage what content appears in search engines using existing web conventions.

As of last week, there is an additional tool available to manage content in Yahoo!. By adding a class=”robots-nocontent” attribute to an html tag, a webmaster can specify that content within the html tag shouldn’t appear in Yahoo! As we note in our related class=”robots-nocontent” article, we welcome the additional granularity that this option offers to specify how a search engine indexes a page. Unfortunately its value will be limited until all the major search engines adopt it or a similar syntax.

So have you protected your sensitive content from search engine crawlers?

Similar Posts:

Registration is now open for the next SEO Course and Google Analytics Course in Milan. Don’t miss the opportunity!


About Sean Carlos

Sean Carlos is a digital marketing consultant & teacher, assisting companies with their Search (SEO + SEA = SEM), Social Media & Digital Media Analytics strategies. Sean first worked with text indexing in 1990 in a project for the Los Angeles County Museum of Art. Since then he worked for Hewlett-Packard Consulting and later as IT Manager of a real estate website before founding Antezeta in 2006. Sean is an official instructor of the Digital Analytics Association and collaborates with the Bocconi University. He is Chairman of the SMX Search and Social Media Conference, 12 & 13 November in Milan. He is also a co-author of the Treccani encyclopedic dictionary of computer science, ICT & digital media. Born in Providence, RI, USA, Sean received Honors in Physics from Bates College, Maine. He speaks English, Italian and German.