Comments on: 6 methods to control what and how your content appears in search engines

By: jaime

jaime — Tue, 07 Feb 2012 19:31:22 +0000

this nice

By: Dhika - Pang Iklan

Dhika - Pang Iklan — Fri, 14 Oct 2011 12:20:16 +0000

in robot.txt file, can I write like http://www.mysite.com/images/ if I want prevent just 1 folder ?

By: William Lake

William Lake — Wed, 21 Sep 2011 18:52:35 +0000

Hi,

I’ve got the following code in my robots.txt

User-agent: *
Disallow: /

Yet google has indexed 44 pages and directories and yahoo 1.

I’m confused, very confused. Any ideas?

By: Katie @ Women Magazine

Katie @ Women Magazine — Tue, 23 Aug 2011 11:29:57 +0000

Obviously, you must be overwhelmed with lots of “thank you for sharing” comments but this is something which is really good and informative.

I would request you to write how to block all unwanted spiders and allow only specific crawlers. Is it even possible?

By: menno

menno — Sat, 23 Jul 2011 00:38:04 +0000

about

what do you need to do to prevent cache on third party websites only but to keep them on the original domain?

By: Chicago With Kids

Chicago With Kids — Fri, 13 May 2011 14:42:59 +0000

Just what I was looking for. Usually I want search engines to find my pages, but not in this case. Glad to know I can keep them at bay.

By: Nur Mohammad

Nur Mohammad — Mon, 09 May 2011 20:04:40 +0000

Thanks for the great article!!!
i am gonna give a try to protect a Folder in my website.

By: liza

liza — Wed, 02 Mar 2011 09:36:15 +0000

This is a very comprehensive resource, I must say. My question is somewhat related to this but here’s the scenario.

We have a website running on two servers. So the files/website is synchronized in these two servers. We have a second server for internal purposes. Let’s name the first server as www and ww2 for the second server. ww2 is automatically updated once the files are updated in www.

Now, Google is indexing the ww2 which I want to stop and just let www be crawled and indexed. My questions are: 1. How can I removed those crawled pages in ww2 removed from Google index? 2. How can I stop google from indexing ww2?

By: Better-IT

Better-IT — Thu, 07 Oct 2010 14:24:03 +0000

A very comprehensive post, which I’m bookmarking as a reference.

Might be good to add techniques such as javascript links which search engines still can’t follow (I think?)

By: John

John — Fri, 01 Oct 2010 17:07:58 +0000

I am also thinking of how to block my content being crawled using .htaccess. Any one has any idea?

By: David

David — Fri, 10 Sep 2010 05:26:27 +0000

Great and comprehensive post. Many thanks for including so much detail!

By: Magson Fernandes

Magson Fernandes — Tue, 07 Sep 2010 20:38:21 +0000

if i use .htaccess to protect a folder will it stop search engines from accessing my content from my folder.

By: Damn Amazing Things

Damn Amazing Things — Wed, 07 Jul 2010 12:37:18 +0000

Thanks For This information about google robots

By: chris

chris — Thu, 22 Apr 2010 09:50:55 +0000

Guys,

I am trying to stop google indexing certain search features, as its indexing them as duplicate content, let me try and explain this.

From Webmaster tools,

Pages with duplicate title tags Pages
Thermolife Products – Sport and Supplements
Go to URL/brands/Thermolife.html
Go to URL/brands Thermolife.html?sort=alphadesc
Go to URL/brands/Thermolife.html?sort=newest Go to URL/brands/Thermolife.html?sort=priceasc
Go to URL/brands/Thermolife.html?sort=pricedesc

Sorry about the text above, but as you can see im getting all these ?String indexed, which is not great, I have tried to search google and use there help docs to restrict it in the robots txt, but Im lost and need a little help please.

Regards

Chris

By: Corl

Corl — Mon, 19 Apr 2010 19:19:23 +0000

Very informative article.

Since I can only use the meta robots tags – I am stuck with a quandary. It’s great that the bots index my site, I just don’t want my images to be indexed and to show up in the Google, Bing, Yahoo!, or others(?) image searches. Is there a tag or another way without access to the robots.txt file, to tell all the search engines not to index the images on my site?

Thanks,
Corl

By: Interface

Interface — Thu, 18 Mar 2010 18:56:25 +0000

Robots.txt has always been very effective for us. That, coupled with Google’s Webmaster Tools, makes ranking management relatively straightforward.

By: Binary Soldier

Binary Soldier — Thu, 08 Oct 2009 18:16:33 +0000

A good selection of the various methods which are often used.

I plan to stop hot-linking of my imges and it seems CPanel makes this fairly easy.

By: Matt

Matt — Tue, 22 Sep 2009 14:20:04 +0000

Great article and very informative. Hopefully in time Bing will improve the way it reads robots.txt

By: uttam Kadam

uttam Kadam — Thu, 23 Jul 2009 05:17:32 +0000

deep and clear description is found.
Which is very useful.
Thanks for this post.

By: bill

bill — Wed, 08 Jul 2009 16:28:51 +0000

Nice summary. I am looking to keep all the engines from indexing a certain block of text on each page. Does anyone know of a bing and google equivalent to yahoo’s class=”robots-nocontent” ?

By: Shelley

Shelley — Tue, 07 Jul 2009 11:53:22 +0000

Nice article!
I’m having the same issue as Vilnis..someone uploaded a PDF and some XLS to the server and they coming up in the search with the usual 2 line blurb. I’ve changed the actual document to have no information in it, but in the search results it is showing information from the old content.
How do I get rid of this, any ideas?

By: Vilnis

Vilnis — Mon, 06 Jul 2009 16:42:37 +0000

Hi. Thanks for he article re “noindex” etc.

We removed a pdf from our server, that had been indexed by Google. The “view as HTML” link still appears in a Google search result, and still shows an HTML rendering of the PDF – how can we get this listing removed from searches?

By: James

James — Wed, 18 Feb 2009 16:22:24 +0000

Great post and exactly what I have been looking for. Thanks.

By: Jody

Jody — Sun, 08 Feb 2009 05:10:37 +0000

Even though, as a newbie, much of this went over my head, it served the teaching function of alerting me to something that I should know about and even provided useful links for further exploration. Thank you!