http://www.jlh-design.com/2008/08/dont-use-robotstxt-to-control-indexing/ Terrible writing and mere conjecture Sun, 14 Mar 2010 11:36:02 +0000 http://wordpress.org/?v=abc hourly 1 http://www.jlh-design.com/2008/08/dont-use-robotstxt-to-control-indexing/comment-page-1/#comment-6044 Patrick Daly Tue, 19 Aug 2008 19:02:22 +0000 http://www.jlh-design.com/?p=492#comment-6044 Great post. Often the ease of a one-liner in the robots.txt seems the easiest and quickest solution, but you convinced me otherwise. Great post. Often the ease of a one-liner in the robots.txt seems the easiest and quickest solution, but you convinced me otherwise.

]]> http://www.jlh-design.com/2008/08/dont-use-robotstxt-to-control-indexing/comment-page-1/#comment-6017 Everett Mon, 04 Aug 2008 20:56:28 +0000 http://www.jlh-design.com/?p=492#comment-6017 That's all fine until you start dealing with eCommerce sites that have a ton of duplicate content created from dynamic functions that are out of your control, for which the "system" is unable or unwilling to allow dynamic insertion of a noindex meta tag (i.e. if you add one to the dynamic variable page ?sort=bestsellers it also adds it to the unsorted page). In those cases, you have to block the content from being indexed by whatever means you have at your disposal - namely robots.txt with wildcard disallows. Ideally, one would have enough control over their site, or their clients' site to do a dynamic insertion, but with big eCommerce soutions this is not often the case. That’s all fine until you start dealing with eCommerce sites that have a ton of duplicate content created from dynamic functions that are out of your control, for which the “system” is unable or unwilling to allow dynamic insertion of a noindex meta tag (i.e. if you add one to the dynamic variable page ?sort=bestsellers it also adds it to the unsorted page). In those cases, you have to block the content from being indexed by whatever means you have at your disposal - namely robots.txt with wildcard disallows. Ideally, one would have enough control over their site, or their clients’ site to do a dynamic insertion, but with big eCommerce soutions this is not often the case.

]]>