X-Robots-Tag in headers – fine grained crawlers control in apache

xrobotsI was asked to implement X-Robots-tag on one of our dedicated servers the other day.
This google document explains in the great details what it does and how it influences your site presentation on the search engine.

I have my doubts about how “otherbot” are willing to conform the google-set rules but that beyond the scope of this post.
The rest is the implementation. There are about 500+ virtual hosts on the server, each of them should have following added to the */public_html/.htaccess

Now, to add these lines to all htaccess files preserving what was there before is a little bit tricky.
First, we need to create small shell script that will be doing the actual adding

And from command line, something like this

Done. I wouldn’t pretend that this is the only optimal way to do it, but it’s quick and it works for me. YMMV.

Leave a Comment

NOTE - You can use these HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">