Block bots and crawlers with .htaccess and robots.txt

0
201

If you want to block search engine and crawler bots from visiting your pages you can do so by uploading the following files to the / directory on your site. Double check you aren’t rewriting over any .htaccess file before you do it.

http://jamesbachini.com/misc/blockbots/robots.txt

http://jamesbachini.com/misc/blockbots/.htaccess

(.htaccess link doesn’t work because it’s a no read file. So I have uploaded them as a zip)

Download both files in a zip

These two files when uploaded to your root directory will block a lot of the common crawlers and bots that are coming on to your site. It is good practice that bots check for the robots.txt file before they scan your site however if they don’t the .htaccess rulesets should prevent them anyway. It won’t however do any good for ad fraud bots unfortunately. If you are getting a lot of bots from a particular traffic source optimise your sources. These type of bots will inevitably cloak the user agent anyway but can be detected by lack of micro-conversions, time on page and mouse actions via javascript.

bots

LEAVE A REPLY

Please enter your comment!
Please enter your name here