Tagged: crawlers


Block bots and crawlers with .htaccess and robots.txt

If you want to block search engine and crawler bots from visiting your pages you can do so by uploading the following files to the / directory on your site. Double check you aren’t rewriting over any .htaccess file before you do it. http://jamesbachini.com/misc/blockbots/robots.txt http://jamesbachini.com/misc/blockbots/.htaccess (.htaccess link doesn’t work because it’s a no read file. So I have uploaded them as a zip) Download both files in a zip These...