How To Use .htaccess & robots.txt To Block Bots
If you want to block search engine and crawler bots from visiting your pages you can do so by uploading the robots.txt and .htaccess files to the root directory on your site. Double check you aren’t rewriting over any existing .htaccess file before you do it.
(.htaccess link doesn’t work because it’s a no read file. So I have uploaded them as a zip)
These two files when uploaded to your root directory will block a lot of the common crawlers and bots that are coming on to your site. It is good practice that bots check for the robots.txt file before they scan your site however if they don’t the .htaccess rulesets should prevent them anyway. It won’t however do any good for ad fraud bots unfortunately.