If you think we have been crawling the wrong parts of your website do use Yahoo, Ask, Google, or any other search engine to discover important information on how to create a ‘robots.txt’ file that will ensure we view only relevant information on your website. We don’t give examples of “robots.txt” files as each website will have different needs depending upon factors such as the CMS used and the nature of the website.

Creating a valid “robots.txt” file

Do bear in mind the “robots.txt” file semantics are more complicated than they first appear. Random changes in your ‘robots.txt’ file or ‘just trying things out’ may inadvertently cause the exact opposite of what you intended. We recommend you hire the services of an SEO specialist who can help you create a properly crafted “robots.txt” file. However, we can give you some general advice. A good “robots.txt” file can significantly improve your rankings in public search engines – it’s definitely worth the effort to get it right.

You can test your “robots.txt” file using an online checker such as http://www.frobee.com/robots-txt-check.

If want to block us from indexing your entire website, you can create a “robots.txt” file to block us completely. However, before you do please do bear in mind the following information.

• We provide information to respected credit reference agencies who may consider a block on indexing your website or significant portions of it as an important risk indicator. If your website is personal in nature, this issue may be safely ignored.
• Google, Yahoo, and other search engines will look at all rules in the “robots.txt”, not just the rules provided for them. If there is a significant difference in permissions in the ‘robot.txt’ file rules for different search engines, this will be seen as an attempt at SEO fraud and will lead to low ranking scores or exclusion from search results. Be careful not to accidentally damage your website ranking by allowing Google all permissions and us none.
• Even for personal sites, if you create ‘robots.txt’ rules that allow, say, Google to index your website but forbid some or all other search engines from doing so, then your activities could be viewed as “anti-competitive”. This could leave you open to litigation. You should take appropriate legal advice before constructing such “robots.txt” rules. Remember, Google and the other major search engines are aware if they index a website with a ‘robots.txt’ file that prevents other search engines from doing the same, that they are also open to litigation. As a result, Google and the other major search engines will not index sites where the ‘robots.txt’ file gives them significantly more permissions than other crawlers – this means your website will at best lose its ranking and at worst mean your website is excluded from Google searches and the loss of any advertising revenue due to you.

We hope you have found this information useful. If you are interested in any of our services, please do visit www.aihit.com.

Contacting us

If you feel we have breached our policy, or believe your web-site may have been crawled by a crawler purporting to be aiHit, please email us at crawler@aihit.com with the following information:

• Your company name
• The domain name (e.g. abc.com) where you believe the abuse occurred
• Proof of domain ownership
• The nature of the abuse (e.g. not following “robots.txt” rules)
• Details from your logs of the date and time the alleged abuse occurred. This is important as we can compare this with our own logs.
• Your contact information if we require further information and to report back the results of our findings

Back...