Update your reporting on a regular basis
This handy tool can help you to understand whether your robots.txt file is set up correctly for all the pages that you actually want Google's spider algorithms to crawl. If you are a tech-savvy person and are familiar with HTML, you can change these tags in the code yourself …
more ...