How to update robots.txt file

As site owner you can simply update the robots.txt file from Site Settings area.

You can use the following content for the robots.txt file:

To allow all robots complete access

(or just create an empty “/robots.txt” file, or don’t use one at all)

To exclude all robots from the entire server

To allow a single robot

Alternatively you can explicitly disallow all disallowed pages:

You can also add a crawl-delay parameter, set to the number of seconds to wait between successive requests to the same server:

Note: robots.txt  file supports a directive for slowing down for certain bots like MSN or Yandex, but Google Bot does not support it.

For advance usage of robots.txt file, please check the following page: http://www.robotstxt.org/robotstxt.html

As an alternative you can check the robots.txt file using the following tool: http://tools.seobook.com/robots-txt/analyzer/