Robots.txt page exclusion

Note: This documentation is for an old version of Webinator. The latest documentaion is here.

Robots.txt page exclusion

On the first access to a site the file /robots.txt will be retrieved, if its exists. Settings there will be respected. Anything in the todo list that is disallowed by robots.txt will be discarded.