|
On the first access to a site the file /robots.txt will be
retrieved, if its exists. Settings there will be respected. Anything
in the todo list that is disallowed by robots.txt will be
discarded.
Copyright © Thunderstone Software Last updated: Tue Nov 6 10:58:47 EST 2007
|