Note: This documentation is for an old version of Webinator.
The latest documentaion is here.
Robots
Syntax: select Yes or No buttons
robots.txt
With this set to Yes, Webinator will initially get
/robots.txt from any site being indexed and respect its settings
for what prefixes to ignore. Ignoring robots.txt is not generally recommended.
See also Robots.txt 5.3.
Meta
Respect the meta tag called robots. With this set to Yes
Webinator will process and respect the robot control information
within each retrieved HTML page.
See also Robots.txt 5.3.
Copyright © Thunderstone Software Last updated: Thu Mar 11 16:13:32 EST 2010