Syntax: select Yes or No buttons
robots.txt
With this set to Yes
Webinator
will initially get
/robots.txt
from any site being indexed and respect its settings
for what prefixes to ignore. Ignoring robots.txt is not generally recommended.
See also Robots.txt
5.3.
Meta
Respect the meta tag called robots
. With this set to Yes
Webinator
will process and respect the robot control information within each retrieved
HTML page.
See also Robots.txt
5.3.