|
Syntax: select Yes or No buttons
robots.txt
With this set to Yes, Webinator will initially get
/robots.txt from any site being indexed and respect its settings
for what prefixes to ignore. Ignoring robots.txt is not generally recommended.
See also Robots.txt 5.4.
Meta
Respect the meta tag called robots. With this set to Yes
Webinator will process and respect the robot control information
within each retrieved HTML page.
See also Robots.txt 5.4.
Copyright © Thunderstone Software Last updated: Thu Dec 22 14:38:01 EST 2011
|