Syntax: -r
Ignore robots.txt. Normally gw
will initially get
/robots.txt
from any site being indexed and respect its settings
for what prefixes to ignore. This option will disable the use of
robots.txt and retrieve everything. Use of this option in not generally
recommended. Any URLs specified with -x
will still be excluded
when using this option.