Ignore robots.txt (-r)

Syntax: -r

Ignore robots.txt. Normally gw will initially get /robots.txt from any site being indexed and respect its settings for what prefixes to ignore. This option will disable the use of robots.txt and retrieve everything. Use of this option in not generally recommended. Any URLs specified with -x will still be excluded when using this option.