Page Fetching Behavior

• Grab off-site pages (-o)
• Don't add to todo (-a)
• Allow an entire domain (-domain)
• Allow an entire network (-network)
• Control allowed extensions (-f and -F)
• Define a plugin (-n)
• Allow cgi-bin paths (-C)
• Allow all file types (-y)
• Add allowed MIME type (-T)
• Delete allowed MIME type (-S)
• Define prefix exclusions (-x)
• Define path pattern exclusions (-x/)
• Define query pattern exclusions (-exquery)
• Define URL prefix restrictions (-j)
• Ignore robots.txt (-r)
• Set login user name (-U)
• Set login password (-P)
• Set proxy to use (-proxy)
• Don't lookup every hostname (-L)
• Don't report as Mozilla (-M)
• Set reported Mozilla version number (-M)
• Set reported User-Agent (-M/)
• Index meta data (-meta)
• Strip query strings (-Q)
• Don't include alt text (-N)
• Don't store refs (-R)
• Force page insertion (-Force)