URL URL

Syntax: an HTTP URL to a plain text file (NOT HTML)

This allows you to specify the URL of a plain text file containing a list of site URLs to walk. This is an additional way of specifying more Base URLs 4.4.4. This URL will be refetched each time a rewalk is started. In the file, the list of URLs can be one URL per line (preferred) or delimited by any number of spaces.

Warning: Due to the nature of Stay Under, a large number of URL URLs (1000+) in different directories will cause the crawl to progress very slowly, as all URLs encountered will need to be checked against every one of those directories. In such a situation, we recommend turning off Stay Under and instead writing your own Required Prefix/Required REX expressions, which will be more efficient.