URL URL
Syntax: an HTTP URL to a plain text file (NOT HTML)
This allows you to specify the URL of a plain text file containing a list of site URLs to walk. This is an additional way of specifying more Base URLs 4.4.4. This URL will be refetched each time a rewalk is started. In the file, the list of URLs can be one URL per line (preferred) or delimited by any number of spaces.
Warning: Due to the nature of Stay Under, a large number of URL
URLs (1000+) in different directories will cause the crawl to progress
very slowly, as all URLs encountered will need to be checked against
every one of those directories. In such a situation, we recommend
turning off Stay Under and instead writing your own
Required Prefix/Required REX expressions, which will be
more efficient.
Copyright © Thunderstone Software Last updated: Thu Mar 11 16:13:32 EST 2010