Multiple Fetches

Note: This documentation is for an old version of Webinator. The latest documentaion is here.

Multiple Fetches

Syntax: select Y or N

Multiple Fetches allows a page to be fetched multiple times, and can potentially slow down a crawl. It should only be used in specific situations in conjunction with Off-Site Pages.

For example, Consider the situation of crawling two sites, a.com and b.com with Off-Site Pages enabled. A link from a page on a.com to b.com/page.htm is considered off-site, so it will be crawled but its links won't. Then, when b.com starts its crawl, b.com/page.htm won't be processed because it's already been done, causing b.com/page.htm's links to not be included.

Multiple Fetches allows the 2nd encounter of b.com/page.htm to be processed again, which will allow its links to be properly processed.