Crawl Delay

Note: This documentation is for an old version of Webinator. The latest documentaion is here.

Crawl Delay

Syntax: a decimal number from 0 to 10

Causes Webinator to wait the specified number of seconds between page fetches. Normally set this to 0, and Webinator will fetch and process pages as quickly as it can. Increase the Crawl Delay if the web server cannot handle being hit rapidly. Increasing this value forces the walk to take at least the following number of seconds to complete: the Crawl Delay number times the number of pages on the site.

Decimal numbers may be specified - 0.1 will cause it to walk no more than 10 pages per second, etc.

Note: Using a delay larger than 0 forces Threads(4.4.10) to 1. A delay defeats the advantage of multiple threads and large delays could cause unexpected page fetch timeouts.