Syntax: one or more file extensions separated by space
A list of the URL extensions that the crawler will accept. The defaults are
.html
.htm
.txt
.pdf
To search MS-Word
documents, use .doc
. For Shockwave/Flash use .swf
. For WordPerfect documents specify whatever extension you use and ensure that the web server returns the
mime type application/wordperfect
as there is no consistent extension for WordPerfect documents. Any extensions not listed here will
not be searched or walked.
A few other extensions you may find useful are
.asp
.jsp
.shtml
.jhtml
.phtml