Allow Extensions

Syntax: one or more file extensions separated by space

A list of the URL extensions that the crawler will accept. The default list is empty, i.e. all extensions are allowed.

To search MS-Word documents, use .doc. For Shockwave/Flash use .swf. For WordPerfect documents specify whatever extension you use and ensure that the web server returns the MIME type application/wordperfect as there is no consistent extension for WordPerfect documents. Any extensions not listed here will not be searched or walked.

A few other extensions you may find useful are

.asp
.cfm
.jsp
.shtml
.jhtml
.phtml