Was one of you New Year's resolutions to improve the efficiency of your I.T. infrastructure? Our expert engineering staff is available to clients with a current maintenance contract, and for the next month you can schedule a free 15-minute consultation on a first come first served basis to discuss your architecture and get immediate advice on improving your solution. If you need more time, that can also be arranged. There are a limited number of time slots available. So, make sure to call today at +1 216 820 2200.
There are a number of ways to specify what URLs you'd like the
software to crawl, and which will be easiest to use can depend on your situation.
The old standby -- URLs listed in the Base URL will be crawled, and the entirety of all pages they link to will be included. If you only have one or two sites and start from the top, this is definitely the way to go.
Sometimes you may have dozens or hundreds of base URLs, maybe for doing many different folders on a site (but not all of them). If putting them all in a text box is starting to feel unwieldy, you can use the URL URL instead.
Create a text file somewhere on your website that contains the URLs you want to use as Base URLs, each on its own line. Then you can specify the URL to _that_ in the URL URL setting. The URL URL is
fetched by the crawler, and every URL is treated as a Base URL. This
can make it easier to manage a frequently-changing list of URLs.
This is the only benefit of URL URL. The pages will still be crawled EXACTLY as if they were all listed as Base URLs. It exists only to make the list easier to manage for you.
Sometimes you have a single page, or a handful of pages, where you
just want that page crawled, but none of its links. This is exactly
what Single Page is for.
The URLs listed in Single Page are fetched, and their links are
Just as "URL URL" is a list of URLs for "Base URL", "Page URL" is a list of URLs for "Single Page". URLs listed here should point to a
plain text file on your server, each URL on its own line. Every one
of those pages is fetched, and their links are completely ignored.
In the coming months you'll find a number of interesting new case studies at the Thunderstone.com website. As usual, we'll also feature links to them them here in this newsletter. Keep an eye out for them. You won't want to miss these case studies of TEXIS, Webinator and Thunderstone Search Appliances "in action".