Search Engine Spiders and Crawler Pages

This week, I'm going to show you a good technique for getting your Web pages listed with Google and other search engines, and which can subsequently help increase the traffic to your site. Specifically, what we're talking about is what is sometimes referred to as a "crawler page."

What's A Crawler Page?
A crawler page makes it easier for all the pages on your site to be crawled by a search spider. A search spider is a device that search engines use to crawl the Web every so often, looking for pages to list. If your page is on the Web, then it will eventually be crawled by one of these spiders. Of course you can also submit your page manually, or pay someone else to do it.

Free, Fee, and Relatively Easy
Most search engines allow you to submit your site along with a description of it for free. Yahoo is an excellent engine to be listed with, which leads to them having a lot of pages to sort through. While suggesting your site to Yahoo won't guarantee inclusion in their directory (or even that they'll look at your page to begin with), you'll have a better chance of getting in if you submit than if you don't.

If you don't want to go to the trouble of submitting your site yourself, you can pay someone else. One of the most popular site submitting services, Submit It, currently offers their services for somewhere around $50. For your money, you get submitted to most of the major search engines and you stay submitted for one year. The site is a bit ambiguous on the question of whether or not you'll actually get listed, but it's probably safe to assume that it's not guaranteed. Still, you'll be upping your chances significantly.

Oh What a Tangled Web We Weave...
Now, about that crawler page we were talking about earlier. The crawler page simply lists all the pages in your site so that a search spider can easily get to all of them. That's all it does. So, instead of trying to track down every single page on your site from your index page, the spider crawls the crawler page and gets to all of them instantly. While this might not mean much to you if you have a very small site with less than 10 pages, more complex sites can have hundreds, or even thousands, of pages -- and all of them might not get crawled without a crawler page.

To set up a crawler page, start with a blank file and call it "archive.htm", "sitemap.htm", or something similar; the contents of this page are going to be nothing but links to all the pages on your site. So, the source might look something like this:

<a href="about_us.htm">About Us</a><br>
<a href="our_services.htm">Our Services</a><br>
<a href="our_services_outreach.htm">Our Services -- Outreach</a><br>
<a href="our_services_in_home.htm">Our Services ? In home</a><br>
<a href="our_services_awards.htm">Our Services -- Awards</a><br>
<a href="board_members.htm">Board Members</a><br>
<a href="activities.htm">Activities</a><br>
<a href="activities_summer.htm">Activities -- Summer</a><br>
<a href="activities_winter.htm">Activities -- Winter</a><br>
<a href="activities_spring.htm">Activities -- Spring</a><br>
<a href="activities_fall.htm">Activities -- Fall</a><br>
<a href="calendar.htm">Calendar</a><br>
<a href="summer_season.htm">Summer Season</a><br>
<a href="staff.htm">Staff</a><br>
<a href="interns.htm">Interns</a><br>
<a href="members_only.htm">Members Only</a><br>
<a href="contact_us.htm">Contact Us</a><br>
<a href="merchandise.htm">Merchandise</a><br>

And that's it. Now, provide a link to this page off of your index page (which is probably the page you'll list with search engines) and call it your archive or site map or whatever you'd like. Alternately, you can include your index page in the crawler page and just list the crawler page with the search engines. Remember, the page really isn't for humans to look at -- it's primarily to be read by a computer, to make it easier to access all the pages in your site. With a crawler page you stand a better chance of having your pages located and used.

Give it a try and see if you don't get better treatment from those search engines. Good luck and happy coding!

Contributions @ Dream City Design