Search Engines

Search engines create their listings automatically; they crawl the web, gathering information about sites and pages and compile this in their databases. People, you and me, can then search through what they have found and hopefully find what we are looking for.

Today’s search engines come in two types.

Directory-based engines, like Yahoo, are still built manually. “What that means,” Giles says, “is that you decide what your directory categories are going to be” — Business, and Health, and Entertainment — “and then you put a person in charge of each category, and that person builds up an index of relevant links. This used to be the function of librarians, right?” Changing your web pages has no effect on your listing. The elements that are useful for improving a listing with a search engine have nothing to do with improving a listing in a directory. The only exception is that a good site, with good content, might be more likely to get reviewed for free than a poor site.

Crawler-based engines, like Google, employ a software program — called a crawler — that goes out and follows links, grabs the relevant information, and brings it back to build your index . If you change your web pages, crawler-based search engines eventually find these changes, and that can affect your listings. Page titles, body copy, links and other elements all play a role.

As information grows deeper, crawler-based engines become more and more popular. But automatic engines have their limitations, too. For one thing, most current crawlers are unable to recognize “spam,” which in this context means unreliable information. An even more basic problem, however, is that crawlers, by their nature, depend on the links between sites to get from one place to another. Sometimes, links don’t exist, and if links don’t exist, sites that might be relevant don’t get found.

( compiled from different sites )