SEO Expert Guide - Search Engines Explained (part 1/10)

Before we explore the world of search engine optimization, it is vital that you know a little about how search engines work and their relative market shares. It will help you to prioritize your activities later!

(a) What are Search Engines and who powers them?

There are essentially four different parts to a typical large search engine; the crawler, the directory, sponsored results and the search engine itself.

Crawlers (e.g. Google) automatically visit web pages to compile their listings, making use of a so-called robot or spider (eg. Googlebot), which follows links from one website to another, ultimately compiling an index of all the pages and sites on the internet. These crawlers provide an index, which can then searched by the search engine. You may find that several or all of the pages on your site are indexed in thisway. Some search engines have their own crawler and others buy-in crawler results from others.

Human-powered directories, such as the Open Directory, rely on submissions from the public, which are reviewed by editors for inlusion in the directory. If you get included in a directory, generally only one page from your site (usually your home - or index - page) will be listed.

Crawled results are combined with sponsored results, supplied by pay-per-click (PPC) advertisers, and the results from human-maintained directories to complete the search engine index. Check out the Search Engine Reationship Chart at Bruce Clay inc. for the latest picture on who powers whom. You will note a couple of things right away. Firstly, the dominance of the Google and Yahoo! crawlers and secondly the importance of DMOZ directory results as a back-door for many search engines.

(b) How do Search Engines find and rank sites?

Search engines do not really search the web directly, but rather an index database of the full text of web pages, which itself is drawn from the billions of web pages on the internet's servers. Search engine databases are selected and built by computer robot programs called spiders.

If a web page is never linked to by any other page, spiders cannot find it, unless the (usually new) site is submitted manually by a human at the search engine's "add URL" page. All search engine companies offer ways to do this.

After spiders find pages, they pass them on to another computer program for "indexing." This program uses an "algorithm" to assess the text, links, and other content in the page for "key words" that might be searched on at the engine. This allows the search engine to order results served by their "relevancy" to the search terms used. As each search engine has a different algorithm, it will index sites in a different way and thus serve up different relevant results.

Some types of pages and links are excluded from most search engines by policy. Others are excluded because search engine spiders cannot access them. Generally, the use of frames, flash graphics and dynamic URLs all get in the way of effective spidering and should thus be avoided.

In addition to indexing pages, most algorithms seek to establish the "authority" of a site. A site which is linked to by many other sites (using keyword-rich anchor text) is assumed to be of greater merit than one with no links at all. This activity is called "ranking" and helps search engines to sort otherwise similar results into ever-more relevant and authoratative results.

(c) Which Search Engines are the most popular?

Based on US analysis in January 2005, the top search engines (by share of total searches at home and work) are as follows:

Google Search - 47%

Yahoo! Search - 21%

MSN Search - 13%

All Others - 19%

These shocking figures do not convey the true dominance of the top players, as you have seen from the interdependence of search provision in section (a) above. You could be searching at AOL (part of the "other" 19%) and viewing Google results, for example.

There is also strong anecdotal evidence that Yahoo! and MSN tend to send more searchers through to their sponsored (or paid) results than do Google (due to the prominence of these results on their results pages). As such, for a typical small webmaster who does not use pay-per-click (PPC) advertising, they might get up to 80% of all their traffic from Google's various sites across the world.

Now you understand the market a little better, you will perhaps understand the obsession many webmasters have with Google! A top-10 position at Google for your key search terms can make your online business fly. If you drop out of that top-10, your business can literally collapse overnight!

Don't forget these key stats as you embark on your optimization journey...

Navigate the guide

Previous : SEO Expert Guide - Index of Contents

Next: SEO Expert Guide - Proposition Development (part 2/10)

About the author:

David Viney ([email protected]) is the author of the Intranet Portal Guide; 31 pages of advice, tools and downloads covering the period before, during and after an Intranet Portal implementation.

Read the guide at http://www.viney.com/DFV/intranet_portal_guide or the Intranet Watch Blog at http://www.viney.com/intranet_watch.

In The News:

How Can I Set Up an SEO Agency?  Search Engine Journal
How to Choose an E-commerce SEO Agency?  https://www.indianretailer.com/
The Ultimate eCommerce SEO Checklist  Business 2 Community
5G Just Got Weird  IEEE Spectrum
3 SEO Principles for New Strategists  Multichannel Merchant
What is SEO?  Business 2 Community
6 Best Hacks For SEO Managers  Business 2 Community

Search Engine Ranking... Oh, the Mystery!

Rankings, Rankings, Rankings!How do you get your website ranked well... Read More

Search Engines from a Webmaster Perspective

By now, virtually every webmaster has heard or read that... Read More

Google Groups

Some very early users of the Internet - not the... Read More

Should You Be Linking for Traffic or Rankings?

Just for a change, rather than a technical article, I... Read More

How To Avoid These Ten Costly Search Engine Mistakes

If you have a website then you already know the... Read More

Google: The Ultimate Web Writer?s Style Guide

Indulge me for a moment.Forget that Google is a search... Read More

Everything You Wanted To Know About Google -- But Were Afraid To Search For!

(A Reflective look at the little search engine that soared!)All... Read More

How to Succeed with the Search Engines

The Cold Hard Facts?..One of the most important factors in... Read More

Google Takes Care of Idiots Too

There's an old saying that goes, "God takes care of... Read More

LSI and Link Popularity

When Paypal's official Web site no longer ranked #1 in... Read More

Search Engines The Masters Of The Internet Universe ? Part 3

This is part three of the article series and deals... Read More

Do-It-Yourself Search Engine Optimization

Search engine optimization, or SEO, is big business. If you... Read More

Getting Honest With The Search Engines

Getting Honest With The Search EnginesI spend a lot of... Read More

Link Building for Hilltop

Hilltop is one of the major concepts underpinning Google's search... Read More

Your Online Home-Based Business Can Lead To Search Engine Stress

As a member of several search engine optimization forums, I... Read More

What Size Body Section Ranks Highest?

This is another one of the controversial questions in many... Read More

List Building vs. Search Engine Optimization

It seems the excitement about search engine optimization fades in... Read More

Googles New SEO Rules

Google has recently made some pretty significant changes in its... Read More

9 Steps to Getting Better Search Engine Rankings

You finally have a website and you are ready to... Read More

How Do I Improve My Web Site Conversion Rate? Part 3

Question 1How do keywords effect your conversion rate in terms... Read More

How To Improve Your Search Engine Ranking

With search engines like Google currently indexing over 8 billion... Read More

Complete Web-Site Optimization For Search Engines (Part 2)

Source code optimization.{title}...{/title}This tag is to be a winner. This... Read More

4 Tricks For Lightning Fast Indexing

The biggest problem that most are running into seems to... Read More

Web Site Copy is about More Than Keywords

Let's say you are writing a web site to sell... Read More

Do You Submit Your Website To All The Search Engines?

There is really only one search engine to worry about... Read More