Do Search Engines Like Your Web Site?

Between 75% and 98.8% of visitors to Web sites come from searches made at search engines. If you're going to get high levels of traffic - and hence the levels of ROI you're looking for - it's very important that the search engines can access all the information on your Web site.

Do the search engines know about all of your pages?

You can find out which pages on your site the search engines know about by using a special search. If you search for 'site:' and your Web site address, the search engine will tell you all of the pages on your Web site it knows about.

For example, search for: site:webpositioningcentre.co.uk in Google. Yahoo or MSN Search, and it will tell you how many pages they know about.

If the search engines haven't found some of the pages on your Web site, it is probably because they are having trouble spidering them. ('Spidering' is when the search engine uses an automated robot to read your Web pages.)

Spiders work by starting off on a page which has been linked to by another Web site, or that has been submitted to the search engine. They then read and follow any links they find on the page, gradually working their way through your whole Web site.

At least, that's the theory.

The problem is, it's easy to confuse the spiders - especially as they are designed to be wary of following certain kinds of link.

Links which confuse spiders

If your links are within a large chunk of JavaScript code, the spider may not be able to find them, and will not be able to follow the links to your other pages.

This can happen if you have 'rollovers' as your navigation - for instance, pictures that change colour or appearance when you hover your mouse pointer over them. The JavaScript code that makes this happen can be convoluted enough for the spiders to ignore it rather than try to find links inside.

If you think your rollovers are blocking your site from being spidered, you will need to talk to your Web designers about changing the code in to a 'clean link' - a standard HTML link, with no extra code around it - that is much easier for the spiders to follow.

Links like these will look something like this:

Home Page

Page addresses to avoid

Spiders will also ignore pages if they don't like the URL (the address needed to find the page).

For example, a Web site that has URLs containing several variables can cause spiders to ignore the page content. You can spot pages like these as they have a ? in them, and &, for instance:

http://webpositioningcentre.co.uk/index.php?page=12&cat=23&jib=c

This URL has three variables, the parts with the = in them, between the ? and &s. We find that if a page has one variable, or even two, the top search engines will spider them without any problems. But if a URL has more than that, often the search engines will not spider them.

Spiders particularly avoid URLs that look like they have 'session IDs' in them. They look something like this:

http://webpositioningcentre.co.uk/index.php?page=12&id=29c8d7r2398jk27897a8

The set of numbers and letters do not make much sense to humans, but some Web sites use them to keep track of who you are, as you click through their Web site.

Spiders will generally avoid URLs with Session IDs in them, so if your Web site has them, you need to talk to the people who developed the site about re-writing it so they do not use these IDs, or at least that you can get around the Web site without them.

Clean links = happy spiders

If you use clean, easy to follow links without several variables in them, your Web site should be spidered without problem. There are, of course, many other facets to successful Search Engine Optimization, but if the search engines can't spider your content, your site will fall at the first hurdle.

Paul Silver and David Rosam are Head of Technical SEO and Head of SEO Copywriting at Web Positioning Centre (http://webpositioningcentre.co.uk). Paul has been involved with the Web commercially since 1996 and David has been writing marketing copy for 20 years, and writing for the Web for a decade.

In The News:

6 Tips for YouTube SEO Optimization  Business 2 Community
What is SEO?  Business 2 Community
How Can I Set Up an SEO Agency?  Search Engine Journal
WhaTech  WhaTech
New Solar Energy Optics  Off Grid Energy Independence
How Google Improves Search Results  Search Engine Journal
How to Dominate Ecommerce in 2020  Search Engine Journal
SEO in 2020: Going Beyond Google  Search Engine Journal

Boost Your Search Engine Ranking And Generate Free Traffic With Reciprocal Links

Reciprocal links are an important step in your overall plan... Read More

Why Do You Want to Link With A Home Business And Affiliate Website?

No, it's not a general question for all and sundry.... Read More

The Myth of Search Engine Submission

Contrary to what most people think, it is not necessary... Read More

How To Become an SEO Expert - 5 Secrets That Will Allow You to Outperform 95% of All Webmasters

Becoming an SEO expert, or a search engine optimization expert,... Read More

What is PageRank?

Chances are you have been on the Internet and have... Read More

Tread Towards A Successful ?Internet Research?

Internet is a terrific resource containing billions of web pages... Read More

Keywords, Choose Them Wisely

By now you have likely heard that keywords and keyword... Read More

SEO Expert Guide - Search Engines Explained (part 1/10)

Before we explore the world of search engine optimization, it... Read More

Search Engine Position Report

Since search engines are the first stop for people on... Read More

Getting Specific With Local Search Engine Optimization

Localized search engine optimization is often overlooked as an excellent... Read More

The Latest Craze: Local Search, 7 Steps to Being #1 in Your Local Market

Anyone would agree that it is much easier to be... Read More

Does Javascript Affect Ranking?

Almost all SEO's agree that using too much javascript can... Read More

Linking for Traffic not Positioning!

With more and more experts and search engine enthusiastsclaiming the... Read More

Search Engine Optimization --- An Overview

Search engine optimization is one of most popular online marketing... Read More

SEOing A Flash Intro

IntroductionWhy does a FLASH intro need SEO? Flash, the great... Read More

Its Not Just All About Google Anymore

Those webmasters that stick to the old ways and focus... Read More

Recommended Tools When You Put On The SEO Cap For Your Web Site

The name of the game is search engine optimization and... Read More

Is Google Having a Tough Time with Their Website Limit?

If you are one to pay attention to what happens... Read More

The Small Screen SEO!

First of all, What is SEO? SEO stands for Search... Read More

SEO Basics - 8 Key Strategies For Online Business Success

The world of search engine optimization (SEO) can be a... Read More

5 Things to Keep an Eye on in the SEO World in 2005...

After the latest PR update at Google and MSN's beta... Read More

Is Reciprocal Link Exchange Dead?

The sun seems to be setting on the world of... Read More

Dont Focus Too Much on Your Internet Business Website Ranking

No doubt, having a high search engine ranking is very... Read More

Submitting Your Site to Search Engines and Directories

Regardless of which type of business you currently have, you... Read More

SEO Expert Explains on how to Restore a Website and Remove Sandbox Effect from Your Website

The sandbox effect or (site getting banned on google) has... Read More