Search Bots, Crawlers, and Spiders

If you are a webmaster and you review your logs, often you will see a bunch of really strange hits. They aren't humans, you can't tell their operating system or their browser! Who are these pesky little creatures who rummage around the internet all the time?

Not quite sure what I am talking about? Here is a few examples of various bots searching my website: (
msnbot/1.0 (+
This is the MSN Search bot. (
Mozilla/5.0 (compatible; Yahoo! Slurp;
This is Yahoos Search Bot. (
This is Googles bot, that searches your webpages for AdSense.

What is a Bot, Crawler, Spider?
These terms are all the same, they all refer to an automated program that goes from website to website caching and processing the pages for search engines. As you know, "WWW" means World Wide Web, thus "Spider" seemed like an appropriate term. Crawler is another term that just describes what it does, crawling from site to site and page to page endlessly. Bot, is actually short for "robot" and again is just an automated program to index websites.

What is the purpose of a Spider?
A spider looks at all the pages of your website, and uses that information to rank you in search engines (how high you will list in a search result), and cache a copy of your page on their server for quick reference, and if your site ever goes down. Spiders jump from link to link on the Internet and run endlessly, even if you never submit your website to a search engine, odds are your site will still be spidered.

Can I stop bots and spiders from searching my website?
Yes and no. Legitimate spiders are run by reputable organizations that follow certain rules. For instance, most companies have a policy that their robot will search for a file called "robots.txt" in the root of your website. This text file is filled with information telling the bots what and what not is allowed to be viewed. Unfortunately, there are also bad bots out there, they search the internet harvesting e-mail addresses for spam and other bad things, these bots often don't comply with the "robots.txt" standard.

How many bots are there?
It's impossible to guess how many bots are out there searching websites. On any given day I will get roughly 10 different ones check my website. Some of them only search one or two pages, others go over my entire website. Not all of them give you a good description of what they do, or who owns them. If you cut and paste their name and IP address in to Google, quite often you can find more information about what they do.

How can I get my site spidered?
As I mentioned before, if your website is up long enough, it "will" get spidered eventually. However, if you want to ensure that it gets done within a few months, go to the various search engine websites and look for the "Add URL" or "Suggest a Link" pages. DMOZ is one of the big directories which you should submit your site. When you sign up for these search engines, your website is automatically queued up to be spidered. It may take several weeks or months to actually start showing up on the search engine, even after you see the robot spidering your website.

What about pay search engines?
There are a bunch of different search engines that make you pay to have your website listed. I personally don't support these search engines, I find that most people use the big free search engines anyway. However, if you do wish to get included in some search engines faster, many have payment options which will get your site listed within a couple of days.

Ken Dennis

In The News:

Can SEO Be Made Predictable?  Search Engine Journal
What Does It Mean to ‘Do SEO’?  Search Engine Journal
SEO Hamilton  Search Engine People
My experience with SEO  Practical Ecommerce
What Is Forensic SEO?  Search Engine Journal
Can SEO Have a Negative Effect?  Search Engine Journal
7 Coding Barriers to SEO Success  Practical Ecommerce
5 Local SEO Tips for Startups  Business 2 Community
5 Big Content Trends for 2020  Search Engine Journal
What Is a Google Penalty in SEO?  Search Engine Journal
The High Cost of Ignoring SEO  Practical Ecommerce
Google Shares Top 3 SEO Factors  Search Engine Journal
What's the Value of SEO?  Practical Ecommerce

HTML Title Tags Dictate Your Rankings

There are a lot of things in Search Engine Optimization... Read More

.com Not Listed in Regional Yahoo? Don?t Despair!

If you're a non-American business with a .com web address,... Read More

Search Engine Optimization --- An Overview

Search engine optimization is one of most popular online marketing... Read More

Keyword Popularity Check

Before using keywords for your site, it is always better... Read More

How Real SEO Analysis Works

If you're serious about SEO, you need to know how... Read More

Its All in the Title

Unfortunately, we don't live in a perfect world. You may... Read More

How Ive Maintained 7 Top Ten Google Rankings For Nine Months

Back in November 2004 I discovered a way to get... Read More

Sales And Crawlers, Update! Update! Update!

The importance to the algorithmic web crawlers that speed throughout... Read More

SEO #4: Off-page Optimization

Yesterday you should have read the third course out of... Read More

Search Engine Spiders Lost Without Guidance - Post This Sign!

The robots.txt file is an exclusion standard required by all... Read More

Increase Web Site Sales with a SEO Proposal - Part 1

You can easily get confused by all the search engine... Read More

These 7 Back Link Strategies Will Get You a Top Ranking on Google Guaranteed

Google use a very complex function to determine which search... Read More

How To Choose Keywords Before they Skyrocket in Popularity

Long before the days of researching phrases with the helpful... Read More

Creative Search Engine Optimization ? A Case Study

Search engine optimization this and search engine optimization that. You... Read More

Copywriters: Make Friends with Search Engine Optimization

I don't know about you, but I felt a lot... Read More

How to Increase Alexa Ranking of Your Website

Alexa toolbar also useful to Browse expired websites database. Many... Read More

Answers Count - Matching Keyword and Phrase Density

What's more important to your business success - the question... Read More

An SEO Checklist

Would you like a checklist of the important steps to... Read More

Ten Critical Website and Search Engine Promotion Mistakes

When it comes to promoting your website, you might think... Read More


The Internet may have started as the fervent brainchild of... Read More

The Purpose of PageRank?

I am sure the first thought provoking question that popped... Read More

Keyword Optimization - How To Achieve It

The most important thing you can do for your business... Read More

SEO: The Good, The Bad And The Ugly

I seem to have created quite a stir, on a... Read More

Are You Losing The Battle For Search Engine Traffic?

Search engine traffic should be a priority for any online... Read More

Dealing With Search Engine Stress In A Home-Based Business

As a member of several search engine optimization forums, I... Read More