Search Bots, Crawlers, and Spiders

If you are a webmaster and you review your logs, often you will see a bunch of really strange hits. They aren't humans, you can't tell their operating system or their browser! Who are these pesky little creatures who rummage around the internet all the time?

Not quite sure what I am talking about? Here is a few examples of various bots searching my website: (
msnbot/1.0 (+
This is the MSN Search bot. (
Mozilla/5.0 (compatible; Yahoo! Slurp;
This is Yahoos Search Bot. (
This is Googles bot, that searches your webpages for AdSense.

What is a Bot, Crawler, Spider?
These terms are all the same, they all refer to an automated program that goes from website to website caching and processing the pages for search engines. As you know, "WWW" means World Wide Web, thus "Spider" seemed like an appropriate term. Crawler is another term that just describes what it does, crawling from site to site and page to page endlessly. Bot, is actually short for "robot" and again is just an automated program to index websites.

What is the purpose of a Spider?
A spider looks at all the pages of your website, and uses that information to rank you in search engines (how high you will list in a search result), and cache a copy of your page on their server for quick reference, and if your site ever goes down. Spiders jump from link to link on the Internet and run endlessly, even if you never submit your website to a search engine, odds are your site will still be spidered.

Can I stop bots and spiders from searching my website?
Yes and no. Legitimate spiders are run by reputable organizations that follow certain rules. For instance, most companies have a policy that their robot will search for a file called "robots.txt" in the root of your website. This text file is filled with information telling the bots what and what not is allowed to be viewed. Unfortunately, there are also bad bots out there, they search the internet harvesting e-mail addresses for spam and other bad things, these bots often don't comply with the "robots.txt" standard.

How many bots are there?
It's impossible to guess how many bots are out there searching websites. On any given day I will get roughly 10 different ones check my website. Some of them only search one or two pages, others go over my entire website. Not all of them give you a good description of what they do, or who owns them. If you cut and paste their name and IP address in to Google, quite often you can find more information about what they do.

How can I get my site spidered?
As I mentioned before, if your website is up long enough, it "will" get spidered eventually. However, if you want to ensure that it gets done within a few months, go to the various search engine websites and look for the "Add URL" or "Suggest a Link" pages. DMOZ is one of the big directories which you should submit your site. When you sign up for these search engines, your website is automatically queued up to be spidered. It may take several weeks or months to actually start showing up on the search engine, even after you see the robot spidering your website.

What about pay search engines?
There are a bunch of different search engines that make you pay to have your website listed. I personally don't support these search engines, I find that most people use the big free search engines anyway. However, if you do wish to get included in some search engines faster, many have payment options which will get your site listed within a couple of days.

Ken Dennis

In The News:

Top 5 Challenges of Enterprise SEO  Search Engine Journal
It's fall: All the warm liquids, please  Chemical & Engineering News
Are Older Domains Better For SEO?  Business 2 Community
SEO in 2020: Going Beyond Google  Search Engine Journal
What Is Enterprise SEO?  Search Engine Journal

Possibly The Biggest Misconception About Ranking Well In The Search Engines

Onpage search engine optimization are things that you can change... Read More

How MSN and Yahoo Sells Your Traffic

Yes, it really happens. Now you might find it hard... Read More

SEO Help: Dont Try to Fool the Search Engines

Writing articles is all the rage these days on the... Read More

Why Is SEO So Important To Your Site?

You have heard the phrase LOCATION LOCATION LOCATION. But wait,... Read More

Keyword Ownership: What It Is And Where Its Headed

Have you ever got one of those silly emails that... Read More

Link Popularity: Why Its The Best Investment You Can Do For Your Business

More and more search engines rank your web pages based... Read More

Link Popularity - Basic Overview

There are many techniques that SEM/SEO experts use to optimize... Read More

Link Building Services

In today scenario when we talk about Search Engine Optimization,... Read More

Does Javascript Affect Ranking?

Almost all SEO's agree that using too much javascript can... Read More

Keywords: The First Step To Recognition

Open Wordtracker [ ] and you'll see... Read More

5 Things to Keep an Eye on in the SEO World in 2005...

After the latest PR update at Google and MSN's beta... Read More

The Business Case for SEO

It's interesting how potential clients have preconceived notions about which... Read More

Effective Search Engine Use

The Internet is a wonderful place full of resources that... Read More

Marketing Articles: Getting A Better Search Engine Rank For All Of Your Pages!

In one of my articles, I discussed how to market... Read More

Making Money with Popular Search Engines

With so many internet and home business opportunities on the... Read More

Link Horse Trading For The PR Challenged

After 105 days Google finally updated PR. And it's about... Read More

SEO for CEOs ? Search Engine Optimization Unmasked for CEOs

If you're like most other CEOs, the term "search engine... Read More

Playing By Googles Rules

As the undisputable leader in search engines, Google places a... Read More

Website Optimization, Good Overall Optimization is Key

Good overall optimization, the right keyword phrases and quality content... Read More

Internet Marketing and SEO

Have you ever seen any email offers of getting you... Read More

Five Ways To Win The Favor Of Search Engines

You've got a cool new website with all the works:... Read More

An Easy Way Not to Get Banned by Google

Strategic search engine optimization involves far more than keyword research,... Read More

SEO Expert Guide - Ongoing Monitoring of Results (part 9/10)

In the Guide, you have, so far, learnt how to... Read More

8 ways to build a really bad web site for Search Engines

Some web sites receive hundreds or thousands of unique visitors... Read More

The Changing Face of Search Engine Optimization

With the ever evolving internet market for just about anything... Read More