Search Bots, Crawlers, and Spiders

If you are a webmaster and you review your logs, often you will see a bunch of really strange hits. They aren't humans, you can't tell their operating system or their browser! Who are these pesky little creatures who rummage around the internet all the time?

Not quite sure what I am talking about? Here is a few examples of various bots searching my website: (
msnbot/1.0 (+
This is the MSN Search bot. (
Mozilla/5.0 (compatible; Yahoo! Slurp;
This is Yahoos Search Bot. (
This is Googles bot, that searches your webpages for AdSense.

What is a Bot, Crawler, Spider?
These terms are all the same, they all refer to an automated program that goes from website to website caching and processing the pages for search engines. As you know, "WWW" means World Wide Web, thus "Spider" seemed like an appropriate term. Crawler is another term that just describes what it does, crawling from site to site and page to page endlessly. Bot, is actually short for "robot" and again is just an automated program to index websites.

What is the purpose of a Spider?
A spider looks at all the pages of your website, and uses that information to rank you in search engines (how high you will list in a search result), and cache a copy of your page on their server for quick reference, and if your site ever goes down. Spiders jump from link to link on the Internet and run endlessly, even if you never submit your website to a search engine, odds are your site will still be spidered.

Can I stop bots and spiders from searching my website?
Yes and no. Legitimate spiders are run by reputable organizations that follow certain rules. For instance, most companies have a policy that their robot will search for a file called "robots.txt" in the root of your website. This text file is filled with information telling the bots what and what not is allowed to be viewed. Unfortunately, there are also bad bots out there, they search the internet harvesting e-mail addresses for spam and other bad things, these bots often don't comply with the "robots.txt" standard.

How many bots are there?
It's impossible to guess how many bots are out there searching websites. On any given day I will get roughly 10 different ones check my website. Some of them only search one or two pages, others go over my entire website. Not all of them give you a good description of what they do, or who owns them. If you cut and paste their name and IP address in to Google, quite often you can find more information about what they do.

How can I get my site spidered?
As I mentioned before, if your website is up long enough, it "will" get spidered eventually. However, if you want to ensure that it gets done within a few months, go to the various search engine websites and look for the "Add URL" or "Suggest a Link" pages. DMOZ is one of the big directories which you should submit your site. When you sign up for these search engines, your website is automatically queued up to be spidered. It may take several weeks or months to actually start showing up on the search engine, even after you see the robot spidering your website.

What about pay search engines?
There are a bunch of different search engines that make you pay to have your website listed. I personally don't support these search engines, I find that most people use the big free search engines anyway. However, if you do wish to get included in some search engines faster, many have payment options which will get your site listed within a couple of days.

Ken Dennis

In The News:

SEO Tools Aren't Enough for Success  Search Engine Journal
How to Do SEO for Niche Markets  Search Engine Journal
How to Use GitHub for Enterprise SEO  Search Engine Journal
3 Must-Know SEO Trends for Marketers  Marketing Tech Outlook
Amazon SEO in 2020  Jungle Scout
SEO in 2020: Going Beyond Google  Search Engine Journal
5 Ways SEO & Web Design Go Together  Search Engine Journal
Can SEO Be Made Predictable?  Search Engine Journal
Google's Advice on How to Hire an SEO  Search Engine Journal
Moving a Company to an SEO Focus  Search Engine Journal

Buzzwords vs Effective SEO Keywords

Ever see a website that seems to speak a foreign... Read More

Optimize your Search Engine Placement Five Easy Ways

If you are like me, you created web site metatags... Read More

SEO Expert Guide - Ongoing Monitoring of Results (part 9/10)

In the Guide, you have, so far, learnt how to... Read More

The Search Engine Optimization Secret that Everybody Ignores

Search engine optimization is a very critical task in the... Read More

How to Increase Alexa Ranking of Your Website

Alexa toolbar also useful to Browse expired websites database. Many... Read More

Which Search Engine Optimization Services to choose, Google OR Yahoo?

Search Engine Optimization is emerging as the most powerful form... Read More

Advertise Locally Using Search Engines

While search engine advertising has been a great advertising medium... Read More

Meta Tags - An Important Part of Every Web Page

Meta tags are an absolute must from a search engine... Read More

The Search Engine Secret That Is No Secret At All

It's common knowledge - we all know that it is... Read More

Diary of a Google Gazumpee

Back in November, when the Google Dance began, Barry Lloyd... Read More

Easy Steps to Get Onto Google Top Search Pages

To get on Google's top pages can be accomplished by... Read More

How Do I Improve My Web Site Conversion Rate? Part 3

Question 1How do keywords effect your conversion rate in terms... Read More

Submit All Of Your Pages And Watch Your Traffic Grow

Everyone is looking for "secrets" about how to get more... Read More

Website Optimization, Good Overall Optimization is Key

Good overall optimization, the right keyword phrases and quality content... Read More

Writing SEO Copy ? 8 Steps to Success

We all know that the lion's share of web traffic... Read More

Getting To Know Google

Having greatly benefited from my relationship with Google in the... Read More

Building Link Popularity with Topical Articles

One of the important factors in ranking high in search... Read More

Picking Keywords for SEO ? A Different View

The first step to developing any search engine optimization effort... Read More

An SEO Checklist

Search engine optimization is on every webmaster's mind these days.... Read More

The Top 3 Mistakes That Can Ruin Your Websites Search Engine Rankings- and How to Fix Them!

Getting your website up and running is hard enough. After... Read More

Search Engine Optimization: What Is It?

Search Engine Optimization is the creation of a web page,... Read More

Duplicate Content Penalty - How to Lose Google Ranking Fast!

Duplicate content penalty. Ever heard of it? This penalty is... Read More

SEO Expert Guide - Conclusions (part 10/10)

As you have seen throughout the guide, search engine optimization... Read More

10 Easy Steps to Boost Your Search Engine Rankings!

In order that someone finds your website and buys your... Read More

8 Essential SEO techniques

1) Title Tag - The title tag is the most... Read More