Deprecated: mysql_connect(): The mysql extension is deprecated and will be removed in the future: use mysqli or PDO instead in /home/a26f9f83/public_html/articles/includes/config.php on line 159
Search Engine Robots - How They Work, What They Do (Part I) > NetSparsh - Viral Content you Love & Share

Search Engine Robots - How They Work, What They Do (Part I)

Automated search engine robots, sometimes called "spiders" or "crawlers", are the seekers of web pages. How do they work? What is it they really do? Why are they important?

You'd think with all the fuss about indexing web pages to add to search engine databases, that robots would be great and powerful beings. Wrong. Search engine robots have only basic functionality like that of early browsers in terms of what they can understand in a web page. Like early browsers, robots just can't do certain things. Robots don't understand frames, Flash movies, images or JavaScript. They can't enter password protected areas and they can't click all those buttons you have on your website. They can be stopped cold while indexing a dynamically generated URL and slowed to a stop with JavaScript navigation. How Do Search Engine Robots Work?

Think of search engine robots as automated data retrieval programs, traveling the web to find information and links.

When you submit a web page to a search engine at the "Submit a URL" page, the new URL is added to the robot's queue of websites to visit on its next foray out onto the web. Even if you don't directly submit a page, many robots will find your site because of links from other sites that point back to yours. This is one of the reasons why it is important to build your link popularity and to get links from other topical sites back to yours.

When arriving at your website, the automated robots first check to see if you have a robots.txt file. This file is used to tell robots which areas of your site are off-limits to them. Typically these may be directories containing only binaries or other files the robot doesn't need to concern itself with.

Robots collect links from each page they visit, and later follow those links through to other pages. In this way, they essentially follow the links from one page to another. The entire World Wide Web is made up of links, the original idea being that you could follow links from one place to another. This is how robots get around.

The "smarts" about indexing pages online comes from the search engine engineers, who devise the methods used to evaluate the information the search engine robots retrieve. When introduced into the search engine database, the information is available for searchers querying the search engine. When a search engine user enters their query into the search engine, there are a number of quick calculations done to make sure that the search engine presents just the right set of results to give their visitor the most relevant response to their query.

You can see which pages on your site the search engine robots have visited by looking at your server logs or the results from your log statistics program. Identifying the robots will show you when they visited your website, which pages they visited and how often they visit. Some robots are readily identifiable by their user agent names, like Google's "Googlebot"; others are bit more obscure, like Inktomi's "Slurp". Still other robots may be listed in your logs that you cannot readily identify; some of them may even appear to be human-powered browsers.

Along with identifying individual robots and counting the number of their visits, the statistics can also show you aggressive bandwidth-grabbing robots or robots you may not want visiting your website. In the resources section of the end of this article, you will find sites that list names and IP addresses of search engine robots to help you identify them. How Do They Read The Pages On Your Website?

When the search engine robot visits your page, it looks at the visible text on the page, the content of the various tags in your page's source code (title tag, meta tags, etc.), and the hyperlinks on your page. From the words and the links that the robot finds, the search engine decides what your page is about. There are many factors used to figure out what "matters" and each search engine has its own algorithm in order to evaluate and process the information. Depending on how the robot is set up through the search engine, the information is indexed and then delivered to the search engine's database.

The information delivered to the databases then becomes part of the search engine and directory ranking process. When the search engine visitor submits their query, the search engine digs through its database to give the final listing that is displayed on the results page.

The search engine databases update at varying times. Once you are in the search engine databases, the robots keep visiting you periodically, to pick up any changes to your pages, and to make sure they have the latest info. The number of times you are visited depends on how the search engine sets up its visits, which can vary per search engine.

Sometimes visiting robots are unable to access the website they are visiting. If your site is down, or you are experiencing huge amounts of traffic, the robot may not be able to access your site. When this happens, the website may not be re-indexed, depending on the frequency of the robot visits to your website. In most cases, robots that cannot access your pages will try again later, hoping that your site will be accessible then.

Resources

*SpiderSpotting - Search Engine Watch http://searchenginewatch.com/webmasters/spiders.html

*Robotstxt.org List of robots and protocols for setting up a robots.txt file. http://www.robotstxt.org/

*Spider-Food Tutorials, forums and articles about Search Engine spiders and Search Engine Marketing. http://spider-food.net/

*Spiderhunter.com Articles and resources about tracking Search Engine spiders. http://www.spiderhunter.com/

*Sim Spider Search Engine Robot Simulator Search Engine World has a spider that simulates what the Search Engine robots read from your website. http://www.searchengineworld.com/cgi-bin/sim_spider.cgi

Daria Goetsch is the founder and Search Engine Marketing Consultant for Search Innovation Marketing, a Search Engine Optimization company serving small businesses. She has specialized in Search Engine Promotion since 1998, including three years as the Search Engine Specialist for O'Reilly Media, Inc., a technical book publishing company.

Copyright © 2002-2005 Search Innovation Marketing. http://www.searchinnovation.com All Rights Reserved.

Permission to reprint this article is granted if the article is reproduced in its entirety, without editing, including the bio information. Please include a hyperlink to http://www.searchinnovation.com when using this article in newsletters or online.

In The News:

This RSS feed URL is deprecated, please update. New URLs can be found in the footers at https://news.google.com/news

Business.com

Time to Hire an SEO Expert? - Business.com
Business.com
You can teach yourself the basics of good search engine optimization, but sometimes it is worth it to hire a professional SEO expert.
The great Google game - The NegotiatorThe Negotiator

all 2 news articles »

Search Engine Journal

7 Signs Your Law Firm May Need a New SEO Partner
Search Engine Journal
The legal industry has a serious problem. Law firms and attorneys around the country continue to get taken advantage of by “churn and burn” SEO agencies. These agencies, which are run by self-proclaimed SEO experts, in most cases, promise big results ...


Forbes

Google's Top Ranking Factors For Local SEO - Forbes
Forbes
With so many potential clients turning to local search to find business or legal services, it's imperative that the online presence of your business or practice is ...
Infographic: 72 Stats to understand SEO in 2018AMEinfo
Tips for helping women in business boost their SEO rankingFinancy
SEO/PPC ManagerBizcommunity.com
Business 2 Community -MarketingProfs.com
all 17 news articles »

Search Engine Journal

YouTube SEO from Basic to Advanced: How to Optimize Your Videos
Search Engine Journal
These basic and advanced video optimization tips and tactics will help ensure your videos will be easily findable on YouTube and other search engines.

and more »

Search Engine Journal

How Important Is an H1 Tag for SEO?
Search Engine Journal
Editor's note: “Ask an SEO” is a weekly column by technical SEO expert Jenny Halasz. Come up with your hardest SEO question and fill out our form. You might see your answer in the next #AskanSEO post!


Practical Ecommerce

SEO Link Building Tactics for Ecommerce Businesses
Practical Ecommerce
But ecommerce sites often struggle with the practice. This is especially true for links to product detail pages. It is nonetheless necessary work. In this post, I'll describe several link-building tactics for ecommerce SEO. While none of these are ...


Search Engine Journal

5 Ways to Move Your SEO Programs to Target the Top of the Buying Funnel
Search Engine Journal
For years, SEO professionals have been tackling the bottom of the sales funnel. We'll give lip service to branding and content marketing targeted at consumers higher in the funnel – those just starting the buyer's journey. But it's much easier to ...


Entrepreneur

5 SEO Mistakes that People are Still Doing in 2018
Entrepreneur
Google has been updating its algorithms with some strict rules and keeping SEO professionals on their toes. Panda, Hummingbird, and Penguin are all the different names given to updates done in the search algorithms by Google. This made SEO serious ...
How to be proactive and prevent a negative SEO campaign — Part 3Search Engine Land
A Beginner's Guide to SEO in a Machine Learning WorldSearch Engine Journal
The core pillars of technical international SEOMarketing Tech
MediaPost Communications
all 14 news articles »

TNW

Listen to Google — UX and SEO are a match made in heaven
TNW
To the layman UX and SEO (if they even understand what they mean) sound like two entirely unrelated things. The one is about some plumber jabbing away at their phone, while the other is about satisfying the algorithms of a company in Silicon Valley.


Entrepreneur

The Startup's Handbook to Perfecting the SEO Strategy
Entrepreneur
Many business owners and founders still make the mistake of assuming SEO to be an offshoot of web development. Understand that SEO is an inseparable and ongoing part of marketing. It is much more than adding titles and meta descriptions to your web ...
4 Very Popular SEO Beliefs That Make No Sense At AllSearch Engine Journal
4 Ways Sentiment Analysis Is Changing SEOSearch Engine People (blog)
What You Should Know About Small Business SEO at a Glance [Infographic]The National Law Review
The SEM Post
all 8 news articles »
Google News

Being dumped by Google? Learn how to avoid becoming a victim next time around!

After Google latest update nicknamed "Florida", many webmasters discovered that... Read More

Is My Search Engine Optimization Working?

You've just built a website and can't wait to start... Read More

Link Popularity: Why Its The Best Investment You Can Do For Your Business

More and more search engines rank your web pages based... Read More

Optimze Your Web Site on a Shoe String Budget

Let me start off by saying I'm not a marketing... Read More

Search Wars! - MSNs Opening Salvo

With all the recent publicity given to Google as the... Read More

Search Engine Monopoly

Ever wondered how one search engine is dominating the entire... Read More

The Other Side of the Search Gods Abracadabra!

Thousands of servers ...billions of web pages.... the possibility of... Read More

Search Engine Optimization: Creative Ways To Acquire Natural Back Links

Search engines use algorithms calculate the order in which the... Read More

What Is Waiting for Us? Tomorrows SEO Industry

Today, SEO is swiftly approaching saturation point. More and more... Read More

Why SEO (as we know it) is Doomed to Failure and How You Can Avoid the Trap

Search Engine Optimization (SEO) has become one of the biggest... Read More

Top Search Engine Ranks- The Only Secret You Need- Explained: Part 1

The top three search pages- the only place you'll be... Read More

Why Search Engine Optimization is Not Enough

OK. So you've created a nice website with lots of... Read More

Webmaster-Friendly Google Operators

There are a lot of operators (commands you can enter... Read More

The Great Search Engine War, Where Content is King

When search engines first appeared, they were simple affairs consisting... Read More

Search Engine Optimization for Everyone

There are many facets to SEO and the search engines... Read More

Search Bots, Crawlers, and Spiders

If you are a webmaster and you review your logs,... Read More

Search Engine Optimization For Blogs

Blogging software is really a simple Content Management System (CMS)... Read More

Alert Marketing - Get Google Search Results By E-mail

Sometimes our jobs as marketers means we need to look... Read More

Link Popularity: Improve Your Search Engine Rankings

What is link popularity?Link Popularity is simply the total number... Read More

How to Prevent Duplicate Content with Effective Use of the Robots.txt and Robots Meta Tag

Duplicate content is one of the problems that we regularly... Read More

Monitor and Increase Your Search Engine Visibility with the DIY SEO Tools

In this three part article, you'll find many tools that... Read More

Search Engine Optimization for Dummies

Lets face it. Search Engines rule the World Wide Web... Read More

Crash Course in Getting a #1 Google Ranking

First, here's the rundown of some of the terminology I'm... Read More

The Google Strategy

Webmasters across the Internet were totally floored by what happened... Read More

These 7 Back Link Strategies Will Get You a Top Ranking on Google Guaranteed

Google use a very complex function to determine which search... Read More