Deprecated: mysql_connect(): The mysql extension is deprecated and will be removed in the future: use mysqli or PDO instead in /home/a26f9f83/public_html/articles/includes/config.php on line 159
Search Engine Robots - How They Work, What They Do (Part I) > NetSparsh - Viral Content you Love & Share

Search Engine Robots - How They Work, What They Do (Part I)

Automated search engine robots, sometimes called "spiders" or "crawlers", are the seekers of web pages. How do they work? What is it they really do? Why are they important?

You'd think with all the fuss about indexing web pages to add to search engine databases, that robots would be great and powerful beings. Wrong. Search engine robots have only basic functionality like that of early browsers in terms of what they can understand in a web page. Like early browsers, robots just can't do certain things. Robots don't understand frames, Flash movies, images or JavaScript. They can't enter password protected areas and they can't click all those buttons you have on your website. They can be stopped cold while indexing a dynamically generated URL and slowed to a stop with JavaScript navigation. How Do Search Engine Robots Work?

Think of search engine robots as automated data retrieval programs, traveling the web to find information and links.

When you submit a web page to a search engine at the "Submit a URL" page, the new URL is added to the robot's queue of websites to visit on its next foray out onto the web. Even if you don't directly submit a page, many robots will find your site because of links from other sites that point back to yours. This is one of the reasons why it is important to build your link popularity and to get links from other topical sites back to yours.

When arriving at your website, the automated robots first check to see if you have a robots.txt file. This file is used to tell robots which areas of your site are off-limits to them. Typically these may be directories containing only binaries or other files the robot doesn't need to concern itself with.

Robots collect links from each page they visit, and later follow those links through to other pages. In this way, they essentially follow the links from one page to another. The entire World Wide Web is made up of links, the original idea being that you could follow links from one place to another. This is how robots get around.

The "smarts" about indexing pages online comes from the search engine engineers, who devise the methods used to evaluate the information the search engine robots retrieve. When introduced into the search engine database, the information is available for searchers querying the search engine. When a search engine user enters their query into the search engine, there are a number of quick calculations done to make sure that the search engine presents just the right set of results to give their visitor the most relevant response to their query.

You can see which pages on your site the search engine robots have visited by looking at your server logs or the results from your log statistics program. Identifying the robots will show you when they visited your website, which pages they visited and how often they visit. Some robots are readily identifiable by their user agent names, like Google's "Googlebot"; others are bit more obscure, like Inktomi's "Slurp". Still other robots may be listed in your logs that you cannot readily identify; some of them may even appear to be human-powered browsers.

Along with identifying individual robots and counting the number of their visits, the statistics can also show you aggressive bandwidth-grabbing robots or robots you may not want visiting your website. In the resources section of the end of this article, you will find sites that list names and IP addresses of search engine robots to help you identify them. How Do They Read The Pages On Your Website?

When the search engine robot visits your page, it looks at the visible text on the page, the content of the various tags in your page's source code (title tag, meta tags, etc.), and the hyperlinks on your page. From the words and the links that the robot finds, the search engine decides what your page is about. There are many factors used to figure out what "matters" and each search engine has its own algorithm in order to evaluate and process the information. Depending on how the robot is set up through the search engine, the information is indexed and then delivered to the search engine's database.

The information delivered to the databases then becomes part of the search engine and directory ranking process. When the search engine visitor submits their query, the search engine digs through its database to give the final listing that is displayed on the results page.

The search engine databases update at varying times. Once you are in the search engine databases, the robots keep visiting you periodically, to pick up any changes to your pages, and to make sure they have the latest info. The number of times you are visited depends on how the search engine sets up its visits, which can vary per search engine.

Sometimes visiting robots are unable to access the website they are visiting. If your site is down, or you are experiencing huge amounts of traffic, the robot may not be able to access your site. When this happens, the website may not be re-indexed, depending on the frequency of the robot visits to your website. In most cases, robots that cannot access your pages will try again later, hoping that your site will be accessible then.

Resources

*SpiderSpotting - Search Engine Watch http://searchenginewatch.com/webmasters/spiders.html

*Robotstxt.org List of robots and protocols for setting up a robots.txt file. http://www.robotstxt.org/

*Spider-Food Tutorials, forums and articles about Search Engine spiders and Search Engine Marketing. http://spider-food.net/

*Spiderhunter.com Articles and resources about tracking Search Engine spiders. http://www.spiderhunter.com/

*Sim Spider Search Engine Robot Simulator Search Engine World has a spider that simulates what the Search Engine robots read from your website. http://www.searchengineworld.com/cgi-bin/sim_spider.cgi

Daria Goetsch is the founder and Search Engine Marketing Consultant for Search Innovation Marketing, a Search Engine Optimization company serving small businesses. She has specialized in Search Engine Promotion since 1998, including three years as the Search Engine Specialist for O'Reilly Media, Inc., a technical book publishing company.

Copyright © 2002-2005 Search Innovation Marketing. http://www.searchinnovation.com All Rights Reserved.

Permission to reprint this article is granted if the article is reproduced in its entirety, without editing, including the bio information. Please include a hyperlink to http://www.searchinnovation.com when using this article in newsletters or online.

In The News:

This RSS feed URL is deprecated, please update. New URLs can be found in the footers at https://news.google.com/news

Vertical Measures (blog)

The Weekly Measure: Writing Tips, Going Viral & Being An SEO Superstar
Vertical Measures (blog)
Every Friday, Vertical Measures gathers all the best content from across the web and compiles it into The Weekly Measure. We feature the entire spectrum of digital marketing, including content marketing, paid media, social media, search engine ...
SEO Strategy: 3 Trade-offs You Must ConsiderSearch Engine Journal
What to look out for when hiring an SEO agencyBorn2Invest
Search Engine Optimisation SEO Software Market Size and Analysis by Leading Manufacturers with Its Application and ...advertisingmarket24.com

all 4 news articles »

Forbes

Four Reasons Why 'Position Zero' Shouldn't Be Your Main SEO Goal
Forbes
It's likely that your SEO goal used to be to reach the top ranking in the SERPs for as many pieces of content as possible. When Google introduced position zero, it's probable that reaching rank zero became your new goal. (Position zero is the snippet ...
5 Critical SEO Considerations When Optimizing News WebsitesSearch Engine Journal

all 5 news articles »

Telegraph.co.uk

Marketing guides
Telegraph.co.uk
It's a bit convoluted to say why this is important for your SEO strategy, and Google denies that it's the case outright, but it's a fair bet that if a SEO or PPC manager, or your company, are paying thousands of pounds for those adverts, there's going ...

and more »

Search Engine Journal

The Ultimate Guide to Seznam SEO
Search Engine Journal
The latest data puts it at around a 16 percent market share, with Google dominating the market at 79 percent. Seznam Search Homepage, September 2018. Seznam includes some SEO guidance through this Napoveda entry. Like other search engines, ...


Local SEO Can Empower Small Businesses To Outrank Bigger Players Convincingly
HostReview.com (press release) (blog)
In the early stages, SEO was much more generic because searches too were generic. As the search pattern evolved with time, it started becoming more specific and geo-centric as searchers started using search engines to gather information about sources ...

and more »

Search Engine Land

Is SEO table stakes? (Hint: No!)
Search Engine Land
Back in late 2006, the topic of the day was “Is SEO rocket science?” It wasn't, of course, unless you were referring to boosting your organic traffic to a rocket-like trajectory or trying to impress someone by spouting off the mathematical equation for ...
3 Ways PPC and SEO Teams Can CollaborateVertical Measures (blog)

all 19 news articles »

New Electrical Contractors SEO Services Announced By Chicago Website Design SEO Company
Digital Journal
Chicago Website Design SEO Company has recently launched new SEO and marketing services that are aimed at increasing the visibility of electrical contractors. The services specifically target the local search because this is what matters. Having a ...


Chicago Website Design SEO Company Announces New SEO and Marketing Services For Roofing Contractors
Digital Journal
Roofing contractors can now access professional SEO and marketing services designed for local businesses courtesy of Chicago Website Design SEO Company. The company has recently included specialized services that target local businesses. This is ...


Forbes

Try These Three SEO Strategies For Best Results
Forbes
What are the top techniques for SEO? originally appeared on Quora: the place to gain and share knowledge, empowering people to learn from others and better understand the world. Answer by Claire Perez, Director of Strategic Account Management at ...
How to recover from a negative SEO attack – Part 5Search Engine Land
SEO Is Back! But Was It Ever Really Gone?Search Engine Journal
SEO: Retaining Performance on Site MigrationsPractical Ecommerce
Business 2 Community
all 11 news articles »

Forbes

Local Reviews Can Help Your Business Rank Higher In Local SEO
Forbes
Many facets go into a marketing strategy, and that's what makes it so interesting. Marketers need to think about who their clients are and what their online search and shopping habits are. They also have to keep a keen eye on the competition, create ...

and more »
Google News

Keyword Ownership: What It Is And Where Its Headed

Have you ever got one of those silly emails that... Read More

Whats Link Popularity?

Link popularity is just one of the ways you can... Read More

Search Engine Optimization for Everyone

There are many facets to SEO and the search engines... Read More

Site-Digest 7 Simple Steps To Place Your Site Map On Steroids

The Site Map is a too often overlooked piece of... Read More

Google Search Algorithm Patent Application Creates Spring Buzz!

Google applied for a patent on their ranking algorithm as... Read More

Optimize Your Website or Get Lost In The Crowd!

It used to be that designing an attractive website to... Read More

Valid HTML code is crucial to Search Engine Optimization

Why valid HTML code is crucial to your web site's... Read More

Torpedo and Sink the Ship SS Search Engine Ranking

I was recently contacted by one of my best clients... Read More

Search Engine Optimization and Submission Tips

Bringing visitors to your site from the main Search engines... Read More

More Hits are Good for My Website - A Myth You Should Avoid

I hear this all the time. "I can get you... Read More

Search Engines The Masters Of The Internet Universe ? Part 2

"This is a continuation of the pervious part, split into... Read More

How to Verify and Monitor Your Search Engine Listing on Google?

Being listed in search engines and ranked high on searches... Read More

SEO Expert Guide - Proposition Development (part 2/10)

It is literally amazing how many people start their online... Read More

Effective Search Engine Use

The Internet is a wonderful place full of resources that... Read More

Is Something Missing From Your Keywords Research? (Part 1)

As you may already know, keywords are an essential part... Read More

Its The Same Deal For Both Search Engines And Real Estate

It's all about location, location, location!In the past...back in the... Read More

Surviving Googles Aging Delay

Google has always been the search industry's innovator and that's... Read More

Which SEO Company/Firm to Choose for SEO Services?

In the last 2-3 years many new companies have mushroomed,... Read More

An Introduction to Google Sitemaps

... and why I 'm dying to get finally in... Read More

Search Engines - The Dominant Factor

Let's face facts - Search engines are starting to rule... Read More

Optimze Your Web Site on a Shoe String Budget

Let me start off by saying I'm not a marketing... Read More

Increase Web Site Sales with a SEO Proposal - Part 2

Part I of this article discussed some of the points... Read More

Google, Yahoo!, Pandoras Box And The Lottery

In the fall of 1990, the musical group Snap had... Read More

How To Conceal Your Website?s ?Fingerprint? From The Search Engines

The internet marketing industry is now flooded with various page... Read More

Linking Strategies to Skyrocket to the Top of Google

If you don't know already, one of the key success... Read More