Search Engine Robots - How They Work, What They Do (Part II)

If your site isn't found in the search engines, it is probably because the robots couldn't deal with it. It could be something as simple as not being able to find the site, or it may be more complicated issues involving the robot's not being able to crawl the site or figure out what your pages are all about.

Submitting your site to the major search engines: that will help with the "can't find it" problem. Even having links pointing back to your site can be enough to attract the search engine robots. Google, for example, suggests that you may not have to submit your pages; they will find your site if you have a link pointing back to it from at least one other site on the web.

If the robots can find your site but can't make sense of it, then you may need to look at the content and technology used on your pages. Frames, Flash, dynamically generated pages, and invalid HTML source code can cause problems when the search engine robot tries to access your web pages. While some search engines are beginning to be able to index dynamically generated pages and Flash (e.g. Google and AllTheWeb), use of some of these technologies can hinder your ability to be indexed by the search engine robots.

Text in images cannot be read by the search engine robots. Using ALT image text is an important way to help the robots "read" your images. Websites with extensive images rely heavily on ALT text to present their content. How Do I Get The Most Out Of Indexing?

If you know what to "feed" the spidering robots you will help yourself with search engine ranking.

Having a website full of good content is the major factor. Search engines exist to serve their visitors, not to rank your website. You need to be sure to present yourself in your site in the way that will be most useful to the search engine visitor. Each search engine has its own idea of what is important in a page, but they all value text highly. Making sure that the text on your pages includes your most important keyword phrases will help the search engine evaluate the content of those pages.

Making sure that you have good title and meta tags will further assist the search engines in understanding what your page is about. If the text on the page is about widgets, the title is about widgets, and the meta tags are about widgets, the search engine will have a pretty good idea that you are all about widgets. When their visitors search for widgets, the search engines know to list your site in the results.

A sitemap page is a very good way of giving the search engine robot every opportunity to reach your website pages. Since robots click through the links of your web pages, make sure that at least your most important pages are included in the sitemap; you may even want to include all your pages there, depending on the size of your site. Be sure to add a link to the sitemap page from each page on your site.

Another important consideration is that of keeping all of your pages within a small number of "clicks" from your top page. Many robots will not follow links more than two or three levels deep, so if your "widgets" page can only be reached from your home page by following multiple links (e.g. home page >> about us page >> products page >> widgets page), the robot may not crawl deep enough to get to the widgets page. Testing Your Website For Search Engine Robot Accessibility

To get an idea just what the search engine robot "sees" on your page, you can look at the Sim Spider tool. You may be surprised at how different your site looks to the robot. You can find this tool at http://www.searchengineworld.com/cgi-bin/sim_spider.cgi

You will see text and ALT image text show up in the results. If your entire website is built in Flash, you will see nothing at all because robots don't understand Flash movies. The Bottom Line

When it comes to search engine robots, think simply. Lots of good content and text, hyperlinks the robots can follow, optimization of your pages, topical links pointing back to your site and a sitemap will help insure the best results when the robots come visiting.

Resources

*SpiderSpotting - Search Engine Watch http://searchenginewatch.com/webmasters/spiders.html

*Robotstxt.org List of robots and protocols for setting up a robots.txt file. http://www.robotstxt.org/

*Spider-Food Tutorials, forums and articles about Search Engine spiders and Search Engine Marketing. http://spider-food.net/

*Spiderhunter.com Articles and resources about tracking Search Engine spiders. http://www.spiderhunter.com/

*Sim Spider Search Engine Robot Simulator Search Engine World has a spider that simulates what the Search Engine robots read from your website. http://www.searchengineworld.com/cgi-bin/sim_spider.cgi

Daria Goetsch is the founder and Search Engine Marketing Consultant for Search Innovation Marketing, a Search Engine Optimization company serving small businesses. She has specialized in Search Engine Promotion since 1998, including three years as the Search Engine Specialist for O'Reilly Media, Inc., a technical book publishing company.

Copyright © 2002-2005 Search Innovation Marketing. http://www.searchinnovation.com All Rights Reserved.

Permission to reprint this article is granted if the article is reproduced in its entirety, without editing, including the bio information. Please include a hyperlink to http://www.searchinnovation.com when using this article in newsletters or online.

In The News:

This RSS feed URL is deprecated, please update. New URLs can be found in the footers at https://news.google.com/news

Business.com

How Artificial Intelligence Is Changing SEO
Business.com
AI is a rapidly evolving technology that will make SEO tools and techniques even more informative and useful for businesses in the near future. Artificial intelligence and digital marketing are quickly becoming intertwined systems. AI development is ...
3 advanced Google SEO strategies for financial marketersCUinsight.com (press release)
Visual Search: What It Is and How It Is Changing the SEO Industry? [Infographic]Business 2 Community

all 6 news articles »

Search Engine Journal

How Reliable is Google's Web.dev SEO Score?
Search Engine Journal
By making sure search engines can find and automatically understand your content, you are improving the visibility of your site for relevant searches. This is called SEO, or search engine optimization, which can result in more interested users coming ...

and more »

ThomasNet News (blog)

What You Need To Know About Google's New SEO Scoring Tool
ThomasNet News (blog)
Cracking the Google SEO algorithm can be a bit daunting, but earlier this week, the search giant launched a new set of tools designed to bring about some clarity. Here's what you need to know about the tools, and how you can incorporate them into your ...


Homesnap Pro+ wants to help bolster your SEO
Inman.com
Homesnap Pro+ is a new product from the search app to help agents establish and manage Google My Business Profiles. Article image credited to Castleski / Shutterstock.com. Subscribe to Inman Select and read the entire article. Get the breaking news, ...


Search Engine Journal

9 Ways to Deal with Unresponsive SEO Clients
Search Engine Journal
Unresponsive SEO clients are a common problem that most agencies have to deal with at some point. Fortunately, re-establishing contact is often simpler than you might think. A little extra preparation can also help you avoid situations like this in the ...


Search Engine Land

5 SEO trends that will matter most in 2019
Search Engine Land
This year, Google's shaken the world with its mobile- and speed-related efforts. As a result, most of next year's SEO efforts are expected in this direction. However, some “non-Google” game-changers will also influence how we build our SEO campaigns.
Google Launches New Tool That Grades A Site's SEO ImplementationSearch Engine Journal
The Search Future: Plan Your 2019 SEO TacticsValueWalk
Get Your Website on Google's Front Page with These SEO ToolsXDA Developers (blog)

all 20 news articles »

Search Engine Roundtable

Google Tool Scores Your SEO: Should Google Be Scoring Your SEO?
Search Engine Roundtable
It's too broad to be a useful metric. It would do more harm than good - SEOs would be judged on it incorrectly. E.g. a small site, perfectly "SEO'd" gets score of 100%. SEO is "complete". Create another 50 pieces of great content, score drops, traffic ...


Search Engine Journal

What Are Entities & Why They Matter for SEO
Search Engine Journal
Entities are, in my not-so-humble opinion, the single most important concept to understand in SEO right now. Full stop. Think I'm just another SEO professional spouting the latest “silver bullet” that will die on the table along with many before it ...


The Drum

Recovering SEO traffic and rankings after a website redesign
Search Engine Land
When building a new website, retaining and improving your SEO and organic traffic should be a key design goal. This requires a clear understanding of how SEO and website design work together and careful planning for the site migration. If everything is ...
The impact of voice search on local SEOThe Drum

all 5 news articles »

Campaign US

SEO jackpot sparks birth of e-commerce agency success
Campaign US
It sparked an SEO explosion and raked in a ridiculous amount of traffic via Google. The article's success was exploited and monetized with affiliate links, which gave the publisher a nice little revenue stream for a while. "Eventually, we came to the ...

Google News

Are You Getting Nuked By Google Lately?

Since the last Google update, there have been many instances... Read More

The Google Sandbox ? A Frustrating Inevitability or a Golden Opportunity?

IntroductionThe Google Sandbox is a term applied to the phenomenon... Read More

The Truth About Search Engines: Playing A Game You Cant Win

If you go strictly by the numbers, Yahoo, MSN and... Read More

Most Overlooked Search Engine Optimization Technique

Some of the basics when it comes to SEO and... Read More

A Naive Mistake Cost Me My Google Rankings

Little do you know but you too could be making... Read More

The Disease of Google Banning: Are You At Risk?

Definition:Google banning occurs when Google removes your site from its... Read More

10 Easy Steps to Boost Your Search Engine Rankings!

In order that someone finds your website and buys your... Read More

Guide to Search Engine Optimization

What is Search Engine Optimization?Search Engine Optimization or SEO for... Read More

Search Engine Optimization SEO Made Easy

SEO is a never ending battle! So is SEO... Read More

Optimum SEO Keyword Density ? A Real-Life Case Study

So you've built your website, you know what keywords you... Read More

SEO Tips - Google Has Changed - Learn Why And What To Do

Google now checks the year your domain name was first... Read More

SEO Expert Guide - Page Optimization (part 5/10)

In parts 1 - 4 you learnt how to develop... Read More

Click Click Boom: a Linking Strategy that will Blow Away Your Competition

Web marketers, do you hear what I hear?"Click-click BOOM"That's the... Read More

What In The World Is This Google Sandbox Theory Thing? And How Do I Beat It?

Ok, so over the past month or so I've been... Read More

21 Search Engine Terms Every Web Marketer Should Know Part 1

1. Search Engine - Is a database of web sites... Read More

The Life Blood of Internet-Based Home Businesses...Search Engines

Anyone involved in an Internet-based home business will soon come... Read More

Writing SEO Copy ? 8 Steps to Success

We all know that the lion's share of web traffic... Read More

Determining the Value of Your SEO Service

Every once in a while--and probably more often than we... Read More

SEO Expert Guide - Search Engines Explained (part 1/10)

Before we explore the world of search engine optimization, it... Read More

7 Steps to Highly Relevant, Search Engine Friendly, and People Useful Web Site Content

The task is to build a Content Rich Search Friendly... Read More

Why Fear? Now Change Your Filenames Without Disturbing Search Engines/Visitors And Rank Well

By now, you know that to rank well in the... Read More

SEO - Get Your Site Out of the Google Sandbox Fast!

Is your new site sitting in the infamous Google "sandbox"?... Read More

What Is Waiting for Us? Tomorrows SEO Industry

Today, SEO is swiftly approaching saturation point. More and more... Read More

Finding Profitable Keyword Phrases

So you have a site concept developed that you are... Read More

A Way for Search Engines to Improve

Wouldn't it be nice if the search engines could comprehend... Read More