Deprecated: mysql_connect(): The mysql extension is deprecated and will be removed in the future: use mysqli or PDO instead in /home/a26f9f83/public_html/articles/includes/config.php on line 159
Search Engine Robots - How They Work, What They Do (Part II) > NetSparsh - Viral Content you Love & Share

Search Engine Robots - How They Work, What They Do (Part II)

If your site isn't found in the search engines, it is probably because the robots couldn't deal with it. It could be something as simple as not being able to find the site, or it may be more complicated issues involving the robot's not being able to crawl the site or figure out what your pages are all about.

Submitting your site to the major search engines: that will help with the "can't find it" problem. Even having links pointing back to your site can be enough to attract the search engine robots. Google, for example, suggests that you may not have to submit your pages; they will find your site if you have a link pointing back to it from at least one other site on the web.

If the robots can find your site but can't make sense of it, then you may need to look at the content and technology used on your pages. Frames, Flash, dynamically generated pages, and invalid HTML source code can cause problems when the search engine robot tries to access your web pages. While some search engines are beginning to be able to index dynamically generated pages and Flash (e.g. Google and AllTheWeb), use of some of these technologies can hinder your ability to be indexed by the search engine robots.

Text in images cannot be read by the search engine robots. Using ALT image text is an important way to help the robots "read" your images. Websites with extensive images rely heavily on ALT text to present their content. How Do I Get The Most Out Of Indexing?

If you know what to "feed" the spidering robots you will help yourself with search engine ranking.

Having a website full of good content is the major factor. Search engines exist to serve their visitors, not to rank your website. You need to be sure to present yourself in your site in the way that will be most useful to the search engine visitor. Each search engine has its own idea of what is important in a page, but they all value text highly. Making sure that the text on your pages includes your most important keyword phrases will help the search engine evaluate the content of those pages.

Making sure that you have good title and meta tags will further assist the search engines in understanding what your page is about. If the text on the page is about widgets, the title is about widgets, and the meta tags are about widgets, the search engine will have a pretty good idea that you are all about widgets. When their visitors search for widgets, the search engines know to list your site in the results.

A sitemap page is a very good way of giving the search engine robot every opportunity to reach your website pages. Since robots click through the links of your web pages, make sure that at least your most important pages are included in the sitemap; you may even want to include all your pages there, depending on the size of your site. Be sure to add a link to the sitemap page from each page on your site.

Another important consideration is that of keeping all of your pages within a small number of "clicks" from your top page. Many robots will not follow links more than two or three levels deep, so if your "widgets" page can only be reached from your home page by following multiple links (e.g. home page >> about us page >> products page >> widgets page), the robot may not crawl deep enough to get to the widgets page. Testing Your Website For Search Engine Robot Accessibility

To get an idea just what the search engine robot "sees" on your page, you can look at the Sim Spider tool. You may be surprised at how different your site looks to the robot. You can find this tool at http://www.searchengineworld.com/cgi-bin/sim_spider.cgi

You will see text and ALT image text show up in the results. If your entire website is built in Flash, you will see nothing at all because robots don't understand Flash movies. The Bottom Line

When it comes to search engine robots, think simply. Lots of good content and text, hyperlinks the robots can follow, optimization of your pages, topical links pointing back to your site and a sitemap will help insure the best results when the robots come visiting.

Resources

*SpiderSpotting - Search Engine Watch http://searchenginewatch.com/webmasters/spiders.html

*Robotstxt.org List of robots and protocols for setting up a robots.txt file. http://www.robotstxt.org/

*Spider-Food Tutorials, forums and articles about Search Engine spiders and Search Engine Marketing. http://spider-food.net/

*Spiderhunter.com Articles and resources about tracking Search Engine spiders. http://www.spiderhunter.com/

*Sim Spider Search Engine Robot Simulator Search Engine World has a spider that simulates what the Search Engine robots read from your website. http://www.searchengineworld.com/cgi-bin/sim_spider.cgi

Daria Goetsch is the founder and Search Engine Marketing Consultant for Search Innovation Marketing, a Search Engine Optimization company serving small businesses. She has specialized in Search Engine Promotion since 1998, including three years as the Search Engine Specialist for O'Reilly Media, Inc., a technical book publishing company.

Copyright © 2002-2005 Search Innovation Marketing. http://www.searchinnovation.com All Rights Reserved.

Permission to reprint this article is granted if the article is reproduced in its entirety, without editing, including the bio information. Please include a hyperlink to http://www.searchinnovation.com when using this article in newsletters or online.

In The News:

This RSS feed URL is deprecated, please update. New URLs can be found in the footers at https://news.google.com/news

Forbes

Four Reasons Why 'Position Zero' Shouldn't Be Your Main SEO Goal
Forbes
It's likely that your SEO goal used to be to reach the top ranking in the SERPs for as many pieces of content as possible. When Google introduced position zero, ...
5 Critical SEO Considerations When Optimizing News WebsitesSearch Engine Journal
6 Ways to Analyze Your Site's SEO - Business 2 CommunityBusiness 2 Community

all 5 news articles »

Search Engine Journal

The Ultimate Guide to Seznam SEO
Search Engine Journal
The latest data puts it at around a 16 percent market share, with Google dominating the market at 79 percent. Seznam Search Homepage, September 2018. Seznam includes some SEO guidance through this Napoveda entry. Like other search engines, ...


Local SEO Can Empower Small Businesses To Outrank Bigger Players Convincingly
HostReview.com (press release) (blog)
In the early stages, SEO was much more generic because searches too were generic. As the search pattern evolved with time, it started becoming more specific and geo-centric as searchers started using search engines to gather information about sources ...


Search Engine Land

Is SEO table stakes? (Hint: No!)
Search Engine Land
Back in late 2006, the topic of the day was “Is SEO rocket science?” It wasn't, of course, unless you were referring to boosting your organic traffic to a rocket-like trajectory or trying to impress someone by spouting off the mathematical equation for ...
3 Ways PPC and SEO Teams Can CollaborateVertical Measures (blog)
3 SEO Strategies That Don't Require an Expert to ManageBusiness 2 Community

all 19 news articles »

socPub (blog)

The aspects to consider before availing the services of an SEO agency
socPub (blog)
The presence of a website on the internet is not enough for driving a suitable amount of traffic on the site and for this purpose it might be appropriate to choose an SEO company. The following factors should be considered before selecting a company ...


Forbes

Local Reviews Can Help Your Business Rank Higher In Local SEO
Forbes
Many facets go into a marketing strategy, and that's what makes it so interesting. Marketers need to think about who their clients are and what their online search and shopping habits are. They also have to keep a keen eye on the competition, create ...

and more »

Forbes

Try These Three SEO Strategies For Best Results
Forbes
What are the top techniques for SEO? originally appeared on Quora: the place to gain and share knowledge, empowering people to learn from others and better understand the world. Answer by Claire Perez, Director of Strategic Account Management at ...
How to recover from a negative SEO attack – Part 5Search Engine Land
SEO Is Back! But Was It Ever Really Gone?Search Engine Journal
SEO: Retaining Performance on Site MigrationsPractical Ecommerce
Search Engine Roundtable
all 11 news articles »

PR Daily

4 SEO trends communicators should track
PR Daily
Ranking first on a Google search engine results page (SERP) is considered an honorable achievement—but it's not the only goal that matters. If you don't rank first, there's a lot you can do to improve your organic search results. Here are four search ...

and more »

Forbes

How Brands Can Optimize SEO With Google's Evolving Algorithm
Forbes
In the world of SEO, technological innovations are constantly driving ideas forward. There are a number of things that are evolving in SEO -- first of which is voice search. Since the inception of Apple's Siri, Amazon's Alexa and Google's voice ...
A Guide to Local SEOSearch Engine Journal

all 2 news articles »

Forbes

How Google Thinks: SEO Done The Right Way
Forbes
People are always on the lookout for the search engine optimization (SEO) trick that will propel them to the top of a Google Search results page. Obviously, this does not exist, but if there were one SEO factor that summed it up, then it would be to ...

and more »
Google News

So, Where Has Your Search Engine Been Today?

Visit Google, Yahoo, MSN or one of the lesser search... Read More

Google Sitemaps Explained - How To Use Google Sitemaps

Three Ways To Index Your Site With Google Sitemaps [Difficult,... Read More

Keywords, Choose Them Wisely

By now you have likely heard that keywords and keyword... Read More

Getting Your Site Seen By Search Engines

Is your website listed on search engines? When people search... Read More

The Matrix Help For Keyphrase Analysis

Keyword analysis is a major part of search engine marketing.... Read More

Google Local Search And The Impact On Natural Optimization

With the advent of Google Local, a service that helps... Read More

SEO Expert Guide - Sitewide Optimization (part 4/10)

In parts 1 and 2 you learnt how to develop... Read More

Are You Hung-Up on Page Rank and Back Links?

It's unfortunate that many website owners are so hung-up on... Read More

World of Website Promotion

Website promotion is a big and ongoing process. Every person... Read More

Search Engine Optimization - Enhancing Web Site Visibility

I've had several prospects and clients say to me "I... Read More

6 Ways To Attract Search Engines To Your Website More Often

Adding fresh, updated content to your website is the surest... Read More

SEO Expert Guide - Paid Site Promotion (Marketing) (part 7/10)

In parts 1 - 6 you learnt how to develop... Read More

All About Google

If you read The Search Engine Showdown, you... Read More

SEO for CEOs ? Search Engine Optimization Unmasked for CEOs

If you're like most other CEOs, the term "search engine... Read More

Getting Specific With Local Search Engine Optimization

Localized search engine optimization is often overlooked as an excellent... Read More

Got Spiders?

Many internet marketers blow mountains of start-up cash on their... Read More

Google Takes Care of Idiots Too

There's an old saying that goes, "God takes care of... Read More

An SEO Checklist

Search engine optimization is on every webmaster's mind these days.... Read More

Website Submission - What You May Not Know About Search Engine Indexing

Getting your website listed in the search engines is relatively... Read More

Googles New SEO Rules

Google has recently made some pretty significant changes in its... Read More

Keyword Density And How To Use It To Keep Traffic Flowing To Your Site!

Generating high traffic to your web site can be costly,... Read More

Getting One Way SEO Links - The Easy and FREE Way

What we are hunting for?Unique Domains,One Way Links,Text Link copy... Read More

Keyword Research Made Simple!

Keyword Research is the first task in optimizing your web... Read More

SEO Training - Avoid Making this Costly Mistake!

SEO training can be overwhelming. There are literally hundreds of... Read More

Search Engine Ranking... Oh, the Mystery!

Rankings, Rankings, Rankings!How do you get your website ranked well... Read More