The Role of the Robots.txt File to Improve Site Ranking!

Not many web master take the time to use a robots.txt file for their website. For search engine spiders that use the robots.txt to see what directories to search through, the robots.txt file can be very helpful in keeping the spiders indexing your actual pages and not other information, such as looking through your stats!

The robots.txt file is useful in keeping your spiders from accessing parts folders and files in your hosting directory that are totally unrelated to your actual web site content. You can choose to have the spiders kept out of areas that contain programming that search engines cannot parse properly, and to keep them out of the web stats portion of your site.

Many search engines cannot view dynamically generated content properly, mainly created by programming languages, such as PHP or ASP. If you have an online store programmed in your hosting account, and it is in a seperate directory, you would be wise to block out the spiders from this directory so it only finds relevant information.

The robots.txt file should be placed in the directory where your main files for your hosting are located. So you would be advised to create a blank text file, and save it as robots.txt, and then upload it to your web hosting to the same directory your index.htm file is located.

Here is examples of the use of the robots.txt file:

To block out a directory in a robots.txt file, such as a subdirectory for your online store called /store/ you would do the following: Disallow: /store/

Another example to block out your stats directory: Disallow: /stats/

You may also want to disallow individual files that you do not want searched by the search engines. For example you dont want search.php to be parsed by the Search Engines. To do this you type in the following on its own line:

Disallow: /search.php

Following the rules outlined and creating the robots.txt file, you will keep search engine spiders out of unwanted files and directories, and letting them go through the important files to see what your web site is all about!

Michael Kralj is owner of Emenki Web Solutions and Domains at Retail. Emenki Web Solutions are web site designers and programmers based in Hamilton, Ontario, providing businesses with an informative and strategic approach to establishing an online presence on the web.

Please visit Emenki Web Solutions & Web Design in Hamilton, Ontario on the web http://www.emenki.com

Please visit Domains at Retail - Cheap Domain Name Registration on the web: http://www.domainsatretail.com

In The News:

SEO Tools Aren't Enough for Success  Search Engine Journal
How to Do SEO for Niche Markets  Search Engine Journal
How to Use GitHub for Enterprise SEO  Search Engine Journal
3 Must-Know SEO Trends for Marketers  Marketing Tech Outlook
Amazon SEO in 2020  Jungle Scout
SEO in 2020: Going Beyond Google  Search Engine Journal
5 Ways SEO & Web Design Go Together  Search Engine Journal
Can SEO Be Made Predictable?  Search Engine Journal
Google's Advice on How to Hire an SEO  Search Engine Journal
Moving a Company to an SEO Focus  Search Engine Journal

World of Website Promotion

Website promotion is a big and ongoing process. Every person... Read More

Soliciting Search Engines

As your guide operator through the web, search engines are... Read More

All About Links -- Interview With Link Building Expert , Bob Gladstein

Julia: Welcome Bob. Thank you for taking the time to... Read More

Link Popularity --- Its Role and Importance In Getting Top Search Engine Rankings

Introduction"Link Popularity" ? these words may have caught your attention... Read More

SEO - Google Sitemaps Explained

Once again I seem to be writing about Google. The... Read More

Google Contest - Nigritude Ultramarine

Search engine optimization experts are having fun with Google. Experts,... Read More

Search Engine Optimization (SEO) - Fix Your OnPage!

Search Engine Optimization (SEO) is something you should be aware... Read More

Your Online Home-Based Business Can Lead To Search Engine Stress

As a member of several search engine optimization forums, I... Read More

Do It Yourself SEO

Internet surfers use search engines more than any other tool... Read More

Finding the Right SEO Company

I often talk to people who have lost faith in... Read More

Pay Per Click Versus ?Organic? Search Engine Listings - The Pros and Cons and Best Uses for Each

What's the difference?For those who aren't quite clear what the... Read More

Google Rank Cake

6 cups thick content mix 1 jar word of mouth,... Read More

Taking Advantage of Googles Sandbox Effect

Most new sites submitted to Google (at least within the... Read More

Easy And Simple Steps To Get Listed In Search Engines

Search engines are one of THE best resource for free... Read More

10 Basic Rules for Where to Place Your Keywords

First of all, Google and most other search engines do... Read More

Finding Profitable Keyword Phrases

So you have a site concept developed that you are... Read More

Goofy Mistakes that Hurt Your Search Engine Rankings

One thing is for sure, you don't want to spend... Read More

The Great Search Engine Experiment Revisited Who is the Coolest Guy in the Universe

A recent Search Engine Experiment Demonstrates how by combining Key... Read More

60 Day Sandbox for Google & AskJeeves; MSN Indexes Quickest, Yahoo Next

Search engine listing delays have come to be called the... Read More

How to Pick an SEO Firm

If you're looking for an SEO firm, we recommend that... Read More

Local Search Optimization - A Guide to Getting Started

While searching the web these days, it's hard not to... Read More

Achieving Better Search Engine Optimization

The search engine giants are locked in an all out... Read More

Buzzwords vs Effective SEO Keywords

Ever see a website that seems to speak a foreign... Read More

Are You Getting Nuked By Google Lately?

Since the last Google update, there have been many instances... Read More

Keywords, Ranking, & Search Engine Optimization Fun

I am a Search Engine Optimization newbie. I have read... Read More