The Role of the Robots.txt File to Improve Site Ranking!

Not many web master take the time to use a robots.txt file for their website. For search engine spiders that use the robots.txt to see what directories to search through, the robots.txt file can be very helpful in keeping the spiders indexing your actual pages and not other information, such as looking through your stats!

The robots.txt file is useful in keeping your spiders from accessing parts folders and files in your hosting directory that are totally unrelated to your actual web site content. You can choose to have the spiders kept out of areas that contain programming that search engines cannot parse properly, and to keep them out of the web stats portion of your site.

Many search engines cannot view dynamically generated content properly, mainly created by programming languages, such as PHP or ASP. If you have an online store programmed in your hosting account, and it is in a seperate directory, you would be wise to block out the spiders from this directory so it only finds relevant information.

The robots.txt file should be placed in the directory where your main files for your hosting are located. So you would be advised to create a blank text file, and save it as robots.txt, and then upload it to your web hosting to the same directory your index.htm file is located.

Here is examples of the use of the robots.txt file:

To block out a directory in a robots.txt file, such as a subdirectory for your online store called /store/ you would do the following: Disallow: /store/

Another example to block out your stats directory: Disallow: /stats/

You may also want to disallow individual files that you do not want searched by the search engines. For example you dont want search.php to be parsed by the Search Engines. To do this you type in the following on its own line:

Disallow: /search.php

Following the rules outlined and creating the robots.txt file, you will keep search engine spiders out of unwanted files and directories, and letting them go through the important files to see what your web site is all about!

Michael Kralj is owner of Emenki Web Solutions and Domains at Retail. Emenki Web Solutions are web site designers and programmers based in Hamilton, Ontario, providing businesses with an informative and strategic approach to establishing an online presence on the web.

Please visit Emenki Web Solutions & Web Design in Hamilton, Ontario on the web http://www.emenki.com

Please visit Domains at Retail - Cheap Domain Name Registration on the web: http://www.domainsatretail.com

In The News:

Can SEO Be Made Predictable?  Search Engine Journal
Can SEO Have a Negative Effect?  Search Engine Journal
What Does It Mean to ‘Do SEO’?  Search Engine Journal
SEO Hamilton  Search Engine People
My experience with SEO  Practical Ecommerce
7 Coding Barriers to SEO Success  Practical Ecommerce
5 Big Content Trends for 2020  Search Engine Journal
What Is a Google Penalty in SEO?  Search Engine Journal
The High Cost of Ignoring SEO  Practical Ecommerce
Google Shares Top 3 SEO Factors  Search Engine Journal
What's the Value of SEO?  Practical Ecommerce
How to Avoid SEO Misinformation  Search Engine Journal

New Free Online Tool for Keyword Research

I came across a new website the other day and... Read More

How To Get Number 1 Spot In Google, Every Time, Guaranteed

Woaah! Wait a minute. Before I exclusively reveal these insider... Read More

See No Google, Hear No Google, Speak No Google

That's right - I dreamt of a World Wide Web... Read More

Absolute & Relative Links How Do They Rank?

The question for this article is whether or not you... Read More

Importance of Keywords in Anchor Text or Title Text

Keywords are indisputably, the single most important element of an... Read More

Optimizing Your Website

Search Engine Optimization, optimizing your website for it to be... Read More

Keyword Targeting Strategy In Your Site

Once the keywords have been decided for the site one... Read More

Google is Quickly Changing...

Google is quickly changing...With the big buzz of Novembers fall... Read More

Link Popularity Explained and How To Build Links

Link popularity is the single most influential factor for determining... Read More

Search Engine Visibility - The Mantra of Corporate Profitability

The corporate fundamentals are par excellence! The product is unsurpassable... Read More

Increase Your Search Engine Ranking

There are methods to increase your search engine rankings which... Read More

If Content is King, then surely Relevance is Queen!

There has been a lot of to-ing and fro-ing in... Read More

Googled

The Internet may have started as the fervent brainchild of... Read More

Make Quality Content Your #1 Priority

It is by now a proven fact that content is... Read More

The Purpose of PageRank?

I am sure the first thought provoking question that popped... Read More

How to Avoid the Google Duplicate Content Filter?

More and more webmasters are building websites with publicly available... Read More

Learn about the Google Search Engine Tools

Think you know everything about searching with Google? Think again.... Read More

The Google Sandbox ? A Frustrating Inevitability or a Golden Opportunity?

IntroductionThe Google Sandbox is a term applied to the phenomenon... Read More

SEO and the Outsourcing of Inbound Link Building

Search Engine Optimization nowadays has a lot to do with... Read More

Monitor and Increase Your Search Engine Visibility with the DIY SEO Tools

In this three part article, you'll find many tools that... Read More

Search Engines 101 - Search Engines Explained

What Are Search Engines?A search engine is a database system... Read More

SEO Quickstart - 3 Things You can Do to Improve Search Engine Rankings Right Now

Increasing your search engine rankings should certainly be one of... Read More

STOP Writing for Search Engines

Back when I was starting out with my first internet... Read More

Top 5 Search Engine Optimization Mistakes

There are a lot of ways to promote your website... Read More

Creating A Search Engine Copywriting Plan

Search engine copywriting has become an extremely important part of... Read More