Deprecated: mysql_connect(): The mysql extension is deprecated and will be removed in the future: use mysqli or PDO instead in /home/a26f9f83/public_html/articles/includes/config.php on line 159
Effectively Using Robots Meta Tags > NetSparsh - Viral Content you Love & Share

Effectively Using Robots Meta Tags

The "robots" meta tag, when used properly, will tell the search engine spiders whether or not to index and follow a particular page. For the purposes of this article, we will be using the "( )" symbols to represent the "< >" in html coding.

Some examples of robot usage are as follows:

(meta name="robots" content="index,follow")

(meta name="robots" content="noindex,follow")

(meta name="robots" content="index,nofollow")

(meta name="robots" content="noindex,nofollow")

Let us first examine what these terms mean before we explain the usage for each one:

"index"- This directive tells the search engine robots (or spiders) that it is okay to index the page. Another words, you are allowing the search engine to include your page within their search directory.

"noindex"- Using this tag, you are informing the robots that this page should not be indexed. Simply put, this page will not appear in their search directory.

"follow"- When you use this tag, you are telling the search engines that you want their robot to follow any links that are found on that page.

"nofollow"- The opposite of the above definition, this directive will tell the robots not to follow any links on your page.

Putting It All Together:

With the robots tags explained, let's examine the usage for each one.

1. (meta name="robots" content="index,follow")

This tag will be used when you want the search engine spiders to index the page and follow the links to other pages. Most search engines use this setting as a "default" setting. It is possible that you may not even need to use this tag if you want the search engines to follow and index the page. However, an article at Search Engine World (searchengineworld.com/metatag/robots.htm) suggests that Inktomi does not use this as their default setting. Instead, they use the "index, nofollow" tag.

Better safe than sorry!

There has been much debate over whether or not it is necessary to use this tag. If there is even a slight possibility that some search engines do not use this as the default setting, then it would only make sense to include this tag if you want your page included in their search directory AND your links to be followed. Do the research and decide for yourself.

2. (meta name="robots" content="noindex,follow")

This tag can be used to tell the search engines that you do not want the page included in their directory, but you DO want them to follow the links that lead to other pages. A good example of its usage would be your disclaimer or privacy policy pages. You may not want these pages to show up in the search engines if they are only important to your actual visitors. However, if the links on these pages point to other pages that you want the search engines to find, then you would still want the spiders to "follow" those links.

3. (meta name="robots" content="index,nofollow")

This tag will allow your page to be indexed in the search engines, but any links on that page will not be followed.

4. (meta name="robots" content="noindex,nofollow")

When using this tag, the search engine spiders will not include this page in their directory and will not follow any links on the page either.

Where does the "robots" tag belong?

The "robots" meta tag should be used within the (head) and (/head) tags of your page. These tags are located at the top of the html coding. It will look something like this:

(html)
(head)
(title)Title of your page goes here(/title)
(meta name="keywords" content="word1,word2,word3,word4")
(meta name="description" content="A brief description of the content of this page.")
(meta name="robots" content="index,follow")
(/head)
(body)
Your webpage information here.
(/body)
(/html)

More Robots Tags

Google automatically archives a page as it crawls it. This is called a "cached" version of the page. Visitors can retrieve the archived version of the page by clicking on the "cached" link within Google's search results. If you do not want your content to be archived, you can use the following tag:

(meta name="robots" content="noarchive")

*This will only prevent your page from being "cached". If you do not want your page to be indexed at all, you will still need to include the "noindex" tag.

Another alternative to the above tag is the tag that specifically addresses Google only. If you want other search engine robots to archive your site, but you would like to prevent Google from doing so, then you can use the following tag:

(meta name="googlebot" content="noarchive")

The Misuse of Robots Tags

Something that has been popping up on websites everywhere is the Google indexing tag. This is a silly little tag that is not necessary. Some people think this tag helps Google to spider your site, but this simply isn't true. The tag looks like this: (meta name="googlebot" content="index,follow"). Some website owners believe that by specifying "googlebot" that their site has the advantage of being spidered faster and listed by Google. According to Google's web crawler information, you may use the noindex, nofollow, or noarchive tags when you DO NOT want Google to cache, index, or follow that page. Google's default setting is to index and follow the links on the page, so this "so called" googlebot index/follow tag that some site owner made up one day is completely unnecessary.

Another silly little tag--- The "Revisit-After" Tag

(meta name="revisit-after" content="90 days")
(meta name="revisit-after" content="15 days")

I'm not sure where this myth was started. Today, you will find this tag all over the Internet. Webmasters have even promoted it, claiming that it actually works. Are we so naive to believe the search engine spiders need to know when to come back? I have never used this tag, and my site has no problem with being crawled on a regular basis. Even some SEO (search engine optimization) sites are claiming its value. This comes back to the importance of researching the topic thoroughly first. My research came from some interesting information on WebmasterWorld.

It is important to examine the correct usage of the "robots" tag before applying it to your website. Incorrect usage of tags could result in spidering errors that cause robots to completely ignore the page all together. If you are interested in learning more about web robots, this great little site will provide you with the information you need to use them effectively: www.robotstxt.org/wc/robots.html

Stephani Richardson is a work at home mother of 4 who has been actively involved with affiliate marketing and home business opportunities since December 1999. She owns and operates several business related websites including 1 Work At Home Dot Com.

In The News:

This RSS feed URL is deprecated, please update. New URLs can be found in the footers at https://news.google.com/news

Android Authority (blog)

Learn the tricks of Web Development in 44 hours
Android Authority (blog)
That' s right for just $29 you get The Complete PHP & MySQL Web Development Bundle has been slashed to this week. This combination of seven online training kits and 44 hours of expert instruction is enough to take you from an absolute newbie to pro web ...


Mashable

Sign up for a web developer bootcamp class online during this course sale and pay just £12
Mashable
Web development sounds like something that should be kept to the confines of tech support HQ — clouded in mystery and something us mere mortals cannot comprehend. But it shouldn't be this way, especially as web applications become a bigger part of ...


Columbus Business First

The List: Shakeup at the top of Central Ohio's busiest web development firms
Columbus Business First
The 2018 list of the busiest Central Ohio web development firms boasts seven companies that did not appear in the list last year, including one in the top five. Two major players in this list have either been acquired or have recently announced an ...


Packt Hub

Use Rust for web development [Tutorial]
Packt Hub
You might think that Rust is only meant to be used for complex system development, or that it should be used where security is the number one concern. Thinking of using it for web development might sound to you like huge overkill. We already have ...


Web Developer (Front End)
Built In Chicago
Through an assortment of platforms, including health stations, mobile apps, and a website, higi works to raise consumer awareness, incentivize adoption of healthy behaviors, and improve life overall. We are looking for an experienced candidate with ...


ADT Magazine

Open Source Dart 2 Revamp Focuses on Mobile, Web Development
ADT Magazine
After a nearly eight-month pre-release preview, the open source Dart 2 programming language has emerged as a stable release that includes many breaking changes in a revamp that focuses on mobile and Web client-side development. Along with that Web ...

and more »

Practical Ecommerce

12 YouTube Channels for Design and Development
Practical Ecommerce
Interested in improving your web design and development skills? There are a wide variety of options to teach beginners and advanced users in all aspects of creating a website. Here is a list of YouTube channels for web design and development. There are ...


Nashville Post

Web development company leaves CBD for North Capitol
Nashville Post
Local web development company DevDigital has left its central business district office in the diminutive Berger Building and will soon be operational in North Capital near First Tennessee Park. Brittany Wegusen, a DevDigital partner and director of ...


RetailTechNews

Why Retailers Need to Care About Web Development Platforms
RetailTechNews
Brands are struggling to build and grow customer loyalty in an increasingly complex omnichannel environment. According to Wanda Cadigan, VP commerce, Sitecore, many are struggling with mobile enablement, in addition to the ultimate vision of delivering ...


Billy Penn

Meme this: How a second-gen Filipino-American landed at the heart of Philly's internet community
Billy Penn
His automotive employer soon caught on that Ilagan had a knack for the net, so he was promoted to selling cars online — where he developed a renewed desire for his original passion. “Car salesmanship is kind of how I started getting back on track for ...

Google News

10 Tips For A Successful Website

Have you ever thought the secret behind the success of... Read More

Optimal Website Design

Optimal website design is the art of logical navigation.It will... Read More

Live Support from Your Website? Can You Do It for Free?

I'm a freelance programmer and somehow I managed to get... Read More

7 Ways to Gain a Professional Online Reputation

A professional online reputation is essential to the flourishing of... Read More

Beyond Web Usability: Web Credibility

If you've been developing websites on Mars for the past... Read More

Entangled in the World Wide Web

I am just a learner who likes to delve in... Read More

How to Create a Professional Web Site in 24 Hours

While creating a Web site may seem like... Read More

The Most Important Aspect of Writing Web Copy

There is an ongoing debate about web copy. Some say... Read More

Five Steps to Create Your Software Product with Outsourcing

Many executives and investors are skeptical that software products can... Read More

Are Web Graphics Stealing Your Money?

They might not be wearing a mask and carrying a... Read More

Traffic for Webmasters

"If you build it, they will come"; is an age... Read More

The Evolution Of Search

There have been many significant changes to the face of... Read More

Preventing Repeat Form Submission Using PHP Sessions

We've all seen those messages on some websites warning not... Read More

Business Website Building Do?s and Don?ts

First, using a lot of web graphics, flash, banners and... Read More

Build a Web Site that Works...Even for a Safari Hobby!

My life's greatest passion is a safari hobby. And what... Read More

Are You Overlooking the Benefits of Using Sub-Domains within Your Web-hosting account?

One of the most useful features offered by some web... Read More

Top 7 Reasons Why Your Business Needs a Website

A website provides invaluable advantages for businesses who have one.7... Read More

Be Creative Before You Purchase Or Create A Website

Before you jump into spending coutnless hours on designing and... Read More

10 Simple Steps: Its Magic If You can E-Mail You can Update Your Web Page Dynamically

Dynamically Update Your Web Pages Via E-MailIf you can Send... Read More

The Web: Your Window of Opportunity

Thinking about starting up a business? The web is the... Read More

Using Psychology Easily on Your Website to Make More Sales!

Psychology is a long word which everyone thinks, "no, that's... Read More

7 Golden Rules For Building Mini Sites

First, an eye opener. Making your fortune through affiliate programs... Read More

Cost Effective and Powerful Dynamic Websites w. Fantastico Web Hosting

If you are developing your website on a host that... Read More

You Dont Know Which One

I still remember it very clear ... It was almost... Read More

Establishing Your Web Presence

Part of the success of most home-based businesses is their... Read More