Proper SEO and the Robots.txt File

When it comes to SEO, most people understand that a Web site must have content, “search engine friendly” site architecture/HTML, and meta data — i.e., title tags, meta description, and meta keywords tags.

But lately, I’m seeing a lot of “optimized” Web sites that have totally disregarded the robots.txt file. When optimizing a Web site, don’t disregard the power of this little text file.

What is a Robots.txt File?

Simply put, if you go to domain.com/robots.txt, you should see a list of directories of the Web site that the site owner is asking the search engines to “skip” (or “disallow”). However, if you’re not careful when editing a robots.txt file, you could be putting information in your robots.txt file that could really hurt your business.

There’s tons of information about the robots.txt file available at the Web Robots Pages, including the proper usage of the disallow feature, and blocking “bad bots” from indexing your Web site.

The general rule of thumb is to make sure a robots.txt file exists at the root of your domain (e.g., domain.com/robots.txt). To exclude all robots from indexing part of your Web site, your robots.txt file would look something like this:

User-agent:
* Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/

The above syntax would tell all robots not to index the /cgi-bin/, the /tmp/, and the /junk/ directories on your Web site.

Real Life Examples of Robots.txt Gone Wrong

I recently reviewed a Web site that had a good amount of content and several high quality backlinks. But, the Web site had virtually no presence in the SERPs. What happened? Well, the site’s owner had included a disallow to “/”. They were telling the search engine robots not to crawl any part of the Web site.

Read Full News

Resources for

My Web Design Source

Advertisements

One Response

  1. best design

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: