Google Makes Robot.txt Generation Easier


If you have worked on a website that you want to control what search engine bots do you probably know just how powerful the robot.txt file can be. Configuring the robot.txt is often a specialty skill as many web masters don't do much with the file. Well, Google has made it much easier to generate a robot.txt file with a Robot.txt Generator tool that is part of it's Webmaster Tools.

Below is an excerpt from the Google Webmaster Central post.

Once you're finished with the generator, feel free to test the effects of your new robots.txt file with our robots.txt analysis tool. When you're done, just save the generated file to the top level (root) directory of your site, and you're good to go. There are a couple of important things to keep in mind about robots.txt files:

* Not every search engine will support every extension to robots.txt files

The Robots.txt Generator creates files that Googlebot will understand, and most other major robots will understand them too. But it's possible that some robots won't understand all of the robots.txt features that the generator uses.

* Robots.txt is simply a request

Although it's highly unlikely from a major search engine, there are some unscrupulous robots that may ignore the contents of robots.txt and crawl blocked areas anyway. If you have sensitive content that you need to protect completely, you should put it behind password protection rather than relying on robots.txt.

You can read the full post here.
To go to the Webmaster Center click here.

Now you can use the Robot.txt Generator to make your site work better on Google and possibly enhance your search engine effectiveness. There are other similar tools available, but a tool from the makers of the most used search engine on the planet is definitely worth a look. If you use the tool I would love to hear your experience with it.

Post new comment

The content of this field is kept private and will not be shown publicly.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd> <pre> <div> <blockquote> <object> <embed> <img> <param>
  • Lines and paragraphs break automatically.
  • Web page addresses and e-mail addresses turn into links automatically.

More information about formatting options

This question is for testing whether you are a human visitor and to prevent automated spam submissions.
Copy the characters (respecting upper/lower case) from the image.