Share

From this page you can share Google Makes Robot.txt Generation Easier to a social bookmarking site or email a link to the page.
Social WebE-mail
Enter multiple addresses on separate lines or separate them with commas.
Google Makes Robot.txt Generation Easier
(Your Name) has forwarded a page to you from Ajaxonomy
(Your Name) thought you would like to see this page from the Ajaxonomy web site.

Google Makes Robot.txt Generation Easier

Tagged:  

If you have worked on a website that you want to control what search engine bots do you probably know just how powerful the robot.txt file can be. Configuring the robot.txt is often a specialty skill as many web masters don't do much with the file. Well, Google has made it much easier to generate a robot.txt file with a Robot.txt Generator tool that is part of it's Webmaster Tools.

Below is an excerpt from the Google Webmaster Central post.

Once you're finished with the generator, feel free to test the effects of your new robots.txt file with our robots.txt analysis tool. When you're done, just save the generated file to the top level (root) directory of your site, and you're good to go. There are a couple of important things to keep in mind about robots.txt files:

* Not every search engine will support every extension to robots.txt files

The Robots.txt Generator creates files that Googlebot will understand, and most other major robots will understand them too. But it's possible that some robots won't understand all of the robots.txt features that the generator uses.

* Robots.txt is simply a request

Although it's highly unlikely from a major search engine, there are some unscrupulous robots that may ignore the contents of robots.txt and crawl blocked areas anyway. If you have sensitive content that you need to protect completely, you should put it behind password protection rather than relying on robots.txt.

You can read the full post here.
To go to the Webmaster Center click here.

Now you can use the Robot.txt Generator to make your site work better on Google and possibly enhance your search engine effectiveness. There are other similar tools available, but a tool from the makers of the most used search engine on the planet is definitely worth a look. If you use the tool I would love to hear your experience with it.