Follow for free plugins, new theme releases and theme news
Plugin Description
When robots (like the Googlebot) crawl your site, they begin by requesting http://example.com/robots.txt and checking it for special instructions. Use this plugin to create and edit your robots.txt file from within Wordpress (using Options -> Robots.txt).
Whenever a user (or a robot, more likely) appends “robots.txt” to your blog URL (e.g. http://blog.example.com/robots.txt), this plugin will serve up the robots.txt file that you created in the Wordpress admin menu.
This plugin should work with most versions of Wordpress, but it is particularly intended for WP-MU installations, since it allows each WPMU blog to have a unique robots.txt file.
Limitations
Note that robots make only top-level requests for robots.txt files. If you have Wordpress installed in a subdomain (e.g. http://blog.example.com/) or in your root (e.g. http://example.com/), this plugin will work as intended. But if you have Wordpress installed in a subdirectory (e.g. http://example.com/blog/), then this plugin won’t do much for you, since the search engines won’t look for http://example.com/blog/robots.txt, only for http://example.com/robots.txt.
Also, requires that you be using some form of permalink structure. If links to blog posts look like http://example.com/index.php?p=36, this won’t work.
Support
If you post your support questions as comments below, I probably won’t see them. If the FAQs don’t answer your questions, you can post support questions at the KB Robots.txt plugin page on my site.
Screenshots
This is a demo robots.txt file that was created using this plugin: KB Robots.txt demo. (This is just a demo; if it were a real robots.txt file, I would have placed it at my root.)
