ROBOTStxt is an editor for robots.txt files which are used to direct bots which pages to fetch from a website. More information regarding robots.txt can be found on the wikipedia article.

Features:

• Edit Robots.txt
• View Robots.txt
• Open Robots.txt from WWW

How To:
This application is a very simple application to edit or view robots.txt files. You can either open an existing robots.txt file (either from a remote location on the www, or from your local disk). You can then start adding entries. The most important entries to add are the User-agent which identifies a user agent (use * to match all) and any Disallow paths. To delete an entry simply right click on it and click on delete. A sample robots.txt file can be found over here. Please note that robots.txt file should reside in the root directory of your website and should be named robots.txt for the bot to find it.