Welcome to the home of the Robotcop project. Robotcop is an open source module for webservers which helps webmasters prevent spiders from accessing parts of their sites they have marked off limits.

Mar 11 - Release 0.6 which adds report pages, log filtering, and important bugfixes.
Feb 21 - A Linux (RH7) DSO is now available on the download page.

Robotcop is now available for Apache 1.3 webservers. This is the first public release of the software, but the basic functions are complete. Help us make it better by trying it out on your site!

The next version will include support for Apache 2.0. Later versions will bring Robotcop to ISAPI webservers such as Zeus and IIS, and add features like distributed lists for server farms.

• Spiders which read the robots.txt file are held to its rules. If a spider breaks a law in that file, further requests from that spider are intercepted by Robotcop.
• The webmaster can create trap directories which are marked off limits in the robots.txt file. If a spider acceses a trap directory in violation of the robots.txt file, further requests from that spider are intercepted.
• Webmasters can respond to misbehaving spiders by trapping them, poisoning their databases of harvested e-mail addresses, or simply block them.
• Robotcop is a webserver module written in C, not a CGI program, which ensures that it does its job very fast and with minimal impact on the site.
• All requests to the site are checked by Robotcop to ensure that misbehaving spiders are intercepted. Robotcop even protects requests for other modules such as PHP.
• Robotcop has a configurable list of known evil spiders which are immediately intercepted.