[pmwiki-users] slurp is broken

[Patrick R. Michaud]

On Wed, Jul 19, 2006 at 11:36:53AM -0500, JB wrote:
> PM,
> 
> Can I please get a copy of your robots.txt file?

Also, for any who are interested, here's the relevant
sections of my root .htaccess file, which denies certain
user agents at the webserver level instead of waiting
for PmWiki to do it:

    # HTTrack and MSIECrawler are just plain annoying
    RewriteEngine On
    RewriteCond %{HTTP_USER_AGENT} HTTrack [OR]
    RewriteCond %{HTTP_USER_AGENT} MSIECrawler
    RewriteRule ^wiki/ - [F,L]
    
    # block ?action= requests for these spiders
    RewriteCond %{QUERY_STRING} action=[^rb]
    RewriteCond %{HTTP_USER_AGENT} Googlebot [OR]
    RewriteCond %{HTTP_USER_AGENT} Slurp [OR]
    RewriteCond %{HTTP_USER_AGENT} msnbot [OR]
    RewriteCond %{HTTP_USER_AGENT} Teoma [OR]
    RewriteCond %{HTTP_USER_AGENT} ia_archive
    RewriteRule .* - [F,L]

Pm