[pmwiki-users] Yahoo! Slurp is broken (was: pmwiki.org performance)

bram brambring bram.brambring at gmail.com
Thu Mar 16 01:33:35 CST 2006


According to slurp slurp  (and slurp only?) supports an extension in the
robots.txt:

User-agent: Slurp
Crawl-delay: 900


I see that you have that already in your robots.txt ( with agent *)


maybe increasing it and specifying it for slurp only.
No idea whether or not it helps

bram




   Yahoo! Slurp is STUPID.
>    Yahoo! Slurp is STUPID.
>    Yahoo! Slurp is STUPID.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: /pipermail/pmwiki-users/attachments/20060316/e0e64be6/attachment.html 


More information about the pmwiki-users mailing list