[pmwiki-users] Active prevention of crawling by web robots
Simon
s-i-m-o-n at paradise.net.nz
Mon Aug 8 02:36:02 CDT 2005
Was there a reason why robots.txt was not used, or the robots meta tag?
S
http://www.searchengineworld.com/robots/robots_tutorial.htm
http://www.robotstxt.org/wc/meta-user.html
Daniel Scheibler wrote:
>At
>
>http://www.pmwiki.org/wiki/Cookbook/BlockCrawler
>
>I describe a recipe to active prevent web crawlers.
>
>Greets,
>
>scheiby.
>
>
More information about the pmwiki-users
mailing list