[pmwiki-users] Active prevention of crawling by web robots

Simon s-i-m-o-n at paradise.net.nz
Mon Aug 8 02:36:02 CDT 2005


Was there a reason why robots.txt was not used, or the robots meta tag?

S

http://www.searchengineworld.com/robots/robots_tutorial.htm
http://www.robotstxt.org/wc/meta-user.html

Daniel Scheibler wrote:

>At
>
>http://www.pmwiki.org/wiki/Cookbook/BlockCrawler
>
>I describe a recipe to active prevent web crawlers.
>
>Greets,
>
>scheiby.
>  
>





More information about the pmwiki-users mailing list