[Pmwiki-users] robots.txt

Patrick R. Michaud pmichaud
Fri Feb 13 10:06:55 CST 2004


On Fri, Feb 13, 2004 at 11:57:05AM +0100, Christian Ridderstr?m wrote:
> Hi
> 
> I just realized that it might be a good idea to supply an example 
> 	robots.txt
> that does something suitable (unless that file is no longer needed because 
> it's controlled through <HEAD>...</HEAD>

By default it's now "controlled" through <meta ... /> tags.  But some
spiders may not recognize robot meta tags, so it may still be useful
to create a robots.txt file.  OTOH, having robots.txt in a pmwiki
directory won't do any good, as it has to be in a server document root
(accessible via the url 'http://someserver.org/robots.txt') to
do any good.  Overall I think it's better to just provide a documentation
page that explains everything and provides a sample robots.txt file.

Pm



More information about the pmwiki-users mailing list