[Pmwiki-users] robots

[Paul Williams]

My robots.txt has the following entry:

User-agent: *
Disallow: *action=*

This works very well for all robots.txt-capable
indexers that have encountered my site to date.  It
does seem to take a few days for the indexers to
figure it out, but it works nonetheless.

--- Christian_Ridderström <chr at home.se> wrote:
> On Thu, 8 Jan 2004, bram brambring (zonnet) wrote:
> 
> > Hi,
> > 
> > 
> > I have been looking into:
> > 
> > http://www.pmichaud.com/wiki/PmWiki/Robots
> > 
> > and used the depicted rules to exclude the action
> and search links. However
> > some searchengine sites still showed up with the
> ?action links.
> > 
> > I talked to the owner of one of them, he pointed
> me to the fact that
> > wildcards are not part of the robot exclusion
> standaard. And although google
> > supports wildcards, he is right.
> > 
> > The disallow section must be a complete path
> starting with '/'
> > 
> > I solved it in the local.php, are there nicer
> solutions?
> > 
> > if ( $action and ( $action!='browse' ) ) {
> >    $robots="noindex,nofollow";
> > } else {
> >    $robots="index,follow";
> > }
> > 
> > $HTMLTitleFmt = array (
> > "<title>\$Titlespaced</title>",
> > "<META CONTENT=\"$robots\" NAME=\"robots\">");
> > 
> 
> Thanks for the tip! (I reformatted it like this and
> it seems to work)
> #
> # Taking care of robots...
> #
> if($action and $action!='browse')
>   $robots="noindex,nofollow";
> else
>   $robots="index,follow";
> $HTMLTitleFmt = array
> ("<title>\$Titlespaced</title>",
>                        "<META CONTENT=\"$robots\"
> NAME=\"robots\">");
> 
> I'd also noticed in the documentation of robots.txt
> that wildcards aren't 
> allowed, but forgotten about the whole thing.
> 
> /Christian
> 
> > 
> > _______________________________________________
> > Pmwiki-users mailing list
> > Pmwiki-users at pmichaud.com
> >
>
http://pmichaud.com/mailman/listinfo/pmwiki-users_pmichaud.com
> > 
> 
> -- 
> Dr. Christian Ridderström, +46-8-768 39 44      
> http://www.md.kth.se/~chr
> 
> 
> _______________________________________________
> Pmwiki-users mailing list
> Pmwiki-users at pmichaud.com
>
http://pmichaud.com/mailman/listinfo/pmwiki-users_pmichaud.com
> 
>