[pmwiki-users] Yahoo! Slurp is broken (was: pmwiki.org performance)

[Robin Sheat]

On Friday 17 March 2006 03:01, Patrick R. Michaud wrote:
> It just seems to me that Slurp is much more aggressive than it
> should be.
Yahoo may have a contact address for that kind of thing. Apparently you 
can ask Google to slow down the rate of spidering of your site, Yahoo may 
do the same. Also, (again, apparently), if you return 302 (IIRC, 'Not 
Modified') to a request the spiders will learn how often your site 
changes, and adjust accordingly. Is there any way to get PmWiki to return 
this to spiders and browsers as appropriate. It would make things faster 
for users, too. I know it's not a trivial problem, but perhaps when a 
page is saved it could work out a list of dependencies, and when a 
request is sent it checks the dates on all those pages, if they're all 
older than the time on the request, then it doesn't need to regenerate 
the page. I guess you'd also need a flag to say that if this page, or any 
of the dependencies has dynamic markup then the page is always sent 
fresh.

-- 
Robin <robin at kallisti.net.nz> JabberID: <eythian at jabber.kallisti.net.nz>

Hostes alienigeni me abduxerunt. Qui annus est?

PGP Key 0xA99CEB6D = 5957 6D23 8B16 EFAB FEF8  7175 14D3 6485 A99C EB6D
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : /pipermail/pmwiki-users/attachments/20060317/742764d4/attachment.bin