[pmwiki-users] Yahoo! Slurp is broken (was: pmwiki.org performance)
Robin Sheat
robin at kallisti.net.nz
Thu Mar 16 16:23:37 CST 2006
On Friday 17 March 2006 03:01, Patrick R. Michaud wrote:
> It just seems to me that Slurp is much more aggressive than it
> should be.
Yahoo may have a contact address for that kind of thing. Apparently you
can ask Google to slow down the rate of spidering of your site, Yahoo may
do the same. Also, (again, apparently), if you return 302 (IIRC, 'Not
Modified') to a request the spiders will learn how often your site
changes, and adjust accordingly. Is there any way to get PmWiki to return
this to spiders and browsers as appropriate. It would make things faster
for users, too. I know it's not a trivial problem, but perhaps when a
page is saved it could work out a list of dependencies, and when a
request is sent it checks the dates on all those pages, if they're all
older than the time on the request, then it doesn't need to regenerate
the page. I guess you'd also need a flag to say that if this page, or any
of the dependencies has dynamic markup then the page is always sent
fresh.
--
Robin <robin at kallisti.net.nz> JabberID: <eythian at jabber.kallisti.net.nz>
Hostes alienigeni me abduxerunt. Qui annus est?
PGP Key 0xA99CEB6D = 5957 6D23 8B16 EFAB FEF8 7175 14D3 6485 A99C EB6D
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : /pipermail/pmwiki-users/attachments/20060317/742764d4/attachment.bin
More information about the pmwiki-users
mailing list