[pmwiki-users] Faster searches and categories

[Martin Fick]

On Mon, Sep 12, 2005 at 01:04:03PM -0500, Patrick R. Michaud wrote:
> On Mon, Sep 12, 2005 at 01:49:45PM -0400, Martin Fick wrote:
> > > The indexes are maintained incrementally, as pages are saved.
> > > 
> > > Creating the initial index of existing pages may take a little 
> > > bit of time, but even this can be made to occur "in passes"
> > > to avoid timeouts.  
> > > ...
> > 
> > Some clever updating mechanisms indeed.  But these sound
> > only optimistic, they do not reduce the worst case scenario
> > at all, do they?
> 
> What are you meaning by "optimistic" and "worst case scenario"?
> If you mean that it doesn't speed up a link= search when no
> index exists, then yes, it's as slow as before.  However, it
> builds the index in the process, and after that initial hit 
> it's optimized again.  It's actually faster than the grep
> version, since the system only opens/reads pages of interest
> instead of opening/reading them all.


I mean optimistic because you are hoping that someone builds the
index before you need it.  The worst case scenario is that there
is no index and the first category pagelist request needs to 
search every page.



> > Does this mechanism work with apges which have not been
> > updated since before you added the link= feature (older
> > betas)?
> 
> It works with any pages created by versions 2.0.devel13 and later
> (including all of the 2.0.betas, which came after the devel series).


Cool, pm205 vs pm2wgrep benchmarks below:

Turns out the grep is still slightly faster in most
situations. The situation where it is slower is actually 
when I just search the Category pages.  My find is not
terribly smart: it does not used the pattern passed in to
limit the pages searched (the filtering is handled by
pmwiki afterwards so it still works).  This means that it
is actually searching the entire site and it is still
within a few percent of the index method's time!

-Martin



Page: Category.Renee

This page has a pagelist with all the Categories in the
sidebar and a pagelist with the Category.Renee which
searches the entire site.

pm205 run 1 (no index):

00.00 MarkupToHTML begin
00.41 MakePageList begin
02.65 BacklinksTo begin
02.66 BacklinksTo end
02.76 MakePageList scan
27.47 MakePageList sort
27.55 MakePageList update
27.55 LinkIndexUpdate begin
30.44 LinkIndexUpdate end
30.45 MakePageList end
31.36 MarkupToHTML end
31.74 MarkupToHTML begin
32.96 MakePageList begin
34.96 MakePageList scan
35.73 MakePageList sort
35.77 MakePageList update
35.77 MakePageList end
37.71 MarkupToHTML end
37.75 now


pm205 run 2:

00.00 MarkupToHTML begin
00.40 MakePageList begin
02.73 BacklinksTo begin
03.80 BacklinksTo end
03.95 MakePageList scan
05.04 MakePageList sort
05.12 MakePageList update
05.12 MakePageList end
05.97 MarkupToHTML end
06.31 MarkupToHTML begin
07.53 MakePageList begin
09.52 MakePageList scan
10.28 MakePageList sort
10.31 MakePageList update
10.31 MakePageList end
12.18 MarkupToHTML end
12.22 now

pm205 run 3:

00.00 MarkupToHTML begin
00.38 MakePageList begin
02.39 BacklinksTo begin
03.43 BacklinksTo end
03.57 MakePageList scan
04.58 MakePageList sort
04.66 MakePageList update
04.66 MakePageList end
05.61 MarkupToHTML end
05.92 MarkupToHTML begin
07.03 MakePageList begin
08.84 MakePageList scan
09.53 MakePageList sort
09.56 MakePageList update
09.56 MakePageList end
11.31 MarkupToHTML end
11.34 now


pm2 with grep run 1:

00.00 MarkupToHTML begin
00.56 MakePageList begin
03.53 MakePageList end
04.34 MarkupToHTML end
04.66 MarkupToHTML begin
05.82 MakePageList begin
09.08 MakePageList end
10.88 MarkupToHTML end
10.92 now

pm2 with grep run 2:

00.00 MarkupToHTML begin
00.41 MakePageList begin
03.36 MakePageList end
04.22 MarkupToHTML end
04.56 MarkupToHTML begin
05.80 MakePageList begin
08.97 MakePageList end
11.00 MarkupToHTML end
11.04 now

------------
Removed SideBar pagelist temporarily:

Category.Renee

pm205 run 1:

00.00 MarkupToHTML begin
00.41 MakePageList begin
02.62 BacklinksTo begin
03.77 BacklinksTo end
03.92 MakePageList scan
05.06 MakePageList sort
05.23 MakePageList update
05.23 MakePageList end
06.09 MarkupToHTML end
06.44 MarkupToHTML begin
08.39 MarkupToHTML end
08.67 now

pm205 run 2:

00.00 MarkupToHTML begin
00.39 MakePageList begin
02.48 BacklinksTo begin
03.52 BacklinksTo end
03.67 MakePageList scan
04.73 MakePageList sort
04.81 MakePageList update
04.81 MakePageList end
05.63 MarkupToHTML end
06.01 MarkupToHTML begin
07.92 MarkupToHTML end
08.19 now


pm2 with grep run 1:

00.00 MarkupToHTML begin
00.40 MakePageList begin
03.57 MakePageList end
04.42 MarkupToHTML end
04.76 MarkupToHTML begin
06.65 MarkupToHTML end
06.98 now

pm2 with grep run 2:

00.00 MarkupToHTML begin
00.39 MakePageList begin
03.37 MakePageList end
04.21 MarkupToHTML end
04.55 MarkupToHTML begin
06.45 MarkupToHTML end
06.78 now


---------------------

Category.SideBar-Menu?action=edit

pm205 run 1:
00.00 MarkupToHTML begin
02.03 MarkupToHTML end
02.40 MarkupToHTML begin
03.64 MakePageList begin
05.69 MakePageList scan
06.41 MakePageList sort
06.45 MakePageList update
06.45 MakePageList end
08.46 MarkupToHTML end
08.57 MarkupToHTML begin
09.52 MarkupToHTML end
09.53 now

pm205 run 2:
00.00 MarkupToHTML begin
02.18 MarkupToHTML end
02.56 MarkupToHTML begin
03.87 MakePageList begin
06.11 MakePageList scan
06.88 MakePageList sort
06.91 MakePageList update
06.91 MakePageList end
09.06 MarkupToHTML end
09.17 MarkupToHTML begin
10.20 MarkupToHTML end
10.22 now


pm2 with grep run 1:

00.00 MarkupToHTML begin
02.05 MarkupToHTML end
02.42 MarkupToHTML begin
03.72 MakePageList begin
07.19 MakePageList end
09.26 MarkupToHTML end
09.37 MarkupToHTML begin
10.42 MarkupToHTML end
10.43 now

pm2 with grep run 2:

00.00 MarkupToHTML begin
01.99 MarkupToHTML end
02.35 MarkupToHTML begin
03.66 MakePageList begin
07.04 MakePageList end
09.04 MarkupToHTML end
09.15 MarkupToHTML begin
10.13 MarkupToHTML end
10.15 now