[pmwiki-users] Trouble with .pageindex when too much _new_ data to index (+ sqlite)

[Peter Bowers]

On Wed, Jan 28, 2015 at 10:10 PM, ABClf <languefrancaise at gmail.com> wrote:

> Is there something to do with the native search engine to avoid it failing
> each time amount of new data is too big ?


Try reindexing by multiple re-loads of the page Site.Reindex using the
following as the contents of Site.Reindex.php:

===(snip)===
<?php

# NOTE: For this to work it expects wiki.d/.reindex to be deleted before
# starting the reindex. Then simply reload this page multiple times until
# you get the message that the reindexing process is complete

#global $PageIndexTime, $WikiDir, $FarmD;
include_once("$FarmD/scripts/stdconfig.php");
include_once("$FarmD/scripts/pagelist.php");

SDV($ReindexFile, "$WorkDir/.reindex");
#echo "DEBUG: Attempting to delete $ReindexFile<br />\n";
#unlink($ReindexFile);

set_time_limit(120);
$PageIndexTime = 60;
$fp = @fopen($ReindexFile, "r");
if (!$fp) { // no .pageindex - start from scratch
    echo "DEBUG: A<br />\n";
    $pagelist = $WikiDir->ls();
    sort($pagelist);
    file_put_contents($ReindexFile, implode("\n", $pagelist));
    fixperms($ReindexFile);
} else { // we are assuming .pageindex has been created in order
    echo "DEBUG: B<br />\n";
    $pagelist = explode("\n", file_get_contents($ReindexFile));
    $lastpage = '';
    $ifp = @fopen($PageIndexFile, 'r');
    if ($ifp) {
        while (!feof($ifp)) {
          $line = fgets($ifp, 4096);
          while (substr($line, -1, 1) != "\n" && !feof($ifp))
            $line .= fgets($ifp, 4096);
          $i = strpos($line, ':');
          if ($i === false) continue;
          $n = substr($line, 0, $i);
          if ($n > $lastpage) $lastpage = $n;
          else break;
        }
        fclose($ifp);
        for ($i = 0; $i < sizeof($pagelist); $i++)
            if ($pagelist[$i] >= $lastpage) break;
        if ($pagelist[$i] == $lastpage)
            $pagelist = array_slice($pagelist, $i+1);
    }
}
echo "DEBUG: count=".count($pagelist)."<br />\n";
if (!count($pagelist)) {
    echo "Indexing complete. Deleting $ReindexFile<br />\n";
    if (file_exists($ReindexFile)) {
        fixperms($ReindexFile);
        unlink($ReindexFile); // for some reason this is giving err in
windows
    }
}
PageIndexUpdate($pagelist);
===(snip)===

I've only done a bare minimum of testing so there will probably be issues.
If you get it working it may be worthwhile to package it as a recipe for
others who may want an explicit reindex.

-Peter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.pmichaud.com/pipermail/pmwiki-users/attachments/20150129/4755e1c5/attachment.html>