[pmwiki-users] Wiki speed diferences, no apparent reason

[Patrick R. Michaud]

On Wed, Mar 22, 2006 at 02:09:05PM +0100, Joachim Durchholz wrote:
> Philip Stitt schrieb:
> > Sometimes my website is unbearably slow. It will get that way for about
> > 20 to 30 minutes at a time. If I spend all day working on my site, I'll
> > see this happen maybe 5 or 6 times during the day, sometimes more.
> 
> I have a similar phenomenon. PmWiki pages are sometimes really, really slow.
> 
> I don't know whether that's Apache, FastCgi, or PmWiki. 

Or PHP.

Over the past few months I've been doing a *lot* of investigation
into why things get slow, and I still don't have any clear answers.
Here are some of my observations thus far:

* It almost has to be related to resource contention of some sort,
  as opposed to PmWiki being inherently slow.  As many have observed,
  there are times when PmWiki does run quite quickly, which means
  that its internal algorithms aren't intrinsically slow.

* pmwiki.org seems to slow down during the middle of the day
  (U.S. timezones), when there are a larger number of incoming
  requests.  This also points to a resource contention issue,
  although I can't be certain if it's PmWiki that is eating up
  the resources or some other application(s) on the same server.

* I think Apache has a bug somewhere that prevents it from
  recognizing dropped connections (and it seems to spinlock).  
  I've never been able to duplicate it on my own, but occasionally 
  when I do an Apache server-status request I'll find processes 
  that have been running for 5 minutes or more.  Often these are 
  PmWiki processes, but sometimes they're simple file requests 
  that seem to have been "hung" somehow.

  I know that for many of the PmWiki processes that seem to be
  taking a long time, it's because they're waiting to send output
  and not because they're actively processing content.

  I also find it bizarre that many of the PmWiki/PHP processes
  continue running beyond the "max execution time" parameter
  (30 seconds on pmwiki.org), which again would seem to indicate 
  a problem in PHP or Apache and not PmWiki.

* Until a few days ago, pmwiki.org's performance was extremely
  sensitive to incoming messages to the pmwiki-users mailing list.
  My server currently uses mailman for mailing lists and 
  (unfortunately) exim for mail transport, and both of those
  packages tend to be disk and memory intensive.  And mailman
  doesn't have strong support for throttling traffic.  So, an incoming
  message to pmwiki-users would rapidly spawn a few dozen exim
  processes that would instantly spike the server load and take
  a long time to complete.  Of course, as they take longer to
  complete, it becomes more likely that another message will arrive
  and spike things even further.

  Late last week I was finally able to find the mailman/exim settings
  needed so that these spikes don't happen, and performance has been
  greatly improved since then.

* My first inclination has been to think that the primary bottleneck
  is memory, since PHP (and PmWiki) can use up a lot of memory while
  running.  Unfortunately, PHP doesn't have good facilities for
  monitoring memory usage, so I haven't been able to confirm this
  theory.  However, I recently upgraded the RAM profile on my
  virtual server, and pmwiki.org still experiences the occasional
  slowdown, and AFAICT I'm not using all of my available memory
  anyway...

* Which leaves us with disk contention as the primary candidate for
  slowness -- i.e., the operating system is slow in responding
  to requests to open or read files.  I suspect this can be a real
  issue in shared hosting or virtual server environments, where
  there are many processes all trying to access disk resources
  in one way or another.  

* Anecdotally, when PmWiki is running slow I also seem to have
  high disk latency from my shell command prompts -- commands that
  don't require any disk accesses will run extremely fast, but
  there's a definite lag in any command that requires opening or
  reading a file of some sort.
  
> Well, for the moment, the best would be some kind of log that writes
> PmWiki's processing times to some kind of CSV file. 

I've been thinking I need to develop a diagnostic log for this
to run on pmwiki.org, but I hadn't thought about doing it as a CSV file.  
That's a really good idea.  

I think the easy way to approach it will be as a wrapper script
to pmwiki.php -- i.e., something that can grab the relevant variables
and save them to a log file.  Either that or I'll find a way to
log PmWiki's internal stopwatch (PmWiki has a "StopWatch" diagnostic
function that I use to bench mark things and find out where things
are slow).

> That way, one can
> see whether the lags are due to PmWiki or outside, and if they are
> within PmWiki, we might be able to correlate the slow requests with
> specific pages, or specific query strings, or various kinds of system
> load (memory/CPU load, number of active Apache processes, and whatnot).

I've looked for a long time and haven't found a correlation to
specific pages, query strings, or requests.  I'm pretty sure it's
more system related, i.e., with Apache, PHP, or the OS.

Pm