[pmwiki-users] Testing and quality control (Was: Pre-announcing 2.2.0 non-beta release)

[Christian Ridderström]

On Tue, 20 Jan 2009, Patrick R. Michaud wrote:

> On Wed, Jan 21, 2009 at 01:15:36PM +1300, John Rankin wrote:

> So, "[there] should be a quality control [...] of PmWiki extensions" 
> is another one of those places that I fear handwaves away the difficult 
> part of the whole problem.  :-)

I'd like to rephrase that as: 'So, "[there] should be a quality control 
[...] of PmWiki and its extensions"'.
 		^^^^^^^

> Indeed, if it was easy, we'd probably already be doing it.  :-)

True, but I believe there are some things we can do, especially in terms 
of testing. I don't think we'd ever be able to do complete tests, but I 
also don't believe this means we should skip trying to do some simple 
tests.

One practice I use for my custom recipes is to simply create 
semi-automatic test cases. Typically this just involves using the recipe 
on one or more wiki pages and the visually inspecting the result. It's not 
automatic because the computer does not verify the result, but the actual 
running of the test is as simple as reloading the wiki page.

So, something that I think would help is setting up separate public areas 
for the testing of different recipes. These areas could be a wiki groups 
or separate wikis and the purpose of being separate is to avoid 
interference between recipes. The public aspect mean that many people can 
contribute with the testing.

I think a test pages should contain something like this:
* Something that identifies the recipe being tested, which could be
   implicit through the name of the group or the wiki.
* Something that identifies what this particular test is testing.
* The expected result, so that a human can verify the result.
* The actual test
* The last date that the test passed and with which revision.
   (This might be annoying to update if you have many pages, in which

To make it easy to inspect the results, it's nice to have a single page 
that pulls together all the test pages.

The drawback with the scheme above is that it'll require at least a little 
bit more involvement by e.g. Petko in terms of setting up the testing 
area.

An advantage is that these test pages also serve as documentation examples 
and possibly also for design testing. However, the latter would be much 
better if it was possible for the recipe designer to import and export a 
bunch of test pages. What I mean is that the designer has his own set of 
test pages while developing a new (version of the) recipe, and the exports 
them to the separate test area.  Running the tests again on a clean 
installation will also detect some problems with dependencies on other 
recipes.

Oh, and finally I should of course say that just by having the recipes 
enabled for their corresponding cookbook page we do have some of this kind 
of testing. A small further step would be to create a '<recipe>-Test' page 
in the cookbook directory, or simply a test page in the group 
'Cookbook-Test/<recipe>'.  For many (most?) recipes this might be enough 
in terms of testing. However, it'll probably not be enough for skins, as 
I've noticed that you sometimes have to add extra pages in Site. etc.

Best regards,
/Christian

-- 
Christian Ridderström, +46-8-768 39 44            http://www.md.kth.se/~chr