[pmwiki-users] Re: RE: Fixed: Error in cookbook rename function

[Joachim Durchholz]

chr at home.se wrote:
> On Tue, 31 May 2005, Patrick R. Michaud wrote:
>>So, if someone knows a good way to do regression tests on web output, we
>>can start building them.
> 
> I assume you mean aside from simply using curl or wget to retrieve the 
> page and compare the output?
> 
> Using a browser like lynx, we could also extract a "text only" version of 
> the page and compare that output.

The problem with skinnable software like PmWiki is how to distinguish 
ephemeral from essential output. Say, if Pm changes the standard skin, 
how should the comparison program determine which changes are due to 
skin changes (hence OK) and which are bugs?
It could be worse though (say, if the program had a real GUI - HTML is, 
thankfully, text-based).

Also, we should avoid simply comparing output. With that approach, 
almost every change in PmWiki would require adapting the test suite to 
match the changed output. This not only is far more work than a single 
person would (or should) care to shoulder, it also introduces the danger 
that regression test failures become all too common and get ignored - 
and if they are ignored during an update, we end up with buggy 
regression tests that aren't reliable enough to be worth the effort.


The skinning variations can be kept under control by using a trivial 
test skin. (Of course, we'd need more elaborate skins for testing the 
skinning mechanisms themselves.)

Still, to avoid overly complex problems, the test tool should extract 
only those passages of the output that are relevant to a particular 
test. E.g. to test the ''' markup, we could write a wiki page like this:

TEST [='''=]

'''this should be strong'''

EXPECT

[=<strong>this should be strong</strong>=]

/TEST

and have the test tool compare the output between TEST and EXPECT with 
the output between EXPECT and /TEST. The ''' on the TEST line is just 
for giving the test a name, so that it can be identified in failure logs 
and such.

The good thing about this kind of writing tests is that contributors can 
easily devise their own tests :-)

Alternatively, we could devise a (:test <name>:) ... (:expect:) ... 
(:testend:) markup that passes through the <name> and anything after 
(:expect:) unchanged. (Actually seems to be simpler than requiring all 
tests to follow the TEST.../TEST form, with proper escaping in just the 
right places.)


Other tests would have to follow different guidelines (e.g. mailpost, 
wikipage storage, history mechanism, skin syntax, etc.). The essential 
thing is to make sure that every test checks just a single issue, and is 
as independent of other PmWiki features as possible - that way, if a 
PmWiki feature changes, there's no need to adapt all the tests that 
depend on that feature. ("Decoupling tests" seems to be the right word 
for that.)


Disclaimer: my experience with regression testing is restricted to a 
single Java application with a GUI front-end. There may be issues that 
we never hit - but decoupling tests was the key lesson that I learned 
from that experience. There's nothing worse than having a test suite 
that breaks because somebody changed something that a test shouldn't 
even check. You're lucky if that change was a bug, but many changes 
aren't, and updating a suite of 100+ tests just to adapt to a changed 
output order or something similarly ephemeral isn't fun if you could be 
doing something productive with your time.

Regards,
Jo