[crossfire] Unit tests

[Alex Schultz]

tchize wrote:

>But i must raise my shield against the idea to make a unit testing plugin. A 
>unit test is by definition a unit. This mean an execution unit. Unit testing 
>is something done at compile time. A unit testing framework gather all the 
>unit test, and runs them one after each other. For isolation purposes, each 
>test is generally run in it's own process and/or memory address space. After 
>each run, it gather informations on success and failures.
>
>This framework http://check.sourceforge.net/ looks very good to me, it has 
>automake integration, a clean api and does fork to have each test on it's own 
>address space. Unfortuately, it's not possible to run those unit test under 
>visual studio (visual studio does not have fork()). But it should be possible 
>to run it under cygwin or alike.
>
>List of existing unit testing frameworks:
>http://www.xprogramming.com/software.htm
>  
>
Actually, I was looking at those a bit, including that one at 
http://check.sourceforge.net/ however I think that using one of those 
frameworks might be somewhat bloated for our needs, and more flexibility 
would be good. I believe that writing our own minimalistic framework, 
tuned to crossfire, would be a better solution. Also, I'm not sure how 
good those frameworks are at doing a core dump in the event that a test 
segfaults, which given that a significant amount of the bugs that are 
getting fixes are segfault ones, is an important feature for any test 
framework that will be used for crossfire.

In terms of isolation, I agree that some is needed. I was thinking that 
forking for each test (perhaps starting a new server, with some argvs, 
would be an alternative method for platforms with no support for fork). 
One idea that Mark Wedel brought up on IRC a few days back, was possibly 
first running each test in isolation, then afterwards, run them in 
series without isolation, to make the test more robust.

In terms of as a plugin, what I meant, is that to test, it could be 
loaded at startup time using some of the plugin code, and essentially 
overrides the normal server operation and reads a text file defining the 
tests. I'm not sure that is the best idea, but that was what I was 
thinking at the time. One flaw with that anyways, is it would be good 
for defining high level tests (see next paragraph), but it would be 
rather poor for defining lower level tests of functions.

I think both high and low level tests are important, perhaps some 
'hybrid' ones too. High level being ones were it loads and runs a map, 
and checks conditions after a certain number of ticks. Low level being 
running an isolated server function. Hybrid ones being ones that load a 
map and check a low level server function on that map.

Also, thanks for clearing up a bit of that in the first paragraph of 
your original reply. I already became aware that some of what I said was 
not entirely accurate, but what you said clears it up further.

    --Alex Schultz