The Test Suite

What is it?

The test suite is a set of scripts and files that enable the testing of every single line of code in POPFile. As every programmer knows, adding a feature to a program or even fixing a bug may make some seemingly unrelated line of code break. The test suite helps to avoid bugs by ensuring that POPFile works as expected.


Test Suite

You will need the files that comprise the test suite which you will need to get with CVS. They do not come as part of the standard POPFile download.

Just checkout or update the POPFile engine:

cvs -d:pserver:[email protected]:/cvsroot/popfile login
cvs -z3 -d:pserver:[email protected]:/cvsroot/popfile co engine

(If this gives you an error just try again a few times, anonymous cvs access to sourceforge is often busy)


If you want to run the test suite or write tests for you own patches, you need a set of tools. In particular, you need the following programs:

  • Perl
  • Make
  • rm
  • cp

I guess that you already have Perl, if you need to get the other tools, take a look at the tools page. Make sure that all of these programs are installed on your system path.

How to run the test suite

Open a command line / console window, change to the directory in which POPFile and the test suite are installed and type:

make test

This will run all of the available tests and may take some time.

If you want to run a specific test, i.e. test a specific module, type:

 make test TESTARGS=ModuleName 

this will run the tests of one module only if the test exists. Note that the module name is case-sensitive.


If you are on Windows and you are not using the cygwin package, you should issue the command

attrib -r -h -s *.* /s /d

before running make. The Windows ports of rm otherwise will not delete certain directories which must be deleted for the test suite to pass without errors.

How it works

When you enter one of the above commands, make will run the script “” that can be found in the POPFile directory. This is a short Perl script that will run the tests, report the results, and which contains some testing functions (see below). The tests themselves are in files that can be globbed by the pattern tests/Test*.tst. I.e. they can be found in the “tests” subdirectory, shoud be prefixed with “Test”, and they have a “.tst” suffix.

The .tst files are actually Perl scripts. They start POPFile and the required modules, do their testing business, and then stop POPFile.

The “testing business” is really quite simple. You feed POPFile some data and see whether the results are correct. There are a couple of ways to do this. Here are some examples:

The first two can be found in TestMailParse.tst”:

 test_assert( $b->start() );

test_assert is one of the testing functions in, mentioned above. It is used to check that the return value of the function start() in is true. Use test_assert() when you want something to be true / non-null. Like the other testing functions, test_assert() will make issue an error message if the assertion is not met.

 test_assert_equal( $cl->map_color( 'red' ), 'ff0000' ); 

test_assert_equal() can be used when you need to make sure that two values are equal. In this case, the return value of the function map_color() is checked against the desired return value. In other words, the line above makes sure that MailParse:map_color() maps the color definition “red” to the hex value “ff0000”.

 test_assert_regexp( $temp, 'before' );

This one is from TestBayes.tst. It asserts that a regular expression matches a string. Here, the string in $temp must match the regular expression “before”.

The counterpart of test_assert_regexp() is test_assert_not_regexp. It asserts that the string is not matched by the regular expression.

More Detail on How it Works

In the engine directory there's a Makefile which does various things but has two targets called test and coverage. Both of these are passed into a Makefile in the tests/ directory which understands what to do about them.

So if you type make test in the engine directory it will execute make test in the tests/ directory which will run the Perl scripts found in the engine directory. controls the tests being run. looks in the tests/ directory for files that end .tst, each file is associated with a POPFile module (e.g. TestWordMangle.tst tests Within the tst (which is basically a Perl script itself) there are assertions of the form test_assert and test_assert_equal which are used to say that something must be true.

For example, in TestWordMangle.tst you'll find:

test_assert_equal( $w->mangle( 'BIGWORD' ), 'bigword' );
test_assert_equal( $w->mangle( 'BIG+/?*|()[]{}^$.WORD' ), 'big..............word');
test_assert_equal( $w->mangle('awordthatisfartolongtobeacceptableforuseinclassification' ), // );
test_assert_equal( $w->mangle( 'A1234BEF66' ), // );
test_assert_equal( $w->mangle( 'BIG:WORD' ), 'bigword' );
test_assert_equal( $w->mangle( 'BIG:WORD', 1 ), 'big:word' );

Each line tests the mangle() API and makes sure that it produces the correct result. If it fails then will make a note and output a detailed error at the end of the run. If it works then just keeps count.

At the end of the run summarizes the situation.


Now to make sure that we test everything we can there's also a coverage option. This counts the percentage of lines in POPFile's source that have actually been executed by the test suite. To run that you do gmake coverage and it runs the test suite and keeps track on a line by line basis of which lines were executed. At the end it summarizes the lines touched. The goal would be to have 100% of the lines of POPFile tested (a tough goal). At the end it writes out an HTML file per file tested so that you can see exactly which lines were executed and which were not. There's an example of this here for Green lines were tested and red were not.

testsuite.txt · Last modified: 2008/02/09 19:09 by

Should you find anything in the documentation that is incomplete, unclear, outdated or just plain wrong, please let us know and leave a note in the Documentation Forum.

Recent changes RSS feed Donate Driven by DokuWiki
The content of this wiki is protected by the GNU Fee Documentation License