Slicer3:Execution Model Testing

The design of Slicer3's Execution Model facilitates testing, since the command line programs (CLP's) can be run as stand-alone programs. CMake, CTest and CDash provide mechanism to build, run and report the results of tests.

Overview

The Execution Model testing uses ITK's test driver. This test driver is located in Insight/Code/Common/itkTestMain.h. The ITK test driver runs each test as a subprocess, collects the output of the test and wraps the test output into an XML description of the test. Optionally, after all tests are run, CTest submits the results to the Slicer3 CDash dashboard. The test driver can also compare the test results with baseline images for the tests.

In ITK, many test are run from a test executable. Currently, in Slicer3, each test executable runs one test.

The CLP testing follows a number of conventions.

1. Tests reside in Slicer3/Applications/CLI/Testing
2. Most test data resides in Slicer3/Testing/Data/Input
3. Baselines for regression tests reside in Slicer3/Testing/Data/Baseline/CLI
4. Naming conventions:
   1. The test driver for a CLP program called Foo is in FooTest.cxx
   2. The baseline image for a CLP program called Foo is in FooTest.nhdr or some other image format.

Creating a Test

There are 5 steps to create and run a test for a CLP program:

1. Create a test driver file
2. Design a test and optionally, create baseline images for the test
3. Configure the test
4. Run the test
5. Check in the test files

The following sections create a test for the CLP program CurvatureAnisotropicDiffusion.

Creating a Test Driver

In Applications/CLI/Testing create the file CurvatureAnisotropicDiffusionTest.cxx

C++ Code	Explanation
#include <iostream> #include "itkTestMain.h" void RegisterTests() { REGISTER_TEST(CurvatureAnisotropicDiffusionTest); } #undef main #define main CurvatureAnisotropicDiffusionTest #include "CurvatureAnisotropicDiffusion.cxx"	This is the ITK test driver. ITK tests usually REGISTER several tests. However, because of the nature of Slicer3 command line programs, only one test can exist in each test driver.

Designing a Test

Find a set of data that you can use for the test. There are several sources of data in Slicer3 including:

1. Testing/Data/Input
2. Modules/EMSegment/Testing/TestData/TutorialTest2/VolumeData and its sub directories.
3. Libs/MRML/Testing/TestData

The dataset should be fairly small, but be large enough to produce a result that you can verify.

1. Now run Slicer3 and load the data.
2. Select the program that you will test.
3. Define appropriate parameters.
4. Run the program.

If the program produces an image, use File->Save to save the resulting image as a Baseline in Testing/Data/Baseline/CLI. The name of the baseline image should follow the naming conventions described above.

Configure Build and Add the test

Edit the CMakeLists.txt file in Applications/CLI/Testing. For the CurvatureAnisotropicDiffusion program the additional lines are:

CMake Code	Explanation
################################## # CurvatureAnisotropicDiffusion tests set (CLP CurvatureAnisotropicDiffusion) add_executable(${CLP}Test ${CLP}Test.cxx) add_dependencies(${CLP}Test ${CLP}) target_link_libraries(${CLP}Test ITKIO)	This cmake code creates the test driver executable for the test. The target_link_libraries should be the same ones specified when the command line program is build in Applications/CLI. This ITK-based command line program only needs the ITKIO library.
add_test(${CLP}Test ${Slicer3_EXE} --launch ${CLP}Test --compare ${BASELINE}/${CLP}Test.nhdr ${TEMP}/${CLP}Test.nhdr ${CLP}Test --conductance 2 --timeStep 0.0625 --iterations 2 ${TEST_DATA}/MRHeadResampled.nhdr ${TEMP}/${CLP}Test.nhdr )	This cmake code creates the test and specifies the command line arguments. Tests are run using the --launch option of the Slicer3 program. This assures that all required environment variables are set properly. The --compare flag and its two arguments ${BASELINE}/${CLP}Test.nhdr and ${TEMP}/${CLP}Test.nhdr are only present for regression tests. Notice that the output of the test is stored in the ${TEMP} directory. For regression tests, this output is compared with the image in ${BASELINE}.[1]

[1] If the output of your command line module is not a volume, but is a different type of file on disk, you can use the cmake compare command to test the output. See an example with the MergeModels test checked in here.

ITK testing framework is defined in $ITK_HOME/Code/Common/itkTestMain.h, which utilizes itkDifferenceImageFilter for baseline comparison. Additional regression testing functionality allows to specify the following comparison parameters (NOTE: the order is important: these parameters must follow in this order after --compare image1 image2):

parameter	explanation
--compareNumberOfPixelsTolerance <integer> (default 0)	if this number (or less) of pixels is different from the baseline, the test will succeed
--compareRadiusTolerance <integer> (default 0)	search in the neighborhood of this radius for the matching pixel (be careful with this, since setting this to non-0 will increase drastically the execution time of your test!)
--compareIntensityTolerance <float> (default 2.0)	pixels with the intensities that differ by the specified value are considered same

Configure the Test

Run CMake followed by make (for linux) or the Visual Studio IDE.

Run the Test

ctest -V -R CurvatureAnisotropicDiffusionTest

If the test passes, proceed.

Check in the Test Files

In Applications/CLI/Testing,

svn add CurvatureAnisotropicDiffusionTest.cxx
svn commit -m"ENH: New test for CurvatureAnisotropicDiffusion."
 CMakeList.txt CurvatureAnisotropicDiffusiontest.cxx

In Testing/Data/Input add and commit any new input data.

In Testing/Data/Baseline/CLI

svn add CurvatureAnisotropicDiffusionTest.nhdr
        CurvatureAnisotropicDiffusionTest.raw.gz
svn commit -m"ENH: Baselines for CurvatureAnisotropicDiffusionTest"
        CurvatureAnisotropicDiffusionTest.nhdr
        CurvatureAnisotropicDiffusionTest.raw.gz

References

• ITK Procedure for adding a test (ITK wiki)
• Regression testing (Wikipedia)