Slicer3:Execution Model Testing
The design of Slicer3's Execution Model facilitates testing, since the command line programs (CLP's) can be run as stand-alone programs. CMake, CTest and CDash provide mechanism to build, run and report the results of tests.
Contents
Overview
The Execution Model testing uses ITK's test driver. This test driver is located in Insight/Code/Common/itkTestMain.h. The ITK test driver runs each test as a subprocess, collects the output of the test and wraps the test output into an XML description of the test. Optionally, after all tests are run, CTest submits the results to the Slicer3 CDash dashboard. The test driver can also compare the test results with baseline images for the tests.
In ITK, many test are run from a test executable. Currently, in Slicer3, each test executable runs one test.
The CLP testing follows a number of conventions.
- Tests reside in Slicer3/Applications/CLI/Testing
- Most test data resides in Slicer3/Testing/Data/Input
- Baselines for regression tests reside in Slicer3/Testing/Data/Baseline/CLI
- Naming conventions:
- The test driver for a CLP program called Foo is in FooTest.cxx
- The baseline image for a CLP program called Foo is in FooTest.nhdr or some other image format.
Creating a Test
There are 5 steps to create and run a test for a CLP program:
- Create a test driver file
- Design a test and optionally, create baseline images for the test
- Configure the test
- Run the test
- Check in the test files
The following sections create a test for the CLP program CurvatureAnisotropicDiffusion.
Creating a Test Driver
In Applications/CLI/Testing create the file CurvatureAnisotropicDiffusionTest.cxx
| C++ Code | Explanation |
|---|---|
#include <iostream>
#include "itkTestMain.h"
void RegisterTests()
{
REGISTER_TEST(CurvatureAnisotropicDiffusionTest);
}
#undef main
#define main CurvatureAnisotropicDiffusionTest
#include "CurvatureAnisotropicDiffusion.cxx"
|
This is the ITK test driver. ITK tests usually REGISTER several tests. However, because of the nature of Slicer3 command line programs, only one test can exist in each test driver. |
Designing a Test
Find a set of data that you can use for the test. There are several sources of data in Slicer3 including:
- Testing/Data/Input
- Modules/EMSegment/Testing/TestData/TutorialTest2/VolumeData and its sub directories.
- Libs/MRML/Testing/TestData
The dataset should be fairly small, but be large enough to produce a result that you can verify.
- Now run Slicer3 and load the data.
- Select the program that you will test.
- Define appropriate parameters.
- Run the program.
If the program produces an image, use File->Save to save the resulting image as a Baseline in Testing/Data/Baseline/CLI. The name of the baseline image should follow the naming conventions described above.
Configure Build and Add the test
Edit the CMakeLists.txt file in Applications/CLI/Testing. For the CurvatureAnisotropicDiffusion program the additional lines are:
| CMake Code | Explanation |
|---|---|
##################################
# CurvatureAnisotropicDiffusion tests
set (CLP CurvatureAnisotropicDiffusion)
add_executable(${CLP}Test ${CLP}Test.cxx)
add_dependencies(${CLP}Test ${CLP})
target_link_libraries(${CLP}Test ITKIO)
|
This cmake code creates the test driver executable for the test. The target_link_libraries should be the same ones specified when the command line program is build in Applications/CLI. This ITK-based command line program only needs the ITKIO library. |
add_test(${CLP}Test ${Slicer3_EXE} --launch ${CLP}Test
--compare ${BASELINE}/${CLP}Test.nhdr
${TEMP}/${CLP}Test.nhdr
${CLP}Test
--conductance 2
--timeStep 0.0625
--iterations 2
${TEST_DATA}/MRHeadResampled.nhdr
${TEMP}/${CLP}Test.nhdr
)
|
This cmake code creates the test and specifies the command line arguments. Tests are run using the --launch option of the Slicer3 program. This assures that all required environment variables are set properly. The --compare flag and its two arguments ${BASELINE}/${CLP}Test.nhdr and ${TEMP}/${CLP}Test.nhdr are only present for regression tests. Notice that the output of the test is stored in the ${TEMP} directory. For regression tests, this output is compared with the image in ${BASELINE}.[1] |
[1] If the output of your command line module is not a volume, but is a different type of file on disk, you can use the cmake compare command to test the output. See an example with the MergeModels test checked in here.
ITK testing framework is defined in $ITK_HOME/Code/Common/itkTestMain.h, which utilizes itkDifferenceImageFilter for baseline comparison. Additional regression testing functionality allows to specify the following comparison parameters (NOTE: the order is important: these parameters must follow in this order after --compare image1 image2):
| parameter | explanation |
|---|---|
|
--compareNumberOfPixelsTolerance <integer> (default 0) |
if this number (or less) of pixels is different from the baseline, the test will succeed |
|
--compareRadiusTolerance <integer> (default 0) |
search in the neighborhood of this radius for the matching pixel (be careful with this, since setting this to non-0 will increase drastically the execution time of your test!) |
|
--compareIntensityTolerance <float> (default 2.0) |
pixels with the intensities that differ by the specified value are considered same |
Configure the Test
Run CMake followed by make (for linux) or the Visual Studio IDE.
Run the Test
ctest -V -R CurvatureAnisotropicDiffusionTest
If the test passes, proceed.
Check in the Test Files
svn add CurvatureAnisotropicDiffusionTest.cxx svn commit -m"ENH: New test for CurvatureAnisotropicDiffusion." CMakeList.txt CurvatureAnisotropicDiffusiontest.cxx
In Testing/Data/Input add and commit any new input data.
svn add CurvatureAnisotropicDiffusionTest.nhdr
CurvatureAnisotropicDiffusionTest.raw.gz
svn commit -m"ENH: Baselines for CurvatureAnisotropicDiffusionTest"
CurvatureAnisotropicDiffusionTest.nhdr
CurvatureAnisotropicDiffusionTest.raw.gz
