Slicer3:Execution Model Testing
The design of Slicer3's Execution Model facilitates testing, since the command line programs (CLP's) can be run as stand-alone programs. CMake, CTest and CDash provide mechanism to build, run and report the results of tests.
Contents
Overview
The Execution Model testing uses ITK's test driver. This test driver is located in Insight/Code/Common/itkTestMain.h. The ITK test driver runs each test as a subprocess, collects the output of the test and wraps the test output into an XML description of the test. Optionally, after all tests are run, CTest submits the results to a CDash dashboard. The test driver can also compare the test results with baseline images for the tests.
In ITK, many test are run from a test executable. Currently, in Slicer3, each test executable runs one test.
The CLP testing follows a number of conventions.
- Tests reside in Slicer3/Applications/CLI/Testing
- Most test data resides in Slicer3/Testing/Data/Input
- Baselines for regression tests reside in Slicer3/Testing/Data/Baseline/CLI
- Naming conventions:
- The test driver for a CLP program called Foo is in FooTest.cxx
- The baseline image for a CLP program called Foo is in FooTest.nhdr or some other image format.
Creating a Test
There are xx steps to create and run a test for a CLP program:
- Create a test driver file in Applications/CLI/Testing
- Design a test and optionally, create baseline images for the test
- Configure the test
- Run the test
- Check in the test files
The following sections create a test for the CLP program CurvatureAnisotropicDiffusion.
Creating a Test Driver
In Applications/CLI/Testing create the file CurvatureAnisotropicDiffusionTest.cxx
| CMake Code | Explanation |
|---|---|
#include <iostream>
#include "itkTestMain.h"
void RegisterTests()
{
REGISTER_TEST(CurvatureAnisotropicDiffusionTest);
}
#undef main
#define main CurvatureAnisotropicDiffusionTest
#include "CurvatureAnisotropicDiffusion.cxx"
|
This is the ITK test driver. ITK tests usually REGISTER several tests. However, because of the nature of Slicer3 command line programs, only one test can exist in each test driver. |
Designing a Test
Find a set of data that you can use for the test. There are several sources of data in Slicer3 including:
- Testing/Data/Input
The dataset should be fairly small, but be large enough to produce a result that you can verify.
- Now run Slicer3 and load the data.
- Select the program that you will test.
- Define appropriate parameters.
- Run the program.
If the program produces an image, use File->Save to save the resulting image as a Baseline in Testing/Data/Baseline/. The name of the baseline image should follow the naming conventions.
Configure Build and Add the test
Edit the CMakeLists.txt file in Application/CLI/Testing. For the CurvatureAnisotropicDiffusion program the additional lines are:
| CMake Code | Explanation |
|---|---|
##################################
# CurvatureAnisotropicDiffusion tests
set (CLP CurvatureAnisotropicDiffusion)
add_executable(${CLP}Test ${CLP}Test.cxx)
add_dependencies(${CLP}Test ${CLP})
target_link_libraries(${CLP}Test ITKIO)
|
This cmake code creates the test driver executable for the test. The target_link_libraries should be the same ones specified when the command line program is build in Applications/CLI. This ITK-based command line program only needs the ITKIO library. |
add_test(${CLP}Test ${Slicer3_EXE} --launch ${CLP}Test
--compare ${BASELINE}/${CLP}Test.nhdr
${TEMP}/${CLP}Test.nhdr
${CLP}Test
--conductance 2
--timeStep 0.0625
--iterations 2
${TEST_DATA}/MRHeadResampled.nhdr
${TEMP}/${CLP}Test.nhdr
)
|
This cmake code creates the test and specifies the command line arguments. Tests are run using the --launch option of the Slicer3 program. This assures that all required environment variables are set properly. The --compare flag and its two arguments ${BASELINE}/${CLP}Test.nhdr and ${TEMP}/${CLP}Test.nhdr are only present for regression tests. Notice that the output of the test is stored in the ${TEMP} directory. For regression tests, this output is compared with the image in ${BASELINE}. |
Configure the Test
Run CMake followed by make (for linux) or the Visual Studio IDE.
Run the Test
ctest -V -R CurvatureAnisotropicDiffusionTest
If the test passes, proceed.
Check in the Test Files
In Applications/CLI/Testing,
svn add CurvatureAnisotropicDiffusionTest.cxx svn commit -m"ENH: New test for CurvatureAnisotropicDiffusion." CMakeList.txt CurvatureAnisotropicDiffusiontest.cxx
In Testing/Data/Input
Add and commit any new input data.
In Testing/data/Baseline/CLI
svn add CurvatureAnisotropicDiffusionTest.nhdr
CurvatureAnisotropicDiffusionTest.raw.gz
svn commit -m"ENH: Baselines for CurvatureAnisotropicDiffusionTest"
CurvatureAnisotropicDiffusionTest.nhdr
CurvatureAnisotropicDiffusionTest.raw.gz
