Slicer3:Execution Model Testing

The design of Slicer3's Execution Model facilitates testing, since the command line programs (CLP's) can be run as stand-alone programs. CMake, CTest and CDash provide mechanism to build, run and report the results of tests.

Overview

The Execution Model testing uses ITK's test driver. This test driver is located in Insight/Code/Common/itkTestMain.h. The ITK test driver runs each test as a subprocess, collects the output of the test and wraps the test output into an XML descriptiomn of the test. Optionally, after all tests are run, CTest submits the results to a CDash dashboard. The test driver can also compare the test results with baseline images for the tests.

In ITK, many test are run from a test executable. Currently, in Slicer3, each test executable runs one test.

The CLP testing follows a number of conventions.

1. Tests reside in Slicer3/Applications/CLI/Testing
2. Most test data resides in Slicer3/Testing/Data/Input
3. Baselines for regression tests reside in Slicer3/Testing/Data/Baseline/CLI
4. Naming conventions:
   1. The test driver for a CLP program called Foo is in FooTest.cxx
   2. The baseline image for a CLP program called Foo is in FooTest.nhdr or some other image format.

Creating a Test

There are xx steps to create and run a test for a CLP program:

1. Create a test driver file in Applications/CLI/Testing
2. Design a test and optionally, create baseline images for the test
3. Configure the test
4. Run the test
5. Check in the test files

The following sections create a test for the CLP program CurvatureAnisotropicDiffusion.

Creating a Test Driver

In Applications/CLI/Testing create the file CurvatureAnisotropicDiffusionTest.cxx

#include <iostream>
#include "itkTestMain.h" 
void RegisterTests()
{
  REGISTER_TEST(CurvatureAnisotropicDiffusionTest);
}
#undef main
#define main CurvatureAnisotropicDiffusionTest
#include "CurvatureAnisotropicDiffusion.cxx"

Designing a Test

Find a set of data that you can use for the test. There are several sources of data in Slicer3 including:

1. Testing/Data/Input

The dataset should be fairly small, but be large enough to produce a result that you can verify.

1. Now run Slicer3 and load the data.
2. Select the program that you will test.
3. Define appropriate parameters.
4. Run the program.

If the program produces an image, use File->Save to save the resulting image as a Baseline in Testing/Data/Baseline/. The name of the baseline image should follow the naming conventions.

Configure Build and Add the test

Edit the CMakeLists.txt file in Application/CLI/Testing. For the CurvatureAnisotropicDiffusion program the additional lines are:

CMake Code	Explanation
################################## # CurvatureAnisotropicDiffusion tests set (CLP CurvatureAnisotropicDiffusion) add_executable(${CLP}Test ${CLP}Test.cxx) add_dependencies(${CLP}Test ${CLP}) target_link_libraries(${CLP}Test ITKIO)	This cmake code creates the test driver executable for the test. The target_link_libraries should be the same ones specified when the command line program is build in Applications/CLI. This ITK-based command line program only needs the ITKIO library.
add_test(${CLP}Test ${Slicer3_EXE} --launch ${CLP}Test --compare ${BASELINE}/${CLP}Test.nhdr ${TEMP}/${CLP}Test.nhdr ${CLP}Test --conductance 2 --timeStep 0.0625 --iterations 2 ${TEST_DATA}/MRHeadResampled.nhdr ${TEMP}/${CLP}Test.nhdr )	This cmake code creates the test and specifies the command line arguments. Tests are run using the --launch option of the Slicer3 program. This assures that all required environment variables are set properly. The --compare flag and its two arguments ${BASELINE}/${CLP}Test.nhdr and ${TEMP}/${CLP}Test.nhdr are only present for regression tests. Notice that the output of the test is stored in the ${TEMP} directory. For regression tests, this output is compared with the image in ${BASELINE}.

Configure the Test

Run CMake followed by make (for linux) or the Visual Studio IDE.

Run the Test

ctest -V -R CurvatureAnisotropicDiffusionTest

If the test passes, proceed.

Check in the Test Files

In Applications/CLI/Testing,

svn add CurvatureAnisotropicDiffusionTest.cxx
svn commit -m"ENH: New test for CurvatureAnisotropicDiffusion."
 CMakeList.txt CurvatureAnisotropicDiffusiontest.cxx

In Testing/Data/Input

Add and commit any new input data.

In Testing/data/Baseline/CLI

svn add CurvatureAnisotropicDiffusionTest.nhdr
        CurvatureAnisotropicDiffusionTest.raw.gz
svn commit -m"ENH: Baselines for CurvatureAnisotropicDiffusionTest"
        CurvatureAnisotropicDiffusionTest.nhdr
        CurvatureAnisotropicDiffusionTest.raw.gz