2009 Winter Project Week Command Line Program Testing

Return to Project Week Main Page

Key Investigators

• Bill Lorensen (His Basement)

Objective

There is NA-MIC funded infrastructure in place to regression test the command line programs found in Applications/CLI. Surprisingly, of the over 50 command line programs in the current Slicer3 distribution, only 11 have tests. This project will illustrate the importance of testing and show how easy it is to create tests for command line programs using the existing infrastructure. The project will document the process, create tests and help others to create tests.

Approach, Plan

Using the existing infrastructure, the project will create regression tests for many of the existing command line programs in Applications/CLI. The project will provide expert assistance to those who feel that creating a test is difficult. A Wiki page will provide step-by-step procedures to test the command line programs.

Progress

There is really no excuse for the lack of testing in Applications/CLI. The infrastructure that has been in place for a few years makes it almost painless to test command line programs. Better documentation and peer pressure will help improve the situation.

Tests have been created for all of the command line programs in Applications/CLI (not including sub directories). Three types of tests exist:

1. --help tests just run the command line program with the --help flag. These tests are as basic as a test can be. All they really test is whether or not the command line program can build, link and report help.
2. Smoke tests run the program put do not check the results of the program. These are one step above the --help tests and should be the minimum test for each program.
3. Regression tests run the program and compare the resulting output with a baseline output. Currently only image regression testing is available. Each command line program that produces an image should have a regression test.

Documentation: Slicer3 Execution Model Testing

Status

Starting (current) status:

1. 5 (20) regression tests
2. 3 (9) smoke tests
3. 30 (9) with no tests

During Project Week, I'll work with those folks responsible for programs whose status is untested.

The table shows each command line program and its testing status. Under Test Type, the item in parentheses is a test that has been added as part of this project. Even the untested programs have at a minimum test that runs the program with --help. We still consider these programs untested since every CLP program responds to --help. However, at least these test programs can link and run.

Program	Test Type	Command Line
Add	Regression	${TEST_DATA}/CTHeadAxial.nhdr ${TEST_DATA}/CTHeadAxial.nhdr ${TEMP}/AddTest.nhdr
AffineRegistration	None (--help)
BSplineDeformableRegistration	Smoke	--resampledmovingfilename ${TEMP}/BSplineDeformableRegistrationTest00.nhdr --outputtransform ${TEMP}/BSplineDeformableRegistrationTest00Transform.txt --default 0 --maximumDeformation 1.0 --constrain --spatialsamples 10000 --histogrambins 32 --gridSize 5 --iterations 50 ${TEST_DATA}/CTHeadAxial.nhdr ${TEST_DATA}/CTHeadAxial.nhdr
CheckerBoard	Regression	--checkerPattern 5,3,2 ${TEST_DATA}/CTHeadAxial.nhdr ${TEST_DATA}/CTHeadAxialDoubled.nhdr ${TEMP}/CheckerBoardTest.nhdr
ConfidenceConnected	None (Regression)	--smoothingIterations 1 --timestep 0.0625 --iterations 5 --multiplier 2.5 --neighborhood 1 --labelvalue 2 --seed 9.71622,-8.1,14.2189 --seed -12.9459,-8.1,18.577 ${TEST_DATA}/MRHeadResampled.nhdr ${TEMP}/ConfidenceConnectedTest.nhdr
CurvatureAnisotropicDiffusion	None (Regression)	--conductance 2 --timeStep 0.0625 --iterations 2 ${TEST_DATA}/MRHeadResampled.nhdr ${TEMP}/CurvatureAnisotropicDiffusionTest.nhdr
DicomToNrrdConverter	None (Smoke)	${TEST_DATA}/CTHeadAxialDicom ${TEMP} CTHeadAxialFromDicom.nhdr
DiffusionTensorEstimation	None (Smoke)	--otsuomegathreshold 0.5 ${MRML_TEST_DATA}/helix-DWI.nhdr ${TEMP}/DiffusionTensorEstimationTest_dti.nhdr ${TEMP}/DiffusionTensorEstimationTest_baseline.nhdr ${TEMP}/DiffusionTensorEstimationTest_otsu.nhdr
DiffusionTensorMathematics	None (Smoke)	--enumeration FractionalAnisotropy ${MRML_TEST_DATA}/helix-DWI.nhdr ${TEMP}/DiffusionTensorMathematicsTest.nhdr
DiffusionTensorTest	None (--help)
DiffusionWeightedTest	None (--help)
DWIDicomLoad	None (--help)
ExecutionModelTour	None (Smoke)	--integer 30 --double 30 -f 1.3,2,-14 --string_vector foo,bar,foobar --enumeration Bill --boolean1 --transform1 ${TEST_DATA}/ExecutionModelTourTest.mrml\#vtkMRMLLinearTransformNode1 --transform2 ${TEST_DATA}/ExecutionModelTourTest.mrml\#vtkMRMLLinearTransformNode2 ${TEST_DATA}/MRHeadResampled.nhdr ${TEST_DATA}/CTHeadAxial.nhdr
FreesurferSurfaceSectionExtraction	None (--help)
GaussianBlurImageFilter	Regression	--sigma 2.5 ${TEST_DATA}/CTHeadAxial.nhdr ${TEMP}/GaussianBlurImageFilterTest.nhdr
GradientAnisotropicDiffusion	None (Regression)	--conductance 2 --timeStep 0.0625 --iterations 2 ${TEST_DATA}/MRHeadResampled.nhdr ${TEMP}/GradientAnisotropicDiffusionTest.nhdr
GrayscaleFillHoleImageFilter	None (Regression)	${TEST_DATA}/MRHeadResampled.nhdr ${TEMP}/GrayscaleFillHoleTest.nhdr
GrayscaleGrindPeakImageFilter	None (Regression)	${TEST_DATA}/MRHeadResampled.nhdr ${TEMP}/GrayscaleGrindPeakTest.nhdr
GrayscaleModelMaker	None (Smoke)	--threshold 300 --name CTFace --smooth 15 --decimate 0.95 --splitnormals --pointnormals ${TEST_DATA}/CTHeadAxial.nhdr ${TEMP}GrayscaleModelMakerTest.vtp
HistogramMatching	None (Regression)	--numberOfHistogramLevels 64 --numberOfMatchPoints 10 ${TEST_DATA}/CTHeadAxial.nhdr ${TEST_DATA}/MRHeadResampled.nhdr
ImageLabelCombine	None (--help)
ImageReadDicomWrite	None (Regression)	--patientName Austrialian --patientID 8775070 --patientComments "A volunteer" --studyID 123456 --studyDate 20090102 --studyComments Resampled --studyDescription None --modality CT --manufacturer "GE Medical Systems" --model CT8800 --seriesNumber 123456 --seriesDescription None --dicomDirectory ${TEMP} --dicomPrefix CTHeadAxialDicom ${TEST_DATA}/CTHeadAxial.nhdr
LabelMapSmoothing	None (Regression)	--labelToSmooth 1 --numberOfIterations 50 --maxRMSError 0.01 --gaussianSigma 3 ${TEST_DATA}/CTHeadResampledOtsuSegmented.nhdr ${TEMP}/LabelMapSmoothingTest.nhdr
LinearRegistration	Smoke	--resampledmovingfilename ${TEMP}/LinearRegistrationTest00.nhdr --outputtransform ${TEMP}/LinearRegistrationTest00Transform.txt --initialtransform ${TEST_DATA}/itkAffineTransform00.txt --spatialsamples 10000 --histogrambins 32 --iterations 50 --learningrate 0.01 --translationscale 100 ${TEST_DATA}/CTHeadAxial.nhdr ${TEST_DATA}/CTHeadAxial.nhdr
MedianImageFilter	Regression	--neighborhood 1,2,3 ${TEST_DATA}/CTHeadAxial.nhdr ${TEMP}/MedianImageFilterTest.nhdr
ModelMaker	None (Smoke)	--labels 2 --modelSceneFile ${TEMP}/ModelMakerTest.mrml\#vtkMRMLModelHierarchyNode1 ${MRML_TEST_DATA}/helix-roi-lable2.nrrd
MultipleModelsExample	None (--help)
OrientImage	Regression	--orientation Axial ${MRML_DATA}/fixed.nrrd ${TEMP}/OrientImageTestAxial.nrrd --orientation Coronal ${MRML_DATA}/fixed.nrrd ${TEMP}/OrientImageTestCoronal.nrrd --orientation Sagittal ${MRML_DATA}/fixed.nrrd ${TEMP}/OrientImageTestSagittal.nrrd
OtsuThresholdImageFilter	None (Regression)	--insideValue 0 --outsideValue 255 --numberOfBins 128 ${TEST_DATA}/CTHeadResampled.nhdr ${TEMP}/OtsuThresholdImageFilterTest.nhdr
OtsuThresholdSegmentation	None (Regression)	--brightObjects --numberOfBins 128 --faceConnected --minimumObjectSize 1000 ${TEST_DATA}/CTHeadResampled.nhdr ${TEMP}/OtsuThresholdSegmentationTest.nhdr
ProbeVolumeWithModel	None (--help)
ResampleVolume2	None (Regression)	--Reference ${TEST_DATA}/MRHeadResampled.nhdr --number_of_thread 0 --transform_matrix 1,0,0,0,1,0,0,0,1,0,0,0 --transform rt --space LPS --interpolation linear --window_function c --spline_order 3 --spacing 0,0,0 --size 0,0,0 --direction_matrix 0,0,0,0,0,0,0,0,0 ${TEST_DATA}/MRHeadResampled.nhdr ${TEMP}/ResampleVolume2Test.nhdr
ResampleVolume	None (Regression)	--spacing 5,5,5 --interpolation linear ${TEST_DATA}/MRHeadResampled.nhdr ${TEMP}/ResampleVolumeTest.nhdr
RigidRegistration	None (--help)
Subtract	None (Regression)	${TEST_DATA}/CTHeadAxial.nhdr ${TEST_DATA}/CTHeadAxialDoubled.nhdr ${TEMP}/SubtractTest.nhdr
TestGridTransformRegistration	Smoke	--gridSize 5 --outputtransform ${TEMP}/TestGridTransformRegistrationTest00Transform.txt ${TEST_DATA}/CTHeadAxial.nhdr ${TEST_DATA}/CTHeadAxial.nhdr
VotingBinaryHoleFillingImageFilter	None (Regression)	--radius 3,3,3 --majorityThreshold 1 --background 0 --foreground 1 ${TEST_DATA}/CTHeadResampledOtsuSegmented.nhdr ${TEMP}/VotingBinaryHoleFillingImageFilterTest.nhdr
ZeroCrossingBasedEdgeDetectionImageFilter	None (Regression)	${TEST_DATA}/CTHeadResampled.nhdr ${TEMP}/ZeroCrossingBasedEdgeDetectionImageFilterTest.nhdr