NA-MIC-kit-curriculum/Testing-Based Programming/How to run Dynamic Analysis

Dynamic Analysis focuses on detecting defects at run time, particularly: Uninitialized variables and Memory leaks.

Introduction

This tutorial illustrates how to run dynamic analysis in a small project.

Requisites

You should have completed first the tutorials:

• How to add Tests in CMake
• How to run Code Coverage

Installing Valgrind

Valgrind is the applications that will check your tests at run time and will report their defects.

In Ubuntu and Debian GNU/Linux you can install Valgrind by doing

  sudo apt-get install valgrind

Configuring the Project

Basic Configuration

Rerun CMake and verify that the variable MEMORYCHECK_COMMAND is set properly:

 cd ${BINARY_DIR}
 make edit_cache

• Hit the "t" key to go to the advanced mode
• Hit the "/" key to search for MEMORYCHECK_COMMAND
• Verify that it points to the valgrind executable that you installed
• Hit the "c" key to configure
• Hit the "g" key to generate and quit

Additional Flags

You can fine tune the behavior of Valgrind by setting additional arguments in the CMake variable MEMORYCHECK_COMMAND_OPTIONS

 --sim-hints=lax-ioctls 
 --trace-children=yes 
 -q 
 --tool=memcheck 
 --leak-check=yes 
 --show-reachable=yes 
 --num-callers=100
 -v

Running the Tests

We present here two methods for running dynamic tests.

Method 1 is best suited for being used with a CDash Dashboard

Method 2 is best suited for being run locally and for tracking specific problems.

If you are having problems with valgrind suppressions, you may need to do the following (see this discussion):

I had to modify the file

   Slicer3-lib/VTK/CMake/VTKValgrindSuppressions.supp

by removing the following lines:

29,35d28
< {
<    <insert a suppression name here>
<    core:PThread
<    fun:pthread_error
<    fun:pthread_attr_setscope
<    fun:_ZN16vtkMultiThreader19SingleMethodExecuteEv
< }

Method 1

Run the following commands:

  cd ${BINARY_DIR}
  make ExperimentalStart
  make ExperimentalConfigure
  make ExperimentalBuild
  make ExperimentalTest
  make ExperimentalMemCheck

Method 2

Run the command locally:

  cd ${BINARY_DIR}
  valgrind -v --leak-check=yes TestMain 5  

The final part of the output will look like:

 ==27817== HEAP SUMMARY:
 ==27817==     in use at exit: 0 bytes in 0 blocks
 ==27817==   total heap usage: 1 allocs, 1 frees, 352 bytes allocated
 ==27817== 
 ==27817== All heap blocks were freed -- no leaks are possible
 ==27817== 
 ==27817== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 19 from 8)
 --27817-- 
 --27817-- used_suppression:     19 dl-hack3-cond-1
 ==27817== 
 ==27817== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 19 from 8)

Purposely Introducing an Error

To get an idea of how a real error will look like, here we introduce an error on purpose.

Edit main.cxx

Edit the main.cxx file and add the line:

   double * data = new double[200];

in line 13 of the file.

Compile the project

and run again the valgrind command

  cd ${BINARY_DIR}
  valgrind -v --leak-check=yes TestMain 5  

This time the output will look similar to:

 ==27897== HEAP SUMMARY:
 ==27897==     in use at exit: 1,600 bytes in 1 blocks
 ==27897==   total heap usage: 2 allocs, 1 frees, 1,952 bytes allocated
 ==27897== 
 ==27897== Searching for pointers to 1 not-freed blocks
 ==27897== Checked 100,724 bytes
 ==27897== 
 ==27897== 1,600 bytes in 1 blocks are definitely lost in loss record 1 of 1
 ==27897==    at 0x4025024: operator new[](unsigned int) (vg_replace_malloc.c:258)
 ==27897==    by 0x8048E51: main (main.cxx:12)
 ==27897== 
 ==27897== LEAK SUMMARY:
 ==27897==    definitely lost: 1,600 bytes in 1 blocks
 ==27897==    indirectly lost: 0 bytes in 0 blocks
 ==27897==      possibly lost: 0 bytes in 0 blocks
 ==27897==    still reachable: 0 bytes in 0 blocks
 ==27897==         suppressed: 0 bytes in 0 blocks
 ==27897== 
 ==27897== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 19 from 8)
 --27897-- 
 --27897-- used_suppression:     19 dl-hack3-cond-1
 ==27897== 
 ==27897== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 19 from 8)

Running Dynamic Analysis for Slicer tests

First, find the exact command line that is used to launch the test. Say, we want to look at N4ITKBiasFieldCorrection.

  ctest -V -R N4ITKBiasFieldCorrection
  test 163
   Start 163: N4ITKBiasFieldCorrection_Full
  163: Test command: /workspace/fedorov/Slicer/Debug/Slicer3-build/Slicer3 --launch N4ITKBiasFieldCorrectionTest \
     ModuleEntryPoint --inputimage /workspace/fedorov/Slicer/Debug/Slicer3/Testing/Data/Input/he3volume.nii.gz \ 
     --maskimage /workspace/fedorov/Slicer/Debug/Slicer3/Testing/Data/Input/he3mask.nii.gz \
     --outputimage /workspace/fedorov/Slicer/Debug/Slicer3-build/Testing/Temporary/he3corrected.nii.gz \
     --outputbiasfield /workspace/fedorov/Slicer/Debug/Slicer3-build/Testing/Temporary/he3biasfield.nii.gz

Next, from Slicer3-build, you can use the convenience script to run the dynamic analysis:

  ./Scripts/runValgrindForOneModule.sh ./bin/N4ITKBiasFieldCorrectionTest \
     ModuleEntryPoint --inputimage /workspace/fedorov/Slicer/Debug/Slicer3/Testing/Data/Input/he3volume.nii.gz \
     --maskimage /workspace/fedorov/Slicer/Debug/Slicer3/Testing/Data/Input/he3mask.nii.gz \
     --outputimage /workspace/fedorov/Slicer/Debug/Slicer3-build/Testing/Temporary/he3corrected.nii.gz \
     --outputbiasfield /workspace/fedorov/Slicer/Debug/Slicer3-build/Testing/Temporary/he3biasfield.nii.gz

Note, that we specify the full path to the test located in bin, and the command line parameters exactly as they were provided by ctest output.