2012 Summer Project Week:SelfTesting

• 

  Projects List

Key Investigators

• Isomics: Steve Pieper
• Kitware: Jean-Christophe Fillion-Robin, Julien Finet, Benjamin Long
• BWH: Sonia Pujol

Objective

BIST (built-in self-test) is an important concept in delivery of high-reliability engineered systems. While Slicer and the packages in the NA-MIC Kit are fairly well tested by open source software standards, the focus has been almost exclusively on build-time testing. But this leaves out many important variables that can lead to user-level issues. A BIST should be runnable from the installed version of Slicer on the user's machine. Desirable features include:

• Ensure that packaging and installation have succeeded correctly, taking into account the variability of the user's platform (OS version, memory, graphics card, drivers, etc)
• Exactly recreate testing steps so that any user-level issues can be narrowed down (e.g. if a tool works on test data, but not on the user's data).
• Support testing in a continuous loop to look for memory corruption or leaks.

Approach, Plan

The goal at project week is to discuss these issues and formulate a standard plan that can be used by all slicer developers.

Some desirable features of a solution:

• easy for developers to write tests that capture the full workflow
  • distribution of tests as extensions as part of the debugging and support process
  • integration with QtTesting framework to define tests 'by doing'
• same tests should be available in CTest and BIST (ideally this will happen automatically)
• interface features:
  • should allow selection of all tests or a subset of tests
  • should allow running in a loop
  • should present a summary of success/failure
• (possible feature) dialog when slicer first installs that suggests running the tests
• (possible feature) integration of performance testing (e.g. render speed)
• (possible feature) integration with CDash so users can submit test results

Progress

• a prototype implementation as a scripted module is being developed using a factory-style mechanism where modules can register self-tests with slicer that will then be optionally run as part of the self-testing process. Prototype scripted module
• The issue is being tracked as a feature request in mantis
• Example of a test
• Discussion with some developers revealed no preferred alternative built in self test architecture
• This approach will be further elaborated and integrated to the slicer trunk.

Delivery Mechanism

This work will be delivered to the NA-MIC Kit as a (please select the appropriate options by noting YES against them below)

1. ITK Module
2. Slicer Module - YES
   1. Built-in - YES
   2. Extension -- commandline
   3. Extension -- loadable
3. Other (Please specify)

References