AHM2010:Non-interactive-tools

Back to AHM 2010

CMake / CPack

Summary

• Release candidate 2.9
• 1200 downloads per day
• Users: KDE, Blizzard, NVidia, ...

Features added

• Library support: Qt, R, ...
• Superbuild: single run to download and install multiple libraries and executables

Use cases

• 3DSlicer, Teem, CLI Modules, MRML, ...

CDash / CTest

Summary

• CDash rewritten
  • Php, Mysql
• 100 projects freely hosted on cdash.org
• Over 1,000 submissions to ITK's dashboard on ITK's 10th birthday

Features added

• CDash
  • New statistics generated
  • Compression
• CTest
  • Parallel test (ctest -j)
  • Dependencies

Use cases

• Over 100 tests added to Slicer
• Increasing code coverage
• Nightly testing on linux, Mac, windows

XNAT

Features added

• XNAT 1.4 RC3 available
• More robust QC modeling via xnat:qcManualAssessorData.
• New 'columns' variable for REST based listings.
• LDAP support.
• Improved memory footprint and efficiency.
• Refactored Search UI in the listing engine.
• Prebuilt XNAT images using VMWare and Virtualbox
• xnatfs provides WebDav access to XNAT instances

Grid Wizard Engine

Summary

GWE is an open source application scheduler that can run multiple (thousands of) commands simultaneously, on multiple clusters of computers, by a single command. It works without requiring any code modification or scripting. It can be used by itself, or as part of a web-based portal environment. It is written in Java and works well with clusters based on Sun Grid Engine and Condor.

Features added

• Interactive experiments' results browser.
  • Next generation: Record set explorer
• CSV and generated commands dumps and preview.
• Support for LSF and Lava cluster DRMs.
• Ability to override auto-discovered DRMs with configured ones.
• Ability to override default database installation location.
• Function to extract experiments' parameters from text files.
• Operations to delete, pause and resume experiments.

Use cases

Screenshot of the prototype implemented in slicer3. The prototype script reads the automatically generated GWE parameter file to determine each valid combination of variables. A preprocessing routine creates images of each volume dataset. Using the sliders, the user can scroll through the preview images in real time. To explore a particular volume in more detail, the user can click the load button to bring the entire volume in to slicer.

BatchMake

Summary

• Scripted description of a processing pipeline that can be applied to a large collection of images or to a large collection of parameter values.

Use cases

• UNC SBIR, see below
• Registration module testing

Related funding

• UNC, Phase 2 SBIR with Kitware: High throughput murine image analysis
  • Slicer modules executed on a grid when mouse MRI are uploaded to a MIDAS server
  • DTI analysis, atlas-based segmentation, brain morphormetry

Renewal

Centralized Testing

• CDash operates as a master for CTest slaves
  • slaves set their availability and resources
  • CDash server allocates tests to the slaves
  • Ensures testing of important configurations
  • More efficient use of available resources
  • Test combinations can change nightly to cover complex combinations

Integrated project tracking

• SVN, CMake, CPack, CTest, CDash, Wiki, and BugTracker will be extended to report to a central server
• Generates a comprehensive timeline of a project.
• Example use cases
  • Commit a code change and a report is sent to the server indicating
    • files changed, by whom, on what date,
    • changes to the dashboard that resulted from the commit,
    • wiki page changes made by that author around that date/time,
    • bug associated with that commit

iPython for distributed computing

• Python chosen as a favored scripting language for Slicer
  • VTK and Qt are well supported by python
  • ITK v4 will have python as a target language
  • Considered perl (biologists), tcl (comp sci), BatchMake, JavaScript, ...
• iPython contains extensions to python for threaded, grid, and distributed computing
  • Reduced setup and learning curve
  • Broad support