Difference between revisions of "AHM2010:Non-interactive-tools"

From NAMIC Wiki
Jump to: navigation, search
 
(8 intermediate revisions by the same user not shown)
Line 2: Line 2:
  
 
= CMake / CPack =
 
= CMake / CPack =
 +
 +
[[Image:NA-MICSoftwareProcess.png|thumb|250px|The NA-MIC software process has CMake, CPack, and CDash as the foundation for cross-platform building, testing, and delivery of 3D Slicer]]
 +
  
 
== Summary ==
 
== Summary ==
Line 14: Line 17:
 
== Use cases ==
 
== Use cases ==
 
* 3DSlicer, Teem, CLI Modules, MRML, ...
 
* 3DSlicer, Teem, CLI Modules, MRML, ...
 +
* Qt-version of 3D Slicer
  
 
= CDash / CTest =
 
= CDash / CTest =
 +
[[Image:2010-AHM-SlicerDashboard.jpg|thumb|250px|This Slicer Dashboard illustrates the many machines used to test Slicer every day.]]
  
 
== Summary ==
 
== Summary ==
−
* CDash rewritten
+
* Regression testing of Slicer and it optional modules on multiple platforms
−
** Php, Mysql
 
 
* 100 projects freely hosted on cdash.org
 
* 100 projects freely hosted on cdash.org
 
* Over 1,000 submissions to ITK's dashboard on ITK's 10th birthday
 
* Over 1,000 submissions to ITK's dashboard on ITK's 10th birthday
Line 25: Line 29:
 
== Features added ==
 
== Features added ==
 
* CDash
 
* CDash
−
** New statistics generated
+
** New statistics generated (computation time, history of pass/fails)
−
** Compression
+
** Compression of historic data
 
* CTest
 
* CTest
 
** Parallel test (ctest -j)
 
** Parallel test (ctest -j)
−
** Dependencies
+
** Inter-test dependencies
  
 
== Use cases ==
 
== Use cases ==
Line 37: Line 41:
  
 
= XNAT =
 
= XNAT =
 +
 +
[[Image:FetchMIDoc_SlicerRegistration.png|thumb|250px|Tutorial 1.4 data on xnd.slicer.org and Pop-up Data I/O Manager showing data transfer status]]
  
 
== Summary ==
 
== Summary ==
 +
The Extensible Neuroimaging Archive Toolkit (XNAT) is an open source software platform designed to facilitate management and exploration of neuroimaging and related data. XNAT includes a secure database backend and a rich web-based user interface.
  
 
== Features added ==
 
== Features added ==
Line 51: Line 58:
  
 
== Use cases ==
 
== Use cases ==
 +
* FetchMI
  
 
= Grid Wizard Engine =
 
= Grid Wizard Engine =
 +
 +
[[Image:GWE-Slicer3 Parameter Explorer Prototype 2009-02-23.png|thumb|right|250px|Screenshot of the prototype implemented in slicer3. The prototype script reads the automatically generated GWE parameter file to determine each valid combination of variables.  A preprocessing routine creates images of each volume dataset.  Using the sliders, the user can scroll through the preview images in real time.  To explore a particular volume in more detail, the user can click the load button to bring the entire volume in to slicer.]]
  
 
== Summary ==
 
== Summary ==
Line 68: Line 78:
  
 
== Use cases ==
 
== Use cases ==
−
[[Image:GWE-Slicer3 Parameter Explorer Prototype 2009-02-23.png|thumb|right|300px|Screenshot of the prototype implemented in slicer3. The prototype script reads the automatically generated GWE parameter file to determine each valid combination of variables.  A preprocessing routine creates images of each volume dataset.  Using the sliders, the user can scroll through the preview images in real time.  To explore a particular volume in more detail, the user can click the load button to bring the entire volume in to slicer.]]
+
* GWE Mini Retreat
−
 
+
* Project weeks
−
== Related funding ==
 
  
 
= BatchMake =
 
= BatchMake =
  
 
== Summary ==
 
== Summary ==
−
* Scripted description of a processing pipeline that can be applied to a large collection of images or to a large collection of parameter values.
+
Scripted description of a processing pipeline that can be applied to a large collection of images or to a large collection of parameter values.
  
 
== Features added ==
 
== Features added ==
 +
* Improvement in the integration of grid computing using condor.
 +
* Semi-Automatic conversion of Slicer modules for batch processing
 +
* Parameter exploration is now possible
 +
* Execution of algorithms are faster
  
 
== Use cases ==
 
== Use cases ==
−
* UNC SBIR, see below
 
 
* Registration module testing
 
* Registration module testing
−
 
−
== Related funding ==
 
 
* UNC, Phase 2 SBIR with Kitware: High throughput murine image analysis
 
* UNC, Phase 2 SBIR with Kitware: High throughput murine image analysis
 
** Slicer modules executed on a grid when mouse MRI are uploaded to a MIDAS server
 
** Slicer modules executed on a grid when mouse MRI are uploaded to a MIDAS server
Line 89: Line 99:
  
 
= Renewal =
 
= Renewal =
 +
[[Image:NAMICKitOverview.png|thumb|250px|The components of the NA-MIC for the 3D Slicer platform]]
  
−
== Centralized Testing ==
+
== Centralized Test Management via CDash ==
 
* CDash operates as a master for CTest slaves
 
* CDash operates as a master for CTest slaves
 
** slaves set their availability and resources
 
** slaves set their availability and resources
Line 99: Line 110:
  
 
== Integrated project tracking ==
 
== Integrated project tracking ==
 +
[[Image:ProjectTracking.jpg|thumb|250px|A depiction of one possible visualization for the project tracker]]
 +
 
* SVN, CMake, CPack, CTest, CDash, Wiki, and BugTracker will be extended to report to a central server
 
* SVN, CMake, CPack, CTest, CDash, Wiki, and BugTracker will be extended to report to a central server
 
* Generates a comprehensive timeline of a project.
 
* Generates a comprehensive timeline of a project.
Line 116: Line 129:
 
** Reduced setup and learning curve
 
** Reduced setup and learning curve
 
** Broad support
 
** Broad support
 +
 +
== XNAT ==
 +
* Desktop
 +
 +
== GWE ==
 +
* Workflows

Latest revision as of 22:47, 6 January 2010

Home < AHM2010:Non-interactive-tools

Back to AHM 2010

CMake / CPack

The NA-MIC software process has CMake, CPack, and CDash as the foundation for cross-platform building, testing, and delivery of 3D Slicer


Summary

  • Release candidate 2.9
  • 1200 downloads per day
  • Users: KDE, Blizzard, NVidia, ...

Features added

  • Library support: Qt, R, ...
  • Superbuild: single run to download and install multiple libraries and executables

Use cases

  • 3DSlicer, Teem, CLI Modules, MRML, ...
  • Qt-version of 3D Slicer

CDash / CTest

This Slicer Dashboard illustrates the many machines used to test Slicer every day.

Summary

  • Regression testing of Slicer and it optional modules on multiple platforms
  • 100 projects freely hosted on cdash.org
  • Over 1,000 submissions to ITK's dashboard on ITK's 10th birthday

Features added

  • CDash
    • New statistics generated (computation time, history of pass/fails)
    • Compression of historic data
  • CTest
    • Parallel test (ctest -j)
    • Inter-test dependencies

Use cases

  • Over 100 tests added to Slicer
  • Increasing code coverage
  • Nightly testing on linux, Mac, windows

XNAT

Tutorial 1.4 data on xnd.slicer.org and Pop-up Data I/O Manager showing data transfer status

Summary

The Extensible Neuroimaging Archive Toolkit (XNAT) is an open source software platform designed to facilitate management and exploration of neuroimaging and related data. XNAT includes a secure database backend and a rich web-based user interface.

Features added

  • XNAT 1.4 RC3 available
  • More robust QC modeling via xnat:qcManualAssessorData.
  • New 'columns' variable for REST based listings.
  • LDAP support.
  • Improved memory footprint and efficiency.
  • Refactored Search UI in the listing engine.
  • Prebuilt XNAT images using VMWare and Virtualbox
  • xnatfs provides WebDav access to XNAT instances

Use cases

  • FetchMI

Grid Wizard Engine

Screenshot of the prototype implemented in slicer3. The prototype script reads the automatically generated GWE parameter file to determine each valid combination of variables. A preprocessing routine creates images of each volume dataset. Using the sliders, the user can scroll through the preview images in real time. To explore a particular volume in more detail, the user can click the load button to bring the entire volume in to slicer.

Summary

GWE is an open source application scheduler that can run multiple (thousands of) commands simultaneously, on multiple clusters of computers, by a single command. It works without requiring any code modification or scripting. It can be used by itself, or as part of a web-based portal environment. It is written in Java and works well with clusters based on Sun Grid Engine and Condor.

Features added

  • Interactive experiments' results browser.
    • Next generation: Record set explorer
  • CSV and generated commands dumps and preview.
  • Support for LSF and Lava cluster DRMs.
  • Ability to override auto-discovered DRMs with configured ones.
  • Ability to override default database installation location.
  • Function to extract experiments' parameters from text files.
  • Operations to delete, pause and resume experiments.

Use cases

  • GWE Mini Retreat
  • Project weeks

BatchMake

Summary

Scripted description of a processing pipeline that can be applied to a large collection of images or to a large collection of parameter values.

Features added

  • Improvement in the integration of grid computing using condor.
  • Semi-Automatic conversion of Slicer modules for batch processing
  • Parameter exploration is now possible
  • Execution of algorithms are faster

Use cases

  • Registration module testing
  • UNC, Phase 2 SBIR with Kitware: High throughput murine image analysis
    • Slicer modules executed on a grid when mouse MRI are uploaded to a MIDAS server
    • DTI analysis, atlas-based segmentation, brain morphormetry

Renewal

The components of the NA-MIC for the 3D Slicer platform

Centralized Test Management via CDash

  • CDash operates as a master for CTest slaves
    • slaves set their availability and resources
    • CDash server allocates tests to the slaves
    • Ensures testing of important configurations
    • More efficient use of available resources
    • Test combinations can change nightly to cover complex combinations

Integrated project tracking

A depiction of one possible visualization for the project tracker
  • SVN, CMake, CPack, CTest, CDash, Wiki, and BugTracker will be extended to report to a central server
  • Generates a comprehensive timeline of a project.
  • Example use cases
    • Commit a code change and a report is sent to the server indicating
      • files changed, by whom, on what date,
      • changes to the dashboard that resulted from the commit,
      • wiki page changes made by that author around that date/time,
      • bug associated with that commit

iPython for distributed computing

  • Python chosen as a favored scripting language for Slicer
    • VTK and Qt are well supported by python
    • ITK v4 will have python as a target language
    • Considered perl (biologists), tcl (comp sci), BatchMake, JavaScript, ...
  • iPython contains extensions to python for threaded, grid, and distributed computing
    • Reduced setup and learning curve
    • Broad support

XNAT

  • Desktop

GWE

  • Workflows
Retrieved from "https://www.na-mic.org/w/index.php?title=AHM2010:Non-interactive-tools&oldid=47291"