Difference between revisions of "AHM2010:Non-interactive-tools"
From NAMIC Wiki
| (14 intermediate revisions by the same user not shown) | |||
| Line 2: | Line 2: | ||
= CMake / CPack = | = CMake / CPack = | ||
| + | |||
| + | [[Image:NA-MICSoftwareProcess.png|thumb|250px|The NA-MIC software process has CMake, CPack, and CDash as the foundation for cross-platform building, testing, and delivery of 3D Slicer]] | ||
| + | |||
== Summary == | == Summary == | ||
| Line 14: | Line 17: | ||
== Use cases == | == Use cases == | ||
* 3DSlicer, Teem, CLI Modules, MRML, ... | * 3DSlicer, Teem, CLI Modules, MRML, ... | ||
| − | + | * Qt-version of 3D Slicer | |
| − | |||
| − | * | ||
= CDash / CTest = | = CDash / CTest = | ||
| + | [[Image:2010-AHM-SlicerDashboard.jpg|thumb|250px|This Slicer Dashboard illustrates the many machines used to test Slicer every day.]] | ||
== Summary == | == Summary == | ||
| − | * | + | * Regression testing of Slicer and it optional modules on multiple platforms |
| − | |||
* 100 projects freely hosted on cdash.org | * 100 projects freely hosted on cdash.org | ||
| − | * Over | + | * Over 1,000 submissions to ITK's dashboard on ITK's 10th birthday |
== Features added == | == Features added == | ||
* CDash | * CDash | ||
| − | ** New statistics generated | + | ** New statistics generated (computation time, history of pass/fails) |
| − | ** Compression | + | ** Compression of historic data |
* CTest | * CTest | ||
** Parallel test (ctest -j) | ** Parallel test (ctest -j) | ||
| − | ** | + | ** Inter-test dependencies |
== Use cases == | == Use cases == | ||
| Line 39: | Line 40: | ||
* Nightly testing on linux, Mac, windows | * Nightly testing on linux, Mac, windows | ||
| − | == | + | = XNAT = |
| − | * | + | |
| + | [[Image:FetchMIDoc_SlicerRegistration.png|thumb|250px|Tutorial 1.4 data on xnd.slicer.org and Pop-up Data I/O Manager showing data transfer status]] | ||
| + | |||
| + | == Summary == | ||
| + | The Extensible Neuroimaging Archive Toolkit (XNAT) is an open source software platform designed to facilitate management and exploration of neuroimaging and related data. XNAT includes a secure database backend and a rich web-based user interface. | ||
| + | |||
| + | == Features added == | ||
| + | * XNAT 1.4 RC3 available | ||
| + | * More robust QC modeling via xnat:qcManualAssessorData. | ||
| + | * New 'columns' variable for REST based listings. | ||
| + | * LDAP support. | ||
| + | * Improved memory footprint and efficiency. | ||
| + | * Refactored Search UI in the listing engine. | ||
| + | * Prebuilt XNAT images using VMWare and Virtualbox | ||
| + | * xnatfs provides WebDav access to XNAT instances | ||
| + | |||
| + | == Use cases == | ||
| + | * FetchMI | ||
= Grid Wizard Engine = | = Grid Wizard Engine = | ||
| + | |||
| + | [[Image:GWE-Slicer3 Parameter Explorer Prototype 2009-02-23.png|thumb|right|250px|Screenshot of the prototype implemented in slicer3. The prototype script reads the automatically generated GWE parameter file to determine each valid combination of variables. A preprocessing routine creates images of each volume dataset. Using the sliders, the user can scroll through the preview images in real time. To explore a particular volume in more detail, the user can click the load button to bring the entire volume in to slicer.]] | ||
== Summary == | == Summary == | ||
| + | GWE is an open source application scheduler that can run multiple (thousands of) commands simultaneously, on multiple clusters of computers, by a single command. It works without requiring any code modification or scripting. It can be used by itself, or as part of a web-based portal environment. It is written in Java and works well with clusters based on Sun Grid Engine and Condor. | ||
== Features added == | == Features added == | ||
| + | * Interactive experiments' results browser. | ||
| + | ** Next generation: Record set explorer | ||
| + | * CSV and generated commands dumps and preview. | ||
| + | * Support for LSF and Lava cluster DRMs. | ||
| + | * Ability to override auto-discovered DRMs with configured ones. | ||
| + | * Ability to override default database installation location. | ||
| + | * Function to extract experiments' parameters from text files. | ||
| + | * Operations to delete, pause and resume experiments. | ||
== Use cases == | == Use cases == | ||
| − | + | * GWE Mini Retreat | |
| − | + | * Project weeks | |
= BatchMake = | = BatchMake = | ||
== Summary == | == Summary == | ||
| − | + | Scripted description of a processing pipeline that can be applied to a large collection of images or to a large collection of parameter values. | |
== Features added == | == Features added == | ||
| + | * Improvement in the integration of grid computing using condor. | ||
| + | * Semi-Automatic conversion of Slicer modules for batch processing | ||
| + | * Parameter exploration is now possible | ||
| + | * Execution of algorithms are faster | ||
== Use cases == | == Use cases == | ||
| − | |||
* Registration module testing | * Registration module testing | ||
| − | |||
| − | |||
* UNC, Phase 2 SBIR with Kitware: High throughput murine image analysis | * UNC, Phase 2 SBIR with Kitware: High throughput murine image analysis | ||
** Slicer modules executed on a grid when mouse MRI are uploaded to a MIDAS server | ** Slicer modules executed on a grid when mouse MRI are uploaded to a MIDAS server | ||
** DTI analysis, atlas-based segmentation, brain morphormetry | ** DTI analysis, atlas-based segmentation, brain morphormetry | ||
| + | |||
| + | = Renewal = | ||
| + | [[Image:NAMICKitOverview.png|thumb|250px|The components of the NA-MIC for the 3D Slicer platform]] | ||
| + | |||
| + | == Centralized Test Management via CDash == | ||
| + | * CDash operates as a master for CTest slaves | ||
| + | ** slaves set their availability and resources | ||
| + | ** CDash server allocates tests to the slaves | ||
| + | ** Ensures testing of important configurations | ||
| + | ** More efficient use of available resources | ||
| + | ** Test combinations can change nightly to cover complex combinations | ||
| + | |||
| + | == Integrated project tracking == | ||
| + | [[Image:ProjectTracking.jpg|thumb|250px|A depiction of one possible visualization for the project tracker]] | ||
| + | |||
| + | * SVN, CMake, CPack, CTest, CDash, Wiki, and BugTracker will be extended to report to a central server | ||
| + | * Generates a comprehensive timeline of a project. | ||
| + | * Example use cases | ||
| + | ** Commit a code change and a report is sent to the server indicating | ||
| + | *** files changed, by whom, on what date, | ||
| + | *** changes to the dashboard that resulted from the commit, | ||
| + | *** wiki page changes made by that author around that date/time, | ||
| + | *** bug associated with that commit | ||
| + | |||
| + | == iPython for distributed computing == | ||
| + | * Python chosen as a favored scripting language for Slicer | ||
| + | ** VTK and Qt are well supported by python | ||
| + | ** ITK v4 will have python as a target language | ||
| + | ** Considered perl (biologists), tcl (comp sci), BatchMake, JavaScript, ... | ||
| + | * iPython contains extensions to python for threaded, grid, and distributed computing | ||
| + | ** Reduced setup and learning curve | ||
| + | ** Broad support | ||
| + | |||
| + | == XNAT == | ||
| + | * Desktop | ||
| + | |||
| + | == GWE == | ||
| + | * Workflows | ||
Latest revision as of 22:47, 6 January 2010
Home < AHM2010:Non-interactive-toolsBack to AHM 2010
CMake / CPack
Summary
- Release candidate 2.9
- 1200 downloads per day
- Users: KDE, Blizzard, NVidia, ...
Features added
- Library support: Qt, R, ...
- Superbuild: single run to download and install multiple libraries and executables
Use cases
- 3DSlicer, Teem, CLI Modules, MRML, ...
- Qt-version of 3D Slicer
CDash / CTest
Summary
- Regression testing of Slicer and it optional modules on multiple platforms
- 100 projects freely hosted on cdash.org
- Over 1,000 submissions to ITK's dashboard on ITK's 10th birthday
Features added
- CDash
- New statistics generated (computation time, history of pass/fails)
- Compression of historic data
- CTest
- Parallel test (ctest -j)
- Inter-test dependencies
Use cases
- Over 100 tests added to Slicer
- Increasing code coverage
- Nightly testing on linux, Mac, windows
XNAT
Summary
The Extensible Neuroimaging Archive Toolkit (XNAT) is an open source software platform designed to facilitate management and exploration of neuroimaging and related data. XNAT includes a secure database backend and a rich web-based user interface.
Features added
- XNAT 1.4 RC3 available
- More robust QC modeling via xnat:qcManualAssessorData.
- New 'columns' variable for REST based listings.
- LDAP support.
- Improved memory footprint and efficiency.
- Refactored Search UI in the listing engine.
- Prebuilt XNAT images using VMWare and Virtualbox
- xnatfs provides WebDav access to XNAT instances
Use cases
- FetchMI
Grid Wizard Engine
Screenshot of the prototype implemented in slicer3. The prototype script reads the automatically generated GWE parameter file to determine each valid combination of variables. A preprocessing routine creates images of each volume dataset. Using the sliders, the user can scroll through the preview images in real time. To explore a particular volume in more detail, the user can click the load button to bring the entire volume in to slicer.
Summary
GWE is an open source application scheduler that can run multiple (thousands of) commands simultaneously, on multiple clusters of computers, by a single command. It works without requiring any code modification or scripting. It can be used by itself, or as part of a web-based portal environment. It is written in Java and works well with clusters based on Sun Grid Engine and Condor.
Features added
- Interactive experiments' results browser.
- Next generation: Record set explorer
- CSV and generated commands dumps and preview.
- Support for LSF and Lava cluster DRMs.
- Ability to override auto-discovered DRMs with configured ones.
- Ability to override default database installation location.
- Function to extract experiments' parameters from text files.
- Operations to delete, pause and resume experiments.
Use cases
- GWE Mini Retreat
- Project weeks
BatchMake
Summary
Scripted description of a processing pipeline that can be applied to a large collection of images or to a large collection of parameter values.
Features added
- Improvement in the integration of grid computing using condor.
- Semi-Automatic conversion of Slicer modules for batch processing
- Parameter exploration is now possible
- Execution of algorithms are faster
Use cases
- Registration module testing
- UNC, Phase 2 SBIR with Kitware: High throughput murine image analysis
- Slicer modules executed on a grid when mouse MRI are uploaded to a MIDAS server
- DTI analysis, atlas-based segmentation, brain morphormetry
Renewal
Centralized Test Management via CDash
- CDash operates as a master for CTest slaves
- slaves set their availability and resources
- CDash server allocates tests to the slaves
- Ensures testing of important configurations
- More efficient use of available resources
- Test combinations can change nightly to cover complex combinations
Integrated project tracking
- SVN, CMake, CPack, CTest, CDash, Wiki, and BugTracker will be extended to report to a central server
- Generates a comprehensive timeline of a project.
- Example use cases
- Commit a code change and a report is sent to the server indicating
- files changed, by whom, on what date,
- changes to the dashboard that resulted from the commit,
- wiki page changes made by that author around that date/time,
- bug associated with that commit
- Commit a code change and a report is sent to the server indicating
iPython for distributed computing
- Python chosen as a favored scripting language for Slicer
- VTK and Qt are well supported by python
- ITK v4 will have python as a target language
- Considered perl (biologists), tcl (comp sci), BatchMake, JavaScript, ...
- iPython contains extensions to python for threaded, grid, and distributed computing
- Reduced setup and learning curve
- Broad support
XNAT
- Desktop
GWE
- Workflows
