2008 Summer Project Week:Batch Processing
- Julien Jomier
- Steve Pieper
- Marco Ruiz
- Jim Miller
- Chris Churas
- Hans Johnson
- Killian Pohl
- Jeff Grethe
- Mark Anderson
Implement tests of batch processing techniques using slicer3 command line modules (EMSegmenter, Registration).
BatchMake and GWE are both ready to apply to sample datasets using distributed grid/cluster computing. This week we will flesh out the use cases and run actual jobs.
Also, review the concepts and issues documented on the Large Scale Experiment Brainstorming page.
- Identified separation between GWE and BatchMake
- GWE: Running Slicer modules on parameter space from within slicer, console utility or scripts.
- BatchMake: Scripting using Slicer3 modules (outside of Slicer3 GUI) and reporting
- 175 cases processed with BSplineDeformableRegistration on the SPL cluster using GWE.
- 2275 cases processed with BSplineDeformableRegistration on the BIRN cluster using GWE.
- 16 cases processed with FreeSurfer on the BIRN cluster using GWE.
- Working out some Condor issues on the SPL cluster.
- Prepared commands to process 50 cases with Killian's pipeline on the SPL cluster using GWE.
- Reviewed Sylvain's cases requirements to verify they could be processed in a cluster using GWE.
- Future Work:
- Implement support to run through GWE
- Provide reporting tool as slicer execution modules
- Detailed on the "future work" section of Project_Week_2008_Special_topic_breakout:_GWE
- Auto-deploy Slicer3 in clusters where module invocations will be parallelize.
- Real time monitor interface to track status of requests submitted to clusters.
- Integrate GWE with the "Slicer3 Chain Module Engine": Slicer3:JavaBasedChainCLMsEngine