LONIPipelineSummary
The LONI Pipeline Processing Environment has been used quite successfully since its release in the neuroimaging field. It has been downloaded over 700 times and there are close to 100 registered users within LONI and its collaborators. The Pipeline has sped up research in this area, allowing researchers to focus upon the research and not the mundane and repetitive administrative tasks of data management and processing. It also has provided an easy mechanism for researchers to disseminate their research processes.
All software grow and evolve along with user feedback, and the Pipeline is no different. Since its initial release, much of the feedback has been constructive, suggesting useful functionalities that could be integrated with the application. We have added features and removed bugs accordingly. The architecture of the environment has been reorganized to aid in accomplishing these, and future, tasks.
A modular architecture, with task specific components that provide quick and easy customization, is currently under development. By cleanly separating these components, the development process has reduced in complexity and speeded up dramatically. Taking advantage of this has allowed us to make other dramatic improvements (which are included in the upcoming v3 release). These new improvements include improved security, grid computing integration, automated process management, automated resource management, advanced debugging, and fault tolerance.
Secure communications is now an integral part of the Pipeline. All messages between a server and a client are encrypted using an AES key, which is established via the Diffie-Hellman algorithm. This will aid researchers in staying compliant with HIPPA and local IRB rules.
Grid computing integration via Sun's Grid Engine has been completed. This allows labs with large but disparate computational resources to make full use of those resources.
Automated process management schedules jobs according to type, user priority, resource availability, etc. Improvements in this version include an advanced, configurable process scheduler, resource locks for ensuring execution correctness, and message handling.
Automated resource management ensures that remotely available files are cached and available when needed. Updates to the original file are also returned automatically, alleviating users of tedious file management.
Advanced debugging features enables a researcher to trace the cause of a failure in the pipeline. This includes informative logs, from both the executables themselves as from errors occurring in the pipeline.
Network fault tolerance is the ability to seamlessly handle a network failure (whether user initiated or not). The stateful processing on a server continues even when a client disconnects. Upon reconnection, a client retrieves all the stateful information left upon the server.
Integration of additional functionalities, which include vizualization, validation, and provenance, are planned. For developers who wish to make their own contributions to the development of the Pipeline, an application programming interface (API) will be made available.
Current development schedule:
- Pre-alpha release (March)
- Alpha release (April)
- Beta release (May)
- Release candidate (June)
The Pipeline in the context of NAMIC:
Example of Slicer in LONI Pipeline 1.0
