2008 Progress Report NIH QnA

Please address promptly (by July 30th, 5pm)

We received this letter from our NIH program officers. Below are excerpted specific questions from this letter, with names of NA-MIC team members responsible for the answers.

MGH

• What progress has been made with the MGH subcontract in the past year?
  • There's no progress report summary following the Statement of Intent for MGH (page 42). We recognize that MGH's budget was significantly reduced this past year, as a good portion was re-budgeted to Washington University beginning in December 2007. Nevertheless, we expect some progress to have been made.
  • The scientific progress report lists MGH as participating in some efforts (Shape Based Segmentation and Registration; Spherical Wavelets; and Shape Analysis with Overcomplete Wavelets), but there are no MGH personnel listed as key investigators for that section.
• With regard to future work (Dr. Fischl's consultation), the statement of work refers to "integration of FreeSurfer with ITK and 3D Slicer." What does that mean? Wasn't that effort abandoned.
• The progress report is somewhat confusing as to whether FreeSurfer is being used to study cortical correspondence. On page 250 (Section 3.2) a collaboration between MGH and MIT is mentioned with respect to cortical correspondence, while the UNC progress report states that MOL is being used to explore cortical correspondence. Please clarify.

MGH Response (Bruce Fischl)

The overcomplete wavelet project has resulted in a conference publication, and we are actively working on it's completion. We are currently working on incorporating geometric invariants to replace the coordinate functions, and also on augmenting our existing neonate datasets with one or two more manually labeled cases so that we can assess how well our model fits the data.

Going forward we intend to improve the FreeSurfer/Slicer/ITK interoperability. Towards this end we have worked with Core 2 to incorporate support for our internal formats in Slicer, and intend to include support for ITK formats such as NRRD directly into FreeSurfer. This will significantly increase the ease of development of algorithms that take advantage of the two platforms.

We also have an ongoing collaboration with Drs. Polina Golland, Mert Sabuncu and Thomas Yeo at MIT regarding improving cortical correspondence. Specifically, the existing FreeSurfer algorithms are being used to generate a baseline state-of-the-art accuracy measure that will be used to assess the performance of novel registration algorithms. Here we are working on both fast diffeormophic registration, and have a recent conference paper describing the "spherical demons" algorithm that improves execution time significantly without sacrificing performance. In addition, we as part of Mr. Yeo's Ph.D. research we are developing techniques for the optimal alignment of either archectonically or functionally defined cortical regions, defined using whole-brain histology and functional MRI respectively. The existing FreeSurfer framework is also being used by UNC to comparatively evaluate the UNC cortical thickness framework, which provides a group-wise cortical correspondence based framework fully within Slicer.

UCLA

• What progress has been made with the UCLA subcontract in the past year?
  • There are no publications listed in progress report. A search of NA-MIC publications database shows only one paper with Art Toga or Nathan Hageman's name on it, and that's the iTools paper (from the Software and Data Integration Working Group).
  • Timeline information is out of date. It has not been updated to reflect the changes in the statement of work that was agreed upon in August 2007.

UCLA Response (Arthur Toga)

• Q1: A paper citing the NAMIC grant that describes our fluid mechanic DTI tractography method has been accepted to IEEE TMI. It will become available in the NAMIC publications database shortly.
• Q2: We have updated the timeline based on the most recent NIH statement of work.

Queens

• Regarding the Brachytherapy Needle Positioning Robot Integration DBP:
  • Which grant is funding the patient data collection? If it is supported by NA-MIC, please let me know because NIBIB will have to approve your Data Safety Monitoring Plan.
  • There seems to be some scientific overlap between this project and other NIH-funded grants, R01 EB002963 (PI: Whitcomb [previously Fichtinger]) and R01 CA111288 (PI:Tempany). The statement of work for the NA-MIC subcontract states, "The deliverables of the contract is professional-grade clinical software engineering of the above modules based on the NA-MIC toolkit (to the extent reasonable and possible) and to develop end applications based on Slicer, for clinical trials in image-gu ided prostate biopsy." However, this goal also falls within the System Integration aim (Aim 3) of the NCI grant and the System Integration aim (Aim 3) of the NIBIB grant. Please provide us with additional clarification to distinguish these projects in terms of their aims.
  • Also, please confirm that Dr. Gobbi and Mr. Vikal are being supported at no more than 3 and 6 months, respectively by other grants (as they are listed for 9 and 6 months on the NAMIC subcontract).

Queens Response (Gabor Fichtinger)

Q1: Which grant is funding the patient data collection? -- The Queen's NAMIC team does not conduct clinical trials. For development and testing purposes, we use previously acquired anonymous image data provided by our clinical collaborators. The data we use does not contain patient identification information. Under the NAMIC grant, we shall not perform clinical trials. We will hand over the developed system to clinical collaborators who are supported by NAMIC.

Q2: There seems to be some scientific overlap -- There are indeed close synergies but no overlap. First: R01 EB002963 (PI: Whitcomb) concentrated on developing a transrectal prostate biopsy robot and its rapid clinical testing. For this purpose, a straightforward and minimalistic system was built to operate the robot. This system is not based on Slicer. The objective of our RoadMap project to empower this robot with superior capaboilities of Slicer and the NAMIC toolkit. The grants therefore are ideally synergistic, without any overlap. Also importantly, R01 EB002963 (PI: Whitcomb) will end on July 31, 2008, before the anniversary date of the NAMIC grant. Second, R01 CA111288 (PI:Tempany) is a Biomedical Research Partnership (BRP) that includes Acoustic Medsystems, Inc., a company whose responsibility is system's integration within the company's quality control environment. Dr. Fichtinger's NAMIC team at Queen's is not involved in system development for R01 CA111288 (PI:Tempany).

Q3: We confirm that Dr. Gobbi and Mr. Vikal are being supported at no more than 3 and 6 months, respectively by other grants (as they are listed for 9 and 6 months on the NAMIC subcontract).

Kitware

• Kitware is working on a text, "Practical Software Process", to document the NA-MIC software process. How will that be distributed? Will NA-MIC funds cover all the costs so that the text can be distributed free of charge?

Kitware Response (Will Schroeder)

Whenever possible, all works created under NA-MIC funding are distributed under an appropriate open source license (of course some publications and data have special requirements due to long-standing journal policy and privacy issues). Thus in the case of the text referred to here, "Practical Software Process", the material will indeed be available in an open source form. The idea is to capture the software process in a single document that will help other researchers and developers replicate the software processes that we use at NA-MIC.

Note that this does not preclude Kitware (and other organizations for that matter) from publishing and selling the book. This is fair game under open source licensing. We have precedent for such an approach, for example the ITK user guide http://www.itk.org/ItkSoftwareGuide.pdf is available on-line, yet the Company still sells the book.

Dissemination Core

• What does it mean when you say that you NA-MIC hosted the Workshop on Open Source and Open Data at MICCAI 2007? Did NA-MIC providing financial support? Set the agenda? Invite participants? Please clarify.

Dissemination Core Response (Tina Kapur)

NA-MIC personnel contributed their time to conduct typical tasks involved in chairing a workshop such as soliciting manuscripts and open reviews, setting the agenda, and inviting speakers for keynote presentations. No financial support was needed or provided by NA-MIC other than travel of NA-MIC presenters to the conference.

Questions for non-urgent consideration:

MIND

• Systemic Lupus Erythematous project
  • Has this project driven any new algorithm development?
  • Have the new tools been tested on other lesions? What other sorts of lesions are likely to benefit from them?
  • Has manual segmentation (which is serving as the gold standard for this project) been shown to have low inter-observer variability?

Structural Image Analysis

• Is NA-MIC supporting the UNC-led (Martin Styner) 3D Segmentation in the Clinic Workshop at MICCAI 2008? If so, in what ways?
• Does NA-MIC have any process planned for eliminating an algorithm from its toolkit if a competing algorithm outperforms it?
• Are there any plans to integrate results from the segmentation workshop into documentation for Slicer in such a way that they are readily accessible to users choosing between Slicer modules?

Structural Image Analysis Response (Martin Styner with input from other Core 1 and 2 people)

Response to #1: NA-MIC is and has been supporting the MICCAI workshops on the comparative evaluation of image analysis tasks with real, clinical data ("3D Segmentation in the Clinic" workshops at MICCAI 2007 and MICCAI 2008). Due to NA-MIC's involvement, the two workshops thus far highlighted evaluations focusing on image analysis tasks of core importance to the NA-MIC DBPs (caudate segmentation in 2007 and lesion segmentation in 2008). NA-MIC's support is provided via time contribution of one of the organizers, Martin Styner, as well as by providing training and testing dataset. More than half of the training and testing datasets in the 2007 caudate segmentation evaluation were provided through NA-MIC by the PNL DBP I site.

Response to #2: NA-MIC is the provider of a research software platform. We do not believe that eliminating algorithms from the NA-MIC kit is in the best interest of the research community because even if a particular algorithm is not actively used as part of an end-to-end clinical solution today, or does not perform as well as another one in the context of a particular task, we want to remain open to the possibility that it could be a key enabler of solutions in the future. However, for algorithms that are part of end-to-end solutions to problems that we are actively working on, we are making a concerted effort to provide publicly accessible tutorial materials to explain and encourage their adoption.

Response to #3: While some participants especially of this year's workshop will make their work available as Slicer modules, this is no requirement on participation. We certainly encourage open source submissions, as well as availability of Slicer modules. The linkage of the online available workshop's results to the documentation of those submissions that are available as Slicer modules will provide information about the module's comparative performance. This information is readily available via the workshop websites that host the continued comparative evaluations (e.g. cause07.org for the caudate segmentation evaluation). Thus there exists an efficient way for the user to select the best performing Slicer module for the task at hand. As the evaluation is continued after the workshop, newly developed Slicer modules can be added at any time to the comparative evaluation websites.

Miscellaneous

Kitware (Will Schroeder)

Some concerns were raised regarding the blurring of the distinction between NA-MIC, ITK and VTK. We recognize the contributions that NA-MIC-funded programmers have made to both ITK and VTK and we recognize that the relationship between NA-MIC and the other toolkits is beneficial to a broad development and user community. However, in some cases there seems to be insufficient acknowledgement in the report of the tools that predated NAMIC and helped lay the groundwork for it; for example CMake and DART.

Response: Yes, in extended, long-lived open source communities it can be difficult to keep track of the original contributors and funding agencies. We believe that this is important, since the contributing parties are often asked to justify their existence; plus in the interest of fairness, not to mention the need to track the source of contributions when problems occur or changes are contemplated, a trail of ownership is desirable. We generally ask all developers to include a "Thanks:" field (or equivalent) in the file documentation to note contributions due to projects. Also, svn and cvs repositories contain extensive historical information about the developer(s) who add, modify and delete material to a project.

Another way of recognizing the contributions of funding agencies and organizations is to keep the historical narrative of a project up to date. For example, the "About" page in the ITK project http://www.itk.org/HTML/About.htm refers to the many projects that it spawned including CMake and DART. However, upon reading this document as it currently exists, it is clear that further historical information would help clarify contributions. Also, the "Sponsors" section (see http://www.itk.org/HTML/Sponsors.htm for an example) also can be helpful in ferreting out contributions.

To confuse matters more, there are many organization beyond NIH and NLM have contributed to these systems. In particular VTK has received large contributions from the user community, as sell as the US National Labs, NSF, Air Force Research Labs, Army Research Labs, DOE, and dozens of commercial entities who do not want their names publicized. For example, in the last two years Sandia National Labs have worked side-by-side with the VTK community to implement an extensive scalable information visualization system, which I predict will eventually by used in biomedical computing applications.

Having said this, we are taking this comment to heart and will revisit the historical descriptions of these projects to provide clarity on the source of contributions. We will also remind our developers that assigning appropriate credit is important and will update the documents as necessary to reflect this.

St. Louis (Dan Marcus)

XNAT is open-source using the XNAT License . What does that mean? Is it different than the NA-MIC license?

The XNAT license is quite similar to the NA-MIC license in that it gives users freedom to modify code without sharing it and to use it for commercial purposes. Its use predates the existence of NA-MIC, so we plan to stick with it. -Dan Marcus 7/28/08

Isomics (Steve Pieper)

Isomics' statement of work states: Significant effort will be devoted to re-architecting core components of 3D Slicer to make them better interoperate with other NA-MIC tools. Wasn't Slicer developed in parallel with NA-MIC tools? How have incompatibilities arisen?

Response: Version 2.x of 3D Slicer existed for a number of years prior to NA-MIC and is still used by NA-MIC participants for specific tasks. Development of version 3.x is now about 2 years old and has been developed with NA-MIC tools as the foundation. A number of functional blocks continue to be ported and re-architected to leverage the new environment. Examples include the interactive diffusion imaging modules, the Label Map Editor, and the Image Guided Therapy interfaces. In addition, as new functionality such as XNAT and Grid Wizard are added to the NA-MIC Kit, new interfaces are required.

UCSD (Jeff Grethe)

The report includes no specific information on progress at UCSD in developing and supporting grid computing for NA-MIC.

Training Core (Randy Gollub)

Please tell us a bit more about the training core's program to provide one-on-one mentoring (it's mentioned in the timeline , but there's no information on the Wiki). How is it structured? Who can be mentored? How does one arrange for mentorship?

Some updates to the timeline are needed:

Bruce Fischl

The timeline for MGH indicates that many of the tasks have been modified, but these modifications aren't listed in the table of timeline modifications.

Dan Marcus

Why not list Wash U in the timeline for the appropriate tasks?

I sent a timeline of Wash U tasks to Ross. I'm not sure why the aren't in the document. Here they are again:

1) XNAT Desktop
1.1) establish requirements for desktop version of XNAT (complete)
1.2) develop implementation plan for prototype (complete)
1.3) implement prototype version (in progress, to be completed 3rd 
quarter, 2008)
1.4) implement alpha version(to begin 4th quarter, 2008)
2) XNAT Central
2.1) deploy XNAT Central, a public access XNAT host (complete)
2.2) coordinate with NAMIC sites to upload project data (ongoing)
2.3) continue developing XNAT Central based on feedback from NAMIC 
sites (ongoing)
3) NAMIC Kit integration
3.1) implement web services to exchange data with Slicer, Batchmake, 
and other client applications (ongoing)
3.2) add XNAT Desktop to standard NAMIC kit distribution (to be 
determined, likely 2nd quarter, 2009) Let me know if you need anything 
else or if these seem incomplete.

Ross Whitaker

Certain tasks have been removed from the Utah aims because they have been "subsumed by Core 1-2 partners", presumably MGH. Now that the plans have changed so that MGH no longer plans to do this work, this needs to be updated.

Response from RTW: The work for head tissue classification was based on the paper "MRI Tissue Classification with Neighborhood Statistics: A Nonparametric, Entropy-Minimizing Approach" by Tasdizen et al. in MICCAI 2005. This was implemented in ITK and was the first tissue classification algorithm in the ITK toolkit (as far as we know). The EM-segmenter work, which came out of MIT and Harvard, did not use learned neighborhood statistics, but used probabilistic atlases more effectively. This is now in ITK, and is undergoing continuous improvements and updates. It works sufficiently well that further engineering of the "Neighborhood Statistics" approach, from Utah, is probably not going to have an impact on the DBPs and is not an effective use of NAMIC resources.

Will Schroeder

• Has there been progress in the migration from LONI to batchmake? It is not yet listed as complete in the Isomics timeline.
• Based on the timeline, Kitware has completed its tasks . Some new tasks are listed in the statement of work and should be entered into the timeline.

Tina Kapur

For the future (assuming and hoping there is a future!) , it would be nice to have publications listed at the end of each relevant section (in addition to the Additional Information links currently provided). Several members of the Center team commented on the lack of details in the progress report and wanted to follow up by reading the relevant publications.

Response: Thanks for the suggestion. We will follow it in the future.