Projects:QIN:3D Slicer Annotation Image Markup:Design and Implementation

Back to Projects:QIN:3D Slicer Annotation Image Markup

Slicer Reporting Module

Terminology

• Annotation: textual structured description of the finding. How do we differentiate those from Slicer Annotations?
• Markup: visual cues placed over the image to assign annotations to image features

Implementation considerations

• We will use existing annotation hierarchies to organize the AM. AM will be organized into findings that will be located in the hierarchy under a particular volume. Each volume can have separate findings. Each finding can be assigned to only one volume.
• We are focused on the support of markup in 2D slices; 3D view annotations are secondary.
• The functionality needed to support connection between the data and AM will be added to Slicer core. User-level interactions will be available in a separate module.
• Markup is defined relative to the active slice of the active volume. AIM/DICOM SR do not have a notion of 3D coordinate space (needs verification). Slicer annotations will be restricted to be in-slice for the use cases of this project.
• Annotations text can be stored in the "description" field of the Slicer Annotations in the XML format, as determined by the annotation template used.
• Annotations text can be formatted and visualized using HTML as a report in the Slicer Annotation description area.

Input from Justin Kirby:

• allow loading reports, editing, and saving again
• allow multiple reports per volume
• allow multiple observers per volume
• include points (use case: renal application in preparation)

Markups to be supported

These will be decided based on the existing requirements from the use cases provided by the QIN community, and based on the existing capabilities of the AIM-enabled tools (OsiriX and ClearCanvas TCGA workstation). The tentative list in the order of priorities (not finalized):

• Point
• Ruler
• 3d segmentation

Example AIM templates

AIM template schema: File:AIM Template v1rv18.xsd.zip

• RANO File:RANO4 AIM template.xml.zip
• BRCA File:BRCA AIM template.xml.zip
• VASARI File:VASARI AIM template.xml.zip

GUI

Ron's suggestions

• will need to revisit the tree -- it is not user-friendly
• use radio boxes instead of dropdowns to minimize the number of clicks
• consider using webkit for form visualization?

User-level (implementation level) Workflow

Creation:

• 1 choose volume being annotated ( implementation --> create corresponding node in the hierarchy, initialize slice viewers, rotate to slice plane)
• 2 select annotation template (default: finding? option to load template?) ( --> populate the GUI with the annotation template contents)
• 3 populate template ( --> NA)
• 3 scroll to the slice
• 5 add new finding into the hierarchy ( --> need to have a button to add new finding? all subsequent markups will go under that tree element)
• 6 place markups ( --> support markup type selection from slicer toolbar? add extra markups compared to Annotation module, and remove them when user leaves the module? for 2d/3d ROIs -- invoke Editor?)
• 7 assign text (annotation of annotation in AIM?) -- all go under the active annotation hierarchy node
• 8 done: save/export results ( --> Slicer save dialog? save annotation as AIM XML or as "AIM-light"? include name of the observer, creation date)

Loading:

• may distribute the template used in annotation together with the scene or not?
• if we allow editing of the annotation/markup, when saving need to update this (think about data provenance/revision history later)

Comments:

• User can switch to multiple volumes over the annotation period. This is ok, separate hierarchy element for each volume.

Progress

Source code on github: https://github.com/fedorov/Reporting

Active task list

Open tasks are now tracked as Issues on the project github repository http://github.org/fedorov/Reporting

• add "observer" name as a Slicer setting, or query from the system settings
• make a demo screencast
• documentation
• support of standard vocabulary for terminology: DICOM SEG uses SNOMED CT (SRT). Options for integrating: drop down? web view + search? direct search? Consider: licensing issues, relation to the color tables, synergy with NAC onthologies -- talk to Mike Halle

Completed tasks

• functionality
  • implement support for DICOM segmentation objects
    • DICOM SEG can keep all the information for the segmentation+term annotation, no need for AIM/SR !
    • integrate with module IO
• Nicole:
  • add extra handling for slice nodes: detect acquisition plane and rotate to volume slice
  • error checking on markup placement: verify that place, export, import will result in same markup locations
  • make sure assumption of single volume in the report does not cause us difficulties later
  • add support for one volume in multiple reports
    • determining which is the active markup list when switching between reports is the current challenge
• error checking on markup placement: detect situations where multipoint markup elements span separate slices (ruler etc. won't be added to report)
• handling of markups placed not on the volume (not added to markup list for this report)
• trace labelmaps that are associated with the volume being annotated (listen to AddNode events, and add all label maps for volume node in the report to the hierarchy)
• add handling of "eye" visibility icon for markup elements
• switched from RANO to a "generic" QIN template (ie, the only annotation element is the free-text name of the entity); RANO-based work is stored in the RANO branch on github
• add jump to the slice where the markup element is located on tree hierarchy click
• consistent user experience (no duplication of hierarchy/volume nodes, correct parenting)
• AIM export that contains image references, annotation, markup (fiducial and ruler types) is implemented using QtXml
• AIM Import
  • AIM import that contains rulers implemented using python dom.minidom
  • referenced series are added automatically using Slicer DICOM module functionality
• Need to support MRML scene round-trip for Reporting (verify this is working after all elements in place: fiducials, rulers, labels)
• scene restore with ruler annotation is crashing
• add module-specific layouts: 2-over-2 slice viewers
• add parameter node to store selected report&volume
• disable GUI elements to make workflow more intuitive
• testing for RANO node
• hide markups that do not belong to the active report
• show only active report in the widget
• RANO MRML node events/serialization completed
• Add functionality to track the UID of the DICOM volume based on the current volume and IJK coordinates
• Add links between AM (hierarchy? finding?) and parent volume
• Start work on the module (loadable module, python GUI scripting)
• decided: we will not be adding any new widgets as part of this project
• decided: no support for generic annotation templates (at least until DICOM SEG is supported and implementation is stable for the finalized features)

Items being researched

XML-related background information

The basic need is given the AIM/AIM template schema, how to correctly parse XML documents that satisfy this schema, and how to create schema-compliant documents.

AIM API tools (Northwestern) use C++ classes corresponding the schema element types generated using Altova XMLSpy. CodeSynthesis provides another solution for this task. Both are not acceptable as they are commercial (perhaps we could reuse code generated for AIM API, but that one would need to be ported to become cross-platform).

Qt has QtXml and QtXmlPatterns modules to support XML interactions, but no C++ binding tools like XMLSpy/CodeSynthesis (see this thread, Jan 2011).

Python seems to have some tools to generate Python classes/modules and parsers for a given schema, e.g., PyXB, PyXSD, generateDS (see the summary of available tools here).

Possible implementation approaches:

• 1. (short-term) create manually the elements and initialize attributes for a subset of AIM that we will need to support the annotations and markup we have right now.
• 2. re-use the C++ classes in NW AIM API.
• 3. use python-based approach to generate classes, validate and support serialization.

Additional considerations for python XML tools:

• both PyXB and generateDS appear to be active projects
• generateDS has limited support for namespaces, see this thread (need to clarify if we need that for AIM schema)
• PyXB generated bindings .py output includes pyxb, introducing a new package dependency; generateDS depends only on the system modules (see comparison here)
• generateDS does not seem to provide schema validation mechanisms (but we can do that using Qt?)
• PyXB: need some special handling of namespaces (?)
• generateDS needs lxml (not included in Slicer)

Items for discussion at the weekly meetings

• Jul 19
  • generation of unique UIDs for AIM
  • how should the series numbers be generated/assigned for the segmentation objects?
  • API for seg object save/import
  • Editor panel customizations
  • Status of automated GUI testing?