Projects:ARRA:miAnnotation

Back to Slicer ARRA home page

Aim

Medical images often contain a wealth of information, such as anatomy and pathology, not explicitly accessible. One way to address this issue is via image annotation and markup. We propose to create a comprehensive framework for annotation and markup within 3D Slicer, enabling users to capture structured information easily. Furthermore, we will develop schemas for saving and recovering this information into and from XNAT, allowing queries of larger data sets of medical scans. This tool will provide clinicians with a relatively simple way to capture information latent in medical scans, and also to select micro-cohorts of medical scans for studying diseases.

Research Plan

3D Slicer currently provides very basic technology for annotating images. This limits users in their ability to properly capture semantic information contained in images and data sets. We propose to address this issue by expanding Slicer's mark up and annotation capabilities. New features will include:

• a rich set of geometric objects for improved visual differentiation between annotations
• markers for measuring anatomical characteristics, such as the volume of an annotated region, to provide patient specific information difficult to extract from visual inspection
• entry fields beyond free-text, such as graphics and external data, to capture comprehensive information and support for emerging domain specific ontologiesand
• a full integration of these capabilities with the mrml tree to support Scenesnapshots, load, save both to disk and XNAT.

We will implement these features by developing two different modules. The first module, called Marker Module, creates different types of markers based on current ITK technology. The user defines the appearance of the marker by specifying its color, size, and shape, such as points and 3D boxes. The user also labels each marker with tags and specifies its function, such as measuring the volume of a region.

The Annotation Module, the second module, provides the interface for annotating images with these markers. Users place the markers on the image and further specify the semantic information through free text, plots, and references to ontology and internet. The annotations are shown both in 3D and 2D viewers. The module also allows annotating entire scenes by linking annotations across images, as well as within an image. All annotations are stored in a database targeted towards medical imaging, called XNAT. The structure of the database is automatically defined by the tags of the markers. Thus, users can query across large image data sets by looking for specific tag values.

Both modules are accompanied by training materials and documentation to ensure usability.

Design of Module

• Current GUI outline - Screenhot in Slicer
• Suggested MRML outline

Key Personnel

60% Kilian Pohl
95% Yong Zhang

Events

• 12/16/09 - 12/17/09 Annotation Brain Storming
• 01/06/10 Break out session at AHM 2010

Progress

(reverse chronological order, most recent on top)

• 01/01/10 Generated a GUI interface via QT for the annotation module. Reviewed all VTK Widgets related to annotation.
• 12/25/09 Created project and organized break out session at AHM meeting. Coordinated efforts with BWH team in preparation of the meeting.
• 12/18/09 Held workshop to integrate AMI into 3D Slicer. Hired Yong Zhang to implement AMI in Slicer.
• 12/11/09 Summary of tools demoed at RSNA
• 12/04/09 Visited RSNA to review annotation tools by GE, Siemens & Phillips. Connected to caBIG AIM project to see how we can make use of their data scheme
• 11/29/09 Created GUI for MarkUp module
• 11/20/09 Design MRML Structure of Annotation and MarkUp Module
• 11/13/09 Organize Annotation Brain Storming Session
• 11/06/09 Designed User Interface , Meet with Julien Finet and Jean-Christophe Fillion-Robin from Kitware to discuss integration of Qt in 3D Slicer
• 10/30/09 Participated in Qt-Tcon, Interviewed candidate at Almaden, coordinated efforts with Nicole Aucoin
• 10/23/09 Organized onsite interview , got in contact with Steve Pieper to discuss next steps, installed Slicer3
• 10/17/09 Started interviewing postdoc as well as solving several HR issues for hiring personal

To Do

• Prepare for AHM
  • Prepare Break out session
  • Define two use cases

Widget related action items

General

WS: revisit the picking behavior in VTK so that the default approach is to use an accelerated method.

Annotations

• vtkCaptionWidget (TestCaptionWidget) :
  • KP: The widget is buggy - the cursor icon seems to suggest a different functionality then what you can do to move the / modify the location of the text . I was only able to move the text by clicking on the lower bar of the text box. Furthermore in TestCaptionWidget.cxx I had to uncomment widget->On(); - so I believe by default it is never tested
  • KK: This has just been fixed. You should now be able to resize the caption etc. VTK/Widgets/vtkCaptionRepresentation.cxx new revision: 1.8
  • KP: Check out change
• vtkTextWidget (TestTextWidget:):
  • KP: Font size customization ?
  • KK: Currently the font size scales to the entire text box size. Check with KP on what he needs

Measurements

• vtkSeedWidget (TestSeedWidget):
  • KP: Custom measurement annotations needed, such as the intensity value
  • KK: I'll put in functionality for a callback mechanism that allows you to optionally update a label each time the seed is moved,so that a probe class can be attached to the seed widget.
• TestPolygonalHandleRepresentations:
  • KP: I will have to update VTK and Test it again
• vtkkPolygonalSurfacePointPlacer (TestPolygonalRepresentationHandleWidget):
  • KP: A little bit Slow
  • KK: the handle is constrained to the surface of the polygonal object. This involves a pick each time the handle is moved, and for a surface with tens of thousands of cells can get slow. An unconstrained handle should be fast.
• vtkDistanceWidget (TestDistance Widget) (2D distance widget):
  • KP: Instead of displaying measurement along line display it outside with dotted line such as in caption widget
  • KP: Measurement should be moved anywhere in the image without obstructing important anatomy
  • KP: Distance text font customization (too small to read)
  • KK: working on it
• vtkLineWidget2 (TestLineWidget2) (3D distance widget):
  • KP: Hard to initialize control points in 3D (difficult to judge behavior)
  • KK: working on it
• vtkAngleWidget
  • vtkAngleRepresentation2D (TestAngleWidget2D)
    • KP: Instead of displaying measurement along line display it outside with dotted line such as in caption widget
    • KP: text font customization (too small to read)
    • KP: Half circle scale customization to create less image obstruction
    • KK: working on it
  • vtkAngleRepresentation3D (TestAngleWidget2D)
    • KP: Instead of displaying measurement along line display it outside with dotted line such as in caption widget
    • KP: Widget measurement disappeared as it was placed inside the sphere (Check with KP. This is indeed the intended behaviour)
    • KP: Half circle scale customization to create less image obstruction
    • KK: working on it
• vtkBiDimensionalWidget (TestBiDimensionalWidget)
  • KP: Instead of displaying measurement along line display it outside with dotted line such as in caption widget
    • KK: working on it
  • KP: It is strange that the fourth point is set automatically but can be modified later. Once the second control point is set I would just draw a line orthogonal to the one already created . Once the third control point is set the line is fixed and only the length of the line can be changed.
  • KK: likes it that way

ROI

• TestImageActorContourWidget
  • KP: the line is drawn once the control point is set. Instead, the line should always follow the cursor - once the MB is pressed the line is fixed up to this control point
  • KK: we should create another representation then. The issue with this is that it will become slow. The contour widget as you know works with several interpolators. Some interpolators are fast, some are slow (such as livewiring etc).. The interpolation is invoked only when a new control point is pressed. Having these be computed on the fly as you move the mouse around, will be slow, depending on the interpolator.
  • KP: missing measurement with length of line
    • KK: Will add option
  • KP: and region of interest (for closed lines)
    • KK: What do you mean by measurement of a "region of interest" ? Area ?
    • KP: a closed curve defines an area - this area I call region of interest
  • KP: do not know how to delete control points
    • KK: do not know how to delete control points
    • KP: Provide Balloon with help text or menu with delete option when one specific MB is pressed. We should try avoiding key combination as they need explanation
• vtkContourWidget (TestContourWidget2):
  • KP: could not set control points as the control points are predefined. Can that be changed ?

• TestDijkstraImageGeodesicPath :
  • KP: the line is drawn once the control point is set. Instead, the line should always follow the cursor - once the MB is pressed the line is fixed up to this control point
  • KP: missing measurement with length of line and region of interest (for closed lines)
  • KP: could not figure out how to modify location of control points
    • KK: You can simply mouse over the control point and move it around. Clicking anywhere on the line adds a new control point, which also can be moved around.
    • KP: Maybe it did not work for me bc it was too slow

• TestDijkstraGraphGeodesicPath:
  • KP: same as TestDijkstraImageGeodesicPath
  • KP: seg faulted on me
    • KK: Really. It works on the dashboard ?

Widgets Completed Reviewed

• vtkBalloonWidget (TestBalloonWidget)  : ok
• TestHandleWidget:
  • KP: did not understand test
  • KK: Added comment