Projects:ARRA:miAnnotation

From NAMIC Wiki
Revision as of 14:23, 29 August 2010 by Pohl (talk | contribs) (→‎Events)
Jump to: navigation, search
Home < Projects:ARRA:miAnnotation

Back to Slicer ARRA home page

Aim

Medical images often contain a wealth of information, such as anatomy and pathology, not explicitly accessible. One way to address this issue is via image annotation and markup. We propose to create a comprehensive framework for annotation and markup within 3D Slicer, enabling users to capture structured information easily. Furthermore, we will develop schemas for saving and recovering this information into and from XNAT, allowing queries of larger data sets of medical scans. This tool will provide clinicians with a relatively simple way to capture information latent in medical scans, and also to select micro-cohorts of medical scans for studying diseases.

Research Plan

3D Slicer currently provides very basic technology for annotating images. This limits users in their ability to properly capture semantic information contained in images and data sets. We propose to address this issue by expanding Slicer's mark up and annotation capabilities. New features will include:

  • a rich set of geometric objects for improved visual differentiation between annotations
  • markers for measuring anatomical characteristics, such as the volume of an annotated region, to provide patient specific information difficult to extract from visual inspection
  • entry fields beyond free-text, such as graphics and external data, to capture comprehensive information and support for emerging domain specific ontologiesand
  • a full integration of these capabilities with the mrml tree to support Scenesnapshots, load, save both to disk and XNAT.

We will implement these features by developing two different modules. The first module, called Marker Module, creates different types of markers based on current ITK technology. The user defines the appearance of the marker by specifying its color, size, and shape, such as points and 3D boxes. The user also labels each marker with tags and specifies its function, such as measuring the volume of a region.

The Annotation Module, the second module, provides the interface for annotating images with these markers. Users place the markers on the image and further specify the semantic information through free text, plots, and references to ontology and internet. The annotations are shown both in 3D and 2D viewers. The module also allows annotating entire scenes by linking annotations across images, as well as within an image. All annotations are stored in a database targeted towards medical imaging, called XNAT. The structure of the database is automatically defined by the tags of the markers. Thus, users can query across large image data sets by looking for specific tag values.

Both modules are accompanied by training materials and documentation to ensure usability.

Design of Module

Documentation

Key Personnel

10% Kilian Pohl (UPenn)
95% Daniel Haehn (UPenn)
5% Dominique Belhachemi (UPenn)

Past Contributors

Yong Zhang (IBM)

Events

Progress

  • 08/26/10
    • added a hack to initialize renderer properly - annotations display now much better
    • cleaned the code (more then 1000 lines of code where obsolete)
    • new property dialog in the works.. including modification of the displayableManagers to support MRML and widget updates
  • 08/19/10
    • fiducials and sticky node work now
    • ruler and spline work now
      • spline widget causes problems b/c after one spline was placed, no interaction with 3D scene possible
    • ROI works now
    • all annotations now can be placed to examine widget behavior
    • started working on a new interface for the property dialog
    • consolidated experts for 2D slice displayableManagers
    • started working on synchronizing the widgets with MRML display nodes
    • fixed GUI problem: the table did not expand even if there was free space in the module frame. now it works.
  • 08/12/10
    • Wendy, Nicole and Daniel discussed how to integrate mouse modes of Slicer into Annotation module and vice versa. Solution is a widget which appears in the Annotation module panel but can also be shown outside of it as a toplevel window in Slicer (maybe with keybinding), see here: http://wiki.na-mic.org/Wiki/index.php/File:Widget.png
    • current status (workflow, user places seeds and widget appears):
      • angle annotations work (note: the widget did not support programmatical placement. a trick is used to place the widget where the user has clicked)
      • text annotations work
    • modified vtkMRMLInteractionNode to have a customTag for place modes not only from annotation module but all modules
  • 08/05/10
    • Enhanced the workflow for creating text annotations. This approach should serve as a general framework for all annotations (sequence diagram available http://wiki.na-mic.org/Wiki/index.php/File:TextAnnotationWorkflow.png)
    • modified vtkMRMLInteractionNode to support the place modes for the annotations
      • meet with Wendy next week to discuss integration
    • added a click counter class to support the creation of widgets with multiple clicks (very generalized to support all annotations)
      • plan is to place seeds before the actual widget gets created
  • 07/29/10
    • Created an extra layer called vtkMRMLAnnotationDisplayableManager which serves as a superclass for all specific Annotation displayableManagers. F.e. the vtkMRMLAnnotationTextDisplayableManager is now really lightweight.
    • Coordinated with Wendy to add additions interaction modes to vtkMRMLInteractionNode to support different interaction types for different annotations. This will happen under the hood, so the user sees only the "place" mode.
    • Got together with members of the 3D & Advanced Imaging Lab at UPENN to investigate how commercial products realize annotations (see http://wiki.na-mic.org/Wiki/index.php/Projects:ARRA:SlicerAM:vtkWidgets#How_do_they_do_it.3F).
    • Interactive placement of text annotations is now possible. This should be the standard way of placing annotations for all available types.
    • Added a test for the creation of a text annotation and its widget without a displayableManager (vtkMRMLAnnotationTextNodeAndWidgetTest1)
    • Removed initial values from AnnotationNodes due to different seg-fault errors when properties where not initialized. To cut down confusion, no initializations are now necessary.
  • 07/22/10
    • Meeting of Nicole, Steve and Daniel to talk about architecture
      • biggest change is the introduction of a hierarchy for annotation
    • fixed AnnotationTextDisplayableManager to support multiple text annotations
    • added a test for the AnnotationTextDisplayableManager
    • property dialog for annotations gets displayed again, nevertheless the connection to the MRML display nodes needs further work
  • 07/15/10
    • add text nodes using the new displayable manager architecture is now possible
      • nevertheless, full functionality is not restored yet
    • added fiducial displayable manager
    • new Slicer bug tracker category: Annotations
      • filed open bugs and feature requests - please add all new requests there
  • 07/08/10
    • moved tests to Testing/Cxx
    • created a superclass for all displayable managers (vtkMRMLAnnotationDisplayableManager)
    • created vtkMRMLAnnotationTextDisplayableManager based on the old qMRMLAnnotationTextManager
    • working on the add text node functionality with the new displayable manager
    • fixed linking errors appearing on the Slicer Dashboard
  • 07/01/10
    • re-factored module from miAnnotation to Annotation
    • re-integrating functionality without KWWidgets
      • MRMLDisplayableManager was created by Kitware guys and includes base functionality for the re-integration
      • starting with adding text nodes
    • added option to set helptext for the module in consistency with qSlicerWidget
  • 06/24/10
    • Compiling without KWWidgets on Mac OS is now possible.
    • Starting to include the functionality of KWWidgets in pure QT code.
    • Daniel Haehn took over project from Yong Zhang.
  • 06/18/10
    • Tutorial is created.
    • GetAnnotationMeasurement is modified to return multiple measurements.
    • ROI annotation is modified to display extension measurements.
    • Bidimensional annotation is modified to display bi-line measurements.
  • 06/11/10
    • Documentation page is created.
    • Bugs for property dialog are fixed.
    • Constructions of Bidimensional and Spline annotations are modified using proper control point initialization.
    • Bugs for showing values for some annotation types are fixed.
  • 06/04/10
    • vtkSlicerAnnotationROIManager is implemented.
    • vtkSlicerAnnotationTextManager is implemented.
    • vtkSlicerAnnotationSplineManager is implemented.
    • vtkSlicerAnnotationBidimensionalManager is implemented.
  • 05/28/10
    • vtkMRMLAnnotationROINode is added;
    • vtkMRMLAnnotationBiDimensionalNode is added;
    • vtkMRMLAnnotationSplineNode is added.
    • Logic and ModuleWidget are modified to include new Nodes
  • 05/21/10
    • ROI, Text, BiDimensional, and Spline Annotations are initialized.
    • fixed BUG: HTML report cannot be saved properly.
  • 05/14/10
    • fixed BUG: When opening display properties from Sticky note -> reduce frame and enlarge it again by pressing triangle next to title : value and coordinates appear
    • fixed BUG: fiducial manager is added.
    • fixed BUG: AnnotationMouseModePick/Place are updated.
  • 05/07/10
    • fixed BUG: Lock does not lock the position of a widget in the 3D Viewer
    • fixed BUG: changing coordinates to AngleNode does not have any effects on the Angle Widget.
    • fixed BUG: fix coloring for annotation selection
  • 04/30/10
    • fixed BUG: changing control points cannot be done when property dialog is open.
    • fixed BUG: annotation is not highlighted when selected.
    • fixed BUG: starting from the 2nd annotation, changing the location of the left point for ruler or center point for angle does not update the table properly.
    • fixed Bug : changing the center point of angle node does not update table properly
  • 04/23/10
    • Created a ruler widget manager
  • 04/16/10
    • Created the new annotation type sticky note, which allows you to enter text independent from the viewer
  • 04/09/10
    • Modified source code to be compliant with VTK 5.6
    • Grid out your buttons (Create Frame)
    • Fixed BUG: When changing line color in Display Properties it changes the Text Color
    • Added script for testing color setting
    • Fixed BUG: Label of line color changes in property window when defining a unselected color
  • 04/02/10
    • Fixed BUG: Nothing happens when Property Window is open and you press the edit button for another annotation
    • Restructured Annotation MRML
    • Fixed Bug: Resetting of Lines and Points does not work correctly GetNumberOfLines > 0
  • 03/26/10
    • The property restoration is now implemented properly.
    • Property Modification is re-designed to simplify the operation.
    • Removed bug: ruler/angle disappear after adding fiducials
    • Removed BUG: in property dialog, when editing the text, the character is entered from right to left.
    • include lock annotation into option button (use same logic as for new (De)select All button)
    • For each annotation type provide list outlining which display property can be changed, which cannot , and which one you do not know
    • Resolved BUG: When saving a report with a file name that already exists then the following message appears in the command shell "Error: cannot make directory" and directory with files is not updated
    • Resolved BUG: Annotation Properties: (Un)selected color is initially white even though widget in 3D viewer is different
    • Resolved BUG: Select Color window does not show the correct color
    • Resolved BUG: Selected and Unselected Color are switched
    • Define Tooltips for fiducial, angle, ruler,report, save mrml button
    • In table: Add proper tool tip for visualization and edit button
    • Resolved BUG: when saving report with out ending html to the file name - icons will not be saved under Linux
    • Created Test Module for testing miAnnotation logic
    • In collaboration with Nicole Aucoin figured out line display bug in 3D Viewer
    • Use the Kitware slider and colorPicker under Display Property
    • Fixed BUG: Slicer crashes when creating an annotation while Display properties is open, and when closing the property window and then pressing edit for the new annotation
  • 03/19/10
    • In the property window display properties are shown via collapsed frame by default
    • List all attributes for text/point/line and implement those that can be modified
    • Fixed BUG: When pressing the property button warning messages pop up in my command shell
    • Annotation Properties: For Angle and Ruler Widget also shows now values of AnnotationDisplayNode and AnnotationControlPointsNode
    • Change design of html report
    • Implemented text edition and update for ruler and angle widgets.
    • Changed the default mouse mode icon to MousePick.
    • Modified Report Dialog to avoid horizontal slider when screen shot is included in the report.
    • Removed multiple warning messages.
    • Fixed Bug in Report: empty text fields are displayed without boundaries
    • Ruler Distance is now displayed just in mm
    • After talking to BWH changed name of tool from Reporting to miAnnotation
  • 03/12/10
    • Created icons for:
      • edit conditioning for all annotation types (done),
      • mouse mode icons, pause, play, cancel, ok/done, (Un)lock (done)
      • new the annotation types poly line, spline with control points, volume information extractor, and sticky note (done)
      • Modify fiducial icon just to show one star and change save annotation icon (done)
      • Modify save annotation and save scene icons (done)
      • Modify move up/down delete icons (done)
    • Write 5 page, double column, AMIA paper about Reporting
  • 03/05/10
    • Removed Warning Messages when compiling
    • Fixed Bug: When entering text in property box (under linux) it is entered from right to left instead left to right
    • Fixed Bug: Annotation Property: funny behavior with selecting set of digits in a numerical value and typing in a numeric value - only the first digit gets change.
    • Fixed Bug: Value of Angle node is not update in table
    • Fixed Bug: Updating the value in the table slows down the operations
    • Changed the default ruler from 1mm to 500mm.
    • Angle Widget is added by manually clicking three points on the viewer. The angle value is now shown in the table. The behavior of the angle button is in the "toggle down" state until all three control points are set
    • When selecting annotation in table, color of annotation in 3D Viewer changes to selected color
    • Revised "add ruler" button to click-style instead of toggle-style
    • When creating a new annotation, set this entry to selected and all other table entries to unselected
    • (De) Select All button: change as discussed with Wendy Plesniak, BWH, to option button with proper logic
    • Setting color/selected color for individual Ruler and Angle widgets in property dialog
    • Fixed Bug: hiding/unhiding ruler and angle annotations
    • Change display property for each annotation individually
    • Lots of warning messages when compiling code
  • 02/26/10
    • Created pop up window showing screen shot
    • Manual editing of coordinates via "Property Window"
    • Resolved column width problem in table (original problem, second screen shot problem)
    • Fixed bug where hide icon changed size in table
    • Include icons when saving html report in a directory with the same name as the html file
    • Reorganized GUI based on feedback from Wendy Plesniak, BWH
    • Include screen shots (if any) in the report (see an example)
    • Fixed the following bug: After resizing the 3D Window taking a screen shot only displays the window with the original size (see screen shot)
    • Fixed Bug: Value update of ruler widget is slow
    • Text defined in property box now shows up in table
    • Integrated vtkMRMLAnnotationAngleNode with AngleWidget
    • Added new buttons to create frame based on feedback from BWH
  • 02/19/10
    • Implement a general structure for vtkMRMLAnnotation*Node
    • Work together with Kitware to resolve QT Table layout problem
    • Allow defining long text annotations for each widget
    • Improve user interaction through pop up windows when trying to modify an existing annotation
    • Creating and removing ruler now works correctly
    • Use correct label format to display measurement values in Annotation Property window
    • Created a "Save annotation" Window
    • Fixed changing icon size in table (see original screen shot )
    • Fixed slicer crashing closing the property window
    • Renamed and resized columns in Table
  • 02/12/10 removed bugs related to table buttons, linked button to angle widget and ruler widget
  • 02/05/10 implemented "Save Annotation Report" button and "Move up/down" buttons; added extra column in the report table for visibility; modified annotation property dialog; implemented button-click in the table.
  • 01/29/10 implemented functionality for "Annotation Property Modification" button, "Create Report" button, and integrated new icons for buttons.
  • 01/22/10 Set up the source control for the reporting module; implemented functionality for "Select All" button, "Visible" button, and "Delete" button in the GUI; created new icons for buttons
  • 01/15/10 Organized biweekly Tcon
  • 01/08/10 Participated in all week All Hands Meeting in Utah; Expanded QT interface with functionality to set seeds in images; Defined a final MRML Structure for storing annotation.
  • 01/01/10 Generated a GUI interface via QT for the annotation module. Reviewed all VTK Widgets related to annotation.
  • 12/25/09 Created project and organized break out session at AHM meeting. Coordinated efforts with BWH team in preparation of the meeting.
  • 12/18/09 Held workshop to integrate AMI into 3D Slicer. Hired new person to implement AMI in Slicer.
  • 12/11/09 Summary of tools demoed at RSNA
  • 12/04/09 Visited RSNA to review annotation tools by GE, Siemens & Phillips. Connected to caBIG AIM project to see how we can make use of their data scheme
  • 11/29/09 Created GUI for MarkUp module
  • 11/20/09 Design MRML Structure of Annotation and MarkUp Module
  • 11/13/09 Organize Annotation Brain Storming Session
  • 11/06/09 Designed User Interface , Meet with Julien Finet and Jean-Christophe Fillion-Robin from Kitware to discuss integration of Qt in 3D Slicer
  • 10/30/09 Participated in Qt-Tcon, Interviewed candidate at Almaden, coordinated efforts with Nicole Aucoin
  • 10/23/09 Organized onsite interview , got in contact with Steve Pieper to discuss next steps, installed Slicer3
  • 10/17/09 Started interviewing postdoc as well as solving several HR issues for hiring personal

Feature Request

VTK Widgets

QT Related

  • Submit bugs and feature request via Mantis
  • Feature requests that may open discussions can be added here

Daniel

  • By Ron: specify Use Cases for different Annotations
    • Text: short and long text (multiple paragraphs) should be possible
      • include some international and some special characters in your long version of a test text for the text annotation tool (e.g. a with Umlaut, c with circumflex, the dollar sign)
  • By Ron: mode for automatic alignment of annotations
  • add a scalar color bar annotation
  • implement hierarchy (waiting for nicole)
  • add MRMLDisplayNodes (HIGH PRIORITY)
  • Screenshots of 2D viewers
  • Polyline does not work (HIGH PRIORITY)
  • ROI does not work (HIGH PRIORITY)
  • ruler node does not work (HIGH PRIORITY)
  • initial values are incorrectly shown in the property dialog (HIGH PRIORITY)
  • delete text in the property dialog may crash Slicer3 (HIGH PRIORITY)
  • icons are not saved properly in the HTML report.
  • entering text and numbers in property dialog has very funny behavior
  • Have Fiducial saved via vtkMRMLAnnotationFiducialNode
  • implement "reset viewer" button - viewer is returned to setting when annotation was last modified
  • Include annotation that automatically reads in information from dicom header file such as patient name , ...
  • Add flag to report so that if it is set only visible annotations are shown

Yong

  • List of Bugs
    • Fiducial is using FiducialListNode instead of using vtkMRMLAnnotationFiducialNode
    • Hide and lock functions apply to all fiducials in the viewers instead of single selected fiducial.
    • When a new ruler node is created, it is not shown in the viewer. It is shown only when another annotation is created (this bug occurs when slicer3 is updated to the latest version, ruler node works fine with the old version of Slicer3).
    • Initial values are incorrectly shown in the property dialog (only shown correctly after changing the annotation in the viewers).
    • Screen shots options are not implemented.
    • resetting viewer button is not implemented.
    • mouse mode buttons are not implemented.
    • icons are not saved properly in the HTML report.
    • delete text in the property dialog may crash Slicer3


  • Major
    • define functionality for the following buttons:
      • mouse mode status icon
      • radio button with play and pause, cancel, done
    • define functionality for the info extractor button which will be a option button that displays all volume nodes -> you select the node to extract the info from the header => annotation is displayed in pop up window (similar to sticky node)
    • Save the view with the annotation it was last time modified in the 3D viewer
    • put options for screen capture into new screen capture pop window


  • Minor
    • Turn Screen Capture button into option button with options: Active viewer, All viewer, slicer window - by default last action
    • Fiducial button: put label under button (after click on) what to do next such as setting seeds in 3D window
    • Create a Slicer widget for a multi column list with convenience button that we currently have - have options for making them visible or not
    • Combine move up and down button into one option button which shows the last action (up or down) by default
    • Create window showing volume tab when pressing Adjust Volume Properties
    • Show current mouse behavior on mouse behavior button - also show menu to change mouse behavior - use QPushButton::setMenu()
    • Show options in Change Viewer Layout use QPushButton::setMenu()
    • Generate testscript to interactively test various widgets and GUI elements
    • create local copy of the doxygen file using getbuildtest.tcl --no-slicer-update -doxy
  • Not Important
    • Have text field in table to fill out space also when changing size of window (see current screen shot )
      • Yong: this requires a re-design of the resizeEvent() of the QTableWidget, do we really need to do it?
    • BUG: When saving a report with a file name without html extension the save window will check for the existence of that file name instead of the file name + .html
      • Yong: tried but cannot fix it unless go into the QFileDialog source code, it is recommended to include the extension name when using in Linux.

Kilian

  • investigate vtkMRMLHierarchicalModelNode for annotations

Wendy

  • Extend the Save Data Widget API to display one data type and hide save options

Steve

  • Bug: Ruler still visible in 2D window after pressing hide button. This bug has been reported to Mantis http://na-mic.org/Mantis/view.php?id=760
  • For compiling Slicer3 in windows OS, has to use "SET PYTHON OFF"

Julien / JC

  • BUG: When pressing save button reporting module is replaced with last module visited before: an email is sent to report the issue to Julien and JC.

Mar 02: Work around : go to File-> Load Scene-> Cancel

Nicole

Mar 02: Adding SeedWidget to Reporting Module - Nicole is waiting for the integration of the new Vtk version into Slicer

Karthik Krishnan

Resolved

Display Properties of Widgets

  • Unavailable / Unknown:
    • Set text colors for Ruler and Angle Widget
    • Get/Set glyph scale / point size for point display node
    • Get/Set text opacity/ambient/diffuse/specular from text display node
    • Color of "angle line" cannot be defined
  • Available:
    • Get/Set text colors from display node
    • Get/Set text scale from display node
    • Get/Set point colors from point display node
    • Get/Set point opacity/ambient/diffuse/specular from point display node
    • Get/Set line colors from line display node
    • Get/Set line width/opacity/ambient/diffuse/specular from line display node
    • Get text format for angle and ruler
Retrieved from "https://www.na-mic.org/w/index.php?title=Projects:ARRA:miAnnotation&oldid=57602"