Projects:ARRA:SlicerAM:vtkWidgets

Back to Slicer Report home page

Design Notes (Nicole & WJP)

See here for some of the preliminary discussions. Partial list of vtk widgets.

APPEARANCE wish list

Widgets and handles should clearly distinguishable and consistently presented. Widgets and handles can be in the following states:

• locked/unlocked (so we know why a widget appears enabled but isn’t pickable.)
• active/inactive (so we know when a widget/handle has been picked and can be manipulated)
• created/nascent (so we know when a widget is not fully specified yet in click-to-place mode)
• There should be widget / handle LOCK color/treatment.

There should be a near-widget near-handle active color/treatment that shows when mouse is hovering over a pickable or picked thing. There should be a nascent color/treatment that shows that a widget is not yet fully defined. These colors/treatments should be consistent in 2D and 3D.

BEHAVIOR wish list

Mousing over a widget should highlight the widget and highlight all handles. User can grab widget and translate it to new location. Mousing over a handle should highlight the individual individual handle. User can grab handle and move it to a new location. Hmmm: how to instrument this at widget-level? --NearWidget and NearHandle events or just booleans on the widgets & handles? Widget creation behavior:

• Default (press button and widget appears at origin)
• Click to globally place (click in viewer and widget appears at that location)
• Click-rubberband to define extent of widget (like endpoints of a ruler or angle widget).
• Click-without-rubberband?
• Create from fiducials? (control-m for ruler implemented in Measurements module)

Hover over widget: cursor changes and color changes,

• Once hovering, right-click raises options menu including
  • Constrain to surface (lock/unlock)
  • Constrain to location (lock/unlock)
  • Set Representation (glyph shape)
  • Set properties (color, visibility)
• Left click allows Pick & Manipulate on handle or widget

Widget editing: for some widgets we may want to change appearance while editing (like text widget, or text fields of a widget.) Ilustrations shown below. Please note any problems!!!

• Handling widgets whose specification needs to be interrupted -- like the vtkContourWidget. A storyboard may go like this:
  • user selects to create contour widget
  • slicer goes into persistent place mode.
  • user clicks to place four control points.
  • user double-clicks to end contour widget definition.
  • slicer switches back into transform view mode.
  • user repositions the view
  • user mouses over the entire contour widget and it highlights.
  • user right clicks for options menu and selects "Add more points"
  • slicer goes into persistent place mode again
  • user clicks to place more control points.
  • etc.

Widget behavior: hover highlights; left mouse button down picks and manipulates.

Handle behavior: hover highlights, left mouse button down picks and manipulates

Locking a Handle: mouse over highlights; right button exposes options menu; Left mouse button selects option.

EVENTS WE NEED (to manage cursor and mouse-modes)

• Handle hover event
• Widget hover event
• Widget created event
• Editing event (we need for widgets with text that can be interactively changed)
• Left-mouse-down
• Left-mouse-release
• Right-mouse-down
• Middle-mouse-down
• Middle-mouse-release
• Key-press
• Mouse-Wheel

This includes events that widgets observe and emit...

LINK to MOUSEMODES

hover and NearHandle:

• Widget and handles highlight.
• Application switches to Transient Pick mode;
• can left-button-press to pick and move handle;
• right-button-press options menu configures individual handle.
• on left-button-release, mouse-mode switches back to default.

hover and NearWidget:

• Widget and handles highlight.
• Application switches to Transient Pick mode;
• can left-button-press to pick and move entire widget;
• right-button-press options menu configures widget and all handles.
• on left-button-release, mouse-mode switches back to default.

Question: Once this works properly, does application NEED toolbar icon for pick mode at all? Maybe it becomes implicit. Place and TransformView are the only two modes needed in toolbar? Yes, we probably need all mouse modes.

API wish list

1. All vtkHandleRepresentations within a widget should have independent accessor methods as well as methods to get/set properties on all. Need accessor methods on properties including:

• color
• text
• process events on/off (for locking things)
• methods for turning on/off zoom-level dependent text scale
• interaction state
  • locked=true/false
  • selected=true/false (right-click could select and configure, e.g. turn lock on/off)
  • mousehover=true/false
  • updating=true/false (left-click could select and drag, causing updates in position
• all handles should have a selected property.
• all handles should have a lock/unlock -- allows handle hover event to be processed, but doesn’t allow handle to be picked and moved.
• all widgets and handles should have adjustable NearHandle and NearWidget heuristics.

2. We need accessor methods to return each widget or handle subclass type -- (to avoid need to SafeDownCast to particular subclass). For instance: widget->GetRepresentation() always returns superclass, and handle->GetRepresentation() always returns superclass. Need convenience methods to avoid needing to do: vtkAbstractPolgonalHandleRepresentation *polyhandleRep = vtkAbstractPolygonalHandleRepresentation::SafeDownCast ( handle->GetRepresentation() );

3. For any vtkHandleWidget, we want to be able to set a new vtkAbstractPolygonalHandleRepresentation or one of its subclasses -- rather then having these be automatically cloned. (Cloning prevents setting different values for properties on different handles.)

4. If multiple handles exist on a single widget, vtkAbstractWidget->ManageCursor() should be respected by all of them -- they should not attempt to override each other! Maybe a right-click should allow us to lock/unlock a handle when we are hovering.

5. We need to be able to distinguish between NearHandle and NearWidget events so Application can provide the right user feedback on what’s pickable (whole widget, or just handle?)

6. Wherever possible, all widgets should have both a 3D and a 2D implementation, and their class naming convention should reflect 2D or 3D appropriately.

PRIORITY WIDGETS

Bold widgets below are being used by Slicer3.6. Other widgets included are widgets that we know we'd like to use in Slicer4.

• vtkAffineWidget (need 3D version: a backport from vtkSlicerBoxWidget2 has required logic)
• vtkAngleWidget
• vtkBalloonWidget (and other special classes of vtkHoverWidget)
• vtkBidimensionalWidget
• vtkBoxWidget (lower priority)
• vtkCaptionWidget
• vtkContourWidget
• vtkDistanceWidget (need to have 3D version of this!)
• vtkHandleWidget
• vtkImplicitPlaneWidget2
• vtkLineWidget
• vtkLineWidget2
• vtkTensorProbeWidget
• vtkTextWidget
• vtkSeedWidget
• vtkSplineWidget2
• vtkSplineWidget
• Expand ROI widget (RON):
  • The most general case is a boolean of multiple prisms (like in the AZE demo on Thursday late afternoon).
  • brick, skewed brick, cone, ellipsoid as special cases.
  • applicable to volumes (with resampling), surface models (properly closing cut surfaces), tensor fields (with resampling).
• cutting planes (any kind of cuts, maybe reformat widget is the right one, with clip outside option)

API Plan

To probe gaps in the API across widgets, plan is to set up testing macros for all of the following vtk classes, from the superclass down to the specific subclass implementation level.

Question: Who does this? Ideally, these tests should be added to VTK testing (not Slicer).

• Macro testing example by Nicole in Slicer3/TestingMacros.h
• used in /Libs/MRML/Testing/(fiducials tests).
• write test, add to CMakeLists.txt and
• run using ctest testname -vv.

Classes to include:

• vtkAbstractWidget-- see link
• vtkWidgetRepresentation and subclasses-- see link
• vtkHandleRepresentation and subclasses-- see link

Notes on current state of writing tests here, first pass of testing checked into SlicerVTK on github, available when building using getbuildtest.tcl --update (have to turn VTK testing on manually).

Review Widgets

Meeting at Summer Project Week 2010

• Attendees: Kilian, Wendy, Nicole and Daniel

VTK Related Issues

If you add any new items in the summary below please add your name to them (KP)

Widget appearance & behavior design goals. All widgets (in 3D and in 2D) should:

• Expose a "hover" event that the application can observe.
• Expose a "pick" event that the application can observe.
• Expose a "start interaction" event (same as above?)...
• Expose an "editing" event (for things like text widgets).
• Expose a "release" event...
• Expose an "end interaction" event (same as above?)...

• Consistently display what parts of them are selectable.
• Consistently display what parts of them are NOT selectable.
• Consistently display what parts of them are selected.
• Consistently display when they are being manipulated.
• Consistently display when they have been released.
• Have a consistent mapping between their presentation in 3D and 2D viewers.

General VTK Issues

• Improve Consistency across Widgets:
  • vtk widgets should map both into 3d viewers and slice viewers with consistent appearance and behavior
    • same API for both viewer modes in VTK
  • same API/variables across widgets
  • controls for glyphs and fonts for all widgets (use current fiducial module as a start point). Meaningful defaults default settings are the same across widgets
• Create/improve Documentation for each widget
• tooltips for every widget
• improve graphical design of widgets to be more pleasing for the eye (KP)
• Placement behavior:
  • rubberband between clicks
  • click points and hotkey, e.g. set two points in 2D viewer , press ctrl M (for measurement) , and then a ruler is placed between the points
  • parameterized place (f.e. from MRML or GUI panel) -> default setting is one case of parameters across
• Manipulation behavior:
  • highlighting when something is selectable
  • user needs to know when something is selected and _which part_ of the widget is selected
    • there is a selected property for most of the widgets
  • left button would move the widget, right button opens context menu for additional configuration (color,...)
• intuitive initialization would be very important (Ron)

Missing Wigets

• Expand ROI widget (RON):
  • The most general case is a boolean of multiple prisms (like in the AZE demo on Thursday late afternoon).
  • brick, skewed brick, cone, ellipsoid as special cases.
  • applicable to volumes (with resampling), surface models (properly closing cut surfaces), tensor fields (with resampling).
• draw with auto-complete (maybe contour widget is the right one), spline open and close
• hold down button and draw
• cutting planes (any kind of cuts, maybe reformat widget is the right one, with clip outside option)
• create Textbox with transparent background color as shown here (KP)
• Bi-Dimensional Measurements: Provide placing mode where the second ruler can only be set perpendicular to the first ruler (KP)
• One widget Ron requests (that could be build out of existing vtkTextWidget) is a high-level description that goes with a scene OR a scene snapshot. Its behavior would be such that it gets optionally displayed when the scene is loaded, or when a scene snapshot is restored, and its text content could be accessed and used to populate a balloon help widget in qt...

Slicer Related Issues

• probably through the Slicer MRML tree, displayNodes should be the same (one manager handles 2D and 3D)
  • KP: I do not understand this - it was listed below "same API for both modes in VTK"
  • DH: I think, it means that the 2D and 3D versions of widgets are accessible through the same API by one manager inside the MRML tree. Does this make sense?

Follow Up

• we need a review session in a smaller group (maybe Nicole, Daniel with JC or who is responsible)
• we need to come up with specific tests and examples for Kitware
• Wendy will create a document specifying the general behavior of widgets (Deadline July 2)
• All participants of the meeting will work on this summary until July 2nd
• We will send the list to Will on July 7th

Widget Specific Issues (updated by Daniel during the implementation of the annotation tools)

1. We tried to standardize the placement of all widgets
2. We chose click to place as a current placement mode
3. We needed to do some hacks to make a possible with the current widgets
4. See also "VTK Widget related action items" further down

• Angle widget:
  • has crosses instead of spheres, should be changed for consistency
    • I manually changed the handles of the representation to red spheres (DH)
  • there are a lot of bugs in the widget
    • indeed: sometimes the angle caption does not appear, only after rotate
  • set position is not possible (realized right now through a hack using vtkInteractorEventRecorder)
• Spline widget:
  • maybe vtkPolygonalSurfaceContourLineInterpolator or vtkBezierContourLineInterpolator are more suitable than vtkSplineWidget2
  • do not know how to quit the SPline interaction after the widget was placed (sent mail to vtk-developers http://www.vtk.org/pipermail/vtk-developers/2010-August/008340.html)
• Ruler widget:
  • when line is resized, text size does not change
  • enable ticks similar to these examples
    • DH: The examples are 3D Slicer scenes? Maybe we should contact the creator and get the code?
• Textbox:
  • set position of the representation is only possible in normalized viewport coordinates (inconvenient)
  • change text by right click or double click
  • move on Mac OS not possible because lack of third mouse button
  • widget does not move when user zooms or rotates the cube. bug or feature?
    • it seems to be a feature but we want this functionality.. other widget?
• ROI (box) widget:
  • do not know how to quit the box interaction after the widget was placed
• Bidimensional widget:
  • very dirty hack to place the widget (1. use vtkInteractorEventRecorder to simulate clicks for the first line, then simulate another click to exit the interaction because the second line has to be placed by strange mouse movements 2. then I can set the position of the second line through the representation)
  • makes Slicer very slow when multiple widgets are placed
    • could be my error (DH) - I will write a test
• Logo widget for sticky notes:
  • like text widget only on a layer on top of the 3D view
• Seed widget:
  • no complaints right now :)

VTK Widget related action items

General

WS: revisit the picking behavior in VTK so that the default approach is to use an accelerated method. NA: is updating Slicer so that Slicer compiles with the newest VTK cvs head and can test new widgets

Annotations

• vtkCaptionWidget (TestCaptionWidget) :
  • KP: The widget is buggy - the cursor icon seems to suggest a different functionality then what you can do to move the / modify the location of the text . I was only able to move the text by clicking on the lower bar of the text box. Furthermore in TestCaptionWidget.cxx I had to uncomment widget->On(); - so I believe by default it is never tested
  • KK: This has just been fixed. You should now be able to resize the caption etc. VTK/Widgets/vtkCaptionRepresentation.cxx new revision: 1.8
  • KP: Check out change
• vtkTextWidget (TestTextWidget:):
  • KP: Font size customization ?
  • KK: Currently the font size scales to the entire text box size. Check with KP on what he needs

Measurements

• vtkSeedWidget (TestSeedWidget):
  • KP: Custom measurement annotations needed, such as the intensity value
  • KK: I'll put in functionality for a callback mechanism that allows you to optionally update a label each time the seed is moved,so that a probe class can be attached to the seed widget.
• TestPolygonalHandleRepresentations:
  • KP: I will have to update VTK and Test it again
• vtkkPolygonalSurfacePointPlacer (TestPolygonalRepresentationHandleWidget):
  • KP: A little bit Slow
  • KK: the handle is constrained to the surface of the polygonal object. This involves a pick each time the handle is moved, and for a surface with tens of thousands of cells can get slow. An unconstrained handle should be fast.
• vtkDistanceWidget (TestDistance Widget) (2D distance widget):
  • KP: Instead of displaying measurement along line display it outside with dotted line such as in caption widget
  • KP: Measurement should be moved anywhere in the image without obstructing important anatomy
  • KP: Distance text font customization (too small to read)
  • KK: working on it
• vtkLineWidget2 (TestLineWidget2) (3D distance widget):
  • KP: Hard to initialize control points in 3D (difficult to judge behavior)
  • KK: working on it
• vtkAngleWidget
  • vtkAngleRepresentation2D (TestAngleWidget2D)
    • KP: Instead of displaying measurement along line display it outside with dotted line such as in caption widget
    • KP: text font customization (too small to read)
    • KP: Half circle scale customization to create less image obstruction
    • KK: working on it
  • vtkAngleRepresentation3D (TestAngleWidget2D)
    • KP: Instead of displaying measurement along line display it outside with dotted line such as in caption widget
    • KP: Widget measurement disappeared as it was placed inside the sphere (Check with KP. This is indeed the intended behaviour)
    • KP: Half circle scale customization to create less image obstruction
    • KK: working on it
• vtkBiDimensionalWidget (TestBiDimensionalWidget)
  • KP: Instead of displaying measurement along line display it outside with dotted line such as in caption widget
    • KK: working on it
  • KP: It is strange that the fourth point is set automatically but can be modified later. Once the second control point is set I would just draw a line orthogonal to the one already created . Once the third control point is set the line is fixed and only the length of the line can be changed.
    • KK: likes it that way
    • KP: Discuss with KK for possible solution

ROI

• TestImageActorContourWidget
  • KP: the line is drawn once the control point is set. Instead, the line should always follow the cursor - once the MB is pressed the line is fixed up to this control point
  • KK: we should create another representation then. The issue with this is that it will become slow. The contour widget as you know works with several interpolators. Some interpolators are fast, some are slow (such as livewiring etc).. The interpolation is invoked only when a new control point is pressed. Having these be computed on the fly as you move the mouse around, will be slow, depending on the interpolator.
  • KP: missing measurement with length of line
    • KK: Will add option
  • KP: and region of interest (for closed lines)
    • KK: What do you mean by measurement of a "region of interest" ? Area ?
    • KP: a closed curve defines an area - this area I call region of interest
  • KP: do not know how to delete control points
    • KK: do not know how to delete control points
    • KP: Provide Balloon with help text or menu with delete option when one specific MB is pressed. We should try avoiding key combination as they need explanation
• vtkContourWidget (TestContourWidget2):
  • KP: could not set control points as the control points are predefined. Can that be changed ?

• TestDijkstraImageGeodesicPath :
  • KP: the line is drawn once the control point is set. Instead, the line should always follow the cursor - once the MB is pressed the line is fixed up to this control point
  • KP: missing measurement with length of line and region of interest (for closed lines)
  • KP: could not figure out how to modify location of control points
    • KK: You can simply mouse over the control point and move it around. Clicking anywhere on the line adds a new control point, which also can be moved around.
    • KP: Maybe it did not work for me bc it was too slow

• TestDijkstraGraphGeodesicPath:
  • KP: same as TestDijkstraImageGeodesicPath
  • KP: seg faulted on me
    • KK: Really. It works on the dashboard ?
    • KP: Try playing around with it a little bit - it seg faulted after a while

Widgets Completed Reviewed

• vtkBalloonWidget (TestBalloonWidget)  : ok
• TestHandleWidget:
  • KP: did not understand test
  • KK: Added comment

How do they do it?

Daniel worked with Scott and Curtis from the UPENN 3D & Advanced Imaging lab to analyze how commercial products realize annotation functionality. In general, Daniel was told that the functionality is not used a lot. Nevertheless, Curtis was able to intuitively figure out how to work with the functions.

SYNGO MULTI-MODALITY WORKPLACE (SIEMENS)

• Overall
  • Screenshot:
    • Annotation tools are under the tab 'Measure'
    • Product link: http://www.medical.siemens.com/webapp/wcs/stores/servlet/ProductDisplay?storeId=10001&langId=-1&catalogId=-1&catTree=100010,1008643,12757,15955&productId=174640
  • labels of annotations are independently movable without moving the annotation
  • workflow to annotate
    • click tab to select toolbox
    • select tool
    • click and drag
    • release
    • annotation widget appears
• Text annotations
  • always come with arrows
  • after overall workflow you can directly type the label
• ROI
  • different types: rectangle, circle, draw yourself
  • overall workflow (f.e. one click to place circle, then drag to change radius)
    • if multi edges, overall workflow and then click for new edge
  • widget displays:
    • MinMax (? maybe the min and max intensity values ?)
    • Area in sq cm
    • amount of pixels
    • mean intensity/standard deviation
• Measurement
  • overall workflow (click one point and hold, then drag around while the ruler already appears)
  • widget displays:
    • MinMax (? maybe the min and max intensity values ?)
    • Distance
• Freehand Measurement
  • measure along a spline
  • overall workflow and click starts a new edge
• Angle
  • draw two lines (workflow like measurement)
  • angle gets calculated where lines meet
• Shutter
  • select a rectangle, all information outside gets hidden
  • like temporary ROI

TeraRecon Aquarius Intuition Workstation

• Overall
  • Modules on the left side like in 3D Slicer
  • Screenshot:
  • Product Link: http://www.terarecon.com/products/aq_ws_prod.html
• Text annotations
  • always comes with arrow
  • accessible by right-click in the image (not via icon)
  • workflow: click, drag and release and type
  • right-click next to arrow or text shows menu to edit properties
• Measurement
  • just shows mm
  • Different types (each has a icon):
    • 1 line (click, drag and release)
    • N lines (click+release, drag, click+release again for new edge, doubleclick to exit)
    • Baseline mode
      • first line is the baseline (workflow as 1 line above), the following lines get labeled with the ratio to the baseline (f.e. 2x or 1.37x)
• Area measurement
  • has its own icons
  • different types: rectangle, circle, freehand
  • click, drag and release mechanism similar to SIEMENS
  • shows in sq cm
• Angle
  • draw two lines with click, drag and release
  • where the lines meet, compute angle
  • exactly like SIEMENS

GE Advantage Workstation

• Product link: http://www.instrumentarium.com/euen/advantage-workstation/index.html
• Screenshot:
• Text annotations
  • Preset mode (has its own icon)
    • lets the user to choose, edit and save text presets in the sidebar
    • select a preset and click into image, text appears
  • Normal mode (has its own icon)
    • type text in sidebar, then click to place
    • can be saved as preset from this GUI
• Measurements
  • 1-line measurements
  • Workflow: click+release, drag, click+release
    • note: that's the workflow we have in mind right now
• Spline
  • Workflow: click+release, drag, click+release for new edge
  • shows measurement directly
  • when closed, show area measurement in sq mm
• Wizard to annotate special use-cases (VesselIQ)
  • Link: http://www.gehealthcare.com/usen/ct/products/vesseliq_xpr_index.html
  • asks the user to click on various anatomic regions inside the image
  • places the label automatically after the wizard ran
  • reformats the image automatically to show f.e. an aorta
    • includes a diameter profile
  • WOW!

Conclusions

• very intuitive
• the workflow: click, drag and release to place a line (ruler..) seems to be the state-of-the-art and is what Curtis always tried at first
• angle measurement is performed in a different way than we had planned
• ability to move the labels next to annotations individually is very useful