Projects:ARRA:SlicerAM:vtkWidgets

Back to Slicer Report home page

Design Notes (Nicole & WJP)

APPEARANCE wish list

Widgets and handles should clearly distinguishable and consistently presented. Widgets and handles can be in the following states:

• locked/unlocked (so we know why a widget appears enabled but isn’t pickable.)
• active/inactive (so we know when a widget/handle has been picked and can be manipulated)
• created/nascent (so we know when a widget is not fully specified yet in click-to-place mode)
• There should be widget / handle LOCK color/treatment.

There should be a near-widget near-handle active color/treatment that shows when mouse is hovering over a pickable or picked thing. There should be a nascent color/treatment that shows that a widget is not yet fully defined. These colors/treatments should be consistent in 2D and 3D.

BEHAVIOR wish list

Mousing over a widget should highlight the widget and highlight all handles. User can grab widget and translate it to new location. Mousing over a handle should highlight the individual individual handle. User can grab handle and move it to a new location. Hmmm: how to instrument this at widget-level? --NearWidget and NearHandle events or just booleans on the widgets & handles? Widget creation behavior:

• Default (press button and widget appears at origin)
• Click to globally place (click in viewer and widget appears at that location)
• Click-rubberband to define extent of widget (like endpoints of a ruler or angle widget).
• Click-without-rubberband?
• Create from fiducials? (control-m for ruler implemented in Measurements module)

Hover over widget: cursor changes and color changes,

• Once hovering, right-click raises options menu including
  • Constrain to surface (lock/unlock)
  • Constrain to location (lock/unlock)
  • Set Representation (glyph shape)
  • Set properties (color, visibility)
• Left click allows Pick & Manipulate on handle or widget

Widget editing: for some widgets we may want to change appearance while editing (like text widget, or text fields of a widget.) Ilustrations shown below. Please note any problems!!!

• Handling widgets whose specification needs to be interrupted -- like the vtkContourWidget. A storyboard may go like this:
  • user selects to create contour widget
  • slicer goes into persistent place mode.
  • user clicks to place four control points.
  • user double-clicks to end contour widget definition.
  • slicer switches back into transform view mode.
  • user repositions the view
  • user selects the entire contour widget.
  • user right clicks for options menu and selects "Add more points"
  • slicer goes into persistent place mode again
  • user clicks to place more control points.
  • etc.

Widget behavior

Handle behavior

Locking a Handle

EVENTS WE NEED (to manage cursor and mouse-modes)

• Handle hover event
• Widget hover event
• Widget created event
• Editing event (we need for widgets with text that can be interactively changed)
• Left-mouse-down
• Left-mouse-release
• Right-mouse-down
• Middle-mouse-down
• Middle-mouse-release
• Key-press
• Mouse-Wheel

This includes events that widgets observe and emit...

LINK to MOUSEMODES

hover and NearHandle:

• Widget and handles highlight.
• Application switches to Transient Pick mode;
• can left-button-press to pick and move handle;
• right-button-press options menu configures individual handle.
• on left-button-release, mouse-mode switches back to default.

hover and NearWidget:

• Widget and handles highlight.
• Application switches to Transient Pick mode;
• can left-button-press to pick and move entire widget;
• right-button-press options menu configures widget and all handles.
• on left-button-release, mouse-mode switches back to default.

Question: Once this works properly, does application NEED toolbar icon for pick mode at all? Maybe it becomes implicit. Place and TransformView are the only two modes needed in toolbar? Yes, we probably need all mouse modes.

API wish list

1. All vtkHandleRepresentations within a widget should have independent accessor methods as well as methods to get/set properties on all. Need accessor methods on properties including:

• color
• text
• process events on/off (for locking things)
• methods for turning on/off zoom-level dependent text scale
• interaction state
  • locked=true/false
  • selected=true/false (right-click could select and configure, e.g. turn lock on/off)
  • mousehover=true/false
  • updating=true/false (left-click could select and drag, causing updates in position
• all handles should have a selected property.
• all handles should have a lock/unlock -- allows handle hover event to be processed, but doesn’t allow handle to be picked and moved.
• all widgets and handles should have adjustable NearHandle and NearWidget heuristics.

2. We need accessor methods to return each widget or handle subclass type -- (to avoid need to SafeDownCast to particular subclass). For instance: widget->GetRepresentation() always returns superclass, and handle->GetRepresentation() always returns superclass. Need convenience methods to avoid needing to do: vtkAbstractPolgonalHandleRepresentation *polyhandleRep = vtkAbstractPolygonalHandleRepresentation::SafeDownCast ( handle->GetRepresentation() );

3. For any vtkHandleWidget, we want to be able to set a new vtkAbstractPolygonalHandleRepresentation or one of its subclasses -- rather then having these be automatically cloned. (Cloning prevents setting different values for properties on different handles.)

4. If multiple handles exist on a single widget, vtkAbstractWidget->ManageCursor() should be respected by all of them -- they should not attempt to override each other! Maybe a right-click should allow us to lock/unlock a handle when we are hovering.

5. We need to be able to distinguish between NearHandle and NearWidget events so Application can provide the right user feedback on what’s pickable (whole widget, or just handle?)

6. Wherever possible, all widgets should have both a 3D and a 2D implementation, and their class naming convention should reflect 2D or 3D appropriately.

PRIORITY WIDGETS

Bold widgets below are being used by Slicer3.6. Other widgets included are widgets that we know we'd like to use in Slicer4.

• vtkAffineWidget (need 3D version: a backport from vtkSlicerBoxWidget2 has required logic)
• vtkAngleWidget
• vtkBalloonWidget (and other special classes of vtkHoverWidget)
• vtkBidimensionalWidget
• vtkBoxWidget (lower priority)
• vtkCaptionWidget
• vtkContourWidget
• vtkDistanceWidget (need to have 3D version of this!)
• vtkHandleWidget
• vtkImplicitPlaneWidget2
• vtkLineWidget
• vtkLineWidget2
• vtkTensorProbeWidget
• vtkTextWidget
• vtkSeedWidget
• vtkSplineWidget2
• vtkSplineWidget
• Expand ROI widget (RON):
  • The most general case is a boolean of multiple prisms (like in the AZE demo on Thursday late afternoon).
  • brick, skewed brick, cone, ellipsoid as special cases.
  • applicable to volumes (with resampling), surface models (properly closing cut surfaces), tensor fields (with resampling).
• cutting planes (any kind of cuts, maybe reformat widget is the right one, with clip outside option)

API Plan

To probe gaps in the API across widgets, plan is to set up testing macros for all of the following vtk classes, from the superclass down to the specific subclass implementation level.

Question: Who does this? Ideally, these tests should be added to VTK testing (not Slicer).

• Macro testing example by Nicole in Slicer3/TestingMacros.h
• used in /Libs/MRML/Testing/(fiducials tests).
• write test, add to CMakeLists.txt and
• run using ctest testname -vv.

Classes to include:

• vtkAbstractWidget-- see link
• vtkWidgetRepresentation and subclasses-- see link
• vtkHandleRepresentation and subclasses-- see link

Review Widgets

Meeting at Summer Project Week 2010

• Attendees: Kilian, Wendy, Nicole and Daniel

VTK Related Issues

If you add any new items in the summary below please add your name to them (KP)

Widget appearance & behavior design goals. All widgets (in 3D and in 2D) should:

• Expose a "hover" event that the application can observe.
• Expose a "pick" event that the application can observe.
• Expose a "start interaction" event (same as above?)...
• Expose an "editing" event (for things like text widgets).
• Expose a "release" event...
• Expose an "end interaction" event (same as above?)...

• Consistently display what parts of them are selectable.
• Consistently display what parts of them are NOT selectable.
• Consistently display what parts of them are selected.
• Consistently display when they are being manipulated.
• Consistently display when they have been released.
• Have a consistent mapping between their presentation in 3D and 2D viewers.

General VTK Issues

• Improve Consistency across Widgets:
  • vtk widgets should map both into 3d viewers and slice viewers with consistent appearance and behavior
    • same API for both viewer modes in VTK
  • same API/variables across widgets
  • controls for glyphs and fonts for all widgets (use current fiducial module as a start point). Meaningful defaults default settings are the same across widgets
• Create/improve Documentation for each widget
• tooltips for every widget
• improve graphical design of widgets to be more pleasing for the eye (KP)
• Placement behavior:
  • rubberband between clicks
  • click points and hotkey, e.g. set two points in 2D viewer , press ctrl M (for measurement) , and then a ruler is placed between the points
  • parameterized place (f.e. from MRML or GUI panel) -> default setting is one case of parameters across
• Manipulation behavior:
  • highlighting when something is selectable
  • user needs to know when something is selected and _which part_ of the widget is selected
    • there is a selected property for most of the widgets
  • left button would move the widget, right button opens context menu for additional configuration (color,...)
• intuitive initialization would be very important (Ron)

Widget Specific Issues

See also "VTK Widget related action items" further down

• Angle widget:
  • has crosses instead of spheres, should be changed for consistency
  • there are a lot of bugs in the widget
• Ruler widget:
  • when line is resized, text size does not change
  • enable ticks similar to these examples
    • DH: The examples are 3D Slicer scenes? Maybe we should contact the creator and get the code?
• Textbox:
  • change text by right click or double click
  • move on Mac OS not possible because lack of third mouse button
  • widget does not move when user zooms or rotates the cube. bug or feature?

Missing Wigets

• Expand ROI widget (RON):
  • The most general case is a boolean of multiple prisms (like in the AZE demo on Thursday late afternoon).
  • brick, skewed brick, cone, ellipsoid as special cases.
  • applicable to volumes (with resampling), surface models (properly closing cut surfaces), tensor fields (with resampling).
• draw with auto-complete (maybe contour widget is the right one), spline open and close
• hold down button and draw
• cutting planes (any kind of cuts, maybe reformat widget is the right one, with clip outside option)
• create Textbox with transparent background color as shown here (KP)
• Bi-Dimensional Measurements: Provide placing mode where the second ruler can only be set perpendicular to the first ruler (KP)

Slicer Related Issues

• probably through the Slicer MRML tree, displayNodes should be the same (one manager handles 2D and 3D)
  • KP: I do not understand this - it was listed below "same API for both modes in VTK"
  • DH: I think, it means that the 2D and 3D versions of widgets are accessible through the same API by one manager inside the MRML tree. Does this make sense?

Follow Up

• we need a review session in a smaller group (maybe Nicole, Daniel with JC or who is responsible)
• we need to come up with specific tests and examples for Kitware
• Wendy will create a document specifying the general behavior of widgets (Deadline July 2)
• All participants of the meeting will work on this summary until July 2nd
• We will send the list to Will on July 7th

VTK Widget related action items

General

WS: revisit the picking behavior in VTK so that the default approach is to use an accelerated method. NA: is updating Slicer so that Slicer compiles with the newest VTK cvs head and can test new widgets

Annotations

• vtkCaptionWidget (TestCaptionWidget) :
  • KP: The widget is buggy - the cursor icon seems to suggest a different functionality then what you can do to move the / modify the location of the text . I was only able to move the text by clicking on the lower bar of the text box. Furthermore in TestCaptionWidget.cxx I had to uncomment widget->On(); - so I believe by default it is never tested
  • KK: This has just been fixed. You should now be able to resize the caption etc. VTK/Widgets/vtkCaptionRepresentation.cxx new revision: 1.8
  • KP: Check out change
• vtkTextWidget (TestTextWidget:):
  • KP: Font size customization ?
  • KK: Currently the font size scales to the entire text box size. Check with KP on what he needs

Measurements

• vtkSeedWidget (TestSeedWidget):
  • KP: Custom measurement annotations needed, such as the intensity value
  • KK: I'll put in functionality for a callback mechanism that allows you to optionally update a label each time the seed is moved,so that a probe class can be attached to the seed widget.
• TestPolygonalHandleRepresentations:
  • KP: I will have to update VTK and Test it again
• vtkkPolygonalSurfacePointPlacer (TestPolygonalRepresentationHandleWidget):
  • KP: A little bit Slow
  • KK: the handle is constrained to the surface of the polygonal object. This involves a pick each time the handle is moved, and for a surface with tens of thousands of cells can get slow. An unconstrained handle should be fast.
• vtkDistanceWidget (TestDistance Widget) (2D distance widget):
  • KP: Instead of displaying measurement along line display it outside with dotted line such as in caption widget
  • KP: Measurement should be moved anywhere in the image without obstructing important anatomy
  • KP: Distance text font customization (too small to read)
  • KK: working on it
• vtkLineWidget2 (TestLineWidget2) (3D distance widget):
  • KP: Hard to initialize control points in 3D (difficult to judge behavior)
  • KK: working on it
• vtkAngleWidget
  • vtkAngleRepresentation2D (TestAngleWidget2D)
    • KP: Instead of displaying measurement along line display it outside with dotted line such as in caption widget
    • KP: text font customization (too small to read)
    • KP: Half circle scale customization to create less image obstruction
    • KK: working on it
  • vtkAngleRepresentation3D (TestAngleWidget2D)
    • KP: Instead of displaying measurement along line display it outside with dotted line such as in caption widget
    • KP: Widget measurement disappeared as it was placed inside the sphere (Check with KP. This is indeed the intended behaviour)
    • KP: Half circle scale customization to create less image obstruction
    • KK: working on it
• vtkBiDimensionalWidget (TestBiDimensionalWidget)
  • KP: Instead of displaying measurement along line display it outside with dotted line such as in caption widget
    • KK: working on it
  • KP: It is strange that the fourth point is set automatically but can be modified later. Once the second control point is set I would just draw a line orthogonal to the one already created . Once the third control point is set the line is fixed and only the length of the line can be changed.
    • KK: likes it that way
    • KP: Discuss with KK for possible solution

ROI

• TestImageActorContourWidget
  • KP: the line is drawn once the control point is set. Instead, the line should always follow the cursor - once the MB is pressed the line is fixed up to this control point
  • KK: we should create another representation then. The issue with this is that it will become slow. The contour widget as you know works with several interpolators. Some interpolators are fast, some are slow (such as livewiring etc).. The interpolation is invoked only when a new control point is pressed. Having these be computed on the fly as you move the mouse around, will be slow, depending on the interpolator.
  • KP: missing measurement with length of line
    • KK: Will add option
  • KP: and region of interest (for closed lines)
    • KK: What do you mean by measurement of a "region of interest" ? Area ?
    • KP: a closed curve defines an area - this area I call region of interest
  • KP: do not know how to delete control points
    • KK: do not know how to delete control points
    • KP: Provide Balloon with help text or menu with delete option when one specific MB is pressed. We should try avoiding key combination as they need explanation
• vtkContourWidget (TestContourWidget2):
  • KP: could not set control points as the control points are predefined. Can that be changed ?

• TestDijkstraImageGeodesicPath :
  • KP: the line is drawn once the control point is set. Instead, the line should always follow the cursor - once the MB is pressed the line is fixed up to this control point
  • KP: missing measurement with length of line and region of interest (for closed lines)
  • KP: could not figure out how to modify location of control points
    • KK: You can simply mouse over the control point and move it around. Clicking anywhere on the line adds a new control point, which also can be moved around.
    • KP: Maybe it did not work for me bc it was too slow

• TestDijkstraGraphGeodesicPath:
  • KP: same as TestDijkstraImageGeodesicPath
  • KP: seg faulted on me
    • KK: Really. It works on the dashboard ?
    • KP: Try playing around with it a little bit - it seg faulted after a while

Widgets Completed Reviewed

• vtkBalloonWidget (TestBalloonWidget)  : ok
• TestHandleWidget:
  • KP: did not understand test
  • KK: Added comment