Projects:ARRA:SlicerEM:Developer:WorkflowManager

Back to SlicerEM:Developer page

Summary

• EMSegmenter has a fairly complicated workflow
• Need a mechanism to validate user input and transition appropriately between steps of the workflow
• Should be dependent on Qt only (and not VTK)
• Plan: use Qt's state machine implementation, with our own workflow manager on top (Qt Wizard doesn't quite work in harmony with the Qt state machine framework, and the UI itself can't be embedded in a pre-existing top-level)

Workflow classes

• Core workflow functionality (no widgets):
  • ctkWorkflowStep: a step in the workflow
  • ctkWorkflow: the workflow
  • ctkWorkflowTransition: a transition in the workflow
• Workflows associated with widgets:
  • ctkWorkflowWidgetStep: a step in the workflow with a user interface
  • ctkWorkflowWidget: a workflow with a user interface
  • ctkWorkflowAbstractPagedWidget: a workflow with a user interface with many pages (abstract class)
  • ctkWorkflowStackedWidget: a workflow with a user interface with many pages, based on a QStackedWidget
  • ctkWorkflowTabWidget: a workflow with a user interface with many pages, based on a QTabWidget

To define a new workflow step with a user interface

• Method 1: derive ctkWorkflowWidgetStep:
  • Implement validate(): evaluates user input and returns true if valid, false if not. Emits validateComplete(int) when finished.
  • Implement entryProcessing(): processing that is done when entering the step. Emits entryProcessingComplete() when finished.
  • Implement exitProcessing(): processing that is done when exiting the step. Emits exitProcessingComplete() when finished.
  • Either:
    • Implement populateStepWidgetsList(QList<QWidget*>& stepWidgetsList): add the step's widgets to the given list; by default they will be displayed vertically in the order in which they were added. Emits populateStepWidgetsListComplete() when finished.
    • Implement showUserInterface() and hideUserInterface(): for more control over the step's UI. Emits showUserInterfaceComplete() and hideUserInterfaceComplete(), respectively, when finished.

• Method 2: use signals and slots
  • Create an object that derives QObject
  • Implement slots that mimic the functionality of validate() / entryProcessing() / exitProcessing() / populateStepWidgetsList() / showUserInterface() / hideUserInterface(), and which emit a signal when completed
  • Use regular ctkWorkflowWidgetSteps in the workflow, and set their hasXXXCommands (ex. hasValidateCommand) to 1
  • Connect your object's signals/slots to those of the steps/workflow
  • When the workflow runs, it will emit the relevant signals to trigger your QObject's slots, which will then send indicate completion by the signals they send.

Examples

• current software is publicly available on github
• examples of how to create a custom workflow step:
  • Method 1: ctkExampleDerivedWorkflowWidgetStep.h and ctkExampleDerivedWorkflowWidgetStep.cpp: custom step created by deriving ctkWorkflowWidgetStep
  • Method 2: ctkExampleWorkflowWidgetStepUsingSignalsAndSlots.h and ctkExampleWorkflowWidgetStepUsingSignalsAndSlots.cpp: custom step created by implementing the step's functionality in a class derived from QObject, and communicating with the workflow using its signal-slot mechanism
• examples of how to create a workflow:
  • Method 1: ctkExampleUseOfWorkflowWidgetUsingDerivedSteps: builds a simple workflow using custom steps that derive ctkWorkflowWidgetStep
  • Method 2: ctkExampleUseOfWorkflowWidgetUsingSignalsAndSlots: builds a simple workflow using custom steps that communicate with the workflow using its signal-slot mechanism

Overview of state diagram underlying ctkWorkflow

• Processing step = handles user interaction, and supports additional processing (ex. image processing) as well
• Validation step = evaluates the success of the processing step
• Currently supported transitions:
  • Transition to the next step
  • Transition to the previous step
  • Transition to a "finish" step
    • Equivalent to doing "next, next, next, next..." until you hit the finish step: completes all entry/exit processing associated with each step encountered on the way to the finish step, without showing user interface.
    • Success relies on the suitability of the default processing, parameters, etc
    • "Blocking" steps, ex. those requiring user interaction, will prohibit success in transitioning to the finish step
    • On success: transitions back to the step where the attempt to go to the finish step was initiated, so that the user can perform additional customization from there if needed.
    • On failure: remains in the step whose validate() returned failure, and shows its user interface.

<graphviz>

digraph workflow { subgraph cluster_0 { style=filled; color=lightgrey; node [style=filled,color=white]; "Processing 1" -> "Validation 1" [label="ValidationTransition"];

               "Validation 1" -> "Processing 1" [label="ValidationFailedTransition"];

label = "ctkWorkflowStep* step1"; }

subgraph cluster_1 {

               style=filled;

color=lightgrey; node [style=filled,color=white]; "Processing 2" -> "Validation 2" [label="ValidationTransition"];

               "Validation 2" -> "Processing 2" [label="ValidationFailedTransition"];

label = "ctkWorkflowStep* step2"; }

       "Validation 1" -> "Processing 2" [label="TransitionToNextStep"];
       "Processing 2" -> "Processing 1" [label="TransitionToPreviousStep"];

} </graphviz>

Signal/slot mechanism to transition to the next step

• ex. user enters parameter values, and then clicks "Next" to go to the next step

<graphviz> digraph signalsAndSlotsNext {

       "SIGNAL: pushButton->clicked()" -> "SLOT: workflow->triggerValidationTransition()" -> "posts ValidationTransition event" -> "SIGNAL: validationState->entered()" -> "SLOT: workflow->attemptToGoToNextStep()" -> "step1->validate()" -> "SIGNAL: validationComplete(int)" -> "SLOT: workflow: evaluateValidationResults(int)" -> "workflow->triggerTransitionToNextStep()" -> "posts TransitionToNextStep event" -> "SIGNAL: transition->triggered()" -> "SLOT: workflow->performTransitionBetweenSteps()" -> "step1->onExit()" -> "step1->exitProcessing()" -> "SIGNAL: step1: exitProcessingComplete()" -> "SLOT: step1: evaluateExitProcessingResults()" -> "step1->hideUserInterface()" -> "SIGNAL: step1: hideUserInterfaceComplete()" -> "SLOT: step1->evaluateHideUserInterfaceResults()" -> "SIGNAL: step1->onExitComplete" -> "SLOT: workflow-> evaluateOnExitResults()" -> "step2->onEntry()" -> "step2->entryProcessing()" -> "SIGNAL: step2: entryProcessingComplete()" -> "SLOT: step2: evaluateEntryProcessingResults()" -> "step2->showUserInterface()" -> "SIGNAL: step2: showUserInterfaceComplete()" -> "SLOT: step2->evaluateShowUserInterfaceResults()" -> "SIGNAL: step2->onEntryComplete" -> "SLOT: workflow-> evaluateOnEntryResults()"

} </graphviz>

Signal/slot mechanism to trigger transition to the previous step

• ex. user clicks "Back" to go to the previous step

GUI implementation in ctkWorkflowWidget

• workflowWidget->addWidget(QWidget* widget): adds a widget to the clientArea
• workflowWidget->showWidget(QWidget* widget): shows the clientArea (i.e. for ctkWorkflowStackedWidget and ctkWorkflowTabWidget, ensures that the correct "page" is shown
• TODO:
  • plus qDialog as well
  • in Qt Designer - could have one .ui file for the entire stacked widget, or, for widgets representing complicated steps, could have a separate .ui

Previous workflow manager implementation in KWWidgets

• Please see the documentation here, as it is already well described

Uses:

• KWWidgets state machines - incorporates states (ex. user interaction within a workflow step, validation of the user input within a workflow step), transitions (between states), and inputs (pushed onto a queue to trigger transitions)
• KWWidgets wizard workflow - provides additional functionality to manage workflow using a state machine (ex. bundles pairs of user interaction and validation states into a workflow "step", handles a navigation stack of steps encountered along the way that triggers updates of widgets and/or dialogs)

Transition from KWWidgets state machines to Qt state machines

• Core functionality in KWWidgets state machines and equivalent (if available) in Qt (In Progress):

KWWidgets functionality	Present in Qt?
State machine involve states, inputs and transitions	State machine involve states, events/signals and transitions
States, inputs and transitions are instantiated separately and added to the state machine	Only states are added to state machine, transitions are added to state, events/signals are implied by transition
Transitions require an input, origin and destination	Transitions can be targetless (stay in place but execute something / emit signal),
State machine runs until there are no more inputs in its queue	State machine runs its own even loop automatically, can be started, stopped/resumed and will run until FinalState is reached or app is quit.
Can define custom inputs	Can define custom events and transitions (by deriving QEvent and QAbstractTransition)
States have enter/leave events & callbacks, reimplementable Enter()/Leave()	States have enter/exit signals, reimplementable onEnter()/on Exit()
Transitions have start/end events & callbacks, reimplementable Start()/End()	Transitions have triggered signal, reimplementable onTransition()
Ordering of state and transition events: transition - start origin state - leave destination state - enter transition - end	... different: origin state - exit transition - triggered destination state - enter
Clusters of states (one level)	A state can have a parent state allowing for arbitrary level grouping; a transition can be shared by many states by assigning it to the parent, but can be redefined by a child (Limitation: entry into the parent's initial state trigger's the initial state's onEntry() function, but not the parent state's onEntry() function)
Can save history of encountered transitions	No but a HistoryState can be automatically maintained to keep track of the last known child state of a grouping state when it is exited. It can be referred to directly to get back in the group and resume.
No	Object properties can be set/restored automatically when entering/leaving a state and animated during transition
No	FinalState class exists, state emits finished signal (parent emits, if final state is in group, allowing for blackblox behavior)
No	Transition can be guarded, i.e. the transition's input (a QEvent) can have a payload that can be tested by a custom transition (say a StringEvent that only triggers when the payload is "Hello")
No	Parallel states can be used to avoid combinatorial explosion. When a parallel state group is entered, all its child states will be simultaneously entered.

• If some of these components are not available in Qt's state machine framework, it means some more refactoring/restructuring work

Transition from KWWidgets workflow manager to a workflow manager using Qt's state machines

• The four KWWidget workflow wizard classes:
  • vtkKWWizardStep: a wizard step
  • vtkKWWizardWorkflow : a wizard workflow
  • vtkKWWizardWidget : a wizard widget, embedding UI (buttons) and a wizard workflow engine
  • vtkKWWizardDialog : a wizard dialog, embedding a wizard widget in a toplevel/dialog window.

Additional ideas and questions

• Image processing throughout - need to deal with failures in image processing that are unrelated to the GUI
• Would be a good idea to make the workflow manager as general as possible for CTK - ex. use for management of IGT workflows in Slicer, where you may be coordinating several modules (ex. calibration module -> registration module -> tracking module), and may need to notify other components of current state (ex. over OpenIGTLink) (ex see this IJ paper)
  • would need to make show/hide user interface functions more generic
  • Ex. make a workflow step more generic: instead of an "interaction" state and a "validation" state, could think about a "do stuff" state and a "validation" state, where "do stuff" can be user interaction, image processing, etc.
  • In validation states, could also have "borderline" success, where you may want to warn the user and allow them to choose whether to go ahead to the next step or to redo the current step with different inputs
• May even like to provide the option to save the MRML tree at the end of each step (to restore state if there is a crash, for example
• Automatic validation at each user input, to enable the "next" button once the user's input is valid
• Logging state transitions, user actions, inputs and results thoughout (logging states?)
• Design / output of state machines in a graphical format
• Replaying workflows
• Branching workflows
• Undo / redo and forward/back transitions
• Automatic transition to the last state ("GoToSelf" in KWWidgets)

API

ctkWorkflow

• addStartStep(ctkWorkflowStep* startStep)

• addTransition(ctkWorkflowStep* originStep, ctkWorkflowStep* destinationStep, const ctkWorkflow::TransitionType& transitionType, const QString& branchingTag)

• goForward() [slot]

• bool canGoForward()

• QList<ctkWorkflowStep*> forwardSteps(ctkWorkflowStep* step) const

• goBackward() [slot]

• bool canGoBackward()

• QList<ctkWorkflowStep*> backwardSteps(ctkWorkflowStep* step) const

• ctkWorkflowStep* previousStep()

• goTo(const QString& stepId) [slot]

• [Future] bool canGoTo(const QString& stepId)

• ctkWorkflowStep * currentStep()

• [Future] currentStepAboutToBeChanged(ctkWorkflowStep* currentStep, ctkWorkflowStep* nextStep) [signal]

• currentStepChanged(ctkWorkflowStep * currentStep) [signal]

TransitionDirectionnality

• Forward
• Backward
• BiDirectional

ctkWorkflowStep

• attemptToValidate() [virtual]

• onEntry() [virtual]

• onExit() [virtual]

Example

ctkworkflow w;
ctkWorkflowStep s1("select_image");
ctkWorkflowStep s2("input_new_size");
ctkWorkflowStep s3("display_resized_image");

w.addTransition(&s1, &s2);
w.addTransition(&s2, &s3);

Workflow layout

Branching workflows

API:

• addTransition(ctkWorkflowStep* originStep, ctkWorkflowStep* destinationStep, const ctkWorkflow::TransitionType& transitionType, const QString& branchId = "")
  • must give a branchId if destinationStep is the 2nd/3rd/4th, etc step added to originStep

                s3----s4    "simple"
               /
  s0----s1----s2
               \
                s5----s6    "advanced"

Associated code:

ctkWorkflow w;

ctkWorkflowStep s0;
...
ctkWorkflowStep s6;

w.setInitialStep(s0); [Ideally - should be optional]

w.addTransition(s0, s1);
w.addTransition(s1, s2);
w.addTransition(s2, s3, ctkWorkflow::BiDirectionnal, "simple");
w.addTransition(s3, s4);
w.addTransition(s2, s5, ctkWorkflow::BiDirectionnal, "advanced");
w.addTransition(s5, s6);

Conditional branching:

[signal] void validationComplete(bool isValid, const QString& branchId)

• ctkWorkflow::validate() doesn't give branchId:
  • Single transition:
    • transition created without branchId: post transition
    • transition created with branchId: post transition

• • Multiple transitions:
    • (incorrect usage) transitions created with branchIds: use first transition that was added

• ctkWorkflow::validate() gives branchId:
  • Single transition:
    • transition created without branchId: post transition (workflow overrides step)
    • transition created with branchId: conditional transition based on match, if no match then stay in current step
  • Multiple transitions:
    • transitions created with branchIds: conditional transition based on match, if no match then stay in current step

Notes

ctkWorkflowWidgetStep ... 
TODO notes
- no need to return value in validate anymore
- do NOT implement ANY processing in show/hide UI - will majorly
screw up the "gotofinish" implementation
- when making onEntry/onExit: remember gotofinish
implementation, and that you might already be done and then going
through it again
- remember not to include blocking steps if you're going to implement
the gotofinish functionality
- remember to look at changeUserInterface when designing custom step
and implementing showuserinterface/hideuserinterface (handled for you
if you're using the defaults with populateStepWidgestLIst)

TODO

Functionality:
- ctkWorkflowButtonBoxWidget, with configurable presence in ctkWorkflowWidget and ctkWorkflowWidgetStep
- branching workflow (validate returns branch tag or nextStep id)
- step based on QDialog? (may be unnecessary if we can put a ctkWorkflowWidget into a dialog box?)
- integrate with Qt designer

API:
- showUserInterface without call to superclass
- use ctkLogger
- look at names in ctkWorkflowTransition enum, and eliminate unnecessary ones
- ctkWorkflow::removeTransition() ???
- split ctkWorkflowTest1 into six tests (one per run of runWorkflowWidgetTest at the end of the file)
- ensure virtual functions make sense
- flags in constructor instead of in separate set/get
- ctkWorkflowWidget: don't need public setTitle/setDescription/setErrorText because they are pulled from the current step
- ctkWorkflow::goForward(bool) - goForward without needing to call validate()
- comments/documentation
- move implementation into Pimpl when possible