Difference between revisions of "Project Week 25/Human-Computer Interaction under sterile conditions"

From NAMIC Wiki
Jump to: navigation, search
 
(18 intermediate revisions by 3 users not shown)
Line 1: Line 1:
−
__NOTOC__
 
  
 +
__TOC__
 
Back to [[Project_Week_25#Projects|Projects List]]
 
Back to [[Project_Week_25#Projects|Projects List]]
  
Line 6: Line 6:
 
*[http://isgwww.cs.uni-magdeburg.de/cas/ Christian Hansen] (University of Magdeburg, Germany)
 
*[http://isgwww.cs.uni-magdeburg.de/cas/ Christian Hansen] (University of Magdeburg, Germany)
 
*[http://isgwww.cs.uni-magdeburg.de/cas/team.php Julian Hettig] (University of Magdeburg, Germany)
 
*[http://isgwww.cs.uni-magdeburg.de/cas/team.php Julian Hettig] (University of Magdeburg, Germany)
 +
*[http://perk.cs.queensu.ca/users/lasso Andras Lasso] (PerkLab, Queen's)
 
*[http://isgwww.cs.uni-magdeburg.de/cas/team.php Benjamin Hatscher] (University of Magdeburg, Germany)
 
*[http://isgwww.cs.uni-magdeburg.de/cas/team.php Benjamin Hatscher] (University of Magdeburg, Germany)
 
*[http://www.researchgate.net/profile/David_Black11 David Black] (University of Bremen; Fraunhofer Institute for Medical Image Computing MEVIS, Bremen, Germany)
 
*[http://www.researchgate.net/profile/David_Black11 David Black] (University of Bremen; Fraunhofer Institute for Medical Image Computing MEVIS, Bremen, Germany)
 
*[http://www.dkfz.de/en/mic/team/people/Marco_Nolden.html Marco Nolden] (German Cancer Research Center (DKFZ), Germany)
 
*[http://www.dkfz.de/en/mic/team/people/Marco_Nolden.html Marco Nolden] (German Cancer Research Center (DKFZ), Germany)
 
*[http://juanruizalzola.com/about-juan/ Juan Ruiz Alzola] (University of Las Palmas de Gran Canaria, Spain)
 
*[http://juanruizalzola.com/about-juan/ Juan Ruiz Alzola] (University of Las Palmas de Gran Canaria, Spain)
 +
*[http://www.imagenglab.com/newsite/salvatore_scaramuzzino/ Salvatore Scaramuzzino] ("Magna Graecia" University - ASL Vercelli, Italy)
  
 
==Project Description==
 
==Project Description==
Line 29: Line 31:
 
## With OSC communication protocol, David will make quick, flexible sound methods
 
## With OSC communication protocol, David will make quick, flexible sound methods
 
## foot/eye interaction (?)
 
## foot/eye interaction (?)
−
# [[File:OSC Test Input.pd.zip|thumb|Test incoming OSC messages using PureData]]
 
  
 
|
 
|
 
<!-- Progress and Next steps (fill out at the end of project week), please start each sentence in a new line. -->
 
<!-- Progress and Next steps (fill out at the end of project week), please start each sentence in a new line. -->
  
 +
# Connected Slicer to OSC: added SlicerSoundControl extension (https://github.com/SlicerIGT/SlicerSoundControl), see below for necessary OSC message names
 +
# Tested avoidance warning with experimental integration into breast lumpectomy navigation module (LumpNav): https://youtu.be/gSz8IHmogMo
 +
# tried using gestures to zoom windows in and out
 +
 +
|}
 +
 +
==Previous Work==
 +
Demo Videos
 +
* ablation needle or general tool placement, with U. Magdeburg and U. Hannover, [https://www.dropbox.com/s/35xzp51tk0rui6x/Singing.m4v?dl=0 “singing" method] and [https://www.dropbox.com/s/05et9abov10yvmk/AblationDemoVideo1.m4v?dl=0 synthetic sounds]
 +
* [https://www.dropbox.com/s/ullaaab4gp9pucq/BWH_Dual-frequency-feedback.mp4?dl=0 ureteroscopy], with Harvard Surgical Planning Lab
 +
* [https://www.dropbox.com/s/cdfc77ugjgg8o96/LIM%20Short.mp4?dl=0 sacral neuromodulation], with University Carlos III Madrid
 +
* [https://www.dropbox.com/s/bn0u0g22yu14uvt/AD%20Resection%20short.m4v?dl=0 resection guidance], with Robert Bosch Hospital Stuttgart ([https://www.youtube.com/watch?v=gCg5nJSI2pY longer version with explanation])
 +
 +
In addition, please see the following articles:
 +
* [https://link.springer.com/article/10.1007/s11548-017-1537-1 Auditory feedback to support image-guided medical needle placement]
 +
* [https://link.springer.com/article/10.1007/s11548-017-1547-z A Survey of auditory display in image-guided interventions]
 +
* [https://www.researchgate.net/publication/233791033_Auditory_support_for_resection_guidance_in_navigated_liver_surgery Auditory support for resection guidance in navigated liver surgery.]
 +
 +
==Illustrations==
 +
 +
{| class="wikitable"
 +
! style="text-align: left; width:50%" |  [[image:OpenSoundControl.png|400px]] OpenSoundControl module for configuring and testing OpenSoundControl server communication
 +
! style="text-align: left; width:50%" |  [[image:SoundNavigation.png|400px]] SoundNavigation module for automatically generating OpenSoundControl messages for tool navigation, from tool and reference transform nodes
 +
|- style="vertical-align:top;"
 
|}
 
|}
 +
 +
<embedvideo service="youtube">https://youtu.be/gSz8IHmogMo</embedvideo>
 +
 +
==SlicerSoundControl==
 +
see https://github.com/SlicerIGT/SlicerSoundControl
 +
 +
=== Environment for Testing Slicer Sound Control ===
 +
 +
OSC Message names:
 +
 +
Base name: /SoundNav/Instrument/
 +
 +
Translation: each fro 0 to 100 mm
 +
* /TranslationX
 +
* /TranslationY
 +
* /TranslationZ
 +
* /Distance  (absolute distance)
 +
 +
Translation: each from 0 to 180 degrees
 +
* /OrientationX
 +
* /OrientationY
 +
* /OrientationZ
 +
* /Orientation (overall orientation)
 +
 +
 +
GUI for test:
 +
[[File:OSCTest GUI.png|500px|frameless|OSCTest GUI]]
 +
 +
=== Environment for Sterile Gestures (planned, not yet implemented) ===
 +
* In/ out
 +
** --> "/InOut 0" out
 +
** --> "/InOut 1" in
 +
* Edge distance (0 to 3 cm?)
 +
** --> "/EdgeDist x" where x is float between 0 and 3
 +
 +
=== Ambient ===
 +
* Overall acceleration
 +
** --> "/OverallAccel x" where x is float between 0 and 1
 +
* Background noise
 +
* Absolute position (x/y, not depth) to show current selected window
 +
** --> "/AbsPolX x" where x is float between 0 and 1
 +
** --> "/AbsPolY x" where x is float between 0 and 1
 +
 +
=== Gestures ===
 +
* select image
 +
** --> "/SelectGesture x" where x is gesture number
 +
* zoom to left window
 +
* zoom to right window
 +
* put back into small viewer
  
 
==Background==
 
==Background==
−
[[File:Screen Shot 2017-06-26 at 12.14.57.png|400px|example HCI with freehand gestures for sound]]
+
[[File:Screen Shot 2017-06-26 at 12.14.57.png|500px|example HCI with freehand gestures for sound]]
  
  

Latest revision as of 13:23, 30 June 2017

Home < Project Week 25 < Human-Computer Interaction under sterile conditions

Back to Projects List

Key Investigators

Project Description

Objective Approach and Plan Progress and Next Steps

Human-Computer Interaction under Sterile Conditions.

  • Review of state of the art (tochless interaction)
  • Development of new user interfaces to support surgical interterventions
  1. Long, intensive conceptualization (2 days) of possibilities for gesture interaction and audio feedback for it
    1. We will need moderation kit (paper, pencil, markers, etc)
  2. Prototypes of possible auditory/visual feedback based on conceptualization
    1. With OSC communication protocol, David will make quick, flexible sound methods
    2. foot/eye interaction (?)
  1. Connected Slicer to OSC: added SlicerSoundControl extension (https://github.com/SlicerIGT/SlicerSoundControl), see below for necessary OSC message names
  2. Tested avoidance warning with experimental integration into breast lumpectomy navigation module (LumpNav): https://youtu.be/gSz8IHmogMo
  3. tried using gestures to zoom windows in and out

Previous Work

Demo Videos

In addition, please see the following articles:

Illustrations

OpenSoundControl.png OpenSoundControl module for configuring and testing OpenSoundControl server communication SoundNavigation.png SoundNavigation module for automatically generating OpenSoundControl messages for tool navigation, from tool and reference transform nodes

SlicerSoundControl

see https://github.com/SlicerIGT/SlicerSoundControl

Environment for Testing Slicer Sound Control

OSC Message names:

Base name: /SoundNav/Instrument/

Translation: each fro 0 to 100 mm

  • /TranslationX
  • /TranslationY
  • /TranslationZ
  • /Distance (absolute distance)

Translation: each from 0 to 180 degrees

  • /OrientationX
  • /OrientationY
  • /OrientationZ
  • /Orientation (overall orientation)


GUI for test: OSCTest GUI

Environment for Sterile Gestures (planned, not yet implemented)

  • In/ out
    • --> "/InOut 0" out
    • --> "/InOut 1" in
  • Edge distance (0 to 3 cm?)
    • --> "/EdgeDist x" where x is float between 0 and 3

Ambient

  • Overall acceleration
    • --> "/OverallAccel x" where x is float between 0 and 1
  • Background noise
  • Absolute position (x/y, not depth) to show current selected window
    • --> "/AbsPolX x" where x is float between 0 and 1
    • --> "/AbsPolY x" where x is float between 0 and 1

Gestures

  • select image
    • --> "/SelectGesture x" where x is gesture number
  • zoom to left window
  • zoom to right window
  • put back into small viewer

Background

example HCI with freehand gestures for sound


Example of previous related work using standard surgical gloves and OR compatible plastic draping confirmed to be compatible with sterile requirements at BWH AMIGO: ]

References

Mewes et al. (2017) Touchless Interaction with Software in Interventional Radiology and Surgery: A Systematic Literature Review

Hettig et al. (2017) Comparison of Gesture and Conventional Interaction Techniques for Interventional Neuroradiology

Retrieved from "https://www.na-mic.org/w/index.php?title=Project_Week_25/Human-Computer_Interaction_under_sterile_conditions&oldid=97018"