View source for Project Week 25/Human-Computer Interaction under sterile conditions


__TOC__
Back to [[Project_Week_25#Projects|Projects List]]

==Key Investigators==
*[http://isgwww.cs.uni-magdeburg.de/cas/ Christian Hansen] (University of Magdeburg, Germany)
*[http://isgwww.cs.uni-magdeburg.de/cas/team.php Julian Hettig] (University of Magdeburg, Germany)
*[http://perk.cs.queensu.ca/users/lasso Andras Lasso] (PerkLab, Queen's)
*[http://isgwww.cs.uni-magdeburg.de/cas/team.php Benjamin Hatscher] (University of Magdeburg, Germany)
*[http://www.researchgate.net/profile/David_Black11 David Black] (University of Bremen; Fraunhofer Institute for Medical Image Computing MEVIS, Bremen, Germany)
*[http://www.dkfz.de/en/mic/team/people/Marco_Nolden.html Marco Nolden] (German Cancer Research Center (DKFZ), Germany)
*[http://juanruizalzola.com/about-juan/ Juan Ruiz Alzola] (University of Las Palmas de Gran Canaria, Spain)
*[http://www.imagenglab.com/newsite/salvatore_scaramuzzino/ Salvatore Scaramuzzino] ("Magna Graecia" University - ASL Vercelli, Italy)

==Project Description==
{| class="wikitable"
! style="text-align: left; width:27%" |   Objective
! style="text-align: left; width:27%" |   Approach and Plan
! style="text-align: left; width:27%" |   Progress and Next Steps
|- style="vertical-align:top;"
|
Human-Computer Interaction under Sterile Conditions.
* Review of state of the art (tochless interaction)
* Development of new user interfaces to support surgical interterventions

|
<!-- Approach and Plan bullet points -->
# Long, intensive conceptualization (2 days) of possibilities for gesture interaction and audio feedback for it
## We will need moderation kit (paper, pencil, markers, etc)
# Prototypes of possible auditory/visual feedback based on conceptualization
## With OSC communication protocol, David will make quick, flexible sound methods
## foot/eye interaction (?)

|
<!-- Progress and Next steps (fill out at the end of project week), please start each sentence in a new line. -->

# Connected Slicer to OSC: added SlicerSoundControl extension (https://github.com/SlicerIGT/SlicerSoundControl), see below for necessary OSC message names
# Tested avoidance warning with experimental integration into breast lumpectomy navigation module (LumpNav): https://youtu.be/gSz8IHmogMo
# tried using gestures to zoom windows in and out

|}

==Previous Work==
Demo Videos
* ablation needle or general tool placement, with U. Magdeburg and U. Hannover, [https://www.dropbox.com/s/35xzp51tk0rui6x/Singing.m4v?dl=0 “singing" method] and [https://www.dropbox.com/s/05et9abov10yvmk/AblationDemoVideo1.m4v?dl=0 synthetic sounds]
* [https://www.dropbox.com/s/ullaaab4gp9pucq/BWH_Dual-frequency-feedback.mp4?dl=0 ureteroscopy], with Harvard Surgical Planning Lab
* [https://www.dropbox.com/s/cdfc77ugjgg8o96/LIM%20Short.mp4?dl=0 sacral neuromodulation], with University Carlos III Madrid
* [https://www.dropbox.com/s/bn0u0g22yu14uvt/AD%20Resection%20short.m4v?dl=0 resection guidance], with Robert Bosch Hospital Stuttgart ([https://www.youtube.com/watch?v=gCg5nJSI2pY longer version with explanation])

In addition, please see the following articles:
* [https://link.springer.com/article/10.1007/s11548-017-1537-1 Auditory feedback to support image-guided medical needle placement]
* [https://link.springer.com/article/10.1007/s11548-017-1547-z A Survey of auditory display in image-guided interventions]
* [https://www.researchgate.net/publication/233791033_Auditory_support_for_resection_guidance_in_navigated_liver_surgery Auditory support for resection guidance in navigated liver surgery.]

==Illustrations==

{| class="wikitable"
! style="text-align: left; width:50%" |  [[image:OpenSoundControl.png|400px]] OpenSoundControl module for configuring and testing OpenSoundControl server communication
! style="text-align: left; width:50%" |  [[image:SoundNavigation.png|400px]] SoundNavigation module for automatically generating OpenSoundControl messages for tool navigation, from tool and reference transform nodes
|- style="vertical-align:top;"
|}

<embedvideo service="youtube">https://youtu.be/gSz8IHmogMo</embedvideo>

==SlicerSoundControl==
see https://github.com/SlicerIGT/SlicerSoundControl

=== Environment for Testing Slicer Sound Control ===

OSC Message names:

Base name: /SoundNav/Instrument/

Translation: each fro 0 to 100 mm
* /TranslationX
* /TranslationY
* /TranslationZ
* /Distance  (absolute distance)

Translation: each from 0 to 180 degrees
* /OrientationX
* /OrientationY
* /OrientationZ
* /Orientation (overall orientation)


GUI for test:
[[File:OSCTest GUI.png|500px|frameless|OSCTest GUI]]

=== Environment for Sterile Gestures (planned, not yet implemented) ===
* In/ out 
** --> "/InOut 0" out
** --> "/InOut 1" in
* Edge distance (0 to 3 cm?)
** --> "/EdgeDist x" where x is float between 0 and 3

=== Ambient ===
* Overall acceleration
** --> "/OverallAccel x" where x is float between 0 and 1
* Background noise
* Absolute position (x/y, not depth) to show current selected window
** --> "/AbsPolX x" where x is float between 0 and 1
** --> "/AbsPolY x" where x is float between 0 and 1

=== Gestures ===
* select image
** --> "/SelectGesture x" where x is gesture number
* zoom to left window
* zoom to right window
* put back into small viewer

==Background==
[[File:Screen Shot 2017-06-26 at 12.14.57.png|500px|example HCI with freehand gestures for sound]]


<!-- Use this space for information that may help people better understand your project, like links to papers, source code, or data -->
Example of previous related work using standard surgical gloves and OR compatible plastic draping confirmed to be compatible with sterile requirements at BWH AMIGO:
]
<embedvideo service="youtube">https://www.youtube.com/watch?v=zSqO2pUEodw</embedvideo>

== References==

[http://isgwww.cs.uni-magdeburg.de/cas/pub/2017_IJCARS_MewesHensenWackerHansen.pdf  Mewes et al. (2017) Touchless Interaction with Software in Interventional Radiology and Surgery: A Systematic Literature Review]

[http://isgwww.cs.uni-magdeburg.de/cas/pub/2017_Hettig_JCARS.pdf Hettig et al. (2017) Comparison  of  Gesture  and  Conventional  Interaction Techniques for Interventional Neuroradiology]