Difference between revisions of "Project Week 25/Human-Computer Interaction under sterile conditions"
From NAMIC Wiki
(28 intermediate revisions by 4 users not shown) | |||
Line 1: | Line 1: | ||
− | |||
+ | __TOC__ | ||
Back to [[Project_Week_25#Projects|Projects List]] | Back to [[Project_Week_25#Projects|Projects List]] | ||
Line 6: | Line 6: | ||
*[http://isgwww.cs.uni-magdeburg.de/cas/ Christian Hansen] (University of Magdeburg, Germany) | *[http://isgwww.cs.uni-magdeburg.de/cas/ Christian Hansen] (University of Magdeburg, Germany) | ||
*[http://isgwww.cs.uni-magdeburg.de/cas/team.php Julian Hettig] (University of Magdeburg, Germany) | *[http://isgwww.cs.uni-magdeburg.de/cas/team.php Julian Hettig] (University of Magdeburg, Germany) | ||
+ | *[http://perk.cs.queensu.ca/users/lasso Andras Lasso] (PerkLab, Queen's) | ||
*[http://isgwww.cs.uni-magdeburg.de/cas/team.php Benjamin Hatscher] (University of Magdeburg, Germany) | *[http://isgwww.cs.uni-magdeburg.de/cas/team.php Benjamin Hatscher] (University of Magdeburg, Germany) | ||
*[http://www.researchgate.net/profile/David_Black11 David Black] (University of Bremen; Fraunhofer Institute for Medical Image Computing MEVIS, Bremen, Germany) | *[http://www.researchgate.net/profile/David_Black11 David Black] (University of Bremen; Fraunhofer Institute for Medical Image Computing MEVIS, Bremen, Germany) | ||
*[http://www.dkfz.de/en/mic/team/people/Marco_Nolden.html Marco Nolden] (German Cancer Research Center (DKFZ), Germany) | *[http://www.dkfz.de/en/mic/team/people/Marco_Nolden.html Marco Nolden] (German Cancer Research Center (DKFZ), Germany) | ||
*[http://juanruizalzola.com/about-juan/ Juan Ruiz Alzola] (University of Las Palmas de Gran Canaria, Spain) | *[http://juanruizalzola.com/about-juan/ Juan Ruiz Alzola] (University of Las Palmas de Gran Canaria, Spain) | ||
+ | *[http://www.imagenglab.com/newsite/salvatore_scaramuzzino/ Salvatore Scaramuzzino] ("Magna Graecia" University - ASL Vercelli, Italy) | ||
==Project Description== | ==Project Description== | ||
Line 19: | Line 21: | ||
| | | | ||
Human-Computer Interaction under Sterile Conditions. | Human-Computer Interaction under Sterile Conditions. | ||
− | * | + | * Review of state of the art (tochless interaction) |
− | * | + | * Development of new user interfaces to support surgical interterventions |
+ | |||
| | | | ||
<!-- Approach and Plan bullet points --> | <!-- Approach and Plan bullet points --> | ||
# Long, intensive conceptualization (2 days) of possibilities for gesture interaction and audio feedback for it | # Long, intensive conceptualization (2 days) of possibilities for gesture interaction and audio feedback for it | ||
## We will need moderation kit (paper, pencil, markers, etc) | ## We will need moderation kit (paper, pencil, markers, etc) | ||
− | # Prototypes of possible auditory feedback based on conceptualization | + | # Prototypes of possible auditory/visual feedback based on conceptualization |
## With OSC communication protocol, David will make quick, flexible sound methods | ## With OSC communication protocol, David will make quick, flexible sound methods | ||
+ | ## foot/eye interaction (?) | ||
+ | |||
| | | | ||
<!-- Progress and Next steps (fill out at the end of project week), please start each sentence in a new line. --> | <!-- Progress and Next steps (fill out at the end of project week), please start each sentence in a new line. --> | ||
+ | |||
+ | # Connected Slicer to OSC: added SlicerSoundControl extension (https://github.com/SlicerIGT/SlicerSoundControl), see below for necessary OSC message names | ||
+ | # Tested avoidance warning with experimental integration into breast lumpectomy navigation module (LumpNav): https://youtu.be/gSz8IHmogMo | ||
+ | # tried using gestures to zoom windows in and out | ||
|} | |} | ||
+ | |||
+ | ==Previous Work== | ||
+ | Demo Videos | ||
+ | * ablation needle or general tool placement, with U. Magdeburg and U. Hannover, [https://www.dropbox.com/s/35xzp51tk0rui6x/Singing.m4v?dl=0 “singing" method] and [https://www.dropbox.com/s/05et9abov10yvmk/AblationDemoVideo1.m4v?dl=0 synthetic sounds] | ||
+ | * [https://www.dropbox.com/s/ullaaab4gp9pucq/BWH_Dual-frequency-feedback.mp4?dl=0 ureteroscopy], with Harvard Surgical Planning Lab | ||
+ | * [https://www.dropbox.com/s/cdfc77ugjgg8o96/LIM%20Short.mp4?dl=0 sacral neuromodulation], with University Carlos III Madrid | ||
+ | * [https://www.dropbox.com/s/bn0u0g22yu14uvt/AD%20Resection%20short.m4v?dl=0 resection guidance], with Robert Bosch Hospital Stuttgart ([https://www.youtube.com/watch?v=gCg5nJSI2pY longer version with explanation]) | ||
+ | |||
+ | In addition, please see the following articles: | ||
+ | * [https://link.springer.com/article/10.1007/s11548-017-1537-1 Auditory feedback to support image-guided medical needle placement] | ||
+ | * [https://link.springer.com/article/10.1007/s11548-017-1547-z A Survey of auditory display in image-guided interventions] | ||
+ | * [https://www.researchgate.net/publication/233791033_Auditory_support_for_resection_guidance_in_navigated_liver_surgery Auditory support for resection guidance in navigated liver surgery.] | ||
==Illustrations== | ==Illustrations== | ||
− | https:// | + | |
+ | {| class="wikitable" | ||
+ | ! style="text-align: left; width:50%" | [[image:OpenSoundControl.png|400px]] OpenSoundControl module for configuring and testing OpenSoundControl server communication | ||
+ | ! style="text-align: left; width:50%" | [[image:SoundNavigation.png|400px]] SoundNavigation module for automatically generating OpenSoundControl messages for tool navigation, from tool and reference transform nodes | ||
+ | |- style="vertical-align:top;" | ||
+ | |} | ||
+ | |||
+ | <embedvideo service="youtube">https://youtu.be/gSz8IHmogMo</embedvideo> | ||
+ | |||
+ | ==SlicerSoundControl== | ||
+ | see https://github.com/SlicerIGT/SlicerSoundControl | ||
+ | |||
+ | === Environment for Testing Slicer Sound Control === | ||
+ | |||
+ | OSC Message names: | ||
+ | |||
+ | Base name: /SoundNav/Instrument/ | ||
+ | |||
+ | Translation: each fro 0 to 100 mm | ||
+ | * /TranslationX | ||
+ | * /TranslationY | ||
+ | * /TranslationZ | ||
+ | * /Distance (absolute distance) | ||
+ | |||
+ | Translation: each from 0 to 180 degrees | ||
+ | * /OrientationX | ||
+ | * /OrientationY | ||
+ | * /OrientationZ | ||
+ | * /Orientation (overall orientation) | ||
+ | |||
+ | |||
+ | GUI for test: | ||
+ | [[File:OSCTest GUI.png|500px|frameless|OSCTest GUI]] | ||
+ | |||
+ | === Environment for Sterile Gestures (planned, not yet implemented) === | ||
+ | * In/ out | ||
+ | ** --> "/InOut 0" out | ||
+ | ** --> "/InOut 1" in | ||
+ | * Edge distance (0 to 3 cm?) | ||
+ | ** --> "/EdgeDist x" where x is float between 0 and 3 | ||
+ | |||
+ | === Ambient === | ||
+ | * Overall acceleration | ||
+ | ** --> "/OverallAccel x" where x is float between 0 and 1 | ||
+ | * Background noise | ||
+ | * Absolute position (x/y, not depth) to show current selected window | ||
+ | ** --> "/AbsPolX x" where x is float between 0 and 1 | ||
+ | ** --> "/AbsPolY x" where x is float between 0 and 1 | ||
+ | |||
+ | === Gestures === | ||
+ | * select image | ||
+ | ** --> "/SelectGesture x" where x is gesture number | ||
+ | * zoom to left window | ||
+ | * zoom to right window | ||
+ | * put back into small viewer | ||
+ | |||
+ | ==Background== | ||
+ | [[File:Screen Shot 2017-06-26 at 12.14.57.png|500px|example HCI with freehand gestures for sound]] | ||
− | |||
<!-- Use this space for information that may help people better understand your project, like links to papers, source code, or data --> | <!-- Use this space for information that may help people better understand your project, like links to papers, source code, or data --> | ||
Example of previous related work using standard surgical gloves and OR compatible plastic draping confirmed to be compatible with sterile requirements at BWH AMIGO: | Example of previous related work using standard surgical gloves and OR compatible plastic draping confirmed to be compatible with sterile requirements at BWH AMIGO: | ||
+ | ] | ||
<embedvideo service="youtube">https://www.youtube.com/watch?v=zSqO2pUEodw</embedvideo> | <embedvideo service="youtube">https://www.youtube.com/watch?v=zSqO2pUEodw</embedvideo> | ||
+ | |||
+ | == References== | ||
+ | |||
+ | [http://isgwww.cs.uni-magdeburg.de/cas/pub/2017_IJCARS_MewesHensenWackerHansen.pdf Mewes et al. (2017) Touchless Interaction with Software in Interventional Radiology and Surgery: A Systematic Literature Review] | ||
+ | |||
+ | [http://isgwww.cs.uni-magdeburg.de/cas/pub/2017_Hettig_JCARS.pdf Hettig et al. (2017) Comparison of Gesture and Conventional Interaction Techniques for Interventional Neuroradiology] |
Latest revision as of 13:23, 30 June 2017
Home < Project Week 25 < Human-Computer Interaction under sterile conditionsContents
Back to Projects List
Key Investigators
- Christian Hansen (University of Magdeburg, Germany)
- Julian Hettig (University of Magdeburg, Germany)
- Andras Lasso (PerkLab, Queen's)
- Benjamin Hatscher (University of Magdeburg, Germany)
- David Black (University of Bremen; Fraunhofer Institute for Medical Image Computing MEVIS, Bremen, Germany)
- Marco Nolden (German Cancer Research Center (DKFZ), Germany)
- Juan Ruiz Alzola (University of Las Palmas de Gran Canaria, Spain)
- Salvatore Scaramuzzino ("Magna Graecia" University - ASL Vercelli, Italy)
Project Description
Objective | Approach and Plan | Progress and Next Steps |
---|---|---|
Human-Computer Interaction under Sterile Conditions.
|
|
|
Previous Work
Demo Videos
- ablation needle or general tool placement, with U. Magdeburg and U. Hannover, “singing" method and synthetic sounds
- ureteroscopy, with Harvard Surgical Planning Lab
- sacral neuromodulation, with University Carlos III Madrid
- resection guidance, with Robert Bosch Hospital Stuttgart (longer version with explanation)
In addition, please see the following articles:
- Auditory feedback to support image-guided medical needle placement
- A Survey of auditory display in image-guided interventions
- Auditory support for resection guidance in navigated liver surgery.
Illustrations
SlicerSoundControl
see https://github.com/SlicerIGT/SlicerSoundControl
Environment for Testing Slicer Sound Control
OSC Message names:
Base name: /SoundNav/Instrument/
Translation: each fro 0 to 100 mm
- /TranslationX
- /TranslationY
- /TranslationZ
- /Distance (absolute distance)
Translation: each from 0 to 180 degrees
- /OrientationX
- /OrientationY
- /OrientationZ
- /Orientation (overall orientation)
Environment for Sterile Gestures (planned, not yet implemented)
- In/ out
- --> "/InOut 0" out
- --> "/InOut 1" in
- Edge distance (0 to 3 cm?)
- --> "/EdgeDist x" where x is float between 0 and 3
Ambient
- Overall acceleration
- --> "/OverallAccel x" where x is float between 0 and 1
- Background noise
- Absolute position (x/y, not depth) to show current selected window
- --> "/AbsPolX x" where x is float between 0 and 1
- --> "/AbsPolY x" where x is float between 0 and 1
Gestures
- select image
- --> "/SelectGesture x" where x is gesture number
- zoom to left window
- zoom to right window
- put back into small viewer
Background
Example of previous related work using standard surgical gloves and OR compatible plastic draping confirmed to be compatible with sterile requirements at BWH AMIGO:
]