2016 Winter Project Week/Projects/TrackedUltrasoundStandardization

• 

  Projects List

Key Investigators

• Andras Lasso, Tamas Ungi (PerkLab, Queen's University)
• Christian Askeland, Ingerid Reinertsen, Ole Vegard Solberg (CustusX, IGT research, SINTEF)
• Simon Drouin (Mcgill University, Montreal, Canada)
• Junichi Tokuda (BWH)
• Steve Pieper (Isomics)
• Adam Rankin (VASST Laboratory, Western University, Canada)
• Thomas Kirchner, Janek Gröhl (MITK, DKFZ, Heidelberg, Germany)

Project Description

Objective

• Establish a common software platform for tracked ultrasound and image-guided interventions by converging existing IGT toolkits:
  • 3D Slicer, SlicerIGT extension
  • Plus toolkit
  • CustusX
  • IBIS
  • Echelon)
  • MITK-IGT

Approach, Plan

Progress

Preparation meetings

Christian's list of potential topics of collaboration

Enhance connection of CustusX client to PLUS server

Background: CustusX has used PLUS to connect to an Ultrasonix scanner with an attached tracking device from Ascension (Sonix GPS), as part of the EU RASimAs project. CustusX also has tracking support from the IGSTK toolkit, and own direct support for a number of video/US devices. We would like to add PLUS to this family, poosibly replacing some of our older components.

Topics:

• Tracking of BrainLab devices.
• Connection to BK medical scanners.
• Enhance transfer of US probe metadata. Might include extending the OpenIGTLink standard.
• Use of PLUS US reconstruction algorithms in CustusX as an alternative to CX internal algorithms, and enable comparison of algorithms.

Standardization of IGT software

Background:

• Emails, major players converge towards IGTLink. Several groups are using PLUS and/or Slicer, but also open-sourcing their own software and want to be compatible with each other.
• Data should be usable by several systems in order to ease comparison in e.g. studies.
• Our group need to communicate models and volumes with Slicer, and would like to use PLUS.

There are several areas where standardization would be beneficial:

• messaging between applications.
• file formats.
• algorithms/shared libraries.

It is important to build as much as possible on existing material and de facto standards.

Topics:

• Messaging: OpenIGTLink is the de facto standard for messaging. Is the standard complete? What do we need? US-probe data, polydata colors, volume transfer functions /LUTs.
• File Formats:  Models, Volumes, Video streams, US metadata, scenegraph. Some of these objects are already somewhat standardized, but not all.
• Algorithms: A library of IGT algorithms would be welcome. It might be included into an existing library (VTK, CTK, …) or emerge as a new one.

Meeting 2015-11-23

Participants: Andras Lasso, Tamas Ungi, Steve Pieper, Junichi Tokuda, Adam Rankin, Simon Drouin, Ole Vegard Solberg, Ingerid Reinertsen, Christian Askeland

Notes:

• Introduction of all present. Everyone seems to have a strong interest in IGT and systems integration.
• Junichi: Funded OpenIGTLink grant scope is well aligned with this standardization effort.
• Christian: project manager for CustusX, Plus is used for data acquisition, 3D Slicer used for pre/post operative processing
• Ingrid: research scientist, IGT and CustusX user, uses Slicer for pre-processing, segmentation, registration, post-processing; clinicians use Slicer for tumor segmentation, becoming a routine
• Simon: IBIS has been developed for 10 years, next step is integration with Plus, plan is to move features to 3D Slicer/SlicerIGT/Plus and use that as a platform (grant application submitted for getting funding)
• Walkthrough of (some of) Christian’s topics
  • CustusX - BrainLab connection through PLUS: BrainLab already supports OpenIGTLink, connection either through Plus or directly to the application should be straightforward.
  • CustusX - BK connection through PLUS: Partially implemented in PLUS. NOTE: Must buy additional license from BK to access continuous image streaming (GRAB ON/OFF command). Without that the maximum acquisition rate is limited to a few FPS. GRAB ON/OFF interface implementation is partially complete in Plus. If a BK machine is available that supports this command then implementation in Plus requires a couple of days.
  • 4D (3D+t) US acquisition support: Robarts has access to a private Philips API, SINTEF has access to a private GE API. Both have converter to OpenIGTLink (Philips converter is implemented in Plus, GE is external).
  • Ultrasound scanner two-way communication support in OpenIGTLink (data query, device control - depth, imaging mode): several of those present (Adam, Christian, ..) and others need this, and several slightly different implementations exist (Plus, CustusX, MUSiiC). Junichi agrees that this should be standardized into OpenIGTLink. We need a backward and forward compatible solution, possibly XML in STRING OpenIGTLink message. Needs further discussion
  • Shared file formats: All IGT data (pose tracking, 2D+t, 3D, 3D+t images) should be stored in a common format. Needs further discussion
    • Steve suggests DICOM. Some work already in progress on using DICOM as interchange format between BrainLab and Slicer (BrainLab already uses DICOM as its internal data representation). Most of the required structures are present, possibly excluding tracking positions. This can be (with time and work) added to the standard. General agreement on DICOM, but more discussion needed. Real-time streaming to DICOM file may be problematic (DICOM header has to be prepended to image data; if image data is several GB then it may take long time, not convenient for intra-operative use).
    • Can also discuss with Michael Onken/DCMTK when in Boston.
• General agreement that standardization and cooperation is needed, and that more discussions are needed before Boston.

TODO:

• Contact others if they were interested to join these efforts (MUSiiC, MITK-IGT, Nifty-IGT) - Andras
• Ask Tina about possibility for Amigo visit (BK ultrasound, BrainLab integration) - Andras
• Talk to BrainLab engineers about tracked ultrasound (and general IGT) information storage in DICOM format - Steve

Next meeting: December 7., 0900 EST (1500 Trondheim time)

Meeting 2015-12-07

Time: December 7., 0900 EST (1500 Trondheim/Heidelberg time) - click here to join the meeting through Google Hangout

Agenda:

• Intro to additional people who may join the meeting (DKFZ, JHU, Kitware, ...)
• Progress since last week:
  • Interest of other groups (MUSiiC / Johns Hopkins, MITK-IGT / DKFZ Heidelberg, Nifty-IGT / UCL London UK)- Andras
  • Amigo visit - Andras
  • BrainLab IGT storage format, DICOM - Steve

Topics of interests: 1. file storage, in-memory storage, and real-time transfer of various data structures; 2. common algorithms

• Tracking and imaging data stream:
  • Storage file format => What formats are used in CustusX, MITK, etc.?
    • MetaImage (mha/mhd), NRRD with custom fields
      • Very simple, flexible, human-readable file format
      • Lossless compression, special image types (4D, color, etc.) are supported
      • Basic support is available in all research software, C++ and Matlab libraries are available for reading/writing
      • Reading/writing is fully supported by Plus, 3DSlicer/SlicerIGT/Sequences
      • Limitations: only one image stream per file, no per-frame fields (simulated by adding per-file custom fields; standard libraries cannot associate them with frames); numbers stored as text, some precision is lost
    • DICOM: Ultrasound image IOD, Ultrasound multi-frame image IOD, Ehanced ultrasound volume IOD
      • use cases, overview
      • Standard exists for tracked 2D, 3D ultrasound sequences (frame of reference relative to probe, patient, other)
      • Physiological waveforms (cardiac, respiratory)
      • Physical units, real world values (speed, etc), color calibration
      • Custom per-frame fields can be specified (could be added to standard)
      • Multiple image regions (biplane, etc)
      • Limitation: no standard for storing navigation data (only probe and image position) - to be confirmed with David Clunie
  • Storage in memory for visualization and processing => What formats are used in CustusX, MITK, etc.?
    • Plus: rotating buffer for image frames (vtkImageData + custom fields), tool tracking data (pose, status); TrackedFrameList (list of image data with string field list)
    • 3D Slicer: sequences of data nodes (vtkMRMLScalarVolume, vtkTransform)
  • Real-time transfer: colors, volume transfer functions / LUTs?
• Calibration info: from/to coordinate system name, linear transform
  • File storage: In Plus: XML, CoordinateDefinitions element
  • In memory: vtkMatrix4x4, TransformRepository
  • Real-time transfer: application receives as TRANSFORM message, set by OpenIGTLink command
• Meshes
  • File storage: vtkPolyData, VTK file formats are OK but there is no coordinate system information; DICOM?
  • Memory: VTK class
  • Real-time transfer: is anything missing in OpenIGTLink?
• Ultrasound metadata: imaging parameters (depth, dynamic range, etc.), image geometry (clipping rectangle, clipping fan, image spacing, axis directions, transducer center position), freeze state, probe button press
  • File storage: currently in XML in Plus
  • In memory storage: XML data elements
  • Real-time transfer: MUSiiC custom OpenIGTLink messages have some info
• Scene:
  • File storage: Slicer uses MRML - alternatives?
  • Memory: MRML library
  • Real-time transfer?
• Common algorithms: => Create a new library (basically the same goals as IGSTK)? Include in CTK (it is already used by Slicer, MITK, etc)? As VTK or ITK remote modules? (both ITK and VTK have disadvantages, MITK is mainly ITK-based, Slicer and Plus is mainly VTK-based)
  • Pivot calibration (with robust point matching, error rejection, error metrics, etc.)
  • Landmark registration (with automatic landmark detection, etc.)
  • Volume reconstruction
  • Scan conversion, brightness conversion
  • Bone surface detection
  • Ultrasound simulation
  • US/MRI registration