2016 Winter Project Week/Projects/TrackedUltrasoundStandardization

• 

  Projects List

Key Investigators

• Andras Lasso, Tamas Ungi (PerkLab, Queen's University)
• Christian Askeland, Ingerid Reinertsen, Ole Vegard Solberg (CustusX, IGT research, SINTEF)
• Simon Drouin (Mcgill University, Montreal, Canada)
• Junichi Tokuda (BWH)
• Steve Pieper (Isomics)
• Adam Rankin (VASST Laboratory, Western University, Canada)
• Thomas Kirchner, Janek Gröhl (MITK, DKFZ, Heidelberg, Germany)

Project Description

Objective

• Establish a common software platform for tracked ultrasound and image-guided interventions by converging existing IGT toolkits:
  • 3D Slicer, SlicerIGT extension
  • Plus toolkit
  • CustusX
  • IBIS
  • Echelon
  • MITK-IGT

Approach, Plan

Progress

Preparation meetings

Christian's list of potential topics of collaboration

Enhance connection of CustusX client to PLUS server

Background: CustusX has used PLUS to connect to an Ultrasonix scanner with an attached tracking device from Ascension (Sonix GPS), as part of the EU RASimAs project. CustusX also has tracking support from the IGSTK toolkit, and own direct support for a number of video/US devices. We would like to add PLUS to this family, poosibly replacing some of our older components.

Topics:

• Tracking of BrainLab devices.
• Connection to BK medical scanners.
• Enhance transfer of US probe metadata. Might include extending the OpenIGTLink standard.
• Use of PLUS US reconstruction algorithms in CustusX as an alternative to CX internal algorithms, and enable comparison of algorithms.

Standardization of IGT software

Background:

• Emails, major players converge towards IGTLink. Several groups are using PLUS and/or Slicer, but also open-sourcing their own software and want to be compatible with each other.
• Data should be usable by several systems in order to ease comparison in e.g. studies.
• Our group need to communicate models and volumes with Slicer, and would like to use PLUS.

There are several areas where standardization would be beneficial:

• messaging between applications.
• file formats.
• algorithms/shared libraries.

It is important to build as much as possible on existing material and de facto standards.

Topics:

• Messaging: OpenIGTLink is the de facto standard for messaging. Is the standard complete? What do we need? US-probe data, polydata colors, volume transfer functions /LUTs.
• File Formats:  Models, Volumes, Video streams, US metadata, scenegraph. Some of these objects are already somewhat standardized, but not all.
• Algorithms: A library of IGT algorithms would be welcome. It might be included into an existing library (VTK, CTK, …) or emerge as a new one.

Meeting 2015-11-23

Participants: Andras Lasso, Tamas Ungi, Steve Pieper, Junichi Tokuda, Adam Rankin, Simon Drouin, Ole Vegard Solberg, Ingerid Reinertsen, Christian Askeland

Notes:

• Introduction of all present. Everyone seems to have a strong interest in IGT and systems integration.
• Junichi: Funded OpenIGTLink grant scope is well aligned with this standardization effort.
• Christian: project manager for CustusX, Plus is used for data acquisition, 3D Slicer used for pre/post operative processing
• Ingrid: research scientist, IGT and CustusX user, uses Slicer for pre-processing, segmentation, registration, post-processing; clinicians use Slicer for tumor segmentation, becoming a routine
• Simon: IBIS has been developed for 10 years, next step is integration with Plus, plan is to move features to 3D Slicer/SlicerIGT/Plus and use that as a platform (grant application submitted for getting funding)
• Walkthrough of (some of) Christian’s topics
  • CustusX - BrainLab connection through PLUS: BrainLab already supports OpenIGTLink, connection either through Plus or directly to the application should be straightforward.
  • CustusX - BK connection through PLUS: Partially implemented in PLUS. NOTE: Must buy additional license from BK to access continuous image streaming (GRAB ON/OFF command). Without that the maximum acquisition rate is limited to a few FPS. GRAB ON/OFF interface implementation is partially complete in Plus. If a BK machine is available that supports this command then implementation in Plus requires a couple of days.
  • 4D (3D+t) US acquisition support: Robarts has access to a private Philips API, SINTEF has access to a private GE API. Both have converter to OpenIGTLink (Philips converter is implemented in Plus, GE is external).
  • Ultrasound scanner two-way communication support in OpenIGTLink (data query, device control - depth, imaging mode): several of those present (Adam, Christian, ..) and others need this, and several slightly different implementations exist (Plus, CustusX, MUSiiC). Junichi agrees that this should be standardized into OpenIGTLink. We need a backward and forward compatible solution, possibly XML in STRING OpenIGTLink message. Needs further discussion
  • Shared file formats: All IGT data (pose tracking, 2D+t, 3D, 3D+t images) should be stored in a common format. Needs further discussion
    • Steve suggests DICOM. Some work already in progress on using DICOM as interchange format between BrainLab and Slicer (BrainLab already uses DICOM as its internal data representation). Most of the required structures are present, possibly excluding tracking positions. This can be (with time and work) added to the standard. General agreement on DICOM, but more discussion needed. Real-time streaming to DICOM file may be problematic (DICOM header has to be prepended to image data; if image data is several GB then it may take long time, not convenient for intra-operative use).
    • Can also discuss with Michael Onken/DCMTK when in Boston.
• General agreement that standardization and cooperation is needed, and that more discussions are needed before Boston.

TODO:

• Contact others if they were interested to join these efforts (MUSiiC, MITK-IGT, Nifty-IGT) - Andras
• Ask Tina about possibility for Amigo visit (BK ultrasound, BrainLab integration) - Andras
• Talk to BrainLab engineers about tracked ultrasound (and general IGT) information storage in DICOM format - Steve

Next meeting: December 7., 0900 EST (1500 Trondheim time)

Meeting 2015-12-07

Date: December 7., 0900 EST (1500 Trondheim/Heidelberg time)

Participants: Andras Lasso, Tamas Ungi, Steve Pieper, Junichi Tokuda, Adam Rankin, Simon Drouin, Ole Vegard Solberg, Christian Askeland, Alfred Franz, Thomas Kirchner, Janek Groehl

Topics of interests

• Tracking and imaging data stream (image includes distance maps, various color images, etc):
  • Storage file format
    • MetaImage (mha/mhd), NRRD with custom fields
      • Very simple, flexible, human-readable file format
      • Lossless compression, special image types (4D, color, etc.) are supported
      • Basic support is available in all research software, C++ and Matlab libraries are available for reading/writing
      • Reading/writing is fully supported by Plus, 3DSlicer/SlicerIGT/Sequences
      • Limitations: only one image stream per file, no per-frame fields (simulated by adding per-file custom fields; standard libraries cannot associate them with frames); numbers stored as text, some precision is lost
    • DICOM: Ultrasound image IOD, Ultrasound multi-frame image IOD, Ehanced ultrasound volume IOD
      • use cases, overview
      • Standard exists for tracked 2D, 3D ultrasound sequences (frame of reference relative to probe, patient, other)
      • Physiological waveforms (cardiac, respiratory)
      • Physical units, real world values (speed, etc), color calibration
      • Custom per-frame fields can be specified (could be added to standard)
      • Multiple image regions (biplane, etc)
      • Limitation: no standard for storing navigation data (only probe and image position) - to be confirmed with David Clunie
  • Storage in memory for visualization and processing
    • Plus: rotating buffer for image frames (vtkImageData + custom fields), tool tracking data (pose, status); TrackedFrameList (list of image data with string field list)
    • 3D Slicer: sequences of data nodes (vtkMRMLScalarVolume, vtkTransform)
  • Real-time transfer: colors, volume transfer functions / LUTs?
• Ultrasound metadata: imaging parameters (depth, dynamic range, etc.), image geometry (clipping rectangle, clipping fan, image spacing, axis directions, transducer center position), freeze state, probe
• Surgical microscope metada: focus, zoom
• Calibration info: from/to coordinate system name, linear transform
  • File storage: In Plus: XML, CoordinateDefinitions element
  • In memory: vtkMatrix4x4, TransformRepository
  • Real-time transfer: application receives as TRANSFORM message, set by OpenIGTLink command
• Meshes, point cloud
  • File storage: vtkPolyData, VTK file formats are OK but there is no coordinate system information; DICOM?
  • Memory: VTK class
  • Real-time transfer: is anything missing in OpenIGTLink?
• Dependency, data provenance
  • Data represenation for segmentation, keeping relationship to source structures in DICOM
  • Make sure there is a way to store DICOM UIDs, use definitions that are already specified in DICOM

button press

• • File storage: currently in XML in Plus
  • In memory storage: XML data elements
  • Real-time transfer: MUSiiC custom OpenIGTLink messages have some info
• Scene:
  • File storage: Slicer uses MRML - alternatives?
  • Memory: MRML library
  • Real-time transfer?
• Common algorithms:
  • Pivot calibration (with robust point matching, error rejection, error metrics, etc.)
  • Landmark registration (with automatic landmark detection, etc.)
  • Volume reconstruction
  • Scan conversion, brightness conversion
  • Bone surface detection
  • Ultrasound simulation
  • US/MRI registration
  • Custom reader/writer classes for tracked image data (supporting compressed streaming, compressed seeking, all necessary metadata, etc.)

Current implementations

• Tracked ultrasound storage in file:
  • Plus: sequence metafile
  • CustusX:
    • Saving during recording: a new folder is created, each image frame is written to a separate file (optionally compressed), separate file stores tracking data and metadata
    • Sequence metafile (MetaImage format with Plus custom fields) is also used
    • Application state is saved to an XML file (similar to MRML scene) that links to other files (sequence metafile, VTK polydata, etc)
    • The entire procedure is also recorded to a separate, simple, compressed file
  • IBIS:
    • Sequence of MINC files: each frame is a separate MINC file, contains both tracking transform and calibration transform that was current when the frame was acquired. All provenance information is kept in the header of each file (up to the original data source, DICOM, etc.)
  • MITK-IGT:
    • There is not really a standard, images can be saved as a 3D NRRD image using ITK, tracking data can be saved as xml or csv
    • Sometimes sequence metafile is used
• Tracked ultrasound storage in memory:
  • Plus: TrackedFrameList class, list of PlusVideoFrame, PlusVideoFrame is a VTK class with vtkImageData (with zero origin and unit spacing), timestamp, and custom string fields (with convenience function to store/retrieve transforms, numerical values, etc.)
  • CustusX:
    • Uses VTK as main backend (with ITK converters for algorithms)
    • Custom image class that contains vtkImageData (with zero position, unit spacing), additional with orientation, in vtk image data position is 0
    • all transforms in eigen
    • sequences of video and ultrasound in a class
  • IBIS:
    • vtkImageData data and separate transform vtkMatrix4x4 for individual frame and separate for registration
    • Everything is kept in memory, no streaming writing to disk
  • MITK-IGT:
    • OpenCV mat image for image acquisition and pre-processing
    • MITK image sequence could be used as well
    • Calibration is a property of the MITK image
    • Tracking data: tracking data set is a separate structure
• Tracked ultrasound streaming:
  • Plus:
    • Transforms: receive TRANSFORM or TDATA, send TRANSFORM (TDATA implementation is in progress)
    • Images: IMAGE, USMESSAGE (MUSiiC ultrasound message), TRACKEDFRAME (Plus tracked frame: image data + string fields)
  • CustusX:
    • OpenIGTLink: has been used for 4 years, Plus for about half year
    • Image: both 3D and 4D US, using own image message type, non-standard message
    • Tracking: TRANSFORM
  • IBIS: everything is in the same process, no IPC
  • MITK: TDATA or TRANSFORM messages, depending on the application

Plans

• We focus on real-time data transfer standardization first, then file storage, in-memory representation, reader/writer and processing software library

Next meeting: December 14., 0900 EST (1500 Trondheim/Heidelberg time)

Homework before next meeting: Add to this wiki page the description of all data to be transferred in real-time using OpenIGTLink

Data necessary to be transferred in real time

• MITK-IGT group:
  • For image data:
    • Option to add meta data for different modalities (e.g: ultrasound- and photoacoustic imaging):
    • Additional metadata for PAI:
      • Wavelength (for multispectral image shots)
      • Laser pulse energy (for normalisation of signals)
      • Adding specific message types based on IMGMETA could be of use [optional/custom fields?]
  • For tracking data:
    • Consistent format
    • TDATA message suits our needs
• IBIS group:
  • Image data:
    • RGB + Depth images
    • synchronized stereo images
    • Meta-data for augmented reality: zoom, focus setting of microscope/endoscope/camera
    • Meta-data for US: depth, mode (b-mode, doppler, power doppler), image mask parameters
  • Tracking data:
    • TDATA
    • Tracker tool state
    • Meta-data for augmented reality: calibration parameters:
      • Camera model type: Zhang, Heikkila, Tsai.
      • focal length
      • Image center
      • distortion coefficients (depends on model type)
    • Meta-data for US : calibration matrix ( tool to image transform )