StandardsInterfaces

We are intreated in getting feedback on creating standards and interfaces for microscopy image analysis in ITK

Key Investigators

• Harvard - Sean Megason

Objective

There are now several groups using ITK to develop filters for microscopy image analysis. However there are a number of possibilities for the input and output formats for these filters. For filters from different groups to be interchangeable, it is important that these filters use a defined interface. The goal of this project is to solicit opinions from both groups working on microscopy image analysis as well as those with experience designing interfaces for image processing in other domains.

Approach, Plan

If you have opinions, suggestions, or interfaces you are already using please talk to Sean and come to the Microscopy Image Analysis breakout session on Wed afternoon.

Issues

Specifically we would like to define functional groups of filters (e.g. denoising, seeding, segmentation, tracking, lineaging, post-processing, quantitation, classification) to allow for filter swapping within a group and then to define input/output types between these groups. ImageToImage filters are a sufficient interface for some filter groups but are problematic for others due to changes in dimension (tracking, lineaging), excess size (seeding), and differences with the "natural" output of an algorithm (e.g. a segmentation algorithm may output a mesh rather than image).

Progress

Image Formats

We discussed standards for image formats at the Microscopy Breakout session. We broke this issue down into 3 components:

• File Types: e.g. jpg, png, tif, lsm. The problem of needing to read a bunch of different file types has been around a long time. This problem is still not ideally solved (there is no BSD licensed library that will read all microscopy formats), but there are enough workarounds that this issue is now more of an inconvenience than a road block. In practice most groups only use 1 maybe 2 types of microscopes and have solved their problem locally. A global solution would make it easier for groups to work together and would thus be preferable. However, current file types do not meet the needs of advanced imaging applications anyway (see "meta" file types below) so efforts should be aimed at both creating BSD readers for existing file types as well as meta file types.
• Metadata: metadata refers to all the nonimage data that is part of an imaging experiment. Metadata include image acquisition parameters (objective, laser power, PMT gain, filters, pixel dwell time) much of which is microscope specific. It can also include information about how the specimen was labeled and and its stage and orientation for imaging. It can even include a full description of the experiment (genotype, drugs...). Current solutions are to store the data in XML files, databases, and image headers. Metadata is very heterogeneous and user-specific and thus needs a flexible format.
• Meta image formats: For advanced microscopy applications, an experimental unit almost never corresponds to a single image file. Many microscopy file formats (e.g. Zeiss LSM) are designed for the one experiment one file paradigm with encapsulated metadata, but these scale very poorly. For advanced microscopy applications, it is more common to use a "meta" format consisting of many image files of some standard file type along with an XML or database schema that organizes the data across experimental dimensions and stores experimental metadata. This meta format approach has been widely and independently adopted because of its flexibility and scalability. In our own work we use something called the MegaCapture format which consists of an XML file for metadata along with a set of image files. These image files are stored as 2d, single channel images in any common format (TIF, JPG, PNG), and their file names reflect their coordinates in the dimensions of z, time, x-tile, y-tile, row, column, and plate. This schema is reflected in the database used by GoFigure. Several other groups have developed similar meta approaches. To the extent possible it would be nice for these formats to be compatible to allow for interchange between groups and perhaps more importantly to allow for the creation of ITK filters where the input is a meta image rather than a single image. ITK filters with meta image input are important for streamlined, HPC based processing for advanced microscopy applications.

Microscopy Filter IO Types

I met with Luis Ibanez and Jim Miller to discuss how the best representation for filter I/O issues have been handled in ITK previously. They suggested itk::SpatialObject and itk::LabelObject. LabelObject seems like a good intermediate between an image based representation and an object(cell) based representation, and allows for either perspective depending on which is more convenient. LabelObject uses run-length enconding of objects in an image which results in a lot of compression. LabelObject also allows for objects to be attributed with values (e.g. cell size, intensity, velocity). One downside is that there are not currently many filters in ITK that use LabelObjects. In our research, we (Kishore) have already been working with LabelObjects as the output to segmentation filters. This seems like a promising approach. We need to standardize the representation of cells in xy, xyt, xyz, xyzt, xyzt+division using LabelObjects and the associated cellular attributes. We are also working on a filter for converting LabelObjects into a GoFigure IO file format.

Greatest Common Denominator

We also discussed what the "greatest common denominator" of interchange between microscopy image analysis groups is.

• ITK filters: A preferred method for sharing microscopy processing filters is by submitting a paper to the Insight Journal and a class or classes to ITK. This mechanism provides for peer reviewed publication, automatic testing, and easy access to the code.
• Plugins for different apps: Another approach for collaboration amongst different groups is through the creation of plugins for existing microscopy image analysis applications. Plugins allow for extending the capabilities of an application to meet the needs of the user without having to recode all the common needs of an application such as visualization and manual editing of segmentation results. Such an approach can aid in the development cycle by providing a means for visualizing and verifying the results of a new filter and makes it easier to tune the parameters to a filter.
• Creation of a "microscopy data interchange format": Another approach is to have a standard interchange format for microscopy data that would encapsulate images, metadata, segmented cells/tracks/lineages, and object annotations. Such an approach would allow for any application in any language to work together as long as it reads and writes this format. The trick of course is defining the format and providing readers/writers. This issue is related to the meta format and filter IO format issues above.