OpenIGTLink/Discussion/NewCommands

<< OpenIGTLink

Objective

• This is a log page for the discussion about new OpenIGTLink messages introduced in protocol version 2.

What's new in OpenIGTLink Protocol version 2 and Library Version 2

• Better support for IGS Systems
  • New IGS specific message types
• Matlab interface support
  • Remote Matlab command execution
  • Matlab interface library
  • IGS support in Matlab
• Other new messages
  • Associative Array message
  • NIfTI support ?

Timeline for V.2 Protocol Release

• Events:
• June 20 - June 26: NA-MIC Summer Project Week in Boston
  • June 22, 10:30 - : OpenIGTLink Update Presentation by Junichi Tokuda
  • June 22, 16:00 - : OpenIGTLink User Group Meeting
  • Release v.2.0!!
• Notes:
  • Because of the software release schedule, we will develop protocol for IGS and protocol for others separately.

Week	4/11 - 4/17	4/18 - 4/24	4/25 - 5/1	5/2 - 5/8	5/9 - 5/15	5/16 - 5/22	5/23 - 5/29	5/30 - 6/5	6/6 - 6/12	6/13 - 6/19	6/20 - 6/26
Events											NA-MIC Project Week
Protocol Draft [IGS]	Review	Review, Freeze
Protocol Draft [other]					Draft	Review	Review	Review			Review, Freeze
IGS System (Alexander)	Dev	Dev	Dev	Test	Test	Test
Library (Junichi)		Dev (IGS)	Dev (IGS)	Test (IGS)	Dev (other)	Dev (other)	Test (other)		Test (other)	Test (other)
Matlab IF (Junichi)						Dev	Dev	Test
3D Slicer (OIGTL IF) (Junichi)						Dev	Dev	Dev	Test	Test
3D Slicer (IGS Module) (Haiying)						Dev	Dev	Dev	Test	Test

Common GET_LIST message ?

A common GET_LIST message could be defined with the data type, like IMAGE, in the device name field. The answer could be a LIST message with ids (each 20 bytes, so they can be used as device names) of available data. Afterwards a GET_xy message with the id could be used to query the details.

Well, I don't think this will help much in practice.

Example: an application wants to show a list of available images before querying them from the server. Ids are not very specific, so an additional meta data message must be defined. So we have a) GET_LIST and b) for each id: GET_IMGMETA. It would be easier to get all the meta data at once.

Next example: all ficucial points shall be queried. A two-step mechanism would result in 1+n queries, whereas getting all points at once would end in only 1 query. Even if the user would like to get only 1 specific point, how could he know which Id to use?

Another reason is that sending all the data at once is much more efficient.

However, there might be use cases, where single items are useful. If the item size is constant, it makes no difference if only one or several items are transmitted. The number is bodySize/itemSize. Using this definion, single items can still be pushed over the network as it's currently done in most OpenIGTLink applications.

Comments

Junichi: I agree that generalized GET_LIST message is not necessary in many case. I would suggest to design GET_IMGMETA to allow requesting either list of metadata or metadata for a specific image. If there is a way to request meta data for a specific image, we can provide two-step approach (GET_LIST->GET_IMGMETA) in the future.

Alexander: Sure, the Id can be entered in device name field. If device name field is empty, all image meta data is returned. If Id is entered, only one image meta data is returned.

GET messages

Get messages are not well defined yet. The following things should be fixed:

* If 0 items are available, the bodysize of the answer is 0.
* If an error occurs while processing the request, a STATUS message shall be returned.

Comments

Junichi: The STATUS message defines status code 4 "Not found (file, configuration, device etc)", which can be used to tell 0 items are available. The problem is that it is not possible to specify the device type in the STATUS message. The requesting host (the host that issues GET_* message) may not be able to identify which GET_* message is associated with the received STATUS message, because the OpenIGTLink message allows having different devices with the same device name. One possible solution is to use "Status name" field in STATUS message for specifying device name. For example:

• Host A requests image (device type: "IMAGE", device name: "diffusion 1") to host B by sending GET_IMAGE message.
• Host B receives the GET_IMAGE message, but it does not have such image.
• Host B sends STATUS message with device type "STATUS", device name "diffusion 1", status code 4, and status message "NO IMAGE".

Fortunately, maximum length of the status name field is 20, longer than the maximum length of device name.

Alexander: Well ok, if 0 items are available, a STATUS msg can be returned, but I would leave the error name as it is already defined - as error name. Imagine you have a GET message with type, device name and some other parameters. You send several of these messages with equal type and device name, but other parameters. You have to associate the STATUS messages... We can avoid this by defining the following:

• A GET message shall be answered by exactly one answer message.
• The answer messages shall be returned in the same sequence as the GET messages were sent.

However, the STATUS message has a device name field. I would write into that field the type name of the GET message, e.g. IMAGE, because this is the "toplevel" information. What do you think?

Junichi: It makes sense to leave the error name as it is. Putting device type into the device name is a bit confusing,if there is a device with a name same as device type. I'm start thinking that your initial idea (message with body size 0) makes sense.

Alexander:

• Ok, let's define if item(s) is/are not available, an answer with body size 0 shall be returned. In this way it's easy to include device type and name.
• If something went wrong during processing the query, a STATUS message could be returned, but as you already wrote, a field is missing for specifying the device type. Maybe the interface needs a STATUS 2.0 message? However, I think about using always the original answer message with appropriate body data on success and body size 0 if an error has occurred, because in almost every case the important information is "data not available" and not "reason abc".

Junichi:

• I totally agree. The STATUS message was originally designed for hardware e.g. biopsy robot and may not fit to other applications.

Image data

I would like to introduce a GET_IMGMETA message. This get-message has no parameters. The answer is IMGMETA:

Data	Type	Description
Name/Description	char[64]	Name or description of the image
Id	char[20]	Id to query the IMAGE and COLORT
Modality	char[32]	String which specifies the modality
Patient name	char[64]	Name of the patient
Patient id	char[64]	Id of the patient
Timestamp	64 bit unsigned	Scan time, see OpenIGTLink/Timestamp
RI, RJ, RK	16 bit unsigned	Number of pixels in each direction (same as in IMAGE)
S	8 bit unsigned	Scalar type (e.g. 3:uint8, 5:uint16, same as in IMAGE)
--	8 bit unsigned	Reserved

• More than one item can be transmitted. The number is bodySize/itemSize.
• To get the IMAGE, GET_IMAGE is used with the Id in the device name field.

To get the COLORTABLE, GET_COLORT is used with the Id in the device name field. The following is copied from Standard OpenIGTLink Protocol description -- Version 2 draft. I have removed the version and reserved fields - or why do we need a version in the header and in the body?

• GET_COLORT has no parameter except the device name field in the header.
• See below the COLORTABLE message:

Data	Type	Description
I	8-bit unsigned int	Index Type (3:uint8 5:uint16)
M	8-bit unsigned int	Map Type (3:uint8 5:uint16 19: RGB color)
TABLE	Array of 8-bit unsigned int	Color index table

Comments

Junichi:

• I would define time stamp as 64 bit unsigned int so that we can also specify time. (this is also used in the OpenIGTLink header. see OpenIGTLink/Timestamp.)
• If a device name is specified in GET_IMGMETA, only one set of meta data for the image with specified device name is returned. (This allows two-step approach (GET_LIST->GET_IMGMETA), while supporting multiple sets of meta data in a single IMGMETA message.)
• I would call the first field "Image description"

Alexander:

• Agree to your first point. I have adapted the table above.
• Agree to your second point.
• Regarding your third point: let's call it "Name or description".

Junichi: Sounds good.

Alexander: I have added the COLORTABLE message above.

Label map / Voxel objects

Voxel objects can be tumors, segmented structures likes eyes, etc. They are typically much smaller than the owning slicesets and can be overlayed on the sliceset data.

To retreive voxel objects or a label map, GET_IMAGE / IMAGE can be used. But the client should be able to get a list of available structures. I suggest a GET_LBMETA message:

LBMETA

Data	Type	Description
Name/Description	char[64]	Name or description of the image
Id	char[20]	Id to query the IMAGE
Label	8 bit unsigned int	Label of structure (0 if unused)
--	8 bit unsigned int	Reserved.
R,G,B,A	8 bit unsigned	Color in RGBA (0 0 0 0 if no color is defined)
RI, RJ, RK	16 bit unsigned	Number of pixels in each direction (same as in IMAGE), bounding box of the structure(s)
Owner image	char[20]	Id of the owner image/sliceset. Voxel objects from different slicesets can be sent if slicesets are fused. Can be empty if n/a.

Point or fiducal data

Currently only transformations can be sent with OpenIGTLink, but especially points like fiducals or trajectories consist of more than a transformation.

GET_POINT and GET_TRAJ are used to get the point data. They have no parameters. The answer is a POINT or TRAJECTORY message:

POINT

Data	Type	Description
Name	char[64]	Name or description of the point
Group name	char[32]	Can be "Labeled Point", "Landmark", "Fiducal", ...
R,G,B,A	8 bit unsigned	Color in RGBA
X,Y,Z	32 bit float	Coordinate of the point
Diameter	32 bit float	Diameter of the point, can be 0.
Owner image	char[20]	Id of the owner image/sliceset. Points from different slicesets can be sent if slicesets are fused.

TRAJECTORY

Data	Type	Description
Name	char[64]	Name or description of the trajectory.
Group name	char[32]	Can be "Trajectory", ...
Type	8 bit unsigned	1: trajectory with only entry point, 2: trajectory with only target point, 3: trajectory with entry and target point
R,G,B,A	8 bit unsigned	Color in RGBA
X1,Y1,Z1	32 bit float	Entry point of the trajectory
X2,Y2,Z2	32 bit float	Target point of a trajectory
Diameter	32 bit float	Diameter of trajectory, can be 0
Owner image	char[20]	Id of the owner image/sliceset. Trajectories from different slicesets can be sent if slicesets are fused.

* More than one item can be transmitted if device name field is empty. The number is bodySize/itemSize.

Comments

Junichi:

• For diameter, I think It's OK to allow value greater than 0 for single point.
• Do we really need group name?
• Is it possible to have TRAJECTORY type independently? I know POINTS and TRAJECTORY are very similar, but a bit confusing for those who new to the protocol.

Alexander:

• Yes, points can have a diameter.
• Group name: Yes, I think so. With the group name it is possible to distinguish e.g. between pre-op and intra-op landmarks, etc.
• If you really think it would be better to have two queries, I'll accept that. In the past we already had two messages. I was not sure if it is better to combine these messages or not. However, I have apapted the table above. If anyone else likes to have one message, please comment!

Start / stop push

Currently there is no way to start or stop a pushing request. The message START_PUSH and STOP_PUSH shall be used for that purpose. The device name field shall contain the data type to be pushed. Everytime this data changes, new messages shall be sent until the pushing is stopped. The body of START_PUSH and STOP_PUSH can contain data specific arguments.

START_PUSH

Data	Type	Description
Max Resolution	32 bit unsigned	Maximum update time in ms. Use 0 for as fast as possible. E.g. if data changes with 20 Hz, 20 messages per second will be sent.
Data specific arguments	char[bodysize - 4]	Can be everything, depends on the specific data type.

STOP_PUSH

Data	Type	Description
Data specific arguments	char[bodysize - 4]	Can be everything, depends on the specific data type.

Comments

Junichi:

• possible fields in START_PUSH message:
  • device type (TRACKINGDATA, TRANSFORM, IMAGE)
  • maximum time resolution (Hz)

Alexander:

• Yes, the device name field of the header could be e.g. TRACKINGDATA.
• Good idea about the maximum resolution!

I have added two tables above.

Junichi:

• I would add device type field in the START_PUSH and STOP_PUSH. The device name field cannot be used for specifying device type, because data source should be specified by a pair of device name and device type in OpenIGTLink.
• But I, at the same time, start thinking that your original idea makes sense. A device type can be START_<device type> / STOP_<device type> instead of START_PUSH / STOP_PULL, because this is consistent with what we do with GET_<device type> message.

Alexander:

• Agreed. We should dismiss START_PUSH and STOP_PUSH due to
  • type and device name must be specified,
  • inconsistencies with GET_<device type>,
  • we need a data specific body.

See below the "new" messages:

STT_TDATA

Data	Type	Description
Resolution	32 bit unsigned	Minimum time between two frames. Use 0 for as fast as possible. If e.g. 50 ms is specified, the maximum update rate will be 20 Hz.
Coordinate system name	char[20]	Coordinate system to use. Can be empty for default coordinate system. (not included if action = 2)

STP_TDATA

Data

Type

Description

Alexander:

• After thinking about the device name, the following can be specified:
  • If device name is empty, all visible trackers/instruments will be pushed.
  • If device name is not empty, only the appropriate tracker/instrument will be pushed.
• We need something to indicate, that pushing is started or stopped. Examples:
  • An error could occur, e.g. the device name = instrument name is not valid, no TDATA message can be sent.
  • After stopping the push, it might be possible that some TDATA messages are still in the pipeline. The stop should be acknowledged so the client can rely on not getting TDATA messages anymore.
• This could be done by a STATUS message, but as already wrote, the STATUS message does not have enough fields. I think about a typical answer message forr STT_TDATA and STP_TDATA:

RTS_TDATA

Data	Type	Description
Status	8 bit unsigned	0: Success 1: Error

Alexander:

• Using the query-answer-mechanism makes starting/stopping the push consistent to e.g. GET_IMAGE/IMAGE, etc.

What do you think about that?

Junichi:

• Sounds good idea to specify what happens if empty device name is specified.
• Yes, I think we need to address how to return status.
  • It's a nice idea to define status messages.
  • I would suggest more consistent naming convention
    • For starting data push START_<device name>
    • For stopping data push STOP_<device name>
    • For status STATUS_<device name>
  • By the way, 'START_' 'STOP_' and 'STATUS_' seem too long, because we only have 12 bytes for device name. Would be nice to have 3-letter codes ex. 'STT_', 'STP_', 'RTS_'. (this improves the consistency with 'GET_')

Alexander:

• I totally agree. I have changed the messages above to STT_TDATA, STP_TDATA and the returning status to RTS_TDATA. The real data, which is pushed is called TDATA now, see below.

Tracking data

In version 1 of the protocol, transformation matrices can be transmitted. But there are some additional requirements.

• Specifing that something is not valid/visible anymore (could be done by 0-matrix)
• Specifing which data is taken at the same time / part of the same camera frame (can be done with the timestamp field of the header)
• Specifing the type of data, e.g. instrument, instrument without a direction, tracker, ... (could be done by a meta-data query)
• Sending all tracking data each frame, e.g. 5 instruments/trackers at 60 Hz result in 300 message per second, which is very much. This should be reduced.

All the points above can be handles with a new message, let's call it TDATA:

TDATA

Data	Type	Description
Name	char[20]	Name (=Id) of the instrument/tracker
Type	8 bit unsigned	1: tracker, 2: 6D instrument (regular instrument), 3: 3D instrument (only tip of the instrument defined), 4: 5D instrument (tip and handle are defined, but not the normal vector)
Matrix	32 bit float	12 values like in TRANSFORM message

* All tracking data from one frame is included.
* Invisible/unavailable trackers/instruments are not included.
* Easy to develop. Sample pseudo code: while(true) { recv(trackingdata); updateView(trackingdata); }

This message cannot be queried with a regular GET message, but with START_PUSH and STOP_PUSH messages. The device name field consists of the string "TDATA".

Usually the tracking data will be sent using the standard coordinate system, which is also used for POINT, IMAGE, ... But this does only work after patient registration. Therefore the body of START_PUSH has an optional field for specifing the coordinate system "CAMERA". To switch back to the standard coordinate system, one has to send STOP_PUSH and afterwards START_PUSH without explicitly specifing the camera coordinate system.

Comments

Junichi:

• How about adding 8- or 16-bit status field in TDATA? This will allow us to indicate that coordinate system is not registered. I would like to keep START_PUSH message simple....

Alexander:

• What status types can be specified?
• In the case the coordinate system is not valid, a STATUS message should be returned.
• Well, I understand that you would like to keep it as simple as possible. We really like to specifiy the coordinate system like "Camera" or "Patient". We could also specifiy a SET_COORD message, but I think this would be overkill. I still vote for a data specific argument field in START_PUSH. Maybe other future data types can also use this field?
• Due to consistency, I think we should add this field to STOP_PUSH, too.
• We would like to allow only one TDATA push for each client at a time, so a second START_PUSH will stop the first and start the second. A STOP_PUSH will stop the push regardless of the arguments in the body. What do you think about that?

Junichi:

• I haven't defined status types... it can be a bit array like: bit 0: registered or not; bit 1: line-of-site error; ....
  • Do you think it is useful? if not, we can omit it.
• I agree that we need a way to specify coordinate system. It's good idea to have data specific field in the START_PUSH.
• I agree with the last comment. I would say one TDATA push for each device name, because multiple data sources may exist. The coordinate system can be overwritten by another START_PUSH message with the same device name and type.

Alexander:

• I think we don't need the status type. But maybe we define a TDATA in v3 after collecting some feedback from different users. I could think about other fields like diameter of instruments, etc. But for now I would like to keep TDATA as simple as possible.
• Ok, starting another trackingdata push with the same device name will implictly stop the first one. But if another device name is used, a second push will be started.