Difference between revisions of "DBP3:Utah:AutoWallSeg"

From NAMIC Wiki
Jump to: navigation, search
Line 23: Line 23:
 
|-
 
|-
 
|}
 
|}
 +
 +
====Isosurface====
 +
 +
Note how the automatic segmentation creates a smooth/complete (mostly) surface on the top side of the atrium, while the bottom has gaps similar to the manual.
 +
 +
{| class="wikitable"
 +
|-
 +
! Auto Wall Segmenation Isosurface (01 3mo s5)
 +
! Manual Wall Segmentation Isosurface (01 3mo s5)
 +
|-
 +
| [[File:01 3mo s5 AutoIso.png|thumb]]
 +
| [[File:01 3mo s5 ManualIso.png|thumb]]
 +
|-
 +
| [[File:01 3mo s5 AutoIso 2.png|thumb]]
 +
| [[File:01 3mo s5 ManualIso 2.png|thumb]]
 +
|-
 +
|}
 +
 +
===Comparison Statistics===
 +
 +
====Inter Auto Seg Comparison====
 +
 +
Before getting the automatic wall segmentation code we had done a small study on our manual segmentations (see [[LA Segmentation reproducibility - intra and inter observer]]), this involved 64 total Endo + Wall Segmentations (4 different segmentors segmenting the same scan, for 16 scans).
 +
 +
To compare them we used the STAPLE algorithm to create a "ground truth" and then calculated overlap and accuracy, among other things, according to that ground truth.
 +
 +
We did the same analysis, but on the automatic wall segmentations (each derived from the different manual endo segmentations), and compared it to the original manual segmentation data.
 +
 +
Below are boxplots of the two data sets, for both overlap and accuracy.
 +
 +
For two sets S1 and S2, overlap is computed:  (2 * ||S1 ^ S2|| ) / ( ||S1|| + ||S2|| )
 +
 +
For two sets S1 and S2, accuracy is computed:  (||S1 ^ S2|| + ||!S1 ^ !S2|| ) / (||S1|| + ||S2||)
 +
 +
(For brevity: ^ is intersection, U is union, and !S is the complement of S)
 +
 +
[[File:AutoSegOverlap.png]]
 +
 +
[[File:AutoSegAccuracy.png]]
 +
 +
 +
As the graphs show the automatic wall segmentations have much better overlap and accuracy with their STAPLE produced ground truths.
 +
 +
However the most these results seem to say is that the automatic algorithm is more consistent than manual segmentations.  We hope to run a new comparison to be able to really evaluate the performance of the automatic segmentation. 
 +
 +
A side note is that the manual segmentations are actually very consistent when compared to the automatic segmentations - one would expect the gap to be larger.
 +
 +
Original data: [[File:Autoseg.xlsx]], [[File:Manual Compare Results.xlsx]]
 +
 +
====Auto to Manual Overlap Comparison====
 +
 +
In this comparison we took each automatic segmentation produced using a manual endo segmentation and compared it with the corresponding manual wall segmentation.
 +
 +
The box plot below is for all 64 automatic-manual segmentation pairs.
 +
 +
[[File:AugoSeg Overlap pairs.png]]
 +
 +
Original data: [[File:Autoseg pairs.xlsx]]

Revision as of 00:52, 22 April 2011

Home < DBP3:Utah:AutoWallSeg

Automatic Segmentation

Automatic Wall Segmentation from GA Tech

GA Tech produced a slicer module to automatically segment the left atrial wall, given the original data AND the endo (blood pool) segmentation. Below is some evaluation of those segmentations.

Auto, Endo, and Manaul Wall (01 3mo s5) Auto, Endo, and Manual Wall (01 3mo s4) Auto, Endo, and Manual Wall (01 3mo s1) Auto, Endo, and 4 Pixel Dilation from Endo (01 3mo s1)
01 3mo s5 Auto Endo Manual slice.png
01 3mo s4 Auto Endo Manual slice.png
01 3mo s1 Auto Endo Manual slice.png
01 3mo s1 Auto Endo 4PixDilate slice.png
This image shows how the automatic wall segmentation creates a complete surface around the endo - while a manual segmentor would cut off the wall around the veins (although the location this cutoff decision is arbitrary). This image shows where the automatic segmentation surrounds a vein, while the manual segmentor decided to not surround the vein. This image shows some island artifacts that may be the result of the process being done in 3D (leaking from the slice above) - note that the data below the islands is lighter. Here we show a 4 pixel dilation from the endo compared to the automatic segmentation. This also image shows more islands.

Isosurface

Note how the automatic segmentation creates a smooth/complete (mostly) surface on the top side of the atrium, while the bottom has gaps similar to the manual.

Auto Wall Segmenation Isosurface (01 3mo s5) Manual Wall Segmentation Isosurface (01 3mo s5)
01 3mo s5 AutoIso.png
01 3mo s5 ManualIso.png
01 3mo s5 AutoIso 2.png
01 3mo s5 ManualIso 2.png

Comparison Statistics

Inter Auto Seg Comparison

Before getting the automatic wall segmentation code we had done a small study on our manual segmentations (see LA Segmentation reproducibility - intra and inter observer), this involved 64 total Endo + Wall Segmentations (4 different segmentors segmenting the same scan, for 16 scans).

To compare them we used the STAPLE algorithm to create a "ground truth" and then calculated overlap and accuracy, among other things, according to that ground truth.

We did the same analysis, but on the automatic wall segmentations (each derived from the different manual endo segmentations), and compared it to the original manual segmentation data.

Below are boxplots of the two data sets, for both overlap and accuracy.

For two sets S1 and S2, overlap is computed: (2 * ||S1 ^ S2|| ) / ( ||S1|| + ||S2|| )

For two sets S1 and S2, accuracy is computed: (||S1 ^ S2|| + ||!S1 ^ !S2|| ) / (||S1|| + ||S2||)

(For brevity: ^ is intersection, U is union, and !S is the complement of S)

AutoSegOverlap.png

AutoSegAccuracy.png


As the graphs show the automatic wall segmentations have much better overlap and accuracy with their STAPLE produced ground truths.

However the most these results seem to say is that the automatic algorithm is more consistent than manual segmentations. We hope to run a new comparison to be able to really evaluate the performance of the automatic segmentation.

A side note is that the manual segmentations are actually very consistent when compared to the automatic segmentations - one would expect the gap to be larger.

Original data: File:Autoseg.xlsx, File:Manual Compare Results.xlsx

Auto to Manual Overlap Comparison

In this comparison we took each automatic segmentation produced using a manual endo segmentation and compared it with the corresponding manual wall segmentation.

The box plot below is for all 64 automatic-manual segmentation pairs.

AugoSeg Overlap pairs.png

Original data: File:Autoseg pairs.xlsx

Retrieved from "https://www.na-mic.org/w/index.php?title=DBP3:Utah:AutoWallSeg&oldid=66823"