Difference between revisions of "Slicer3:Remote Data Handling"

From NAMIC Wiki
Jump to: navigation, search
m (Text replacement - "http://www.slicer.org/slicerWiki/index.php/" to "https://www.slicer.org/wiki/")
 
(113 intermediate revisions by 4 users not shown)
Line 1: Line 1:
'''[[Slicer3:Developers#Slicer_3_Projects | Back to Slicer3 Projects List ]]'''
+
<big>'''Note:''' We are migrating this content to the slicer.org domain - <font color="orange">The newer page is [https://www.slicer.org/wiki/Slicer3:Remote_Data_Handling here]</font></big>
 
 
= Current status of Slicer's (local) data handling =
 
Currently, MRML files, XCEDE catalog files, XNAT archives and individual datasets are all loaded from local disk. All remote datasets are downloaded (via web interface or command line) outside of Slicer. In the BIRN 2007 AHM we demonstrated downloading .xar files from a remote database, and loading .xar and .xcat files into Slicer from local disk using Slicer's XNAT archive reader and XCEDE2.0 catalog reader. Slicer's current scheme for data handling is shown below:
 
 
 
[[image:DataLoadingCurrent.png]]
 
 
 
= Goal for how Slicer would upload/download from remote data stores =
 
Eventually, we would like to download uris remotely or locally from the Application itself, and have the option of uplaoding to remote stores as well. A sketch of architecture planned in a meeting (on 2/14/08 with Steve Pieper, Nicole Aucoin and Wendy Plesniak) is shown below, including:
 
* a collection of vtkURIHandlers,
 
* an (asynchronous) Data I/O Manager and
 
* a Cache manager,
 
 
 
all created by the main application and pointed to by the MRMLScene.:
 
 
 
[[image:DataLoadingTarget.png ]]
 
 
 
= TWO Use CASES can drive a first pass implementation =
 
For BIRN, we'd like to demonstrate two use cases:
 
 
 
* '''First''', is loading a combined FIPS/FreeSurfer analysis, specified in an Xcede catalog (.xcat) file that contains uris pointing to remote datasets, and view this with Slicer's QueryAtlas. (...if we cannot get an .xcat via the HID web GUI, our approach would be to manually upload a test Xcede catalog file and its constituent datasets to some place on the SRB. We'll keep a copy of the catalog file locally, read it and query SRB for each uri in the .xcat file.)
 
* '''Second''', is running a batch job in Slicer that processes a set of remotely held datasets. Each iteration would take as arguments the XML file parameterizing the EMSegmenter, the uri for the remote dataset, and a uri for storing back the segmentation results.
 
 
The schematic of the functionality we'll need is shown below:
 
 
 
[[image:DataLoadingStartPlan.png]]
 
 
 
= Target pieces to implement prior to upcoming BIRN Meetings =
 
 
 
'''RemoteIO Library''', including the following classes:
 
* vtkURIHandler
 
** vtkFileHandler
 
** vtkFtpHandler
 
** vtkHttpHandler
 
* vtkPasswordPrompter ''Wendy''
 
* vtkDataIOManager ''Wendy''
 
* vtkDataTransfer ''Wendy''
 
* vtkCacheManager ''Wendy''
 
 
 
URIs that we’d want to handle:
 
 
 
* http://www.slicer.org/example.mrml
 
* ftp://user:pw@host:port/path/to/volume.nrrd (read and write)
 
* http://host/file.utp (read only)
 
 
 
'''MRML-specific implementations and extensions''' of the following classes:
 
* vtkMRMLScene ''Steve''
 
* vtkMRMLDataIOHandler ''Wendy''
 
* vtkMRMLStorageNode methods ''Nicole''
 
* vtkMRML<DataType>StorageNode methods ''Nicole''
 
 
 
'''GUI implementations''' of the following classes:
 
* vtkSlicerDataTransferWidget ''Wendy''
 
* vtkSlicerDataIOHandlerGUI ''Wendy''
 
* vtkSlicerPasswordPrompter ''Wendy''
 
 
 
'''New Application Interface Settings''': ''Nicole''
 
* Set CacheDirectory
 
* Set CacheFreeBufferSize
 
* Set CacheLimit
 
* Enable/Disable CacheOverwriting
 
* Enable/Disable ForceRedownload
 
* Enable/Disable asynchronous I/O
 
* Instance & Register URI Handlers (?)
 
 
 
'''Important notes:'''
 
* MRML Scene file reading will always be synchronous.
 
* Read/write of individual datasets referenced in the scene should work with or without the asynchronous read/write turned on, and with or without the dataIOManager GUI interface.
 
 
 
= pseudocode & notes worked out in 2/14/08 meeting =
 
 
 
== MRMLScene ==
 
 
 
// example: pseudocode for reading remote mrml scene,
 
// parsing and loading data synchronously or asynchronously.
 
//------------------------------------------------------------
 
//-- hypothetical dataset: http://*/example.mrml
 
//------------------------------------------------------------
 
 
//--- In Slicer3.cxx main application
 
//--- Create Application and read registry
 
 
//--- Create MRML scene
 
vtkMRMLScene *scene = vtkMRMLScene::New();
 
   
 
 
//--- create cache manager
 
CacheManager = vtkCacheManager::New();
 
CacheManager->Configure();
 
 
//--- create asynchronous IO manager
 
DataIOManager = vtkMRMLDataIOManager::New();
 
DataIOManager->Configure();
 
 
//--- create  uri handler collection and populate
 
URIHandlerCollection = vtkCollection::New()
 
//--- discover and instance all vtkURIHandlers  and add them to collection
 
 
//--- set members of mrml scene
 
scene->SetCacheManager (CacheManager);
 
scene->SetDataIOManager (DataIOManager);
 
scene->SetURIHandlerCollection ( URIHandlerCollection );
 
scene->SetURI ( *uri );
 
 
//--- parse scene file and load data
 
MRMLScene::Connect()
 
{
 
  filename = GetCacheManager()->GetFilenameFromURI ( *uri );
 
  handler = this->FindURIHandler ( *uri );
 
  handler->StageFile ( *uri, *filename );
 
  //…
 
  //--- synchronous read until we have the file on disk
 
  //…
 
  //--- then parse
 
  parser = vtkMRMLParser::New();
 
  parser->SetMRMLScene ( this);
 
  parser->Parse();
 
 
  //--- and create nodes
 
  //…
 
  {
 
  vtkMRML<DataType>StorageNode->ReadData();
 
}
 
 
 
'''Note:'''
 
* scene->GetRootDir() should return a uri
 
* When scene->Delete() is called:
 
 
 
if (CacheManager != NULL )
 
{
 
  CacheManager->Delete();
 
  CacheManager = NULL;
 
}
 
if (DataIOManager != NULL )
 
{
 
  DataIOManager->Delete();
 
  DataIOManager = NULL;
 
}
 
//--- and clean up the URIHandlerCollection here too?
 
 
 
== vtkMRMLStorageNode class ==
 
 
 
StageReadData ( vtkMRMLNode *node )
 
{
 
vtkMRMLDataIOManager *iomanager = this->GetMRMLScene()->GetDataIOManager();
 
if ( iomanager != NULL )
 
{
 
  int asynch = this->GetMRMLScene()->GetDataIOManager()->GetAsynchronousEnabled();
 
}
 
if ( iomanager != NULL && asynch && this->GetReadState() != PENDING )
 
{
 
  this->SetReadState(PENDING);
 
  this->handler = this->GetMRMLScene()->FindURIHandler ( this->URI );
 
  iomanager->Queue ( *node );
 
}
 
else
 
{
 
  filename = this->GetMRMLScene()->GetCacheManager()->GetFilenameFromURI ( this->URI );
 
  handler->StageFile ( this->URI, *filename );
 
  this->SetReadState ( DONE );
 
}
 
 
 
StageWriteData ( vtkMRMLNode *node )
 
  {
 
  // similar to above
 
  }
 
 
 
'''Notes:'''
 
* Need progress feedback -- vtkFilterWatcher on handler?
 
 
 
== vtkMRML<DataType>StorageNode classes ==
 
 
 
ReadData ( vtkMRMLNode *node )
 
{
 
  Superclass::StageReadData ( node );
 
  if ( this->GetReadState() == PENDING )
 
  {
 
  return;
 
  }
 
  else
 
  {
 
  Read;
 
  }
 
 
 
WriteData ( vtkMRMLNode *node)
 
  {
 
  // figure out logic wrt checking GetWriteState
 
  Write to file;
 
  SuperClass::StageWriteData(node);
 
  }
 
 
 
'''Notes:'''
 
* Need progress feedback -- vtkFilterWatcher on Read?
 
 
 
== vtkCacheManager class ==
 
 
 
vtkCachedFileCollection = vtkCollection::New();
 
 
 
== vtkDataIOManager and vtkMRMLDataIOManager  ==
 
 
 
Are there some set of basic methods good to break into Libs/RemoteIO/vtkDataIOManager class?
 
 
 
In vtkMRML(orSlicer)DataIOManager class:
 
 
 
Queue ( vtkMRMLNode *node )
 
{
 
  //--- creates a vtkDataTransferObject, sets values, adds to collection
 
  this->AddDataTransfer( node );
 
  //--- if ForceReUpload flag is set:
 
  this->DeleteFileInCache ( );
 
  handler = node->GetStorageNode()->GetURIHandler();
 
 
 
  //--- need to add an observer so can monitor the read progress...
 
  //--- maybe on handler, maybe on storage node?
 
  handler->AddObserver(...);?
 
  node->GetStorageNode()->AddObserver(...);?
 
 
 
  //--- create a new processing thread in which to call:
 
  node->GetStorageNode()->ReadData ( node );
 
  return;
 
}
 
 
 
'''Notes:'''
 
* Check CommandLineModuleLogic for threading examples
 
* or ApplicationLogic
 
* vtkFilterWatcher examples are in model maker.
 
 
 
= Developing... RemoteIO library classes, members and methods depend on vtk and libcmcurl=
 
 
 
== New /Libs/RemoteIO/'''vtkDataTransfer''' class: ==
 
This class describes each upload/download load/save handled by the DataIOManager.
 
 
 
'''vtkDataTransfer members:'''
 
* SourceURI
 
* DestinationURI
 
* uriHandler
 
* Status ( cancelled, finished, running, timed out)
 
* TransferID
 
* TransferType (upload, download, fromdisk, todisk)
 
* Timestamp
 
 
 
'''vtkDataTransfer methods:'''
 
* Get/Set on members
 
 
 
 
 
 
 
== New /Libs/RemoteIO/'''vtkDataIOManager''' class: ==
 
 
 
'''vtkDataIOManager members:'''
 
* vtkCollection *DataTransferCollection
 
* BOOL Asynchronous
 
 
 
'''vtkDataIOManager methods:'''
 
* Get/Set DataTransferCollection
 
* virtual Queue ( )
 
* ...
 
 
 
 
 
== New /Libs/RemoteIO/'''vtkCacheManager''' class: ==
 
 
 
'''vtkCacheManager members:'''
 
* CacheDirectory
 
* CacheLimit
 
* FreeBufferSize
 
* EnableOverwrite
 
* ForceRedownload
 
 
 
'''vtkCacheManager methods:'''
 
* ClearCache();
 
* CheckCacheSize();
 
* GetFilenameFromURI ();
 
 
 
== New /Libs/RemoteIO '''vtkPasswordPrompter''' class: ==
 
 
 
'''vtkPasswordPrompter members:'''
 
* Title
 
* Prompt
 
* UserName
 
* Password
 
* HelpString
 
* Service
 
* SavePasswordService (?)
 
 
 
'''vtkPasswordPrompter methods:'''
 
* Get/Set on members
 
* Prompt () ( ?stdio text prompting?)
 
 
 
== New /Libs/RemoteIO/'''vtkURIHandler''' class ==
 
 
 
'''vtkURIHandler members:'''
 
* vtkDataTransferLog *DataLog
 
* URI
 
* CacheDirectory
 
* LocalFileName
 
* vtkPasswordPrompter *PasswordPrompter
 
* ErrorString
 
* FileSize
 
* ThreadID (?)
 
 
 
'''vtkURIHandler (virtual) methods:'''
 
* Get/Set on members ...
 
* int CanHandleURI ()
 
* QueryFileSize ()
 
* Download (filter watcher?)
 
* Upload (filter watcher?)
 
* (virtual) Notify ( int status )
 
 
 
== New /Libs/RemoteIO classes, derived from '''vtkURIHandler'''==
 
 
 
* vtkFileHandler
 
** int CanHandleURI()
 
** PasswordPrompter->Prompt() if required
 
** Download ( ) (filter watcher?)
 
*** Separate thread of control for asynchronous download
 
*** Returns some kind of threadID (not sure how to do this?)
 
** Upload ( ) (filter watcher?)
 
*** Separate thread of control for asynchronous download
 
*** Returns some kind of threadID (not sure how to do this?)
 
** FinishTransfer()
 
*** Returns some value to the caller saying transfer is complete
 
** QueryProgress ()
 
* vtkHttpHandler
 
* vtkFtpHandler
 
* vtkSRBHandler
 
* vtkXNATHandler
 
* vtkS3Handler
 
* vtkSTDIOHandler
 
 
 
= Developing... Extending MRML classes, members and methods =
 
 
 
==vtkMRMLScene methods (and members) ==
 
 
 
'''vtkMRMLScene members: '''
 
* CacheDirectory
 
* URIHandlerCollection
 
* CacheManager
 
* DataIOManager
 
 
 
'''Slicer3 Application will have:'''
 
* CacheManager
 
* DataIOManager
 
 
 
 
 
'''vtkMRMLScene methods: '''
 
* Get/Set on members...
 
* RegisterURIHandler ( *vtkURIHandler )
 
** ''Adds a URIHandler to the collection''
 
* vtkURIHandler *FindURIHandler (uri)
 
 
 
 
 
 
 
==vtkMRMLDataIOManager  class, derived from vtkDataIOManager class:==
 
 
 
== New vtkMRMLStorageNode''' members and methods:''' ==
 
 
 
'''vtkMRMLStorageNode members:'''
 
* URI
 
* ErrorString?
 
 
 
'''vtkMRMLStorageNode methods:'''
 
* Get/Set URI
 
* Get/Set ErrorString
 
* void StageReadData ( vtkMRMLNode *node )
 
* void StageWriteData ( vtkMRMLNode *node )
 
 
 
== New vtkMRML<DataType>StorageNode''' members and methods:'''==
 
 
 
''Do we need a CacheFileName? Won't the local location just be stored in FileName? StageDataRead could set the FileName after DL from the URI.''
 
 
 
'''vtkMRML<DataType>StorageNode members:'''
 
* CacheFileName
 
 
 
'''vtkMRML<DataType>StorageNode methods:'''
 
* Get/Set CacheFileName()
 
* ReadData( vtkMRMLNode *node )
 
** Superclass::StageDataRead ( *node )
 
* WriteData( vtkMRMLNode *node )
 
** Superclass::StageDataWrite ( *node )
 
 
 
= Developing... Changes to /Base/GUI classes, members and methods =
 
== New /Base/GUI/'''vtkSlicerDataIOManagerGUI''' class''': ==
 
 
 
'''vtkSlicerDataIOManagerGUI members:'''
 
* vtkKWTopLevel
 
* ...
 
'''vtkSlicerDataIOManagerGUI methods:'''
 
 
 
== New /Base/GUI/vtkSlicerDataTransferWidget class, derived from vtkKWWidget==
 
 
 
'''vtkSlicerDataTransferWidget members:'''
 
* ...(fills out interface to interact with one data transfer)
 
 
 
'''vtkSlicerDataTransfer methods:'''
 
*...
 
 
 
== New /Base/GUI/vtkSlicerPasswordPrompter class, derived from '''/Libs/RemoteIO/vtkPasswordPrompter'''==
 
 
 
'''vtkSlicerPasswordPrompter members:'''
 
* vtkKWDialog *PromptWindow
 
*...
 
 
 
'''vtkSlicerPasswordPrompter methods:'''
 
* DisplayPrompt () (KW dialog-based prompt?)
 
 
 
= ITK-based mechanism handling remote data (for command line modules, batch processing, and grid processing) (Nicole) =
 
 
 
This one is tenatively on hold for now.
 
 
 
= Workflows to support =
 
 
 
The first goal is to figure out what workflows to support, and a good implementation approach.
 
 
 
Currently, '''Load Scene''', '''Import Scene''', and '''Add Data''' options in Slicer all encapsulate two steps:
 
* locating a dataset, usually accomplished through a file browser, and
 
* selecting a dataset to initiate loading.
 
Then MRML files, Xcede catalog files, or individual datasets are loaded from local disk.
 
 
 
For loading remote datasets, the following options are available:
 
* break these two steps apart explicitly (easiest option),
 
* bind them together under the hood,
 
* or support both of these paradigms.
 
 
 
For now, we choose the first option.
 
 
 
==Breaking apart "find data" and "load data":==
 
 
 
'''Possible workflow A'''
 
* User downloads .xcat or .xml (MRML) file to disk using the HID or an XNAT web interface
 
* From the Load Scene file browser, user selects the .xcat or .xml archive. If no locally cached versions exist, each remote file listed in the archive is downloaded to /tmp directory (always locally cached) by the Data I/O Manager, and then cached (local) uri is passed to vtkMRMLStorageNode method when download is complete.
 
 
 
'''Possible workflow B'''
 
* User downloads .xcat or .xml (MRML) file to disk using the HID or an XNAT web interface
 
* From the Load Scene file browser, user selects the .xcat or .xml archive. If no locally cached versions exist, each remote file in the archive is downloaded to /tmp (only if a flag is set) by the Data IO Manager, and loaded directly into Slicer via a vtkMRMLStorageNode method when download is complete. (How does load work if we don't save to disk first?)
 
 
 
'''Workflow C'''
 
* describe batch processing example here, which includes saving to local or remote.
 
 
 
In each workflow, the data gets saved to disk first and then loaded into StorageNode or uploaded to remote location from cache.
 
 
 
== What data do we need in an .xcat file? ==
 
For the fBIRN QueryAtlas use case, we need a combination of '''FreeSurfer morphology analysis''' and a '''FIPS analysis''' of the same subject. With the combined data in Slicer, we can view activation overlays co-registered to and overlayed onto the high resolution structural MRI using the FIPS analysis, and determine the names of brain regions where activations occur using the co-registered morphology analysis.
 
 
 
The required analyses including all derived data are in two standard directory structures on local disk, and *hopefully* somewhere on the HID within a standard structure (check with Burak). These directory trees contain a LOT of files we don't need... Below are the files we *do* need for fBIRN QueryAtlas use case.
 
 
 
===FIPS analysis (.feat) directory and required data===
 
For instance, the FIPS output directory in our example dataset from Doug Greve at MGH is called sirp-hp65-stc-to7-gam.feat. Under this directory, QueryAtlas needs the following datasets:
 
* sirp-hp65-stc-to7-gam.feat/reg/example_func.nii
 
* sirp-hp65-stc-to7-gam.feat/reg/freesurfer/anat2exf.register.dat
 
* sirp-hp65-stc-to7-gam.feat/stats/(all statistics files of interest)
 
* sirp-hp65-stc-to7-gam.feat/design.gif (this image relates statistics files to experimental conditions)
 
 
 
===FreeSurfer analysis directory, and required data ===
 
For instance, the FreeSurfer morphology analysis directory in our example dataset from Doug Greve at MGH is called fbph2-000670986943. Under this directory, QueryAtlas needs the following datasets:
 
 
 
* fbph2-000670986943/mri/brain.mgz
 
* fbph2-000670986943/mri/aparc+aseg.mgz
 
* fbph2-000670986943/surf/lh.pial
 
* fbph2-000670986943/surf/rh.pial
 
* fbph2-000670986943/label/lh.aparc.annot
 
* fbph2-000670986943/label/rh.aparc.annot
 
 
 
= What do we want HID webservices to provide? =
 
 
 
* Question: are FIPS and FreeSurfer analyses (including QueryAtlas required files listed above) for subjects available on the HID yet? --Burak says not yet.
 
 
 
* Given that, can we manually upload an example .xcat and the datasets it points to the SRB, and download each dataset from the HID in Slicer, using some helper application (like curl)?
 
 
 
* (Eventually.) The BIRN HID webservices shouldn't really need to know the subset of data that QueryAtlas needs... maybe the web interface can take a BIRN ID and create a FIPS/FreeSurfer xcede catalog with all uris (http://....) in the FIPS and FreeSurfer directories, and package these into an Xcede catalog.
 
 
 
* (Eventually.) The catalog could be requested and downloaded from the HID web GUI, with a name like .xcat or .xcat.gzip or whatever. QueryAtlas could then open this file (or unzip and open) and filter for the relevant uris for an fBIRN or Qdec QueryAtlas session.
 
 
 
 
 
* Then, for each uri in a catalog (or .xml MRML file), we'll use (curl?) to download; so we need all datasets to be publicly readable.
 
 
 
* Can we create a directory (even a temporary one) on the SRB/BWH HID for Slicer data uploads?
 
 
 
* We need some kind of upload service, a function call that takes a dataset and a BIRNID and uploads data to appropriate remote directory.
 
 
 
[[ Slicer3:XCEDE_use_cases | For more info, see this page for more discussion of QueryAtlas's current use of Xcede catalogs, and assumptions... ]]
 
 
 
= Slicer's Data I/O Manager =
 
 
 
[[image:DataIOManager.png]]
 

Latest revision as of 18:07, 10 July 2017

Home < Slicer3:Remote Data Handling

Note: We are migrating this content to the slicer.org domain - The newer page is here