View source for CTSC Slicer XNE

= UXP & functionality planning for Slicer's XNE client =

== Workflow & Implementation approaches ==

=== All Workflow Steps: ===
* Select XNAT Enterprise server
* Authentication dialog presented
* XNAT login performed and session opened;
* Response is checked for error codes
* XNAT queried for projects 
* response is checked for http error codes
* xml parsed to get list of projects
* GUI populated
* One project selected
* XNAT queried for subjects and experiments in that project
* GUI populated
* User performs search for data
* User selects one scan and "download"
* Scan is loaded into Slicer

This workflow is detailed below. '''Note:''' Slicer uses the curl_easy library for client calls to RESTful web services. So the unix commandline curl calls below will need to be packaged up appropriately (including whatever basic authentication header information is required).

=== 1. Select XNAT Enterprise Server ===
This sets the current URI handler to be the XNE_HttpHandler for subsequent REST-based client calls to XNE web services.

=== 2. Authentication Dialog Presented ===

[[image:CTSC_Slicer_XNE_PermissionPrompter.png ]]

* Enter user name and password

=== 3. Process Dialog Input ===

* Save the username (XNE_UserName) and password (XNE_Password)
* Authenticate with $XNE_Svr and create a session:
 
 curl -d POST $XNE_Svr/REST/JSESSION -u $XNE_UserName:$XNE_Password

''Comment (DM): Will the username/password be encrypted in this exchange?''

The response from this request is a session ID that should be stored in $JSessionID; that ID can be used in all subsequent queries so authentication doesn't have to be performed at each transaction.

=== 4. Query XNAT for Projects Available to User ===

* XNAT queried for appropriate projects

 curl $XNE_Srv/REST/projects?format=xml --cookie JSESSIONID=$JSessionID

* response is checked for http error codes 
** if error is found, report to user
** else xml is parsed to get list of <del>scans<del> subjects 

''Comment (DM): You'll need to continue drilling down to get to the 'scan' level.  The typical hierarchy is project>subject>session>scan.  I believe that there are shortcuts in the API to skip the subject level but not to skip all the way to scans (which are the individual runs witin a session (for example, a single T1 or BOLD acquisition.  Typical sessions include multiple (someimtes many) scans.''

'''Question:''' Eventually, it would be nice to get the same info that the web GUI presents on its project view. Can we get a set of queries from Tim that give us the info there? including:
* Project Name
* Project ID
* PI
* Description
* Permissions

=== 5. User Selects One Project from GUI and clicks "Search Button" ===

* XNAT-E notebook panel is displayed with the '''"Choose Project"''' panel raised.
* GUI is populated with list of projects the user is permitted to see.
* User selects a project (only one for now)

Mockup:

[[image: CTSC_Slicer_XNE_ProjectPanel.png ]]

'''Question:''' Would be nice to search across multiple projects at once. Ask Tim for help on formulating a single query to search for subjects, experiments, etc., for projects {P1, P2,...,PN}.

=== 6. Query XNAT for Subjects in Selected Project ===

* XNAT queried for subjects in the selected project
 
 curl $XNE_Svr/REST/projects/$projectID/subjects?format=xml --cookie JSESSIONID=$JSessionID

* response is checked for http error codes 
** if error is found, report to user
** else xml is parsed to get list of scans
* '''"Browse & Download"''' panel is automatically raised
* Selected Project Name and Project ID are shown
* subject menu in '''"Browse & Download"''' panel is populated with list of subjects.
* "none" is selected in subject menu by default.
* experiment menu in '''"Browse & Download"''' panel is disabled until a subject is selected.
* '''Resource Filters''' panel is closed by default.

Mockup:

[[image: CTSC_Slicer_XNE_BrowsePanel.png ]]

=== 7. User Selects One Subject, Configures Filters to Narrow Search & Selects Search ===
Initial view presents a search that '''only''' requires SubjectID to be selected in order to populate the list of resources, but each additional selection (corresponding to the basic XNE data model) can be used to focus the search. Eventually add funcationality that allows the inclusion/exclusion of "shared subjects".

''''' 7a. Initial Presentation'''''
* Subject: { none } default selection
* Experiments: { none } (disabled)
** Scans: on (disabled)
** Reconstructions: on (disabled)
** Imaging Assessors: on (disabled)

 '''query:''' curl $XNE_Svr/REST/projects/$ProjectID/subjects?format=xml --cookie JSESSIONID=$JSessionID
 '''parse response''' into list of SubjectIDs
 update GUI

Mockup (user has opened the Resource Filters panel which is collapsed by default):

[[image: CTSC_Slicer_XNE_BrowsePanelFilters.png ]]


''''' 7b. Just a Subject ID is selected; Filter state & GUI are updated'''''

User Selects a subject, experiments menu is populated with all experiments for that subject and "all" is selected by default. All Resource Filters default to "all". This view provides a list of all resources for the selected SubjectID across all experiments.
* Subject: $SubjectID
* Experiments:  { all } by default
** Scans: on by default
** Reconstructions: on by default
** Assessors: on by default

 '''query:''' curl $XNE_Svr/REST/projects/$ProjectID/subjects/$SubjectID/experiments?format=xml --cookie JSESSIONID=$JSessionID
 '''parse response''' into list of ExperimentIDs
 for each id in ExperimentIDs
 {
  '''query:''' curl $XNE_Svr/REST/projects/$ProjectID/subjects/subjects/$SubjectID/experiments/$id/scans?format=xml --cookie JSESSIONID=$JSessionID
  '''parse response''' and add to list of ScanIDs
  '''query:''' curl $XNE_Svr/REST/projects/$ProjectID/subjects/subjects/$SubjectID/experiments/$id/reconstructions?format=xml --cookie JSESSIONID=$JSessionID
  '''parse response''' and add to list of ReconstructionIDs
  '''query:''' curl $XNE_Svr/REST/projects/$ProjectID/subjects/subjects/$SubjectID/experiments/$id/assessors?format=xml --cookie JSESSIONID=$JSessionID
  '''parse response''' and add to list of AssessorIDs
 }
 combine all IDs into resource list
 update GUI

Resource list presented in GUI might need to be constructed like one of these to be unique: 
* '''ExperimentID:Scan:ID''' or
* '''ExperiementID:Recon:ID''' or
* '''ExperimentID:Assr:ID'''

Then the resource list could be a '''std::map<std::string, std::vector<std::string> >''' that maps a uniquely named resource to a vector of strings that contains the following info, to help query building:
* projectID
* subjectID
* experimentID
* scanID (or NULL)
* reconstructionID (or NULL)
* assessorID (or NULL)
'''Question:''' Do we need anything else here?

Mockup:

[[image: CTSC_Slicer_XNE_BrowsePanelFiltersSubjectOnly.png ]]


''''' 7c. Both SubjectID && Experiment ID have been selected; Filter state & GUI are updated'''''

User selects an experiment and Scan, Reconstruction, and Imaging Assessor menus are populated.
* Subject: $SubjectIDs
* Experiments:  $ExperimentID(s)
** Scans: on (focused for $SubjectID && $ExperimentID(s) )
** Reconstructions: on (focused for $SubjectID && $ExperimentID(s) )
** Assessors: on (focused for $SubjectID && $ExperimentID(s) )

 for each $id in $ExperimentID(s)
 {
  '''query:''' curl $XNE_Svr/REST/projects/$ProjectID/subjects/subjects/$SubjectID/experiments/$id/scans?format=xml --cookie JSESSIONID=$JSessionID
  '''parse response''' and add to list of ScanIDs
  '''query:''' curl $XNE_Svr/REST/projects/$ProjectID/subjects/subjects/$SubjectID/experiments/$id/reconstructions?format=xml --cookie JSESSIONID=$JSessionID
  '''parse response''' and add to list of ReconstructionIDs
  '''query:''' curl $XNE_Svr/REST/projects/$ProjectID/subjects/subjects/$SubjectID/experiments/$id/assessors?format=xml --cookie JSESSIONID=$JSessionID
  '''parse response''' and add to list of AssessorIDs
 }
 combine all IDs into resource list
 update GUI

Mockup:

[[image: CTSC_Slicer_XNE_BrowsePanelFiltersSubjectExperiment.png ]]

''''' 7d. Filter is adjusted to refine search; Resource is selected;'''''
* Subject: $SubjectIDs
* Experiments: $ExperimentID 
** Scans: on 
** Reconstructions: off 
** Assessors: off 

  '''query:''' curl $XNE_Svr/REST/projects/$ProjectID/subjects/subjects/$SubjectID/experiments/$id/scans?format=xml --cookie JSESSIONID=$JSessionID
  '''parse response''' and add to list of ScanIDs
 combine all ScanIDs into resource list
 update GUI

Mockup:

[[image: CTSC_Slicer_XNE_BrowsePanelDownload.png ]]

=== 8. User Clicks Download ===

'''Questions:'''
* Once we get this far, presume we don't know the format of the resource (DICOM, Analyze, NRRD, etc.) so we can't just use the following to retrieve a list of files:

 curl $XNE_Svr/projects/$ProjectID/subjects/$SubjectID/experiments/$ExpID/scans/$ScanID/resources/DICOM/files/format=xml --cookie JSESSIONID=$JSessionID

* What is the recommended way to download a resource? In response, we want query to return a list of URIs which Slicer uses to populate a vtkMRMLVolumeArchetypeStorageNode:URI and vtkMRMLVolumeArchetypeStorageNode::URIListMembers

=== 9. Scan is Loaded into Slicer ===

vtkMRMLVolumeArchetypeStorageNode::ReadData( vtkMRMLVolumeArchetypeStorage *node) is called, which calls vtkMRMLStorageNode::StageReadData (*node) to download data using the XNE_HttpHandler.

'''Question for Nicole:''' Does the approach here seem to fit with the multi-file volume code you've written?

We can also ask for a .zip (or .xar?) file and use something like the code in Base/GUI/Tcl/Loader.tcl to load the archive from cache.

== Questions ==

* We'll need an XNETagTable to hold metadata for all resources coming from and going up to $XNE_Svr:
** We'll need to decide what metadata to put on resources downloaded from $XNE_Srv (stored in each vtkMRMLStorableNode's UserTagTable) 
** need to know what metadata is put to stuff going back up to XNE
** need to translate this metadata into appropriate URIs to PUT or POST back to $XNE_Svr.
* XNE web services allows us to DL multiple resources using one URI (see [http://nrg.wikispaces.com/XNAT+REST+Download here]); should we implement this in a first pass, or later?
* Custom tags are present in XNE -- how can we query for all custom tags and their values for a selected $XNE_Svr? This would allow us to populate an "advanced" search widget.

== Things to Build ==

* Authentication dialog (try using existing Base/GUI/vtkSlicerXNATPermissionPrompterWidget code) '''(wjp)'''
* extend FetchMI infrastructure to admit XNE web services '''(wjp)'''
* Improve/generalize http error checking 
** bring to superclass vtkFetchMIParser (right now, just looking for error string in html title in vtkFetchMIParserXND)
** recognize and report specific http error codes via vtkFetchMINode::ErrorMessage and vtkFetchMINode::RaiseErrorEvent
* Extend FetchMI's GUI to include an XNE notebook, switch notebooks when serverType changes. '''(wjp)'''
* Build vtkFetchMIQueryBuilder superclass '''(wjp)'''
* Derive vtkFetchMIQueryBuilderXND class and move XND query formulation into here. '''(wjp)'''
* Add vtkFetchMIQueryBuilderXNE class
* Add vtkFetchMIWriterXNE class
* Add vtkFetchMIParserXNE class
* Add vtkFetchMIWebServicesClientXNE class to encapsulate 
* Add vtkXNETagTable in Libs/RemoteIO
* Add vtkXNEHandler in Libs/RemoteIO