View source for Slicer3:Build/Modules

==News==

* TCON Tursday 03/13/08: Steve Pieper, Terry G Lorber 2nd, Alex Yarmarkovich, Sebastien BARRE et al.
* TCON Monday 01/28/08: Steve Pieper, Terry G Lorber 2nd, Alex Yarmarkovich, Sebastien BARRE

==Description==

In an effort to streamline the process of downloading, building, maintaining, testing and installing the Slicer3 project we are aiming at "deconstructing" the current <tt>[http://www.na-mic.org/ViewVC/index.cgi/trunk/Scripts/getbuildtest2.tcl?view=markup getbuildtest2.tcl]</tt> Tcl script into a set of smaller, re-usable and documented CMake scripts. This new framework will act as a backend to provide the same functionalities as <tt>getbuildtest2.tcl</tt> and extend it to support a new modular approach and additional features. 

===What <tt>getbuildtest2.tcl</tt> can do===

* download all libraries,<br><small>([http://cmake.org CMake], [http://tcl.tk Tcl/Tk], IncrTcl/Tk, IWidgets, BLT, [http://python.org Python], numpy, scipy, freetype2+matplotlib, [http://vtk.org VTK], [http://kwwidgets.org KWWidgets], [http://itk.org ITK], Teem, [http://igstk.org IGSTK], NaviTrack, DCMTK, [http://batchmake.org BatchMake], cmCurl)</small>
* update all libraries,
* configure all libraries,
* buid (and/or clean) all libraries,
* build Slicer3 (debug or release),
* test Slicer3,
* create (and upload) an installer for Slicer3,
* create the Slicer3 Doxygen documentation.

===What <tt>getbuildtest2.tcl</tt> can't do (so well)===

* it doesn't provide an easy, user-friendly way to specify your own support libraries, i.e.
** a specific version/tag, 
** a local source tree, 
** a local build tree, 
** a local installation.
''these are all configurable using the slicer_variables2.tcl script''
* it doesn't mesh so great within the CMake/CTest/CDash framework as a Tcl script, ''can you explain this one? -SP''
* it requires Tcl knowledge for part of the build process, whereas most support/external libraries and the rest of Slicer3 is using CMake.

===What we would like to do===

While we would like to address all the requirements above, some new concerns have also arisen over the past year as the Slicer3 project significantly grew up in size and exposure:
* the larger number of dependencies and support libraries provides a significant challenge in term of build process and maintenance,
* the larger number of external developers makes it increasingly difficult to make sure everyone can ''easily'' plug their own code/module inside Slicer3 with ''minimal'' impact on the application's stability and ''minimal'' knowledge of either Tcl and/or CMake.

===The modular approach===

In order to address the issues described so far, we are currently experimenting with the following concepts and creating the corresponding backend in CMake:
* treat support libs and external code as ''modules'',
* provide a simple way to describe a module and its dependencies in term of other modules,
* provide CMake scripts/functions to:
** parse a module description (local or remote),
** download the corresponding source repository (CVS or SVN) or specify your own,
** update the corresponding source repository,
** retrieve its dependencies automatically,
** allow modules to be enabled/disabled,
** configure all enabled modules,
** build all enabled modules,
** build, test, install and pack Slicer3 and its modules.

Scripts will be provided to allow developers to create various front-ends (see [http://na-mic.org/Wiki/index.php/Slicer3:Build_Instructions#SBuild SBuild]) and provide control at each step of the process (i.e., turn some modules ON/OFF through a list of checkboxes, solve dependencies locks, pick a specific version of a module, etc).

==Status==

* Experiments with the new BuildSystem and downloadable modules can be found in the [http://www.na-mic.org/ViewVC/index.cgi/trunk/BuildSystem/?root=NAMICSandBox NAMICSandBox/BuildSystem] directory.
* Given the growing complexity of Slicer3 (Python, really?), let's focus on the modules only, not the core libraries or the core Slicer; modules will be downloaded/configured/built against a Slicer3 that is built or installed already. We will tackle Slicer3 later.

===Requirements===

Slicer3 is one of the most complex application CMake is currently being used on. As such, we found ourselves adding new CMake functionalities and fixing new bugs, which will hopefully benefit others projects. This however means that you will need the latest and greatest CMake version to play with this example (i.e. the current [http://cmake.org/HTML/Download.html#cvs CMake CVS HEAD]). New features include the FILE(DOWNLOAD...) and LIST(REMOVE_DUPLICATES...) subcommands, better SET/GET_PROPERTY support, cleaner scoping through FUNCTION, etc. ''UPDATE'': you need a CMake updated as of March 12th, 2008.

===Quick Example===

Here is an excerpt from the  [http://www.na-mic.org/ViewVC/index.cgi/trunk/BuildSystem/CMakeLists.txt?root=NAMICSandBox&view=markup NAMICSandBox/BuildSystem/CMakeLists.txt] file. It will be updated as more functionalities are added to the scripts.

This example is pretty simplistic and has been tested on Linux and Win32 (nmake mode). It will automatically download and/or update several modules and external libraries from different locations.

<pre>
set(SLICER_LOCAL_MODULES_PATH "${CMAKE_CURRENT_SOURCE_DIR}/Modules")
set(SLICER_DOWNLOADED_SOURCES_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}/Sources")

# Bring the macros

include(SlicerParseModule)
include(SlicerEnableModule)
include(SlicerDownloadModule)

# Parse a remote module

slicer_parse_module_url("http://www.na-mic.org/ViewVC/index.cgi/trunk/BuildSystem/Modules/TestModule1/TestModule1.xml?root=NAMICSandBox&view=co")

# Parse a local module

slicer_parse_module_file("${SLICER_LOCAL_MODULES_PATH}/TestModule2/")

# Create all download/update targets
# "make update_modules" will download and update everything

slicer_create_download_and_update_modules_targets(${SLICER_DOWNLOADED_SOURCES_DIRECTORY})
</pre>

Run CMake on this <tt>BuildSystem</tt> source directory, from a new build-directory, then type:
   make update_modules

===Roadmap===

Development is likely to speed up as we learned how to stretch CMake capabilities and expand some of its features accordingly. Building blocks have been committed (see the 'References' section below), and we are building upon them. 

* resolution of inter-modules dependencies will be shown soon (not committed yet),
* simple frontend example from the CMake GUI itself: select module ON/OFF, show/hide experimental modules, etc. (soon),
* configuration of all modules,
* build all modules,
* etc.

==References==

===Modules===

Modules XML descriptions can be found in the [http://www.na-mic.org/ViewVC/index.cgi/trunk/BuildSystem/Modules/?root=NAMICSandBox NAMICSandBox/BuildSystem/Modules] directory. Each module description is kept in a separate subdirecty (as more files *may* crop up, per module). Keep in mind that the module description here is separate from the module source code itself, especially for those modules or external libraries that we do not have control of, for example.

CMake can parse a module description either from a string, a local file or a remote file. Later on, it is possible we maintain a ''module index'' either in the main Slicer3 SVN or online, which will point to different modules locations (think of it as a master index, the same way [http://cygwin.com/ Cygwin]'s setup.exe retrieves its list of packages). External developers should be able to expose their modules that way.

Here is the contents of the KWWidgets module description, which can be found in the [http://www.na-mic.org/ViewVC/index.cgi/trunk/BuildSystem/Modules/TestModule2/TestModule2.xml?root=NAMICSandBox&view=markup NAMICSandBox/BuildSystem/Modules/TestModule2/TestModule2.xml] file.

<pre>
<Name>TestModule2</Name>
<Group>Segmentation</Group>
<Description>Test module #2</Description>
<SourceLocation>:pserver:anoncvs:@public.kitware.com:/cvsroot/Sandbox</SourceLocation>
<CVSModule>Sandbox/barre/slicer3/TestModule2</CVSModule>
<HomePage>http://na-mic.org/Wiki/index.php/Slicer3:Build/Modules</HomePage>
<Version>1.0</Version>
<Dependency>TestModule1</Dependency>
<Author>Sebastien BARRE</Author>
<Acknowledgement>Kitware, Inc.</Acknowledgement>
</pre>

i.e. the name of the module is ''TestModule2'', part of the ''Segmentation'' group, and described as ''Test module #2''; it can be retrieved by checking out the CVS Module ''Sandbox/barre/slicer3/TestModule2'' from the CVS repository '':pserver:anoncvs:@public.kitware.com:/cvsroot/Sandbox''; its home page is ''http://na-mic.org/Wiki/index.php/Slicer3:Build/Modules''; Version ''1.0'' depends on the module ''TestModule1''; it was written by ''Sebastien BARRE'', with help ''Kitware, Inc.''.

Note: this is likely to change for a more readable format:

<pre>
Name: TestModule2
Group: Segmentation
Description: Test module #2
SourceLocation: :pserver:anoncvs:@public.kitware.com:/cvsroot/Sandbox
CVSModule: Sandbox/barre/slicer3/TestModule2
HomePage: http://na-mic.org/Wiki/index.php/Slicer3:Build/Modules
Version: 1.0
Dependency: TestModule1
Author: Sebastien BARRE
Acknowledgement: Kitware, Inc.
</pre>

===Introspecting a module===

See [http://www.na-mic.org/ViewVC/index.cgi/trunk/BuildSystem/CMake/SlicerSetGetModule.cmake?root=NAMICSandBox&view=markup NAMICSandBox/BuildSystem/CMake/SlicerSetGetModule.cmake] file.

<pre>
# slicer_set_module_value: set a module value.
# slicer_get_module_value: get a module value.
# slicer_unset_module_value: unset a module value.
# slicer_is_module_unknown: check if a module is unknown.
# slicer_get_module_short_description: get a module short description.
# slicer_get_module_source_repository_type: get a module source repository type.
</pre>

Example:

<pre>
slicer_set_module_value(TestModule Author "John Doe")
slicer_get_module_value(TestModule Author authors)
message("Author(s): ${author}")

slicer_set_module_value(TestModule MyList Elem1 Elem2 Elem3)
...
</pre>

===Parsing a module===

See [http://www.na-mic.org/ViewVC/index.cgi/trunk/BuildSystem/CMake/SlicerParseModule.cmake?root=NAMICSandBox&view=markup NAMICSandBox/BuildSystem/CMake/SlicerParseModule.cmake] file.

<pre>
# slicer_parse_module: parse a module.
# slicer_parse_module_file: parse a module from a file.
# slicer_parse_module_url: parse a module from a remote file.
# slicer_get_modules_list: get the list of parsed/known modules.
</pre>

Example:

<pre>
slicer_parse_module_file("C:/foo/TestModule/TestModule.xml" TestModule)
slicer_get_module_value(TestModule Name name)
message("Module name: ${name}")

slicer_parse_module_url("http://www.na-mic.org/modules/test/test.xml" TestModule)
...
</pre>

===Downloading a module===

See [http://www.na-mic.org/ViewVC/index.cgi/trunk/BuildSystem/CMake/SlicerDownloadModule.cmake?root=NAMICSandBox&view=markup NAMICSandBox/BuildSystem/CMake/SlicerDownloadModule.cmake] file.

<pre>
# slicer_create_download_module_target: create a download module target.
# slicer_get_download_module_target: get the name of a download module target.
# slicer_create_update_module_target: create an update module target.
# slicer_get_update_module_target: get the name of a update module target.
# slicer_create_download_and_update_modules_targets: create download and update
targets for all known modules.
</pre>

Please note that most of the smaller functions can be safely ignored but act as helper functions for higher-level commands. For example, one does not need to create each download and update targets for each module: <tt>slicer_create_download_and_update_modules_targets</tt> will do it for you, for all modules. But still, a finer granularity can't hurt and is exposed for frontends.

Example:

<pre>
slicer_parse_module_file("C:/foo/TestModule/TestModule.xml" TestModule)
slicer_parse_module_url("http://foo/bar/module/module.xml" TestModule2)
...
slicer_create_download_and_update_modules_targets("/src")
</pre>

===Enabling a module===

See [http://www.na-mic.org/ViewVC/index.cgi/trunk/BuildSystem/CMake/SlicerEnableModule.cmake?root=NAMICSandBox&view=markup NAMICSandBox/BuildSystem/CMake/SlicerEnableModule.cmake] file.

<pre>
# slicer_create_use_module_option: create an option to use a module.
</pre>

Example:

<pre>
slicer_parse_module_file("C:/foo/TestModule/TestModule.xml" TestModule)
slicer_create_use_module_option(TestModule USE_TEST_MODULE)
if(USE_TEST_MODULE)
  ...
endif(USE_TEST_MODULE)
</pre>

This function is a no-op at the moment, but once a module is turned ON, it will likely solve its dependencies automatically and bring the corresponding modules in.

===Module XML Description===

Things that the module's (XML) description should be able to tell:

* Name
* Group
* Description
* Source Location
* Home Page
* Dependencies (on other Groups or Modules and what versions and/or options)
* Version #
[and maybe:]
* Icon
* Author(s)
* Acknowledgment(s)

===Module Groups===

* Core (for support libraries)
* Base
* Segmentation
* Registration
* Filtering
* Diffusion Imaging/Tractography
* Modeling
* Meshing
* Image Guided Therapy
* Rendering
* Radiation Treatment
* Microscopy
* Astronomy
* Utilities
* Databases (XCEDE?)
* Other