Slicer3:Build/Modules

News

• TCON Wednesday 02/04/08: Steve Pieper, Terry G Lorber 2nd, Sebastien BARRE.
  • Steve to create a new SVN repository for modules,
  • Terry to finish converting GradientAnisotropicDiffusionFilter,
  • Sebastien to make sure GradientAnisotropicDiffusionFilter can be build against Slicer3 using the FIND_PACKAGE() framework, then use it as first example of modules discovered/downloaded/configured/built.
• TCON Tursday 03/13/08: Steve Pieper, Terry G Lorber 2nd, Alex Yarmarkovich, Sebastien BARRE et al.
• TCON Monday 01/28/08: Steve Pieper, Terry G Lorber 2nd, Alex Yarmarkovich, Sebastien BARRE

Description

In an effort to streamline the process of downloading, building, maintaining, testing and installing the Slicer3 project we are aiming at "deconstructing" the current getbuildtest2.tcl Tcl script into a set of smaller, re-usable and documented CMake scripts. This new framework will act as a backend to provide the same functionalities as getbuildtest2.tcl and extend it to support a new modular approach and additional features.

What getbuildtest2.tcl can do

• download all libraries,
  (CMake, Tcl/Tk, IncrTcl/Tk, IWidgets, BLT, Python, numpy, scipy, freetype2+matplotlib, VTK, KWWidgets, ITK, Teem, IGSTK, NaviTrack, DCMTK, BatchMake, cmCurl)
• update all libraries,
• configure all libraries,
• buid (and/or clean) all libraries,
• build Slicer3 (debug or release),
• test Slicer3,
• create (and upload) an installer for Slicer3,
• create the Slicer3 Doxygen documentation.

What getbuildtest2.tcl can't do (so well)

• it doesn't provide an easy, user-friendly way to specify your own support libraries, i.e.
  • a specific version/tag,
  • a local source tree,
  • a local build tree,
  • a local installation.

these are all configurable using the slicer_variables2.tcl script

• it doesn't mesh so great within the CMake/CTest/CDash framework as a Tcl script, can you explain this one? -SP
• it requires Tcl knowledge for part of the build process, whereas most support/external libraries and the rest of Slicer3 is using CMake.

What we would like to do

While we would like to address all the requirements above, some new concerns have also arisen over the past year as the Slicer3 project significantly grew up in size and exposure:

• the larger number of dependencies and support libraries provides a significant challenge in term of build process and maintenance,
• the larger number of external developers makes it increasingly difficult to make sure everyone can easily plug their own code/module inside Slicer3 with minimal impact on the application's stability and minimal knowledge of either Tcl and/or CMake.

The modular approach

In order to address the issues described so far, we are currently experimenting with the following concepts and creating the corresponding backend in CMake:

• treat support libs and external code as modules,
• provide a simple way to describe a module and its dependencies in term of other modules,
• provide CMake scripts/functions to:
  • parse a module description (local or remote),
  • download the corresponding source repository (CVS or SVN) or specify your own,
  • update the corresponding source repository,
  • retrieve its dependencies automatically,
  • allow modules to be enabled/disabled,
  • configure all enabled modules,
  • build all enabled modules,
  • build, test, install and pack Slicer3 and its modules.

Scripts will be provided to allow developers to create various front-ends (see SBuild) and provide control at each step of the process (i.e., turn some modules ON/OFF through a list of checkboxes, solve dependencies locks, pick a specific version of a module, etc).

Workflow

Tentative workflow:

1. download Slicer3 base,
2. configure Slicer3,
   1. configure Slicer3 base itself, so that the export files are created (and used by the modules using FIND_PACKAGE()).
   2. configure the modules automatically, create the targets to download, configure (i.e. run a CMake instance on each module separately with a parameter pointer at the Slicer3 directory to help FIND_PACKAGE), and build the modules.
   3. link the targets together so that 'make modules' will download/config/build each module

At the end of the day, 'make all' should build Slicer3 with default modules, but one should be able to type 'make configure_modules' to turn modules ON/OFF (which will eventually download more modules).

Status

• Experiments with the new BuildSystem and downloadable modules can be found in the NAMICSandBox/BuildSystem directory.
• Given the growing complexity of Slicer3 (Python, really?), let's focus on the modules only, not the core libraries or the core Slicer; modules will be downloaded/configured/built against a Slicer3 that is built or installed already. We will tackle Slicer3 later.

Requirements

Slicer3 is one of the most complex application CMake is currently being used on. As such, we found ourselves adding new CMake functionalities and fixing new bugs, which will hopefully benefit others projects. This however means that you will need the latest and greatest CMake version to play with this example (i.e. the current CMake CVS HEAD). New features include the FILE(DOWNLOAD...) and LIST(REMOVE_DUPLICATES...) subcommands, better SET/GET_PROPERTY support, cleaner scoping through FUNCTION, etc. UPDATE: you need a CMake updated as of March 12th, 2008.

Quick Example

Here is an excerpt from the NAMICSandBox/BuildSystem/CMakeLists.txt file. It will be updated as more functionalities are added to the scripts.

This example is pretty simplistic and has been tested on Linux and Win32 (nmake mode). It will automatically download and/or update several modules and external libraries from different locations.

set(SLICER_LOCAL_MODULES_PATH "${CMAKE_CURRENT_SOURCE_DIR}/Modules")
set(SLICER_DOWNLOADED_SOURCES_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}/Sources")

# Bring the macros

include(SlicerParseModule)
include(SlicerEnableModule)
include(SlicerDownloadModule)

# Parse a remote module

slicer_parse_module_url("http://www.na-mic.org/ViewVC/index.cgi/trunk/BuildSystem/Modules/TestModule1/TestModule1.xml?root=NAMICSandBox&view=co")

# Parse a local module

slicer_parse_module_file("${SLICER_LOCAL_MODULES_PATH}/TestModule2/")

# Create USE_FOOBAR options for all modules

slicer_get_parsed_modules(modules)
slicer_create_use_modules_options("${modules}")
slicer_get_used_modules("${modules}" used_modules)

# Find the unresolved dependencies, and the resolved modules

slicer_get_unresolved_modules_dependencies(
  "${used_modules}" unresolved_dependencies)
slicer_get_resolved_modules(
  "${used_modules}" resolved_modules)

# Create all download/update targets for the resolved modules

slicer_create_download_and_update_modules_targets(
  "${resolved_modules}" "${SLICER_DOWNLOADED_SOURCES_DIRECTORY}")

Run CMake on this BuildSystem source directory, from a new build-directory, then type:

  make update_modules

Roadmap

Development is likely to speed up as we learned how to stretch CMake capabilities and expand some of its features accordingly. Building blocks have been committed (see the 'References' section below), and we are building upon them.

• resolution of inter-modules dependencies will be shown soon (not committed yet),
• simple frontend example from the CMake GUI itself: select module ON/OFF, show/hide experimental modules, etc. (soon),
• configuration of all modules,
• build all modules,
• etc.

References

Modules

Modules XML descriptions can be found in the NAMICSandBox/BuildSystem/Modules directory. Each module description is kept in a separate subdirecty (as more files *may* crop up, per module). Keep in mind that the module description here is separate from the module source code itself, especially for those modules or external libraries that we do not have control of, for example.

CMake can parse a module description either from a string, a local file or a remote file. Later on, it is possible we maintain a module index either in the main Slicer3 SVN or online, which will point to different modules locations (think of it as a master index, the same way Cygwin's setup.exe retrieves its list of packages). External developers should be able to expose their modules that way.

Here is the contents of the KWWidgets module description, which can be found in the NAMICSandBox/BuildSystem/Modules/TestModule2/TestModule2.xml file.

<Name>TestModule2</Name>
<Group>Segmentation</Group>
<Description>Test module #2</Description>
<SourceLocation>:pserver:anoncvs:@public.kitware.com:/cvsroot/Sandbox</SourceLocation>
<CVSModule>Sandbox/barre/slicer3/TestModule2</CVSModule>
<HomePage>http://na-mic.org/Wiki/index.php/Slicer3:Build/Modules</HomePage>
<Version>1.0</Version>
<Dependency>TestModule1</Dependency>
<Author>Sebastien BARRE</Author>
<Acknowledgement>Kitware, Inc.</Acknowledgement>

i.e. the name of the module is TestModule2, part of the Segmentation group, and described as Test module #2; it can be retrieved by checking out the CVS Module Sandbox/barre/slicer3/TestModule2 from the CVS repository :pserver:anoncvs:@public.kitware.com:/cvsroot/Sandbox; its home page is http://na-mic.org/Wiki/index.php/Slicer3:Build/Modules; Version 1.0 depends on the module TestModule1; it was written by Sebastien BARRE, with help Kitware, Inc..

Note: this is likely to change for a more readable format:

Name: TestModule2
Group: Segmentation
Description: Test module #2
SourceLocation: :pserver:anoncvs:@public.kitware.com:/cvsroot/Sandbox
CVSModule: Sandbox/barre/slicer3/TestModule2
HomePage: http://na-mic.org/Wiki/index.php/Slicer3:Build/Modules
Version: 1.0
Dependency: TestModule1
Author: Sebastien BARRE
Acknowledgement: Kitware, Inc.

Introspecting a module

See NAMICSandBox/BuildSystem/CMake/SlicerSetGetModule.cmake file.

# slicer_set_module_value: set a module value.
# slicer_get_module_value: get a module value.
# slicer_unset_module_value: unset a module value.
# slicer_is_module_unknown: check if a module is unknown.
# slicer_get_module_short_description: get a module short description.
# slicer_get_module_source_repository_type: get a module source repository type.
# slicer_get_module_source_tag: get a module source tag.
# slicer_get_module_cache_directory: get a module cache directory.

Example:

slicer_set_module_value(TestModule Author "John Doe")
slicer_get_module_value(TestModule Author authors)
message("Author(s): ${author}")

slicer_set_module_value(TestModule MyList Elem1 Elem2 Elem3)
...

Parsing a module

See NAMICSandBox/BuildSystem/CMake/SlicerParseModule.cmake file.

# slicer_parse_module: parse a module.
# slicer_parse_module_file: parse a module from a file.
# slicer_parse_module_url: parse a module from a remote file.
# slicer_parse_modules_directory: parse modules sub-directories in a directory.
# slicer_get_parsed_modules: get the list of parsed modules.

Example:

slicer_parse_module_file("C:/foo/TestModule/TestModule.xml" TestModule)
slicer_get_module_value(TestModule Name name)
message("Module name: ${name}")

slicer_parse_module_url("http://www.na-mic.org/modules/test/test.xml" TestModule)
...

Downloading a module

See NAMICSandBox/BuildSystem/CMake/SlicerDownloadModule.cmake file.

# slicer_create_download_module_target: create a download module target.
# slicer_get_download_module_target: get the name of a download module target.
# slicer_create_update_module_target: create an update module target.
# slicer_get_update_module_target: get the name of a update module target.
# slicer_create_download_and_update_modules_targets: create download and update
targets for all known modules.

Please note that most of the smaller functions can be safely ignored but act as helper functions for higher-level commands. For example, one does not need to create each download and update targets for each module: slicer_create_download_and_update_modules_targets will do it for you, for all modules. But still, a finer granularity can't hurt and is exposed for frontends.

Example:

slicer_parse_module_file("C:/foo/TestModule/TestModule.xml" TestModule)
slicer_parse_module_url("http://foo/bar/module/module.xml" TestModule2)
...
slicer_create_download_and_update_modules_targets("/src")

Enabling a module

See NAMICSandBox/BuildSystem/CMake/SlicerEnableModule.cmake file.

# slicer_create_use_module_option: create an option to use a module.
# slicer_create_use_modules_options: create options to use specific modules.
# slicer_get_used_modules: get the list of used modules.
# slicer_get_unresolved_modules_dependencies: get the unresolved modules dependencies for specific modules 
# slicer_get_resolved_modules: get the list of resolved modules for specific modules 

Example:

slicer_parse_module_file("C:/foo/TestModule/TestModule.xml" TestModule)
slicer_create_use_module_option(TestModule USE_TEST_MODULE)
if(USE_TEST_MODULE)
  ...
endif(USE_TEST_MODULE)
slicer_get_parsed_modules(modules)
slicer_get_used_modules("${modules}" used_modules)
slicer_get_resolved_modules("${used_modules}" resolved_modules)

Module XML Description

Things that the module's (XML) description should be able to tell:

• Name
• Group
• Description
• Source Location
• Home Page
• Dependencies (on other Groups or Modules and what versions and/or options)
• Version #

[and maybe:]

• Icon
• Author(s)
• Acknowledgment(s)

Module Groups

• Core (for support libraries)
• Base
• Segmentation
• Registration
• Filtering
• Diffusion Imaging/Tractography
• Modeling
• Meshing
• Image Guided Therapy
• Rendering
• Radiation Treatment
• Microscopy
• Astronomy
• Utilities
• Databases (XCEDE?)
• Other