2006 IGT Workshop Panel Discussion

Back to workshop agenda

Panel Chair: Jim Duncan

Panelists: Russ Taylor, Steve Pizer, and Presenters.

Guidelines

Guidelines for panel discussion:

1. Questions from the audience on the morning session
2. Challenges proposed by presenters in the morning session
3. Are there any low-hanging fruit? anything that NCIGT and NA-MIC can help with?
4. Are there any specific issues that should be added to the breakout discussions agenda?
5. Any opportunities for problems that could be solved with industry-academia partnerships?

Jim's Introduction

Some Common Themes in Challenges proposed by Speakers

1. we need to do a better job at defining IGT systems requirements
2. we need to develop useful data for evaluation and validation based on models, simulations, actual patient data; also need performance standards/gold standards to compare to (breakout session 2)
3. we need to develop better/additional hardware and software standards for IGT (breakout 1 ?)
4. we need to develop algorithm repositories for open source IGT software/hardware solutions– within this want to be able to leverage existing toolkits and fill toolkit gaps furthermore, need to coordinate research in tracking technologies (breakout 4)
5. we need to provide a strong rationale/motivation to manufacturers to provide API’s/ research interfaces (part of breakout 3 ?)
6. need to think about system design approaches that indicate algorithm confidence in produced result;
7. need to develop systems that capitalize on incorporating robust training data into system decision making (robust across normal/abnormal)

Discussion

(from Jim's slide) Common issue

• System requirements is needed
• Needs to have "flight recording"

---

• How to specify requirements?
  • Is there something inbetween highly rigorous industrial practices and back of the envelope specifications?
  • synchronization of code and requirements
  • different scenarios, could be iterative process = no single requirement specification
    • often need refinement
    • ISO9000 lite
  • see FDA requirements
    • need method for reporting problems

• Should we collect and disseminate case studies?
  • Workshop?
  • record processes = best practices manual?
  • sell the development process...
• How do we develop guidelines for validation?
  • need mechanism to pool our results
  • testing: gathing information, evaluation metrics
  • case data costs money
  • uniform methods/tools for gathering data (e.g., OpenTracker)
  • see Nafis study for methodology for tracking validation - example of how to share methods?
  • NEED consensus on how to evaluate "correctness"
  • "Open Data", just like "Open Source" used to be?
  • suggestion: when designing IRBs, try to broaden a little bit so that the results can be used more widely or more effectively.
  • patient advocacy groups - mechanisms to make data available for patients to share if they wish,

• Do we need to develop better/additional hardware and software standards for IGT?
  • let the software be the driver by using "open source", at least initially
  • standards problem = no standards and too many standards simultaneously (standards, but not enough agreement) - this is probably not a research problem - where should it come from?
  • e.g., motivation for APIs exist for trackers (due to demand of customer), but for others (e.g., interventional imaging systems) there is less motivation for the manufacturer.
  • NIH play a role in funding industry to develop these standard interfaces (APIs?)
  • portability of standards is important (e.g., explicit support of legacy systems)
  • what is the value proposition for industry to invest into opening interfaces or developing standards?
    • there will always be competing standards - it is up to the marketplace which will win - where is the marketplace located in our community?
    • therapy is smaller niche than diagnostic, how to convince manufacturers of new methods
    • work with imaging people who also need to open up interfaces for new diagnostic methods
    • Chris Hasser: incentive to undustry - take advantage of resources, brain power that are being brought to bear by the research community - low cost and accelerated prototyping
    • shouldn't underestimate amount of engineering required to produce stable API
    • existing hooks exist in most vendor's equipment - until vendors agree on standards, community can contribute work to interface with these interfaces under research agreements,
    • Guy Schechter (Philips): these things are understood
      • need to take the lead and chrystallize the needs (leverage the power of the group)
      • but real power is buying power? research community is often weak!
      • critical mass (need to convince a few companies)
      • standards may need to be narrow in their respective areas in order to be feasible (broad standards will take too much time, or not happen at all)

• Develop algorithm repositories?
  • awareness - disseminate links to information about things like OpenTracker
  • participants of this workshop should add these links on the workshop wiki
  • perhaps recast: ask companies to permit us to provide a uniform interface, with having them be forced to comply with the interface.
  • hardware knowledge repositories should also be considered,
  • Open Source (1. freely available use) (2. distributed community of contributors)
    • should we choose prototype applications to drive an open interaction?
    • heirarchy of need - simple lab-based tests, versus complex assembly of many modules (IGT-lite and IGT-serious)
    • evolutionary process of developing open collaborations
  • PeterK - how about a forum for user evaluation and comments? (e.g., just like Amazon customer reviews)
    • Insight Journal could host such discussions. Forum for discussion or review.
      • what is the incentive for the review? danger in review bias.
      • won't this just end up evolving into peer review?
  • This community could come together to write requirements to give to industry - agree on common elements of functionality.
    • Larry Clarke: there is dialog between NIST and FDA on this topic
  • FDA/legal/regulatory impact on opening access to machine

• Database of simulators (see work in the MRI neuro-imaging world)

• Is there some need for focused study section in this area?
  • absence of reference standards for judging IGT research proposals
  • the community is its own worst enemy - study section
  • what role can NIH play?
  • new study section?
  • how many IGT proposals are actually going in.
    • there is a coding for diseases, but not for technologies, so this is difficult to track. keywords? this group could help define these keywords.
    • e.g., what should we call this community?
    • goal for breakouts: identify keywords