Difference between revisions of "NaviTrack Tutorial:Integrating:Passing image data"

From NAMIC Wiki
Jump to: navigation, search
 
(14 intermediate revisions by 2 users not shown)
Line 1: Line 1:
You can pass image data from one application to another through NaviTrack in almost same way as passing coordinate data.
+
=Overview for passing image data=
 +
 
 +
NaviTrack have a capability to send any C++ data types through NaviTrack data stream.
 +
That means, once you define a C++ class, you can transfer any kind of data even through
 +
a network, in quite same way as you did for coordinate data in previous tutorial pages.
 +
 
 +
For an image transfer, NaviTrack has already had a "Image" class to pass image data.
 +
This class has a few member variables including image size and image itself,
 +
as well as memberfunctions to serialize image data.
 +
Serialized image data is pushed into NaviTrack data flow as an event, and transfered to
 +
other nodes.
 +
 
 +
Image class is defined in include/OpenTracker/types/Image.h and src/input/types/Image.cxx.
  
 
=Sending side=
 
=Sending side=
 
==Modify MyTutorialModule==
 
==Modify MyTutorialModule==
To send data through NaviTrack, it is necessary to generate a new event, which contains the data.
+
To send data through NaviTrack, a new event is created.
The following code is implemented into MyTutorialModule to be called from your application.
+
The following code shows a basic idea of implementation:
void MyTutorialModule::setTracker(std::vector<float> pos,std::vector<float> quat)
+
 
{
+
  void MyTutorialModule::setImage(Image& img, int w, int h)
  if (pos.size() != 3 || quat.size() != 4) {
+
  {   
    std::cout << "MyTutorialModule::setTracker(): illegal vector size." << std::endl;
+
    if(source!=NULL)
    return;
+
      {
  }
+
        Event event;
   
+
        event.setAttribute("image", img);
  if(source!=NULL)
+
        event.setAttribute("xsize", w);
    {
+
        event.setAttribute("ysize", h);
      ot::Event *event = new ot::Event();
+
        event.timeStamp();
      event->setAttribute("position",pos);
+
        source->updateObservers( event );
      event->setAttribute("orientation",quat);
+
      }
      event->timeStamp();
+
  }
      source->updateObservers( *event );
+
 
    }
 
}
 
  
==pushpos.cxx==
+
Due to NaviTrack's bug, however, above code might not work well for lage image in Windows environment (see the last section in this page). To avoid this problem, we need to split a image into several fragments as following code:
 +
 
 +
  void MyTutorialModule::setImage(Image& img, int w, int h)
 +
  {
 +
    //static short* pix = NULL;
 +
   
 +
    if(source!=NULL)
 +
      {
 +
        // split a image
 +
        int nlines  = MAXIMUM_IMAGE_SIZE / (w*sizeof(short));
 +
        int nblocks = h / nlines;
 +
        if (h % nlines > 0) nblocks ++;
 +
 
 +
        for (int n = 0; n < nblocks; n ++)
 +
          {
 +
            int ysize;
 +
            int endflag = 0;
 +
 
 +
            if (n == nblocks-1)
 +
              {
 +
                ysize = h-nlines*n;
 +
                endflag = 1;
 +
              }
 +
            else
 +
              ysize = nlines;
 +
 
 +
            short* ba = (short*)img.image_ptr;
 +
            Image fragment(w,ysize,sizeof(short),&ba[w*n*nlines]);
 +
             
 +
 
 +
            //ot::Event event;
 +
            ot::Event event;
 +
            event.setAttribute("endflag",  endflag);
 +
            event.setAttribute("xsize",    w);
 +
            event.setAttribute("ysize",    ysize);
 +
            event.setAttribute("totalysize",h);
 +
            event.setAttribute("line",      n*nlines);
 +
            event.setAttribute("image",    fragment);
 +
            event.setAttribute("index",    n);
 +
 
 +
            event.timeStamp();
 +
            source->updateObservers( event );
 +
            this->context->loopOnce();
 +
            //usleep(pushImageInterval);
 +
            OSUtils::sleep(pushImageInterval);
 +
          }
 +
      }
 +
  }
 +
 
 +
==pushimage.cxx==
 +
 
 +
The core part of image sending would be like following code:
 +
 
 +
  MyTutorialModule* mtm = (MyTutorialModule*) context.getModule("MyTutorialConfig");
 +
 
 +
  int stopflag = 0;
 +
  int i = 0;
 +
  while (stopflag == 0) {
 +
   
 +
    // load image data to image_data here.
 +
      ....
 +
   
 +
    Image img(width, heigt, sizeof(short), (char*)image_data);
 +
   
 +
    if (mtm) {
 +
      mtm->setImage(img, w, h);
 +
    }
 +
   
 +
    stopflag = context.loopOnce();
 +
   
 +
    usleep(rate);
 +
  }
  
 
=Receiving side=
 
=Receiving side=
 
==Modify MyTutorialSink and MyTutorialModule==
 
==Modify MyTutorialSink and MyTutorialModule==
  
An event to pull coordinate data is handled in onEventGenerated() function in MyTutorialSink.
+
MyTutorialSink.h: Add member variables to hold the coordinate data:
There are two steps to pass the coordinate data to your application: STEP 1) onEventGenerated() stores the data into member variables in MyTutorialSinke; STEP 2) your application fetch the data from MyTutorialSink through MyTutorialModule.
+
  private:
 +
    Image image;
 +
    int width;
 +
    int height;
  
Bellows are examples to implement STEP 1 and STEP 2.
+
MyTutorialSink.cxx (basic idea):
 +
  void MyTutorialSink::onEventGenerated( Event& event, Node& generator)
 +
  {
 +
    if (event.hasAttribute("image") &&
 +
        event.hasAttribute("xsize") &&
 +
        event.hasAttribute("ysize"))
 +
      {
 +
        image = event.getAttribute((Image*)NULL, "image");
 +
        width  = event.getAttribute(std::string("xsize"),0);
 +
        height = event.getAttribute(std::string("ysize"),0);
 +
      }
 +
    else
 +
      {
 +
        width  = 0;
 +
        height = 0;
 +
      }
 +
  }
  
MyTutorialSink.h: Add member variables to hold the coordinate data:
 
  private:
 
    std::vector<float> position;
 
    std::vector<float> orientation;
 
  
MyTutorialSink.cxx:
+
MyTutorialSink.cxx (implemented for this tutorial):
 
   void MyTutorialSink::onEventGenerated( Event& event, Node& generator)
 
   void MyTutorialSink::onEventGenerated( Event& event, Node& generator)
 
   {
 
   {
 +
    //std::cerr << "MyTutorialSink::onEventGenerated()" << std::endl;
 
     if (event.hasAttribute("position"))
 
     if (event.hasAttribute("position"))
 
       for(int i = 0; i < 3; i ++)
 
       for(int i = 0; i < 3; i ++)
Line 55: Line 152:
 
           orientation[i]= event.getOrientation()[i];
 
           orientation[i]= event.getOrientation()[i];
 
         }
 
         }
        std::cout << "orientation !!!" << std::endl;
 
 
       }
 
       }
 
     else
 
     else
Line 63: Line 159:
 
         orientation[2]=0.0;
 
         orientation[2]=0.0;
 
         orientation[3]=0.0;
 
         orientation[3]=0.0;
 +
      }
 +
     
 +
 
 +
    if (event.hasAttribute("image") &&
 +
        event.hasAttribute("xsize") &&
 +
        event.hasAttribute("ysize") &&
 +
        event.hasAttribute("totalysize") &&
 +
        event.hasAttribute("line") //&&
 +
        //event.hasAttribute("endflag")
 +
        )
 +
      {
 +
        Image segimg  = event.getAttribute((Image*)NULL, "image");
 +
        width_buf      = event.getAttribute(std::string("xsize"),0);
 +
        height_buf    = event.getAttribute(std::string("totalysize"),0);
 +
        int segheight  = event.getAttribute(std::string("ysize"),0);
 +
        int line      = event.getAttribute(std::string("line"),0);
 +
        int endflag    = event.getAttribute(std::string("endflag"), 0);
 +
        int index      = event.getAttribute(std::string("index"), 0);
 +
   
 +
        if (!image_buf) {
 +
          image_buf = new Image();
 +
          image_buf->SetSize(width_buf, height_buf, sizeof(short));
 +
          image_buf->image_ptr = (void*)malloc(image_buf->size());
 +
        } else if (image_buf->size() != width_buf * height_buf * sizeof(short)) {
 +
          free(image_buf->image_ptr);
 +
          image_buf->SetSize(width_buf, height_buf, sizeof(short));
 +
          image_buf->image_ptr = (void*)malloc(image_buf->size());
 +
        }
 +
        short* dest = (short*)image_buf->image_ptr;
 +
 
 +
        if (width_buf*segheight > 0)
 +
          memcpy(&dest[width_buf*line], segimg.image_ptr, width_buf*segheight*sizeof(short));
 +
        if (endflag)
 +
          {
 +
            // switch image buffer
 +
            Image* tmp;
 +
            int    tmp_width, tmp_height;
 +
            tmp      = image;
 +
            image    = image_buf;
 +
            image_buf = tmp;
 +
 
 +
            tmp_width = width;
 +
            width    = width_buf;
 +
            width_buf = tmp_width;
 +
 
 +
            tmp_height= height;
 +
            height    = height_buf;
 +
            height_buf= tmp_height;
 +
 
 +
            timestamp = event.time;
 +
          }
 +
      }
 +
    else
 +
      {
 +
        width  = 0;
 +
        height = 0;
 
       }
 
       }
 
   }
 
   }
  
MyTutorialModule.cxx (and .h): add the follwing function:
+
==pullimage.cxx==
   void MyTutorialModule::getTracker(std::vector<float>& pos,std::vector<float>& quat)
+
 
  {
+
   MyTutorialModule* mtm = (MyTutorialModule*) context.getModule("MyTutorialConfig");
    //std::cout << "MyTutorialModule::getTracker() is called." << std::endl;
 
 
    
 
    
     if(sink!=NULL)
+
  while (stopflag == 0) {
       {
+
   
        sink->getTracker(pos, quat);
+
    stopflag = context.loopOnce();  // push and pull Event
 +
   
 +
     if (mtm) {
 +
      Image img;
 +
      int w, h;
 +
        
 +
      mtm->getImage(img, w, h);
 +
      if (img.size() != 0) {
 +
       
 +
        // do something for the received image here
 +
        ...
 +
     
 
       }
 
       }
 +
    }
 +
 
 +
    usleep(rate);
 
   }
 
   }
 
==pullpos.cxx==
 
  
 
=Testing=
 
=Testing=
Line 85: Line 248:
  
 
In terminal 2 (receiving side: pullimage)
 
In terminal 2 (receiving side: pullimage)
  $ ./pullimage tutorial_source.xml
+
  $ ./pullimage tutorial_sink.xml
  
 +
=Current Probelms on Image Transfer using NaviTrack=
  
!!NOTE!! You may encounter application errors, while passing image data through a network using NaviTrack NetworkModule.
+
==Microsoft Windows issue==
 +
In case that the image size exceeds approx 8000 bytes (uncertain), data transfer from NetworkSink to
 +
NetworkSource might fail in Windows environment. This is thought to be due to an implementation of OpenTracker or
 +
ACE library, which NaviTrack depends on, but we are not sure the reason, and we continue to work on this issue.
 +
At this moment, to avoid this problem, the programs used in this tutorial splits image data into several
 +
fragments with size of less than 8000 bytes in a pushimage program, and reconstruct a complete image from
 +
fragments in a pullimage program.
 +
 
 +
==UDP issue==
 +
You may encounter application errors, while passing image data through a network using NaviTrack NetworkModule.
 
This is because NaviTrack NetworkModule uses UDP for a communication between Sink and Source.
 
This is because NaviTrack NetworkModule uses UDP for a communication between Sink and Source.
 
Each data has header information in this communication including entire size of the data,
 
Each data has header information in this communication including entire size of the data,
 
and packet loss causes inconsistency between data size information in header and actual data size.
 
and packet loss causes inconsistency between data size information in header and actual data size.
NaviTrack has no capability to recover this kind of error in the current version.
+
Currently, NaviTrack has no capability to recover this kind of errors.
 +
 
 +
=Notes=
 +
If you don't have a good viewer for '''pgm''' imges on your machine, try to download and install ImageJ:
 +
http://rsb.info.nih.gov/ij/download.html
  
 
Back to [[NaviTrack_Tutorial:Integrating_into_your_application|Integrating into your application]].
 
Back to [[NaviTrack_Tutorial:Integrating_into_your_application|Integrating into your application]].

Latest revision as of 14:48, 15 June 2007

Home < NaviTrack Tutorial:Integrating:Passing image data

Overview for passing image data

NaviTrack have a capability to send any C++ data types through NaviTrack data stream. That means, once you define a C++ class, you can transfer any kind of data even through a network, in quite same way as you did for coordinate data in previous tutorial pages.

For an image transfer, NaviTrack has already had a "Image" class to pass image data. This class has a few member variables including image size and image itself, as well as memberfunctions to serialize image data. Serialized image data is pushed into NaviTrack data flow as an event, and transfered to other nodes.

Image class is defined in include/OpenTracker/types/Image.h and src/input/types/Image.cxx.

Sending side

Modify MyTutorialModule

To send data through NaviTrack, a new event is created. The following code shows a basic idea of implementation:

 void MyTutorialModule::setImage(Image& img, int w, int h)
 {  
   if(source!=NULL)
     {
       Event event;
       event.setAttribute("image", img);
       event.setAttribute("xsize", w);
       event.setAttribute("ysize", h);
       event.timeStamp();
       source->updateObservers( event );
     }
 }


Due to NaviTrack's bug, however, above code might not work well for lage image in Windows environment (see the last section in this page). To avoid this problem, we need to split a image into several fragments as following code:

 void MyTutorialModule::setImage(Image& img, int w, int h)
 {
   //static short* pix = NULL;
   
   if(source!=NULL)
     {
       // split a image
       int nlines  = MAXIMUM_IMAGE_SIZE / (w*sizeof(short));
       int nblocks = h / nlines;
       if (h % nlines > 0) nblocks ++;
 
       for (int n = 0; n < nblocks; n ++) 
         {
           int ysize;
           int endflag = 0;
 
           if (n == nblocks-1)
             {
               ysize = h-nlines*n;
               endflag = 1;
             }
           else
             ysize = nlines;
 
           short* ba = (short*)img.image_ptr;
           Image fragment(w,ysize,sizeof(short),&ba[w*n*nlines]);
             
 
           //ot::Event event;
           ot::Event event;
           event.setAttribute("endflag",   endflag);
           event.setAttribute("xsize",     w);
           event.setAttribute("ysize",     ysize);
           event.setAttribute("totalysize",h);
           event.setAttribute("line",      n*nlines);
           event.setAttribute("image",     fragment);
           event.setAttribute("index",     n);
 
           event.timeStamp();
           source->updateObservers( event );
           this->context->loopOnce();
           //usleep(pushImageInterval);
           OSUtils::sleep(pushImageInterval);
         }
     }
 }

pushimage.cxx

The core part of image sending would be like following code:

 MyTutorialModule* mtm = (MyTutorialModule*) context.getModule("MyTutorialConfig");
 
 int stopflag = 0;
 int i = 0;
 while (stopflag == 0) {
   
    // load image data to image_data here.
     ....
   
   Image img(width, heigt, sizeof(short), (char*)image_data);
   
   if (mtm) {
     mtm->setImage(img, w, h);
   }
   
   stopflag = context.loopOnce();
   
   usleep(rate);
 }

Receiving side

Modify MyTutorialSink and MyTutorialModule

MyTutorialSink.h: Add member variables to hold the coordinate data:

 private:
   Image image;
   int width;
   int height;

MyTutorialSink.cxx (basic idea):

 void MyTutorialSink::onEventGenerated( Event& event, Node& generator)
 {
   if (event.hasAttribute("image") &&
       event.hasAttribute("xsize") &&
       event.hasAttribute("ysize"))
     {
       image = event.getAttribute((Image*)NULL, "image");
       width  = event.getAttribute(std::string("xsize"),0);
       height = event.getAttribute(std::string("ysize"),0);
     }
   else
     {
       width  = 0;
       height = 0;
     }
 }


MyTutorialSink.cxx (implemented for this tutorial):

 void MyTutorialSink::onEventGenerated( Event& event, Node& generator)
 {
   //std::cerr << "MyTutorialSink::onEventGenerated()" << std::endl;
   if (event.hasAttribute("position"))
     for(int i = 0; i < 3; i ++)
       position[i]=event.getPosition()[i];
   else
     {
       position[0]=0.0;
       position[1]=0.0;
       position[2]=0.0;
     }
 
   if (event.hasAttribute("orientation"))
     {
       for  (int i = 0; i < 4; i ++) {
         orientation[i]= event.getOrientation()[i];
       }
     }
   else
     {
       orientation[0]=0.0;
       orientation[1]=0.0;
       orientation[2]=0.0;
       orientation[3]=0.0;
     }
     
 
   if (event.hasAttribute("image") &&
       event.hasAttribute("xsize") &&
       event.hasAttribute("ysize") &&
       event.hasAttribute("totalysize") &&
       event.hasAttribute("line") //&&
       //event.hasAttribute("endflag")
       )
     {
       Image segimg   = event.getAttribute((Image*)NULL, "image");
       width_buf      = event.getAttribute(std::string("xsize"),0);
       height_buf     = event.getAttribute(std::string("totalysize"),0);
       int segheight  = event.getAttribute(std::string("ysize"),0);
       int line       = event.getAttribute(std::string("line"),0);
       int endflag    = event.getAttribute(std::string("endflag"), 0);
       int index      = event.getAttribute(std::string("index"), 0);
   
       if (!image_buf) {
         image_buf = new Image();
         image_buf->SetSize(width_buf, height_buf, sizeof(short));
         image_buf->image_ptr = (void*)malloc(image_buf->size());
       } else if (image_buf->size() != width_buf * height_buf * sizeof(short)) {
         free(image_buf->image_ptr);
         image_buf->SetSize(width_buf, height_buf, sizeof(short));
         image_buf->image_ptr = (void*)malloc(image_buf->size());
       }
       short* dest = (short*)image_buf->image_ptr;
 
       if (width_buf*segheight > 0)
         memcpy(&dest[width_buf*line], segimg.image_ptr, width_buf*segheight*sizeof(short));
       if (endflag)
         {
           // switch image buffer
           Image* tmp;
           int    tmp_width, tmp_height;
           tmp       = image;
           image     = image_buf;
           image_buf = tmp;
 
           tmp_width = width;
           width     = width_buf;
           width_buf = tmp_width;
 
           tmp_height= height;
           height    = height_buf;
           height_buf= tmp_height;
 
           timestamp = event.time;
         }
     }
   else
     {
       width  = 0;
       height = 0;
     }
 }

pullimage.cxx

 MyTutorialModule* mtm = (MyTutorialModule*) context.getModule("MyTutorialConfig");
 
 while (stopflag == 0) {
   
   stopflag = context.loopOnce();   // push and pull Event
   
   if (mtm) {
     Image img;
     int w, h;
     
     mtm->getImage(img, w, h);
     if (img.size() != 0) {
        
       // do something for the received image here
        ...
     
     }
   }
 
   usleep(rate);
 }

Testing

In terminal 1 (sending side: pushimage)

$ ./pushimage tutorial_source.xml

In terminal 2 (receiving side: pullimage)

$ ./pullimage tutorial_sink.xml

Current Probelms on Image Transfer using NaviTrack

Microsoft Windows issue

In case that the image size exceeds approx 8000 bytes (uncertain), data transfer from NetworkSink to NetworkSource might fail in Windows environment. This is thought to be due to an implementation of OpenTracker or ACE library, which NaviTrack depends on, but we are not sure the reason, and we continue to work on this issue. At this moment, to avoid this problem, the programs used in this tutorial splits image data into several fragments with size of less than 8000 bytes in a pushimage program, and reconstruct a complete image from fragments in a pullimage program.

UDP issue

You may encounter application errors, while passing image data through a network using NaviTrack NetworkModule. This is because NaviTrack NetworkModule uses UDP for a communication between Sink and Source. Each data has header information in this communication including entire size of the data, and packet loss causes inconsistency between data size information in header and actual data size. Currently, NaviTrack has no capability to recover this kind of errors.

Notes

If you don't have a good viewer for pgm imges on your machine, try to download and install ImageJ: http://rsb.info.nih.gov/ij/download.html

Back to Integrating into your application.