PDA

View Full Version : Capture to Mat Array for OpenCV



ak_25
October 14, 2015, 02:05:41
Hello,

We have a DFK Z30GP031 and would like to interface it with OpenCV. I understand that your driver supports directshow but we cannot seem to view the captured images; the camera light blinks that data transfer is in progress.

Is there sample code that I could use with Imaging Control 3.3 or 3.4 and capture a single image for processing and convert it to the Mat Object in openCV in RGB mode?

Is there any API +SDK in works for OpenCV?

Also, I cannot find the samples mentioned here
http://www.theimagingsource.com/en_US/support/documentation/icimagingcontrol-class/AcquireVidFmtInfo.htm

Thanks

Stefan Geissler
October 14, 2015, 10:51:56
Hello

I have a very old sample, using IPLImge instead of the modern Mat. The trick is forwarding the IC Membuffer::getPtr() data to the OpenCV image.

ak_25
October 14, 2015, 18:20:59
Thank you, I'll try this out.. IpplImage is the old C struct i believe. Probably a function to convert it.. Ill have a look and let you know if I have any issues!

ak_25
October 15, 2015, 23:05:24
Hello Stefan,

I'm trying to first learn how to snap the image using the snapImages() command and saving it to .bmp to compare.

Sadly, all I get is a grey image. The sample program that Visual Studio Creates works fine but I do not want to create and MFC application..

The grey image that is saved has the right format and size however//

Here is the code


#include "opencv2/highgui/highgui.hpp"
#include <opencv2/imgproc/imgproc.hpp>
#include <iostream>
#include "stdafx.h"
using namespace cv;
using namespace std;
using namespace DShowLib;

int main()
{
//Init Library
if (!DShowLib::InitLibrary())
{
return FALSE;
}

//Instiantiate Obeject
DShowLib::Grabber* m_pGrabber;
m_pGrabber = new DShowLib::Grabber();
ASSERT(m_pGrabber);

//List avail Devices
Grabber::tVidCapDevListPtr pVidCapDevList = m_pGrabber->getAvailableVideoCaptureDevices();
if (pVidCapDevList == 0 || pVidCapDevList->empty())
{
return -1; // No device available.
}

// Open the selected video capture device.createDevice
m_pGrabber->openDev(pVidCapDevList->at(0));


// Begin Listing FrameRates and video Formats so I can set them
Grabber::tVidFmtListPtr pVidFmtList = m_pGrabber->getAvailableVideoFormats();
Grabber::tFrameRateListPtr pVidFrmList = m_pGrabber->getAvailableFrameRates();

if (pVidFmtList == 0) // No video formats available?
{
std::cerr << "Error : " << m_pGrabber->getLastError().toString() << std::endl;
}
else
{
unsigned int counter = 0;
// List the available video formats.

for (Grabber::tVidFmtList::iterator it = pVidFmtList->begin();
it != pVidFmtList->end();
++it)
{
std::cout << "\t[" << counter++ << "] " << it->toString() << std::endl;
}
m_pGrabber->setVideoFormat(pVidFmtList->at(125));
}

if (pVidFrmList == 0) // No video formats available?
{
std::cerr << "Error : " << m_pGrabber->getLastError().toString() << std::endl;
}
else
{
unsigned int counter = 0;
// List the available Frame formats.
for (Grabber::tFrameRateList::iterator it = pVidFrmList->begin();
it != pVidFrmList->end();
++it)
{
std::cout << "\t[" << counter++ << "] " << *it << std::endl;
}
m_pGrabber->setFrameRate(pVidFrmList->at(0));
}





//Create Sink
tFrameHandlerSinkPtr pSink = FrameHandlerSink::create(eRGB32, 1);

// Enable snap mode
pSink->setSnapMode(true);

// Set the sink.
m_pGrabber->setSinkType(pSink);

// Prepare the live mode, to get the output size if the sink.
if (!m_pGrabber->prepareLive(false))
{
std::cerr << "Could not render the VideoFormat into a RGB32 sink.";
return -1;
}

BYTE* buf[5];



// Retrieve the output type and dimension of the handler sink.
DShowLib::FrameTypeInfo info;
pSink->getOutputFrameType(info);

// Allocate 5 image buffers of the above calculate buffer size.
for (int i = 0; i < 5; ++i)
{
buf[i] = new BYTE[info.buffersize];
}

Mat newImg = Mat(info.dim.cy, info.dim.cx, CV_8UC1, Scalar(0));
// Create a new MemBuffer collection that uses our own image buffers.

tMemBufferCollectionPtr pCollection = MemBufferCollection::create(info, 5, buf);
if (pCollection == 0 || !pSink->setMemBufferCollection(pCollection))
{
std::cerr << "Could not set the new MemBufferCollection, because types do not match.";
return -1;
}

//Start Live But dont dislay it
m_pGrabber->startLive(false);

//Snap
pSink->snapImages(1, 5);
m_pGrabber->stopLive();
m_pGrabber->closeDev();

pCollection->save("file*.bmp");
//memcpy(newImg.ptr(), buf[0], info.buffersize);


//namedWindow("Display window", WINDOW_AUTOSIZE);

// imshow("cap",newImg);
// waitKey(0);


DShowLib::ExitLibrary;

}


Also I list out the camera formats and then manually select their ID for resolution and frame rate.



[0] RGB64 (96x2)
[1] RGB64 (128x96)
[2] RGB64 (160x120)
[3] RGB64 (176x144)
[4] RGB64 (240x176)
[5] RGB64 (240x180)
[6] RGB64 (320x240)
[7] RGB64 (320x480)
[8] RGB64 (352x240)
[9] RGB64 (352x288)
[10] RGB64 (384x288)
[11] RGB64 (640x240)
[12] RGB64 (640x288)
[13] RGB64 (640x480)
[14] RGB64 (704x576)
[15] RGB64 (720x240)
[16] RGB64 (720x288)
[17] RGB64 (720x480)
[18] RGB64 (720x576)
[19] RGB64 (768x576)
[20] RGB64 (1024x768)
[21] RGB64 (1280x960)
[22] RGB64 (1280x1024)
[23] RGB64 (1296x972)
[24] RGB64 (1600x1200)
[25] RGB64 (1920x1080)
[26] RGB64 (2048x1536)
[27] RGB64 (2592x1944)
[28] Y16 (96x2)
[29] Y16 (128x96)
[30] Y16 (160x120)
[31] Y16 (176x144)
[32] Y16 (240x176)
[33] Y16 (240x180)
[34] Y16 (320x240)
[35] Y16 (320x480)
[36] Y16 (352x240)
[37] Y16 (352x288)
[38] Y16 (384x288)
[39] Y16 (640x240)
[40] Y16 (640x288)
[41] Y16 (640x480)
[42] Y16 (704x576)
[43] Y16 (720x240)
[44] Y16 (720x288)
[45] Y16 (720x480)
[46] Y16 (720x576)
[47] Y16 (768x576)
[48] Y16 (1024x768)
[49] Y16 (1280x960)
[50] Y16 (1280x1024)
[51] Y16 (1296x972)
[52] Y16 (1600x1200)
[53] Y16 (1920x1080)
[54] Y16 (2048x1536)
[55] Y16 (2592x1944)
[56] Y800 (96x2)
[57] Y800 (128x96)
[58] Y800 (160x120)
[59] Y800 (176x144)
[60] Y800 (240x176)
[61] Y800 (240x180)
[62] Y800 (320x240)
[63] Y800 (320x480)
[64] Y800 (352x240)
[65] Y800 (352x288)
[66] Y800 (384x288)
[67] Y800 (640x240)
[68] Y800 (640x288)
[69] Y800 (640x480)
[70] Y800 (704x576)
[71] Y800 (720x240)
[72] Y800 (720x288)
[73] Y800 (720x480)
[74] Y800 (720x576)
[75] Y800 (768x576)
[76] Y800 (1024x768)
[77] Y800 (1280x960)
[78] Y800 (1280x1024)
[79] Y800 (1296x972)
[80] Y800 (1600x1200)
[81] Y800 (1920x1080)
[82] Y800 (2048x1536)
[83] Y800 (2592x1944)
[84] RGB24 (96x2)
[85] RGB24 (128x96)
[86] RGB24 (160x120)
[87] RGB24 (176x144)
[88] RGB24 (240x176)
[89] RGB24 (240x180)
[90] RGB24 (320x240)
[91] RGB24 (320x480)
[92] RGB24 (352x240)
[93] RGB24 (352x288)
[94] RGB24 (384x288)
[95] RGB24 (640x240)
[96] RGB24 (640x288)
[97] RGB24 (640x480)
[98] RGB24 (704x576)
[99] RGB24 (720x240)
[100] RGB24 (720x288)
[101] RGB24 (720x480)
[102] RGB24 (720x576)
[103] RGB24 (768x576)
[104] RGB24 (1024x768)
[105] RGB24 (1280x960)
[106] RGB24 (1280x1024)
[107] RGB24 (1296x972)
[108] RGB24 (1600x1200)
[109] RGB24 (1920x1080)
[110] RGB24 (2048x1536)
[111] RGB24 (2592x1944)
[112] RGB32 (96x2)
[113] RGB32 (128x96)
[114] RGB32 (160x120)
[115] RGB32 (176x144)
[116] RGB32 (240x176)
[117] RGB32 (240x180)
[118] RGB32 (320x240)
[119] RGB32 (320x480)
[120] RGB32 (352x240)
[121] RGB32 (352x288)
[122] RGB32 (384x288)
[123] RGB32 (640x240)
[124] RGB32 (640x288)
[125] RGB32 (640x480)
[126] RGB32 (704x576)
[127] RGB32 (720x240)
[128] RGB32 (720x288)
[129] RGB32 (720x480)
[130] RGB32 (720x576)
[131] RGB32 (768x576)
[132] RGB32 (1024x768)
[133] RGB32 (1280x960)
[134] RGB32 (1280x1024)
[135] RGB32 (1296x972)
[136] RGB32 (1600x1200)
[137] RGB32 (1920x1080)
[138] RGB32 (2048x1536)
[139] RGB32 (2592x1944)
[0] 66
[1] 71
[2] 83
[3] 100
[4] 111
[5] 125
[6] 142
[7] 166
[8] 200
[9] 250
[10] 333
[11] 500
[12] 1000
[13] 1146


These are the formats and the frame Rates. I wonder if i can select something less than 66.

I was trying to use RGB32 640x480 to start..

Thanks,

Akshay

Here is the image it saves
https://app.box.com/s/7unb8zxjrih6lv86ao9ol2tdn7ign3js

Stefan Geissler
October 16, 2015, 10:39:53
Hello

The frame rates are the time spans in milli seconds between two frames.

That means 66ms = 1000/66 = 15 fps.

I suggest to not close the device while you want to access the image buffer.

The steps

// Allocate 5 image buffers of the above calculate buffer size.
for (int i = 0; i < 5; ++i)
{
buf[i] = new BYTE[info.buffersize];
}

tMemBufferCollectionPtr pCollection = MemBufferCollection::create(info, 5, buf);


Are not necessary, the buffers will be created automatically.

ak_25
October 19, 2015, 21:55:05
Hello Stefan,

Things are working!!!

I ended up using the CListner Class and instantiating it and using the callback function frameReady to access the camera.


#include "opencv2/highgui/highgui.hpp"
#include <opencv2/imgproc/imgproc.hpp>
#include <iostream>
#include "stdafx.h"
using namespace cv;
using namespace std;
using namespace DShowLib;

int main()
{
//Init Library
if (!DShowLib::InitLibrary())
{
return FALSE;
}

// At the end of the program, the IC Imaging Control Class Library must be cleaned up
// by a call to ExitLibrary().
atexit(DShowLib::ExitLibrary);


DShowLib::Grabber m_pGrabber; // The instance of the Grabber class.

DShowLib::FrameHandlerSink::tFHSPtr m_pSink;
CListener m_cListener; // Image processing will be done by this object.

//Instiantiate Obeject
//DShowLib::Grabber* m_pGrabber;
//m_pGrabber = new DShowLib::Grabber();
//ASSERT(m_pGrabber);

// Add the CListener object to the the Grabber object.
m_pGrabber.addListener(&m_cListener);

//m_cListener.SetParent(this);


//List avail Devices
Grabber::tVidCapDevListPtr pVidCapDevList = m_pGrabber.getAvailableVideoCaptureDevices();
if (pVidCapDevList == 0 || pVidCapDevList->empty())
{
return -1; // No device available.
}

// Open the selected video capture device.createDevice
m_pGrabber.openDev(pVidCapDevList->at(0));


// Begin Listing FrameRates and video Formats so I can set them
Grabber::tVidFmtListPtr pVidFmtList = m_pGrabber.getAvailableVideoFormats();
Grabber::tFrameRateListPtr pVidFrmList = m_pGrabber.getAvailableFrameRates();

if (pVidFmtList == 0) // No video formats available?
{
std::cerr << "Error : " << m_pGrabber.getLastError().toString() << std::endl;
}
else
{
unsigned int counter = 0;
// List the available video formats.

for (Grabber::tVidFmtList::iterator it = pVidFmtList->begin();
it != pVidFmtList->end();
++it)
{
std::cout << "\t[" << counter++ << "] " << it->toString() << std::endl;
}
m_pGrabber.setVideoFormat(pVidFmtList->at(109));
}

if (pVidFrmList == 0) // No video formats available?
{
std::cerr << "Error : " << m_pGrabber.getLastError().toString() << std::endl;
}
else
{
unsigned int counter = 0;
// List the available Frame formats.
for (Grabber::tFrameRateList::iterator it = pVidFrmList->begin();
it != pVidFrmList->end();
++it)
{
std::cout << "\t[" << counter++ << "] " << *it << std::endl;
}
m_pGrabber.setFrameRate(pVidFrmList->at(0));
}





//Create Sink
m_pSink = FrameHandlerSink::create(DShowLib::eRGB24, 3);

m_pSink->setSnapMode(false); // Automatically copy every frame to the sink and call CListener::frameReady().
m_pGrabber.setSinkType(m_pSink);
m_pGrabber.startLive(false);
while (1);

//using at exit
//DShowLib::ExitLibrary;

}



I reused the doProcessing method in your example and as you suggested passed the buffer pointer to the cv:MAT


void CListener::DoImageProcessing( smart_ptr<MemBuffer> pBuffer)
{

// Get the bitmap info header from the membuffer. It contains the bits per pixel,
// width and height.
smart_ptr<BITMAPINFOHEADER> pInf = pBuffer->getBitmapInfoHeader();

// Now retrieve a pointer to the image. For organization of the image data, please
// refer to:
// http://www.imagingcontrol.com/ic/docs/html/class/Pixelformat.htm

DShowLib::MemBuffer::tBmInfHPtr bmpHeader = pBuffer->getBitmapInfoHeader();
LONG height = bmpHeader->biHeight;
LONG width = bmpHeader->biWidth;

cv::Mat img((int)height, (int)width, CV_8UC3);
cv::Mat img2;
BYTE* pImageData = pBuffer->getPtr();

img.data = (unsigned char*)pImageData;
cv::namedWindow("Display window", CV_WINDOW_AUTOSIZE);

cv::cvtColor(img, img2, cv::COLOR_RGB2GRAY); //Convert to Gray
cv::imshow("Display window", img2);
cvWaitKey(3);

I was wondering how to access greater frame rates at lower resolutions? Right now the lowest it lists is 66 which as you said is 15fps. Thanks a lot to your forum. Now off to actual image processing and interfacing this with multiple cameras!

Do you also have any other tips?

Regards,

Akshay

ak_25
October 19, 2015, 23:20:19
Ok, Can used setFPS to get a max speed of 132fps realistically at 640x480

Thanks. Akshay

Daniel Dundee
December 26, 2015, 20:10:52
Hi Akshay,

could you share this project?
I try to avoid using MFC as well, but always end up with some smart pointer invalid casting..

Using MS VS 2013, but also tried in 2005..

Thanks!!!
Daniel



Hello Stefan,

Things are working!!!

I ended up using the CListner Class and instantiating it and using the callback function frameReady to access the camera.


#include "opencv2/highgui/highgui.hpp"
#include <opencv2/imgproc/imgproc.hpp>
#include <iostream>
#include "stdafx.h"
using namespace cv;
using namespace std;
using namespace DShowLib;

int main()
{
//Init Library
if (!DShowLib::InitLibrary())
{
return FALSE;
}

// At the end of the program, the IC Imaging Control Class Library must be cleaned up
// by a call to ExitLibrary().
atexit(DShowLib::ExitLibrary);


DShowLib::Grabber m_pGrabber; // The instance of the Grabber class.

DShowLib::FrameHandlerSink::tFHSPtr m_pSink;
CListener m_cListener; // Image processing will be done by this object.

//Instiantiate Obeject
//DShowLib::Grabber* m_pGrabber;
//m_pGrabber = new DShowLib::Grabber();
//ASSERT(m_pGrabber);

// Add the CListener object to the the Grabber object.
m_pGrabber.addListener(&m_cListener);

//m_cListener.SetParent(this);


//List avail Devices
Grabber::tVidCapDevListPtr pVidCapDevList = m_pGrabber.getAvailableVideoCaptureDevices();
if (pVidCapDevList == 0 || pVidCapDevList->empty())
{
return -1; // No device available.
}

// Open the selected video capture device.createDevice
m_pGrabber.openDev(pVidCapDevList->at(0));


// Begin Listing FrameRates and video Formats so I can set them
Grabber::tVidFmtListPtr pVidFmtList = m_pGrabber.getAvailableVideoFormats();
Grabber::tFrameRateListPtr pVidFrmList = m_pGrabber.getAvailableFrameRates();

if (pVidFmtList == 0) // No video formats available?
{
std::cerr << "Error : " << m_pGrabber.getLastError().toString() << std::endl;
}
else
{
unsigned int counter = 0;
// List the available video formats.

for (Grabber::tVidFmtList::iterator it = pVidFmtList->begin();
it != pVidFmtList->end();
++it)
{
std::cout << "\t[" << counter++ << "] " << it->toString() << std::endl;
}
m_pGrabber.setVideoFormat(pVidFmtList->at(109));
}

if (pVidFrmList == 0) // No video formats available?
{
std::cerr << "Error : " << m_pGrabber.getLastError().toString() << std::endl;
}
else
{
unsigned int counter = 0;
// List the available Frame formats.
for (Grabber::tFrameRateList::iterator it = pVidFrmList->begin();
it != pVidFrmList->end();
++it)
{
std::cout << "\t[" << counter++ << "] " << *it << std::endl;
}
m_pGrabber.setFrameRate(pVidFrmList->at(0));
}





//Create Sink
m_pSink = FrameHandlerSink::create(DShowLib::eRGB24, 3);

m_pSink->setSnapMode(false); // Automatically copy every frame to the sink and call CListener::frameReady().
m_pGrabber.setSinkType(m_pSink);
m_pGrabber.startLive(false);
while (1);

//using at exit
//DShowLib::ExitLibrary;

}



I reused the doProcessing method in your example and as you suggested passed the buffer pointer to the cv:MAT


void CListener::DoImageProcessing( smart_ptr<MemBuffer> pBuffer)
{

// Get the bitmap info header from the membuffer. It contains the bits per pixel,
// width and height.
smart_ptr<BITMAPINFOHEADER> pInf = pBuffer->getBitmapInfoHeader();

// Now retrieve a pointer to the image. For organization of the image data, please
// refer to:
// http://www.imagingcontrol.com/ic/docs/html/class/Pixelformat.htm

DShowLib::MemBuffer::tBmInfHPtr bmpHeader = pBuffer->getBitmapInfoHeader();
LONG height = bmpHeader->biHeight;
LONG width = bmpHeader->biWidth;

cv::Mat img((int)height, (int)width, CV_8UC3);
cv::Mat img2;
BYTE* pImageData = pBuffer->getPtr();

img.data = (unsigned char*)pImageData;
cv::namedWindow("Display window", CV_WINDOW_AUTOSIZE);

cv::cvtColor(img, img2, cv::COLOR_RGB2GRAY); //Convert to Gray
cv::imshow("Display window", img2);
cvWaitKey(3);

I was wondering how to access greater frame rates at lower resolutions? Right now the lowest it lists is 66 which as you said is 15fps. Thanks a lot to your forum. Now off to actual image processing and interfacing this with multiple cameras!

Do you also have any other tips?

Regards,

Akshay