PDA

View Full Version : Problems with filters and avi



marco.hi-tec
April 11, 2008, 10:36:05
Hi support. I'm a newbie in c# programming and also in imagingcontrols. I needed to save an avi file from a 21BF04 camera (current video format is BY8) and at same time stream bitmap (no more then 5 frame per sescond, actually compressed as jpeg) over network.

I solved the unability of imagincontrol of saving an avi and call imageAvailable callback using a device filter, based on your binarization sample.

Initially this approach seems to work fine. But playing the generated avi result in time-compressed avi of about x 2. So, for example, if I record 10 second of movie, the result is about 5 second (time compressed, not cutted, it seems it is loosing frames without time-compensate them).

This is how apply the filter:



// Create an instance of the frame filter implementation
binFilterImpl = new BinarizationFilter();
// Create a FrameFilter object wrapping the implementation
m_FrameFilter = camera.FrameFilterCreate(binFilterImpl);
// Set the FrameFilter as display frame filter.
camera.DeviceFrameFilters.Add(m_FrameFilter);


this is my binfilter implementation:



namespace Binarization
{
/// <summary>
/// This frame filter applies a binarization on the image data.
/// If enabled, every gray value greater of equal to a specified threshold is changed to
/// the maximum gray value, every other gray value is changed to zero.
///
/// Allowed input types: RGB24
///
/// Output types: RGB24, the input type determines the output type.
///
/// Parameters:
/// enable:
/// send a frame over network
/// threshold:
/// not used
/// </summary>
public class BinarizationFilter : FrameFilterImpl
{
private bool m_bEnabled = false;
private int m_threshold = 127;

private NetworkStream m_stream;
public MemoryStream stream = new MemoryStream();
private ASCIIEncoding ascii = new ASCIIEncoding();

public static Bitmap bmp = null;

public int echo()
{
return 0;
}

public BinarizationFilter()
{
AddBoolParam( "enable", new SetBoolParam( setEnable ), new GetBoolParam( getEnable ) );
AddIntParam( "threshold", new SetIntParam( setThreshold ), new GetIntParam( getThreshold ) );
AddDataParam("netstream", new SetDataParam(setNetStream), new GetDataParam(getNetStream));
}

[DllImport("kernel32.dll")]
static extern unsafe void RtlMoveMemory(byte* dest, byte* src, uint len);

void setNetStream(byte[] stream)
{
MemoryStream ms = new MemoryStream(stream);
BinaryFormatter bf1 = new BinaryFormatter();
ms.Position = 0;

IntPtr sptr = (IntPtr)bf1.Deserialize(ms);

NetworkStream ns = (NetworkStream)Marshal.GetObjectForIUnknown(sptr);

m_stream = ns;
}

byte[] getNetStream()
{
return null;
}


/*
* Enables or disables the binarization.
*
* Only call this method in a beginParamTransfer/endParamTransfer block.
*/
void setEnable( bool bEnable )
{
m_bEnabled = bEnable;
}

/*
* Get the current enabled state of the binarization filter.
*
* Only call this method in a beginParamTransfer/endParamTransfer block.
*/
bool getEnable()
{
return m_bEnabled;
}

/*
* Sets the threshold value for the binarization.
*
* Only call this method in a beginParamTransfer/endParamTransfer block.
*/
void setThreshold( int threshold )
{
m_threshold = threshold;
}

/*
* Get the current threshold value of the binarization filter.
*
* Only call this method in a beginParamTransfer/endParamTransfer block.
*/
int getThreshold()
{
return m_threshold;
}

/*
* This method fills the ArrayList arr with the frame types this filter
* accepts as input.
*
* For the binarization filter, only the gray color formats eY800 and eRGB8 are accepted.
*/

public override void GetSupportedInputTypes(System.Collections.ArrayLis t frameTypes)
{
frameTypes.Add( new TIS.Imaging.FrameType( TIS.Imaging.MediaSubtypes.RGB24 ) );
}

/*
* This method returns the output frame type for a given input frame type.
*
* The binarization filter does not change size or color format,
* so the only output frame type is the input frame type.
*/
public override bool GetTransformOutputTypes(FrameType inType, System.Collections.ArrayList outTypes)
{
// We don't change the image type, output = input
outTypes.Add( inType );

return true;
}

/*
* This method is called to copy image data from the src frame to the dest frame.
*
* Depending on the value of m_bEnabled, this implementation applies a binarization or
* copies the image data without modifying it.
*/
public override bool Transform(IFrame src, IFrame dest)
{
unsafe
{
// Check whether the destination frame is available
if( dest.Ptr == null ) return false;

// Copy the member variables to the function's stack, to protect them from being
// overwritten by parallel calls to setThreshold() etc.
//
// beginParamTransfer/endParamTransfer makes sure that the values from various
// member variables are consistent, because the user of this filter must enclose
// writing parameter access into beginParamTransfer/endParamTransfer, too.

BeginParameterTransfer();
int threshold = m_threshold;
bool enabled = m_bEnabled;
EndParameterTransfer();

byte* pIn = src.Ptr;
byte* pOut = dest.Ptr;


if (enabled)
{
int imgSize;
string sDesc;

if (bmp == null) bmp = new Bitmap(640, 480, System.Drawing.Imaging.PixelFormat.Format24bppRgb) ;

BitmapData bData = bmp.LockBits(new Rectangle(new Point(), bmp.Size), ImageLockMode.WriteOnly, PixelFormat.Format24bppRgb);
// For each byte in the input buffer, check whether it is greater or
// equal to the threshold.
byte* ptr = (byte*)bData.Scan0.ToPointer();
RtlMoveMemory(ptr, pIn, (uint)src.FrameType.BufferSize);
bmp.UnlockBits(bData);
bmp.RotateFlip(RotateFlipType.Rotate180FlipNone);

stream.Position = 0;
bmp.Save(stream, System.Drawing.Imaging.ImageFormat.Jpeg);
imgSize = (int)stream.Position;
byte[] sockBuff = new byte[imgSize];

stream.Position = 0;
stream.Read(sockBuff, 0, imgSize);

sDesc = "JPEG:" + imgSize + "!";

// Send the descriptor...
try
{
byte[] buff = ascii.GetBytes(sDesc);
m_stream.Write(buff, 0, buff.Length);

// Send the bitmap
m_stream.Write(sockBuff, 0, imgSize);
}
catch
{
}
m_bEnabled = false;
}

dest.CopyFrom( src );
}

return true;
}
}
}



note that if I disable all image processing (simply use dest.CopyFrom( src ) ) the issue is the same. If I disable the filter, all works fine.

Thank's in advance for any idea to help me.

Marco T.
hi-tec srl
www.hi-tec.it

Sascha Schmidt
April 11, 2008, 14:36:39
Hello Marco,

The frame rate of replay of the AVI file is written once in the header of the AVI file. In IC Imaging Control the frame rate of the camera is used. This means, if your frame rate is set to 7.5 frames per second, but you only save 3 frames per second, then then AVI is replayed with double speed.

On the internet are tools available, that can be used to change the frame rate of replay of an AVI file.

Otherwise, if you use the standard IC functionality for AVI capture and do the image processing in a frame filter, then this may is easier. The "Image and AVI Capture" sample (http://www.imagingcontrol.com/library/dotnet/section/video-aquisition/example/image-and-avi-capture) shows this.

marco.hi-tec
April 11, 2008, 15:00:58
The camera is fixed (and forced) at 30 fps...


camera.DeviceFrameRate = 30.0F;

If I disable this code



// Create an instance of the frame filter implementation
binFilterImpl = new BinarizationFilter();
// Create a FrameFilter object wrapping the implementation
m_FrameFilter = camera.FrameFilterCreate(binFilterImpl);
// Set the FrameFilter as display frame filter.
camera.DeviceFrameFilters.Add(m_FrameFilter);


the created avi (tested with camera.AviStartCapture and also with MediaStreamSink method) is perfect.

If I enable the filter code (just dest.CopyFrom( src ) in Transform() is sufficient to reproduce the issue) the frame rate is not consistent... in the avi file header is written 30fps, but the movie is accelerated, as frames are missing without time compensation. I know that I could miss frames during video acquisition, but, as I said above, if I acquire 5 second of video at 30fps, the avi is complete (not cutted) but its duration is about 2.5-3.0 seconds, always at 30fps...

Thanks
Marco

Sascha Schmidt
April 11, 2008, 16:55:34
Hallo Marcus,

Please try placing your filter into the mediastream sink and enable frame time correction(aviSink.FrameTimeCorrection = true).
This should correct your trouble:


private TIS.Imaging.FrameFilter m_FrameFilter;
private TestFilter curTestFilter;

private TIS.Imaging.MediaStreamSink aviSink;
private TIS.Imaging.BaseSink oldSink;
...

// Create an instance of the frame filter implementation
curTestFilter = new TestFilter();
// Create a FrameFilter object wrapping the implementation
m_FrameFilter = icImagingControl1.FrameFilterCreate(curTestFilter) ;
curTestFilter.m_bEnabled = true;


private void StartAvi()
{
aviSink = new TIS.Imaging.MediaStreamSink();
aviSink.FrameFilters.Add(m_FrameFilter);

aviSink.Codec = (TIS.Imaging.AviCompressor)cmbCodec.SelectedItem;
aviSink.Filename = "test.avi";

aviSink.FrameTimeCorrection = true;

aviSink.SinkModeRunning = true;
bool wasRunning = icImagingControl1.LiveVideoRunning;

if (wasRunning)
icImagingControl1.LiveStop();

oldSink = icImagingControl1.Sink;
icImagingControl1.Sink = aviSink;

if (wasRunning)
icImagingControl1.LiveStart();
}

private void StopAvi()
{
bool wasRunning = icImagingControl1.LiveVideoRunning;
if (wasRunning)
icImagingControl1.LiveStop();

aviSink.SinkModeRunning = false;
icImagingControl1.Sink = oldSink;
aviSink = null;

if (wasRunning)
icImagingControl1.LiveStart();
}


Here is the complete sample:

marco.hi-tec
April 11, 2008, 17:17:21
Hallo Marcus,

Please try placing your filter into the mediastream sink and enable frame time correction(aviSink.FrameTimeCorrection = true).
This should correct your trouble:




Thank's you... I will try as I have the camera back...

Marco.

marco.hi-tec
May 15, 2008, 14:04:57
Seems to works fine with aviSink.FrameTimeCorrection = true.

Anyway I hope that future imagecontrols library will support imageAvailable callback during avi processing.

Thank's you.

Stefan Geissler
May 15, 2008, 14:11:55
Marco,

thank you for your feedback. We will have a look on this.