how can convert raw data frames into Mat

kishan patel
9 ●2 ●6

Hello, I have using gstreamer application to convert frames into BGR format and i can use VPE/VIDEOCONVERT in gstreamer pipeline to convert data in BGR format.

Please, check some lines of codes as below:

// Taken one variable std::atomic<cv::mat*> atomicFrame;

/* * @brief Check preroll to get a new frame using callback * https://gstreamer.freedesktop.org/doc... * @return GstFlowReturn */ GstFlowReturn new_preroll(GstAppSink /appsink/, gpointer /data/) { return GST_FLOW_OK; }

/* * @brief This is a callback that get a new frame when a preroll exist * * @Param appsink * @return GstFlowReturn */ GstFlowReturn new_sample(GstAppSink *appsink, gpointer /data*/) { static int framecount = 0;

// Get caps and frame
GstSample *sample = gst_app_sink_pull_sample(appsink);
GstCaps *caps = gst_sample_get_caps(sample);
GstBuffer *buffer = gst_sample_get_buffer(sample);
GstStructure *structure = gst_caps_get_structure(caps, 0);
const int width = g_value_get_int(gst_structure_get_value(structure, "width"));
const int height = g_value_get_int(gst_structure_get_value(structure, "height"));

// Print dot every 30 frames
if(!(framecount%30)) {
    g_print(".");
}

// Show caps on first frame
if(!framecount) {
    g_print("caps: %s\n", gst_caps_to_string(caps));
}
framecount++;

// Get frame data
#ifdef FACE_DETECTION
GstMapInfo map;
gst_buffer_map(buffer, &map, GST_MAP_READ);

// Convert gstreamer data to OpenCV Mat
cv::Mat* prevFrame;
prevFrame = atomicFrame.exchange(new cv::Mat(cv::Size(width, height), CV_8UC3, (char*)map.data, cv::Mat::AUTO_STEP));
if(prevFrame) 
{
    cv::Mat picBGR;
    //cv::cvtColor(prevFrame, prevFrame, COLOR_YUV2BGR_YV12);
    delete prevFrame;
}

gst_buffer_unmap(buffer, &map);
#endif
gst_sample_unref(sample);

return GST_FLOW_OK;

}

If we use videoconvert in gstreamer pipeline, all frames can convert into Mat(atomicFrame.exchange) properly and we can see it on display. But when use VPE in gstreamer pipeline, i got "segmentation fault" when application try to convert frame data into Mat(atomicFrame.exchange).

So, what should be wrong in this function?.

Regards, Kishan Patel.

answered 2018-10-22 08:10:58 -0600

berak
32993 ●7 ●81 ●312

updated 2018-10-22 08:34:18 -0600

do not use pointers to cv::Mat, that's always a terrible idea (defeats the internal refcounting)

you also need to deep-copy your prevFrame, because the external gstreamer buffer will no more be valid. so:

cv::Mat prevFrame; // NO pointer !

some loop {

    GstMapInfo map;
    gst_buffer_map(buffer, &map, GST_MAP_READ);
    // Convert gstreamer data to OpenCV Mat
    Mat frame(cv::Size(width, height), CV_8UC3, (char*)map.data, cv::Mat::AUTO_STEP); 
    // NO new !

    if (! prevFame.empty()) {
        // do calculation with frame & prevFrame
    }
    frame.copyTo(prevFrame); // deep copy !

    // no delete !

    gst_buffer_unmap(buffer, &map);    
}

edit flag offensive delete link

Comments

Now, segmantation fault not occuring.But can not see video-frames on display. I have taken pionter of Mat before, because of i have save data when new frame received(call-back function), and display it when received.

kishan patel ( 2018-10-23 00:20:03 -0600 )edit

// Main loop while(1) { g_main_iteration(false);

    cv::Mat* frame = atomicFrame.load();
    if(frame) 
    {

ifdef FACE_DETECTION

        detectAndDisplay( atomicFrame.load()[0]);

endif

        cv::waitKey(30);
    }
}

kishan patel ( 2018-10-23 00:22:25 -0600 )edit

your pointers are wrongn and the reason for the segfaults

berak ( 2018-10-23 01:24:39 -0600 )edit

Okay. If i use imshow for frame/prevFrame, it can not display frame.

kishan patel ( 2018-10-23 01:29:53 -0600 )edit

Moreover when i have used "frame.copyTo(prevFrame); // deep copy !", it takes more times to copy frame-data.

kishan patel ( 2018-10-23 05:58:19 -0600 )edit

Hello Berak, Have you any idea about 2 function as below: -prevFrame = atomicFrame.exchange(new cv::Mat(cv::Size(width, height), CV_8UC3, (char)map.data, cv::Mat::AUTO_STEP)); -cv::Mat frame = atomicFrame.load();

1st function didi not generate "Segmentation fault", when convert raw data into BGR format using "videoconvert". But this function has generate "segmentation fault" only when i have use "VPE" and convert raw data in BGR format.

kishan patel ( 2018-10-23 06:02:42 -0600 )edit

your main problem is: you have a borrowed pointer to the image data. if you don't make a deep copy of it, your Mat is invalid, once you call unmap, or leave the scope. using pointers tp cv::Mat does not cure it, and your atomic something does not , either (it even hides the fundamental problem)

imho, you have a design problem here. either you do everything in the callback, or you would also need proper locks/mutexes

maybe have another look, at how opencv's cap_gstreamer is implemented here.

berak ( 2018-10-23 06:09:58 -0600 )edit

Okay Berak, It means if i will use imshow in callback, then do unmap. Its fine. Right?

kishan patel ( 2018-10-23 06:19:06 -0600 )edit

i hope so (as long as this happens on the main thread !) ;)

berak ( 2018-10-23 06:23:25 -0600 )edit

Okay. And i have to use lock&unlock before&after "gst_buffer_map" and "gst_buffer_unmap". Right?

kishan patel ( 2018-10-23 06:52:47 -0600 )edit

see more comments

how can convert raw data frames into Mat

1 answer

Comments

ifdef FACE_DETECTION

endif

Links

Question Tools

Stats

Related questions

how can convert raw data frames into Mat edit

1 answer

Comments

ifdef FACE_DETECTION

endif

Links

Question Tools

Stats

Related questions

how can convert raw data frames into Mat