Converting camera frames to float on GPU?

asked 2016-01-24 10:38:11 -0500

mynameisjohnj gravatar image

updated 2016-01-24 10:47:26 -0500

Hi All,

I've got a very basic example that grabs frames from a camera, uploads them to a GpuMat, and displays them in a namedWindow with CV_WINDOW_OPENGL as its type. The code below works as expected, but I don't understand why I have to perform the data type conversion using the host mat.

#include <opencv2/core/core.hpp>
#include <opencv2/core/cuda.hpp>
#include <opencv2/highgui/highgui.hpp>

#include <iostream>

int main() {
    using namespace cv;
    using cv::cuda::GpuMat;

    VideoCapture cap(0);
    if (!cap.isOpened())
        return -1;

    // create opengl window
    namedWindow("webcam", CV_WINDOW_OPENGL);

    // Gpu mat to display
    GpuMat g;

    // Host frame buf
    Mat frame;

    bool grabFrame = true;
    while (grabFrame) {
        // Grab frame
        cap >> frame;

        // Why is this line necessary?
        frame.convertTo(frame, CV_32F);

        // Upload to gpu

        // convert to normalized float
        g.convertTo(g, CV_32F, 1.f / 255);

        // show in opengl window
        imshow("webcam", g);

        // maybe quit
        if (cv::waitKey(30) >= 0)
            grabFrame = false;

    return 0;

If I comment out that line and try to perform the conversion and division in one step, I get a black image. I thought it could be the order in which the conversion happens (i.e scale followed by conversion), so I tried this

g.convertTo(g, CV_32F);
g.convertTo(g, CV_32F, 1.f / 255);

but had no luck. Checking the type of image the camera yields returns 16, which is a 3 channel single byte image. I tried replacing CV_32F with CV_32FC3 but it didn't make a difference. For what it's worth, though, the final dataType of the gpumat is 21, which is CV_32FC3. The thing that breaks it seems to be the type conversion.

Are there limits on the data types we can convert?

edit retag flag offensive close merge delete


Before we dig into technical details, is there any good reason why you would convert an image using GPU? The overload of pushing it from CPU to GPU and back will never outweigh doing it on the CPU...

StevenPuttemans gravatar imageStevenPuttemans ( 2016-01-24 12:30:38 -0500 )edit

Fair question. My intent is to perform some further processing on the image, and I assume there is some added benefit to displaying the GpuMat inside a namedWindow created with OpenGL (maybe that assumption is wrong.) My thinking was that if I'm displaying the image, it's going on the GPU somehow, so if I can I may as well get it there as soon as possible.

Furthermore I plan on running this code on a Jetson TX1, and my intent is to verify that data transfer is cheaper on that device, given that the memory storage for the CPU and GPU are physically unified, but that is beside the point.

mynameisjohnj gravatar imagemynameisjohnj ( 2016-01-24 13:23:07 -0500 )edit

@StevenPuttemans So do you have any suggestions? And if my comment wasn't clear... I'm never actually "pushing the data" back from the GPU. I agree that in most cases it wouldn't be worth the effort, but I'm kind of using this for a larger project that will involve using thrust and context sharing with OpenGL, so I'd definitely like to get it on my graphics card as soon as possible (especially for something like this type conversion and division, which is what you'd call embarrassingly parallel.)

mynameisjohnj gravatar imagemynameisjohnj ( 2016-01-25 09:13:25 -0500 )edit