Ask Your Question
1

Android Camera2 YUV to RGB conversion turns out green?

asked 2015-05-11 04:37:39 -0500

franz gravatar image

updated 2015-05-11 08:49:22 -0500

So I'm getting Image objects from Android's Camera2 API, then I convert them to OpenCV Mat objects via their byte buffers. The YUV_420_888 format is what I set as the output of the camera as recommended by the docs, but when I try converting the Mat from YUV to RGB, all it shows is green.

Following the answers from this thread, this is how I convert the Mat:

Image image = reader.acquireLatestImage();

ByteBuffer buffer = image.getPlanes()[0].getBuffer();
byte[] bytes = new byte[buffer.remaining()];
buffer.get(bytes);

Mat mat = new Mat(image.getHeight()+image.getHeight()/2, image.getWidth(), CvType.CV_8UC1);
mat.put(0, 0, bytes);
Mat rgb = new Mat(image.getHeight(), image.getWidth(), CvType.CV_8UC4);
Imgproc.cvtColor(mat, rgb, Imgproc.COLOR_YUV420sp2BGR, 4);

After these lines, all I did next was use imwrite to write the mats to disk. For reference, here's some sample images resulting from the writes:

YUV - http://i.imgur.com/qm765AZ.jpg (straight from the Camera2 API, no processing yet)

RGB - http://i.imgur.com/FzLx2Cc.jpg (the exact same image, but converted from YUV to RGB)

Any insights as to why the RGB image looks the way it does? I've also tried a whole lot of other conversion options besides COLOR_YUV420sp2BGR, but they all seem to have the same effect, which is a green image. Thank you in advance!

EDIT: As has been pointed out in the comments, it seems I need to use all 3 planes of the YUV image, and not just the first one. I know how to convert each plane into a byte array, and now I have 3 byte arrays each representing a plane, but my question is now how do I create a Mat from these 3 byte arrays? The put() method I'm familiar with only accepts a single byte array. Do I concatenate or combine them somehow?

edit retag flag offensive close merge delete

Comments

1

I guess (but just a guess) the problem is here ByteBuffer buffer = image.getPlanes()[0].getBuffer(); You're just using the 0-plane of the image, but you have 3 planes: Y, U and V.

LorenaGdL gravatar imageLorenaGdL ( 2015-05-11 04:51:47 -0500 )edit

Thanks for the reply! That part of the code I actually based it from the official Camera2 example found here. It does initiate it with the 0-plane, but I just assume buffer.getBytes() does the rest of the job as when I run this example, the results are fully colored. Though the difference is the format was set to JPEG, not YUV. So are you saying that since I changed the format from JPEG to YUV, I must retrieve not just the 0-plane?

franz gravatar imagefranz ( 2015-05-11 04:59:57 -0500 )edit

Also, if I do indeed get byte arrays for the 3 planes, how would I create a Mat from that? From the examples I saw, using the Mat's put() method accepts a single byte array. Now that I have 3 byte arrays, how do I combine them into a single one in order to pass them to put() ? Or should I create Mats using a different method or constructor? I browsed the docs for the Mat object, but I don't see any method or constructor which accepts 3 byte arrays simultaneously?

franz gravatar imagefranz ( 2015-05-11 05:06:53 -0500 )edit
1

I'm not familiarized with the API, so I can't really help, but it seems unnatural to me to use just one single plane when you're working with the YUV color space. Anyway I may be fully wrong.

LorenaGdL gravatar imageLorenaGdL ( 2015-05-11 06:29:32 -0500 )edit
1

Another thing, why do you have a 4 here?Imgproc.cvtColor(mat, rgb, Imgproc.COLOR_YUV420sp2BGR, 4); Doesn't the 4 indicate the number of channels in the ouput image? If so, you only have 3 (you can even omit the number I think)

LorenaGdL gravatar imageLorenaGdL ( 2015-05-11 06:33:37 -0500 )edit

Alright, thank you for your insights, they're very helpful! Anyway, yes I think you're right, that should be 3 and not 4.

franz gravatar imagefranz ( 2015-05-11 08:31:32 -0500 )edit

OK, after a new look at your code I've spotted another important thing: in the definition of mat, there's this CvType.CV_8UC1 when you should have CvType.CV_8UC3 as you're storing 3 channels (same on the output mat)

LorenaGdL gravatar imageLorenaGdL ( 2015-05-12 02:52:30 -0500 )edit

5 answers

Sort by » oldest newest most voted
2

answered 2016-01-18 23:17:15 -0500

I have been able to do it successfully.

    Image.Plane Y = image.getPlanes()[0];
    Image.Plane U = image.getPlanes()[1];
    Image.Plane V = image.getPlanes()[2];

    int Yb = Y.getBuffer().remaining();
    int Ub = U.getBuffer().remaining();
    int Vb = V.getBuffer().remaining();

    byte[] data = new byte[Yb + Ub + Vb];

    Y.getBuffer().get(data, 0, Yb);
    U.getBuffer().get(data, Yb, Ub);
    V.getBuffer().get(data, Yb+ Ub, Vb);

I'm using OpenCV JNI and was able to convert it to BGR image using CV_YUV2BGR_I420 with cvtColor

Here's my C++ code.

jboolean Java_com_fenchtose_myPackage_myClass_myMethod(
        JNIEnv* env, jobject thiz,
        jint width, jint height,
        jbyteArray YUVFrameData)
{
    jbyte * pYUVFrameData = env->GetByteArrayElements(YUVFrameData, 0);

    double alpha;
    alpha = (double) alphaVal;

    Mat mNV(height + height/2, width, CV_8UC1, (unsigned char*)pYUVFrameData);
    Mat mBgr(height, width, CV_8UC3);

    cv::cvtColor(mNV, mBgr, CV_YUV2BGR_I420);

    env->ReleaseByteArrayElements(YUVFrameData, pYUVFrameData, 0);

    return true;
}
edit flag offensive delete link more

Comments

this will not work if the pixel stride is 2, see http://answers.opencv.org/question/61... (awaiting moderation)

zarokka gravatar imagezarokka ( 2016-08-16 03:11:54 -0500 )edit
0

answered 2018-09-21 22:10:23 -0500

I have a image(YUV_420_888) that captured by ARCore in pixel, and want to covert it to Mat.

I have try the first answer(http://answers.opencv.org/ques..., but it failed.I can't get the rgb image by save the picture using the opencv function: imwrite().

And then, I try the second answer(http://answers.opencv.org/ques..., I failed too. because in the image that saved by imwrite(), red object covert to green.

Finally, because of a little accident, I try the code that YUV to data(byte[]) that write in the first answer, and try the code that data(byte[]) to Mat that writes in the second answer, and I successd.

---------- Java

    Image.Plane Y = image.getPlanes()[0];
    Image.Plane U = image.getPlanes()[1];
    Image.Plane V = image.getPlanes()[2];

    int Yb = Y.getBuffer().remaining();
    int Ub = U.getBuffer().remaining();
    int Vb = V.getBuffer().remaining();

    byte[] data = new byte[Yb + Ub + Vb];

    Y.getBuffer().get(data, 0, Yb);
    U.getBuffer().get(data, Yb, Ub);
    V.getBuffer().get(data, Yb+ Ub, Vb);

    return data;

--------- OpenCV JNI ,c++

extern "C"
JNIEXPORT jintArray JNICALL
Java_com_google_ar_core_examples_java_helloar_HelloArActivity_getKeyPoint2
    (JNIEnv *env, jobject jobj, jbyteArray _pixels1, jbyteArray _pixels2, jint width, jint height)
{

jbyte *pixels1 = env->GetByteArrayElements(_pixels1 , NULL);
jbyte *pixels2 = env->GetByteArrayElements(_pixels2 , NULL);

if(pixels1==NULL){
    int size = -1;
    jintArray result = env->NewIntArray(size);
    return result;
}
Mat lyuvMat(height + (height / 2), width, CV_8UC1, (unsigned char*)pixels1);
imwrite("/storage/emulated/0/camoutput/xly/lyuv.jpg", lyuvMat);
Mat lrgbMat(height, width, CV_8UC3);
cvtColor(lyuvMat, lrgbMat, COLOR_YUV2RGB_NV21, 3);
imwrite("/storage/emulated/0/camoutput/xly/lrgb.jpg", lrgbMat);



.......

}
edit flag offensive delete link more
0

answered 2018-08-30 04:43:13 -0500

COLOR_YUV420sp2BGRA 或许你应该使用这个答案,我在face 上也有部分有色变绿,希望对你有用。

edit flag offensive delete link more
0

answered 2016-08-16 03:05:14 -0500

zarokka gravatar image

The answer http://answers.opencv.org/question/61... did not fully work for me, it seems that the data format can vary between devices.

The mentioned answer probably works if the pixel stride is 1, but all devices I have available produce images with pixel stride 2 (Nexus 5, Sony Xperia Z3). The pixel stride means that the U and V planes are padded. The full buffer has the size of 2 bytes per pixel.

There is no way to directly convert this padded format with OpenCV currently, but it can be converted to YUV NV21 before converting it to a RGB/BGR image. (NV21 because it requires less work to convert to NV21 than to normal YUV420).

I do not know if a different pixel stride than 1 or 2 is possible (i guess not).

@TargetApi(21)
public Mat convertYuv420888ToMat(Image image, boolean isGreyOnly) {
    int width = image.getWidth();
    int height = image.getHeight();

    Image.Plane yPlane = image.getPlanes()[0];
    int ySize = yPlane.getBuffer().remaining();

    if (isGreyOnly) {
        byte[] data = new byte[ySize];
        yPlane.getBuffer().get(data, 0, ySize);

        Mat greyMat = new Mat(height, width, CvType.CV_8UC1);
        greyMat.put(0, 0, data);

        return greyMat;
    }

    Image.Plane uPlane = image.getPlanes()[1];
    Image.Plane vPlane = image.getPlanes()[2];

    // be aware that this size does not include the padding at the end, if there is any
    // (e.g. if pixel stride is 2 the size is ySize / 2 - 1)
    int uSize = uPlane.getBuffer().remaining();
    int vSize = vPlane.getBuffer().remaining();

    byte[] data = new byte[ySize + (ySize/2)];

    yPlane.getBuffer().get(data, 0, ySize);

    ByteBuffer ub = uPlane.getBuffer();
    ByteBuffer vb = vPlane.getBuffer();

    int uvPixelStride = uPlane.getPixelStride(); //stride guaranteed to be the same for u and v planes
    if (uvPixelStride == 1) {
        uPlane.getBuffer().get(data, ySize, uSize);
        vPlane.getBuffer().get(data, ySize + uSize, vSize);

        Mat yuvMat = new Mat(height + (height / 2), width, CvType.CV_8UC1);
        yuvMat.put(0, 0, data);
        Mat rgbMat = new Mat(height, width, CvType.CV_8UC3);
        Imgproc.cvtColor(yuvMat, rgbMat, Imgproc.COLOR_YUV2RGB_I420, 3);
        yuvMat.release();
        return rgbMat;
    }

    // if pixel stride is 2 there is padding between each pixel
    // converting it to NV21 by filling the gaps of the v plane with the u values
    vb.get(data, ySize, vSize);
    for (int i = 0; i < uSize; i += 2) {
        data[ySize + i + 1] = ub.get(i);
    }

    Mat yuvMat = new Mat(height + (height / 2), width, CvType.CV_8UC1);
    yuvMat.put(0, 0, data);
    Mat rgbMat = new Mat(height, width, CvType.CV_8UC3);
    Imgproc.cvtColor(yuvMat, rgbMat, Imgproc.COLOR_YUV2RGB_NV21, 3);
    yuvMat.release();
    return rgbMat;
}
edit flag offensive delete link more
0

answered 2018-09-24 05:20:58 -0500

shyam kumar gravatar image

Camera2 YUV_420_888 to RGB Mat(opencv) in Java

@Override
public void onImageAvailable(ImageReader reader){
    Image image = null;

    try {
        image = reader.acquireLatestImage();
        if (image != null) {

            byte[] nv21;
            ByteBuffer yBuffer = mImage.getPlanes()[0].getBuffer();
            ByteBuffer uBuffer = mImage.getPlanes()[1].getBuffer();
            ByteBuffer vBuffer = mImage.getPlanes()[2].getBuffer();

            int ySize = yBuffer.remaining();
            int uSize = uBuffer.remaining();
            int vSize = vBuffer.remaining();

            nv21 = new byte[ySize + uSize + vSize];

            //U and V are swapped
            yBuffer.get(nv21, 0, ySize);
            vBuffer.get(nv21, ySize, vSize);
            uBuffer.get(nv21, ySize + vSize, uSize);

            Mat mRGB = getYUV2Mat(nv21);



        }
    } catch (Exception e) {
        Log.w(TAG, e.getMessage());
    }finally{
        image.close();// don't forget to close
    }
}



public Mat getYUV2Mat(byte[] data) {
Mat mYuv = new Mat(image.getHeight() + image.getHeight() / 2, size.getWidth(), CV_8UC1);
mYuv.put(0, 0, data);
Mat mRGB = new Mat();
cvtColor(mYuv, mRGB, Imgproc.COLOR_YUV2RGB_NV21, 3);
return mRGB;}
edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2015-05-11 04:37:39 -0500

Seen: 12,606 times

Last updated: Sep 24 '18