Ask Your Question

VideoCapture from HTC Vive camera?

asked 2017-02-21 06:28:54 -0500


I'm trying to get frames from the HTC Vive front facing camera but I'm only getting seemingly gray frames out (though they are theoretically not empty).

My code looks like this:

int main() {
VideoCapture videoSource(0);

    Size size{

namedWindow("input", WINDOW_NORMAL);
resizeWindow("input", size.width, size.height);

Mat frameIn;

auto p1 = videoSource.get(CAP_PROP_MODE);
auto p2 = videoSource.get(CAP_PROP_FORMAT);
auto p3 = videoSource.get(CAP_PROP_FPS);
unsigned f = (unsigned)videoSource.get(CV_CAP_PROP_FOURCC);

char fourcc[] = {
    (char)(f >> 8),
    (char)(f >> 16),
    (char)(f >> 24),

cout << "\n\nCAPTURE DEVICE\n---------------"
    << "\nmode: " << p1
    << "\nformat: " << p2
    << "\nfps: " << p3
    << "\nFOURCC: " << string(fourcc)
    << "\nsize: " << size;

// One frame, to check info:
videoSource >> frameIn;

cout << "\n\nFRAME IN (MAT):\n--------------" 
    << "\ntype: " << frameIn.type() 
    << "\ndepth: " << frameIn.depth() 
    << "\nsize: " << frameIn.size();

while (!frameIn.empty()) {
    imshow("input", frameIn);

    char key = waitKey(10);

    if (key == 27) {

    videoSource >> frameIn;

return 0;

And the output (other than a gray window) is:


mode: 0 format: 0 fps: 30.0003 FOURCC: YUY2 size: [612 x 460]


type: 16 depth: 0 size: [612 x 460]

I'm new to OpenCV so I might be missing something obvious. But from the above, it seems like the camera is being detected. The 4cc is yuy2 though, but it still returns 0 for mode and format.

When I check frameIn.empty() is returns false, but as you can see, there doesn't seem to be anything in the mats.

Any help would be greatly appreciated.

edit retag flag offensive close merge delete


I'll check what I'm doing when i get home. I know I can get frames, but I might be using the OpenVR api.

Tetragramm gravatar imageTetragramm ( 2017-02-21 18:10:49 -0500 )edit

1 answer

Sort by ยป oldest newest most voted

answered 2017-02-21 19:33:47 -0500

Tetragramm gravatar image

Ok, I'm not getting OpenCV not throw exceptions, but OpenVR integrates nicely with OpenCV.

Take a look at the OpenVR examples HERE. The relevant one is trackedCamera_vr.

Following the sample, you get the header, if it's changed you get the buffer. However, before then you create an OpenCV mat.

    if (m_nCameraFrameBufferSize / (m_nCameraFrameHeight*m_nCameraFrameWidth) == 4)
        image.create(m_nCameraFrameHeight, m_nCameraFrameWidth, CV_8UC4);
    else if (m_nCameraFrameBufferSize / (m_nCameraFrameHeight*m_nCameraFrameWidth) == 3)
        image.create(m_nCameraFrameHeight, m_nCameraFrameWidth, CV_8UC3);

    nCameraError = m_pVRTrackedCamera->GetVideoStreamFrameBuffer(m_hTrackedCamera, vr::VRTrackedCameraFrameType_Undistorted, (uint8_t*), m_nCameraFrameBufferSize, &frameHeader, sizeof(frameHeader));

You'll notice that unlike the sample, this copies into, which is your Mat. It has a reversed RGB color array, so do a cv::cvtColor(image, image, cv::COLOR_BGR2RGB); and you're good. The sample also has all the code for getting camera pose (Remember it's OpenGL, so it won't match OpenCV's rotation and translation) and camera matrix.

edit flag offensive delete link more


Ok so I need to do it thru OpenVR? Meaning, OpenCV itself cannot really access the camera frames?

saldavonschwartz gravatar imagesaldavonschwartz ( 2017-02-21 20:43:22 -0500 )edit

I think it ought to be able to, but right now I can't get it to work no matter what method I use.

All I can say is that I have used this method before, and it worked then.

Tetragramm gravatar imageTetragramm ( 2017-02-21 21:51:01 -0500 )edit

Someone from Valve suggested that I do pretty much what you suggested. So I'll make your answer as correct. Though it might be the case then that it is not possible just thru OpenCV.

saldavonschwartz gravatar imagesaldavonschwartz ( 2017-02-21 22:06:43 -0500 )edit

I think it can be done, but OpenVR has all the status and error checking that will tell you when it will and won't work and why.

Tetragramm gravatar imageTetragramm ( 2017-02-22 17:45:01 -0500 )edit

Thanks Tetragramm, your method works for me.

sonnguyen512 gravatar imagesonnguyen512 ( 2017-10-24 21:02:13 -0500 )edit

I'm wondering about the integration of opencv, because it also seems to work without it. (e.g. grapping the frames with vlc (dshow) and ffmpeg also works for me). Could you be so gently to provide a larger code sample / git project that I can checkout?

rearranged gravatar imagerearranged ( 2017-10-30 09:14:39 -0500 )edit

Here's the code I used to save the camera intrinsics, extrinsics, and images for a project. It's a modification of the tracked_camera_openvr_sample that I linked to in the answer.

If you don't need the camera information and the cv::VideoCapture works, go for it. It ought to work, it just didn't for me.

EDIT: It helps to actually include the link...

Tetragramm gravatar imageTetragramm ( 2017-10-30 09:56:43 -0500 )edit

another example for reference:

virtual_dario gravatar imagevirtual_dario ( 2018-05-02 14:21:02 -0500 )edit

Question Tools

1 follower


Asked: 2017-02-21 06:28:54 -0500

Seen: 1,055 times

Last updated: Feb 21 '17