After almost two years of trying, I gave up to do this in a neat way. Most android-phones have the v4l2-kernel modules loaded, so I just used that one. Big disadvantage: On most phones, the devices nodes in dev (like /dev/videoX on the galaxy S5) are not readable by apps, so you have to root your phone and access them with root privileges (or change the rights on those files before accessing them).
This is the example I used to base my code on.
There is also an other library out there that tries to do what the v4l2-kernel extension does by using libusb and libuvc (you request the rights to the USB device via androids mechanisms, then pass the native pointer to libusb to read from it), but it is really buggy and I rarely got the cameras running for more than a few seconds, so I decided against it. Maybe it is stable now, so maybe if you want to do this as well you want to try it out.