Normally I would switch to Linux, but due to the nature of the application I have no choice but to work with Win10.
Currently, the only backend that I managed to make work was DirectShow. While it does work, it's not something I can actually use, because it's entirely CPU-driven and due to me using 4 cameras running at 48FPS it grinds the application to a halt.
So far I've tried using:
- ffmpeg
- MSMF
- Intel MFX
- OpenCV MJPEG (I'm not entirely sure if it's hw accelerated but I tried it anyway)
None of the above actually manage to open the camera, and end up resulting in an empty (0 cols, 0 rows) frame being retrieved.
I've yet to try CUDA because I cannot install it on my system without wiping my OS - I tried multiple versions and different installers and every single time the CUDA SDK fails to install without giving me an actual reason.
I'm including the specifications of the camera:
Model ELP-USBFHD08S
Sensor OV4689
Lens Size 1/3 inch
Pixel Size 2.0µm X 2.0µm
image area 5440 µm x 3072 µm
Max. Resolution 1280(H)X720(V)
Compression format MJPEG / YUV2（YUYV）
Resolution & frame 1920X1080 MJPEG@ 60fps
1280X720 MJPEG@ 120fps
640X360MJPEG@ 260fps
Mini illumination 0. 1lux
And my OpenCV build info:
General configuration for OpenCV 4.1.1 =====================================
Version control: 4.1.1
Platform:
Timestamp: 2019-07-26T03:48:44Z
Host: Windows 10.0.18362 AMD64
CMake: 3.14.4
CMake generator: Visual Studio 15 2017
CMake build tool: C:/Program Files (x86)/Microsoft Visual Studio/2017/Professional/MSBuild/15.0/Bin/MSBuild.exe
MSVC: 1916
CPU/HW features:
Baseline: SSE SSE2 SSE3
requested: SSE3
Dispatched code generation: SSE4_1 SSE4_2 FP16 AVX AVX2 AVX512_SKX
requested: SSE4_1 SSE4_2 AVX FP16 AVX2 AVX512_SKX
SSE4_1 (13 files): + SSSE3 SSE4_1
SSE4_2 (1 files): + SSSE3 SSE4_1 POPCNT SSE4_2
FP16 (0 files): + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 AVX
AVX (4 files): + SSSE3 SSE4_1 POPCNT SSE4_2 AVX
AVX2 (27 files): + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 FMA3 AVX AVX2
AVX512_SKX (2 files): + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 FMA3 AVX AVX2 AVX_512F AVX512_COMMON AVX512_SKX
C/C++:
Built as dynamic libs?: YES
C++ Compiler: C:/Program Files (x86)/Microsoft Visual Studio/2017/Professional/VC/Tools/MSVC/14.16.27023/bin/Hostx86/x64/cl.exe (ver 19.16.27030.1)
C++ flags (Release): /DWIN32 /D_WINDOWS /W4 /GR /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _SCL_SECURE_NO_WARNINGS /Gy /bigobj /Oi /fp:precise /EHa /wd4127 /wd4251 /wd4324 /wd4275 /wd4512 /wd4589 /MP8 /MD /O2 /Ob2 /DNDEBUG
C++ flags (Debug): /DWIN32 /D_WINDOWS /W4 /GR /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _SCL_SECURE_NO_WARNINGS /Gy /bigobj /Oi /fp:precise /EHa /wd4127 /wd4251 /wd4324 /wd4275 /wd4512 /wd4589 /MP8 /MDd /Zi /Ob0 /Od /RTC1
C Compiler: C:/Program Files (x86)/Microsoft Visual Studio/2017/Professional/VC/Tools/MSVC/14.16.27023/bin/Hostx86/x64/cl.exe
C flags (Release): /DWIN32 /D_WINDOWS /W3 /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _SCL_SECURE_NO_WARNINGS /Gy /bigobj /Oi /fp:precise /MP8 /MD /O2 /Ob2 /DNDEBUG
C flags (Debug): /DWIN32 /D_WINDOWS /W3 /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _SCL_SECURE_NO_WARNINGS /Gy /bigobj /Oi /fp:precise /MP8 /MDd /Zi /Ob0 /Od /RTC1
Linker flags (Release): /machine:x64 /INCREMENTAL:NO
Linker flags (Debug): /machine:x64 /debug /INCREMENTAL
ccache: NO
Precompiled headers: NO
Extra dependencies:
3rdparty dependencies:
OpenCV modules:
To be built: calib3d core dnn features2d flann gapi highgui imgcodecs imgproc ml objdetect photo stitching video videoio world
Disabled: python2 python3
Disabled by dependency: -
Unavailable: java js ts
Applications: apps
Documentation: NO
Non-free algorithms: NO
Windows RT support: NO
GUI:
Win32 UI: YES
VTK support: NO
Media I/O:
ZLib: build (ver 1.2.11)
JPEG: build-libjpeg-turbo (ver 2.0.2-62)
WEBP: build (ver encoder: 0x020e)
PNG: build (ver 1.6.37)
TIFF: build (ver 42 - 4.0.10)
JPEG 2000: build (ver 1.900.1)
OpenEXR: build (ver 2.3.0)
HDR: YES
SUNRASTER: YES
PXM: YES
PFM: YES
Video I/O:
DC1394: NO
FFMPEG: YES (prebuilt binaries)
avcodec: YES (58.35.100)
avformat: YES (58.20.100)
avutil: YES (56.22.100)
swscale: YES (5.3.100)
avresample: YES (4.0.0)
GStreamer: NO
DirectShow: YES
Media Foundation: YES
DXVA: YES
Parallel framework: Concurrency
Trace: YES (with Intel ITT)
Other third-party libraries:
Intel IPP: 2019.0.0 Gold [2019.0.0]
at: C:/build/master_winpack-build-win64-vc15/build/3rdparty/ippicv/ippicv_win/icv
Intel IPP IW: sources (2019.0.0)
at: C:/build/master_winpack-build-win64-vc15/build/3rdparty/ippicv/ippicv_win/iw
Eigen: NO
Custom HAL: NO
Protobuf: build (3.5.1)
OpenCL: YES (NVD3D11)
Include path: C:/build/master_winpack-build-win64-vc15/opencv/3rdparty/include/opencl/1.2
Link libraries: Dynamic load
Python (for build): C:/utils/soft/python27-x64/python.exe
Java:
ant: C:/utils/soft/apache-ant-1.9.7/bin/ant.bat (ver 1.9.7)
JNI: C:/Program Files/Java/jdk1.8.0_112/include C:/Program Files/Java/jdk1.8.0_112/include/win32 C:/Program Files/Java/jdk1.8.0_112/include
Java wrappers: NO
Java tests: NO
Install to: C:/build/master_winpack-build-win64-vc15/install
-----------------------------------------------------------------
I thought I would include as much info as I can, so here's a function I use to initialize the camera capture:
void ACameraAvatar::CameraInit(int CameraID, FString DeviceName, ACameraManager* _CameraManager)
{
//Get the reference to the overarching manager class:
CameraManager = _CameraManager;
//Get the distortion information about the cameras
cv::FileStorage fs("C:/mocapSettings/cameraParams.xml", cv::FileStorage::READ);
fs["cameraMatrix"] >> cameraMatrix;
fs["distCoeffs"] >> distCoeffs;
//Open the capture with appropriate settings
int API = cv::CAP_DSHOW;
//int API = cv::CAP_FFMPEG;
//int API = cv::CAP_INTEL_MFX;
//int API = cv::CAP_MSMF;
//int API = cv::CAP_OPENCV_MJPEG;
cv::Size size(1280, 720);
UE_LOG(LogTemp, Warning, TEXT("[MOTION CAPTURE] OPENING CAMERA - Name: %s - id: %d"), *DeviceName, CameraID);
CameraCapture = new cv::VideoCapture;
CameraCapture->open(CameraID, API);
if (!CameraCapture->isOpened())
{
UE_LOG(LogTemp, Warning, TEXT("[MOTION CAPTURE] FAILED TO OPEN CAMERA - Name: %s - id: %d"), *DeviceName, CameraID);
return;
}
CameraCapture->set(cv::CAP_PROP_FOURCC, cv::VideoWriter::fourcc('M', 'J', 'P', 'G'));
CameraCapture->set(cv::CAP_PROP_FRAME_WIDTH, size.width);
CameraCapture->set(cv::CAP_PROP_FRAME_HEIGHT, size.height);
CameraCapture->set(cv::CAP_PROP_BUFFERSIZE, 3);
CameraCapture->set(cv::CAP_PROP_FPS, CameraManager->FPS);
while (!CameraCapture->grab())
{
FPlatformProcess::Sleep(0.020);
}
}
In the above code, swapping out the backend to any of the commented ones ends up returning an empty frame.
There's no way I can get around the fact that I need to have a hardware accelerated video capture for my cameras, so here's hoping you guys can help me troubleshoot and figure this problem out.
Thanks in advance!