Why OpenCV building is so slow with CUDA?

edit

I get pretty long compilation times for OpenCV even on an Intel Sandy Bridge server with 64gb of ram. My speculation is that it's a combination of:

• Lots of C++ templates (see this thread for some insight into why C++ templates can be slow to build).
• Lots of cuda kernels -- remember, the kernels themselves are not polymorphic (unless you do some really advanced tricks), so OpenCV routines often have one kernel for each data type (CV_8UC1, CV_32FC3, etc). This adds up quickly.
• As Vladislav said, building for several architectures (Compute 1.0, 1.1, 1.2, 2.0, etc) increases build time, but you can avoid this by just selecting your architecture in the CUDA_ARCH_BIN flag.
• I think there's also some code generation going on at compile-time. I don't remember the details, but I remember seeing a bunch of printouts about code generation during the OpenCV GPU compilation.

You may have already tried this, but building in multithread mode (e.g. use the flag -j8 for 8 threads, -j16 for 16 threads, pick your favorite number) can help. I've noticed that builds sometimes fail in multithreaded mode, but this may just be coincidence. Anyway, it's worth a try.

A couple of somewhat related comments:

I believe TBB (threaded building blocks) would utilise all cores, but maybe that is in run-time and not when compiling. I haven't fiddled with TBB myself yet.

Another thing: Make sure the charger is plugged in. My i7 laptop runs at 1.0 ghz if it is running on battery, but at a full 2.4 ghz on the charger.

I'm also noticing this (e.g. several hours to build with CUDA)

I understand that it's building many versions of each kernel for different template types and different compute targets.

However what is confusing is that this part of the build is not multi-threaded (unlike the rest of the build which is multi-threaded). I presume either this is intentional because NVCC doesn't like being run in parallel (it seems to actually do something with the GPU device whilst building), or because the CMake system doesn't generate the correct project settings to perform the NVCC instructions in parallel (e.g. this is built using some sort of long batch script which just executes instructions one after another).

Testing here with VS2019, cmake, OpenCV 4.4.0

Edit: And it seems I was somewhat correct. The cuda files are all built using <CustomBuild> tags in the opencv world project, which Visual Studio does not perform in parallel (since often this feature is used for actions that need to be run in the correct sequence). If you're not building as opencv world, then you might be able to perform some parallel compilation here and same some time (since projects can be compiled in parallel to one another and I presume the build events don't lock out build events from other projects being compiled).