unfortunately the cuda::CascadeClassifier only understands the old format, so you can use HAAR features only, and have to use the -baseformat true param when calling the opencv_traincascade tool.