Errors in detectMultiScale() for face detection

asked 2017-03-08 08:17:30 -0600

updated 2017-03-09 03:29:44 -0600

Hi guys,

My research team and i are working on a JRuby multithreaded application, that calls in an on-demand way a simple java class for face-recognition.

the JVM is always Oracle jdk 1.8.0_121
the JRuby version is always jruby 9.1.7.0

About openCV, we have tested the 3.2.0 and 3.1.0 versions, building it with java bindings in the following way:

cmake -DMAKE_BUILD_TYPE=DEBUG -DBUILD_SHARED_LIBS=OFF

putting opencv_320.jar in /usr/share/java, and libopencv_java3**.so in /usr/local/lib .

The java code is the following ( we use two face-detectors ):

public class FaceRecognition {

  static{
    System.load("/usr/local/lib/libopencv_java310.so");
  }

  public static String doFaceRec(byte[] img_stream) {
    Mat frame = Imgcodecs.imdecode(new MatOfByte(img_stream), Imgcodecs.IMREAD_UNCHANGED);
    CascadeClassifier faceDetector1 = new CascadeClassifier("haarcascade_profileface.xml");
    faceDetector1.load("haarcascade_profileface.xml");
    CascadeClassifier faceDetector2 = new CascadeClassifier("haarcascade_frontalface_alt.xml");
    faceDetector2.load("haarcascade_frontalface_alt.xml");
    if (faceDetector1.empty() || faceDetector2.empty()) {
        frame.release();
System.out.println("faceDetector is empty");
        return "0";
    }

    MatOfRect faceDetections = new MatOfRect();
    MatOfRect faceDetections2 = new MatOfRect();
    faceDetector1.detectMultiScale(frame, faceDetections, 1.3, 3, 0, new Size(), new Size());
    faceDetector2.detectMultiScale(frame, faceDetections2, 1.1, 3, 0, new Size(), new Size());

    int found = faceDetections.toArray().length + faceDetections2.toArray().length;
    faceDetections.release();
    faceDetections2.release();
    frame.release();

    return ""+found;
  }
}

The application generates different threads that call the doFaceRec(..) method provided by FaceRecognition class; through a concurrent-lock mechanism we are sure that only one thread per time calls the doFaceRec(..) method.

We tested the application on different machines with same conditions and/or parameters, with no errors:

Raspberry Pi 3 (quad-core, 1GB of RAM), Raspbian 8.0 / linux 4.4.38
Sony vaio (i7 dual-core, 12GB of RAM) Ubuntu 16.04 / linux 4.4.0
Dell XPS 13 9360 (i7 dual-core, 16GB of RAM), Ubuntu 16.04 / linux 4.8.4

Unfortunately, we had some errors testing the code in other machines:

Lenovo with i7 quad-core processor, 16GB of RAM, the only machine with openjdk 8
Amazon m4.4xlarge EC2 instance (Intel Xeon 16 vCPU, 64gb of RAM) Ubuntu 16.04 / linux 4.4.0

Without any limitation on the number of used threads, and using both opencv 3.2.0 and opencv 3.1.0 , we have two different errors ( we show only the errors on opencv3.2.0 because those with opencv3.1.0 are the same ):

Error 1:

[thread 140705678751488 also had an error][thread 140706560714496 also had an error][thread 140705812969216 also had an error][thread 140705804576512 also had an error][thread 140705787791104 also had an error]#

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007ff8dcb7bdcb, pid=3047, tid=0x00007ff89effd700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_121-b13) (build 1.8.0_121-b13)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.121-b13 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# java: /dev/opencv/opencv-3.2.0/modules/objdetect/src/cascadedetect.cpp:922: int cv::CascadeClassifierImpl::runAt(cv::Ptr<cv::FeatureEvaluator>&, cv::Point, int, double&): Assertion `!oldCascade && (data.featureType == FeatureEvaluator::HAAR || data.featureType == FeatureEvaluator ...

(more)

edit retag flag offensive close merge delete

Comments

Maybe try if the problem persists with pure C++ code to eliminate the Java component?

Also, maybe try to use the same OpenCV configuration everywhere and or compare the different OpenCV configurations to see if the issue is due to a particular OpenCV configuration?

Eduardo ( 2017-03-08 14:04:40 -0600 )edit

Thank you for the answer,

Maybe try if the problem persists with pure C++ code to eliminate the Java component?

We can try this way, but what do you think could change? Do you suspect a problem with the Java bindings?

Also, maybe try to use the same OpenCV configuration everywhere and or compare the different OpenCV configurations to see if the issue is due to a particular OpenCV configuration?

What do you mean for '"configuration"? Because we have built the two opencv versions always in the same way, and the Java code is always the same..

marcogovo ( 2017-03-09 03:17:51 -0600 )edit

Debug + multithreading is tricky. The best option for me is to eliminate the most possible number of variables and keep a minimal reproducible code everywhere with the same data (no camera stream, one video and copy the mat to all the threads).

In your code, you load two times the cascade file but it should be useless. I suspect you did that because it did not work with only one call, as other people have experienced it. This is not a normal behavior.

Possible causes:

CascadeClassifier is not thread safe?
your code is not thread safe? (I don't know Java unfortunately but you should post also the Java code to create the threads)

Also, is it normal an i7 with only two cores? Otherwise, number of physical cores could be a hint?

Eduardo ( 2017-03-09 04:15:02 -0600 )edit

btw, you should use either the constructor or the load function, not both (double work done)

can you try to comment the cascade.load() functions there ?

berak ( 2017-03-10 08:56:16 -0600 )edit

Are you sure that the constructor and load() are 'mutually exclusive'? because in some cases, only with the constructor, it not works..Is necessary to call load() to load correctly the xml..

marcogovo ( 2017-03-10 09:58:05 -0600 )edit

because in some cases, only with the constructor, it not work -- that would be very weird.

"mutually exclusive" might not be the right word here. but it's a bad idea, to throw the whole classifier away, and load it a 2nd time, no ? in general, you'd even want to keep the instance alive for your whole program, not construct and load a classifier per image to process.

berak ( 2017-03-10 10:39:22 -0600 )edit

I still don't get the whole idea completly. From what I understand, you launch N threads and N threads execute doFaceRec(). But you say also that doFaceRec() is protected by a mutex. I suppose you did that just for debugging and it is not the real code for the production, otherwise I don't see the point of the multithreading.

For what it's worth, the few advices I could give:

minimal test code in C++
same data for reproducibility
same OpenCV configuration
OpenCV and test code compiled in Debug mode
default parameters for detectMultiScale() to see if it changes something maybe

Eduardo ( 2017-03-10 11:59:54 -0600 )edit

add a comment

answered 2017-03-10 08:40:47 -0600

Amorelli
1

Hello. I work with Marco, who opened this thread. I would like to give some additional information, hoping that it will help. I have run gdb on the core dump of the error.

The issue is triggered by a failing assert() at line 922 of file modules/objdetect/src/cascadedetect.cpp. This is the assert call:

assert( !oldCascade &&
       (data.featureType == FeatureEvaluator::HAAR ||
        data.featureType == FeatureEvaluator::LBP ||
        data.featureType == FeatureEvaluator::HOG) );

Using GDB, I was able to verify the value of the involved variables. More specifically, oldCascade is correctly set to false (making !oldCascade equals to true), while the value of data.featureType is 32729, which does not correspond to any of the values rapresented by the enum within the FeatureEvaluator class, defined as follows:

class FeatureEvaluator
{
public:
    enum
    {
        HAAR = 0,
        LBP  = 1,
        HOG  = 2
    };
    ....
}

Here are the values of the other attributes that compose the structure "CascadeClassifierImpl::data":

{stageType = -2012718384, featureType = 32729, ncategories = 0, minNodesPerTree = 1, maxNodesPerTree = 1, origWinSize = {width = 20, height = 20}, stages = std::vector of length 22, capacity 22 = {... }, classifiers = std::vector of length 2135, capacity 2135 = {...}, nodes = std::vector of length 2135, capacity 2135 = {...}, leaves = std::vector of length 4270, capacity 4270 = {...}, subsets = std::vector of length 0, capacity 0,
stumps = std::vector of length 2135, capacity 4096 = {...}}

We verified experimentally that no two calls to the detectMultiScale() method can occur at once (we have a mutex that controls the access to the doFaceRec() method, which is defined as written by my colleague in the post above). I am thinking that there might some kind of error like an array overflow at some point in the code, that overwrites part of the structure "CascadeClassifierImpl::data"?

The "suspicious" data structure is initialized during the call to the method load() of the CascadeClassifierImpl class, which calls data = Data(), which, in turn, executes the following line of code:

stageType = featureType = ncategories = maxNodesPerTree = 0;

I have looked at the code in the modules/objdetect/src/cascadedetect.cpp file, and I cannot seem to find any place in the code where data.featureType could be changed into any value like 32729. Hence, my concerns on the possibility of an array overflow that overwrites the content of CascadeClassifierImpl::data.

Does anybody have any suggestions for us? Is there anything else that I might discover using GDB?

Thank you everybody for your help and time!

Ciao, Alessandro

edit flag offensive delete link

Comments

Have you tried to switch off OpenCL?

It should exist some tool to detect issues like buffer overflow or memory access violation.

Maybe try with LBP cascade file instead of Haar?

Eduardo ( 2017-03-10 12:10:40 -0600 )edit

add a comment

Errors in detectMultiScale() for face detection

Comments

1 answer

Comments

Links

Question Tools

Stats

Related questions

Errors in detectMultiScale() for face detection edit

Comments

1 answer

Comments

Links

Question Tools

Stats

Related questions

Errors in detectMultiScale() for face detection