Hey,
I'm using OpenCV 3.1 in Java under xubuntu 14.10 to match the camera input with a given image database using the FlannBased DescriptorMatcher (Descriptors precomputed with SURF). The code works perfectly with good results, but after some time (ranging from 10sec to 5min) my JRE crashes with different errors dealing with memory access/allocation, mostly [1] and [2].
Simplified version of my Code:
Thread 1:
while(running) {
synchronized (cameraImage) {
videoCapture.read(cameraImage);
camImage.copyTo(screenImage);
}
matView.show(screenImage); //some selfwritten imshow function (swing JFrame)
}
Thread 2:
while(running) {
synchronized (cameraImage) {
cameraImage.copyTo(matchImage);
}
surfDescriptorExtractor.detectAndCompute(matchImage, descriptors);
flannMatcher.match(descriptor, matches);
return bestMatch(matches);
}
My assumptions:
At first I thought it would be a problem of concurrent access of the camImage, but I have completly synchronized the copy parts.
It does not crash (only tested for ~30min), if I run both parts in one thread or if I do not update the JFrame.
I executed the jar with -verbose:gc option and noticed, that Garbage Collection (gc) is called quite frequently (~every 3sec, depending on JRE's -Xmx memory size) and that every error is directly after a gc. If I do not update the JFrame the gc is called almost never.
So here is my assumption: Thread 1 updates the Swing GUI and creates some Java Objects (mainly a BufferedImage of the Mat), invokes the GC as there is not enough space, and GC crashes the nativly allocated memory on which Thread 2 is currently working on.
One work around which did not crash yet is, that I manually call the GC in Thread 2 before calling any OpenCV methods to hope that it is not called again during those methods.
Thanks in advance for any help!
It would be nice to have a solution that really is safe and still works with two threads.
Errors:
[1]
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f182db1b7e3, pid=28910, tid=139740120311552
#
# JRE version: Java(TM) SE Runtime Environment (8.0_40-b25) (build 1.8.0_40-b25)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C [libc.so.6+0x7e7e3]
[2] *** Error in `java': malloc(): memory corruption: 0x00007efdfc01f780 ***
Edit
I have broken down my code into the essential part. If I compile the following code with
javac -classpath /path/to/opencv-310.jar Main.java
and run
java -Djava.library.path=/path/to/lib -cp .:/path/to/opencv-310.jar Main
it crashes
withing within the first two minutes.
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Random;
import org.opencv.core.Core;
import org.opencv.core.CvType;
import org.opencv.core.DMatch;
import org.opencv.core.Mat;
import org.opencv.core.MatOfDMatch;
import org.opencv.core.MatOfKeyPoint;
import org.opencv.features2d.DescriptorExtractor;
import org.opencv.features2d.DescriptorMatcher;
import org.opencv.features2d.FeatureDetector;
import org.opencv.videoio.VideoCapture;
public class Main {
private static final Mat cameraMat;
static {
System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
cameraMat = new Mat();
}
public static void main(String args[]) {
new Thread(Main::grabCamera).start();
new Thread(Main::workingOnSomething).start();
}
public static void grabCamera() {
VideoCapture videoCapture = new VideoCapture(0);
while (true) {
synchronized (cameraMat) {
videoCapture.read(cameraMat);
}
Runtime.getRuntime().gc();
}
}
public static void workingOnSomething() {
// Init DescriptorExtractor and Matcher
FeatureDetector featureDetector = FeatureDetector.create(FeatureDetector.AKAZE);
DescriptorExtractor descriptorExtractor = DescriptorExtractor.create(DescriptorExtractor.AKAZE);
DescriptorMatcher descriptorMatcher = DescriptorMatcher.create(DescriptorMatcher.BRUTEFORCE);
// Generate some random images as train images
List<mat> images = new ArrayList<>();
Random random = new Random();
byte[] bytes = new byte[10000];
for (int i = 0; i < 100; i++) {
random.nextBytes(bytes);
Mat img = new Mat(100, 100, CvType.CV_8UC1);
img.put(0, 0, bytes);
images.add(img);
}
// Compute descriptors of random images
List<mat> descriptorsList = new ArrayList<>();
List<matofkeypoint> keysList = new ArrayList<>();
featureDetector.detect(images, keysList);
descriptorExtractor.compute(images, keysList, descriptorsList);
// Tran the matcher
descriptorMatcher.add(descriptorsList);
descriptorMatcher.train();
final short[] matchesPerImage = new short[descriptorsList.size()];
final Mat currentImage = new Mat();
while (true) {
// Copy current camera image
synchronized (cameraMat) {
cameraMat.copyTo(currentImage);
}
if (!currentImage.empty()) {
// Extract descriptors of current image
MatOfKeyPoint keys = new MatOfKeyPoint();
Mat descriptors = new Mat();
featureDetector.detect(currentImage, keys);
descriptorExtractor.compute(currentImage, keys, descriptors);
// Compute matches
MatOfDMatch matches = new MatOfDMatch();
descriptorMatcher.match(descriptors, matches);
// Find best match
Arrays.fill(matchesPerImage, (short) 0);
DMatch[] matchesArray = matches.toArray();
for (DMatch match : matchesArray) {
matchesPerImage[match.imgIdx]++;
}
int index = 0;
for (int i = 0; i < matchesPerImage.length; i++) {
if (matchesPerImage[i] > matchesPerImage[index]) {
index = i;
}
}
}
}
}
}
