Ask Your Question
0

Create DLL using TrackerKCF for a Unity Project - Crash when updating the tracker with the DLL

asked 2018-11-02 10:07:40 -0500

DJ7179D gravatar image

updated 2018-11-02 11:50:41 -0500

Hi guys !

I'd like to use some OpenCV Tracker's inside a Unity Project. To do so, I created a .dll file that I imported inside my unity project.

Here is my code to create the DLL :

cv::Ptr<cv::TrackerKCF> Tracker;
cv::Rect2d roi;

// To use image date from Unity
struct Color32
{
    uchar r;
    uchar g;
    uchar b;
    uchar a;
};

// To pass ROI's position information beetwen c# and c++
struct Square
{
    Square(int x, int y, int width, int height) : X(x), Y(y), Width(width), Height(height) {}
    int X, Y, Width, Height;
};

extern "C" bool __declspec(dllexport) __stdcall InitTracker(Color32* frameData, int cameraWidth, int cameraHeight, Square & square)
{
    Tracker = cv::TrackerKCF::create();
    cv::Mat frame(cameraHeight, cameraWidth, CV_8UC4, frameData);

    roi = selectROI("Selection", frame);
    square = Square(roi.x, roi.y, roi.width, roi.height);

    // return true if the init is OK, false otherwise
    bool success = Tracker->init(frame, roi);

    cv::rectangle(frame, roi, cv::Scalar(0, 0, 255));
    cv::imshow("Init", frame);

    return success;
}

extern "C" bool __declspec(dllexport) __stdcall UpdateTracker(Color32* frameData, int cameraWidth, int cameraHeight, Square & square)
{
    cv::Mat frame(cameraHeight, cameraWidth, CV_8UC4, frameData);
    cv::Mat copyToCheck(frame);

    cv::rectangle(copyToCheck, roi, cv::Scalar(0, 0, 255));
    cv::imshow("before Update", copyToCheck);
    cv::waitkey(0);

    bool success = Tracker->update(frame, roi);

    cv::rectangle(copyToCheck, roi, cv::Scalar(0, 0, 255));
    cv::imshow("after Uptade", copyToCheck);
    cv::waitkey(0);

    square = Square(roi.x, roi.y, roi.width, roi.height);
    return success;
}

extern "C" void __declspec(dllexport) __stdcall Close()
{
    cv::destroyAllWindows();
}

And here is my c# script :

using System;
using System.Runtime.InteropServices;

using Uk.Org.Adcock.Parallel;

using UnityEngine;

// Define the structure to be sequential and with the correct byte size (4 ints = 4 bytes * 4 = 16 bytes)
[StructLayout(LayoutKind.Sequential, Size = 16)]
public struct CvSquare
{
    public int X, Y, Width, Height;

    public CvSquare(int x, int y, int width, int height) : this()
    {
        this.X = x;
        this.Y = y;
        this.Width = width;
        this.Height = height;
    }
}

// To import the DLL's methods
internal static class TrackingCpp
{
    [DllImport("TrackingCppToDLL")]
    internal static extern void Close();

    [DllImport("TrackingCppToDLL")]
    internal static extern bool InitTracker(Color32[] frameData, int cameraWidth, int cameraHeight, ref CvSquare square);

    [DllImport("TrackingCppToDLL")]
    internal static extern bool UpdateTracker(Color32[] frameData, int cameraWidth, int cameraHeight, ref CvSquare square);
}

public class TrackingFromDLL : MonoBehaviour
{
    private Color32[] pixels32;
    private WebCamTexture _webcamTexture;
    private const int imWidth = 640;
    private const int imHeight = 480;
    private CvSquare square;

    void Start()
    {
        // initialized the webcam texture
        WebCamDevice[] devices = WebCamTexture.devices;
        _webcamTexture = new WebCamTexture(devices[0].name, imWidth, imHeight);

        // Play the video source
        _webcamTexture.Play();
    }

    void Update()
    {
        pixels32 = _webcamTexture.GetPixels32();
        int nbPixel = pixels32.Length;
        Color32[] reversePixels32 = new Color32[nbPixel];
        Parallel.For(0, imHeight, i =>
        {
            for (int j = 0; j < imWidth; ++j)
            {
                // OpenCV's pixel (0,0) is the top-left pixel
                int newPos = nbPixel - (i + 1) * imWidth + j;
                reversePixels32[newPos] = pixels32[i * imWidth + j];

                // OpenCV prefers BGRA instead of RBGA
                byte tmp = reversePixels32[newPos].b;
                reversePixels32[newPos].b = reversePixels32[newPos].r;
                reversePixels32[newPos].r = tmp;
            }
        });

        if (!isInitDone)
            isInitDone = TrackingCpp.InitTracker(reversePixels32 ...
(more)
edit retag flag offensive close merge delete

Comments

we probably can't help you with a clearly unity related problem, but:

"crash" means what, exactly ?

opencv version ?

is there any chance, your c++ code is used from more than 1 thread ?

(the Tracker is for sure not threadsafe ! (and you got a global variable there ...))

berak gravatar imageberak ( 2018-11-02 10:26:29 -0500 )edit
1

Thx for the answer !

There is no chance that my c++ code is used from more than 1 thread, but still, I have no idea how to not have a global variable as the tracker needs an init and then to be updated.

OpenCV 3.4.1

"Crash" means that a window "Unity has stopped working" with the sentence "There is no fix available" appears and then Unity is closed

DJ7179D gravatar imageDJ7179D ( 2018-11-02 10:35:42 -0500 )edit

i hate to say that, but: would you be able to (remote) debug it ? (Visual Studio's debugger is great at that !)

berak gravatar imageberak ( 2018-11-02 11:01:48 -0500 )edit

let me pick a last time on the threading issue. all of opencv's gui functions (imshow(), waitKey(), namedWindow()) MUST be on the main thread. (try to remove them, for a second)

how do you even see anything, not calling cv::waitKey() anywhere ?

and how do you know, it's the Tracker, which fails ?

(and, btw, you should draw your rectangle AFTER you init the tracker, not before (else it tries to track that, too))

berak gravatar imageberak ( 2018-11-02 11:27:37 -0500 )edit
1

I'll remotely debug it during the night and I'll keep you updated.

The selectROI method blocks the processing until an initial box is selected. As for the cv::waitKey, there is actually 1 just before the Tacker->Update(...) method (thx for noticing, I forget to put it, I'll update the post)

As for the knowledge that it's the Tracker that's fail, I just commented the updated method and as expected, I have my webcam running with the choosen box added on the images.

But when I'm actually trying to update the box's positions and size, it fails

DJ7179D gravatar imageDJ7179D ( 2018-11-02 11:37:01 -0500 )edit

Normally, I should use the Tracker->Update(..) return's value (true/false) to display or not the updated box in the image (false means that no good match has been found and that the ROI hadn't been modified)

But for now, I'm just trying to have a return without unity stops to work ^^'

DJ7179D gravatar imageDJ7179D ( 2018-11-02 11:43:29 -0500 )edit

yea, true. the update() return value has some flaws, too, you can't re-init() the tracker, but have to create a new one (bug, somewhat)

and 3.4.1 won't ever return anything but true from there (bug, too, but at least fixed in recent 3.4.3)

berak gravatar imageberak ( 2018-11-02 11:46:50 -0500 )edit
1

I uptaded the post with the good code (my mistake with the rectangle in the Init and the Displays I didn't copy/paste in the update)

Btw, this is not the subject of this topic but are you sure about the flaws of the Update's return values ? I already noticed about the impossibility of re-initiating without re-creating but when I use it in my c++ project, if the tracking is lost I do not have any boxes on the screen. (I didn't check the return's values in a log or something, I just coded to stop the display if a false is return).

DJ7179D gravatar imageDJ7179D ( 2018-11-02 11:57:25 -0500 )edit

i'm not sure when it exactly it was fixed for the KCF tracker (we'd have to look up the pr's), but afaik it was done way after the 3.4.1 release you're using

strike that, i'm wrong.

berak gravatar imageberak ( 2018-11-02 12:02:08 -0500 )edit

About the debugging in mixed mode, do you know how to do that within this kind of project ?

I'm using VS 2017, and I have the feeling that mixed mode debugging doesn't work. Here, a guy said that "You can either debug managed code in VS, or debug native code, but not both at the same time." (last answer of the topic).

And I need to debug both because it is when my managed code calls my native code (dll) that Unity crashes and says "There is no fix available"

This so frustrating to have a code working in C++ and not being able to use it inside a Unity Project :/

DJ7179D gravatar imageDJ7179D ( 2018-11-04 10:46:48 -0500 )edit

1 answer

Sort by ยป oldest newest most voted
2

answered 2018-11-06 08:49:13 -0500

DJ7179D gravatar image

updated 2018-11-06 08:51:22 -0500

I finally found out the solution and I'd like to thanks berak for his reactivity and advices. First of all, I updated my OpenCV version to the 3.4.3 to avoid any issues with any bugs in the 3.4.1.. I still had the same issue.


To understand the following corrections :

Inside my .dll, the failure was there : bool success = Tracker->update(frame, roi); I then went in github to check the Tracker.cpp src code and the method updateImpl of trackerKCF.cpp. In order to check the whole code. The assert CV_Assert(img.channels() == 1 || img.channels() == 3); reminded me that in my code, as Unity gives RGBA images, I naturally used RGBA images in my .dll... This why it crashed !

Remove the uchar a; in the struct Color32 inside the DLL :

struct Color32
{
    uchar b;
    uchar g;
    uchar r;
};

Create a another struct CvColor and perform the compute the image for the .dll in the c# script :

...

// OpenCV prefers BGR instead of RGB
[StructLayout(LayoutKind.Sequential, Size = 3)]
public struct CvColor
{
    public byte B, G, R;
}

...

public class TrackingFromDLL : MonoBehaviour
{    
    ...

    void Update()
    {
        pixels32 = _webcamTexture.GetPixels32();
        int nbPixel = pixels32.Length;
        CvColor[] reverseCvPixels32 = new CvColor[nbPixel];
        Parallel.For(0, imHeight, i =>
        {
            for (int j = 0; j < imWidth; ++j)
            {
                Color32 currentColor = pixels32[i * imWidth + j];

                // OpenCV's pixel (0,0) is the top-left pixel
                int newPos = nbPixel - (i + 1) * imWidth + j;
                reverseCvPixels32[newPos].B = currentColor.b;
                reverseCvPixels32[newPos].G = currentColor.g;
                reverseCvPixels32[newPos].R = currentColor.r;
            }
        });

        // Do whatever you want with the DLL and reverseCvPixels32
        ...
    }
...
}

Everything works fine now and you can do very various things within unity and the tracking from OpenCV ! (A Space Invader where the spaceship move with your hand or your face for example).

edit flag offensive delete link more

Comments

thanks for solving this mystery ;)

berak gravatar imageberak ( 2018-11-06 08:50:32 -0500 )edit

It is still weird and frustrating that Unity prefers to crash and say that no fixes are available instead of showing the CV_ASSERT :/

DJ7179D gravatar imageDJ7179D ( 2018-11-06 08:53:10 -0500 )edit

yea indeed ;( and it's not open source, so no way to send proper issues, i guess ;(

berak gravatar imageberak ( 2018-11-06 09:03:32 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2018-11-02 10:07:40 -0500

Seen: 111 times

Last updated: Nov 06 '18