Ask Your Question
2

cv::dnn::Net::forward() returns wrong output for intermediate output layer of network

asked 2018-12-06 06:21:11 -0500

f4f gravatar image

updated 2018-12-10 00:55:38 -0500

I try to launch intel inference engine example under OpenCV.

Link to example info: https://software.intel.com/en-us/arti...

Security Barrier Camera Demo

I use (OS: Win10) OpenCV 3.4.3 compiled with inference engine.

In particular I have problems with following model /intel_models/vehicle-attributes-recognition-barrier-0039. This pretrained model (.xml + .bin files) runs successfully in intel inference engine demo app. This net has two output softmax layers ("color" and "type", "type" is the final network layer so its result is returned from net.forward())

Piece of code:

 try
{
    cv::dnn::Net net = cv::dnn::readNet(model, config);
    net.setPreferableBackend(cv::dnn::DNN_BACKEND_DEFAULT);
    net.setPreferableTarget(cv::dnn::DNN_TARGET_CPU);


    if (net.empty())
    {
        std::cerr << "Failed! Net is empty!\n";
        return EXIT_FAILURE;
    }
    else
    {
        std::cout << "Success!\n";

        cv::Mat car = cv::imread(file);

        if (car.empty())
        {
            std::cerr << "Failed to load " << file << '\n';
            return EXIT_FAILURE;
        }
        cv::Mat conv_car;
        car.convertTo(conv_car, CV_32FC3);

        int w = 72, h = 72;
        std::cout
            << "car type: " << car.type() << '\n'
            << "conv car type: " << conv_car.type() << '\n';

        while (true)
        {

            auto input = cv::dnn::blobFromImage(
                conv_car,
                1.,
                cv::Size((int)w, (int)h),
                cv::Scalar(0, 0 ,0),
                false,
                false,
                CV_32F
            );
            net.setInput(input);
            cv::Mat output = net.forward();
            output = output.reshape(1, 1);
            std::cout
                << "output.size(): " << output.size() << '\n'
                << "output.elemSize(): " << output.elemSize() << '\n'
                << "output.data():\n";
            for (size_t i = 0; i < output.cols; ++i)
                std::cout << output.at<float>(0, i) << ' ';

            cv::Point classIdPoint;
            double confidence;
            minMaxLoc(output, 0, &confidence, 0, &classIdPoint);
            int classId = classIdPoint.x;
            std::cout << "\nclass id: " << classId << " confidence: " << confidence << '\n';

            cv::imshow("img", car);
            char k = cv::waitKey(0) % 256;
            if (k == 27)
                break;
        }
    }
}
catch (cv::Exception& e)
{
    std::cout << "Failed!\n";
    std::cout << e.what() << '\n';
}

When I call

net.forward()

or

net.forward("type")

I get a reasonable output which match one of intel native sample

output.size(): [4 x 1]
output.elemSize(): 4
output.data():
0.999998 2.33588e-09 6.7388e-09 2.31872e-06

But when I call

net.forward("color")

I get strange output different every time. Though output size looks correct.

output.size(): [7 x 1]
output.elemSize(): 4
output.data():
2.57396e-12 2.61858e+20 7.58788e+31 1.94295e-19 7.71303e+31 5.08303e+31 1.80373e+28
class id: 4 confidence: 7.71303e+31

UPDATE (see berak's comments):

It's possible to get correct output from both layers using following syntax:

vector<Mat> outputs;
vector<String> names {"type","color"};
net.forward(outputs, names);

??? But how to get correct result when call:

net.forward("color")

The only idea I have is that to get the output of intermediate layer in some specific way.

Linkto pretrained network: https://download.01.org/openvinotoolk...

edit retag flag offensive close merge delete

Comments

just a bold guess: can you try to print out: output.size instead of output.size() ? (it might have more than 2 dims)

I have also failed to correctly use net.forward() overloads for multiple outputs.

does it mean, you don't know how to ? or did you get any errors ?

berak gravatar imageberak ( 2018-12-06 06:27:55 -0500 )edit
1

call to output.size returns 1x7 and 1x4 respectively so it's similar to output.size()

f4f gravatar imagef4f ( 2018-12-06 06:41:07 -0500 )edit

I am not sure that I have used net.forward(std::vector< std::vector< Mat > > & outputBlobs, const std::vector< String > & outBlobNames) correctly. Since conversion to std::vector< std::vector< Mat > > is not straightforward for me.

f4f gravatar imagef4f ( 2018-12-06 06:44:53 -0500 )edit
1

forward() for multiple outputs would be:

vector<Mat> outputs;
vector<String> names {"type","color"};
net.forward(outputs, names);

(try with a single vector, not a nested one)

see example here

as much as i'd like to test your code / model, i can't, lacking IE support ;(

berak gravatar imageberak ( 2018-12-06 06:47:37 -0500 )edit
1

You are right. This is a proper way to use net.forward() to get multiple outputs. But I still have a question why attempt to get output of intermediate layer gives wrong results.

f4f gravatar imagef4f ( 2018-12-06 06:59:56 -0500 )edit

@f4f, Which version of IE is used? Have you tried OpenCV 3.4.4? Have you tried to use OpenCV from OpenVINO package? BTW, OpenCV 3.4.3 released in August but OpenVINO R4 has been released in 12 Nov 2018. You'd better build OpenCV 3.4.3 with R3 or 3.4.4 with R4

dkurt gravatar imagedkurt ( 2018-12-06 07:11:19 -0500 )edit

Opencv is build with IE 1.0.13911 However I took MKLDNNPlugin.dll from fresh IE 1.0.17328 . Maybe it causes the problem.

f4f gravatar imagef4f ( 2018-12-06 07:25:22 -0500 )edit

@f4f, Just try to keep things synced. We can support compatibility of fresh versions of OpenCV with already released versions of Inference Engine but using old OpenCV with IEs from future is a kind of magic right now because both are actively developed.

dkurt gravatar imagedkurt ( 2018-12-06 07:28:52 -0500 )edit

Wow, and it'd be better if you mentioned Windows OS in question but not in comments :)

dkurt gravatar imagedkurt ( 2018-12-06 07:30:30 -0500 )edit

@dkurt, Should have mentioned Windows OS, its my fault.

I also get (in different working pipeline) following "assertion failed" inputs.size() == requiredOutputs in function 'cv::dnn::experimental_dnn_34_v8::DataLayer::getMemoryShapes'

What can cause it ? I will try to build a debug build and see what goes wrong, but for now debug build is broken.

f4f gravatar imagef4f ( 2018-12-06 07:38:43 -0500 )edit

1 answer

Sort by ยป oldest newest most voted
1

answered 2018-12-10 00:57:34 -0500

f4f gravatar image

According to berak's comment:

forward() for multiple outputs would be:

vector<Mat> outputs;
vector<String> names {"type","color"};
net.forward(outputs, names);

Example link:https://github.com/opencv/opencv/blob...

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2018-12-06 06:21:11 -0500

Seen: 28 times

Last updated: Dec 10