cv::dnn::Net::forward() returns wrong output for intermediate output layer of network
I try to launch intel inference engine example under OpenCV.
Link to example info: https://software.intel.com/en-us/arti...
Security Barrier Camera Demo
I use (OS: Win10) OpenCV 3.4.3 compiled with inference engine.
In particular I have problems with following model /intel_models/vehicle-attributes-recognition-barrier-0039. This pretrained model (.xml + .bin files) runs successfully in intel inference engine demo app. This net has two output softmax layers ("color" and "type", "type" is the final network layer so its result is returned from net.forward())
Piece of code:
try
{
cv::dnn::Net net = cv::dnn::readNet(model, config);
net.setPreferableBackend(cv::dnn::DNN_BACKEND_DEFAULT);
net.setPreferableTarget(cv::dnn::DNN_TARGET_CPU);
if (net.empty())
{
std::cerr << "Failed! Net is empty!\n";
return EXIT_FAILURE;
}
else
{
std::cout << "Success!\n";
cv::Mat car = cv::imread(file);
if (car.empty())
{
std::cerr << "Failed to load " << file << '\n';
return EXIT_FAILURE;
}
cv::Mat conv_car;
car.convertTo(conv_car, CV_32FC3);
int w = 72, h = 72;
std::cout
<< "car type: " << car.type() << '\n'
<< "conv car type: " << conv_car.type() << '\n';
while (true)
{
auto input = cv::dnn::blobFromImage(
conv_car,
1.,
cv::Size((int)w, (int)h),
cv::Scalar(0, 0 ,0),
false,
false,
CV_32F
);
net.setInput(input);
cv::Mat output = net.forward();
output = output.reshape(1, 1);
std::cout
<< "output.size(): " << output.size() << '\n'
<< "output.elemSize(): " << output.elemSize() << '\n'
<< "output.data():\n";
for (size_t i = 0; i < output.cols; ++i)
std::cout << output.at<float>(0, i) << ' ';
cv::Point classIdPoint;
double confidence;
minMaxLoc(output, 0, &confidence, 0, &classIdPoint);
int classId = classIdPoint.x;
std::cout << "\nclass id: " << classId << " confidence: " << confidence << '\n';
cv::imshow("img", car);
char k = cv::waitKey(0) % 256;
if (k == 27)
break;
}
}
}
catch (cv::Exception& e)
{
std::cout << "Failed!\n";
std::cout << e.what() << '\n';
}
When I call
net.forward()
or
net.forward("type")
I get a reasonable output which match one of intel native sample
output.size(): [4 x 1]
output.elemSize(): 4
output.data():
0.999998 2.33588e-09 6.7388e-09 2.31872e-06
But when I call
net.forward("color")
I get strange output different every time. Though output size looks correct.
output.size(): [7 x 1]
output.elemSize(): 4
output.data():
2.57396e-12 2.61858e+20 7.58788e+31 1.94295e-19 7.71303e+31 5.08303e+31 1.80373e+28
class id: 4 confidence: 7.71303e+31
UPDATE (see berak's comments):
It's possible to get correct output from both layers using following syntax:
vector<Mat> outputs;
vector<String> names {"type","color"};
net.forward(outputs, names);
??? But how to get correct result when call:
net.forward("color")
The only idea I have is that to get the output of intermediate layer in some specific way.
Linkto pretrained network: https://download.01.org/openvinotoolk...
just a bold guess: can you try to print out:
output.size
instead ofoutput.size()
? (it might have more than 2 dims)does it mean, you don't know how to ? or did you get any errors ?
call to
output.size
returns1x7
and1x4
respectively so it's similar tooutput.size()
I am not sure that I have used
net.forward(std::vector< std::vector< Mat > > & outputBlobs, const std::vector< String > & outBlobNames)
correctly. Since conversion tostd::vector< std::vector< Mat > >
is not straightforward for me.forward() for multiple outputs would be:
(try with a single vector, not a nested one)
see example here
as much as i'd like to test your code / model, i can't, lacking IE support ;(
You are right. This is a proper way to use
net.forward()
to get multiple outputs. But I still have a question why attempt to get output of intermediate layer gives wrong results.@f4f, Which version of IE is used? Have you tried OpenCV 3.4.4? Have you tried to use OpenCV from OpenVINO package? BTW, OpenCV 3.4.3 released in August but OpenVINO R4 has been released in 12 Nov 2018. You'd better build OpenCV 3.4.3 with R3 or 3.4.4 with R4
Opencv is build with IE 1.0.13911 However I took MKLDNNPlugin.dll from fresh IE 1.0.17328 . Maybe it causes the problem.
@f4f, Just try to keep things synced. We can support compatibility of fresh versions of OpenCV with already released versions of Inference Engine but using old OpenCV with IEs from future is a kind of magic right now because both are actively developed.
Wow, and it'd be better if you mentioned Windows OS in question but not in comments :)
@dkurt, Should have mentioned Windows OS, its my fault.
I also get (in different working pipeline) following "assertion failed" inputs.size() == requiredOutputs in function 'cv::dnn::experimental_dnn_34_v8::DataLayer::getMemoryShapes'
What can cause it ? I will try to build a debug build and see what goes wrong, but for now debug build is broken.
@f4f, let's see. Perhaps we can catch it if one forgot to call
setInput
of something else. This way we can modify this assertion to something more useful.@dkurt , It was a typo in my source code which cause a call to uninitialized network, thanks a lot.
I will update the topic to note whether there is a correct output from
net.forward("intermediate_layer_name")
with synced inference engine versions.@dkurt, I have tested my app after rebuild opencv 3.4.3 and intel IE 1.0.17328. Also I have rechecked consistency off all used IE's libs and dlls (now its surely consistent). There is still an wrong output for:
cv::Mat output = net.forward("intermideate_layer_name";
output = output.reshape(1, 1);
for (size_t i = 0; i < output.cols; ++i)
std::cout << output.at<float>(0, i) << ' ';
Even though there is a correct output for:
vector<Mat> outputs;
vector<String> names {"type","color"};
net.forward(outputs, names);
@f4f, Could you please clarify why you cannot use OpenCV 3.4.4. and Inference Engine from OpenVINO R4? And what is
1.0.17328
?@f4f,
17328
is a build number of Inference Engine from OpenVINO R4 (November, 2018). OpenCV 3.4.3 released in August. If you want to use OpenCV 3.4.3 - build it with Inference Engine from OpenVINO R3 (build number13911
). If you want to use IE from OpenVINO R4 - build it with OpenCV 3.4.4.@dkurt, yes its a build number for IE shipped with Intel OpenVINO toolkit. I will use 13911 for OpenCV 3.4.3 as you suggest. Thanks a lot.