Multi-person poses' detection using OpenCV + OpenPose
Hello, dear colleagiues!
I recently started studying computer vision. I'm very interested this direction. Now I'm studying the recognition of human poses, and I'm trying to remake some examples in C # to experiment further in Unity.
However, I can not find documentation or examples of how I could improve my code anywhere to find the poses of all people in the image. Could you tell me how I should change my code so that I can recognize the poses for all people in the image?
CODE FRAGMENT:
private void RecognizePose( Mat image, Mat output_image, List<point> points, float[] data ) {
for( int i = 0; i < BODY_PARTS_MPI.Count; i++ ) {
output_image.get( i, 0, data );
Mat heat_map = new Mat ( 1, data.Length, CvType.CV_32FC1 );
heat_map.put( 0, 0, data );
//Originally, we try to find all the local maximums. To simplify a sample
//we just find a global one. However only a single pose at the same time
//could be detected this way.
Core.MinMaxLocResult result = Core.minMaxLoc( heat_map );
heat_map.Dispose();
double x = (
(image.cols() *
(result.maxLoc.x % matrix_output_columns)) /
matrix_output_columns
);
double y = (
(image.rows() *
(result.maxLoc.x / matrix_output_rows)) /
matrix_output_rows
);
if( result.maxVal > 0.1d ) points.Add( new Point( x, y ) );
else points.Add( null );
}
}
Line "Core.MinMaxLocResult result = Core.minMaxLoc( heat_map );" and next processing gets us only single person's pose. And comment for this line says the same. But how can I get the all local maximumns and detect all poses?
With best regards and best wishes, Sergey Netyagin
does this (java?) code even work for a single person ?
i'm having some doubts, if
output_image.get( i, 0, data );is retrieving the right thing, imho, there should be aoutput_image=output_image.reshape(1,19);before trying to access it's dataYes..!! It works perfectly fine for a single person.
The code is available at this link: https://github.com/opencv/opencv/blob...
The step to reproduce is as follows: python3 openpose.py --input=COCO_val2014_000000000589.jpg --proto=openpose_pose_coco.prototxt --model=pose_iter_440000.caffemodel --dataset=COCO
The data is available at:
--> COCO_val2014_000000000589.jpg
https://github.com/CMU-Perceptual-Com...
--> openpose_pose_coco.prototxt
https://github.com/opencv/opencv_extr...
--> pose_iter_440000.caffemodel
http://posefs1.perception.cs.cmu.edu/...
i know, i know.
i was only questioning your java version of it. (are you sure, you got the heatmap extraction right ?)
I am trying to implement it on android (java). The code works on ubuntu with opencv 3.4.1 but I am getting error on android with opencv 3.4.1. I have explained the error here: https://github.com/opencv/opencv/issu...
I tried the code on some images till now. It does fail sometimes.
Any help would be appreciated.
could you add that part of the code to your question ?
(btw, your github repo is no more available)
Should I upload the Android project on my Github repo or should I paste the code here?
I think uploading would be a better option because the entire code would be visible there with indentation.
a few lines with the result parsing here AND your whole code on github, maybe ?
I am getting error in net.forward()
I have uploaded the entire code on github.
github link ?
maybe you can try this one in the meantime
The android project is about 700mb. Can we upload that on github?
no, you can't.
put your main java file somewhere. above code isn't all too helpful
"I am getting error in net.forward()" -- do we have to guess the error ?
I am extremely sorry. I have uploaded the main java file here . I have uploaded a file named error.txt also which contains the error.
error(): OpenCV(3.4.1) Error: Assertion failed (inputs[0]->size[1] % blobs[0].size[1] == 0) in virtual void cv::dnn::ConvolutionLayerImpl::forward(std::vector<cv::mat*>&, std::vector<cv::mat>&, std::vector<cv::mat>&), file /build/master_pack-android/opencv/modules/dnn/src/layers/convolution_layer.cpp, line 962
@Gajjar_Mihir ,
1.0/255 , not 1 here
you need to reshape the output to 19 rows here
float, not double here
again, take a look here, it worked ! (also somewhat easier, without the string lookups)
I think the point float, not double here does not point to the correct line. I tried with the previous two changes but the image remains CV_8UC4 only and the same error persists.
And the code openpose.java sample. Should I try it with android or with normal java compiler?
indeed, l. 153 instead.
you need to cvtColor() your image, COLOR_ARGB2BGR
And also for lines 157 and 158 ?
I cannot use Float at line 153. It shows that out.get(0,i) is a double[] and hence I cannot use MatofFloat there.
no, it does not matter there. (your're puttiing x,y in an integer Point anyway later)
line 153 should probably be:
get(i,j) only returns a single pixel (as an array of doubles) , so that's wrong anyway.
did not see it before, sorry.
I am not able to find COLOR_ARGB2BGR also.
Can we reduce the number of channels using mat1 = mat1.reshape(3) ?
no you can't reshape that, you have to throw away the (unused) A channel. it's probably BGRA2BGR then, but idk. (not using android) try to see, if the image looks right, afterwards.
Okay so I think there is some progress. Now I am getting the points but only the x coordinates. All the y coordinates are zero. And also I am getting no error.
yea, use heatmap.reshape(1,46) instead, so you have a 46x46 image for minmaxloc. that will also have proper x,y coords
Yes. Now it works perfectly fine. I tried it out on several images and it works fine. Thank you very much for your help. I really appreciate it. Is there any way to improve the predictions. Like should I have a better quality image or something like that?
ah, cool. how long does it take ?
It takes about 40 seconds on my Xiaomi Redmi Note 4 with the following specifications: 1. Snapdragon 625 processor 2. 4GB RAM 3. CPU: Octa-core 2.0 GHz Cortex-A53 4. GPU: Adreno 506
I am just getting the points. Is there any way to show these points in the image like draw the skeleton overlapping the image?
just draw lines connecting them ? look at the c++ / python / java samples
Yeah the lines can be drawn but the coordinates which I am getting do not correspond to the pixels of the original image. So how can the coordinates be overlapped on the image?
And are you also getting the same time about 40 seconds or is there anyway to optimize the time?
i get like 30 seconds on a laptop (there is another , tf trained network, that can run in 3 seconds, but it is less accurate)
"but the coordinates which I am getting do not correspond to the pixels of the original image." -- how so ? did you miss the scaling from heatmap to img coords ? again, look at the existing examples, it's a solved problem.
Yes I haven't scaled the heatmap. I tried to find it in the example openpose.java. How should I convert them so that they correspond to the pixels of the original image?
https://gist.github.com/berak/eef67a1...
https://gist.github.com/berak/eef67a1...
Yes. It works perfectly fine. Thank you for your help. Again I really appreciate your help :) :)
I am just curious to know how opencv does this.
I am also trying to use the .pb file provided by ildoonet on Github to get the predictions. I was able to get the output of the last node but I still dont know how to process it to get the coordinates(predictions of the joints).
Does opencv use this .pb file or it does something else?
again, look at the java example, and toggle the comments for that
ohh ok. So is there any model which gives better predictions than this (pose_iter_440000.caffemodel) ?