bfc_opencv's profile - activity

I want the consumer process to block on get() because it should wait until there's something for it to pass to the caffe net. The main process doesn't block because I use empty() to check before calling get(). But I was wondering why I got the same result when I ran them separately. Why would caffes data transfer interfere with video capture?

I'm not sure if it's the structure of the code itself because when I ran a program that just streamed from the webcam without doing anything else and in a separate window ran a caffe prediction in a loop over and over I got a slowdown in the webcam stream.

I have 2 cores and 4 with hyperthreading (i7 6500U CPU). i don't think they're holding up each other since I'm using multiprocessing instead of multi threading. I also don't believe I'm running any blocking methods.

I timed both the get and put calls and both of them are pretty much very close to 0 in their timings. I don't think that's the bottle neck. I know a lot of it is coming from waitKey() but I know a lot of the rendering for imshow() happens there. Any idea what might be going on here? Been stuck on this one for months

Well I passed dummy text from the consumer process back to the main process (in lieu of doing the actual neural net prediction) and experienced no slowdown. I've added that test code to the answer.

It's not consistent. The fps varies between 13-18 ish. I'd say the average is around 15.

Not sure that would make sense in this context since I'm trying to do object classification in real time. The GPU predictions are slower than the frame rate, but that's why I'm doing it in a separate python process that shouldn't really be using any CPU at all. I'm ok with lag between the caption on the stream and the what the stream is showing, since the predictions are slower. But I don't want the webcam stream itself to lag, which is what seems to be happening. It happens even when I run the two programs separately.

Wrapping with main didn't make a difference. Any other suggestions?

I can't use mutithreading for this purpose because of python's GIL lock. I'll try the main wrapping and report back to you.

The first was a type I meant to use frame instead of image. I know putText() needs more parameters, but this is just a stub. The code runs. It's just slow.

I'm using the multiprocessing module in python to help speed up my main program. However I believe I might be doing something incorrectly, as I don't think the computations are happening quite in parallel.

I want my program to read in images from a video stream in the main process, and pass on the frames to two child processes that do computations on them and send text back (containing the results of the computations) to the main process.

However, the main process seems to lag when I use multiprocessing, running about half as fast as without it, leading me to believe that the processes aren't running completely in parallel.

After doing some research, I surmised that the lag may have been due to communicating between the processes using a queue (passing an image from the main to the child, and passing back text from child to main).

However I commented out the computational step and just had the main process pass an image and the child return blank text, and in this case, the main process did not slow down at all. It ran at full speed.

Perhaps an easier way to think of this is that I'm doing image recognition on what the webcam sees. I send one frame at a time to a separate process to do recognition analysis on the frame, and send the text back to be put as a caption on the live feed. Obviously the processing takes more time than simply grabbing frames from the webcam and showing them, so if there is a delay in what the caption is and what the webcam feed shows, that's acceptable and expected.

What's happening now is that the live video I'm displaying is lagging due to the other processes (when I don't send frames to the process for computing, there is no lag). I've also ensured only one frame is enqueued at a time so avoid overloading the queue and causing lag.

Here's the code, if anyone can help me use OpenCV to stream video from the webcam without lag, that would greatly be appreciated.

class Consumer(multiprocessing.Process):

    def __init__(self, task_queue, result_queue):
        multiprocessing.Process.__init__(self)
        self.task_queue = task_queue
        self.result_queue = result_queue
        #other initialization stuff

    def run(self):
        while True:
            image = self.task_queue.get()
            #Do computations on image
            self.result_queue.put("text")

        return

import cv2

tasks = multiprocessing.Queue()
results = multiprocessing.Queue()
consumer = Consumer(tasks,results)
consumer.start()

#Creating window and starting video capturer from camera
cv2.namedWindow("preview")
vc = cv2.VideoCapture(0)
#Try to get the first frame
if vc.isOpened():
    rval, frame = vc.read()
else:
    rval = False

while rval:
    if tasks.empty():
       tasks.put(frame)
    else:
       text = tasks.get()
       #Add text to frame
       cv2.putText(frame,text,(50,50), cv2.FONT_HERSHEY_SIMPLEX, 1, 255)

    #Showing the frame with all the applied modifications
    cv2.imshow("preview", frame)

    #Getting next frame from camera
    rval, frame = vc.read()

I tried using that flag but the code still seems to hang on the calibrateCamera() step. At 150 images its been running for a while now and still no result. At about 750 frames (the length of the full video) I left it running all night and it still was on the same step. For reference, just 100 frames took 6 minutes.

The only thing I saw that might throw it off was that some frames had the chessboard corners found in a rotated order compared to others. Is that an issue?

Also, when I used CALIB_USE_INTRINSIC_GUESS for 100 frames (my guess being the results from the original calibration of 10 images), my results were quite different from when I didn't use the flag. Which is better?

So I've been working on this, and I noticed the calibrateCamera function freezes if I try to pass more than about a 100 sets of objectPoints and imagePoints to it. Why is that? Should I just take batches of 100 random frames from the video and average the resulting cameraMatrix and distortion outputs?

Got it, thanks!

So if I understand you correctly, I just need to rotate and translate the axis using rvecs and tvecs, and that should be what I need.

The problem is, I use solvePnP to get the rotation and translation for a face. And I draw an axis on the face and use projectPoints to draw it on the image plane. I want to find the world coordinates of the axis after its rotated. What I did was find the rotation matrix using Rodrigues, and do rotation_matrix*axis_point + tvec. And then used projectPoints to project the result of that onto the image plane (passing identity matrix and zero matrix for rotation, tvec respectively). But the result of that was different than what I got from just using projectPoints normally. Why is that?

I'm doing something similar to the tutorial here: http://docs.opencv.org/3.1.0/d7/d53/t... regarding pose estimation. Essentially, I'm creating an axis in the model coordinate system and using ProjectPoints, along with my rvecs, tvecs, and cameraMatrix, to project the axis onto the image plane.

In my case, I'm working in the world coordinate space, and I have an rvec and tvec telling me the pose of an object. I'm creating an axis using world coordinate points (which assumes the object wasn't rotated or translated at all), and then using projectPoints() to draw the axes the object in the image plane.

I was wondering if it is possible to eliminate the projection, and get the world coordinates of those axes once they've been rotated and translated. To test, I've done the rotation and translation on the axis points manually, and then use projectPoints to project them onto the image plane (passing identity matrix and zero matrix for rotation, translation respectively), but the results seem way off. How can I eliminate the projection step to just get the world coordinates of the axes, once they've been rotation and translated? Thanks!

Were you using mm or chessboard units in your code? I've been using mm since I need the final results in mm, and my results are way off. I even applied all the fixes you mentioned, including changing the order of transforms, more pics for primary camera, and averaging the results produced by the rvec, tvec pairs for the secondary camera. Here's my code and new pics, and clue what's wrong?

Primary cam New pics only used for new cam matrix and distortion.

Secondary cam to chessboard in space

Secondary cam to chessboard on screen

Updated Python code

Thanks for the update! I'll go ahead and try this out and report back, but just a few quick q's about what you said.

1. If I move the chessboard around with the primary camera, will I have to take new pics with the secondary camera, as the chessboard is stationary in all the secondary camera pics.

2. Why would the y coordinate be negative in your answer, as the y axis is positive moving down from the camera.

3. The z coordinate is approximately 16. Would it be the same for all corners? Or should I just assume 0 for z?

Thanks so much! I've tried to comment my code as best as possible, let me know if there are any issues.

Pictures of Primary Camera to Chessboard in Space

Pictures of Secondary Camera to Chessboard in Space

Pictures of Secondary Camera to Chessboard on Screen

Python Code

Sample of chessboard points

Sample of axes

I fixed the axes on the first (mirrored) chessboard and my results are still far off.

I'm using multiple pics for each setup to get the camera and distoration matrix, I only included the first one for each in the links I showed you. But I'm confused, doesn't the calibrateCamera() function give you the camera matrix and distortion, along with the rvecs and tvecs, all in one go? Or is my logic there wrong?

Right now I'm generating objectPoints based on the mm of each square, then getting imagePoints with findChessBoardCorners(). Then I pass both into calibrateCamera() to get rvecs, tvecs. Then I use Rodgrigues on the rvec to get rot_mat, and then do rot_mat*[0,0,0]T + tvec. I take that result and pass it into the second (inverted) transform, and then the third after. Am I missing something

I believe my laptops webcam mirrored the chessboard on the chair.

And I have the camera matrix and distortion matrix of both cameras via the calibrateCamera() function I used to get the rvecs and tvecs.

I made sure all 3 chessboards had the same axes, but actually rotated the first and third to match the second, as the second had x and y axes that matched my world system x and y, and its origin was closest to the top left corner of the chessboard.

However, my results are still far off from the tape measured (-165,5,0), as I’m getting (294.49, -60.72, 2030.44) after passing (0,0,0) and translating to the top left corner.

Any ideas what might be off? Is it the axes? The z axis of my world system is the opposite of that on the pics, but I don’t think that should affect the results like this. I’ve included updated pics below.

Chessboard Points

Axes

I've kept the units consistent and done as you've said. The code finds the chessboard points, and draws the axes correct. Although I'm not sure about the blue one since I'm not familiar with the drawAxis function, maybe you can verify.

Original Pics

Pics of Axes

Pics of Chessboard Points

If you have any idea what might be wrong, or how to proceed, please let me know. Thank you!

I’ve performed the calculations, and even after verifying the math independently, my results are quite far off.

Using a tape measure, I obtained (-165,5,0) as my upper left screen corner (in mm with origin as camera).

If I stay consistent with chessboard units (i.e. passing (0,0,0), (1,0,0) etc)) into calibrateCamera() as objectPoints, feeding (0,0,0) into the transform yields (-17,18,100) as the top left corner.

If I use mm as objectPoints for calibrateCamera, noting that my real chessboard has a square length 50 mm and screen chessboard has length 20 mm, it yields (-32,846,2114) as the top left corner.

This is how I got rvecs, tvecs. Pics

Thank you for that detailed answer, I am close and just have two small questions before I can finish this calibration.

1. findChessBoardCorners() only gets the internal corners of the screen. If the corner of the chessboard and the screen should match, how can I get that top right corner?

2. Once I've rotated and translated to get the first corner of the screen, what's the most precise way of getting the other 2 corners? I know the length in mm of a side of one of the chessboard squares. Do I divide that by the length in pixels to get a ratio, and then multiply it by 2880? (my screen is 2880x1800)

I'm confused, are you saying the chessboard on screen and the chessboard in space are different? Otherwise if not, as you can see by the images: http://imgur.com/a/6oMEy it may be impossible to you use the secondary camera to take an image of both the chessboard on screen and the chessboard in space without it being angled.

But if I'm understanding you correctly, the first image I took was correct, but on the second the chessboard on screen should have been a chessboard image, not the actual chessboard in space as captured by the primary camera.