R.Saracchini's profile - activity

Hi. I'm sorry for the late answer ! I'm quite overwhelmed these months. For sure I would like to share it, I would need to strip out some libraries which I attached, that we use in my research group, and replace with a simple version of it. Still, I'm not sure if you would be able to compile it in a non-Linux platform, and surely making it compile in Visual Studio would be a feat. The thing is tangled with a bunch of boost mutexes and shared pointers...

Sadly a lot of good algorithms have a ROS dependency tacked on. ..It is a pain to port such algorithms (I did remove recently the ROS dependency of LSD-SLAM, took one week). Anyway, several of such algorithms have dependencies here and there that makes it not worthy to you just add as library into your code as it will be a huge "Frankenstein" code... for example, the PTAMM has a lot of functionality that is present in OpenCV implemented by a bunch of uncommon libs. It takes a long time, but pays off read the papers and if possible the code, and write the thing yourself.

I really recommend read the thesis above if you have time. It have a very good explanation how to compute the Motion Estimator. I hope that this helps you in your project

I believe, from your posts that you are trying to implement a sort of SLAM solution. This is not an easy task, and I recommend you to read the works of LSD-SLAM (semi-dense approach), and PTAMM (feature-based approach) to have an idea of two very robust (albeit distinct) solutions.

Regarding this particular issue:

Determining the camera pose by feature-matching is extremely jittery. A small amount of camera noise can make a calibration feature "disappear" and affect the solvePNP. This is even aggravated if you are using RANSAC. Keep in mind that RANSAC is non-deterministic algorithm. It will find the solution with least error from a a subset of your input. This means that if a small group of features with a certain error provides a solution with small error (but wrong because their positions are contaminated with noise), your camera pose will "jump".

I fast solution that come to my mind is perform some feature matching and then pass the estimated position of the features as initial guess of an optical flow algorithm. Note that optical flow algorithms are not robust to occlusion, so you will need to make a further check if the tracker feature matches with the original one. Then you can use the input to solvePNP. A better yet solution would use a weighted motion estimator instead solvePNP, so features with low confidence would affect the camera calibration less than properly matched points, but keeping all the information into account. This isn`t implemented in OpenCV, but there is a complete description in a the thesis "Visual Tracking for Augmented Reality" from Georg Klein (Univ. of Cambridge), about how to implement one.

I hope that this helps.

If you are using the camera centre as origin, it means that you are using the coordinates in camera space instead world space. You wont be able to compute the motion of the camera in this way. Since you NEED a reference, set the world coordinate space as the camera pose of the first triangulation. Now you should compute the camera pose and rotation relative to the camera pose of this first triangulation. It means that after every triangulation, you must convert such triangulated points to world space coordinates. For this, just use the matrix T = E^(-1) (that is the inverse of the camera extrinsic matrix given my solvePnp) to transform every triangulated point. Note that transforming the point (0,0,0) by T will give you the camera centre in world coordinates.

Well, the feature will match with such background and the fork, so if you change the background that feature wont be present at all. Of course, there is the possibility of false matching, but this happens even with features found within the fork. At first, you should not match directly a feature with the closest feature found in the image, rather do a more robust comparison do eliminate ambiguous matches (check this book here : http://szeliski.org/Book/ , it is very helpfull)

FAST is a corner detector for tracking, so it is natural that it will find only corners. SURF, for another side detects regions that are identifiable in several scales, such that it can be relatively scale independent for feature matching. You will be able to see that if you draw the circles according the size attribute of cv::Keypoint.

Well, solvePNPRansac is quite robust, if you give enough iterations. So your problem is somehow in your input data or how you are processing it: 1) You must be aware that the input and output matrices (camera matrix,, rvec,tvec) are type CV_64F (64 bit float), so not try to read their values using float in the typecasting. 2) You triangulation method is not giving you good results. If your 3d points aren't good, solvePNP wont give you a good movement. Additionally, 3d points should be in metres and in world coordinates. 3)The tracking method is giving you some improper results, so that the 2d input points are wrong. 4) The matching between 2d and 3d points passed as input is somehow wrong. Each i-th element on the 2d vector corresponds to i-th element in the 3d vector. They MUST match.

The function solvePnp (or solvePnPRansac) consider that the 3D points given are in absolute world coordinates, thus it will return a rotation and translation matrix of the extrinsic matrix of the camera. That is, a matrix that will convert 3D world coordinates to 3D coordinates relative to the camera centre. If you compute the inverse of said matrix, you will have the camera transform matrix, which will state the camera rotation and translation in relation to the "world".

Note that the rotation is given in Euler angles, so you will need use cv::Rodrigues to convert it to a 3x3 rotation matrix. The extrinsic matrix is then a 4x4 matrix in the form

R00 R01 R02 T0 
R10 R11 R12 T1
R20 R21 R22 T2
0   0   0   1

you can just use cv::Mat::inv() to compute the inverse.

The match procedure of the BFMatcher class will take two inputs, queryDescriptors and trainDescriptors, which are arrays of descriptors (normally each line of a cv::Mat object corresponds to a individual descriptor), and return a std::vector object of type cv::DMatch.

Each element of this output array corresponds to the correlation of a matched query descriptor to a train descriptor. So matches[i] has tree important attributes: trainIdx, queryIdx and distance. This element states that the line queryIdx of queryDescriptors matches with the line trainIdx of trainDescriptors with distance distance.

Well, after matching you can assemble the inputs of solvePnp (which are : array of 2d positions, an array with the corresponding 3d positions) using those indexes. This depends if the 3D positions are from the query or train descriptors. If you know the 3d positions of the train descriptors, you will use the 2d positions of the matched query descriptors and then compute the camera pose of the query image with this matching data.

Normally you use the distance attribute to filter bad matches.

I hope that this helps.

Truncating will occur if you use a cv::Mat object of type CV_8U (or its variants CV_8UC1, etc...) and try to write an value larger than 255. Create a cv::Mat object of a larger signed type, for example CV_32S or CV_32U instead CV_8U. CV_32S will use 32bit integer type. The assignment of values can be done in the following way:

cv::Mat my_matrix_of_integers(200,200,CV_32S); //creating a 200x200 matrix of 32 bit signed values
my_matrix_of_integers.at<int32_t>(30,40) = 1034; //assignent of a signed 32 bit value

It is VERY important to you know the type of your matrix before assigning a value. If you try to assign a large integer value using "at.< int32_t >" in a matrix of type CV_8U, your program is very likely to crash at certain point, since OpenCV don't verify this and will write outside the allocated memory boundaries...or worse, invading a memory block of another allocated data structure of your software. ALWAYS check the type of your matrix with the cv::Mat::type() function.

Another thing: cv::imshow seems to not like any matrices that arent of type CV_8U and CV_32F. Use cv::Mat::convertTo to convert the matrix to a compatible type for displaying and scale its values.

Well, I'm never worked with Kinect 2, but I suppose that it is very similar to the original Kinect. The fact is that the Kinect has 2 cameras (one IR and another RGB), and it generates it depth map by projecting a pattern over the scene using a IR projector and computing the depths by using the input of the IR camera.

That said, the raw data from the RGB camera wont be aligned with the depth data because the cameras are distinct. It is not a mere matter of just resizing the images. The depth map has to be properly mapped , or better yet, registrated. This means that you have to re-project the depth points into the RGB camera point of view and assign the color

As I said, I'm not sure how you capture the data from the Kinect 2. If you are using the OpenNI backend of OpenCV, you can do just as I explained in this another question HERE, and the depth map will be properly registered with the RGB image.

If the way that you are extracting this data from the Kinect is not this, you must determine the camera pose of the RGB camera relative to the IR camera pose to do the reprojection.

Calibrating the cameras (only once !):

Calibrate both cameras as a stereo pair, using cv::stereoCalibrate with the camera matrices that you computed in the previous step and a calibration pattern viewed by the same cameras. You will obtain the rotation, translation needed to re-project the points. If you have the camera matrices of the RGB and IR camera a priori, it will be much better. The ones from Kinetic 1 are quite well known and can be found in internet. I believe that it is simple to find the ones from Kinetic 2 at this point...

Colouring the depth map (for every frame):

With the procedure above, you have the rotation and translation matrices and the camera matrix of the RGB camera. All that you have to do is project the 3D depth point into the RGB camera space using cv::projectPoints, and assign the colour of the pixel (or interpolation of the nearby pixels). Note that some points will be colourless since they aren't visible by the RGB camera.

OpenCV works very well in parallel processing, except with anything related to GUI management ... are you calling "cv::imshow" and "cv::waitKey" in multiple threads ? If so, you should refrain to use such functions in only one thread.

Depends on how much the markers will rotate or change the appearance, but multi-scale template matching or feature matching (finely tuned) should work much better than classifiers.

I noticed that in Windows my output videos would be of size 0 when I didn't had the codec or even using some options from the codec settings. There is not such problem when a codec pack is installed, thought.

I have edited the answer in order to clarify better your doubts.

The registered depth data is given in a cv::Mat structure with data in several types. If you retrieve it by using CV_CAP_OPENNI_DEPTH_MAP or CV_CAP_OPENNI_POINT_CLOUD_MAP, this matrix will be respectively of CV_16UC1 or CV_32FC3. The first is a 16-bit integer with the depth in millimetres, and the second each element will be an X,Y,Z vector of the relative 3D position of the point to the camera in metres . You can easily access the data by accessing the individual elements of the Mat object.

see the example:

cv:.Mat image3D, imageBGR;
cv::VideoCapture videoReader;
//Init camera
videoReader.open( CV_CAP_OPENNI );
videoReader.set( CV_CAP_OPENNI_IMAGE_GENERATOR_OUTPUT_MODE, CV_CAP_OPENNI_VGA_30HZ );
videoReader.set(CV_CAP_OPENNI_DEPTH_GENERATOR_REGISTRATION,1); 
//grab a frame
if (videoReader.grab()){
    videoReader.retrieve(imageBGR,CV_CAP_OPENNI_BGR_IMAGE);
    videoReader.retrieve(image3D,CV_CAP_OPENNI_POINT_CLOUD_MAP);
   //Here you get the depth
   int y = 200;
   int x = 100;
  cv::Vec3f pt_3d = image3D.at<cv::Vec3f>(y,x); //0 - X, 1 - Y, 2 - Z
else{
  std::cerr << "Error capturing frame !" << std::endl;
}

I hope that it helps.

About the alignment of the images:

The flags sent in the "set" command to the "VideoCapture " structure in the example above (more specifically "CV_CAP_OPENNI_DEPTH_GENERATOR_REGISTRATION") ensure that the RGB and the depth frame will be properly registered/aligned by the OpenNI backend. As side effect of the registration, there will be a black border around the scene which corresponds to regions where the field of view of the IR sensor does not overlap with the RGB sensor.

See the example bellow. As you can see, there is a surrounding area where there isn't any depth information (the black regions):

As far I cannot disclose the code directly (there is some IP over its library), I would be more than happy to aid in write a OpenCV module which implements a Vocabulary Tree training, and search with distinct type of descriptors (floating point,binary) and distance functions(l2,l1, hamming). It would be nice to add to at least a basic Inverted File implementation which is used by the Nister´s algorithm and compatible with a BoW approach.

I wish that I had more time to write a proper Photometric Stereo module...

It is not implemented in OpenCV. I wrote this VT algorithm some years ago using OpenCV structures based both in a code that I found long time ago (C++) and in DBOW. Keep in mind that you will need to compute a Vocabulary Tree using a large amount of descriptors and with sufficient variation if you want reasonable results, as tree should be well populated (most of leaves with an assigned descriptor value). It works very well for tracking loss recovery for SLAM algorithms :)

Interesting ! How much delay you achieve with this set-up ?

I dont known much aboout the Python implementation of OpenCV, as I use only C++. In my experience, in most cases where VideoWriter fails to write any frame silently and generates just the file header is when the width and height of the written frame is not the same as the specified in the VideoWriter constructor. Also, you should verify if Mat type of "edges" is CV_8UC3 (8bit, 3 channel). Sending a CV_32F(float) or a CV_64F(double) frame may cause problems.

This is very weird. I use extensively the C920 and C930 with OpenCV in Debian and Windows 7 for Augmented Reality applications using either 848x480 or higher resolutions, without installing anything exceptional (default drivers or Logitech ones). Note that OpenCV 2.4.3 is VERY outdated. You should be using 2.4.9 or 2.4.10.

Yes you can perfectly use such algorithms with still images. In fact all that they expect in OpenCV is the previous and the next frame images. So it does not make any differentiation. Of course, you have to take into account that Optical Flow works better when the images differs not much from the previous one.

If your image is of type 8UC1 (thats why I suppose that you are using uchar), its values are integers, not floating point, and is very likely that they are between 0(black) and 255(white). The least value that you can add is 1. Attributing 0.01 to a uchar variable will round it to 0.

I do not think this approach wrong. It is possible to even measure (very roughly) its distance, since it is trivial match the characters with real world coordinates. Like the others it has its own drawbacks. One that bothers me, is that the license plate can be easily occluded by a car/bike/pedestrian ... at least in my city, they are placed quite low. Without know more details, it is hard to tell a good solution.

The "real" OpenCV function is the cv::remap function. All that you have to compute the mapping matrices only once, as initialization, and give it as parameter to the remap function. It will un-distort the frame using any optimization options available in OpenCV. If you read the documentation, you will see that "cv::undistort", for lens distortion removal, uses "remap" internally.

Maybe the attribution that you doing within the "ifs" is not really copying the data from the temporary matrix which you created, just giving the reference, and the object points to unallocated memory after the "temp" variable is released. Note that type = 0 corresponds to unsigned integer , which is the constant equivalent to CV_8U (at least, in my platform) .

And yes, Converting binary descriptors to CV_32F is kind of wasteful, since it discards the potential advantages of the binary representation.

I´m not exactly sure about this, but I believe that the descriptors from BRISK are integer which should be compared by Hamming Distance, when FlannMatcher expects floating point descriptors which are compared with L1/L2 distance.

Barrel distortion removal can be considered as remapping a given pixel (x,y) from the original image f into a image g, by a function h(x,y), where the mapping function is the one used by ImageMagick (which you can see here ).

In OpenCV, there is the function cv::remap where you give two mapping images for the x and y coordinates, with the desired interpolating function (nearest neighbour, bilinear, etc.). There is a pretty good tutorial in the OpenCV documentation here showing how you can generate custom maps for remapping of images. Instead translation/rotation in the example you should use the ImageMagick equations, with the parameters that you use.

I hope that this helps.

In order to compute a transform you have to know the correspondence of points in each image. In fact, you should detect the points in a reference frame, compute the correspondence in the following frame by an approach such as optical-flow or feature matching. When you have a set of points that were successfully matched, you compute the desired transform. The algorithm of choice to you compute this correspondences depends a lot of the nature of the application. Optical-flow works well when you have frames which do not change very much and you need fast computation. Feature-matching is more suited to stitching or other tasks where the frame varies a lot and you need invariance regarding rotation, translation or scale. This function wont help you with tracking directly.

I used the C920 and C930 in commercial applications in Debian 64 bit (wheezy and squeeze). This is very unusual, since I can leave the camera on in large amount of time. Seems me more a hardware issue.."VIDIOC_DQBUF: no such device" means that the camera disconnected itself (maybe overheat). Did you tested this camera in applications such as guvcview ?

It is a bit confusing at first, but this kind of input gives you some freedom about the input type. You do not pass a image as input, rather a array of elements which represents your 2D point set.

It can be:

1. std::vector<cv::Point2f>
2. std::vector<cv::Vec2f>
3. cv::Mat with N lines and 1 column, with type CV_32FC2, each element consists in 2 floating point members) , such that can be represented by cv::Vec2f

Particularly, I use mostly the first option to represent the point set. If you don't know what is std::vector container, I suggest you give a look in some tutorials of STL containers here. Understand those containers is helpful to you understand certain OpenCV functions and structures.

Edit: Sorry, now I noticed that you are using C interface. I don't recommend it very much, however, if you do not have choice about this, use the 3rd option.

I had the same issues. Plus the fact that I rely a lot exactly in the re-factored algorithms and the ones that was moved away from the main code. This broke compatibility of a quite complex project of mine. A pity, considering that the transition from 2.4.2 up to 2.4.9 was for me mostly seamless.

I do agree. I have software already in production that will require significant rewriting because the changes in the API, and much needed functionalities from the original libraries where moved to contrib. I do not understand why original functionality is moved away from the code to be put in a separated repository and separated namespace (which often do not even compile because seems to be developed separately)."xfeatures2d" ? Really ??? At moment I´m keeping my usage restricted to 2.4.9 and 2.4.10.