bad_keypoints's profile - activity

@esihaj have you open sourced your own version warpPerspective ? Perhaps, added it to OpenCV ? I'm facing the similar problem of warpPerspective taking horrible ~250ms times on 960x720 dest-size warps. If you can't release your code source, could you atleast hint at how you made it work under 5ms? Thanks.

@Possede are you saying that you specified the dest size in warpPerspective to be of the bounding rect of the corners of the object in your main image upon which you paste your warped image?

Hi, were you able to do anything about it? I too, am getting ~250ms on such resolutions, in warpPerspective alone. Were you able to optimized or find a better alternative? Please do tell, it's a really critical bottleneck in my application. Thank you.

Hi, I've already posted this question on StackOverflow, but didn't get any answers there. Hope someone here takes notice. In my application:

1. I track an object.

2. Get where its corners are coming in this frame.

3. I find the homography between its corners from last frame and current frame.

4. Use that homography to do a perspectiveTransform on the corners found in the current frame, to get transformed_corners.

5. Use the transformed_corners to find the homography between them and the overlay_image.

6. Apply above homography M to overlay_image, to get what would be called the warped_image using warpPerspective. This is the slow part.

7. And then using masking operations, I print the warped_image onto the current frame where the object was found.

Now I know after reading this blog article here why warpPerspective is slow.

And I'm getting ~300ms per frame in just the 6th step above, all because of warpPerspective. It's significantly affecting the FPS output of my application. Basically, it went down to 2FPS from 12 FPS without warping on every frame.

Is there any faster alternative to this? It's all done on Android, using NDK r9. What are some fast alternatives, and optimizations to reduce the warp time from 300ms to sub 50ms times?

My code is exactly what is being done here:

http://ramsrigoutham.com/2014/06/14/p...

Except that its executed on every frame in my tracking application, and the homography is always new because the detected corners of the tracked object in each new frame will be different.

Is camera calibration required everytime I run my app? What if the device changes? (obviously the focal length and some other things might change)

I'm building an app which would on detection of an object/image in the camera preview on mobile, will draw a 3D overlay over the object in the direction pose of the object. Just what augmented reality is.

On searching for topics on getting started with pose estimation of object on which to overlay, I came across that we have to have to do camera calibration to get camera intrinsic params.

After getting those, do we have to do calibration to get intrinsic and extrinsic params everytime when the app is run? Or can I calculate them once with the chessboard and store and use them henceforth?

Please forgive if this seems silly, but the idea of calibrating camera everytime is not something the enduser should be subjected to, and thus I don't want to put this kind of system on my app. But after reading up on camera intrinsic param, I gathered that this is something that we probably have to do on every run.

And from each image, if we save only 200 descriptors, then we will have 200 * 1000 * 10 = 2,000,000 descriptors clustered into 1 million clusters. Which is a lot of spread out area of clusters for an SVM to draw classification distinctiveness between.)

For your second concern: The goal here is to maximize the potential of a query image to be classified very well. Classification is done good when a sweet spot of number of clusters are formed which SVM classifier (or whatever else you're using) is able to use distinctly classify images belonging to different classes. Thus, its good to have a good number of training images, (I'd say go for 1000+ per class), and have clusters = no. of classes * 100.

This could be wrong, or work well. I've not done classification, so I really can' tell, but if you think logically, 1000 clusters having the common descriptors from images of 10 classes will be a really good train data compared to 100 clusters (too much overlapping) for the same, or 1,000,000 (1000 images for 10 classes, continued...)

Disclaimer: I have not implemented a very good working system of BoW as of yet, but I'm currently working on one.

So far, my findings are these:

1) Good keypoints (obviously). What this means is, keypoints invariant to affine transforms. SIFT/SURF/ORB are good. ORB might be the fastest, but it might not give you the best feature matching quality wise. (Take the hint)

2) Number of clusters. This is the most important. I myself haven't figured out yet what a good number is, given the number of training images. Given that I have N images, I've for now tested and found out that N*100 clusters works well for my purpose (which is not classification, mind you).

I'll update this answer in tomorrow with some links to talk more about the second point. Too many will give you a very hard time detecting/classifying anything. Too less will give you overlapping bad results.

See this, I found it very useful in understanding these problems.

I'll update the answer soon enough with more details. Right now I'm going somewhere.

Is there any way I can limit the no. of keypoints detected by a detector I created using the FeatureDetector interface in OpenCV's C++ SDK? I see the function, it has no other param besides the string that's to be passed. But I do need to limit the no. of keypoints the detector detects.

I'm using it for BRISK, and I need to limit the max. no. of keypoints the detector detects in an image, preferably detecting the N best keypoints.

I'm not using cv::BRISK because it's broken, (saw in an issue), and thus I've been using the one which can be instantiated from the FeatureDetector interface, which works great.

did you try googling "opencv save descriptors to file"?

I see this http://stackoverflow.com/questions/76... 4 upvotes to the answer says it was useful for them.

I'll get back to you once I check the results, of getting the nearest matches as expected of a unique image I would add to the database and without re clustering again, instead doing what you said. Thanks man.

@Guanta, and how will I know which cluster mean would be old_mean_i in new_mean_i = ((old_mean_i*N_i) + sum_new_descriptors_assigned_to_cluster_i ) / new_N_i ??

How would I know which clusters the descriptors of the new image should fall in?

Maybe you didn't search well enough.

Here's something to get you started (and I started off with that too, except that I don't need classification).

I still don't understand the "recompute the means from each descriptor assigned to each cluster individually (you even can compute the means incrementally if you save the number of descriptors assigned to each cluster <- which you know from the inverted index)." part.

I've gone over it more than 10 times, I'm not getting it through my thick skull. Can you elaborate that in layman-terms/ELI5 type? As far as I understand it, computing the means from each descriptor assigned to each cluster is what the BOWKMeansTrainer does, right?

@Guanta, first of all, sorry for replying so late. I was on another project. Got back to this today.

Secondly, I just want to make it clear one more time - images will be added to my database everyday. I will need to make sure that query images are matched with their corressponding target images. Say, I have an image of Tom Cruise advertising some brand X's shirt. So, that particular photo with Tom Cruise in it and brand X's shirt on him, should come up as the top match in my results, if I send that exact photo, but as captured from a mobile camera, to my server. This will be the scenario for all the other thousands of images I will have on my server. Each totally different and not related at all.

Now, assuming what you said was keeping in mind what I just wrote above, (cont..)

@Guanta, also, with regards to this:

for each cluster where you raise the bin count you also save the image index (this is how you create the inverted index file), so you just need to update it with the new infos. This is possible, e.g. in flann w. addPoints()-method

Where would I do that, even if I do have the addPoints() method available? Because I set the FlannBasedMatcher in the instance of BOWImgDescriptorExtractor as the matcher. And the BOWImgDescriptorExtractor instance has only compute, descriptorSize, descriptorType, getVocabulary and setVocabulary methods.

For flann, i use an instance of cv2.FlannBasedMatcher and then add descriptor arrays to it, then train it. I have never seen a addPoints() function. Not even in pyflann (atleast not from an object created by pyflann.FLANN() Maybe it's in the C++ api.

Now, without 1, 2 is not possible, or, say, how does it make sense to match a query image with a matcher that isn't even trained with the target image's descriptors? (because we didn't add descriptors of the new image to bowtrainer and then call cluster on it)

I think you still mix some parts up: you don't need to retrain the BoWKMeansTrainer() (=vocabulary), this is needed only once (if your new images are not too much different from the images used when training it)!

That's the thing dude! All my images will always be different so much so that atleast I as a person even wouldn't saw that any two of them belong in a same class or can be classified as "similar".

So, for every image added to my server, I will need to: 1) Update the vocab with the descriptors of those new images. 2) Get the nearest clusters for those new images and then add them to inverted index.

Continuing..

@lilouch, discovered any other way to save the flann index scalable, savable and editable to disk?

For the last part: How would I even update the vocabulary to include the new set of descriptors from the newly added image without retraining the BOWKMeansTrainer object to get a new vocab? And then only can I get BOW descriptor of the new image with BOWImgDescriptorExtractor and use the result to update the inverted index. Please correct me if I'm missing something from the flow.

I don't understand this part: Of course you could also cluster the local descriptors differently and use thus a different number of clusters, but why? What local descriptors? The one I take from the image by SURF/ORB ? And differently how? Actually, I just don't understand any of it. Sorry.

And this: I would go with the simple way. Are you referring to the FLANN based image index which I mentioned above or else what?

And then: For updating: I wouldn't retrain the image database and just update the inverted index file You mean I just add the SURF descriptors of a new image added to my server to my BoW trainer object and then somehow without retraining it to cluster the new set of descriptors into the existing clusters, I should update my inverted index? Seems crazy.

@Christopher, Ah shit yes!! I guess I missed that too. I have working code for my app and I just couldn't see what I was missing. Indeed, the descriptors need to be passed as list

Also, maybe I missed something: my main motive is to have BoW inverted index replace my current FLANN based image index. Because the flann matcher I use to match query image with many images currently is not savable or updatable (one cannot simply add another image's descriptors and use it without retraining to use it for matching, and deleting a particular set of descriptors is not possible). And I want to save it to disk, not generate it from saved descriptor data everytime my application starts.

@Guanta, I don't think I misunderstood BOWKMeansTrainer. It clusters train descriptors into words for the vocabulary, given how many clusters you want. Also, repeating again, I do not want to classify images. I want to leverage the ease of use and fast indexing from the inverted index approach in BoW to find out which clusters the query images falls most into, then find those clusters' associated database images, and then find out which image has the highest frequency of occurrence from the lookup. Maybe this would make it more clear to you: http://www.cs.utexas.edu/~grauman/courses/fall2009/slides/lecture16_bow.pdf goto slide 25 in it, labelled inverted index. Edit: Thanks for answering though, @Guanta.

@berak, I don't want classification. I need to find a match of an incoming query image from a database of unique images. Edit: Also, thanks for the size explanation @berak. That was very clear. Edit 2: To answer your second question, I'd try to make an inverted index table from the bow vocabulary.

@Guanta, I'd search a query image in a database of unique images. For those unique images, I'd construct a BoW vocabulary with N number of clusters where N 1)either would be as large as the final number of images I will have in database, 2) or be change dynamically and correspond to the number of images in my database. Does it make sense to create a codebook from BoW visual words and use that to match query images with only one unique image in database? Also, if you can, please answer my question here: http://answers.opencv.org/question/44670/bow-number-of-words-in-vocabulary-for-unique/

Also, I think you should try out BRISK from here, see if that works, with your 3.0 build: https://github.com/Itseez/opencv/blob/master/samples/python2/find_obj.py

It could be a problem with LSH. Try with KDTREE algo param in your FLANN params, and you'll go farther. Also, you should see this: http://stackoverflow.com/a/13654489

EDIT: I think it's a bug in opencv. Training a flann based matcher results in segmentation fault (some pointer trying to access a memory location something that doesn't exist, probably). Try with simply matching two images one on one, and it works.

So, I guess the fault is somewhere in OpenCV. I used the features2d interfaces for creating detectors and descriptors extractors.

@lilouch, so what did you do for your app? Did you use BoW or did you use FLANN index?

I'm using BoW for an image matching application, in which basically a query image will be matched against thousands of images, (could be millions too). I'll not implement the SVM part, because I don't need classfication. I only need to make a codebook for a fast inverted index implementation to search my images by BOW image descriptors.

First question

BoW vocabulary will consist of N visual words (clusters) as specified in clusterCount param in BOWKMeansTrainer instance creation.

For an image matching application, where there will be only thousands of unique images, and will forever be increased (or mass decreased sometimes), what should the clusterCount be? If for 100k images I specify it to be a mere 1000 clusters, I'm sure that I'll get a lot of false matches in the ranks of my result.

Second question: Also, what would you think the size of the vocabulary will be (in bytes), for a 100k images? I figured if one cluster takes about 0.5 kB, then if I create a vocabulary of 1 million images, giving clusterCount = 1 million, then the vocabulary size should not be more than ~490MB, which is reasonable enough to be used in processing query images on a simple machine with 8GB RAM.

How I calculated cluster size: I made a vocabulary of 8 words from 16 images, and then dumped the vocabulary to disk (numpy.save). The size on disk was S. Assuming each cluster was of same size, dividing S by clusterCount always gave me the value of 512 bytes, for different sets of images and clusterCount and different sizes of images. I used simple toy implementation of @berak's, which can be found here

Thus, for 1 million clusters from 1 million images, my calculations above should hold. Actually, for 1 million images, I'm very sure that most of the descriptors from many images would fall together, so the size could be smaller.

@berak help on bow_train and bow_extract objects.

@berak help on those objects shows nothing.

@berak thanks a tonne man. I checkout the 2.4 branch, everything builds out. SVM, SIFT/SURF, BOW modules. All in one place, finally.

But still though, in the build from the 3.0 alpha release (or the current working master branch on git), where would I find the ML python methods (like SVM()) ? I didn't find anything in cv2.ml or in opencv_contrib modules. And cv2.SVM definitely didn't exist. Just saying, because I'll like to upgrade to 3.0 for all the improvements.