We are effectively trying to do the same thing on an Android handset as this OpenCV tutorial (http://docs.opencv.org/3.1.0/dc/d16/tutorial_akaze_tracking.html) , using the camera of the phone and ORB features.
We have moved onto using the Android native sdk elements , but even taking this approach and using a 640x480 grayscale image from the camera we get a low FPS. We have tried different feature extractors which are quicker e.g. FAST but this then incurs more work during the 'match' phase and also isn't very rotation tolerant.
We have also looked to adjust the number of features extracted.
Does anyone have any suggestions on how we could speed this up? I have seen this demo video using vuforia https://www.youtube.com/watch?v=q3vcDGm-EAg which appears to have a very high framerate. Special affects aside this is the level of tracking we would ideally like to see.