1 | initial version |
The simple answer is Optical Flow + Depth gives you 3d displacement vectors. Cluster those, and you have your background and your objects. Of course, doing that well is difficult.
Did you perchance try Google the question to see what has already been done? Here are results 2, 4, and 5 from the search for moving objects from RGBD data.