Perspective Transform (Homography) on rgb-d images

edit

I'm using a RGB-D camera (Intel Realsense D345) to implement a table top projected augmented reality system. Using chessboard calibration I obtain a transformation matrix which I use to transform each incoming frame using warpPerspective from openCV. It works really well for the color frames. The problem is, am I allowed to do this for depth images as well ? Considering depth images are 3D geometrical data. What's the right way to apply a transformation matrix to depth images?