Stereo rectification: undistortPoints() different results from initUndistortRectifyMap() + remap()

edit

I have a calibrated stereo camera rig and want to rectify the coordinates of some tracked image points.

From the stereo calibration the camera matrices (ML, MR), the distortion coefficients (DL, DR), the rotation matrix (R) between those cameras and the translation vector (T) are obtained.

To get the parameters for the rectification the following function is called

RL, RR, PL, PR, Q, _, _ = cv2.stereoRectify(ML, DL, MR, DR, IMG_SIZE, R, T, alpha=1)

Rectify whole image

Now i want to rectify the images from the left and right camera. Before doing so, red circles are drawn around the tracked coordinates. To rectify the images I call:

map_l = cv2.initUndistortRectifyMap(ML, DL, RL, PL, IMG_SIZE, cv2.CV_32FC1)
map_r = cv2.initUndistortRectifyMap(MR, DR, RR, PR, IMG_SIZE, cv2.CV_32FC1)

img_l = draw_circles(img_l, coordinates_left, "red")  # white image with red circles
img_r = draw_circles(img_r, coordinates_right, "red") 

img_rect_l = cv2.remap(img_l, map_l[0], map_l[1], cv2.INTER_LINEAR)
img_rect_r = cv2.remap(img_r, map_r[0], map_r[1], cv2.INTER_LINEAR)

Rectify tracked points

Instead of rectifying the whole image I just want to rectify the tracked point coordinates, therefore I use undistortPoints() and draw those over the rectified images in green.

coordinates_left_rectified = cv2.undistortPoints(coordinates_left, ML, DL, R=RL, P=PL)
coordinates_right_rectified = cv2.undistortPoints(coordinates_right, MR, DR, R=RR, P=PR)

img_rect_l = draw_circles(img_rect_l, coordinates_left_rectified, "green")
img_rect_l = draw_circles(img_rect_l, coordinates_right_rectified, "green")

Unfortunately I don't get the same results as with remap (image below) - The green circles should align with the red ones. So what am I doing wrong? Any ideas?

Thank you!