World coordinate space from camera coordinate space?

edit

I am doing structure from motion from multiple images using OpenCV. I have 8 images, and I have generated point clouds for each pair of images (img1&2, img2&3, img3&4 etc.). I know that each individual point cloud is correct because they look good when displayed in VTK / OpenGL.

I want to combine all of the point clouds to get a denser reconstruction. From what I understand, all my point clouds right now are on a different camera coordinate system. I am trying to use cv2.solvePnP to compute my camera poses and transform my 3D points with the inverse of the camera pose so that my point clouds are all on the same coordinate system, but the points from each successive 3D point set still have coordinates that are orders of magnitude away from those of the previous set.

Here is my code:

images = ['00.jpg', '01.jpg', '02.jpg', '03.jpg']
cam_poses = []

for i in range(len(images)-1):
    K, pts_3D, img_pts = gen_pt_cloud(i, images[i], images[i+1])

    if i == 0:
        # first 2 images
        pt_cloud = np.array(pts_3D)
        colours = np.array(img_colours)

    elif i >= 1:
        # get the pose of the camera
        rvec, tvec = cv2.solvePnP(pts_3D, img_pts, K, None)[1:3]
        pose = np.vstack((np.hstack((cv2.Rodrigues(rvec)[0], tvec)), np.array([0,0,0,1])))
        cam_poses.append(pose)

        # transform the 3D points using the camera pose
        for j in reversed(range(i)):
            pts_3D = np.array([ np.dot(np.linalg.inv(cam_poses[j]), pts_3D) ])[0]
            pt_cloud = np.vstack((pt_cloud, pts_3D))

vtk_cloud.vtk_show_points(pt_cloud)

Is there something I'm doing wrong?

Also, I checked the OpenCV documentation and saw that objectPoints is an array of object points in the object coordinate space. What does that mean? My objectPoints (pts_3D) are on their specific camera coordinate system. Is that why my points are off? If it is, what should I do to transform the objectPoints so that they are in the object coordinate space?