Create column pointers just like TheMatrix.ptr<int>(vi);

edit

In image processing generally you cannot directly access columns as continuous vectors. However calculating the transpose of an image should be really fast, and I recommend it if you need to access many elements.

Otherwise you have to check first if the image is continuous in the memory (TheMatrix.isContinuous()). Sometimes OpenCV doesn't keep all the image data in the same place.

Now, if the image is continuous, you can calculate directly the pixel position in the memory:

p=TheMatrix.ptr<uchar>();
for(y=0;y<TheMatrix.rows();y++)
     uchar a=*(p+y*TheMatrix.step[1]*y+col);  //get quickly the column number col

Otherwise you need to use the TheMatrix.at<uchar>(y,x) function to get an element.

Anyway, as a general rule I recommend to write your algorithm, and optimize it at the end, if it's really needed. Today's computers are quite fast for image processing. Sometimes parallelization (or using the GPU) gives you much better performance than a heavy optimization (e.g. in this case, you can transpose one image while processing another one).