I know three ways to reduce such problem.
First, create new thread to grab frame and save the frame into your queue from the thread to process your algorithm such as image enhancement, feature calculation or displaying. Moreover, you could adjust thread priorities if necessary. Then you can get much better frames.
Second, if there is network problem, you can rebuild opencv library after fix some codes in cap_ffmpeg_impl.hpp which is one of the sources for opencv_highgui module. There are two ways of RTSP protocol, over TCP and over UDP. Other people have said opencv adapted only RTSP over UDP, which doesn't keep reliability. The stream using UDP can be always lost.
In the function CvCapture_FFMPEG::open in cap_ffmpeg_impl.hpp, you can add several lines (see http://www.qapit.com/qap/opencv-rtsp-...). And rebuild opencv.
OR, if your situation doesn't require a scenario of getting full good stream , you can drop such screwed frames by comparing between frames. I use this way and it seems to work well. I attach some codes as follows,
qm->clear(); // qm is a queue storing frames as I said in first way.
while (1) {
frmPrv = qm->safe_pop(); // get one frame.
// Check whether grabbed frames are bad or not.
frmNxt = qm->safe_pop();
cv::absdiff(frmPrv, frmNxt, frmTmp);
double sum_m = cv::sum(frmTmp)[0];
double dm = sum_m / (frmTmp.cols * frmTmp.rows);
if (dm < 32 /* you can choose threshold */) break;
}
take a look at my answer