Opencl computation time increases after few iterations

edit

I am using opencl module provided by opencv on Qualcomm Snapdragon 805 Inforce 6501.The problem is that the time taken for the execution of opencv's remap function with oclMat after about 20 iterations is around 700 microseconds but if I call it for 50 iteration or more execution time rises to 50ms after about 30 iterations. The code i am using is as follows

Mat xMap1,yMap1,mDst;
Mat xMap2,yMap2;
Mat xMap3,yMap3;
bool first = true;
oclMat osXMap,osYMap;
oclMat conSrc;
oclMat conDst;
oclMat oclXMap,oclYMap;
oclMat oclSrc;
oclMat oclDst1;
oclMat oclDst2;

Mat imageSrc = cv::imread("/storage/sdcard0/DCIM/Camera/2048.jpg");
FileStorage fsx("/storage/sdcard0/DCIM/Camera/FishEyeConversionXmap4.yml",FileStorage::READ);
FileStorage fsy("/storage/sdcard0/DCIM/Camera/FishEyeConversionYmap4.yml",FileStorage::READ);
fsx["xMap"] >> xMap1;
fsy["yMap"] >> yMap1;
fsx.release();
fsy.release();
oclSrc = imageSrc;
oclDst1 = oclSrc.clone();
oclDst2 = oclSrc.clone();
oclXMap= xMap1;
oclYMap= yMap1;

cout<<"Remaping\n";
for(int i=0; i<50; i++)
//while(1)
{
    unsigned long long t1 = GetTime();
    remap(oclSrc,oclDst1,oclXMap,oclYMap,CV_INTER_LINEAR,BORDER_CONSTANT);
    remap(oclSrc,oclDst2,oclXMap,oclYMap,CV_INTER_LINEAR,BORDER_CONSTANT);
    unsigned long long t2 = GetTime();
    printf("%d Time taken remap : %llu microseconds for %f ms\n",i, t2-t1,(float)((t2-t1)/(1000))); 
}

And this is the output I get.

0 Time taken remap : 11946 microseconds for 11.000000 ms
1 Time taken remap : 817 microseconds for 0.000000 ms
2 Time taken remap : 800 microseconds for 0.000000 ms
3 Time taken remap : 258 microseconds for 0.000000 ms
4 Time taken remap : 754 microseconds for 0.000000 ms
5 Time taken remap : 885 microseconds for 0.000000 ms
6 Time taken remap : 764 microseconds for 0.000000 ms
7 Time taken remap : 966 microseconds for 0.000000 ms
8 Time taken remap : 783 microseconds for 0.000000 ms
9 Time taken remap : 757 microseconds for 0.000000 ms
10 Time taken remap : 1050 microseconds for 1.000000 ms
11 Time taken remap : 445 microseconds for 0.000000 ms
12 Time taken remap : 798 microseconds for 0.000000 ms
13 Time taken remap : 758 microseconds for 0.000000 ms
14 Time taken remap : 765 microseconds for 0.000000 ms
15 Time taken remap : 978 microseconds for 0.000000 ms
16 Time taken remap : 1120 microseconds for 1.000000 ms
17 Time taken remap : 763 microseconds for 0.000000 ms
18 Time taken remap : 766 microseconds for 0.000000 ms
19 Time taken remap : 607 microseconds for 0.000000 ms
20 Time taken remap : 258 microseconds for 0.000000 ms
21 Time taken remap : 305 microseconds for 0.000000 ms
22 Time taken remap : 767 microseconds for 0.000000 ms
23 Time taken remap : 750 microseconds for 0.000000 ms
24 Time taken remap : 783 microseconds for 0.000000 ms
25 Time taken remap : 769 microseconds for 0.000000 ms
26 Time taken remap : 782 microseconds for 0.000000 ms
27 Time taken remap : 4588 microseconds for 4.000000 ms
28 Time taken remap : 301 microseconds for 0.000000 ms
29 Time taken remap ...