phirestalker's profile - activity

If I compiled the opencv3.1 library without opengl support, is opencl through the TAPI still able to use the GPU accelerated code? If I have an NVIDIA based card do I need cuda compiled in to see the benefits?

thanks, edited

here is some code that will solve the question

bool goodMatch(int distance, float percent, int descriptornum)
{
    int descriptorbits = 256 * descriptornum;
    int result = descriptorbits - (descriptorbits * percent);
    return (distance <= result);
}

distance - the value returned in dists percent - float value of percent (i.e. 90% = 0.9) descriptornum - the number of descriptors

ok I figured it out this code should work when plugged with a qstringlist of files

QStringList videos;
Mat indexTemp;
struct frm
{
    QString file;
    int position;
    cv::Mat descriptors;
};
QList<frm> frames;

foreach (video, videos)
{
    process(video);
}

for (i = 0; i < indexTemp.rows; i++)
{
    Mat mat = indexTemp.row(i);
    Mat indices;
    Mat dists;
    int k = 3;
    if (!mat.isContinuous() || mat.empty())
        return;
    try
    {
        mat = mat.reshape(1,1);
        index.knnSearch(mat,indices,dists,k);
    }
    catch (Exception e)
    {
        qWarning() << "index search failure" << e.err.c_str() << e.msg.c_str();
    }
    catch (exception& e)
    {
        qWarning() << "index search failure" << e.what();
    }
    for (int j = 0; j < indices.cols; j++)
    {
        if (indices.at<int>(0,i) == -1 || dists.at<int>(0,j) < 12800) //knnsearch returns -1 for all cols after the matches it was able to find. 12800 is about 90% match with 500 ORB descriptors
            continue;
        try
        {
            QTime matchtime(0,0,0,0);
            int matchms = frames.at(i).position;
            matchtime = matchtime.addMSecs(matchms);
            int temp = dists.at<int>(0,j);
            qDebug() << frames.at(i).file << "frame"<<  timestamp.toString("hh:mm:ss") << "match at"<< frames.at(indices.at<int>(0,j)).file << matchtime.toString("hh:mm:ss") << "distance" << temp;
        }
        catch (Exception e)
        {
            qWarning() << "failure in indices" << e.err.c_str() << e.msg.c_str() << e.func.c_str();
        }
        catch (exception &e)
        {
            qWarning() << "failure in indices" << e.what();
        }
    }
}
index.build(indexTemp,cv::flann::LshIndexParams(12,20,3), cvflann::FLANN_DIST_HAMMING);


void process(Qstring video)
{
    if (video.isEmpty())
        return;
    int interval = 500;
    int current_interval = 0;
    UMat frame;
    vector< cv::KeyPoint > keypoints;
    Ptr<DescriptorExtractor> odetector = cv::ORB::create(500);
    UMat imgDescriptor;
    VideoCapture videocap(video.toStdString());
    if (!videocap.isOpened())
    {
        failedstream << video << " could not be opened. Check permissions" << endl;
        return;
    }
    for(; ; )
    {
        if (current_interval != 0)
            videocap.set(CV_CAP_PROP_POS_MSEC,current_interval);
        if (current_interval > videocap.get(CV_CAP_PROP_POS_MSEC) + 400) // I found some videos needed this in case the last from is not quite 500 ms away from the previous one
            break;
        if (!videocap.read(frame))
            break;
        current_interval += interval;
        if (frame.empty() || !frame.isContinuous()) //skipping bad frames
            continue;
        odetector->detect(frame,keypoints);
        odetector->compute(frame,keypoints,imgDescriptor);
        if (imgDescriptor.empty() || !imgDescriptor.isContinuous()) // don't ask me how but even with all the frame checking this is still needed
        {
            qDebug() << "descriptors empty or corrupted" << imgDescriptor.empty() << imgDescriptor.isContinuous();
            continue;
        }
        {
        Mat imgDescriptorMat = imgDescriptor.getMat(cv::ACCESS_RW);
        Mat matonerow;
        frm tmp;
        tmp.file = video;
        tmp.position = current_interval;
        if (imgDescriptorMat.rows < 500)
            imgDescriptorMat.push_back(Mat::zeros((500 - imgDescriptorMat.rows),32,CV_8U));
        matonerow = imgDescriptorMat.reshape(1,1);
        indexTemp.push_back(matonerow);
        imgDescriptorMat.copyTo(tmp.descriptors);
        frames.append(tmp);
    }
}

it returns the amount of differing bits between the 2 matches. so for 500 descriptors at 32 BYTES each so 256 bits, 256 X 500 descriptors 128,000 bits per set of descriptors so a 90% match would mean a value of 12,800 so anything lower than that is a good match

Thanks I did some reading and I think I've got a formula

ok given the following code

    index.knnSearch(mat,indices,dists,3);

when using binary descriptors how do I make dists results into a useful ratio or percentage match?

ok I made the index data global and now I get

Assertion failed: (queries.cols == veclen()), function knnSearch, file /tmp/opencv320160131-91598-1dax8xj/opencv-3.1.0/modules/flann/include/opencv2/flann/lsh_index.h, line 192.

@StevenPuttemans, yes I do a lot of checking on the frames. I check the frame mat for empty, I check iscontinuous then I check the descriptors mat after computing for the same checks and drop them if they fail. I also pad them with zeros to make them the same size for the index. The console shows video errors from the ffmpeg backend and I don't see any. Also ffprobe shows no errors: /usr/local/bin/ffprobe -v quiet -print_format json -show_error /Users/phire/Dropbox/Geoengineering\ 1\ hour\ vid.m4v had blank output

@berak, wait, do you mean that the data I pass to the index() call needs to be global? or at least accesible from the function that I call knnsearch from? If that's the case why wouldn't it crash every time?

that's just it, there is no minimal. I can try to spend the whole day tomorrow copying and pasting trying to give a compilable example so that it can be reproduced

the problem is I don't know what to leave out, because you have to run the program with the video file and it has to process it like my whole program would to get the error.

I've looked at the source code here: https://github.com/Itseez/opencv/blob...

but... where's the beef? it doesn't seem to do any actual work? it seems circular to me, does anyone know where the code that actually modifies dists and indices is?

ok here is what I hope is a simple question.

Does anyone know if flann uses the transparent api? In other words would I benefit from passing a UMat to flann::index::index() and flann::index::knnsearch() ?

That's one of the first pages that I looked at but it doesn't tell you what the format of the dists and indices mats are in. Like each row is a one byte or two byte representation of whatever. Is there nowhere to find that out

And ya that page is what I was going by trying to use the overloaded function. I tried with vector <uchar> int nothing worked

EDIT: after looking through that page again I clicked on ElementType and found: typedef L2<t>::ElementType cv::flann::Index_< T >::ElementType

and L2 has to do with sift and surf type features, so maybe that overloaded function doesn't work with binary descriptors?

thanks, but is there any documentation on the format of indices and dists that you can point me to?

got it, all I had to do was add the lines

        if (mat.rows < 500)
        mat.push_back(Mat::zeros((500 - mat.rows),32,mat.type()));

I am putting all the descriptors for multiple images from an ORB detector into one Mat to feed to cv::flann::index->build. Here is the code I am trying.

EDIT: I guess the main question right now. Is it safe to pad the descriptors with zeros at the end to make them the same number of columns?

while (true)
{
    buffer = indexBuilder.value("descriptors").toString().toStdString();
    Mat mat;
    Mat matonerow;
    FileStorage fs(buffer,FileStorage::READ | FileStorage::MEMORY | FileStorage::FORMAT_YAML);
    fs["descriptors"] >> mat;
    fs.release();
    matonerow =  mat.reshape(1,1);
    qDebug() << "matonerow cols" << matonerow.cols << "rows" << matonerow.rows;
    indexTemp.push_back(matonerow);
    progress->setValue(iterations);
    iterations++;
    if (!indexBuilder.next())
        break;
}

the problem is that some of the descriptors are smaller, so the call to push_back fails with

OpenCV Error: Sizes of input arguments do not match () in push_back, file /tmp/opencv320160111-57911-i2x9h0/opencv-3.1.0/modules/core/src/matrix.cpp, line 834
libc++abi.dylib: terminating with uncaught exception of type cv::Exception: /tmp/opencv320160111-57911-i2x9h0/opencv-3.1.0/modules/core/src/matrix.cpp:834: error: (-209)  in function push_back

so how do I copy the one row mat to the tempindex mat? NOTE: I am using mat because I intend to convert it to UMat before passing it to build the index so I can take advantage of opencl through TAPI.

UPDATE: ok it works now but only for rows with the same amount of columns, which is no better than push_back so I still need another method!

    try
    {
        if (iterations == 1)
            indexTemp = Mat(row_count,matonerow.cols,CV_8U);

            matonerow.copyTo(indexTemp.row(iterations - 1));
    }
    catch (Exception e)
    {
        qCritical() << "failed to copy row of Mat at loop" << iterations << e.err.c_str() << e.msg.c_str();
    }

NOTE: iterations starts at 1

those help, but what I really need now is some documentation on how to build the index with more than one image. I know it takes descriptors in the arguments, but can this be an array of arrays so each images descriptors are separate enough to tell which files match?

I am writing a program to find duplicate videos, I am pulling frames every 500ms and using ORB to compute. I was looking at using FLANN for matching and heard about index. Reading further it says that you sacrifice accuracy, but I wonder how much. Since I am matching videos I only want 95% or higher matches of frames and will determine video matches by how many frames in common at that accuracy.

EDIT: I have been reading and see that orb is a binary descriptor and the flann matcher I was thinking of using is not for binary descriptors. Does the index even work with binary descriptors? Do I need an index or is matching binary descriptors by hamming distance fast enough not to need one?