Ask Your Question
8

SVM bias on weights of positives and negatives

asked 2014-01-20 09:36:09 -0500

updated 2016-01-14 06:31:20 -0500

For the purpose of algorithm comparison I want to compare the detection result of my cascade classifier trained using the viola & jones framework versus an SVM HOG classifier I have trained.

However, the VJ framework has no problem with an uneven ratio for positives and negatives, for example 500 positives and 100 negatives (don't tell me this is wrong please, my whole PhD is about proving that context awareness can reduce the negative numbers effectively. .

If I would like to convert this to SVM classification however, I am reading at many blogposts that using CvSVM should be done with an equal number of positive and negative training samples. If not I should add a bias, but however, I cannot find an example that does exactly that.

Could anyone give me some directions on how I should apply this? Is this even possible within the OpenCV framework?

Kind regards,

Steven

EDIT 1: I have noticed people suggest using libSVM or SVMlight for training the model. However, I would like to stick to openCV only, which should actually be possible.

edit retag flag offensive close merge delete

Comments

2

Never used SVM so I can't really help you there, but please post your conclusions when you finish this study. Good luck. I can refer you to this study by Piotr Dollar https://www.google.pt/url?sa=t&rct=j&q=&esrc=s&source=web&cd=3&ved=0CEIQFjAC&url=https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fmlsurveys%2F97.pdf&ei=IP3fUr-hFsSX1AX26YC4Bg&usg=AFQjCNEHJ_cBNqOAmMV_GE4q6plvpmmveA&sig2=BLIfURmlWm4u_33KLLJbRg&bvm=bv.59568121,d.bGQ&cad=rja where he compares the performance of various detectors on multiple pedestrian datasets, among them the SVM HOG and VJ algorithms. It is a pretty good evaluation.

Pedro Batista gravatar imagePedro Batista ( 2014-01-22 11:16:50 -0500 )edit
1

I know of the work of Dollár which is one of the main paper resources I am using for my own research :) I have succesfully created a wrapper around the output of the CvSVM training to create an output that can be used by the HOGDescriptor interface. Master branch even got a recently merged version for that done by @Mathieu Burchanon. However, as many of the ML functionality in OpenCV, the parameter tuning and study is enourmous :) I will see how far I get, feel free to contact me from time to time to ask about progress!

StevenPuttemans gravatar imageStevenPuttemans ( 2014-01-23 02:59:24 -0500 )edit

3 answers

Sort by » oldest newest most voted
6

answered 2014-04-15 10:08:01 -0500

Hansg91 gravatar image

updated 2014-04-17 05:25:30 -0500

Regarding the weights, I did some small tests to check the influence. Here are my results:

No weights:

image description

With weights [0.9, 0.1] (0.9 for the largest class, 0.1 for the smallest class):

image description

You can see the change of the weights clearly in these pictures. I hope this clears things up a bit.

On a side note: I tried to do this in python but the class_weights variable does not appear to get set properly (see this question for more information). To circumvent this I had hardcoded the weights in the opencv source, using c++ would've most likely given the correct results as well, as the weights seem to get properly set for c++.

Code used to generate the example:

import numpy as np
import cv2
import matplotlib.pyplot as plt
from os.path import isfile
import scipy.misc
import xml.dom.minidom

"""Originally from: http://stackoverflow.com/questions/8687885/python-opencv-svm-implementation"""
class StatModel(object):
    """parent class - starting point to add abstraction"""
    def load(self, fn):
        self.model.load(fn)
    def save(self, fn):
        self.model.save(fn)

"""Wrapper for OpenCV SimpleVectorMachine algorithm"""
class SVM(StatModel):
    def __init__(self):
        self.model = cv2.SVM()

    def train(self, X, Y):
        #setting algorithm parameters
        params = dict( kernel_type = cv2.SVM_LINEAR, 
                       svm_type = cv2.SVM_C_SVC,
                       C = 1,
                       class_weights = [0.9, 0.1],) # THIS DOES NOT WORK IN OPENCV PYTHON

        self.model.train(X.astype(np.float32), Y.astype(np.float32), params = params)

# generates Gaussian dataset
def gen_gauss_dat(mu1, cov1, N1, mu2, cov2, N2):
    X1 = np.random.multivariate_normal(mu1, cov1, (N1))
    X2 = np.random.multivariate_normal(mu2, cov2, (N2))
    X = np.vstack([X1,X2])
    Y = np.hstack([np.ones((X1.shape[0]))*-1, np.ones((X2.shape[0]))])
    return X, Y

def load_model(path):
    if path == None:
        return None

    if isfile(path) == False:
        return None

    print "Loading model from {}".format(path)
    model = xml.dom.minidom.parse(path)

    # TODO: handle multiple support vectors?
    weights = np.fromstring(model.getElementsByTagName("_")[0].childNodes[0].nodeValue, sep=" ")
    alpha = float(model.getElementsByTagName("alpha")[0].childNodes[0].nodeValue)
    rho = float(model.getElementsByTagName("rho")[0].childNodes[0].nodeValue)
    weights = weights * -alpha
    return weights, rho

# generate Gaussian data
X, Y = gen_gauss_dat([0,0], np.identity(2)*1.5, 1000, [2,2], np.identity(2) * 0.5, 100)

# train on the generated Gaussian data
svm = SVM()
svm.train(X, Y)
svm.save("svm.xml")
w, b = load_model("svm.xml", )

# plot the data
plt.scatter(X[Y==-1,0],X[Y==-1,1], c="g", alpha=0.6, s=100)
plt.scatter(X[Y==1,0],X[Y==1,1], c="r", alpha=0.6, s=100)

x = [-10, 10]
y = [(-w[0] * x[0] - b) / w[1], \
     (-w[0] * x[1] - b) / w[1]]
plt.plot(x, y)
plt.axis("off")
plt.title("Linear SVM classifier Gaussian dataset")
plt.show()
edit flag offensive delete link more

Comments

Nice example, thank you!

Guanta gravatar imageGuanta ( 2014-04-15 10:13:51 -0500 )edit
1

Thank you for the sample, but it might be nice if you could add some of the code with the weights leading to these results :)

StevenPuttemans gravatar imageStevenPuttemans ( 2014-04-17 03:24:02 -0500 )edit
2

As requested, I added the code. Do note, adding these weights should _in theory_ work, but the python opencv bindings have a bug which causes the weights to not be set (as mentioned earlier, I hardcoded them in the OpenCV source just to test the weights).

Hansg91 gravatar imageHansg91 ( 2014-04-17 05:24:57 -0500 )edit

just out of curiosity: since class_weights is of type cv::Mat and not of type std::vector<>, have you tried np.array( [0.9, 0.1] ) instead of [0.9, 0.1] ? Maybe I am wrong here and it would also be converted implicitly...

Guanta gravatar imageGuanta ( 2014-04-17 06:31:13 -0500 )edit

... or since in the docs it actually says CvMat: cv.cvmat (don't know how to initialize an old opencv-mat under python)

Guanta gravatar imageGuanta ( 2014-04-17 06:35:41 -0500 )edit
2

Yes I tried that. The thing is the code that handles the conversion for CvSVMParams skips class_weights because of a missing flag for that parameter (CV_PROP_RW). If you add that, a linking error occurs that it cannot convert CvMat* to PyObj and vice versa. Same thing for non pointer, cv::Mat or cv::Mat*.

Hansg91 gravatar imageHansg91 ( 2014-04-17 06:35:59 -0500 )edit
3

answered 2014-01-24 03:20:06 -0500

updated 2014-01-24 06:27:47 -0500

Let's wrap some stuff up, any remarks more than welcome. I tried applying a per element weight factor, however this didn't seem to work, giving raise to an error in OpenCV execution:

OpenCv Error: Bad argument < params.class_weights must be 1d floating-pint vector containing as many elements as the number of classes >

This however means that it doesn't follow the default approach of LibSVM.

However, by specifying an equal weight or different weights for each class, it seems that my support vectors do not change in my XML file, which is quite weird as I see it. Or does anyone know why the support vectors should be identical and is this behaviour meant to be?

UPDATE 1

I have downloaded the Daimler person detector dataset. It has about 15.000 positive images (precropped to 48x96 resolution) and 7000 negative images (full size background images). If have taken the following steps:

Took the 15000 positive windows and created the HOG descriptor from it. Parameters are windowSize 48x96 cellSize 8x8 blockSize 16x16 and windowStride 8x8, which are the official parameters used.

Took the 7000 negatives and cut 15000 random 48x96 windows from those. Also put them through the descriptor creation process.

Have added both sets to a training vector matrix and created the corresponding labels.

Performed the SVM training using the following parameters

CvSVMParams params;
params.svm_type = CvSVM::C_SVC;
params.kernel_type = CvSVM::LINEAR; 
params.C = 1000; //high misclassification cost
params.term_crit = cvTermCriteria(CV_TERMCRIT_ITER, 1000, 1e-6); //stopping criteria loops = 1000 or EPS = 1e-6

Used a conversion function, inside a wrapper_class to generate a single vector for the HOG descriptor using the following code

void LinearSVM::get_primal_form_support_vector(std::vector<float>& support_vector) const {
    int sv_count = get_support_vector_count();
    const CvSVMDecisionFunc* df = decision_func;
    const double* alphas = df[0].alpha;
    double rho = df[0].rho;
    int var_count = get_var_count();
    support_vector.resize(var_count, 0);
    for (unsigned int r = 0; r < (unsigned)sv_count; r++) {
      float myalpha = alphas[r];
      const float* v = get_support_vector(r);
      for (int j = 0; j < var_count; j++,v++) {
        support_vector[j] += (-myalpha) * (*v);
      }
    }
    support_vector.push_back(rho);
}

Then loaded in a set of test images and performed the multiscale detector on it which is integrated in the HOGDescriptor function.

vector<float> support_vector;
SVM_model.get_primal_form_support_vector(support_vector);
hog_detect.setSVMDetector(support_vector);
hog_detect.detectMultiScale(test_image, locations, 0.0, Size(), Size(), 1.01);

However, the result is far from satisfying ... as you can see below

image description

So this raises some questions:

  1. Is it okay to just sample negative images randomly? Or should i first make sure that pedestrians in the original image are about the model size, so that negative samples are more representative?
  2. @Mathieu Barnachon, as you submitted a sortlike interface, do you maybe see anything I am doing wrong? This problem has been giving me a headache for over a week now.
  3. Also, training with equal weights for each class OR training with unequal (0.8 vs 0.2) weights for each class doesn't change the vector values outputted in the XML model... Is this even normal? I would expect the ...
(more)
edit flag offensive delete link more

Comments

1

I can tell by the result that some of the false positives are on objects with a clear vertical expression that may resemble a pedestrian, so that hints me that what you need is to add more significant samples on the training stage. Try bootstrapping the classifier, meaning, run that classifier on negative images, and add the false positives to your negative training set. When I trained my pedestrian classifier I repeated this process 3 times before having a decent detector

Pedro Batista gravatar imagePedro Batista ( 2014-01-27 05:07:49 -0500 )edit

Hmm could be a good idea indeed, however, I do find it quite weird, that the actual pedestrian isn't detected once.. this is not the behaviour I would expect, expecially not when using 15k+ examples to train. I like your suggestion and will try it out, but I am sure there are other things going wrong. Did you experiment yet with the weight bias?

StevenPuttemans gravatar imageStevenPuttemans ( 2014-01-27 05:48:12 -0500 )edit

I am trying an experiment first with 15000 positives en 45000 negatives. Let's see which result this creates :) I do like your hard negative training suggestion, just need to re-implement some of my code so that I can get this automated.

StevenPuttemans gravatar imageStevenPuttemans ( 2014-01-27 06:49:41 -0500 )edit

@Median, actually I was wondering, what kernel type have you been using for your SVM training? And how about your parameters of the SVM training? I am wondering if a linear kernel is even what I am looking for ...

StevenPuttemans gravatar imageStevenPuttemans ( 2014-01-27 08:05:53 -0500 )edit

I didn't use the bootstrapping technique on an SVM classifier, I did it in an adaboost infrastructure, which is less likely to get oversampled, so I guess I had more freedom to add samples. I really have no idea about optimal SVM parameters for pedestrian detection :s but I recall reading about non-linear svm working better for the task. About the result, I am guessing you are letting opencv group rectangles? You'll never know if the pedestrian wasn't actually detected until you see all the detections clearly, since it may be grouping similar rectangles in a way that the pedestrian doesn't seem to be detected.

Pedro Batista gravatar imagePedro Batista ( 2014-01-27 08:38:28 -0500 )edit

Actually the HOGDetector has innergrouping, and you cannot deactivate it by calling the function. It does detect the pedestrian after some tests, but still the main problem of weighting doesn't really get me anywhere. I am doing pure theoretical tests, meaning that I want to have a universal way of SVM training, not specific for pedestrians. However, I tought using an available dataset would help clear things out (since pedestrian sets are widely spread). Seems I was wrong ... :D

StevenPuttemans gravatar imageStevenPuttemans ( 2014-01-27 08:43:37 -0500 )edit

@StevenPuttemans Hey Steven, I'm wondering what kinda results you got at the end. I've computed some accuracy using DET curve with the INRIA dataset. The only difference are the detection parameters are scale and window. strides. Using the HOG INRIA parameters (1.2, 8x8) gives poor results on the INRIA dataset. Using HOG Opencv parameters (1.05, 4x4) gives better results. I'm not sure how they have manage to get good accuracy with scale that high. But anyway the best I could get is 0.35 (for FNR = FPPW). Both used the default opencv SVM

Romanzo gravatar imageRomanzo ( 2016-06-09 22:33:55 -0500 )edit

O well I skipped that part somewhere around februari 2014 and never got back to it, so cant give you more details ...

StevenPuttemans gravatar imageStevenPuttemans ( 2016-06-19 15:28:11 -0500 )edit

@Romanzo, @StevenPuttemans if you have time could you take a look at my question your remarks will be greatly appreciated

sturkmen gravatar imagesturkmen ( 2016-06-19 21:37:10 -0500 )edit

No worries I actually answer to my question myself. Looks like HOG OpenCV can't get better than 46% log-average miss-rate. I achieved the same results with my own SVM. However it's not accurate enough. I found the paper comparing the different pedestrian detection algorithm so I will probably have a look at that! Any hint on some algorithm you tried and work better?

Romanzo gravatar imageRomanzo ( 2016-06-20 20:43:01 -0500 )edit
3

answered 2014-01-22 10:24:58 -0500

Guanta gravatar image

If I understood http://pyml.sourceforge.net/doc/howto.pdf and http://www.csie.ntu.edu.tw/~cjlin/papers/libsvm.pdf correctly, this should work with the class_weights-parameter, which serves as a penalty, i.e. assign a higher weight to the class having more samples.

Example: 400 pos, 100 neg --> class_weight for positive class = 4/5 = 0.8 and 0.2 for the negative class.

Note that for other classifiers this can often be achieved by giving a class-prior. Also note that if you have an unbalanced test (not training) set then a proper error measurement (e.g. unweighted average precision/recall, Vinciarelli et al. "The INTERSPEECH 2012 Speaker Trait Challenge") should be used.

Btw. as far as I know OpenCV's SVM is based on libSVM.

edit flag offensive delete link more

Comments

Well I am convinced it is actually based on libSVM since it is stated in the original submission of the code to the OpenCV repository. I will try out your suggestion of using the class_weights parameter and see if that actually does the trick! Thank you very much for the suggestion! Will keep you posted about progress!

StevenPuttemans gravatar imageStevenPuttemans ( 2014-01-23 03:02:17 -0500 )edit

Btw thanks for the resources! They will become usefull in the future :)

StevenPuttemans gravatar imageStevenPuttemans ( 2014-01-23 03:05:52 -0500 )edit

CvMat* weights = cvCreateMat(1, 2, CV_32FC1);

// Have any idea how you fill this kind of stuff? The C - API is not my thing and I am kinda stuck :P

params.class_weights = weights;

StevenPuttemans gravatar imageStevenPuttemans ( 2014-01-23 06:51:39 -0500 )edit
1

guessing from the documentation: CV_MAT_ELEM(&weights, float, 0, 0) = 0.8; CV_MAT_ELEM(&weights, float, 0, 1) = 0.2; .

You can also stick to C++ and write: cv::Mat1f weights(1,2); weights(0,0) = 0.8; weights(0,1) = 0.2; CvMat old_weights = weights; (this will be then cast to the C-mat) and then you can call CvSVMParams(... , &old_weights, ...);

Guanta gravatar imageGuanta ( 2014-01-23 07:07:10 -0500 )edit

Thanks! Will try it out!

StevenPuttemans gravatar imageStevenPuttemans ( 2014-01-23 08:29:33 -0500 )edit

Okay a small update on this. It seems that weights are instance specific and not class specific. This means that you need to add a weight vector just like we add a label vector. Or this is at least what I grasp from LibSVM guidelines. Will keep you up to date on the result!

StevenPuttemans gravatar imageStevenPuttemans ( 2014-01-24 02:57:26 -0500 )edit

Actually after multiple tests, I am not yet receiving good results. @Guanta, I am wondering if we are taking a wrong approach at the weights. For the moment you are assigning a larger weight (in your example) for the positive class then for the negative class. However, having a larger positive class, doesn't it seem suitable to give that a smaller weight, and a larger weight to the negative ones?

StevenPuttemans gravatar imageStevenPuttemans ( 2014-01-27 07:32:27 -0500 )edit

Documentation states punishing value, so I think you want a higher punishment for the larger class.

Guanta gravatar imageGuanta ( 2014-01-27 07:44:55 -0500 )edit

Hmm, I just succeeded in starting a training with weight 3 versus 1, I am wondering what the result will be.

StevenPuttemans gravatar imageStevenPuttemans ( 2014-01-27 07:53:17 -0500 )edit

But the punishment is about wrong classification right? I do want all those negatives to be punished as hard as the positives for wrong classification :) This doesn't seem as easy as I tought :)

StevenPuttemans gravatar imageStevenPuttemans ( 2014-01-27 07:55:13 -0500 )edit
Login/Signup to Answer

Question Tools

3 followers

Stats

Asked: 2014-01-20 09:36:09 -0500

Seen: 11,573 times

Last updated: Apr 17 '14