I will try to be brief, this is a toughy though so don't hesitate to ask questions or feel bad if your not sure!
So I run some smoothing algorithms on my data before I send to opencv mlp for classification. While I was optimizing the smoothing parameters I noticed this problem. I iterate through each parameter as a variable then output the success percent from mlp, looking for which variable gives the best results. The problem is when I go back and retest that same parameter outside of the loop the results are different.
When I compare the data that got sent to mlp from the loop, to the data when it was called just one time outside of the loop; the data is exactly the same! For some reason opencv mlp classified them differently though! For example, on var 25 in the Main Loop code below, it gave me 72.3% success rate, but if i go back and redo the test, setting it to 25 outside of the loop like in the Singularity block code it gives me 71.4%. Strangely enough though, var 10 in the loop (the very first iteration) gives me 73.4% and in the singularity block it still gives me exactly the same 73.4%. So it makes me think there is something that is stacking each iteration.
Each iteration of the loop I used a completely different mlp object, so its not like I'm using the previous iteration's trained mlp. As said earlier, data looks the same rather it was the 5th iteration or the first, even after running PCA, so its not the smoothing or PCA functions.
This problem is bad, as I can not optimize the other parameters if they give me dubious results.
I am so lost... does mlp change how they use weights or some other variable after being called? Like it initializes using the same weights as the last mlp call or something like that? Even though they are different objects?? If so can I turn this feature off? Thanks so much for any help, I really need it!
Main Loop:
// loop through different variables for golay window size
for (double var = 10; var <= 40; var += 5)
{
SpecData trainData = SpecData.grabData("EICTrain"); // grab our training data
SavitzkyGolay::run(trainData, var, 4); // apply smoothing to data with variable parameter
PCA pca(trainData); // build a pca basis
pca.apply(trainData, 15); // apply training data to pca basis
MultiLayerPerceptron mlp(trainData, testClass); // build and train our opencv mlp
SpecData testData = SpecData.grabData("EICTest"); // grab testing data
SavitzkyGolay::run(testData, var, 4); // smooth testing data with same paramters as training data
pca.apply(testData, 15); // apply testing data to same pca basis
testData = mlp.classify(testData, false, testClass); // test data on trained mlp
}
'
Singularity call:
SpecData trainData = SpecData.grabData("EICTrain"); // grab our training data
SavitzkyGolay::run(trainData, 25, 4); // apply smoothing to data with variable parameter
PCA pca(trainData); // build a pca basis
pca.apply(trainData, 15); // apply training data to pca basis
MultiLayerPerceptron mlp(trainData, testClass); // build and train our opencv mlp
SpecData testData = SpecData.grabData("EICTest"); // grab testing data
SavitzkyGolay::run(testData, 25, 4); // smooth testing data with same paramters as training data
pca.apply(testData, 15); // apply testing data to same pca basis
testData = mlp.classify(testData, false, testClass); // test data on trained mlp
'
MLP Constructor/trainer:
// find how many different classifications we have (unique classes contains all the different classifications our mlp can make)
for (map<string, int>::iterator ucIt = uniqueClasses.begin(); ucIt != uniqueClasses.end(); ucIt++) {
// construct our layers to use in mlp
cv::Mat layers = cv::Mat(2 + 1, 1, CV_32SC1);
layers.row(0) = cv::Scalar(numDataPoints); // equal to number of data points in feature vector
for (int i = 1; i <= numHiddenLayers; i++)
layers.row(i) = cv::Scalar(2 + (numDataPoints / 2));
layers.row(1 + numHiddenLayers) = cv::Scalar(1); // equal to number of classifications in training data
// create training classes matrix
cv::Mat trainingClasses(numVectors, 1, cv::DataType<float>::type, -1);
for (int rI = 0; rI < numVectors; rI++)
if (trainingData.identifiers[trainingClass][rI] == ucIt->first)
trainingClasses.at<float>(rI, 0) = 1;
// create a multilayer perceptron (mlp) with specified layers and nodes
if (mlpByClass.find(ucIt->first) != mlpByClass.end()) {
mlpByClass[ucIt->first]->clear();
delete mlpByClass[ucIt->first];
}
mlpByClass[ucIt->first] = new CvANN_MLP(layers);
mlpByClass[ucIt->first]->train(trainingData.mat.mat, trainingClasses, cv::Mat(), cv::Mat());
}
'
MLP classify:
map<string, map<int, double>> testingResults;
// get a score for each classification from our trained mlp
for (map<string, int>::iterator ucIt = uniqueClasses.begin(); ucIt != uniqueClasses.end(); ucIt++) {
for (int r = 0; r < testingData.mat.numRows(); r++) {
// make a cv mat to store the score for this classification
cv::Mat response(1, 1, CV_32FC1);
// sample is feature vector, consisting of one row from testing data
cv::Mat sample = testingData.mat.mat.row(r);
// predict what the sample is, returning the classification(s) to response
mlpByClass[ucIt->first]->predict(sample, response);
// store response in testing results
float score = response.at<float>(0, 0);
testingResults[ucIt->first][r] = score;
}
}
// now make predictions off of responses
for (int rI = 0; rI < testingData.mat.numRows(); rI++) {
// check for high score
float maxScore = FLT_MIN;
string prediction = "";
for (map<string, map<int, double>>::iterator trIt = testingResults.begin(); trIt != testingResults.end(); trIt++) {
double score = trIt->second[rI];
if (score > maxScore) {
maxScore = score;
prediction = trIt->first;
}
}
// log prediction
testingData.identifiers["Prediction"][rI] = prediction;
// check if right
if (prediction == testingData.identifiers[trainTestClass][rI])
properlyPredicted++;
}
// calculate our success percent in classification
double successPercent = 100 * (double)((double)properlyPredicted / (double)numRows);
// log success percent in output object
testingData.successPercent = successPercent;
// concat testing results to end of classifications edge in return object
testingData.edges[SpecData::CLASS].insert(testingResults.begin(), testingResults.end());
// send back our tested data
return testingData;
}