Revision history [back]

I guess the names are misleading in the old OpenCV API. Let me attempt to explain it, but first give you an advice on using the OpenCV2 C++ API. I write this as a reference to people coming here from Google, so excuse if this is a lengthy answer. This may not directly apply to you using JavaCV, but for everyone else: I strongly suggest using the new OpenCV2 interface, just because the old C API is probably not supported in the future anymore.

Instead of cvCalcEigenObjects, cvEigenDecomposite and cvEigenProjection OpenCV now comes with the cv::PCA class, which makes performing a Principal Component Analysis really simple. Instead of going through it in-depth I am now pasting a sample (so people have it as a starting point) and then I'll answer your question on the old API.

This is an example to perform a Principal Component Analysis on a given set of images, in this example either with a hardcoded set of images or a CSV file. I think a modified version of this was added to the OpenCV documentation recently. By the way you'll find a source code example on using cv::PCA::project and cv::PCA::backProject in the documentation on cv::PCA:

/*
 * Copyright (c) 2012. Philipp Wagner <bytefish[at]gmx[dot]de>.
 * Released to public domain under terms of the BSD Simplified license.
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions are met:
 *   * Redistributions of source code must retain the above copyright
 *     notice, this list of conditions and the following disclaimer.
 *   * Redistributions in binary form must reproduce the above copyright
 *     notice, this list of conditions and the following disclaimer in the
 *     documentation and/or other materials provided with the distribution.
 *   * Neither the name of the organization nor the names of its contributors
 *     may be used to endorse or promote products derived from this software
 *     without specific prior written permission.
 *
 *   See <http://www.opensource.org/licenses/bsd-license>
 */

#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>

#include <fstream>
#include <sstream>

using namespace cv;
using namespace std;

// Reads the images and labels from a given CSV file, a valid file would
// look like this:
//
//      /path/to/person0/image0.jpg;0
//      /path/to/person0/image1.jpg;0
//      /path/to/person1/image0.jpg;1
//      /path/to/person1/image1.jpg;1
//      ...
//
void read_csv(const string& filename, vector<Mat>& images, vector<int>& labels) {
    std::ifstream file(filename.c_str(), ifstream::in);
    if(!file)
        throw std::exception();
    std::string line, path, classlabel;
    // For each line in the given file:
    while (std::getline(file, line)) {
        // Get the current line:
        std::stringstream liness(line);
        // Split it at the semicolon:
        std::getline(liness, path, ';');
        std::getline(liness, classlabel);
        // And push back the data into the result vectors:
        images.push_back(imread(path, IMREAD_GRAYSCALE));
        labels.push_back(atoi(classlabel.c_str()));
    }
}

// Normalizes a given image into a value range between 0 and 255.
Mat norm_0_255(const Mat& src) {
    // Create and return normalized image:
    Mat dst;
    switch(src.channels()) {
    case 1:
        cv::normalize(src, dst, 0, 255, NORM_MINMAX, CV_8UC1);
        break;
    case 3:
        cv::normalize(src, dst, 0, 255, NORM_MINMAX, CV_8UC3);
        break;
    default:
        src.copyTo(dst);
        break;
    }
    return dst;
}

// Converts the images given in src into a row matrix.
Mat asRowMatrix(const vector<Mat>& src, int rtype, double alpha = 1, double beta = 0) {
    // Number of samples:
    size_t n = src.size();
    // Return empty matrix if no matrices given:
    if(n == 0)
        return Mat();
    // dimensionality of (reshaped) samples
    size_t d = src[0].total();
    // Create resulting data matrix:
    Mat data(n, d, rtype);
    // Now copy data:
    for(int i = 0; i < n; i++) {
        //
        if(src[i].empty()) {
            string error_message = format("Image number %d was empty, please check your input data.", i);
            CV_Error(CV_StsBadArg, error_message);
        }
        // Make sure data can be reshaped, throw a meaningful exception if not!
        if(src[i].total() != d) {
            string error_message = format("Wrong number of elements in matrix #%d! Expected %d was %d.", i, d, src[i].total());
            CV_Error(CV_StsBadArg, error_message);
        }
        // Get a hold of the current row:
        Mat xi = data.row(i);
        // Make reshape happy by cloning for non-continuous matrices:
        if(src[i].isContinuous()) {
            src[i].reshape(1, 1).convertTo(xi, rtype, alpha, beta);
        } else {
            src[i].clone().reshape(1, 1).convertTo(xi, rtype, alpha, beta);
        }
    }
    return data;
}

int main(int argc, const char *argv[]) {
    // Holds some images:
    vector<Mat> db;

    // Load the greyscale images. The images in the example are
    // taken from the AT&T Facedatabase, which is publicly available
    // at:
    //
    //      http://www.cl.cam.ac.uk/research/dtg/attarchive/facedatabase.html
    //
    // This is the path to where I stored the images, yours is different!
    //
    string prefix = "/home/philipp/facerec/data/at/";

    db.push_back(imread(prefix + "s1/1.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s1/2.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s1/3.pgm", IMREAD_GRAYSCALE));

    db.push_back(imread(prefix + "s2/1.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s2/2.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s2/3.pgm", IMREAD_GRAYSCALE));

    db.push_back(imread(prefix + "s3/1.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s3/2.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s3/3.pgm", IMREAD_GRAYSCALE));

    db.push_back(imread(prefix + "s4/1.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s4/2.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s4/3.pgm", IMREAD_GRAYSCALE));

    // The following would read the images from a given CSV file
    // instead, which would look like:
    //
    //      /path/to/person0/image0.jpg;0
    //      /path/to/person0/image1.jpg;0
    //      /path/to/person1/image0.jpg;1
    //      /path/to/person1/image1.jpg;1
    //      ...
    //
    // Uncomment this to load from a CSV file:
    //

    /*
    vector<int> labels;
    read_csv("/home/philipp/facerec/data/at.txt", db, labels);
    */

    // Build a matrix with the observations in row:
    Mat data = asRowMatrix(db, CV_32FC1);

    // Number of components to keep for the PCA:
    int num_components = 10;

    // Perform a PCA:
    PCA pca(data, Mat(), CV_PCA_DATA_AS_ROW, num_components);

    // And copy the PCA results:
    Mat mean = pca.mean.clone();
    Mat eigenvalues = pca.eigenvalues.clone();
    Mat eigenvectors = pca.eigenvectors.clone();

    // The mean face:
    imshow("avg", norm_0_255(mean.reshape(1, db[0].rows)));

    // The first three eigenfaces:
    imshow("pc1", norm_0_255(pca.eigenvectors.row(0)).reshape(1, db[0].rows));
    imshow("pc2", norm_0_255(pca.eigenvectors.row(1)).reshape(1, db[0].rows));
    imshow("pc3", norm_0_255(pca.eigenvectors.row(2)).reshape(1, db[0].rows));

    // Show the images:
    waitKey(0);

    // Success!
    return 0;
}

So let's have a look how to map cv::PCA to the old OpenCV API. First of all you would need to find the orthonormal basis W. This is done by using CalcEigenObjects, quoting the documentation:

Calculates orthonormal eigen basis and averaged object for group of input objects

void cvCalcEigenObjects( int nObjects, void* input, void* output, int ioFlags, int ioBufSize, void* userData, CvTermCriteria* calcLimit, IplImage* avg, float* eigVals );

• nObjects: Number of source objects.
• input: Pointer either to the array of IplImage input objects or to the read callback function according to the value of the parameter ioFlags.
• output: Pointer either to the array of eigen objects or to the write callback function according to the value of the parameter ioFlags .
• ioFlags: Input/output flags.
• ioBufSize: Input/output buffer size in bytes. The size is zero, if unknown.
• userData: Pointer to the structure that contains all necessary data for the callback functions.
• calcLimit: Criteria that determine when to stop calculation of eigen objects.
• avg: Averaged object.
• eigVals: Pointer to the eigenvalues array in the descending order; may be NULL .

The function cvCalcEigenObjects calculates orthonormal eigen basis and the averaged object for a group of the input objects. Depending on ioFlags parameter it may be used either in direct access or callback mode. Depending on the parameter calcLimit, calculations are finished either after first calcLimit.maxIters dominating eigen objects are retrieved or if the ratio of the current eigenvalue to the largest eigenvalue comes down to calcLimit.epsilon threshold. The value calcLimit -> type must be CV_TERMCRIT_NUMB, CV_TERMCRIT_EPS, or CV_TERMCRIT_NUMB | CV_TERMCRIT_EPS . The function returns the real values calcLimit -> maxIter and calcLimit -> epsilon .

The function also calculates the averaged object, which must be created previously. Calculated eigen objects are arranged according to the corresponding eigenvalues in the descending order.

The parameter eigVals may be equal to NULL, if eigenvalues are not needed.

The function cvCalcEigenObjects uses the function cvCalcCovarMatrixEx.

Once you have calculated the orthonormal basis W of your data (in this case your input images), you can either project your data X into the subspace:

• Y = (X - mean) * W

Or you can reconstruct your data X from its lower-dimensional representation Y as:

• X = W*Y + mean

So the question is, which of the function does the projection and which one reconstructs? With cv::PCA it is clear. cv::PCA::project projects your data, while cv::PCA::backProject restores your data.

First let's have a look at cvEigenDecomposite. The old documentation states:

Calculates all decomposition coefficients for input object

void cvEigenDecomposite( IplImage* obj, int nEigObjs, void* eigInput, int ioFlags, void* userData, IplImage* avg, float* coeffs );

obj: Input object. nEigObjs: Number of eigen objects. eigInput: Pointer either to the array of IplImage input objects or to the read callback function according to the value of the parameter ioFlags. ioFlags: Input/output flags. userData: Pointer to the structure that contains all necessary data for the callback functions. avg: Averaged object. coeffs: Calculated coefficients; an output parameter.

The function cvEigenDecomposite calculates all decomposition coefficients for the input object using the previously calculated eigen objects basis and the averaged object. Depending on ioFlags parameter it may be used either in direct access or callback mode.

So cvEigenDecomposite performs the mapping to the lower dimensional space, while cvEigenProjection (quoting from the documentation) restores an object using previously calculated eigen objects basis, averaged object, and decomposition coefficients of the restored object. As a reference I am again quoting from the documentation:

Calculates object projection to the eigen sub-space

void cvEigenProjection( int nEigObjs, void* eigInput, int ioFlags, void* userData, float* coeffs, IplImage* avg, IplImage* proj );

• nEigObjs: Number of eigen objects.
• eigInput: Pointer either to the array of IplImage input objects or to the read callback function according to the value of the parameter ioFlags.
• ioFlags: Input/output flags.
• userData: Pointer to the structure that contains all necessary data for the callback functions.
• coeffs: Previously calculated decomposition coefficients.
• avg: Averaged object.
• proj: Decomposed object projection to the eigen sub-space.

The function cvEigenProjection calculates an object projection to the eigen sub-space or, in other words, restores an object using previously calculated eigen objects basis, averaged object, and decomposition coefficients of the restored object. Depending on ioFlags parameter it may be used either in direct access or callback mode.

The functions of the eigen objects group have been developed to be used for any number of objects, even if their total size exceeds free RAM size. So the functions may be used in two main modes. Direct access mode is the best choice if the size of free RAM is sufficient for all input and eigen objects allocation. This mode is set if the parameter ioFlags is equal to CV_EIGOBJ_NO_CALLBACK. In this case input and output parameters are pointers to arrays of input/output objects of IplImage* type. The parameters ioBufSize and userData are not used.

I guess the names are misleading in the old OpenCV API. Let me attempt to explain it, but first give you an advice on using the OpenCV2 C++ API. I write this as a reference to people coming here from Google, so excuse if this is a lengthy answer. This may not directly apply to you using JavaCV, but for everyone else: I strongly suggest using the new OpenCV2 interface, just because the old C API is probably not supported in the future anymore.

Instead of cvCalcEigenObjects, cvEigenDecomposite and cvEigenProjection OpenCV now comes with the cv::PCA class, which makes performing a Principal Component Analysis really simple. Instead of going through it in-depth I am now pasting a sample (so people have it as a starting point) and then I'll answer your question on the old API.

This is an example to perform a Principal Component Analysis on a given set of images, in this example either with a hardcoded set of images or a CSV file. I think a modified version of this was added to the OpenCV documentation recently. By the way you'll find a source code example on using cv::PCA::project and cv::PCA::backProject in the documentation on cv::PCA:

/*
 * Copyright (c) 2012. Philipp Wagner <bytefish[at]gmx[dot]de>.
 * Released to public domain under terms of the BSD Simplified license.
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions are met:
 *   * Redistributions of source code must retain the above copyright
 *     notice, this list of conditions and the following disclaimer.
 *   * Redistributions in binary form must reproduce the above copyright
 *     notice, this list of conditions and the following disclaimer in the
 *     documentation and/or other materials provided with the distribution.
 *   * Neither the name of the organization nor the names of its contributors
 *     may be used to endorse or promote products derived from this software
 *     without specific prior written permission.
 *
 *   See <http://www.opensource.org/licenses/bsd-license>
 */

#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>

#include <fstream>
#include <sstream>

using namespace cv;
using namespace std;

// Reads the images and labels from a given CSV file, a valid file would
// look like this:
//
//      /path/to/person0/image0.jpg;0
//      /path/to/person0/image1.jpg;0
//      /path/to/person1/image0.jpg;1
//      /path/to/person1/image1.jpg;1
//      ...
//
void read_csv(const string& filename, vector<Mat>& images, vector<int>& labels) {
    std::ifstream file(filename.c_str(), ifstream::in);
    if(!file)
        throw std::exception();
    std::string line, path, classlabel;
    // For each line in the given file:
    while (std::getline(file, line)) {
        // Get the current line:
        std::stringstream liness(line);
        // Split it at the semicolon:
        std::getline(liness, path, ';');
        std::getline(liness, classlabel);
        // And push back the data into the result vectors:
        images.push_back(imread(path, IMREAD_GRAYSCALE));
        labels.push_back(atoi(classlabel.c_str()));
    }
}

// Normalizes a given image into a value range between 0 and 255.
Mat norm_0_255(const Mat& src) {
    // Create and return normalized image:
    Mat dst;
    switch(src.channels()) {
    case 1:
        cv::normalize(src, dst, 0, 255, NORM_MINMAX, CV_8UC1);
        break;
    case 3:
        cv::normalize(src, dst, 0, 255, NORM_MINMAX, CV_8UC3);
        break;
    default:
        src.copyTo(dst);
        break;
    }
    return dst;
}

// Converts the images given in src into a row matrix.
Mat asRowMatrix(const vector<Mat>& src, int rtype, double alpha = 1, double beta = 0) {
    // Number of samples:
    size_t n = src.size();
    // Return empty matrix if no matrices given:
    if(n == 0)
        return Mat();
    // dimensionality of (reshaped) samples
    size_t d = src[0].total();
    // Create resulting data matrix:
    Mat data(n, d, rtype);
    // Now copy data:
    for(int i = 0; i < n; i++) {
        //
        if(src[i].empty()) {
            string error_message = format("Image number %d was empty, please check your input data.", i);
            CV_Error(CV_StsBadArg, error_message);
        }
        // Make sure data can be reshaped, throw a meaningful exception if not!
        if(src[i].total() != d) {
            string error_message = format("Wrong number of elements in matrix #%d! Expected %d was %d.", i, d, src[i].total());
            CV_Error(CV_StsBadArg, error_message);
        }
        // Get a hold of the current row:
        Mat xi = data.row(i);
        // Make reshape happy by cloning for non-continuous matrices:
        if(src[i].isContinuous()) {
            src[i].reshape(1, 1).convertTo(xi, rtype, alpha, beta);
        } else {
            src[i].clone().reshape(1, 1).convertTo(xi, rtype, alpha, beta);
        }
    }
    return data;
}

int main(int argc, const char *argv[]) {
    // Holds some images:
    vector<Mat> db;

    // Load the greyscale images. The images in the example are
    // taken from the AT&T Facedatabase, which is publicly available
    // at:
    //
    //      http://www.cl.cam.ac.uk/research/dtg/attarchive/facedatabase.html
    //
    // This is the path to where I stored the images, yours is different!
    //
    string prefix = "/home/philipp/facerec/data/at/";

    db.push_back(imread(prefix + "s1/1.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s1/2.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s1/3.pgm", IMREAD_GRAYSCALE));

    db.push_back(imread(prefix + "s2/1.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s2/2.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s2/3.pgm", IMREAD_GRAYSCALE));

    db.push_back(imread(prefix + "s3/1.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s3/2.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s3/3.pgm", IMREAD_GRAYSCALE));

    db.push_back(imread(prefix + "s4/1.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s4/2.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s4/3.pgm", IMREAD_GRAYSCALE));

    // The following would read the images from a given CSV file
    // instead, which would look like:
    //
    //      /path/to/person0/image0.jpg;0
    //      /path/to/person0/image1.jpg;0
    //      /path/to/person1/image0.jpg;1
    //      /path/to/person1/image1.jpg;1
    //      ...
    //
    // Uncomment this to load from a CSV file:
    //

    /*
    vector<int> labels;
    read_csv("/home/philipp/facerec/data/at.txt", db, labels);
    */

    // Build a matrix with the observations in row:
    Mat data = asRowMatrix(db, CV_32FC1);

    // Number of components to keep for the PCA:
    int num_components = 10;

    // Perform a PCA:
    PCA pca(data, Mat(), CV_PCA_DATA_AS_ROW, num_components);

    // And copy the PCA results:
    Mat mean = pca.mean.clone();
    Mat eigenvalues = pca.eigenvalues.clone();
    Mat eigenvectors = pca.eigenvectors.clone();

    // The mean face:
    imshow("avg", norm_0_255(mean.reshape(1, db[0].rows)));

    // The first three eigenfaces:
    imshow("pc1", norm_0_255(pca.eigenvectors.row(0)).reshape(1, db[0].rows));
    imshow("pc2", norm_0_255(pca.eigenvectors.row(1)).reshape(1, db[0].rows));
    imshow("pc3", norm_0_255(pca.eigenvectors.row(2)).reshape(1, db[0].rows));

    // Show the images:
    waitKey(0);

    // Success!
    return 0;
}

So let's have a look how to map cv::PCA to the old OpenCV API. First of all you would need to find the orthonormal basis W. This is done by using CalcEigenObjects, quoting the documentation:

Calculates orthonormal eigen basis and averaged object for group of input objects

void cvCalcEigenObjects( int nObjects, void* input, void* output, int ioFlags, int ioBufSize, void* userData, CvTermCriteria* calcLimit, IplImage* avg, float* eigVals );

• nObjects: Number of source objects.
• input: Pointer either to the array of IplImage input objects or to the read callback function according to the value of the parameter ioFlags.
• output: Pointer either to the array of eigen objects or to the write callback function according to the value of the parameter ioFlags .
• ioFlags: Input/output flags.
• ioBufSize: Input/output buffer size in bytes. The size is zero, if unknown.
• userData: Pointer to the structure that contains all necessary data for the callback functions.
• calcLimit: Criteria that determine when to stop calculation of eigen objects.
• avg: Averaged object.
• eigVals: Pointer to the eigenvalues array in the descending order; may be NULL .

The function cvCalcEigenObjects calculates orthonormal eigen basis and the averaged object for a group of the input objects. Depending on ioFlags parameter it may be used either in direct access or callback mode. Depending on the parameter calcLimit, calculations are finished either after first calcLimit.maxIters dominating eigen objects are retrieved or if the ratio of the current eigenvalue to the largest eigenvalue comes down to calcLimit.epsilon threshold. The value calcLimit -> type must be CV_TERMCRIT_NUMB, CV_TERMCRIT_EPS, or CV_TERMCRIT_NUMB | CV_TERMCRIT_EPS . The function returns the real values calcLimit -> maxIter and calcLimit -> epsilon .

The function also calculates the averaged object, which must be created previously. Calculated eigen objects are arranged according to the corresponding eigenvalues in the descending order.

The parameter eigVals may be equal to NULL, if eigenvalues are not needed.

The function cvCalcEigenObjects uses the function cvCalcCovarMatrixEx.

Once you have calculated the orthonormal basis W of your data (in this case your input images), you can either project your data X into the subspace:

• Y = (X - mean) * W

Or you can reconstruct your data X from its lower-dimensional representation Y as:

• X = W*Y + mean

So the question is, which of the function does the projection and which one reconstructs? With cv::PCA it is clear. cv::PCA::project projects your data, while cv::PCA::backProject restores your data.

First let's have a look at cvEigenDecomposite. The old documentation states:

Calculates all decomposition coefficients for input object

void cvEigenDecomposite( IplImage* obj, int nEigObjs, void* eigInput, int ioFlags, void* userData, IplImage* avg, float* coeffs );

• obj: Input object.
• nEigObjs: Number of eigen objects.
• eigInput: Pointer either to the array of IplImage input objects or to the read callback function according to the value of the parameter ioFlags.
• ioFlags: Input/output flags.
• userData: Pointer to the structure that contains all necessary data for the callback functions.
• avg: Averaged object.
• coeffs: Calculated coefficients; an output parameter.

The function cvEigenDecomposite calculates all decomposition coefficients for the input object using the previously calculated eigen objects basis and the averaged object. Depending on ioFlags parameter it may be used either in direct access or callback mode.

So cvEigenDecomposite performs the mapping to the lower dimensional space, while cvEigenProjection (quoting from the documentation) restores an object using previously calculated eigen objects basis, averaged object, and decomposition coefficients of the restored object. As a reference I am again quoting from the documentation:

Calculates object projection to the eigen sub-space

void cvEigenProjection( int nEigObjs, void* eigInput, int ioFlags, void* userData, float* coeffs, IplImage* avg, IplImage* proj );

• nEigObjs: Number of eigen objects.
• eigInput: Pointer either to the array of IplImage input objects or to the read callback function according to the value of the parameter ioFlags.
• ioFlags: Input/output flags.
• userData: Pointer to the structure that contains all necessary data for the callback functions.
• coeffs: Previously calculated decomposition coefficients.
• avg: Averaged object.
• proj: Decomposed object projection to the eigen sub-space.

The function cvEigenProjection calculates an object projection to the eigen sub-space or, in other words, restores an object using previously calculated eigen objects basis, averaged object, and decomposition coefficients of the restored object. Depending on ioFlags parameter it may be used either in direct access or callback mode.

The functions of the eigen objects group have been developed to be used for any number of objects, even if their total size exceeds free RAM size. So the functions may be used in two main modes. Direct access mode is the best choice if the size of free RAM is sufficient for all input and eigen objects allocation. This mode is set if the parameter ioFlags is equal to CV_EIGOBJ_NO_CALLBACK. In this case input and output parameters are pointers to arrays of input/output objects of IplImage* type. The parameters ioBufSize and userData are not used.

I guess the names are misleading in the old OpenCV API. Let me attempt to explain it, but first give you an advice on using the OpenCV2 C++ API. I write this as a reference to people coming here from Google, so excuse if this is a lengthy answer. This may not directly apply to you using JavaCV, but for everyone else: I strongly suggest using the new OpenCV2 interface, just because the old C API is probably not supported in the future anymore.

Instead of cvCalcEigenObjects, cvEigenDecomposite and cvEigenProjection OpenCV now comes with the cv::PCA class, which makes performing a Principal Component Analysis really simple. Instead of going through it in-depth I am now pasting a sample (so people have it as a starting point) and then I'll answer your question on the old API.

This is an example to perform a Principal Component Analysis on a given set of images, in this example either with a hardcoded set of images or a CSV file. I think a modified version of this was added to the OpenCV documentation recently. By the way you'll find a source code example on using cv::PCA::project and cv::PCA::backProject in the documentation on cv::PCA:

/*
 * Copyright (c) 2012. Philipp Wagner <bytefish[at]gmx[dot]de>.
 * Released to public domain under terms of the BSD Simplified license.
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions are met:
 *   * Redistributions of source code must retain the above copyright
 *     notice, this list of conditions and the following disclaimer.
 *   * Redistributions in binary form must reproduce the above copyright
 *     notice, this list of conditions and the following disclaimer in the
 *     documentation and/or other materials provided with the distribution.
 *   * Neither the name of the organization nor the names of its contributors
 *     may be used to endorse or promote products derived from this software
 *     without specific prior written permission.
 *
 *   See <http://www.opensource.org/licenses/bsd-license>
 */

#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>

#include <fstream>
#include <sstream>

using namespace cv;
using namespace std;

// Reads the images and labels from a given CSV file, a valid file would
// look like this:
//
//      /path/to/person0/image0.jpg;0
//      /path/to/person0/image1.jpg;0
//      /path/to/person1/image0.jpg;1
//      /path/to/person1/image1.jpg;1
//      ...
//
void read_csv(const string& filename, vector<Mat>& images, vector<int>& labels) {
    std::ifstream file(filename.c_str(), ifstream::in);
    if(!file)
        throw std::exception();
    std::string line, path, classlabel;
    // For each line in the given file:
    while (std::getline(file, line)) {
        // Get the current line:
        std::stringstream liness(line);
        // Split it at the semicolon:
        std::getline(liness, path, ';');
        std::getline(liness, classlabel);
        // And push back the data into the result vectors:
        images.push_back(imread(path, IMREAD_GRAYSCALE));
        labels.push_back(atoi(classlabel.c_str()));
    }
}

// Normalizes a given image into a value range between 0 and 255.
Mat norm_0_255(const Mat& src) {
    // Create and return normalized image:
    Mat dst;
    switch(src.channels()) {
    case 1:
        cv::normalize(src, dst, 0, 255, NORM_MINMAX, CV_8UC1);
        break;
    case 3:
        cv::normalize(src, dst, 0, 255, NORM_MINMAX, CV_8UC3);
        break;
    default:
        src.copyTo(dst);
        break;
    }
    return dst;
}

// Converts the images given in src into a row matrix.
Mat asRowMatrix(const vector<Mat>& src, int rtype, double alpha = 1, double beta = 0) {
    // Number of samples:
    size_t n = src.size();
    // Return empty matrix if no matrices given:
    if(n == 0)
        return Mat();
    // dimensionality of (reshaped) samples
    size_t d = src[0].total();
    // Create resulting data matrix:
    Mat data(n, d, rtype);
    // Now copy data:
    for(int i = 0; i < n; i++) {
        //
        if(src[i].empty()) {
            string error_message = format("Image number %d was empty, please check your input data.", i);
            CV_Error(CV_StsBadArg, error_message);
        }
        // Make sure data can be reshaped, throw a meaningful exception if not!
        if(src[i].total() != d) {
            string error_message = format("Wrong number of elements in matrix #%d! Expected %d was %d.", i, d, src[i].total());
            CV_Error(CV_StsBadArg, error_message);
        }
        // Get a hold of the current row:
        Mat xi = data.row(i);
        // Make reshape happy by cloning for non-continuous matrices:
        if(src[i].isContinuous()) {
            src[i].reshape(1, 1).convertTo(xi, rtype, alpha, beta);
        } else {
            src[i].clone().reshape(1, 1).convertTo(xi, rtype, alpha, beta);
        }
    }
    return data;
}

int main(int argc, const char *argv[]) {
    // Holds some images:
    vector<Mat> db;

    // Load the greyscale images. The images in the example are
    // taken from the AT&T Facedatabase, which is publicly available
    // at:
    //
    //      http://www.cl.cam.ac.uk/research/dtg/attarchive/facedatabase.html
    //
    // This is the path to where I stored the images, yours is different!
    //
    string prefix = "/home/philipp/facerec/data/at/";

    db.push_back(imread(prefix + "s1/1.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s1/2.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s1/3.pgm", IMREAD_GRAYSCALE));

    db.push_back(imread(prefix + "s2/1.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s2/2.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s2/3.pgm", IMREAD_GRAYSCALE));

    db.push_back(imread(prefix + "s3/1.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s3/2.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s3/3.pgm", IMREAD_GRAYSCALE));

    db.push_back(imread(prefix + "s4/1.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s4/2.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s4/3.pgm", IMREAD_GRAYSCALE));

    // The following would read the images from a given CSV file
    // instead, which would look like:
    //
    //      /path/to/person0/image0.jpg;0
    //      /path/to/person0/image1.jpg;0
    //      /path/to/person1/image0.jpg;1
    //      /path/to/person1/image1.jpg;1
    //      ...
    //
    // Uncomment this to load from a CSV file:
    //

    /*
    vector<int> labels;
    read_csv("/home/philipp/facerec/data/at.txt", db, labels);
    */

    // Build a matrix with the observations in row:
    Mat data = asRowMatrix(db, CV_32FC1);

    // Number of components to keep for the PCA:
    int num_components = 10;

    // Perform a PCA:
    PCA pca(data, Mat(), CV_PCA_DATA_AS_ROW, num_components);

    // And copy the PCA results:
    Mat mean = pca.mean.clone();
    Mat eigenvalues = pca.eigenvalues.clone();
    Mat eigenvectors = pca.eigenvectors.clone();

    // The mean face:
    imshow("avg", norm_0_255(mean.reshape(1, db[0].rows)));

    // The first three eigenfaces:
    imshow("pc1", norm_0_255(pca.eigenvectors.row(0)).reshape(1, db[0].rows));
    imshow("pc2", norm_0_255(pca.eigenvectors.row(1)).reshape(1, db[0].rows));
    imshow("pc3", norm_0_255(pca.eigenvectors.row(2)).reshape(1, db[0].rows));

    // Show the images:
    waitKey(0);

    // Success!
    return 0;
}

So let's have a look how to map cv::PCA to the old OpenCV API. First of all you would need to find the orthonormal basis W. This is done by using CalcEigenObjects, quoting the documentation:

Calculates orthonormal eigen basis and averaged object for group of input objects

void cvCalcEigenObjects( int nObjects, void* input, void* output, int ioFlags, int ioBufSize, void* userData, CvTermCriteria* calcLimit, IplImage* avg, float* eigVals );

• nObjects: Number of source objects.
• input: Pointer either to the array of IplImage input objects or to the read callback function according to the value of the parameter ioFlags.
• output: Pointer either to the array of eigen objects or to the write callback function according to the value of the parameter ioFlags .
• ioFlags: Input/output flags.
• ioBufSize: Input/output buffer size in bytes. The size is zero, if unknown.
• userData: Pointer to the structure that contains all necessary data for the callback functions.
• calcLimit: Criteria that determine when to stop calculation of eigen objects.
• avg: Averaged object.
• eigVals: Pointer to the eigenvalues array in the descending order; may be NULL .

The function cvCalcEigenObjects calculates orthonormal eigen basis and the averaged object for a group of the input objects. Depending on ioFlags parameter it may be used either in direct access or callback mode. Depending on the parameter calcLimit, calculations are finished either after first calcLimit.maxIters dominating eigen objects are retrieved or if the ratio of the current eigenvalue to the largest eigenvalue comes down to calcLimit.epsilon threshold. The value calcLimit -> type must be CV_TERMCRIT_NUMB, CV_TERMCRIT_EPS, or CV_TERMCRIT_NUMB | CV_TERMCRIT_EPS . The function returns the real values calcLimit -> maxIter and calcLimit -> epsilon .

The function also calculates the averaged object, which must be created previously. Calculated eigen objects are arranged according to the corresponding eigenvalues in the descending order.

The parameter eigVals may be equal to NULL, if eigenvalues are not needed.

The function cvCalcEigenObjects uses the function cvCalcCovarMatrixEx.

Once you have calculated the orthonormal basis W of your data (in this case your input images), you can either project your data X into the subspace:

• Y = (X - mean) * W

Or you can reconstruct your data X from its lower-dimensional representation Y as:

• X = W*Y + mean

So the question is, which of the function does the projection and which one reconstructs? With cv::PCA it is clear. cv::PCA::project projects your data, while cv::PCA::backProject restores your data.

First let's have a look at cvEigenDecomposite. The old documentation states:

Calculates all decomposition coefficients for input object

void cvEigenDecomposite( IplImage* obj, int nEigObjs, void* eigInput, int ioFlags, void* userData, IplImage* avg, float* coeffs );

• obj: Input object.
• nEigObjs: Number of eigen objects.
• eigInput: Pointer either to the array of IplImage input objects or to the read callback function according to the value of the parameter ioFlags.
• ioFlags: Input/output flags.
• userData: Pointer to the structure that contains all necessary data for the callback functions.
• avg: Averaged object.
• coeffs: Calculated coefficients; an output parameter.

The function cvEigenDecomposite calculates all decomposition coefficients for the input object using the previously calculated eigen objects basis and the averaged object. Depending on ioFlags parameter it may be used either in direct access or callback mode.

So cvEigenDecomposite performs the mapping to the lower dimensional space, while cvEigenProjection (quoting from the documentation) restores an object using previously calculated eigen objects basis, averaged object, and decomposition coefficients of the restored object. As a reference I am again quoting from the documentation:

Calculates object projection to the eigen sub-space

void cvEigenProjection( int nEigObjs, void* eigInput, int ioFlags, void* userData, float* coeffs, IplImage* avg, IplImage* proj );

• nEigObjs: Number of eigen objects.
• eigInput: Pointer either to the array of IplImage input objects or to the read callback function according to the value of the parameter ioFlags.
• ioFlags: Input/output flags.
• userData: Pointer to the structure that contains all necessary data for the callback functions.
• coeffs: Previously calculated decomposition coefficients.
• avg: Averaged object.
• proj: Decomposed object projection to the eigen sub-space.

The function cvEigenProjection calculates an object projection to the eigen sub-space or, in other words, restores an object using previously calculated eigen objects basis, averaged object, and decomposition coefficients of the restored object. Depending on ioFlags parameter it may be used either in direct access or callback mode.

The functions of the eigen objects group have been developed to be used for any number of objects, even if their total size exceeds free RAM size. So the functions may be used in two main modes. Direct access mode is the best choice if the size of free RAM is sufficient for all input and eigen objects allocation. This mode is set if the parameter ioFlags is equal to CV_EIGOBJ_NO_CALLBACK. In this case input and output parameters are pointers to arrays of input/output objects of IplImage* type. The parameters ioBufSize and userData are not used.

I guess the names are misleading in the old OpenCV API. Let me attempt to explain it, but first give you an advice on using the OpenCV2 C++ API. I write this as a reference to people coming here from Google, so excuse if this is a lengthy answer. This may not directly apply to you using JavaCV, but for everyone else: I strongly suggest using the new OpenCV2 interface, just because the old C API is probably not supported in the future anymore.

Instead of cvCalcEigenObjects, cvEigenDecomposite and cvEigenProjection OpenCV now comes with the cv::PCA class, which makes performing a Principal Component Analysis really simple. Instead of going through it in-depth I am now pasting a sample (so people have it as a starting point) and then I'll answer your question on the old API.

This is an example to perform a Principal Component Analysis on a given set of images, in this example either with a hardcoded set of images or a CSV file. I think a modified version of this was added to the OpenCV documentation recently. By the way you'll find a source code example on using cv::PCA::project and cv::PCA::backProject in the documentation on cv::PCA:

/*
 * Copyright (c) 2012. Philipp Wagner <bytefish[at]gmx[dot]de>.
 * Released to public domain under terms of the BSD Simplified license.
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions are met:
 *   * Redistributions of source code must retain the above copyright
 *     notice, this list of conditions and the following disclaimer.
 *   * Redistributions in binary form must reproduce the above copyright
 *     notice, this list of conditions and the following disclaimer in the
 *     documentation and/or other materials provided with the distribution.
 *   * Neither the name of the organization nor the names of its contributors
 *     may be used to endorse or promote products derived from this software
 *     without specific prior written permission.
 *
 *   See <http://www.opensource.org/licenses/bsd-license>
 */

#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>

#include <fstream>
#include <sstream>

using namespace cv;
using namespace std;

// Reads the images and labels from a given CSV file, a valid file would
// look like this:
//
//      /path/to/person0/image0.jpg;0
//      /path/to/person0/image1.jpg;0
//      /path/to/person1/image0.jpg;1
//      /path/to/person1/image1.jpg;1
//      ...
//
void read_csv(const string& filename, vector<Mat>& images, vector<int>& labels) {
    std::ifstream file(filename.c_str(), ifstream::in);
    if(!file)
        throw std::exception();
    std::string line, path, classlabel;
    // For each line in the given file:
    while (std::getline(file, line)) {
        // Get the current line:
        std::stringstream liness(line);
        // Split it at the semicolon:
        std::getline(liness, path, ';');
        std::getline(liness, classlabel);
        // And push back the data into the result vectors:
        images.push_back(imread(path, IMREAD_GRAYSCALE));
        labels.push_back(atoi(classlabel.c_str()));
    }
}

// Normalizes a given image into a value range between 0 and 255.
Mat norm_0_255(const Mat& src) {
    // Create and return normalized image:
    Mat dst;
    switch(src.channels()) {
    case 1:
        cv::normalize(src, dst, 0, 255, NORM_MINMAX, CV_8UC1);
        break;
    case 3:
        cv::normalize(src, dst, 0, 255, NORM_MINMAX, CV_8UC3);
        break;
    default:
        src.copyTo(dst);
        break;
    }
    return dst;
}

// Converts the images given in src into a row matrix.
Mat asRowMatrix(const vector<Mat>& src, int rtype, double alpha = 1, double beta = 0) {
    // Number of samples:
    size_t n = src.size();
    // Return empty matrix if no matrices given:
    if(n == 0)
        return Mat();
    // dimensionality of (reshaped) samples
    size_t d = src[0].total();
    // Create resulting data matrix:
    Mat data(n, d, rtype);
    // Now copy data:
    for(int i = 0; i < n; i++) {
        //
        if(src[i].empty()) {
            string error_message = format("Image number %d was empty, please check your input data.", i);
            CV_Error(CV_StsBadArg, error_message);
        }
        // Make sure data can be reshaped, throw a meaningful exception if not!
        if(src[i].total() != d) {
            string error_message = format("Wrong number of elements in matrix #%d! Expected %d was %d.", i, d, src[i].total());
            CV_Error(CV_StsBadArg, error_message);
        }
        // Get a hold of the current row:
        Mat xi = data.row(i);
        // Make reshape happy by cloning for non-continuous matrices:
        if(src[i].isContinuous()) {
            src[i].reshape(1, 1).convertTo(xi, rtype, alpha, beta);
        } else {
            src[i].clone().reshape(1, 1).convertTo(xi, rtype, alpha, beta);
        }
    }
    return data;
}

int main(int argc, const char *argv[]) {
    // Holds some images:
    vector<Mat> db;

    // Load the greyscale images. The images in the example are
    // taken from the AT&T Facedatabase, which is publicly available
    // at:
    //
    //      http://www.cl.cam.ac.uk/research/dtg/attarchive/facedatabase.html
    //
    // This is the path to where I stored the images, yours is different!
    //
    string prefix = "/home/philipp/facerec/data/at/";

    db.push_back(imread(prefix + "s1/1.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s1/2.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s1/3.pgm", IMREAD_GRAYSCALE));

    db.push_back(imread(prefix + "s2/1.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s2/2.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s2/3.pgm", IMREAD_GRAYSCALE));

    db.push_back(imread(prefix + "s3/1.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s3/2.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s3/3.pgm", IMREAD_GRAYSCALE));

    db.push_back(imread(prefix + "s4/1.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s4/2.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s4/3.pgm", IMREAD_GRAYSCALE));

    // The following would read the images from a given CSV file
    // instead, which would look like:
    //
    //      /path/to/person0/image0.jpg;0
    //      /path/to/person0/image1.jpg;0
    //      /path/to/person1/image0.jpg;1
    //      /path/to/person1/image1.jpg;1
    //      ...
    //
    // Uncomment this to load from a CSV file:
    //

    /*
    vector<int> labels;
    read_csv("/home/philipp/facerec/data/at.txt", db, labels);
    */

    // Build a matrix with the observations in row:
    Mat data = asRowMatrix(db, CV_32FC1);

    // Number of components to keep for the PCA:
    int num_components = 10;

    // Perform a PCA:
    PCA pca(data, Mat(), CV_PCA_DATA_AS_ROW, num_components);

    // And copy the PCA results:
    Mat mean = pca.mean.clone();
    Mat eigenvalues = pca.eigenvalues.clone();
    Mat eigenvectors = pca.eigenvectors.clone();

    // The mean face:
    imshow("avg", norm_0_255(mean.reshape(1, db[0].rows)));

    // The first three eigenfaces:
    imshow("pc1", norm_0_255(pca.eigenvectors.row(0)).reshape(1, db[0].rows));
    imshow("pc2", norm_0_255(pca.eigenvectors.row(1)).reshape(1, db[0].rows));
    imshow("pc3", norm_0_255(pca.eigenvectors.row(2)).reshape(1, db[0].rows));

    // Show the images:
    waitKey(0);

    // Success!
    return 0;
}

So let's have a look how to map cv::PCA to the old OpenCV API. First of all you would need to find the orthonormal basis W. This is done by using CalcEigenObjects, quoting the documentation:

Calculates orthonormal eigen basis and averaged object for group of input objects

void cvCalcEigenObjects( int nObjects, void* input, void* output, int ioFlags, int ioBufSize, void* userData, CvTermCriteria* calcLimit, IplImage* avg, float* eigVals );

• nObjects: Number of source objects.
• input: Pointer either to the array of IplImage input objects or to the read callback function according to the value of the parameter ioFlags.
• output: Pointer either to the array of eigen objects or to the write callback function according to the value of the parameter ioFlags .
• ioFlags: Input/output flags.
• ioBufSize: Input/output buffer size in bytes. The size is zero, if unknown.
• userData: Pointer to the structure that contains all necessary data for the callback functions.
• calcLimit: Criteria that determine when to stop calculation of eigen objects.
• avg: Averaged object.
• eigVals: Pointer to the eigenvalues array in the descending order; may be NULL .

The function cvCalcEigenObjects calculates orthonormal eigen basis and the averaged object for a group of the input objects. Depending on ioFlags parameter it may be used either in direct access or callback mode. Depending on the parameter calcLimit, calculations are finished either after first calcLimit.maxIters dominating eigen objects are retrieved or if the ratio of the current eigenvalue to the largest eigenvalue comes down to calcLimit.epsilon threshold. The value calcLimit -> type must be CV_TERMCRIT_NUMB, CV_TERMCRIT_EPS, or CV_TERMCRIT_NUMB | CV_TERMCRIT_EPS . The function returns the real values calcLimit -> maxIter and calcLimit -> epsilon .

The function also calculates the averaged object, which must be created previously. Calculated eigen objects are arranged according to the corresponding eigenvalues in the descending order.

The parameter eigVals may be equal to NULL, if eigenvalues are not needed.

The function cvCalcEigenObjects uses the function cvCalcCovarMatrixEx.

Once you have calculated the orthonormal basis W of your data (in this case your input images), you can either project your data X into the subspace:

• Y = (X - mean) * W

Or you can reconstruct your data X from its lower-dimensional representation Y as:

• X = W*Y + mean

So the question is, which of the function does the projection and which one reconstructs? With cv::PCA it is clear. cv::PCA::project projects your data, while cv::PCA::backProject restores your data.

First let's have a look at cvEigenDecomposite. The old documentation states:

Calculates all decomposition coefficients for input object

void cvEigenDecomposite( IplImage* obj, int nEigObjs, void* eigInput, int ioFlags, void* userData, IplImage* avg, float* coeffs );

• obj: Input object.
• nEigObjs: Number of eigen objects.
• eigInput: Pointer either to the array of IplImage input objects or to the read callback function according to the value of the parameter ioFlags.
• ioFlags: Input/output flags.
• userData: Pointer to the structure that contains all necessary data for the callback functions.
• avg: Averaged object.
• coeffs: Calculated coefficients; an output parameter.

The function cvEigenDecomposite calculates all decomposition coefficients for the input object using the previously calculated eigen objects basis and the averaged object. Depending on ioFlags parameter it may be used either in direct access or callback mode.

So cvEigenDecomposite performs the mapping to the lower dimensional space, while cvEigenProjection (quoting from the documentation) restores an object using previously calculated eigen objects basis, averaged object, and decomposition coefficients of the restored object. As a reference I am again quoting from the documentation:

Calculates object projection to the eigen sub-space

void cvEigenProjection( int nEigObjs, void* eigInput, int ioFlags, void* userData, float* coeffs, IplImage* avg, IplImage* proj );

• nEigObjs: Number of eigen objects.
• eigInput: Pointer either to the array of IplImage input objects or to the read callback function according to the value of the parameter ioFlags.
• ioFlags: Input/output flags.
• userData: Pointer to the structure that contains all necessary data for the callback functions.
• coeffs: Previously calculated decomposition coefficients.
• avg: Averaged object.
• proj: Decomposed object projection to the eigen sub-space.

The function cvEigenProjection calculates an object projection to the eigen sub-space or, in other words, restores an object using previously calculated eigen objects basis, averaged object, and decomposition coefficients of the restored object. Depending on ioFlags parameter it may be used either in direct access or callback mode.

The functions of the eigen objects group have been developed to be used for any number of objects, even if their total size exceeds free RAM size. So the functions may be used in two main modes. Direct access mode is the best choice if the size of free RAM is sufficient for all input and eigen objects allocation. This mode is set if the parameter ioFlags is equal to CV_EIGOBJ_NO_CALLBACK. In this case input and output parameters are pointers to arrays of input/output objects of IplImage* type. The parameters ioBufSize and userData are not used.

For an in-depth explanation of how such a system might work with the mentioned functions, I just found this nice github page:

• https://github.com/octavian-sima/Face-Recognition/wiki/Face-Recognition#training

I guess the names are misleading in the old OpenCV API. Let me attempt to explain it, but first give you an advice on using the OpenCV2 C++ API. I write this as a reference to people coming here from Google, so excuse if this is a lengthy answer. This may not directly apply to you using JavaCV, but for everyone else: I strongly suggest using the new OpenCV2 interface, just because the old C API is probably not supported in the future anymore.

Instead of cvCalcEigenObjects, cvEigenDecomposite and cvEigenProjection OpenCV now comes with the cv::PCA class, which makes performing a Principal Component Analysis really simple. Instead of going through it in-depth I am now pasting a sample (so people have it as a starting point) and then I'll answer your question on the old API.

This is an example to perform a Principal Component Analysis on a given set of images, in this example either with a hardcoded set of images or a CSV file. I think a modified version of this was added to the OpenCV documentation recently. By the way you'll find a source code example on using cv::PCA::project and cv::PCA::backProject in the documentation on cv::PCA:

/*
 * Copyright (c) 2012. Philipp Wagner <bytefish[at]gmx[dot]de>.
 * Released to public domain under terms of the BSD Simplified license.
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions are met:
 *   * Redistributions of source code must retain the above copyright
 *     notice, this list of conditions and the following disclaimer.
 *   * Redistributions in binary form must reproduce the above copyright
 *     notice, this list of conditions and the following disclaimer in the
 *     documentation and/or other materials provided with the distribution.
 *   * Neither the name of the organization nor the names of its contributors
 *     may be used to endorse or promote products derived from this software
 *     without specific prior written permission.
 *
 *   See <http://www.opensource.org/licenses/bsd-license>
 */

#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>

#include <fstream>
#include <sstream>

using namespace cv;
using namespace std;

// Reads the images and labels from a given CSV file, a valid file would
// look like this:
//
//      /path/to/person0/image0.jpg;0
//      /path/to/person0/image1.jpg;0
//      /path/to/person1/image0.jpg;1
//      /path/to/person1/image1.jpg;1
//      ...
//
void read_csv(const string& filename, vector<Mat>& images, vector<int>& labels) {
    std::ifstream file(filename.c_str(), ifstream::in);
    if(!file)
        throw std::exception();
    std::string line, path, classlabel;
    // For each line in the given file:
    while (std::getline(file, line)) {
        // Get the current line:
        std::stringstream liness(line);
        // Split it at the semicolon:
        std::getline(liness, path, ';');
        std::getline(liness, classlabel);
        // And push back the data into the result vectors:
        images.push_back(imread(path, IMREAD_GRAYSCALE));
        labels.push_back(atoi(classlabel.c_str()));
    }
}

// Normalizes a given image into a value range between 0 and 255.
Mat norm_0_255(const Mat& src) {
    // Create and return normalized image:
    Mat dst;
    switch(src.channels()) {
    case 1:
        cv::normalize(src, dst, 0, 255, NORM_MINMAX, CV_8UC1);
        break;
    case 3:
        cv::normalize(src, dst, 0, 255, NORM_MINMAX, CV_8UC3);
        break;
    default:
        src.copyTo(dst);
        break;
    }
    return dst;
}

// Converts the images given in src into a row matrix.
Mat asRowMatrix(const vector<Mat>& src, int rtype, double alpha = 1, double beta = 0) {
    // Number of samples:
    size_t n = src.size();
    // Return empty matrix if no matrices given:
    if(n == 0)
        return Mat();
    // dimensionality of (reshaped) samples
    size_t d = src[0].total();
    // Create resulting data matrix:
    Mat data(n, d, rtype);
    // Now copy data:
    for(int i = 0; i < n; i++) {
        //
        if(src[i].empty()) {
            string error_message = format("Image number %d was empty, please check your input data.", i);
            CV_Error(CV_StsBadArg, error_message);
        }
        // Make sure data can be reshaped, throw a meaningful exception if not!
        if(src[i].total() != d) {
            string error_message = format("Wrong number of elements in matrix #%d! Expected %d was %d.", i, d, src[i].total());
            CV_Error(CV_StsBadArg, error_message);
        }
        // Get a hold of the current row:
        Mat xi = data.row(i);
        // Make reshape happy by cloning for non-continuous matrices:
        if(src[i].isContinuous()) {
            src[i].reshape(1, 1).convertTo(xi, rtype, alpha, beta);
        } else {
            src[i].clone().reshape(1, 1).convertTo(xi, rtype, alpha, beta);
        }
    }
    return data;
}

int main(int argc, const char *argv[]) {
    // Holds some images:
    vector<Mat> db;

    // Load the greyscale images. The images in the example are
    // taken from the AT&T Facedatabase, which is publicly available
    // at:
    //
    //      http://www.cl.cam.ac.uk/research/dtg/attarchive/facedatabase.html
    //
    // This is the path to where I stored the images, yours is different!
    //
    string prefix = "/home/philipp/facerec/data/at/";

    db.push_back(imread(prefix + "s1/1.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s1/2.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s1/3.pgm", IMREAD_GRAYSCALE));

    db.push_back(imread(prefix + "s2/1.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s2/2.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s2/3.pgm", IMREAD_GRAYSCALE));

    db.push_back(imread(prefix + "s3/1.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s3/2.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s3/3.pgm", IMREAD_GRAYSCALE));

    db.push_back(imread(prefix + "s4/1.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s4/2.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s4/3.pgm", IMREAD_GRAYSCALE));

    // The following would read the images from a given CSV file
    // instead, which would look like:
    //
    //      /path/to/person0/image0.jpg;0
    //      /path/to/person0/image1.jpg;0
    //      /path/to/person1/image0.jpg;1
    //      /path/to/person1/image1.jpg;1
    //      ...
    //
    // Uncomment this to load from a CSV file:
    //

    /*
    vector<int> labels;
    read_csv("/home/philipp/facerec/data/at.txt", db, labels);
    */

    // Build a matrix with the observations in row:
    Mat data = asRowMatrix(db, CV_32FC1);

    // Number of components to keep for the PCA:
    int num_components = 10;

    // Perform a PCA:
    PCA pca(data, Mat(), CV_PCA_DATA_AS_ROW, num_components);

    // And copy the PCA results:
    Mat mean = pca.mean.clone();
    Mat eigenvalues = pca.eigenvalues.clone();
    Mat eigenvectors = pca.eigenvectors.clone();

    // The mean face:
    imshow("avg", norm_0_255(mean.reshape(1, db[0].rows)));

    // The first three eigenfaces:
    imshow("pc1", norm_0_255(pca.eigenvectors.row(0)).reshape(1, db[0].rows));
    imshow("pc2", norm_0_255(pca.eigenvectors.row(1)).reshape(1, db[0].rows));
    imshow("pc3", norm_0_255(pca.eigenvectors.row(2)).reshape(1, db[0].rows));

    // Show the images:
    waitKey(0);

    // Success!
    return 0;
}

So let's have a look how to map cv::PCA to the old OpenCV API. First of all you would need to find the orthonormal basis W. This is done by using CalcEigenObjectscvCalcEigenObjects, quoting the documentation:

Calculates orthonormal eigen basis and averaged object for group of input objects

void cvCalcEigenObjects( int nObjects, void* input, void* output, int ioFlags, int ioBufSize, void* userData, CvTermCriteria* calcLimit, IplImage* avg, float* eigVals );

• nObjects: Number of source objects.
• input: Pointer either to the array of IplImage input objects or to the read callback function according to the value of the parameter ioFlags.
• output: Pointer either to the array of eigen objects or to the write callback function according to the value of the parameter ioFlags .
• ioFlags: Input/output flags.
• ioBufSize: Input/output buffer size in bytes. The size is zero, if unknown.
• userData: Pointer to the structure that contains all necessary data for the callback functions.
• calcLimit: Criteria that determine when to stop calculation of eigen objects.
• avg: Averaged object.
• eigVals: Pointer to the eigenvalues array in the descending order; may be NULL .

The function cvCalcEigenObjects calculates orthonormal eigen basis and the averaged object for a group of the input objects. Depending on ioFlags parameter it may be used either in direct access or callback mode. Depending on the parameter calcLimit, calculations are finished either after first calcLimit.maxIters dominating eigen objects are retrieved or if the ratio of the current eigenvalue to the largest eigenvalue comes down to calcLimit.epsilon threshold. The value calcLimit -> type must be CV_TERMCRIT_NUMB, CV_TERMCRIT_EPS, or CV_TERMCRIT_NUMB | CV_TERMCRIT_EPS . The function returns the real values calcLimit -> maxIter and calcLimit -> epsilon .

The function also calculates the averaged object, which must be created previously. Calculated eigen objects are arranged according to the corresponding eigenvalues in the descending order.

The parameter eigVals may be equal to NULL, if eigenvalues are not needed.

The function cvCalcEigenObjects uses the function cvCalcCovarMatrixEx.

Once you have calculated the orthonormal basis W of your data (in this case your input images), you can either project your data X into the subspace:

• Y = (X - mean) * W

Or you can reconstruct your data X from its lower-dimensional representation Y as:

• X = W*Y + mean

So the question is, which of the function does the projection and which one reconstructs? With cv::PCA it is clear. cv::PCA::project projects your data, while cv::PCA::backProject restores your data.

First let's have a look at cvEigenDecomposite. The old documentation states:

Calculates all decomposition coefficients for input object

void cvEigenDecomposite( IplImage* obj, int nEigObjs, void* eigInput, int ioFlags, void* userData, IplImage* avg, float* coeffs );

• obj: Input object.
• nEigObjs: Number of eigen objects.
• eigInput: Pointer either to the array of IplImage input objects or to the read callback function according to the value of the parameter ioFlags.
• ioFlags: Input/output flags.
• userData: Pointer to the structure that contains all necessary data for the callback functions.
• avg: Averaged object.
• coeffs: Calculated coefficients; an output parameter.

The function cvEigenDecomposite calculates all decomposition coefficients for the input object using the previously calculated eigen objects basis and the averaged object. Depending on ioFlags parameter it may be used either in direct access or callback mode.

So cvEigenDecomposite performs the mapping to the lower dimensional space, while cvEigenProjection (quoting from the documentation) restores an object using previously calculated eigen objects basis, averaged object, and decomposition coefficients of the restored object. As a reference I am again quoting from the documentation:

Calculates object projection to the eigen sub-space

void cvEigenProjection( int nEigObjs, void* eigInput, int ioFlags, void* userData, float* coeffs, IplImage* avg, IplImage* proj );

• nEigObjs: Number of eigen objects.
• eigInput: Pointer either to the array of IplImage input objects or to the read callback function according to the value of the parameter ioFlags.
• ioFlags: Input/output flags.
• userData: Pointer to the structure that contains all necessary data for the callback functions.
• coeffs: Previously calculated decomposition coefficients.
• avg: Averaged object.
• proj: Decomposed object projection to the eigen sub-space.

The function cvEigenProjection calculates an object projection to the eigen sub-space or, in other words, restores an object using previously calculated eigen objects basis, averaged object, and decomposition coefficients of the restored object. Depending on ioFlags parameter it may be used either in direct access or callback mode.

The functions of the eigen objects group have been developed to be used for any number of objects, even if their total size exceeds free RAM size. So the functions may be used in two main modes. Direct access mode is the best choice if the size of free RAM is sufficient for all input and eigen objects allocation. This mode is set if the parameter ioFlags is equal to CV_EIGOBJ_NO_CALLBACK. In this case input and output parameters are pointers to arrays of input/output objects of IplImage* type. The parameters ioBufSize and userData are not used.

For an in-depth explanation of how such a system might work with the mentioned functions, I just found this nice github page:

• https://github.com/octavian-sima/Face-Recognition/wiki/Face-Recognition#training

Although I still recommend using the new API.

As far as I know Samuel Audet (the JavaCV lead dev) has written a wrapper for the cv::FaceRecognizer:

• opencv_contrib.java

You can find an example on how to use it in Petter Christian Bjelland blog:

• http://pcbje.com/2012/12/doing-face-recognition-with-javacv/

Now to your original question. I guess the names are misleading in the old OpenCV API. Let me attempt to explain it, but first give you an advice on using the OpenCV2 C++ API. I write this as a reference to people coming here from Google, so excuse if this is a lengthy answer. This may not directly apply to you using JavaCV, but for everyone else: I strongly suggest using the new OpenCV2 interface, just because the old C API is probably not supported in the future anymore. anymore.

Instead of cvCalcEigenObjects, cvEigenDecomposite and cvEigenProjection OpenCV now comes with the cv::PCA class, which makes performing a Principal Component Analysis really simple. Instead of going through it in-depth I am now pasting a sample (so people have it as a starting point) and then I'll answer your question on the old API.

This is an example to perform a Principal Component Analysis on a given set of images, in this example either with a hardcoded set of images or a CSV file. I think a modified version of this was added to the OpenCV documentation recently. By the way you'll find a source code example on using cv::PCA::project and cv::PCA::backProject in the documentation on cv::PCA:

/*
 * Copyright (c) 2012. Philipp Wagner <bytefish[at]gmx[dot]de>.
 * Released to public domain under terms of the BSD Simplified license.
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions are met:
 *   * Redistributions of source code must retain the above copyright
 *     notice, this list of conditions and the following disclaimer.
 *   * Redistributions in binary form must reproduce the above copyright
 *     notice, this list of conditions and the following disclaimer in the
 *     documentation and/or other materials provided with the distribution.
 *   * Neither the name of the organization nor the names of its contributors
 *     may be used to endorse or promote products derived from this software
 *     without specific prior written permission.
 *
 *   See <http://www.opensource.org/licenses/bsd-license>
 */

#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>

#include <fstream>
#include <sstream>

using namespace cv;
using namespace std;

// Reads the images and labels from a given CSV file, a valid file would
// look like this:
//
//      /path/to/person0/image0.jpg;0
//      /path/to/person0/image1.jpg;0
//      /path/to/person1/image0.jpg;1
//      /path/to/person1/image1.jpg;1
//      ...
//
void read_csv(const string& filename, vector<Mat>& images, vector<int>& labels) {
    std::ifstream file(filename.c_str(), ifstream::in);
    if(!file)
        throw std::exception();
    std::string line, path, classlabel;
    // For each line in the given file:
    while (std::getline(file, line)) {
        // Get the current line:
        std::stringstream liness(line);
        // Split it at the semicolon:
        std::getline(liness, path, ';');
        std::getline(liness, classlabel);
        // And push back the data into the result vectors:
        images.push_back(imread(path, IMREAD_GRAYSCALE));
        labels.push_back(atoi(classlabel.c_str()));
    }
}

// Normalizes a given image into a value range between 0 and 255.
Mat norm_0_255(const Mat& src) {
    // Create and return normalized image:
    Mat dst;
    switch(src.channels()) {
    case 1:
        cv::normalize(src, dst, 0, 255, NORM_MINMAX, CV_8UC1);
        break;
    case 3:
        cv::normalize(src, dst, 0, 255, NORM_MINMAX, CV_8UC3);
        break;
    default:
        src.copyTo(dst);
        break;
    }
    return dst;
}

// Converts the images given in src into a row matrix.
Mat asRowMatrix(const vector<Mat>& src, int rtype, double alpha = 1, double beta = 0) {
    // Number of samples:
    size_t n = src.size();
    // Return empty matrix if no matrices given:
    if(n == 0)
        return Mat();
    // dimensionality of (reshaped) samples
    size_t d = src[0].total();
    // Create resulting data matrix:
    Mat data(n, d, rtype);
    // Now copy data:
    for(int i = 0; i < n; i++) {
        //
        if(src[i].empty()) {
            string error_message = format("Image number %d was empty, please check your input data.", i);
            CV_Error(CV_StsBadArg, error_message);
        }
        // Make sure data can be reshaped, throw a meaningful exception if not!
        if(src[i].total() != d) {
            string error_message = format("Wrong number of elements in matrix #%d! Expected %d was %d.", i, d, src[i].total());
            CV_Error(CV_StsBadArg, error_message);
        }
        // Get a hold of the current row:
        Mat xi = data.row(i);
        // Make reshape happy by cloning for non-continuous matrices:
        if(src[i].isContinuous()) {
            src[i].reshape(1, 1).convertTo(xi, rtype, alpha, beta);
        } else {
            src[i].clone().reshape(1, 1).convertTo(xi, rtype, alpha, beta);
        }
    }
    return data;
}

int main(int argc, const char *argv[]) {
    // Holds some images:
    vector<Mat> db;

    // Load the greyscale images. The images in the example are
    // taken from the AT&T Facedatabase, which is publicly available
    // at:
    //
    //      http://www.cl.cam.ac.uk/research/dtg/attarchive/facedatabase.html
    //
    // This is the path to where I stored the images, yours is different!
    //
    string prefix = "/home/philipp/facerec/data/at/";

    db.push_back(imread(prefix + "s1/1.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s1/2.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s1/3.pgm", IMREAD_GRAYSCALE));

    db.push_back(imread(prefix + "s2/1.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s2/2.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s2/3.pgm", IMREAD_GRAYSCALE));

    db.push_back(imread(prefix + "s3/1.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s3/2.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s3/3.pgm", IMREAD_GRAYSCALE));

    db.push_back(imread(prefix + "s4/1.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s4/2.pgm", IMREAD_GRAYSCALE));
    db.push_back(imread(prefix + "s4/3.pgm", IMREAD_GRAYSCALE));

    // The following would read the images from a given CSV file
    // instead, which would look like:
    //
    //      /path/to/person0/image0.jpg;0
    //      /path/to/person0/image1.jpg;0
    //      /path/to/person1/image0.jpg;1
    //      /path/to/person1/image1.jpg;1
    //      ...
    //
    // Uncomment this to load from a CSV file:
    //

    /*
    vector<int> labels;
    read_csv("/home/philipp/facerec/data/at.txt", db, labels);
    */

    // Build a matrix with the observations in row:
    Mat data = asRowMatrix(db, CV_32FC1);

    // Number of components to keep for the PCA:
    int num_components = 10;

    // Perform a PCA:
    PCA pca(data, Mat(), CV_PCA_DATA_AS_ROW, num_components);

    // And copy the PCA results:
    Mat mean = pca.mean.clone();
    Mat eigenvalues = pca.eigenvalues.clone();
    Mat eigenvectors = pca.eigenvectors.clone();

    // The mean face:
    imshow("avg", norm_0_255(mean.reshape(1, db[0].rows)));

    // The first three eigenfaces:
    imshow("pc1", norm_0_255(pca.eigenvectors.row(0)).reshape(1, db[0].rows));
    imshow("pc2", norm_0_255(pca.eigenvectors.row(1)).reshape(1, db[0].rows));
    imshow("pc3", norm_0_255(pca.eigenvectors.row(2)).reshape(1, db[0].rows));

    // Show the images:
    waitKey(0);

    // Success!
    return 0;
}

So let's have a look how to map cv::PCA to the old OpenCV API. First of all you would need to find the orthonormal basis W. This is done by using cvCalcEigenObjects, quoting the documentation:

Calculates orthonormal eigen basis and averaged object for group of input objects

void cvCalcEigenObjects( int nObjects, void* input, void* output, int ioFlags, int ioBufSize, void* userData, CvTermCriteria* calcLimit, IplImage* avg, float* eigVals );

• nObjects: Number of source objects.
• input: Pointer either to the array of IplImage input objects or to the read callback function according to the value of the parameter ioFlags.
• output: Pointer either to the array of eigen objects or to the write callback function according to the value of the parameter ioFlags .
• ioFlags: Input/output flags.
• ioBufSize: Input/output buffer size in bytes. The size is zero, if unknown.
• userData: Pointer to the structure that contains all necessary data for the callback functions.
• calcLimit: Criteria that determine when to stop calculation of eigen objects.
• avg: Averaged object.
• eigVals: Pointer to the eigenvalues array in the descending order; may be NULL .

The function cvCalcEigenObjects calculates orthonormal eigen basis and the averaged object for a group of the input objects. Depending on ioFlags parameter it may be used either in direct access or callback mode. Depending on the parameter calcLimit, calculations are finished either after first calcLimit.maxIters dominating eigen objects are retrieved or if the ratio of the current eigenvalue to the largest eigenvalue comes down to calcLimit.epsilon threshold. The value calcLimit -> type must be CV_TERMCRIT_NUMB, CV_TERMCRIT_EPS, or CV_TERMCRIT_NUMB | CV_TERMCRIT_EPS . The function returns the real values calcLimit -> maxIter and calcLimit -> epsilon .

The function also calculates the averaged object, which must be created previously. Calculated eigen objects are arranged according to the corresponding eigenvalues in the descending order.

The parameter eigVals may be equal to NULL, if eigenvalues are not needed.

The function cvCalcEigenObjects uses the function cvCalcCovarMatrixEx.

Once you have calculated the orthonormal basis W of your data (in this case your input images), you can either project your data X into the subspace:

• Y = (X - mean) * W

Or you can reconstruct your data X from its lower-dimensional representation Y as:

• X = W*Y + mean

So the question is, which of the function does the projection and which one reconstructs? With cv::PCA it is clear. cv::PCA::project projects your data, while cv::PCA::backProject restores your data.

First let's have a look at cvEigenDecomposite. The old documentation states:

Calculates all decomposition coefficients for input object

void cvEigenDecomposite( IplImage* obj, int nEigObjs, void* eigInput, int ioFlags, void* userData, IplImage* avg, float* coeffs );

• obj: Input object.
• nEigObjs: Number of eigen objects.
• eigInput: Pointer either to the array of IplImage input objects or to the read callback function according to the value of the parameter ioFlags.
• ioFlags: Input/output flags.
• userData: Pointer to the structure that contains all necessary data for the callback functions.
• avg: Averaged object.
• coeffs: Calculated coefficients; an output parameter.

The function cvEigenDecomposite calculates all decomposition coefficients for the input object using the previously calculated eigen objects basis and the averaged object. Depending on ioFlags parameter it may be used either in direct access or callback mode.

So cvEigenDecomposite performs the mapping to the lower dimensional space, while cvEigenProjection (quoting from the documentation) restores an object using previously calculated eigen objects basis, averaged object, and decomposition coefficients of the restored object. As a reference I am again quoting from the documentation:

Calculates object projection to the eigen sub-space

void cvEigenProjection( int nEigObjs, void* eigInput, int ioFlags, void* userData, float* coeffs, IplImage* avg, IplImage* proj );

• nEigObjs: Number of eigen objects.
• eigInput: Pointer either to the array of IplImage input objects or to the read callback function according to the value of the parameter ioFlags.
• ioFlags: Input/output flags.
• userData: Pointer to the structure that contains all necessary data for the callback functions.
• coeffs: Previously calculated decomposition coefficients.
• avg: Averaged object.
• proj: Decomposed object projection to the eigen sub-space.

The function cvEigenProjection calculates an object projection to the eigen sub-space or, in other words, restores an object using previously calculated eigen objects basis, averaged object, and decomposition coefficients of the restored object. Depending on ioFlags parameter it may be used either in direct access or callback mode.

The functions of the eigen objects group have been developed to be used for any number of objects, even if their total size exceeds free RAM size. So the functions may be used in two main modes. Direct access mode is the best choice if the size of free RAM is sufficient for all input and eigen objects allocation. This mode is set if the parameter ioFlags is equal to CV_EIGOBJ_NO_CALLBACK. In this case input and output parameters are pointers to arrays of input/output objects of IplImage* type. The parameters ioBufSize and userData are not used.

For an in-depth explanation of how such a system might work with the mentioned functions, I just found this nice github page:

• https://github.com/octavian-sima/Face-Recognition/wiki/Face-Recognition#training

Although I still recommend using the new API.