Seeking help with converting opencv webcam output to pixel coordinates for integrating with sensory substitution program

asked 2013-05-26 19:20:12 -0500

Pranav Lal gravatar image

updated 2013-05-27 23:58:10 -0500

strong textHi all,

I am trying to create a basic version of the vOICe ( to run on Linux and devices similar to the raspberry pi. This involves writing C code. The last time I did anything in C was 1994 or perhaps 1996. Any how, I have most of the stuff worked out. There is basic soundscape generation code at That code works on Linux.

I am using the opencv library to capture images from a webcam on Linux. I am unable to integrate opencv's output with the code of the vOICe. The vOICe expects pixel coordinates as numbers. I need to replace the hard coded image coordinates with the output from a webcam. I am not sure how openccv gives said coordinates. Yes, I did search and found the answer but I cannot understand the data type conversions involved. The answer is at the opencv faq.

I am pasting the opencv code and then the code of the vOICe. Note: If extracting the code from this message becomes a problem, I have the 2 files in dropbox. Openn cv code: the link to the code for the vOICe

//text of opencv code

#include <stdio.h> #include <stdlib.h> #include "stdafx.h" #include <math.h> #include <cv.h> #include <highgui.h>

void main(int argc,char *argv[])
int c;
IplImage* color_img;
CvCapture* cv_cap = cvCaptureFromCAM(0);
color_img = cvQueryFrame(cv_cap); // get frame

    int nl= color_img->height;
    int nc= color_img->width * color_img->nChannels;

    unsigned char *data= reinterpret_cast<unsigned char *>(color_img->imageData);

cvNamedWindow("Video",0); // create window
for(;;) {
if(color_img != 0)
cvShowImage("Video", color_img); // show frame
c = cvWaitKey(10); // wait 10 ms or for key stroke
if(c == 27)
break; // if ESC, break and quit
/* clean up */
cvReleaseCapture( &cv_cap );

//The code of the vOICe
/* Simple C program for .wav soundscape generation. (C) P.B.L. Meijer 1996 */
#define FNAME "arti1.wav"
#define N     64   /* Resolution, i.e., # rows and columns   */
#define FL   500   /* Lowest  frequency (Hz) in visual sound */
#define FH  5000   /* Highest frequency (Hz)                 */
#define FS 20000   /* Sample  frequency (Hz)                 */
#define T   1.05   /* Image to sound conversion time (s)     */
#define D      1   /* Linear|Exponential=0|1 distribution    */

#include <stdio.h>
#include <stdlib.h>
#include <math.h>

#define C_ALLOC(number, type) ((type *) calloc((number),sizeof(type)) )
#define TwoPi 6.283185307179586476925287
#define WHITE 1.00
#define DGREY 0.25
#define BLACK 0.00

FILE *fp; unsigned long ir=0L, ia=9301L, ic=49297L, im=233280L;
void wi(unsigned int i) { int b1,b0; b0=i%256; b1=(i-b0)/256; putc(b0,fp); putc(b1,fp); }
void wl(long l) { unsigned int i1,i0; i0=l%65536L; i1=(l-i0)/65536L; wi(i0); wi(i1); }
double rnd(void){ ir = (ir*ia+ic) % im; return ir / (1.0*im); }

int main(int argc, char *argv[])
   int    i, j, b, d ...
edit retag flag offensive close merge delete



hi pranav, use code tag to make it easier to read

orochi gravatar imageorochi ( 2013-05-26 21:30:31 -0500 )edit

just to clarify:

  1. your hardcoded image is uchar/gray. wouldn't you want to convert the cam-image to gray as well, before getting the pixels/processing that ?

  2. my camera has a resolution of 640x480, your example image is 64x64 so that needs downsampling ?

  3. in the end, you're not going to write wav-files to disk, are you ? no idea, if RtAudio runs on the pi, but it probably needs waiting for one soundbuffer to finish, before aquiring the next image ?

  4. some hint about the algorithm for the sound generation would be nice, too. src ? links ?

berak gravatar imageberak ( 2013-05-27 03:50:55 -0500 )edit

@berak: 1. Yes. 2. Yes. 3. Probably. 4.

seeingwithsound gravatar imageseeingwithsound ( 2013-05-27 15:09:54 -0500 )edit

thanks for the nice link. haven't got an out-of-the-box answer ready, but started playing with your idea there ( getting the numbers right is not so easy ). but in general ( that's about 3 , again) would you follow the flow of this "thevoice" program ? that is: generate sound for some time, say a second, then make a "blip" to signalize the image is changing, and then hop on to the next img ?

your main question seems to be on "how to get a nice realtime solution" (with images from a webcam) from your fragments, right ?

berak gravatar imageberak ( 2013-05-27 15:24:54 -0500 )edit

If Pranav can put this together, using the Raspberry -Pi with either camera spy glasses (single source) or a cheap webcam , plus a rudimentary keypad and earbuds... we have a sight substitution system which is as good as prosthetics costing $ 50,000. The vOICe is DIY, free, collegiate..disruptive. At present, The vOICe is offered on Windows and on Android. Both systems have problems with cost or portability . A wearable system with low battery consumption, modularity and low price..what could be bad about that?

finner gravatar imagefinner ( 2013-05-27 16:22:07 -0500 )edit

yea, let's go ;)

berak gravatar imageberak ( 2013-05-27 16:23:54 -0500 )edit

Hi berak and all, Some of your questions have already been answered in previous comments.

I do not plan to write the wave files to disk. <smile The idea is to write to the Linux audio device. If I understand things correctly, all devices in Linux are files. The vOICe sounds one frame per second so, I plan to capture one frame and then render that as a soundscape.


Pranav Lal gravatar imagePranav Lal ( 2013-05-28 00:03:01 -0500 )edit

hi Pranav, i think i'm onto something now ;)

got a realtime version working (using opencv for cam, rtaudio for sound ).

i'll give it a bit of polish, and put it on github !

berak gravatar imageberak ( 2013-05-29 04:40:38 -0500 )edit

Many many thanks Berak. I'll give it a go over the weekend!

Pranav Lal gravatar imagePranav Lal ( 2013-05-31 00:17:26 -0500 )edit

Hi all, I am trying to compile the program. I have opencv installed on Ubuntu 12.04 LTS. I am using the following commandline.

g++ -i /usr/include/opencv2 -lopencv_core -lopencv_imgproc -lopencv_highgui -lopencv_ml -lopencv_video -lopencv_features2d -lopencv_calib3d -lopencv_objdetect -lopencv_contrib -lopencv_legacy -lopencv_flann ./main.cpp

I am getting the following error. cc1plus: error: unrecognized command line option ‘-i’

I am not sure what is happening. I suspect I am making an error in including the libraries. I do not know which ones to include.


Pranav Lal gravatar imagePranav Lal ( 2013-06-01 08:16:21 -0500 )edit