# Seeking help with converting opencv webcam output to pixel coordinates for integrating with sensory substitution program

strong textHi all,

I am trying to create a basic version of the vOICe (http://www.seeingwithsound.com) to run on Linux and devices similar to the raspberry pi. This involves writing C code. The last time I did anything in C was 1994 or perhaps 1996. Any how, I have most of the stuff worked out. There is basic soundscape generation code at http://www.seeingwithsound.com. That code works on Linux.

I am using the opencv library to capture images from a webcam on Linux. I am unable to integrate opencv's output with the code of the vOICe. The vOICe expects pixel coordinates as numbers. I need to replace the hard coded image coordinates with the output from a webcam. I am not sure how openccv gives said coordinates. Yes, I did search and found the answer but I cannot understand the data type conversions involved. The answer is at the opencv faq. http://opencv.willowgarage.com/wiki/faq#Howtoaccessimagepixels

I am pasting the opencv code and then the code of the vOICe. Note: If extracting the code from this message becomes a problem, I have the 2 files in dropbox. Openn cv code: https://dl.dropboxusercontent.com/u/3688386/ot.c the link to the code for the vOICe https://dl.dropboxusercontent.com/u/3688386/i2s.c

//text of opencv code


 #include <stdio.h> #include <stdlib.h> #include "stdafx.h" #include <math.h> #include <cv.h> #include <highgui.h>

 void main(int argc,char *argv[]) { int c; IplImage* color_img; CvCapture* cv_cap = cvCaptureFromCAM(0); color_img = cvQueryFrame(cv_cap); // get frame int nl= color_img->height; int nc= color_img->width * color_img->nChannels; unsigned char *data= reinterpret_cast<unsigned char *>(color_img->imageData); cvNamedWindow("Video",0); // create window for(;;) { if(color_img != 0) cvShowImage("Video", color_img); // show frame c = cvWaitKey(10); // wait 10 ms or for key stroke if(c == 27) break; // if ESC, break and quit } /* clean up */ cvReleaseCapture( &cv_cap ); cvDestroyWindow("Video"); } </code> //The code of the vOICe /* Simple C program for .wav soundscape generation. (C) P.B.L. Meijer 1996 */ <code> #define FNAME "arti1.wav" #define N 64 /* Resolution, i.e., # rows and columns */ #define FL 500 /* Lowest frequency (Hz) in visual sound */ #define FH 5000 /* Highest frequency (Hz) */ #define FS 20000 /* Sample frequency (Hz) */ #define T 1.05 /* Image to sound conversion time (s) */ #define D 1 /* Linear|Exponential=0|1 distribution */ #include <stdio.h> #include <stdlib.h> #include <math.h> #define C_ALLOC(number, type) ((type *) calloc((number),sizeof(type)) ) #define TwoPi 6.283185307179586476925287 #define WHITE 1.00 #define DGREY 0.25 #define BLACK 0.00 FILE *fp; unsigned long ir=0L, ia=9301L, ic=49297L, im=233280L; void wi(unsigned int i) { int b1,b0; b0=i%256; b1=(i-b0)/256; putc(b0,fp); putc(b1,fp); } void wl(long l) { unsigned int i1,i0; i0=l%65536L; i1=(l-i0)/65536L; wi(i0); wi(i1); } double rnd(void){ ir = (ir*ia+ic) % im; return ir / (1.0*im); } int main(int argc, char *argv[]) { int i, j, b, d ...
edit retag close merge delete

1

hi pranav, use code tag to make it easier to read

( 2013-05-26 21:30:31 -0500 )edit
1

just to clarify:

1. your hardcoded image is uchar/gray. wouldn't you want to convert the cam-image to gray as well, before getting the pixels/processing that ?

2. my camera has a resolution of 640x480, your example image is 64x64 so that needs downsampling ?

3. in the end, you're not going to write wav-files to disk, are you ? no idea, if RtAudio runs on the pi, but it probably needs waiting for one soundbuffer to finish, before aquiring the next image ?

4. some hint about the algorithm for the sound generation would be nice, too. src ? links ?

( 2013-05-27 03:50:55 -0500 )edit

@berak: 1. Yes. 2. Yes. 3. Probably. 4. http://www.seeingwithsound.com/im2sound.htm

( 2013-05-27 15:09:54 -0500 )edit

thanks for the nice link. haven't got an out-of-the-box answer ready, but started playing with your idea there ( getting the numbers right is not so easy ). but in general ( that's about 3 , again) would you follow the flow of this "thevoice" program ? that is: generate sound for some time, say a second, then make a "blip" to signalize the image is changing, and then hop on to the next img ?

your main question seems to be on "how to get a nice realtime solution" (with images from a webcam) from your fragments, right ?

( 2013-05-27 15:24:54 -0500 )edit

If Pranav can put this together, using the Raspberry -Pi with either camera spy glasses (single source) or a cheap webcam , plus a rudimentary keypad and earbuds... we have a sight substitution system which is as good as prosthetics costing \$ 50,000. The vOICe is DIY, free, collegiate..disruptive. At present, The vOICe is offered on Windows and on Android. Both systems have problems with cost or portability . A wearable system with low battery consumption, modularity and low price..what could be bad about that?

( 2013-05-27 16:22:07 -0500 )edit

yea, let's go ;)

( 2013-05-27 16:23:54 -0500 )edit

I do not plan to write the wave files to disk. <smile The idea is to write to the Linux audio device. If I understand things correctly, all devices in Linux are files. The vOICe sounds one frame per second so, I plan to capture one frame and then render that as a soundscape.

Pranav

( 2013-05-28 00:03:01 -0500 )edit
1

hi Pranav, i think i'm onto something now ;)

got a realtime version working (using opencv for cam, rtaudio for sound ).

i'll give it a bit of polish, and put it on github !

( 2013-05-29 04:40:38 -0500 )edit

Many many thanks Berak. I'll give it a go over the weekend!

( 2013-05-31 00:17:26 -0500 )edit

Hi all, I am trying to compile the program. I have opencv installed on Ubuntu 12.04 LTS. I am using the following commandline.

g++ -i /usr/include/opencv2 -lopencv_core -lopencv_imgproc -lopencv_highgui -lopencv_ml -lopencv_video -lopencv_features2d -lopencv_calib3d -lopencv_objdetect -lopencv_contrib -lopencv_legacy -lopencv_flann ./main.cpp

I am getting the following error. cc1plus: error: unrecognized command line option â€˜-iâ€™

I am not sure what is happening. I suspect I am making an error in including the libraries. I do not know which ones to include.

Pranav

( 2013-06-01 08:16:21 -0500 )edit