Seeking help with converting opencv webcam output to pixel coordinates for integrating with sensory substitution program [closed]
strong textHi all,
I am trying to create a basic version of the vOICe (http://www.seeingwithsound.com) to run on Linux and devices similar to the raspberry pi. This involves writing C code. The last time I did anything in C was 1994 or perhaps 1996. Any how, I have most of the stuff worked out. There is basic soundscape generation code at http://www.seeingwithsound.com. That code works on Linux.
I am using the opencv library to capture images from a webcam on Linux. I am unable to integrate opencv's output with the code of the vOICe. The vOICe expects pixel coordinates as numbers. I need to replace the hard coded image coordinates with the output from a webcam. I am not sure how openccv gives said coordinates. Yes, I did search and found the answer but I cannot understand the data type conversions involved. The answer is at the opencv faq. http://opencv.willowgarage.com/wiki/faq#Howtoaccessimagepixels
I am pasting the opencv code and then the code of the vOICe. Note: If extracting the code from this message becomes a problem, I have the 2 files in dropbox. Openn cv code: https://dl.dropboxusercontent.com/u/3688386/ot.c the link to the code for the vOICe https://dl.dropboxusercontent.com/u/3688386/i2s.c
//text of opencv code
#include <stdio.h>
#include <stdlib.h>
#include "stdafx.h"
#include <math.h>
#include <cv.h>
#include <highgui.h>
void main(int argc,char *argv[])
{
int c;
IplImage* color_img;
CvCapture* cv_cap = cvCaptureFromCAM(0);
color_img = cvQueryFrame(cv_cap); // get frame
int nl= color_img->height;
int nc= color_img->width * color_img->nChannels;
unsigned char *data= reinterpret_cast<unsigned char *>(color_img->imageData);
cvNamedWindow("Video",0); // create window
for(;;) {
if(color_img != 0)
cvShowImage("Video", color_img); // show frame
c = cvWaitKey(10); // wait 10 ms or for key stroke
if(c == 27)
break; // if ESC, break and quit
}
/* clean up */
cvReleaseCapture( &cv_cap );
cvDestroyWindow("Video");
}
</code>
//The code of the vOICe
/* Simple C program for .wav soundscape generation. (C) P.B.L. Meijer 1996 */
<code>
#define FNAME "arti1.wav"
#define N 64 /* Resolution, i.e., # rows and columns */
#define FL 500 /* Lowest frequency (Hz) in visual sound */
#define FH 5000 /* Highest frequency (Hz) */
#define FS 20000 /* Sample frequency (Hz) */
#define T 1.05 /* Image to sound conversion time (s) */
#define D 1 /* Linear|Exponential=0|1 distribution */
#include <stdio.h>
#include <stdlib.h>
#include <math.h>
#define C_ALLOC(number, type) ((type *) calloc((number),sizeof(type)) )
#define TwoPi 6.283185307179586476925287
#define WHITE 1.00
#define DGREY 0.25
#define BLACK 0.00
FILE *fp; unsigned long ir=0L, ia=9301L, ic=49297L, im=233280L;
void wi(unsigned int i) { int b1,b0; b0=i%256; b1=(i-b0)/256; putc(b0,fp); putc(b1,fp); }
void wl(long l) { unsigned int i1,i0; i0=l%65536L; i1=(l-i0)/65536L; wi(i0); wi(i1); }
double rnd(void){ ir = (ir*ia+ic) % im; return ir / (1.0*im); }
int main(int argc, char *argv[])
{
int i, j, b, d ...
hi pranav, use code tag to make it easier to read
just to clarify:
your hardcoded image is uchar/gray. wouldn't you want to convert the cam-image to gray as well, before getting the pixels/processing that ?
my camera has a resolution of 640x480, your example image is 64x64 so that needs downsampling ?
in the end, you're not going to write wav-files to disk, are you ? no idea, if RtAudio runs on the pi, but it probably needs waiting for one soundbuffer to finish, before aquiring the next image ?
some hint about the algorithm for the sound generation would be nice, too. src ? links ?
@berak: 1. Yes. 2. Yes. 3. Probably. 4. http://www.seeingwithsound.com/im2sound.htm
thanks for the nice link. haven't got an out-of-the-box answer ready, but started playing with your idea there ( getting the numbers right is not so easy ). but in general ( that's about 3 , again) would you follow the flow of this "thevoice" program ? that is: generate sound for some time, say a second, then make a "blip" to signalize the image is changing, and then hop on to the next img ?
your main question seems to be on "how to get a nice realtime solution" (with images from a webcam) from your fragments, right ?
If Pranav can put this together, using the Raspberry -Pi with either camera spy glasses (single source) or a cheap webcam , plus a rudimentary keypad and earbuds... we have a sight substitution system which is as good as prosthetics costing $ 50,000. The vOICe is DIY, free, collegiate..disruptive. At present, The vOICe is offered on Windows and on Android. Both systems have problems with cost or portability . A wearable system with low battery consumption, modularity and low price..what could be bad about that?
yea, let's go ;)
Hi berak and all, Some of your questions have already been answered in previous comments.
I do not plan to write the wave files to disk. <smile The idea is to write to the Linux audio device. If I understand things correctly, all devices in Linux are files. The vOICe sounds one frame per second so, I plan to capture one frame and then render that as a soundscape.
Pranav
hi Pranav, i think i'm onto something now ;)
got a realtime version working (using opencv for cam, rtaudio for sound ).
i'll give it a bit of polish, and put it on github !
Many many thanks Berak. I'll give it a go over the weekend!
Hi all, I am trying to compile the program. I have opencv installed on Ubuntu 12.04 LTS. I am using the following commandline.
g++ -i /usr/include/opencv2 -lopencv_core -lopencv_imgproc -lopencv_highgui -lopencv_ml -lopencv_video -lopencv_features2d -lopencv_calib3d -lopencv_objdetect -lopencv_contrib -lopencv_legacy -lopencv_flann ./main.cpp
I am getting the following error. cc1plus: error: unrecognized command line option ‘-i’
I am not sure what is happening. I suspect I am making an error in including the libraries. I do not know which ones to include.
Pranav