Seeking help with converting opencv webcam output to pixel coordinates for integrating with sensory substitution program [closed]
strong textHi all,
I am trying to create a basic version of the vOICe (http://www.seeingwithsound.com) to run on Linux and devices similar to the raspberry pi. This involves writing C code. The last time I did anything in C was 1994 or perhaps 1996. Any how, I have most of the stuff worked out. There is basic soundscape generation code at http://www.seeingwithsound.com. That code works on Linux.
I am using the opencv library to capture images from a webcam on Linux. I am unable to integrate opencv's output with the code of the vOICe. The vOICe expects pixel coordinates as numbers. I need to replace the hard coded image coordinates with the output from a webcam. I am not sure how openccv gives said coordinates. Yes, I did search and found the answer but I cannot understand the data type conversions involved. The answer is at the opencv faq. http://opencv.willowgarage.com/wiki/faq#Howtoaccessimagepixels
I am pasting the opencv code and then the code of the vOICe. Note: If extracting the code from this message becomes a problem, I have the 2 files in dropbox. Openn cv code: https://dl.dropboxusercontent.com/u/3688386/ot.c the link to the code for the vOICe https://dl.dropboxusercontent.com/u/3688386/i2s.c
//text of opencv code
#include <stdio.h>
#include <stdlib.h>
#include "stdafx.h"
#include <math.h>
#include <cv.h>
#include <highgui.h>
void main(int argc,char *argv[])
{
int c;
IplImage* color_img;
CvCapture* cv_cap = cvCaptureFromCAM(0);
color_img = cvQueryFrame(cv_cap); // get frame
int nl= color_img->height;
int nc= color_img->width * color_img->nChannels;
unsigned char *data= reinterpret_cast<unsigned char *>(color_img->imageData);
cvNamedWindow("Video",0); // create window
for(;;) {
if(color_img != 0)
cvShowImage("Video", color_img); // show frame
c = cvWaitKey(10); // wait 10 ms or for key stroke
if(c == 27)
break; // if ESC, break and quit
}
/* clean up */
cvReleaseCapture( &cv_cap );
cvDestroyWindow("Video");
}
</code>
//The code of the vOICe
/* Simple C program for .wav soundscape generation. (C) P.B.L. Meijer 1996 */
<code>
#define FNAME "arti1.wav"
#define N 64 /* Resolution, i.e., # rows and columns */
#define FL 500 /* Lowest frequency (Hz) in visual sound */
#define FH 5000 /* Highest frequency (Hz) */
#define FS 20000 /* Sample frequency (Hz) */
#define T 1.05 /* Image to sound conversion time (s) */
#define D 1 /* Linear|Exponential=0|1 distribution */
#include <stdio.h>
#include <stdlib.h>
#include <math.h>
#define C_ALLOC(number, type) ((type *) calloc((number),sizeof(type)) )
#define TwoPi 6.283185307179586476925287
#define WHITE 1.00
#define DGREY 0.25
#define BLACK 0.00
FILE *fp; unsigned long ir=0L, ia=9301L, ic=49297L, im=233280L;
void wi(unsigned int i) { int b1,b0; b0=i%256; b1=(i-b0)/256; putc(b0,fp); putc(b1,fp); }
void wl(long l) { unsigned int i1,i0; i0=l%65536L; i1=(l-i0)/65536L; wi(i0); wi(i1); }
double rnd(void){ ir = (ir*ia+ic) % im; return ir / (1.0*im); }
int main(int argc, char *argv[])
{
int i, j, b, d ...
hi pranav, use code tag to make it easier to read
just to clarify:
your hardcoded image is uchar/gray. wouldn't you want to convert the cam-image to gray as well, before getting the pixels/processing that ?
my camera has a resolution of 640x480, your example image is 64x64 so that needs downsampling ?
in the end, you're not going to write wav-files to disk, are you ? no idea, if RtAudio runs on the pi, but it probably needs waiting for one soundbuffer to finish, before aquiring the next image ?
some hint about the algorithm for the sound generation would be nice, too. src ? links ?
@berak: 1. Yes. 2. Yes. 3. Probably. 4. http://www.seeingwithsound.com/im2sound.htm
thanks for the nice link. haven't got an out-of-the-box answer ready, but started playing with your idea there ( getting the numbers right is not so easy ). but in general ( that's about 3 , again) would you follow the flow of this "thevoice" program ? that is: generate sound for some time, say a second, then make a "blip" to signalize the image is changing, and then hop on to the next img ?
your main question seems to be on "how to get a nice realtime solution" (with images from a webcam) from your fragments, right ?
If Pranav can put this together, using the Raspberry -Pi with either camera spy glasses (single source) or a cheap webcam , plus a rudimentary keypad and earbuds... we have a sight substitution system which is as good as prosthetics costing $ 50,000. The vOICe is DIY, free, collegiate..disruptive. At present, The vOICe is offered on Windows and on Android. Both systems have problems with cost or portability . A wearable system with low battery consumption, modularity and low price..what could be bad about that?
yea, let's go ;)
Hi berak and all, Some of your questions have already been answered in previous comments.
I do not plan to write the wave files to disk. <smile The idea is to write to the Linux audio device. If I understand things correctly, all devices in Linux are files. The vOICe sounds one frame per second so, I plan to capture one frame and then render that as a soundscape.
Pranav
hi Pranav, i think i'm onto something now ;)
got a realtime version working (using opencv for cam, rtaudio for sound ).
i'll give it a bit of polish, and put it on github !
Many many thanks Berak. I'll give it a go over the weekend!
Hi all, I am trying to compile the program. I have opencv installed on Ubuntu 12.04 LTS. I am using the following commandline.
g++ -i /usr/include/opencv2 -lopencv_core -lopencv_imgproc -lopencv_highgui -lopencv_ml -lopencv_video -lopencv_features2d -lopencv_calib3d -lopencv_objdetect -lopencv_contrib -lopencv_legacy -lopencv_flann ./main.cpp
I am getting the following error. cc1plus: error: unrecognized command line option ‘-i’
I am not sure what is happening. I suspect I am making an error in including the libraries. I do not know which ones to include.
Pranav
you only need the first 3. -lopencv_core -lopencv_imgproc -lopencv_highgui
also : g++ -I /usr/include/opencv2 (big I )
btw, irc://irc.freenode.net/opencv might be a faster way to communicate
Hi all,
I have found the problem. I was misinterpreting what g++ was telling me. The problem was with the path in the #include directive. highgui etc have their own directories in the /usr/include/opencv2 directory. I have now got past that problem but am getting scope related errors. I am still continuing to do something incorrectly since from what I understand from Berak's documentation, the code runs on his setup.
main.cpp: In function ‘int main(int, char**)’:
main.cpp:143:18: error: ‘CAP_PROP_FRAME_HEIGHT’ was not declared in this scope
main.cpp:144:18: error: ‘CAP_PROP_FRAME_WIDTH’ was not declared in this scope
sorry, should be CV_CAP_PROP_FRAME_WIDTH (was running against another version)
Hi Pranav, first of all, sorry for luring you into something overengineered and badly tested. i promise to try harder!
i found out, that you can just write the soundbytes to stdout and pipe that into the "aplay" program.
please look at the updated "simple.cpp" prog, and try that first, it's much easier
Hi Berak, Many thanks for your efforts. I updated the ghitub repository. g++ -I /usr/include -I /home/noddy/vOICe/s eeingwithsound/RtAudio simple.cpp -lopencv_core -lopencv_imgproc -lopencv_highgui
simple.cpp: In function ‘int main(int, char*)’:
simple.cpp:58:59: error: cannot convert ‘_IO_FILE’ to ‘const char’ for argument ‘1’ to ‘int printf(const char, ...)’
Are we using different versions of opencv by any chance?
I agree irc could be a faster way to communicate however, we run into the issue of timezones. I am 5.5 hours ahead of UTC. Do you have a website or e-mail address where I can reach you?