Libbux's profile - activity

This seems like a serious undertaking - and one that you could easily write a thesis on, let alone a high school science fair project.

If you want to finish this in the next hundred years, I'd say that you'll need to drop a couple of the features that were shown in the promotional video. First and foremost will be the 3-dimensional tools floating around showing you what to do - you may be able to work something out in OpenCV with image overlays for directions, but 3d is a bit far fetched; remember, you don't have an unlimited supply of computational power here; and you'll need every last drop of it for the object detection and recognition you're doing. And also, you won't be able to write this in Python. You'll have to bite the bullet and go C/C++, to leverage the full possibilities of OpenCV and other libraries. Sorry bud. :D

The barcode reading could be a challenge. I think you could use zxing or something like that to read the barcode and identify what the puzzle is, (although keep in mind that you'll only be able to have a few puzzles to solve, unless you want to go writing 'technician' AIs for every single electrical device with a barcode on it - which I doubt you want to do). Not only will you only be able to have a couple of puzzles, but also only a couple of solutions - just like in the pictures you show, only a single blatantly obvious loose cable or something - because you'll have to manually teach the machine what a puzzle looks like when it's solved, and what it looks like when it hasn't been solved yet.

The object detection and recognition, like the screwdriver, is actually not terribly difficult. You could adapt the method shown in this video. Like you said, you'd probably have to paint the tools a distinct colour so that other objects in the surrounding area aren't mistaken for tools or other objects of interest. Detecting the screws/bolts/nuts/other small things could also be a challenge, because they're small and monotonous and not distinct enough from the surrounding area; unless of course you painted them some wacky colour that made them really stand out. Colour matching is generally the way to go, because to detect objects based on their size and appearance (other than colour) is pretty much h*ll. You'd have to write your own algorithms to detect the patterns in the objects, and it's completely unnecessary. Not to mention how computationally intense detection and recognition algorithms (not based on colour) are. Colours are the way to go.

I'm not entirely sure how this whole system works with the funky glasses - are they a small embedded system that you develop on top of? Or is this whole system powered by a Raspberry Pi? Actually, with this level of detail a ... (more)