Multitouch Tables

I've been messing around on and off in the past year or so, learning about multipoint input techniques, and trying to implement some in .Net.
Aspired by Microsoft Surface and NUI Group, I tried to build my own test-bed. So far, the success is, to say the least, limited :-)

It is made of a carton box used to accommodate my vacuum cleaner, a 5mm clear perspex board I bought for too much money (these things really are expensive), a web camera I once bought (I tend to buy a new one for experiments every few months or so, usually in the local supermarket) and a 48 LEDs infrared projector I bought on eBay.

Problem was I could never set it up right, although I had some success doing fiducial recognition and tracking completely in .Net. The only problem was performance, mainly of the filters: I tired to implement a Constant Time Median Filter, but even though this is the most efficient method to implement Median Filter, it still runs too slow (about 1 FPS), even on my computer (Intel QuadCore Q6600).
Then I tried to change the implementation to use Mono SIMD, but the camera code (COM) didn't work and the program didn't run, and then I put it aside.

Funny thing is, that the thing I really want to do is to develop controls and infrastructure for such a device, but I don't want not to be able to experiment with it myself, so I just don't get there.
I did write some demos to see I can get the basics to work (two markers that I can drag around with the mouse, representing two fingers), and using touchlib I also managed to play around with some more complex techniques, and finally got a nice picture scaling and rotating on screen. But setting up the environment was such a pain, plus I wanted to see if I can do it all purely in .Net.

To sum things up - I do think it is possible, but it will require some more work. I would either get the camera working in mono one day, or maybe microsoft will release a service pack for .net that implements SIMD, I'll learn to use my GPU for that task (CUDA vision, perhaps?), or try to do all that magic directly on Linux.