Gesture Controls - Computerphile

Key Takeaways

use neighboring troughs and Peaks as a good approximation of where the finger starts and ends and then we can trace along those to finally create a hand pose and that hand pose we can then use as a driver for input for games so we can work out if the things are extended if they're closed the system just uses the Nintendo Wii remote from the Wii uh it connects to the computer via Bluetooth for markers on the end of the fingertips are then picked up by a camera on the Nintendo wiot it's an infrared camera and the system utilizes a library developed by Brian Peak called weote lib which is freely available via codeplex we built the three-dimensional tracking capability on top of that Library it uses plane of homography to be able to work out where the points are in 3D space so we have a plane at the front and a plane at the back and we do an intersection of lines to each of the LEDs via those two planes and we can work out where things are in 3D there are four LEDs on the end of fingers they're powered by this unit on the back here essentially it's a USB rechargeable unit as well when the kits out in the home they can just plug it into the PC and charge it via the PC the gloves has been designed so we can swap for different Siz finger pieces so we can detach and reattach that there so for different users we can use different lengths of wire we can also use individual finger attachments as well the games which you can see on on the screen are essentially they're produced in uh Microsoft's XNA framework uh which is a game development framework which people can also use to distribute to Xbox 360 around 15 million people have a stroke every year the kind of movements people have to make poststroke are pronation superation reach to grasp and then grasp pick an object up and put it down in another place you have the peg test where you have to pinch pick up and drop down and what we've done is we've mapped that interaction uh to uh pinch and release like so so if I'll just show you that now essentially what I do is I I pinch to activate the ball and then I release to uh to fire the ball it's a fun and interesting project to do the main difficulties I would say would be actually building a a system which can be deployed into the home so rather than having a system just works in a research lab we had to actually make it so it was robust enough to work in the real world environment and that's where the the the difficulty came in as far as coding and that sort of thing goes so part of the problems that we had with this system were things like Phantom IR reflection so this Works off a series of infrared emitting diodes but we're getting other um infrared sources from around the house such as perhaps a television remote if you're wearing a glove there might be patient discomfort there might be hygiene issues you might find it difficult to get the glove on there's lots of little technical Parts on the glove which can get broken so we're looking now whether we can use the connect camera as a a markless system where you don't have to wear anything at all for fast and robust tracking in a home environment all right what we've got in front of us now is essentially a hand tracking system that takes advantage of the depth map features of the connect camera so in computer vision research traditionally there's been a lot of background in using Color based cameras to extract people's hands and hand pose information from a scene and they work really really well in a tightly controlled lab setting so when you've got things like green screen very tight control of the lighting conditions that doesn't translate well into to a real home setting and a practical kind of use scenario for a homebased stroke Rehabilitation system now the advantage of the connect is that instead of giving you a color feed it actually gives you a a depth map of the scene so instead of having color information encoded in pixels it encodes the distance in millimeters of each pixel in the scene and it's much more tolerant to environmental conditions and uh things like skin tone this is a proof of concept game designed to test the tracking system so it's based on the slingshot game we saw earlier using the wiot and this is designed to try the the exercise some arm Synergy so we're using the shoulder elbow that reach and the pinch and grasp the game randomly positions a ball on the screen and I just need to move my hand over the ball grab it and throw to the Target and then a ball will reappear we've got kind of a a developer mode to help us see what's going on so you can see it's finding kind of that large red circle which is really the key element of the hand tracking where it's trying to find the center of the Palm we then try to look at nearby information determine fingertips pits of the fingers and we can use that to eventually create these kind of purple lines that we can trace along to find the depth information about the fingers and from that calculate a hand pose we essentially locate the hand in the scene segment all the other information remove everything we're not interested in background your arm your face so you've only got the hand left on the depth MK we then Trace around that hand Contour so we create a kind of a contour rout around the outside of your hand and this helps account for a lot of the missing information so you might have blank spots within the hand but because we're now only looking at the Contour we kind of eliminate all those we then try and find the largest area with inside that Contour cuz that invariably turns out to be the palm of the hand we then do some distance analysis of the Contour away from the center of the Palm we do some filtering to smooth that and this helps us classify the distance graph into Peaks and troughs and we then classify maximal points as fingertips and minimal points as finger troughs and we can simply use neighboring troughs and Peaks as a good approximation of where the finger starts and ends and then we can trace along those to finally create a hand pose and that hand pose we can then use as a driver for input for games so we can work out if the things are extended if they're closed okay so connect is a a novel depth based camera so so it uses a structured light approach so instead of actually just being one camera that you might think of traditionally it has an infrared projector so it's projecting a pattern in infrared light onto everything in the scene that we can't see it then uses an infrared camera to look at that pattern and it looks for distortions in the pattern and eventually uses triangulation to calculate the depth in the scene the down side to that is that it's not particularly accurate we have done some experiments to try and verify the accuracy because using it for clinical purposes it also tends to generate a lot of missing information especially on small objects which is quite a big problem when you're looking at something like the hand because fingertips can quite often disappear now a time of flight camera like this one I have here has traditionally been a lot more expensive so this camera is in the region of several thousand pounds to buy but this actually projects infrared light and measures the time it takes for it to travel back into the camera so that creates a much more accurate dep map because it's not reliant on other information and the size of objects within the scene the connect 2 camera is reported to use a very very similar camera to this I believe the camera develops even by the same company so we're really excited to see kind of what we'll be able to do the same kind of technology and lowcost camera as a connect that's definitely suitable for home use CU it's designed for playing games in the home but with the kind of the accuracy we've got used to from these kind of cameras a camera like this would traditionally cost between 2 to 4,000 so the these are the kind of cameras that have been applied for industrial use in production lines monitoring but the connect really revolutionized the depth camera scene so it was the the first kind of real lowcost consumer avable depth camera avable and it's had a huge impact on computer vision technology is just getting better and better and I think it will make a huge impact not just on our research team but on the kind of this whole field in general connect 2 will be out of Christmas with the launch of the new Xbox one but we're hoping that we can get on the um developer list so there is a list for people to get the SDK early if they're doing interesting enough projects so hopefully you know I think it realiz on something like Photon shift but really I'm more interested in using it than i' leave that to physicists to work it's where they can write some code show me your code and I can give you a job