Jnect: Kinect and Eclipse

You might have seen me waving and moving around a lot recently during presentations at Democamps and during my talks at EclipseCon’s. The reason is not a new style in giving talks, but it is the topic: the Jnect project.

The goal of this project is to connect the Microsoft Kinect SDK with Eclipse/Java. In case you do not know the Kinect, it is a device originally built for the Xbox to track human bodies. As persons can be tracked in real-time it allows you to control games, menus and more, just by moving your body. Additionally the Kinect supports speech recognition for pre-defined phrases. Microsoft has also released an SDK for Windows, which allows access to the features of the Kinect in C++/C#. However, there was one important thing missing: Eclipse

Therefore, about a year ago, I and a group of interested students started to work on a way to make the features of Kinect available in Eclipse. Unlike other approaches, we did not want to connect directly to the device and process the raw data. Instead we decided that it would be nice to reuse Microsoft’s SDK which delivers the data processed and on a very high level of abstraction. As an example the SDK delivers the exact positions of different parts of the body of a tracked person.

Building on the SDK, the tricky part was obviously the bridge from C to Java. In a first prototype we used a small C program running in the background and a socket connection to transfer the data from the original SDK to a plugin in Eclipse. This is what I demonstrated last fall at EclipseCon Europe:

In winter another team of students worked on an improvement – the adaptation via JNI. This is more stable and it doesn’t require a second program running, and therefore creates real interoperability.

Not surprisingly we created Jnect as an Eclipse plugin. Recently we created an Eclipse Labs project hosting the plugin (see jnect.org) and we already have a beta-release. It is not perfect yet, but you can test the capabilities of the Kinect and find your use case. Of course, we welcome any feedback and possible contributions. At EclipseCon North America, we were even able to implement a game based on Jnect, which was played during the awards ceremony. The goal was to step thru code as fast as possible using gestures and voice commands. Of course, we have chosen more difficult gestures then just moving a hand. Here you see one of the participants executing the “Step Over” Gesture 

During on our session at EclipseCon, we demonstrated how to move and resize windows in Eclipse 4 using your hands (a little like minority report). This and other code examples are available on jnect.org. You’ll also find videos demonstrating the capabilities of the Kinect as it controls different parts of Eclipse.

Have fun!