Kinect as Sign Language Interpreter?

Microsoft has submitted a very interesting patent application for its Kinect system. The technology is called “Gesture Keyboarding,” which is defined in the abstract as follows:

“A user makes a gesture by either making a pose or moving in a pre-defined way that is captured by a depth camera. The depth information provided by the depth camera is parsed to determine at least that part of the user that is making the gesture. When parsed, the character or action signified by this gesture is identified.”

Among the claims, it is mentioned that these gestures may include sign language and specifically American Sign Language (ASL). In the descriptions of drawings that were included with the patent, Microsoft says that “ASL has the advantage of having a large number of people who are already facile in using it. To that end, a user who is facile in ASL will have an easy time inputting characters to a system that accepts ASL gestures as input.”

The possibilities here are very interesting. For example, there could be games intended to teach ASL, where you get feedback tailored to your exact movements. Or perhaps deaf and other signing gamers may be able to send information via sign language instead of using menu selections or onscreen keyboards.

Gesture Keyboarding on Kinect could also have non-gaming applications, similar to what we’ve discussed here before. An easy-to-use sign language interpreting system could have a place in hospitals, doctor’s offices, governmental and educational institutions, and just about anywhere that doesn’t have an interpreter on staff but serves deaf clients. 

But it’s important to be aware of the possible limitations of this system. ASL is not merely about manual signs and gestures. For one thing, ASL grammar is very different from English grammar; you cannot simply replace words in an English sentence with corresponding signs and call it American Sign Language. This means that unless the application using Gesture Keyboarding is programmed with both grammars, it’s more likely to produce English-to-signed-English transliterations, and vice versa.

Another thing to keep in mind is that ASL makes use of what is called “facial grammar.” For example, the type of question you’re asking is marked by whether your eyebrows are raised or lowered. Kinect will have facial recognition abilities, but will it be sensitive enough to pick up on expressions that make up an integral part of ASL grammar?

These two considerations, among many others such as cultural issues, are why no electronic interpreting system has yet, or will likely ever take the place of human sign language interpreters. However, in a medical emergency before an interpreter can be brought in, or if a deaf person is unable to write in order to communicate with a hearing person, even a potentially limited system like Kinect could be a boon.

Finally, sign language is not universal. The patent application mentions sign language in general as well as ASL specifically, but what about other national sign languages? The manual alphabets alone are very different between American and British sign, for example. Games and other programs that use only ASL will exclude deaf gamers and signers in other countries (except for most of Canada).

It’s exciting that Microsoft has included in its patent application the potential for sign language to be used with Kinect. While it will not replace the need for human sign language interpreters, it’s certainly a great step in the right direction when it comes to accessibility. If Gesture Keyboarding is included with Kinect, the possibilities for both gaming and non-gaming applications are many, and we can hope that some of the issues discussed here will be addressed.

(Thanks to Adam Milam for the tip.)