Vocal Joystick Software Revolutionizes Vocal Mouse Control

{sidebar id=1}For many with severe mobility impairment, use of the mouse can be extremely challenging if not downright impossible. Unlike the keyboard, whose functionality can easily be augmented or replaced by switch interfaces, speech recognition software, and mouse-driven on-screen keyboards, the mouse is quite difficult to fully and naturally emulate.

Admittedly, it's possible to control a computer's cursor in a very sharp and geometric manner using keys and switches; the most pervasive example of this is probably Windows' "mousekeys" feature. Speech recognition software can do a passable job of mouse control as well, such as Dragon NaturallySpeaking's MouseGrid functionality.

However, these all approach cursor control in a highly linear manner, losing the fluidity of motion that makes a normal mouse such a natural input device. While this deficit may be acceptable for your standard computing application, not being able to naturally move the cursor from one point to another without needing to move in straight horizontal and vertical lines or memorizing coordinates can be a killer when it comes to gaming.

Enter the Vocal Joystick: a revolutionary piece of software being developed at the University of Washington. Where traditional voice recognition software hinges upon recognizing discrete words and commands, Vocal Joystick directly translates vowel sounds into cursor movement. Unlike normal speech, vowel sounds (e.g. "aaah," "aaay," "oooh") can be sustained as long as the user has air to expel, meaning the user is able to make the cursor move exactly as long as they want it to move.

Furthermore, it is reasonably seamless to transition from one vowel sound to another. This translates to easy, instantaneous directional changes to the cursor's motion. Together, these facets of vowel vocalizations afford seamless control of a cursor in the 2D space of a desktop.

Basic operation of the Vocal Joystick is very straightforward. The image to the right shows example words which contain the vowel sounds required to move the cursor in each direction. As the user produces one of these sounds, the cursor will move in the corresponding direction.

Varying the volume of the vowel sound varies the speed with which the cursor moves. In order to click the mouse, the user need only make a "ck" noise as made when saying the word "click."

"The Vocal Joystick is to a mouse what speech recognition is to a keyboard" commented Prof. Jeff Bilmes, the project's lead scientist, in the above video.  And this

comparison, while it may seem oversimplified, is actually quite truthful. By using the same input (audible sound) but approaching its interpretation in a fundamentally different way than traditional speech recognition, the Vocal Joystick has opened up an entirely new realm of human-computer interaction.

While this technology's application to improving the accessibility of the gaming world is clear, it is important to note that the Vocal Joystick and other systems that will use it in the future go far beyond even computing.

The team working on the Vocal Joystick is currently working on a system that allows the user to control a fully articulated desk-mounted robotic arm using the same vowel sounds one can currently use to control the computer cursor.

It is foreseeable that such technology could eventually find its way into the homes and onto the wheelchairs of the severely mobility impaired to increase independence and allow users to physically interact with the world in a way they cannot with their actual limbs. In theory, this technology could even serve as a highly agile hands-free control system for electric wheelchairs with the necessary noise canceling considerations.

You don't need to wait for these futuristic applications to try out the Vocal Joystick for yourself, though! An alpha release of the software is freely available from the project's website here. All you need is a soundcard and microphone, and within a few minutes, you'll be cruising all over your computer hands-free.

I tested the Vocal Joystick using a range of microphones from high-quality $100+ noise canceling headsets to cheap $10 clip-on lapel microphone and my results were good regardless of which microphone I was using.

Unfortunately, the Vocal Joystick's current release is Windows-only, though it's possible that you could run it on a Mac OS or Linux distribution using WINE or similar software. I haven't personally tested this yet, though, so I can't comment on this method's stability. If you're feeling adventurous and try it out, though, post a comment and let us know how it works!