Making use of sleek black gloves, sophisticated sensors, a microcontroller and a smartphone, students from the Ukraine have created a device that translates sign language into speech.
Called “Enable Talk,” the system won first place in the software design category at Microsoft’s 10th annual Imagine Cup, held this year in Sydney, Australia.
The team of four students, called “QuadSquad,” will take home US$25,000 and have the opportunity to apply for further grants and startup assistance from Microsoft.
Motion to Voice
The students got the idea for the device when trying to interact with deaf classmates. The prototype has 11 flex sensors, eight touch sensors, an accelerometer/compass and accelerometer gyroscope built into the gloves. These sense where the hands are in space and transmit this data to a controller that uses Bluetooth to connect to a mobile phone.
Enable Talk’s software translates the symbol into a letter or word, and the phone’s text-to-speech API gives it a computerized voice.
The system, which the team has estimated will cost about $75 to build and $250 to purchase, has initially focused on recognizing finger-spelling commands. It’s being expanded it to include more complex American Sign Language symbols, and users will potentially be able to program it further.
“It’s exciting,” John Miller, co-owner of the ASL resource site SigningSavvy, told TechNewsWorld. “It’s a step in the right direction. It gives a voice to the deaf and hard-of-hearing population.”
Real-World Challenges
The move from finger signing to full signing in a system like this will be tricky, however.
“The problem is that hands are individually shaped, so that’s always caused problems,” said Miller. “Everyone signs a little different, and that’s why [the translation] has always been done with a human eye.”
Just like spoken languages, sign language is complex and contextual, so any program translating it must be able to pick up on the nuances of movements and contexts.
“In sign language, there is one motion for many different words and feelings,” explained Miller. “It’s all about context.”
Sign language, like spoken language, also has numerous dialects and versions, and those will need to be taken into account for any full-fledged sign-recognition program.
“There are certain dialects [or] regional signs that may be missed due to not being ‘known’ to the computer,” Liz Stevens, project manager for AllWorld Language Consultants, told TechNewsWorld. “Also, within each region, there are various ways to sign one word. For example, the sign for ‘hotel’ is made by waving the index and middle fingers in the handshape of ‘H.’ Another way to sign ‘hotel’ is done by way of contact on the face. So if these dialects are not programmed into the computer that is translating the signs into speech, there will still be a miscommunications throughout the conversation.”
Because of these technical limitations, it might be a long time before Enable Talk can be adapted for widespread use.
“There are a lot of handshapes — shapes of our hands which we use to form specific signs — and classifiers — [shapes that] represent persons, places or things — nouns — that may differ from one individual to the next,” explained Stevens.
Enable Talk is not likely to do away with the need for a human interpreter any time soon.
“We as an agency have already seen this product being solicited as a replacement to the in-person interpreter,” said Robbi Crockett, executive director of the ASL Interpreter Network.
“This is definitely not the case, [since] each face, each set of hands carries different gestures and speed of use,” Crockett told TechNewsWorld. “This device is only good, maybe, for one-on-one — and at a much slower pace since each individual style, body language is unique.”
Early Communication
Though it’s in its early stages, this technology nonetheless promises to further communication and understanding between deaf and hearing people.
“This promotes awareness,” said Miller. “Anything that promotes communication between sign language and other languages is a good thing.”
Loading ...
