AI meets ISL: Technology to translate sign language and the spoken word
Researchers in Ireland are developing a new technology that can translate Irish Sign Language into spoken words, and vice versa. The app, which is now at prototype stage, could help deaf people who use ISL when they are in social situations, meetings or medical appointments where an interpreter is needed but not available.
The technology has grown through a collaboration between researchers at Lero the SFI Research Centre for Irish Software, University College Dublin, and at Microsoft.
Dr Anthony Ventresque, who founded and directs the UCD Complex Software Lab, explains, “The idea came through the Skype4Good programme and we have developed an application that can translate speech into ISL and it can translate ISL into speech. We have made the software for use with a phone, with a Microsoft Hololens (a virtual reality headset) or with a laptop.”
If a Deaf person wants to use they app to translate speech to ISL, they would see an avatar signing the translation on their device. Then if a person using ISL wants to translate signing to speech, they would stand in front of a Microsoft Kinect that can register their movements and expressions and this would render the speech. “We hope that as the cameras on phones continue to develop and capture depth, that the approach may one day be a case of holding the phone in front of the person who is signing."
In order to effectively translate ISL, the researchers needed to figure out how to capture information such as facial expressions and the positions of the hands relative to the body. “I cannot sign well, but I am told that about 70% of the communication in sign language comes from the facial expressions - for example the person who is signing might raise their eyebrows to indicate a question. So we used computer vision and a deep-learning approach for the AI to detect and learn the facial expressions.”
The resulting AI-based technology could provide a useful aid to Deaf people when they need a quick translation on the spot. "We know from working with Deaf students that, unlike lectures where ISL interpreters are provided, sometimes they need to go to meetings at short notice and it’s not possible to book an interpreter. So a technology like this could be very helpful.”
Dr Ventresque also notes that wider accessibility for Deaf communication is an important facet of workplaces being more inclusive. “This is the kind of application where AI could help to move things in the right direction."