Augmented reality (AR) has become an attractive trend in the field of language learning. Publications throughout the years 2014 to 2019 show the popularity of mobile-based AR mainly for learning the vocabulary, reading and speaking. There are multiple choices when searching for apps to learn English, Spanish, Portuguese and many more but what if the language you want to get to know to communicate with our opponent is a visual language which depends on signing (forming words by gesticulation with hands)? The main reason for the chosen topic is to alleviate the communication to and with deaf people by making this visual language more accessible in a learning application by implementing augmented reality.
State of the art
The work on technologies, algorithms and language AI that tracks and translates hand gestures into speech developed a lot in the last years. In hope that developers will use them to make own translating apps, Google has published algorithms which can perceive the shape and motion of hands in 2019. Problems the technology is currently facing are hidden fingers and regionalisms in sings meaning that signs differ in specific local areas. Furthermore signing depends not only on hand signs but more importantly the meaning can vary through facial expressions or speed of signing.
To close the gap and enable communication to family members, innovators around the world have created own solutions and technologies like Roy Allela who made a pair of gloves for his hearing-impaired niece. His application reads the text aloud from the movement of the haptic gloves.
Current apps use pictures of hand guestures or videos and many do not include the upper body which is as already mentioned important for the signing context.
Goal
The app should include different possibilities for both sides, people which are hearing and which are not. For example features for learning the guestures/signs with the help of augmented realities to get in touch with the actual situation and speed of gesticulation. For communicating the app should translate the video information of recorded movements to text or audio.
Sources
https://onlinelibrary.wiley.com/doi/10.1111/jcal.12486
https://gebaerdenlernen.de/index.php?article_id=112
https://www.geo.de/geolino/mensch/1854-rtkl-gebaerden-wie-gebaerdensprache-funktioniert
https://www.gehoerlosen-bund.de/faq/deutsche%20geb%C3%A4rdensprache%20(dgs)
