Deaf or hard-of-hearing individuals rely on complex sign language techniques to facilitate effective communication, utilizing a range of expressive tools including gestures, facial cues, and bodily postures to convey subtle meanings with precision. American Sign Language showcases its linguistic sophistication through unique grammatical structures and syntactical rules.
Sign language’s rarity notwithstanding, the vast array of distinct sign languages worldwide is a testament to their linguistic diversity, each boasting its unique grammar, syntax, and vocabulary, underscoring the remarkable complexity and richness of global sign languages.
Real-time innovations aim to translate sign language hand movements seamlessly into written text or verbal communication. To improve communication accessibility for individuals who are deaf or hard of hearing, there is a need for a reliable, real-time system capable of accurately detecting and monitoring American Sign Language (ASL) gestures in real-time. This innovative approach plays a pivotal role in dismantling communication barriers and fostering more inclusive dialogue.
Researchers at Florida Atlantic University’s School of Engineering and Computer Science conducted a pioneering study aimed at recognizing American Sign Language (ASL) alphabet gestures using computer vision. Developed was a meticulously curated dataset comprising 29,820 high-quality still images of American Sign Language (ASL) hand gestures, meticulously capturing the nuances and subtleties of this vital mode of communication. Through the integration of MediaPipe, each image was meticulously labeled with 21 distinctive hand landmarks, providing a comprehensive array of spatial information regarding its architecture and positioning.
The proposed annotations played a vital role in refining the accuracy of YOLOv8, the trained deep learning model, enabling it to more effectively identify subtle nuances in hand movements and gestures.
The outcomes published in the Elsevier journal demonstrate that incorporating precise hand pose information enables the model to achieve a more nuanced detection process, successfully capturing the intricate structure of American Sign Language gestures. By integrating MediaPipe’s hand motion monitoring capabilities with YOLOv8-based coaching, we developed a highly accurate system capable of effectively recognising American Sign Language (ASL) alphabet gestures.
“With the fusion of MediaPipe and YOLOv8, accompanied by meticulous optimization of hyperparameters to optimize performance, this innovative approach marks a significant milestone in AI development,” said Dr. Bader Alsharif, pioneer and Ph.D. researcher. Candidate pursuing a degree within the Florida Atlantic University (FAU) Division of Electrical Engineering and Computer Science. This novel approach remains uninvestigated in prior research, offering a pioneering pathway for further advancements.
Studies indicate that the artificial figure achieved a remarkable accuracy rate of 98%, demonstrated exceptional recall capabilities, accurately replicating gestures at 98%, and garnered a near-perfect overall efficiency score of 99% with its F1 rating. The system successfully obtained an average Common Precision (mAP) of 98% and a more granular mAP50-95 score of 93%, underscoring its impressive reliability and accuracy in detecting American Sign Language hand movements.
According to Alsharif, the findings of our analysis demonstrate that our model is capable of accurately detecting and categorizing American Sign Language gestures with minimal mistakes. “Notably, research results highlight both the resilience and versatility of the system, enabling seamless integration into everyday applications that facilitate more natural human-computer interaction.”
The seamless fusion of MediaPipe’s landmark annotations with the YOLOv8 coaching course significantly enhanced both bounding box accuracy and gesture classification capabilities, enabling the model to discern subtle nuances in hand postures. The combination of landmark monitoring and object detection played a crucial role in ensuring the system’s exceptional precision and efficacy in practical applications, thereby justifying its importance. The mannequin’s capacity to manage high-volume recognition fees with ease under diverse hand positions and gestural inputs showcases its remarkable versatility across a range of practical applications.
According to Dr. Mohammad Ilyas, our study reveals that integrating advanced object detection techniques with landmark monitoring capabilities can yield real-time gesture recognition capabilities, ultimately providing a reliable solution for American Sign Language interpretation. The pivotal factors contributing to this model’s triumph are the judicious incorporation of transfer learning, the painstaking compilation of datasets, and the precise calibration of hyperparameters. This breakthrough has resulted in an exceptionally accurate and reliable system for identifying American Sign Language (ASL) gestures, marking a significant achievement in the field of assistive technology.
Future initiatives will focus on expanding the dataset to encompass a broader range of hand configurations and movements, thereby allowing the model to better differentiate between gestures that may appear visually similar, ultimately leading to enhanced recognition precision. Optimizing the mannequin’s deployment on edge units takes precedence, ensuring seamless performance in resource-constrained environments with minimal latency and maximum efficacy.
“With this breakthrough in American Sign Language recognition, we’re one step closer to developing innovative tools that can significantly enhance communication pathways for individuals who are deaf or hard of hearing.” The prospect of reliable gesture interpretation via mannequins unlocks innovative possibilities for greater inclusivity, fostering seamless daily interactions – whether in education, healthcare or social environments – for individuals reliant on sign language, thereby promoting enhanced accessibility. This progress holds great promise for fostering an even more inclusive society where communication barriers are reduced.
Easa Alalwany, Ph.D., a current doctoral candidate. Graduate of Florida Atlantic University’s School of Engineering and Computer Science and an assistant professor at Taibah University in Saudi Arabia.
