Real-time recognition of Indonesian sign language using recurrent neural network

Abstract

Hand gestures serve as a vital means of communication for deaf individuals. They often face communication challenges in their daily interactions due to the language barrier. This underscores the necessity of sign language interpreters. However, prevailing methods primarily rely on the Indonesian Sign Language System (SIBI), despite the widespread use of Indonesian Sign Language (BISINDO) for communication. Additionally, the effectiveness of these methods hinges greatly on the accuracy of feature extraction. To address this limitation, this study introduces a Recurrent Neural Network (RNN) approach for BISINDO interpretation. Data acquisition involved the use of a webcam to capture video data, subsequently transformed into frames and
arrays. Collected from three respondents, the dataset comprises 3,240 videos and 97,200 array data points, encompassing letters and numbers. Among the tested parameters, training results indicate that utilizing the Adam optimizer with a learning rate of 0.0001 and 500 epochs yields the highest accuracy and minimal loss compared to other configurations. Subsequently, this model
underwent real-time testing, conducted five times for 36 classes, achieving an accuracy of 81.67%. It is important to note that errors may arise due to similarities within hand signal language, particularly involving characters such as I, J, D, P, M, and N.