Deteksi Objek bertumpuk Gerak Tangan Bahasa Isyarat SIBI dengan Algoritma LSTM-Holistic
Abstract
Tantangan yang dihadapi individu dengan gangguan pendengaran dalam berkomunikasi, terutama di lingkungan yang kurang mendukung komunikasi visual, menegaskan pentingnya bahasa isyarat sebagai media utama. Di Indonesia, dua sistem bahasa isyarat yang menonjol adalah SIBI (Sistem Isyarat Bahasa Indonesia) dan BISINDO (Bahasa Isyarat Indonesia). Namun, kompleksitas dan nuansa bahasa isyarat, termasuk gerakan bertumpuk dan gerakan dinamis, menjadi hambatan signifikan dalam pengenalan dan penerjemahan yang akurat. Penelitian ini mengusulkan sistem pengenalan bahasa isyarat SIBI secara real-time dengan memanfaatkan algoritma Long Short-Term Memory (LSTM) dan kerangka Mediapipe-Holistic. Teknologi ini diintegrasikan untuk memproses gerakan dinamis dan gerakan tangan bertumpuk dengan efisien. Sistem ini dirancang untuk menerjemahkan gerakan bahasa isyarat menjadi teks. Dengan teknik pembelajaran mesin, sistem ini akan menggunakan data yang diambil secara privat sebanyak 14,400 untuk setiap gerak bahasa isyarat. untuk melatih model agar dapat mengenali gerakan serta mampu beradaptasi dengan berbagai gaya komunikasi dan kondisi lingkungan data akan dibagi menjadi 80:15. Penelitian ini mencapai akurasi deteksi 99.3% berdasarkan F1-Score yang memungkinkan model untuk melakukan deteksi gerak bahasa isyarat SIBI secara akurat.
References
[1] I. I. Arifah, F. N. Fajri, and G. Q. O. Pratamasunu, “Deteksi Tangan Otomatis Pada Video Percakapan Bahasa Isyarat Indonesia Menggunakan Metode YOLO Dan CNN,” J. Appl. Inform. Comput., vol. 6, no. 2, pp. 171–176, Nov. 2022, doi: 10.30871/jaic.v6i2.4694.
[2] World Health Organization, Addressing the rising prevalence of hearing loss. Geneva: World Health Organization, 2018. Accessed: Nov. 29, 2024. [Online]. Available: https://iris.who.int/handle/10665/260336
[3] N. Adaloglou et al., “A Comprehensive Study on Deep Learning-Based Methods for Sign Language Recognition,” IEEE Trans. Multimed., vol. 24, pp. 1750–1762, 2022, doi: 10.1109/TMM.2021.3070438.
[4] A. Mittal, P. Kumar, P. P. Roy, R. Balasubramanian, and B. B. Chaudhuri, “A Modified LSTM Model for Continuous Sign Language Recognition Using Leap Motion,” IEEE Sens. J., vol. 19, no. 16, pp. 7056–7063, Aug. 2019, doi: 10.1109/JSEN.2019.2909837.
[5] C. K. M. Lee, K. K. H. Ng, C.-H. Chen, H. C. W. Lau, S. Y. Chung, and T. Tsoi, “American sign language recognition and training method with recurrent neural network,” Expert Syst. Appl., vol. 167, p. 114403, Apr. 2021, doi: 10.1016/j.eswa.2020.114403.
[6] B. Triwijoyo, L. Karnaen, and A. Adil, “Deep Learning Approach For Sign Language Recognition,” vol. 9, pp. 12–21, Mar. 2023, doi: 10.26555/jiteki.v9i1.25051.
[7] A. Akdag and O. K. Baykan, “Enhancing Signer-Independent Recognition of Isolated Sign Language through Advanced Deep Learning Techniques and Feature Fusion,” Electronics, vol. 13, no. 7, p. 1188, Mar. 2024, doi: 10.3390/electronics13071188.
[8] M. C. Bagaskoro, F. Prasojo, A. N. Handayani, E. Hitipeuw, A. P. Wibawa, and Y. W. Liang, “Hand image reading approach method to Indonesian Language Signing System (SIBI) using neural network and multi layer perseptron,” Sci. Inf. Technol. Lett., vol. 4, no. 2, Art. no. 2, Nov. 2023, doi: 10.31763/sitech.v4i2.1362.
[9] A. Duarte et al., “How2Sign: A Large-Scale Multimodal Dataset for Continuous American Sign Language,” presented at the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 2735–2744. Accessed: Nov. 11, 2024. [Online]. Available: https://openaccess.thecvf.com/content/CVPR2021/html/Duarte_How2Sign_A_Large-Scale_Multimodal_Dataset_for_Continuous_American_Sign_Language_CVPR_2021_paper.html?ref=https://githubhelp.com
[10] K. Goyal, “INDIAN SIGN LANGUAGE RECOGNITION USING MEDIAPIPE HOLISTIC”.
[11] G. H. Samaan et al., “Mediapipe’s Landmarks with RNN for Dynamic Sign Language Recognition,” Electronics, vol. 11, no. 19, p. 3228, Oct. 2022, doi: 10.3390/electronics11193228.
[12] P. Selvaraj, G. NC, P. Kumar, and M. Khapra, “OpenHands: Making Sign Language Recognition Accessible with Pose-based Pretrained Models across Languages,” Oct. 12, 2021, arXiv: arXiv:2110.05877. doi: 10.48550/arXiv.2110.05877.
[13] K. Roh, H. Lee, E. J. Hwang, S. Cho, and J. C. Park, “Preprocessing Mediapipe Keypoints with Keypoint Reconstruction and Anchors for Isolated Sign Language Recognition”.
[14] R. M. Abdulhamied, M. M. Nasr, and S. N. A. Kader, “Real-time recognition of American sign language using long-short term memory neural network and hand detection,” Indones. J. Electr. Eng. Comput. Sci., vol. 30, no. 1, Art. no. 1, Apr. 2023, doi: 10.11591/ijeecs.v30.i1.pp545-556.
[15] P. Sheth and S. Rajora, Sign Language Recognition Application Using LSTM and GRU (RNN). 2023. doi: 10.13140/RG.2.2.18635.87846.
[16] Y. Obi, K. Claudio, V. Budiman, S. Achmad, and A. Kurniawan, “Sign language recognition system for communicating to people with disabilities,” Procedia Comput. Sci., vol. 216, pp. 13–20, Jan. 2023, doi: 10.1016/j.procs.2022.12.106.
[17] R. S. L. Murali, “Sign Language Recognition System Using Convolutional Neural Network And ComputerVision,” vol. 4, no. 4, 2022.
[18] D. Bragg et al., “Sign Language Recognition, Generation, and Translation: An Interdisciplinary Perspective,” in Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility, in ASSETS ’19. New York, NY, USA: Association for Computing Machinery, Oct. 2019, pp. 16–31. doi: 10.1145/3308561.3353774.
[19] N. C. Camgoz, O. Koller, S. Hadfield, and R. Bowden, “Sign Language Transformers: Joint End-to-End Sign Language Recognition and Translation,” presented at the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 10023–10033. Accessed: Nov. 11, 2024. [Online]. Available: https://openaccess.thecvf.com/content_CVPR_2020/html/Camgoz_Sign_Language_Transformers_Joint_End-to-End_Sign_Language_Recognition_and_Translation_CVPR_2020_paper.html
[20] S. Jiang, B. Sun, L. Wang, Y. Bai, K. Li, and Y. Fu, “Skeleton Aware Multi-Modal Sign Language Recognition,” presented at the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 3413–3423. Accessed: Nov. 11, 2024. [Online]. Available: https://openaccess.thecvf.com/content/CVPR2021W/ChaLearn/html/Jiang_Skeleton_Aware_Multi-Modal_Sign_Language_Recognition_CVPRW_2021_paper.html
[21] H. Zhou, W. Zhou, Y. Zhou, and H. Li, “Spatial-Temporal Multi-Cue Network for Continuous Sign Language Recognition,” Proc. AAAI Conf. Artif. Intell., vol. 34, no. 07, Art. no. 07, Apr. 2020, doi: 10.1609/aaai.v34i07.7001.
[22] M. Ismail, S. Dawwd, and F. Ali, “Static hand gesture recognition of Arabic sign language by using deep CNNs,” Indones. J. Electr. Eng. Comput. Sci., vol. 24, p. 178, Oct. 2021, doi: 10.11591/ijeecs.v24.i1.pp178-188.
[23] D. Li, C. Rodriguez, X. Yu, and H. Li, “Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison,” presented at the Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2020, pp. 1459–1469. Accessed: Nov. 11, 2024. [Online]. Available: https://openaccess.thecvf.com/content_WACV_2020/html/Li_Word-level_Deep_Sign_Language_Recognition_from_Video_A_New_Large-scale_WACV_2020_paper.html
[2] World Health Organization, Addressing the rising prevalence of hearing loss. Geneva: World Health Organization, 2018. Accessed: Nov. 29, 2024. [Online]. Available: https://iris.who.int/handle/10665/260336
[3] N. Adaloglou et al., “A Comprehensive Study on Deep Learning-Based Methods for Sign Language Recognition,” IEEE Trans. Multimed., vol. 24, pp. 1750–1762, 2022, doi: 10.1109/TMM.2021.3070438.
[4] A. Mittal, P. Kumar, P. P. Roy, R. Balasubramanian, and B. B. Chaudhuri, “A Modified LSTM Model for Continuous Sign Language Recognition Using Leap Motion,” IEEE Sens. J., vol. 19, no. 16, pp. 7056–7063, Aug. 2019, doi: 10.1109/JSEN.2019.2909837.
[5] C. K. M. Lee, K. K. H. Ng, C.-H. Chen, H. C. W. Lau, S. Y. Chung, and T. Tsoi, “American sign language recognition and training method with recurrent neural network,” Expert Syst. Appl., vol. 167, p. 114403, Apr. 2021, doi: 10.1016/j.eswa.2020.114403.
[6] B. Triwijoyo, L. Karnaen, and A. Adil, “Deep Learning Approach For Sign Language Recognition,” vol. 9, pp. 12–21, Mar. 2023, doi: 10.26555/jiteki.v9i1.25051.
[7] A. Akdag and O. K. Baykan, “Enhancing Signer-Independent Recognition of Isolated Sign Language through Advanced Deep Learning Techniques and Feature Fusion,” Electronics, vol. 13, no. 7, p. 1188, Mar. 2024, doi: 10.3390/electronics13071188.
[8] M. C. Bagaskoro, F. Prasojo, A. N. Handayani, E. Hitipeuw, A. P. Wibawa, and Y. W. Liang, “Hand image reading approach method to Indonesian Language Signing System (SIBI) using neural network and multi layer perseptron,” Sci. Inf. Technol. Lett., vol. 4, no. 2, Art. no. 2, Nov. 2023, doi: 10.31763/sitech.v4i2.1362.
[9] A. Duarte et al., “How2Sign: A Large-Scale Multimodal Dataset for Continuous American Sign Language,” presented at the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 2735–2744. Accessed: Nov. 11, 2024. [Online]. Available: https://openaccess.thecvf.com/content/CVPR2021/html/Duarte_How2Sign_A_Large-Scale_Multimodal_Dataset_for_Continuous_American_Sign_Language_CVPR_2021_paper.html?ref=https://githubhelp.com
[10] K. Goyal, “INDIAN SIGN LANGUAGE RECOGNITION USING MEDIAPIPE HOLISTIC”.
[11] G. H. Samaan et al., “Mediapipe’s Landmarks with RNN for Dynamic Sign Language Recognition,” Electronics, vol. 11, no. 19, p. 3228, Oct. 2022, doi: 10.3390/electronics11193228.
[12] P. Selvaraj, G. NC, P. Kumar, and M. Khapra, “OpenHands: Making Sign Language Recognition Accessible with Pose-based Pretrained Models across Languages,” Oct. 12, 2021, arXiv: arXiv:2110.05877. doi: 10.48550/arXiv.2110.05877.
[13] K. Roh, H. Lee, E. J. Hwang, S. Cho, and J. C. Park, “Preprocessing Mediapipe Keypoints with Keypoint Reconstruction and Anchors for Isolated Sign Language Recognition”.
[14] R. M. Abdulhamied, M. M. Nasr, and S. N. A. Kader, “Real-time recognition of American sign language using long-short term memory neural network and hand detection,” Indones. J. Electr. Eng. Comput. Sci., vol. 30, no. 1, Art. no. 1, Apr. 2023, doi: 10.11591/ijeecs.v30.i1.pp545-556.
[15] P. Sheth and S. Rajora, Sign Language Recognition Application Using LSTM and GRU (RNN). 2023. doi: 10.13140/RG.2.2.18635.87846.
[16] Y. Obi, K. Claudio, V. Budiman, S. Achmad, and A. Kurniawan, “Sign language recognition system for communicating to people with disabilities,” Procedia Comput. Sci., vol. 216, pp. 13–20, Jan. 2023, doi: 10.1016/j.procs.2022.12.106.
[17] R. S. L. Murali, “Sign Language Recognition System Using Convolutional Neural Network And ComputerVision,” vol. 4, no. 4, 2022.
[18] D. Bragg et al., “Sign Language Recognition, Generation, and Translation: An Interdisciplinary Perspective,” in Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility, in ASSETS ’19. New York, NY, USA: Association for Computing Machinery, Oct. 2019, pp. 16–31. doi: 10.1145/3308561.3353774.
[19] N. C. Camgoz, O. Koller, S. Hadfield, and R. Bowden, “Sign Language Transformers: Joint End-to-End Sign Language Recognition and Translation,” presented at the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 10023–10033. Accessed: Nov. 11, 2024. [Online]. Available: https://openaccess.thecvf.com/content_CVPR_2020/html/Camgoz_Sign_Language_Transformers_Joint_End-to-End_Sign_Language_Recognition_and_Translation_CVPR_2020_paper.html
[20] S. Jiang, B. Sun, L. Wang, Y. Bai, K. Li, and Y. Fu, “Skeleton Aware Multi-Modal Sign Language Recognition,” presented at the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 3413–3423. Accessed: Nov. 11, 2024. [Online]. Available: https://openaccess.thecvf.com/content/CVPR2021W/ChaLearn/html/Jiang_Skeleton_Aware_Multi-Modal_Sign_Language_Recognition_CVPRW_2021_paper.html
[21] H. Zhou, W. Zhou, Y. Zhou, and H. Li, “Spatial-Temporal Multi-Cue Network for Continuous Sign Language Recognition,” Proc. AAAI Conf. Artif. Intell., vol. 34, no. 07, Art. no. 07, Apr. 2020, doi: 10.1609/aaai.v34i07.7001.
[22] M. Ismail, S. Dawwd, and F. Ali, “Static hand gesture recognition of Arabic sign language by using deep CNNs,” Indones. J. Electr. Eng. Comput. Sci., vol. 24, p. 178, Oct. 2021, doi: 10.11591/ijeecs.v24.i1.pp178-188.
[23] D. Li, C. Rodriguez, X. Yu, and H. Li, “Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison,” presented at the Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2020, pp. 1459–1469. Accessed: Nov. 11, 2024. [Online]. Available: https://openaccess.thecvf.com/content_WACV_2020/html/Li_Word-level_Deep_Sign_Language_Recognition_from_Video_A_New_Large-scale_WACV_2020_paper.html
Published
2025-03-12
Section
Articles
Copyright (c) 2025 Dadang Iskandar Mulyana, Guruh Taruno Putra
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
