SignSetu-AI Powered Sign Language Translation Bridge

SignSetu demonstrates that a coherent, end-to-end sign language translation system can be built using accessible, open-source components and deployed on standard web infrastructure at a cost and complexity level within reach of academic and non-governmental organizations. The system's pipeline — from multi-modal input through Whisper transcription, spaCy-based NLP gloss conversion, and Three.js avatar animation — achieved an overall gloss mapping accuracy of 91.7%, an average end-to-end latency of 1.83 seconds, and a SUS usability score of 78.4, all without requiring users to install any software beyond a standard web browser. The most direct impact of this work is a reusable architectural blueprint. Every component of the SignSetu pipeline — the FastAPI endpoint structure, the NLP grammar rules, the AnimationMixer scheduling logic, and the Gloss Overlay synchronization mechanism — is documented and modular, enabling researchers and developers to extend individual stages without rebuilding the surrounding system. The planned roadmap includes expanding the animation dictionary to over 2,000 ISL and ASL signs, integrating facial expression morphing, adding real-time camera input for bidirectional translation, and pursuing WCAG 2.1 compliance certification. At a broader level, SignSetu challenges the assumption that accessibility features must be retrofitted onto existing systems after the fact. By embedding accessibility as the primary architectural goal rather than a secondary consideration, the project illustrates what becomes possible when engineering students and researchers treat communication equity as a first-class technical problem.

1.       Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. (2022). Robust speech recognition via large-scale weak supervision. arXiv preprint arXiv:2212.04356. 2.       Camgoz, N. C., Hadfield, S., Koller, O., Ney, H., & Bowden, R. (2018). Neural sign language translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 7784–7793. 3.       Camgoz, N. C., Koller, O., Hadfield, S., & Bowden, R. (2020). Sign language transformers: Joint end-to-end sign language recognition and translation. In Proceedings of CVPR, 10023–10033. 4.       Kucherenko, T., Hasegawa, D., Henter, G. E., Kaneko, N., & Kjellstrom, H. (2020). Gesticulator: A framework for semantically-aware speech-driven gesture generation. In Proceedings of the ACM International Conference on Multimodal Interaction, 242–250. 5.       McDonald, J., Wolfe, R., Schnepp, J., Hochgesang, J., Jamrozik, D. G., Stumbo, M., ... & Berke, L. (2016). An automated technique for real-time production of lifelike animations of American Sign Language. Universal Access in the Information Society, 15(4), 551–566. 6.       Kipp, M., Nguyen, Q., Heloir, A., & Matthes, S. (2011). Assessing the deaf user perspective on sign language avatars. In Proceedings of the 13th International ACM SIGACCESS Conference on Computers and Accessibility, 107–114. 7.       Morrissey, S., & Way, A. (2005). An example-based approach to translating sign language. In Proceedings of the 10th Annual Conference of the European Association for Machine Translation, 8.       Kacorri, H., Huenerfauth, M., Ebling, S., Patel, K., & Willard, M. (2015). Demographic and experiential factors influencing acceptance of sign language animation by deaf users. In Proceedings of the 17th International ACM SIGACCESS Conference on Computers and Accessibility, 203–210. 9.       Kennaway, R. (2001). Synthetic animation of deaf signing gestures. In Proceedings of the International Gesture Workshop: Gesture and Sign Language in Human-Computer Interaction, 146–157. Springer. 10.    Bragg, D., Koller, O., Bellard, M., Berke, L., Boudreault, P., Braffort, A., ... & Ladner, R. E. (2019). Sign language recognition, generation, and translation: An interdisciplinary perspective. In Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility, 16–31. 11.    Luzardo, G., Freire, M., Huenerfauth, M., & Kacorri, H. (2015). Toward making sign language synthesis usable: Evaluating and improving avatar quality. In Proceedings of the 17th International ACM SIGACCESS Conference on Computers and Accessibility, 211–218. 12.    World Health Organization. (2021). Deafness and hearing loss fact sheet. Retrieved from https://www.who.int/news-room/fact-sheets/detail/deafness-and-hearing-loss 13.    Sims, E. (2001). 3D sign language synthesis and animation. VCOM3D Technical Report, Orlando, FL. 14.    spaCy Development Team. (2023). spaCy: Industrial-strength Natural Language Processing in Python (v3.6). Explosion AI. https://spacy.io 15.    Three.js Authors. (2023). Three.js — JavaScript 3D Library (r158). https://threejs.org 16.    ISLRTC. (2020). Indian Sign Language Dictionary. Indian Sign Language Research and Training Centre, Ministry of Social Justice and Empowerment, Government of India