SignSync is the first ever innovative real-time sign language translation tool integrated directly into the open-source video conferencing platform Jitsi. It bridges the communication gap between the Deaf and Hard of Hearing (D/HH) community and the general public by making sign language visible, understandable, and inclusive during online meetings.
This project is also selected for University of Technology Sydney AI Showcase 2025.
Imagine you have lost your voice—how will you express your thoughts and communicate with those around you? This is a daily dilemma for the D/HH community.
While sign language is widely used within the D/HH community, over 95% of the hearing population is unfamiliar with it, leading to communication barriers in schools, workplaces, and social settings.
As part of an inclusive future, SignSync empowers the D/HH community by allowing their expressions to be translated in real time and understood by everyone in a video call.
While tools like Zoom and Microsoft Teams excel in text and voice communication, they lack built-in support for real-time sign language interpretation. That’s the gap we’re aiming to bridge.
🎥 Checkout the full demo video here
demo_vid_git.mp4
https://drive.google.com/file/d/1Oz-TD5LHIdJV16Md-ZBkN2JFY-CPcHRx/view?usp=sharing
🤟 Real-time Sign Language Recognition Translates sign gestures into readable text using a deep learning model (CNN-based).
📡 Seamless Jitsi Integration Embedded into the Jitsi interface with no need for separate software.
🌍 Privacy-Conscious All processing can run locally without uploading sensitive data to external servers.
🖥️ Cross-Platform Works on modern web browsers across operating systems.
Frontend: JavaScript, React (Jitsi Meet UI)
Backend/ML: Python, OpenCV, PyTorch/TensorFlow
Model: Mediapipe for landmakr extraction + CNN-based Sign Language Translation for fingerspelling
The webcam captures sign gestures from the user.
Mediapipe hands is called to extract hand landmarks.
Extracted landmarks are being detected by model hosted on frontend device.
Calling Gemini API to separate glued text and correct wrong words.
Display translated text on subtitle overlay.
By making video communication accessible to the D/HH community, SignSync doesn’t just enable conversation—it promotes inclusion, equity, and visibility.
• Chong Fai (Jason) Wong
LinkedIn | GitHub
• Robayed Ashraf
LinkedIn | GitHub
• Asim Santos Poudel
LinkedIn | GitHub
This project was undertaken under the supervision and mentorship of Dr. Nabin Sharma at the University of Technology Sydney (UTS).
Selected for UTS AI Showcase 2025 https://utsfeitshowcases.org/ai-showcase/