Late last year I started building something that had no job description attached to it, no business case ready, and no guarantee that anyone would use it.
I built Signlytic, a Chrome extension that intercepts live captions from streaming platforms and translates them into British Sign Language signing animations in real time, rendered as a floating overlay panel directly on top of the video.
Not because someone commissioned it. Not because it was the safe, hireable thing to build. Because it felt like a problem worth solving and because I wanted to know if it was even technically possible.
This is the story of what I built, how I built it, what repeatedly broke, and what the whole experience taught me about the kind of engineering that actually matters.
The problem I kept thinking about
British Sign Language is the primary language of a significant deaf community in the UK. Most digital content is not natively accessible in BSL. Subtitles exist on most platforms, but reading subtitles while watching video is a fundamentally different cognitive experience from watching sign language, which is the natural language for many BSL users.
The question I kept sitting with was this: what would it look like if a browser could intercept live captions and render them as BSL signing in real time, without requiring any changes from the streaming platform, without cloud APIs, and without meaningful delay?
There was no off-the-shelf answer. So I started building one.
The architecture
Signlytic is a multi-model pipeline connecting several distinct systems:
Sign recognition: A Video-SWIN-T model trained to 100% Top-1 accuracy on 5,203 BSL signs, handling the recognition direction for BSL input.
Gloss translation: Groq-hosted Llama 3.3 70B converting English captions to BSL gloss, the written intermediate representation of sign language grammar that bridges English word order and BSL sentence structure.
Voice synthesis: Coqui XTTS v2 for speech output in the reverse direction of the pipeline.
The Chrome extension: Captures live captions across streaming platforms, renders BSL animations as a floating panel, and handles fallback when captions are unavailable.
The four components talk to each other through a local FastAPI backend that manages the pipeline and keeps the extension from needing cloud access for core functionality.
How caption detection actually works
The first engineering problem was fundamental: how do you reliably intercept captions across platforms that all structure their DOM differently?
The answer is a MutationObserver watching the document body for text node additions, with platform-specific filtering to reduce false positives:
const observer = new MutationObserver((mutations) => {
for (const mutation of mutations) {
for (const node of mutation.addedNodes) {
if (node.nodeType === Node.TEXT_NODE) {
const text = node.textContent.trim();
if (text.length > 2) {
sendToTranslationPipeline(text);
}
}
}
}
});
observer.observe(document.body, {
childList: true,
subtree: true,
characterData: true
});
The subtree: true flag is not optional. Caption nodes on most streaming platforms are deeply nested inside shadow DOM structures that change between platform updates. Targeting a specific element selector breaks the moment the platform updates their frontend. Observing the full document body and filtering by text content is more resilient, even though it means processing more mutation events.
When captions are not available at all, the extension falls back to the Web Speech API using the en-GB locale. This covers cases like live streams without auto-captions, or platforms the MutationObserver has not been configured for yet.
How the gloss translation works
Raw English caption text cannot be directly rendered as BSL. BSL has its own grammar, its own word order, and its own sentence structure that differs from English in significant ways. The intermediate representation is called gloss, a written form that maps English concepts to BSL grammar.
The Llama 3.3 70B model running on Groq handles this translation step. A simplified version of that API call looks like this:
from groq import Groq
client = Groq()
def translate_to_bsl_gloss(english_text: str) -> str:
response = client.chat.completions.create(
model="llama-3.3-70b-versatile",
messages=[
{
"role": "system",
"content": (
"You are a British Sign Language linguist. "
"Convert the following English text to BSL gloss notation. "
"Preserve meaning. Adjust word order to BSL grammar. "
"Output gloss only, no explanation."
)
},
{"role": "user", "content": english_text}
]
)
return response.choices[0].message.content.strip()
The system prompt is doing significant work here. Without explicit instruction to output gloss only, the model tends to explain its translation choices. In a real-time pipeline with caption text arriving every few seconds, you need clean output on every call.
What broke repeatedly
Platform inconsistency was relentless. YouTube structures its caption DOM differently from Netflix, which differs from BBC iPlayer, which differs from Amazon Prime. There is no standard. Each platform required individual testing and observer tuning. Some platforms dynamically recreate their caption containers on seek or quality change, which kills a targeted observer but survives a body-level one.
Timing and queue management were harder than expected. Live captions arrive in short, irregular bursts. BSL animation rendering takes time, especially for 3D Mixamo avatar playback via Three.js. If the animation queue grows faster than it can drain, the signing falls behind the audio and the experience breaks down completely.
The fix was a priority queue that drops the oldest pending animation segment when the queue depth exceeded a threshold, similar to the bounded queue pattern I used in my ROS 2 perception work. In real-time human-facing systems, falling behind and catching up is a worse experience than occasionally dropping a segment cleanly.
The gap between recognition accuracy and production performance. The Video-SWIN-T model achieves 100% Top-1 accuracy on the trained sign vocabulary. That number is real. It is also a controlled dataset metric.
Live, unconstrained video input introduces lighting variation, angle variation, background clutter, partial hand occlusion, and signer-to-signer variation that does not exist cleanly in training data. Every computer vision engineer knows this gap exists. Every product builder has to decide honestly how to communicate it. I chose to scope the extension's recognition capabilities explicitly rather than imply a performance that only holds under ideal conditions.
The rendering layer: 2D versus 3D
The extension supports two rendering modes for BSL animation output.
The 2D mode uses MediaPipe Holistic to render pose landmark skeletons, a lightweight representation that works well on lower-end hardware and loads fast. The 3D mode uses Mixamo avatars rendered through Three.js r128 for a more naturalistic signing appearance.
The tradeoff is real. 3D rendering is more legible for BSL users because it more closely resembles actual human signing. But it is heavier, slower to initialise, and can introduce rendering latency that compounds with translation latency if the system is already under load.
The draggable and resizable overlay panel is built using an iframe-based architecture inside Chrome MV3. This is the current Chrome extension manifest standard, and it changes how content scripts, service workers, and injected iframes communicate compared to MV2. That migration from MV2 patterns to MV3 constraints was its own engineering challenge that deserved more preparation time than I gave it initially.
What this project taught me about multi-model AI systems
Multi-model pipelines fail at the connections, not inside the models. The Video-SWIN-T, the Llama translation layer, and the XTTS synthesis are all individually strong. The hard engineering is connecting them without introducing unacceptable cumulative latency, handling failure at each handoff gracefully, and making the end-to-end system feel coherent to someone using it in real time.
Fallback design is product design. What happens when captions are unavailable? What happens when the translation is ambiguous? What happens when the animation queue falls behind? If you have not designed the failure states, you have not designed the product. You have designed the happy path and called it done.
Accessibility is a technical discipline. It is not enough to have good intentions and an interesting architecture. Accessible technology requires rigorous engineering around latency, consistency, reliability, and graceful degradation. The users who depend on it deserve systems that are actually dependable, not just impressively described.
174 offline signs versus 5,203 via backend. The extension ships 174 BSL signs bundled for completely offline use. The full 5,203-sign vocabulary requires the local FastAPI backend to be running. This two-tier architecture was a deliberate product decision: the extension should work in a degraded but functional state without any setup, and scale up when the backend is available. That kind of progressive capability is harder to build than a single-mode system but meaningfully better for real users.
What comes next
A Windows desktop application built in Electron is in development. The goal is system-wide caption capture across any application, not just browser-based streaming. This requires a fundamentally different approach to caption interception since there is no DOM to observe, which means working with Windows accessibility APIs and screen reader hooks.
The longer-term vision is a fully bidirectional BSL communication system: sign-to-speech for BSL users communicating outward and speech-to-sign for hearing people communicating inward, running locally without cloud dependency.
I am documenting the engineering on GitHub as I go. If you are working in accessibility technology, computer vision, browser extension development, or local AI deployment, I would genuinely like to connect.
Closing thought
There is a category of project nobody has a template for. No tutorial, no course, no job description points you toward it. You find it by sitting with a question long enough that building becomes the only honest answer.
Signlytic was that kind of project for me.
It is technically demanding, genuinely unfinished, and one of the most meaningful things I have built. Not because it is perfect. Because the problem is real, the technology is ready enough to try, and the engineering forces you to think at a level that standard work never demands.
Build the thing you cannot stop thinking about.
The engineering will follow.
Top comments (0)