DEV Community: Iyanuoluwa Enoch Oke

I Built a BSL Translation Extension That Nobody Asked For. Here Is Why I Did It Anyway.

Iyanuoluwa Enoch Oke — Sat, 01 Aug 2026 10:00:00 +0000

Late last year I started building something that had no job description attached to it, no business case ready, and no guarantee that anyone would use it.

I built Signlytic, a Chrome extension that intercepts live captions from streaming platforms and translates them into British Sign Language signing animations in real time, rendered as a floating overlay panel directly on top of the video.

Not because someone commissioned it. Not because it was the safe, hireable thing to build. Because it felt like a problem worth solving and because I wanted to know if it was even technically possible.

This is the story of what I built, how I built it, what repeatedly broke, and what the whole experience taught me about the kind of engineering that actually matters.

The problem I kept thinking about

British Sign Language is the primary language of a significant deaf community in the UK. Most digital content is not natively accessible in BSL. Subtitles exist on most platforms, but reading subtitles while watching video is a fundamentally different cognitive experience from watching sign language, which is the natural language for many BSL users.

The question I kept sitting with was this: what would it look like if a browser could intercept live captions and render them as BSL signing in real time, without requiring any changes from the streaming platform, without cloud APIs, and without meaningful delay?

There was no off-the-shelf answer. So I started building one.

The architecture

Signlytic is a multi-model pipeline connecting several distinct systems:

Sign recognition: A Video-SWIN-T model trained to 100% Top-1 accuracy on 5,203 BSL signs, handling the recognition direction for BSL input.

Gloss translation: Groq-hosted Llama 3.3 70B converting English captions to BSL gloss, the written intermediate representation of sign language grammar that bridges English word order and BSL sentence structure.

Voice synthesis: Coqui XTTS v2 for speech output in the reverse direction of the pipeline.

The Chrome extension: Captures live captions across streaming platforms, renders BSL animations as a floating panel, and handles fallback when captions are unavailable.

The four components talk to each other through a local FastAPI backend that manages the pipeline and keeps the extension from needing cloud access for core functionality.

How caption detection actually works

The first engineering problem was fundamental: how do you reliably intercept captions across platforms that all structure their DOM differently?

The answer is a MutationObserver watching the document body for text node additions, with platform-specific filtering to reduce false positives:

const observer = new MutationObserver((mutations) => {
    for (const mutation of mutations) {
        for (const node of mutation.addedNodes) {
            if (node.nodeType === Node.TEXT_NODE) {
                const text = node.textContent.trim();
                if (text.length > 2) {
                    sendToTranslationPipeline(text);
                }
            }
        }
    }
});

observer.observe(document.body, {
    childList: true,
    subtree: true,
    characterData: true
});

The subtree: true flag is not optional. Caption nodes on most streaming platforms are deeply nested inside shadow DOM structures that change between platform updates. Targeting a specific element selector breaks the moment the platform updates their frontend. Observing the full document body and filtering by text content is more resilient, even though it means processing more mutation events.

When captions are not available at all, the extension falls back to the Web Speech API using the en-GB locale. This covers cases like live streams without auto-captions, or platforms the MutationObserver has not been configured for yet.

How the gloss translation works

Raw English caption text cannot be directly rendered as BSL. BSL has its own grammar, its own word order, and its own sentence structure that differs from English in significant ways. The intermediate representation is called gloss, a written form that maps English concepts to BSL grammar.

The Llama 3.3 70B model running on Groq handles this translation step. A simplified version of that API call looks like this:

from groq import Groq

client = Groq()

def translate_to_bsl_gloss(english_text: str) -> str:
    response = client.chat.completions.create(
        model="llama-3.3-70b-versatile",
        messages=[
            {
                "role": "system",
                "content": (
                    "You are a British Sign Language linguist. "
                    "Convert the following English text to BSL gloss notation. "
                    "Preserve meaning. Adjust word order to BSL grammar. "
                    "Output gloss only, no explanation."
                )
            },
            {"role": "user", "content": english_text}
        ]
    )
    return response.choices[0].message.content.strip()

The system prompt is doing significant work here. Without explicit instruction to output gloss only, the model tends to explain its translation choices. In a real-time pipeline with caption text arriving every few seconds, you need clean output on every call.

What broke repeatedly

Platform inconsistency was relentless. YouTube structures its caption DOM differently from Netflix, which differs from BBC iPlayer, which differs from Amazon Prime. There is no standard. Each platform required individual testing and observer tuning. Some platforms dynamically recreate their caption containers on seek or quality change, which kills a targeted observer but survives a body-level one.

Timing and queue management were harder than expected. Live captions arrive in short, irregular bursts. BSL animation rendering takes time, especially for 3D Mixamo avatar playback via Three.js. If the animation queue grows faster than it can drain, the signing falls behind the audio and the experience breaks down completely.

The fix was a priority queue that drops the oldest pending animation segment when the queue depth exceeded a threshold, similar to the bounded queue pattern I used in my ROS 2 perception work. In real-time human-facing systems, falling behind and catching up is a worse experience than occasionally dropping a segment cleanly.

The gap between recognition accuracy and production performance. The Video-SWIN-T model achieves 100% Top-1 accuracy on the trained sign vocabulary. That number is real. It is also a controlled dataset metric.

Live, unconstrained video input introduces lighting variation, angle variation, background clutter, partial hand occlusion, and signer-to-signer variation that does not exist cleanly in training data. Every computer vision engineer knows this gap exists. Every product builder has to decide honestly how to communicate it. I chose to scope the extension's recognition capabilities explicitly rather than imply a performance that only holds under ideal conditions.

The rendering layer: 2D versus 3D

The extension supports two rendering modes for BSL animation output.

The 2D mode uses MediaPipe Holistic to render pose landmark skeletons, a lightweight representation that works well on lower-end hardware and loads fast. The 3D mode uses Mixamo avatars rendered through Three.js r128 for a more naturalistic signing appearance.

The tradeoff is real. 3D rendering is more legible for BSL users because it more closely resembles actual human signing. But it is heavier, slower to initialise, and can introduce rendering latency that compounds with translation latency if the system is already under load.

The draggable and resizable overlay panel is built using an iframe-based architecture inside Chrome MV3. This is the current Chrome extension manifest standard, and it changes how content scripts, service workers, and injected iframes communicate compared to MV2. That migration from MV2 patterns to MV3 constraints was its own engineering challenge that deserved more preparation time than I gave it initially.

What this project taught me about multi-model AI systems

Multi-model pipelines fail at the connections, not inside the models. The Video-SWIN-T, the Llama translation layer, and the XTTS synthesis are all individually strong. The hard engineering is connecting them without introducing unacceptable cumulative latency, handling failure at each handoff gracefully, and making the end-to-end system feel coherent to someone using it in real time.

Fallback design is product design. What happens when captions are unavailable? What happens when the translation is ambiguous? What happens when the animation queue falls behind? If you have not designed the failure states, you have not designed the product. You have designed the happy path and called it done.

Accessibility is a technical discipline. It is not enough to have good intentions and an interesting architecture. Accessible technology requires rigorous engineering around latency, consistency, reliability, and graceful degradation. The users who depend on it deserve systems that are actually dependable, not just impressively described.

174 offline signs versus 5,203 via backend. The extension ships 174 BSL signs bundled for completely offline use. The full 5,203-sign vocabulary requires the local FastAPI backend to be running. This two-tier architecture was a deliberate product decision: the extension should work in a degraded but functional state without any setup, and scale up when the backend is available. That kind of progressive capability is harder to build than a single-mode system but meaningfully better for real users.

What comes next

A Windows desktop application built in Electron is in development. The goal is system-wide caption capture across any application, not just browser-based streaming. This requires a fundamentally different approach to caption interception since there is no DOM to observe, which means working with Windows accessibility APIs and screen reader hooks.

The longer-term vision is a fully bidirectional BSL communication system: sign-to-speech for BSL users communicating outward and speech-to-sign for hearing people communicating inward, running locally without cloud dependency.

I am documenting the engineering on GitHub as I go. If you are working in accessibility technology, computer vision, browser extension development, or local AI deployment, I would genuinely like to connect.

Closing thought

There is a category of project nobody has a template for. No tutorial, no course, no job description points you toward it. You find it by sitting with a question long enough that building becomes the only honest answer.

Signlytic was that kind of project for me.

It is technically demanding, genuinely unfinished, and one of the most meaningful things I have built. Not because it is perfect. Because the problem is real, the technology is ready enough to try, and the engineering forces you to think at a level that standard work never demands.

Build the thing you cannot stop thinking about.

The engineering will follow.

I Built a Real-Time Perception Stack on CARLA. Here Is What Actually Happened.

Iyanuoluwa Enoch Oke — Tue, 21 Jul 2026 16:03:28 +0000

I want to tell you about the three weeks I spent making YOLO and ROS 2 talk to each other inside a simulator, and why it was nothing like the tutorials made it look.

This is not a clean architecture post. This is a "here is what broke, here is what I changed, here is what I actually learned" post.

If you are building robotics perception systems, or you are just curious what it looks like to wire YOLOv11, ByteTrack, StrongSORT, CARLA, and ROS 2 into something that runs in real time, this one is for you.

What I was building and why

The goal was a real-time robotic perception system that could:

detect objects in a moving vehicle simulation
track those objects across frames with stable identities
feed structured perception data into a ROS 2 navigation stack
run fast enough to be genuinely useful for autonomous decision making

CARLA was the simulator. YOLOv11 was the detector. ByteTrack and StrongSORT handled multi-object tracking. ROS 2 was the middleware. ONNX and TensorRT handled edge optimisation.

On paper this is a well-defined problem. In practice, every one of those components has opinions about data formats, timing, and threading that you only discover by running them together.

Problem one: getting camera data out of CARLA cleanly

CARLA has its own Python API and its own event-driven callback system. ROS 2 has its own message types, publisher model, and executor. Getting them to cooperate without dropping frames or blocking the main thread is not automatic.

Here is a simplified version of how I handled the camera subscription and published frames into ROS 2:

import carla
import rclpy
from rclpy.node import Node
from sensor_msgs.msg import Image
from cv_bridge import CvBridge
import numpy as np

class CARLACameraNode(Node):
    def __init__(self):
        super().__init__('carla_camera_node')
        self.publisher = self.create_publisher(Image, '/carla/camera/rgb', 10)
        self.bridge = CvBridge()

    def camera_callback(self, image):
        array = np.frombuffer(image.raw_data, dtype=np.uint8)
        array = array.reshape((image.height, image.width, 4))
        bgr = array[:, :, :3]
        msg = self.bridge.cv2_to_imgmsg(bgr, encoding='bgr8')
        msg.header.stamp = self.get_clock().now().to_msg()
        self.publisher.publish(msg)

The key thing I got wrong initially: I was not stamping the header with the ROS 2 clock.

That one missing line caused timestamp mismatches downstream that made the tracker behave erratically. ByteTrack and StrongSORT both calculate displacement and velocity across frames using time deltas. If those deltas are wrong or inconsistent, the association algorithm produces noisy, unstable tracks that look like a model problem but are actually a data problem.

That bug cost me two days.

Add the timestamp. Always. It is not optional in robotics.

Problem two: YOLO runs fast, ROS 2 callbacks do not wait

YOLOv11 at a reasonable resolution on a GPU is quick. But "quick" in model benchmarking terms and "quick" in robotics middleware terms are genuinely different things.

ROS 2 uses a callback-based execution model. If your inference call is blocking the subscription callback thread, you are missing messages while you compute. The subscriber queue fills. You start processing frames that are already stale by the time inference finishes. Your tracker is now seeing the world as it was, not as it is.

The fix was decoupling ingestion from inference using a bounded queue and a dedicated inference thread:

import threading
import queue
from rclpy.node import Node
from sensor_msgs.msg import Image

class PerceptionNode(Node):
    def __init__(self):
        super().__init__('perception_node')
        self.frame_queue = queue.Queue(maxsize=5)
        self.subscription = self.create_subscription(
            Image,
            '/carla/camera/rgb',
            self.image_callback,
            10
        )
        self.inference_thread = threading.Thread(
            target=self.run_inference,
            daemon=True
        )
        self.inference_thread.start()

    def image_callback(self, msg):
        if not self.frame_queue.full():
            self.frame_queue.put(msg)

    def run_inference(self):
        while rclpy.ok():
            msg = self.frame_queue.get()
            # run YOLOv11 inference here

The maxsize=5 on the queue is deliberate. When inference cannot keep up, I drop old frames rather than accumulate a growing backlog of stale images.

This is a real-time systems principle worth internalising: a stale frame processed late is usually worse than a dropped frame. Timeliness matters as much as completeness in a live perception pipeline.

Problem three: tracking across frames is not free

ByteTrack and StrongSORT both work by associating new detections to existing tracks across frames using motion models and, in StrongSORT's case, appearance embeddings. This sounds clean until you hit occlusion, rapidly moving objects, or a detector that briefly misses something due to a lighting change or partial obstruction.

What I observed specifically in CARLA: when the ego vehicle turned sharply, vehicles and pedestrians would leave the camera frustum and re-enter within a few frames. The tracker sometimes assigned fresh IDs to those returning objects, breaking the continuity that downstream navigation reasoning depended on.

The parameters that mattered most here were:

tracker_config = {
    "max_age": 30,        # frames a track survives without a detection match
    "min_hits": 3,        # detections needed before a track is confirmed
    "iou_threshold": 0.3  # minimum overlap for detection-to-track association
}

Increase max_age too much and you get ghost tracks for objects that have genuinely left the scene. Decrease it too much and you lose real tracks during brief occlusions. Tune iou_threshold too high and the tracker drops tracks during fast motion. Too low and it starts merging distinct objects.

There is no universally correct set of values. They depend on your scene density, your detector confidence, your frame rate, and what your downstream system needs from the tracker. The only way to find the right values is to run scenarios, observe failures, and adjust deliberately.

Problem four: edge hardware changes the problem completely

Everything above ran well on my development GPU. Then I exported the model and tested closer to constrained hardware conditions.

The frame rate dropped. The tracker fell behind. The pipeline that felt smooth in development started introducing the kind of latency that makes real-time perception meaningless.

The fix was ONNX export followed by TensorRT optimisation:

import torch

model = load_yolov11("yolov11n.pt")
model.eval()

dummy_input = torch.randn(1, 3, 640, 640).cuda()

torch.onnx.export(
    model,
    dummy_input,
    "perception.onnx",
    opset_version=17,
    input_names=["images"],
    output_names=["output0"],
    dynamic_axes={"images": {0: "batch"}}
)

After TensorRT conversion and INT8 quantisation, inference time dropped enough to bring the full pipeline back into a usable range. But the deeper lesson is not about TensorRT specifically. It is that edge deployment is a different problem to development, and discovering that at the end of the project means redesigning under pressure.

Test on target hardware early. Not day one necessarily, but not month three either.

Problem five: your confidence threshold is a system design decision

This one is subtle and I did not fully appreciate it until I had the full pipeline running.

YOLO outputs detections with confidence scores. It is tempting to pick a threshold like 0.5 and move on. But in a live robotics system, that threshold is not a model hyperparameter. It is an architectural decision about when your robot is allowed to act on what it sees.

A threshold that is too low sends noisy, unreliable detections to the tracker. The tracker generates unstable tracks. The navigation stack gets bad inputs. The robot behaves erratically.

A threshold that is too high causes the detector to miss real objects in challenging conditions. Tracks are lost. The robot navigates as if obstacles do not exist.

What helped was treating the threshold not as a single number but as part of a validation layer:

def is_valid_detection(detection, track_age, platform_stable):
    if detection.confidence < CONFIDENCE_THRESHOLD:
        return False
    if track_age < MIN_TRACK_AGE:
        return False
    if not platform_stable:
        return False
    return True

Multiple signals need to agree before a detection influences navigation. This pattern, requiring consensus across confidence, temporal stability, and platform state, is something I now consider non-negotiable in any perception pipeline with real-world consequences.

Five things I know now that I did not know at the start

Timestamps are not optional. Misaligned timestamps produce bugs that look like model bugs and take days to diagnose.

Decouple your threads early. Blocking the callback thread is an easy mistake with expensive downstream consequences.

The tracker is part of your perception system. Not an add-on you configure at the end. Design around it from day one.

Test on target hardware before you think you are done. Edge deployment is a different engineering problem to development.

Log everything during development. You cannot replay failures you did not record, and in robotics you will need to replay failures.

Closing thought

Building this system taught me that robotics perception is not a model problem. It is a systems problem where the model is one moving part among many.

The interesting engineering is in how you connect those parts, how you handle failures at the boundaries between them, and how you make the full pipeline behave reliably when nothing goes quite according to plan.

Which is always.

If you are building something similar or want to dig into any part of this, drop a comment. I am always happy to talk robotics and perception systems.