DEV Community

Cover image for AI audit: test what your worker writes when the model times out
MatIntellect
MatIntellect

Posted on AI-assisted

AI audit: test what your worker writes when the model times out

Your subtitle worker says "done". A video lands in Drive. Can you tell whether the model corrected the transcript or the worker rendered the original? Both runs are supposed to produce a video. If your smoke test only checks for a file, both runs look identical.

You can check that boundary with a local fixture: give the worker timestamped words, stand in for the proofreader, then compare what reaches the renderer. The example below runs on Python 3.12 with no packages, accounts or network calls. It checks the handoff to the renderer; it doesn't test speech recognition, video encoding or the model's judgment.

Mat adjusting a caged lamp on a steel structure

Keep the input fixed and change the proofreader

My Auto Subtitles workflow takes about 2 minutes from raw clip to finished video. Telegram starts the job, Drive holds the files, Whisper supplies word timings, Claude proofreads the transcript and FFmpeg puts the subtitles on the video. That timing describes the whole workflow. It isn't a timeout setting for the proofreader.

In the proofreader code, correct_words takes word records and an injected correction function. It joins the words for the correction call, then aligns the returned text with the input so downstream code still has timestamped words. If the correction function raises or sends back an empty reply, the original word records go through instead. In a small test, use a one-word typo with a recorded start and end time. A correction from "teh" to "the" should change the text at that position while leaving its time interval alone. After a timeout the same record should still say "teh". If your fixture starts with a clean transcript, you can miss a proofreader that never runs.

Technician examining a circuit board under a microscope, Pexels

Run both paths against the same destination

Save this as audit_render_input.py and run python3 audit_render_input.py without -O so the assertions stay enabled. The local worker is a teaching fixture. It handles one-for-one substitutions and writes JSON where the real worker would hand word records to the renderer. Each invocation gets its own output path, so a passing test can't read a stale artifact left by the previous run.

import json
from pathlib import Path
from tempfile import TemporaryDirectory

original = [("Check", 0.0, 0.25), ("teh", 0.25, 0.45), ("render", 0.45, 0.9)]
expected = [["Check", 0.0, 0.25], ["the", 0.25, 0.45], ["render", 0.45, 0.9]]

def worker(words, proofread, output):
    try:
        reply = proofread(" ".join(word for word, _, _ in words))
    except TimeoutError:
        reply = ""
    edited = reply.split()
    if len(edited) == len(words):
        words = [(word, start, end) for word, (_, start, end) in zip(edited, words)]
    output.write_text(json.dumps(words), encoding="utf-8")

def corrected(text):
    assert text == "Check teh render", text
    return "Check the render"

def timed_out(text):
    raise TimeoutError("fixture: proofreader unavailable")

with TemporaryDirectory() as tmp:
    for name, proofread, want in (
        ("corrected", corrected, expected),
        ("timeout", timed_out, [list(word) for word in original]),
        ("empty", lambda text: "", [list(word) for word in original]),
    ):
        dest = Path(tmp) / f"{name}.json"
        worker(original, proofread, dest)
        got = json.loads(dest.read_text(encoding="utf-8"))
        assert got == want, (name, got, want)
        print(f"{name}: {got[1][0]} [{got[1][1]}, {got[1][2]}]")
Enter fullscreen mode Exit fullscreen mode

Expected output:

corrected: the [0.25, 0.45]
timeout: teh [0.25, 0.45]
empty: teh [0.25, 0.45]
Enter fullscreen mode Exit fullscreen mode

An exception raised by a stub tests what happens after a timeout. It doesn't prove the client will stop waiting. The real helper passes a timeout value to the injected function; the caller has to enforce it. To check that separately, point the client at a local stub that delays its response past the configured deadline. Record when the call starts and when the fallback reaches the renderer. If the client retries, measure the total wait across attempts. A timeout per attempt can still leave the worker busy much longer than you meant it to be.

For a staging run, keep one short test clip and a separate output folder. Start with a fixed proofreader response so you know which word should change. Open the rendered clip and check the line at the expected point in the audio. Then run the same clip with correction unavailable and look for the original line. If the input records match expectations but the words look wrong on screen, investigate the render handoff before trying another model. The local test cannot tell you whether FFmpeg drew a legible subtitle.

A fallback can still be a failed audit

Rendering from the original text is acceptable for this workflow. Apply the same idea to publication and the expected result flips. In my Content Factory, it takes about 5 minutes from idea to publication, with a person approving the draft. For a platform-specific draft the Telegram handler looks up and consumes the pending draft on confirmation before calling the publisher. A preview on screen is therefore a poor substitute for a test of publication: it says the draft exists, not that it was approved or sent.

Workers inspecting and packing items on a production line, Pexels

Use a private test destination. Before approval, the destination should have no item for that draft. After approval, read back the item and compare it with the approved text. For a timeout after a publish request, check whether the destination accepted the item before retrying. A lost reply and a failed write look the same to the caller, but retrying the first one can publish twice. Consuming a pending draft tests a local state transition; it doesn't establish what a remote service did.

If you also want to review the design, I put the checklist behind my /audit command in a separate write-up. For the runtime check, keep the fixture and the expected artifact together. That gives you something concrete to rerun when you swap the proofreader or change how the worker handles a timeout.

This article was created with the help of AI.

Top comments (0)