Prompt engineering couldn't fix this LLM bug. 20 lines of binary parsing did.

Joel — Wed, 05 Aug 2026 02:23:37 +0000

New here (hey 👋) and wanted to share something. We recently made public an ad intel app we'd been using internally in our agency. The app/system runs video ads through Gemini to pull out a transcript, the hook, and a few timed moments. It works fairly well - especially for ideation. The transcription is genuinely accurate.

A user opened a 2:22 video and saw a moment marked at 3:37.

Not a rounding error. Fifty percent past the end of a video the user was looking at.

The part that makes it interesting

The obvious guess is that the model didn't know how long the video was. That would be a reasonable bug and an easy fix.

But in the same JSON response, Gemini reported duration_seconds correctly.

We measured it properly on one ad, four identical runs, same file, same prompt:

Real duration: 142.08 seconds
duration_seconds reported by the model: correct, every run
Transcript timestamps: drifted past the end on 1 run in 4, reaching t=214

The words were right. The clock was 50% long. The model was holding the correct duration and writing impossible timestamps anyway.

That is not a knowledge problem. It's a consistency problem, and those don't respond to being asked nicely.

Prompt hardening didn't work

We tried the things you'd try. State the duration in the prompt. Tell it no timestamp may exceed that. Restate the constraint after the schema. Put it in caps.

Failure rate went down. It didn't go to zero. A hardened run still drifted to t=218 on a 142-second video.

This is where I'd argue the general lesson lives:

If a model can violate a constraint, prompting reduces how often it does. Only code makes it impossible.

For most output that trade is fine, because most output is hard to falsify. Nobody checks whether a "psychological driver" is really the third most important one.

Timestamps are different. A timestamp is falsifiable in one click. The user scrubs to 3:37 on a 2:22 video and the whole analysis loses credibility, including all the parts that were correct. One checkable wrong number poisons everything unverifiable around it.

So we needed a ground truth the model couldn't wander away from.

Ground truth: read it out of the file

The video is already in memory as a Buffer before it goes anywhere near the model. MP4 files carry their own duration in the mvhd atom (movie header). No API call, no dependency, and ffprobe isn't installed on our host anyway.

The layout after the 4-byte mvhd type is: version (1 byte), flags (3), then for version 0 — created (4), modified (4), timescale (4), duration (4). For version 1, the timestamps are 8 bytes and duration is 8.

Duration in seconds is duration / timescale.

function mp4DurationSeconds(buf: Buffer): number | null {
  const limit = buf.length - 24;
  for (let i = 0; i < limit; i++) {
    // scan for the ASCII bytes 'm','v','h','d'
    if (buf[i] !== 0x6d || buf[i+1] !== 0x76 || buf[i+2] !== 0x68 || buf[i+3] !== 0x64) continue;

    const version = buf[i + 4];
    try {
      const base = i + 8; // past 'mvhd' + version(1) + flags(3)
      if (version === 1) {
        const timescale = buf.readUInt32BE(base + 16);
        const duration  = Number(buf.readBigUInt64BE(base + 20));
        if (timescale > 0 && duration > 0) return duration / timescale;
      } else {
        const timescale = buf.readUInt32BE(base + 8);
        const duration  = buf.readUInt32BE(base + 12);
        if (timescale > 0 && duration > 0) return duration / timescale;
      }
    } catch {
      // malformed atom, keep scanning for a valid one
    }
  }
  return null;
}

Two deliberate choices worth explaining.

We scan for the atom rather than walking the box tree. Walking is more correct in principle. Scanning handles both faststart files (moov at the front) and files with moov at the end, without implementing a parser for a container format we only need one number out of.

A malformed atom keeps the loop going instead of throwing. The four bytes mvhd can appear inside compressed video data by coincidence. If you bail on the first match that doesn't parse, a random byte sequence takes down your duration check.

Then enforce it, rather than trusting the enforcement

With a real ceiling, every timestamp gets validated server-side before it reaches the client. Same doctrine as validating any untrusted input, because that's what model output is.

The interesting decisions were about what to do with a bad value, not how to spot one.

Keep the words, drop the clock. The transcription is accurate. Only the timing is untrustworthy. So an impossible timestamp is deleted and the text survives. Our transcript component already renders a blank gutter for a missing t, so a stripped timestamp degrades into a clean script instead of a wrong number.

Null both ends of a pair, not just the bad one. We track "curiosity loops" with an open and a close time. If the close is impossible, nulling only that one leaves a surviving open time next to a blank close, which renders as precision we don't have. Both go.

The bug inside the fix

This one cost me an afternoon and it's pure JavaScript, nothing to do with AI.

The first version coerced timestamps with Number(x). Looks harmless:

// WRONG
const t = Number(chunk.t);

Number(null) is 0.

A model that honestly returns null for "I don't know when this happened" gets that turned into 0, which renders as 0:00, which then reads as a loop closing before it opens, which trips the validation that was supposed to be protecting us. An honest null became a confident lie in one implicit coercion.

const asSeconds = (x: any): number | null => {
  if (x === null || x === undefined || (typeof x === 'string' && !x.trim())) return null;
  const n = Number(x);
  return Number.isFinite(n) ? n : null;
};

Caught by a unit test, thankfully, and not by a user. If you're sanitising LLM output, null and 0 mean completely different things and JavaScript will happily merge them for you.

Not all bad timestamps mean the same thing

The last piece is the one I'd have got wrong without looking at the actual failures.

The observed mode is a drifting tail. The model transcribes accurately the whole way through, but its clock stretches as it goes, so the late chunks are impossible while every earlier timestamp is fine. Those early ones are good data and worth keeping.

Scattered bad timestamps through the body are a different signal entirely. That means the timing pass itself is unreliable, and presenting a mix of right and wrong as though it were precise is worse than presenting nothing.

So: a contiguous run of bad timestamps ending at the final chunk is treated as tail drift and trimmed. Anything else strips all timings.

My first instinct was a percentage threshold, something like "if more than 25% are impossible, drop everything". That's wrong, and the reason is worth internalising:

The same failure would be judged differently depending on how many chunks the model happened to emit. A five-chunk drift is 19% of a 26-chunk transcript and 28% of an 18-chunk one. Identical failure, opposite verdicts, decided by something you don't control. Threshold on the shape of the failure, not its proportion.

What I'd take from this

A model can hold a fact and violate it in the same response. Reporting the duration correctly told us nothing about whether it would respect it.

Prompting sets a tendency. Code sets a guarantee. For anything a user can check in one click, you need the guarantee.

Find the ground truth in the artefact you already have. The buffer was in memory the whole time. We were asking a language model for a number that was sitting in a binary header 20 lines of code away.

Partial output usually beats no output, if you're honest about which part you dropped. The words were always right. Only the clock was wrong. Shipping an accurate transcript with no timestamps is better than shipping either a wrong timestamp or nothing at all.

I build SOCIALFUEL, which pulls the live ads any brand is running and decodes what makes them work. The video pipeline behind it is where this bug lived. If you're doing timed analysis on video with an LLM, I'd check your timestamps against the container before you trust them.

DEV Community: Joel