Your Chars-Per-Token Estimate Is Wrong (Mine Was, By Exactly One Context Window)

A user saved a long YouTube video. The pipeline pulled the transcript — about 1 MB of text — and sent it to the summarizer, which came back with:

400 Bad Request
This model's maximum context length is 128000 tokens,
however you requested 130700 tokens.

Which is annoying, because the truncation code that exists specifically to prevent this had run. It did its math. It confidently produced a prompt ~2,700 tokens too big.

Two bugs were hiding in that math. Both are the kind that live happily in production until an input arrives that's big enough to find them.

---

Bug 1: one setting doing two jobs it was never meant to do

My AI capabilities — summarize, classify, tag, extract entities, translate — run through tenant-configurable connectors. Each connector has a maxTokens setting.

The handlers were using that one value for two completely different jobs:

// what I had. both of these are wrong.
var maxOutput = connector.MaxTokens;                 // sent to the model
var maxInputChars = connector.MaxTokens * 4 * 3;     // used to truncate input

That second line is a formula with the structure of a guess. But the deeper problem is the first one, and it's conceptual.

A connector is shared. Its maxTokens might be set high because a chat-style consumer somewhere wants long completions. Feed that same number into a summarization call and you have:

1. Reserved an enormous output budget the task will never use,
2. Inflated your input allowance from that number,
3. Blown clean through the context window,

…for a task that was never going to emit more than a page of text.

The fix: budget per capability, from the two numbers that are actually true

public static class AiTokenBudget
{
    // output is a property of the TASK, not of the connector
    private static readonly Dictionary<string, int> OutputCaps = new()
    {
        ["summarize"] = 4_096,   // nobody wants a 20k-token summary
        ["classify"]  = 4_096,
        ["tags"]      = 4_096,
        ["entities"]  = 4_096,
        ["translate"] = 16_384,  // output scales with input here
    };

    private const int ContextWindow = 128_000;
    private const double CharsPerToken = 2.5;   // see Bug 2

    public static int OutputTokens(string capability) =>
        OutputCaps.TryGetValue(capability, out var cap) ? cap : 4_096;

    public static int MaxInputChars(string capability)
    {
        var reserved = OutputTokens(capability) + 1_000;   // prompt overhead
        var inputTokens = ContextWindow - reserved;
        return (int)(inputTokens * CharsPerToken);
    }
}

Output is capped by what the task needs. Input gets whatever room is left in the real context window. The connector's maxTokens goes back to being what it always should have been: a setting for consumers that need it, not a load-bearing constant for every capability in the system.

---

Bug 2: real text is denser than prose

The truncation estimate assumed 3.0 characters per token.

English prose runs around 4. So 3.0 feels comfortably pessimistic. It feels like the safe choice a careful person makes.

Then I actually measured the transcript that failed:

1,048,576 chars / 360,335 tokens = 2.91 chars per token

2.91. Below my "safe" floor.

And of course it is. A timestamped YouTube transcript is not prose:

00:14:22.480 --> 00:14:25.120
- yeah, so the, um, the thing is —
00:14:25.120 --> 00:14:27.900
right, exactly, 40%, roughly

Short lines. Numbers. Timestamps. Speaker fragments. Punctuation everywhere. Exactly the kind of text a BPE tokenizer shreds into tiny pieces. At 1 MB, the difference between "3.0, feels safe" and "2.91, measured" is thousands of tokens — which is to say, it was the entire overflow.

The constant is now 2.5, bought by measuring the input that actually failed rather than by picking a rounder number and feeling better about it.

When a heuristic fails, the fix is a datum, not a more confident-sounding guess.

---

The safety net: halve and retry

Here's the uncomfortable truth sitting underneath both fixes:

Any chars-per-token constant is wrong for some input. Code samples. Dense CJK. Tables. Base64. Some transcript format I haven't met yet. One of them will eventually beat whatever ratio I pick, and I will not know in advance which.

So the last line of defence doesn't estimate at all:

try
{
    return await SummarizeAsync(text, ct);
}
catch (AiContextOverflowException)
{
    // don't re-estimate. just take half.
    var half = text[..(text.Length / 2)];
    _log.LogWarning("Context overflow at {Len} chars; retrying with half.", text.Length);
    return await SummarizeAsync(half, ct);
}

Crude — and that's precisely the virtue. It makes no assumption about why the estimate lost. It converges in a step or two. And it turns "job failed" into "summary of most of the transcript," which for summarization is an easy trade: a slightly shorter input is invisible in the output, while a hard failure is extremely visible in the app.

---

Three layers, and you need all three

1. Prevent the systematic error — per-capability budgets, so one shared setting can't size every call.
2. Narrow the guess — a ratio measured from inputs that actually failed, not one that sounded prudent.
3. Recover without the estimate — halve-and-retry, which is right even when layers 1 and 2 are wrong.

Estimate conservatively. Verify against reality. Keep one recovery path that doesn't depend on your estimate being right.

The 1 MB transcript summarizes fine now.

---

The takeaway you can steal

• Don't derive your input budget from your output setting. They're different numbers with different owners.
• Cap output by capability. A summarizer does not need 16k tokens of headroom.
• Measure your chars/token on the inputs that break you. Your content type has its own density, and it is not prose.
• Keep a dumb recovery path. The clever one will eventually be wrong.