I capped extraction at 250 concepts, told the model to obey the cap, and let the app report the result as complete

This is a submission for DEV's Summer Bug Smash: Smash Stories.

TL;DR. I am building a study app whose entire premise is that it knows what you are about to forget. Extraction caps a source at 250 concepts, for good cost reasons, and I wrote that cap into the system prompt so the model would stop there. It does stop there. Which means a 300 page book comes back as exactly 250 concepts, the hard slice that would have told me something was dropped never fires, and the screen renders "250 concepts extracted." in the same neutral tone it uses for a two page note. The app was not losing your material by accident. It was losing it exactly as instructed, and then reporting a clean number over the top. The fix is one character, and the version almost everyone would write instead is the version that detects nothing.

---

The setup

Vesprit reads your study material and pulls out testable concepts, then tracks which ones are fading from your memory and puts those in front of you. The fade model only knows about concepts that exist in the database. That matters for what follows.

Extraction runs in a Cloudflare Worker against Claude, and it has a ceiling:

/**
 * A ceiling on concepts per source. This is the real cost lever: output tokens
 * cost 5x input, and output scales with concept count rather than source length.
 * Also keeps a single recall backlog from being unusable after one upload.
 */
export const MAX_CONCEPTS = 250;

That comment is mine, it is honest, and I still agree with every word of it. The cap is a deliberate design decision, not a bug. Two things follow from it that I did not think about together until much later: the system prompt tells the model to return at most 250, and the parser ends with a hard slice(0, MAX_CONCEPTS) as a backstop.

The assumption, stated plainly

A documented, intentional limit is a product constraint, not a defect. If it ever actually bites, the slice will be the thing that fires, and that is where I would put the detection.

I never wrote that sentence down. That is rather the point. It sat underneath the design as an unexamined belief, and it is wrong in a specific and interesting way.

Why it was wrong

The slice almost never fires.

The system prompt instructs the model to stop at 250. On a long source the model does what it is told. It returns exactly 250 concepts, parsed.length is exactly 250, and slice(0, 250) removes nothing at all. The backstop I was relying on to notice the problem is a backstop against the model disobeying me, which is the rare case. The common case, the model obeying, produces a response that is byte for byte indistinguishable from complete coverage of a short source.

So the boundary was invisible on both sides. The Worker had no idea it had hit a ceiling, because nothing was cut. The app had no idea, because the Worker only ever sent it a concepts array. The user had no idea, because the screen rendered a count, and a count with no qualifier reads as a total.

And in this product specifically, the consequence is not "a list is short." Material that never entered the system is invisible to the fade model, so the user gets a confident, well ranked revision queue built over a fraction of their syllabus, while every signal the app gives them says coverage is complete. The app's one job is to know what you are missing. It would have been the thing hiding it.

The fix, and the one character that carries it

-  return parsed.slice(0, MAX_CONCEPTS);
+  const truncated = parsed.length >= MAX_CONCEPTS;
+
+  return { concepts: parsed.slice(0, MAX_CONCEPTS), truncated };

>=, not >. That is the whole thing.

> is the version that looks correct in review. It asks "did the slice drop anything," which is the natural question, and it passes a test written by the same person who wrote the check, because that person reaches for the same mental model twice. It would have caught only the case where the model ignored its own instruction, and missed essentially every real truncation.

>= knowingly accepts a false positive. A source that genuinely yields exactly 250 concepts gets flagged as possibly incomplete when it is not. That trade is one sided and I took it on purpose: the cost is one hedged sentence on a rare exact hit, against silently handing someone a fifth of their material and letting them revise from it.

The reasoning is in the file rather than in a commit message nobody will read again, because the next person to look at this line will feel the same pull toward > that I did:

  // -------------------------------------------------------------------------
  // WHY `>=` AND NOT `>`
  //
  // This is the load-bearing line. The system prompt tells the model to return
  // at most MAX_CONCEPTS, so on a large source the model obeys and stops at
  // exactly MAX_CONCEPTS — the slice below never fires, and a `>` test would
  // report nothing. In other words `>` detects only the rare case where the
  // model ignored its own instruction, and misses essentially every real
  // truncation: a 300-page book comes back with 250 concepts, looking for all
  // the world like complete coverage.

The tests name the case rather than describing the mechanics, which is the only reason I trust them here:

it('flags a response that lands exactly on the ceiling', () => {
  // THE case this feature exists for, and the one a `>` comparison misses.
  // The system prompt tells the model to stop at MAX_CONCEPTS, so a large
  // source comes back at exactly the cap with nothing sliced off — which is
  // indistinguishable from complete coverage unless we say so here.
  expect(parseConcepts({ concepts: conceptList(MAX_CONCEPTS) })?.truncated).toBe(true);
});

it('counts survivors, not raw entries', () => {
  // Malformed entries are dropped before the count is taken, so a payload
  // that only reaches the ceiling by including junk is not a truncation.
  const withJunk = [...conceptList(MAX_CONCEPTS - 1), null, { prompt: '', expectedAnswer: 'x' }];

  expect(parseConcepts({ concepts: withJunk })?.truncated).toBe(false);
});

The first of those names is the whole argument. A > implementation passes every other test on this screen.

The part of the fix that nearly caused a worse bug

The flag has to persist. Intake navigates straight to the per source view on success, so a toast would be destroyed by the navigation on the exact path large sources arrive by. That meant a new field on the source document, and a new field means a parser change:

     createdAt,
     status,
+    // Deliberately NOT part of the validation block above. An older document has
+    // no `truncated` field and must still be a perfectly valid source — treating
+    // its absence as malformed would drop every pre-existing row and blank the
+    // timeline, which is precisely what this function's skip-one-row design
+    // exists to prevent.
+    truncated: data.truncated === true,

parseSource returns null for a malformed document and the list path skips nulls, by design, so one corrupt row cannot take down a timeline. Add truncated to the validation block above that line and every document written before this commit becomes malformed. Every row gets skipped. The user's timeline empties itself, silently, with no error anywhere, which is a considerably worse version of the bug I was in the middle of fixing.

It is pinned by a test that exists purely to stop someone tidying that comment away:

it('does NOT reject a document that lacks the field', () => {
  // The failure this guards against is severe and quiet: treating absence as
  // malformed would make parseSource return null for every pre-existing
  // source, and the list path skips nulls — so the user's timeline would
  // simply empty itself with no error anywhere.
  expect(parseSource('src_1', valid)).not.toBeNull();
});

What the user gets, finally, sits directly under the concept count, because that number is where the belief "this is my material" actually forms:

{source.truncated ? (
  <View className="mt-3 rounded-card border border-line bg-canvas-sunken p-gutter">
    <Text className="text-sm font-medium text-ink">Partial coverage</Text>
    <Text className="mt-1 text-sm leading-5 text-ink-muted">
      This source held more testable material than one extraction pass covers, so these{' '}
      {concepts.length} concepts are the most testable ones rather than all of them. Recall
      and fade tracking work normally on what is here.
    </Text>
  </View>
) : null}

It does not promise a fix. Complete coverage of a long source needs chunked extraction, which is scoped and not built, and offering a remedy that does not exist would be worse than the silence it replaces.

The honest note

This was caught by reading my own code while scoping chunked extraction on 2026-08-04, not by a user and not on a device. The app is pre launch, nobody was affected, there was no incident, and no source in the database has ever been long enough to trigger the flag, so the notice above has never rendered outside a fixture. The fix also does nothing until the Worker is redeployed, since the client reads a missing field as false.

What I take from it

An instruction to a model can hide the very boundary it creates. I asked for a limit and I got one, cleanly, every time. Obedience is what made this silent. If I had a sloppier model that overran the cap, my slice backstop would have fired and I would have found this on day one. The better the model followed instructions, the more invisible the problem became, and I do not think I would have predicted that.

The obvious implementation of a check is worth one minute of suspicion. Not the logic, which was fine, but the question it asks. > asks "did I cut anything." >= asks "am I at the edge." Those sound like the same question and they are not, and the difference is the entire feature.

A cap you documented is not a cap the user knows about. I had written the number down three times: in a constant, in a comment, in a system prompt. All three were facing me. None of them were facing them.

Silence is a claim. Rendering "250 concepts extracted." with no qualifier is not a neutral act. It asserts completeness by omission, and I would never have written that assertion out as a sentence.

---

One thing I have not solved, and would genuinely like to hear about.

When a model complies with your own constraint, how do you detect that you were at the boundary at all? The >= trick works here because my ceiling is a discrete count I control. It does not generalise to a token budget, a truncated context window, or a summarisation pass where the model quietly compresses instead of stopping. In all of those, obedience and completeness produce the same shaped output, and I do not have a general technique for telling them apart. Is there a real pattern for this, or is everyone hand rolling a sentinel per call site the way I just did?