48-Hour Field Notes: My Chunker Cut JSON in Half, and Every Retry Made It Worse

The question I gave myself

Could I run a 40-document extraction batch overnight without a paid model key, and without waking up to a folder of unparseable text?

That sounds like a tooling question, but over two days it turned out to be three smaller questions instead: how large should a chunk be, what exactly counts as a valid answer, and what should happen when the answer is invalid?

I blocked off 48 hours, kept a log, and wrote down every change I made — including the two that made the pipeline measurably worse before it got better.

Constraints I refused to relax

• No paid key, no GPU, and no local weights large enough to be useful on the machine I actually had.
• Every extracted record had to pass a schema check before it touched SQLite.
• The whole pipeline had to be re-runnable offline, so one bad night would not cost me a second night.
• Nothing was allowed to depend on a model endpoint staying warm for the entire batch.

For the overnight runs I used MonkeyCode's free model access, and I deployed the harness on its free server option so my laptop could stay closed. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Free-tier availability can change at any time, which is precisely why rule four sits in that list.

Hours 0–6: my chunker measured characters, the model counted tokens

The first six hours destroyed my confidence in a single constant. I sized chunks by character count, assumed 3.2 characters per token, and cut each document into pieces I believed were safely inside my context budget.

They were not, and here is the part that stung: the failure produced no error message at all. The model simply stopped mid-object, and my parser later received something like {"records": [{"id": "INV-118 — a valid prefix, invalid JSON, and no exception until far downstream.

What made diagnosis harder? Truncation only appeared on pages with long numeric tables, so for hours it looked like a content problem rather than a sizing problem. Isn't that always the shape of a sizing bug — it hides behind the data that happens to be longest?

The guard I added afterwards

# harness/guard.py
CHARS_PER_TOKEN = 3.2  # deliberately pessimistic; overestimating here truncates JSON

def chunk_text(text: str, budget_tokens: int = 1200) -> list[str]:
    size = max(400, int(budget_tokens * CHARS_PER_TOKEN))
    return [text[i:i + size] for i in range(0, len(text), size)]

def assert_estimate_holds(chunk: str, reported_prompt_tokens: int) -> None:
    estimate = len(chunk) / CHARS_PER_TOKEN
    if reported_prompt_tokens > estimate:
        raise BudgetMiss(
            f'~{estimate:.0f} tokens estimated, provider counted {reported_prompt_tokens}'
        )

I now call assert_estimate_holds on the first three chunks of every run, using the token count the provider itself reports back. If a chunk trips the guard, I halve the budget and re-chunk from the original text instead of trimming a half-written object afterwards.

Hours 6–18: a strict parser told me almost nothing useful

My second mistake was treating validation as a boolean. I wrote a check that returned True or False, logged invalid, and moved on to regeneration.

That single design choice cost me most of the second afternoon. A boolean tells you that you must spend another generation, while a path-level error tells you what to put in the repair prompt — and those are very different costs when you are trying to stay inside free access.

# harness/schema.py
from jsonschema import Draft202012Validator

SCHEMA = {
    'type': 'object',
    'required': ['records'],
    'additionalProperties': False,
    'properties': {
        'records': {
            'type': 'array',
            'items': {
                'type': 'object',
                'required': ['id', 'amount', 'currency'],
                'additionalProperties': False,
                'properties': {
                    'id': {'type': 'string', 'minLength': 1},
                    'amount': {'type': 'number'},
                    'currency': {'type': 'string', 'pattern': '^[A-Z]{3}$'},
                },
            },
        },
    },
}

_VALIDATOR = Draft202012Validator(SCHEMA)

def problems_in(payload) -> list[str]:
    if payload is None:
        return ['response is not parseable JSON']
    out = []
    for err in sorted(_VALIDATOR.iter_errors(payload), key=lambda e: list(e.path)):
        path = '/'.join(str(p) for p in err.path) or '<root>'
        out.append(f'{path}: {err.message}')
    return out

The payoff is a message like records/3/currency: 'usd' does not match '^[A-Z]{3}$'. That string is not a diagnostic for me — it is the literal content of my next prompt.

Hours 18–32: I regenerated on failure and paid twice

My first retry strategy re-sent the original prompt unchanged. Honestly, that is not a retry; it is a reroll, and it throws away the only new information you have, which is why the previous answer was rejected.

The fix is small, but it changes the cost profile of the whole batch. Instead of describing the task again, I append the specific problems and ask for a corrected object only.

# harness/extract.py
from .schema import problems_in

PROMPT = '''Extract line items from the text below.
Return one JSON object with a 'records' array; each record needs id, amount, currency.
Text:
{chunk}'''

REPAIR = '''
Your previous answer was rejected for these reasons:
{problems}
Return only the corrected JSON object, with no commentary and no code fence.'''

def extract(client, chunk: str, max_repairs: int = 2):
    base = PROMPT.format(chunk=chunk)
    prompt = base
    for attempt in range(max_repairs + 1):
        raw = client.complete(prompt)
        payload = parse_or_none(raw)
        problems = problems_in(payload)
        if not problems:
            return payload, attempt
        prompt = base + REPAIR.format(problems='\n'.join(f'- {p}' for p in problems))
    raise Unrepairable(chunk_id=chunk[:24], last_prompt=prompt)

Note the cap, because the cap is the interesting part. Two repairs handles shape errors well and handles genuinely ambiguous documents not at all — and those two cases deserve different treatment. After two failures I mark the chunk needs_review and keep going, because a document my prompt cannot describe is not a parsing problem I should solve with more retries.

Hours 32–48: fixture replay made the test suite free

The last improvement is the one I would keep if I could keep only one. I keyed every recorded response by a hash of the exact prompt, so repair prompts get their own fixtures automatically and nothing has to be matched by hand.

# harness/fixtures.py
import hashlib, pathlib

def key_for(prompt: str) -> str:
    return hashlib.sha256(prompt.encode()).hexdigest()[:16]

class ReplayClient:
    def __init__(self, root: pathlib.Path):
        self.root = root
    def complete(self, prompt: str) -> str:
        return (self.root / f'{key_for(prompt)}.txt').read_text()

class RecordingClient:
    def __init__(self, inner, root: pathlib.Path):
        self.inner, self.root = inner, root
    def complete(self, prompt: str) -> str:
        raw = self.inner.complete(prompt)
        (self.root / f'{key_for(prompt)}.txt').write_text(raw)
        return raw

python -m venv .venv && . .venv/bin/activate
pip install jsonschema

# once, against live free model access
HARNESS_MODE=record python -m harness.run --input docs/ --fixtures fixtures/

# forever after: no network, no quota, same assertions
HARNESS_MODE=replay python -m harness.run --input docs/ --fixtures fixtures/

The test plan I actually ran

1. Record one live pass and commit the fixture files, so the recorded shapes are reviewable in a diff.
2. Replay the whole batch and assert that every response which passed once still passes after a schema edit.
3. Inject five mutations — fenced block, dropped required field, stringified amount, trailing prose, empty records — and assert the repair loop names the right path each time.
4. Assert that the repair cap fires on a deliberately ambiguous fixture instead of looping.
5. Time-box one live smoke run per change, not per commit, and never let it gate CI.

I did count how many chunks needed zero, one, or two repairs, and I am deliberately not quoting those rates here. A single batch of documents collected on one afternoon is not a benchmark, so treat any number I could give you as anecdote rather than evidence.

The failure table I now keep pinned above my desk

What came back	What broke	What I do now
A fenced

json block | parser choked on backticks | strip a leading and trailing fence before parsing |
| Object plus "let me know if…" | trailing text after the JSON | slice from the first `{` to the last `}` |
| `"amount": "1,200.00"` | schema rejected the string | repair prompt names the exact path |
| An empty `records` array | silently counted as valid | compare against a page heuristic, flag for review |
| Truncated mid-object | invalid JSON, late exception | token-budget guard, then re-chunk smaller |

## Notes on the free server, briefly

Running this on a free server changed less than I expected, mostly because the harness keeps no important state in memory. Everything expensive lives in files I can rebuild from fixtures, and the free tier's behaviour never became part of the code's contract.

That is the honest limitation of this whole setup. Nothing here gives me a latency guarantee, a quota guarantee, or any promise that a free tier keeps working the way it did last week. What it gives me is a pipeline where a bad night costs time rather than money.

## What I would repeat, and what I would never repeat

- Repeat: measuring the token estimate against the provider's own counter on the first few chunks.
- Repeat: schema errors that name a path, because they double as repair instructions.
- Repeat: fixture replay keyed by prompt hash, since it made the free tier optional for testing.
- Never repeat: trimming a truncated response and hoping the remainder parses.
- Never repeat: re-sending the original prompt and calling it a retry.
- Never repeat: treating an empty array as success without a second check.

## Who should not use this approach

If you need contractual latency, guaranteed quota, or a fixed model version for a year, this is the wrong foundation and no amount of retry logic will fix that. If your documents are ambiguous enough that a human must label them anyway, the repair loop just adds steps to the same conversation you were going to have.

Also skip it if you cannot store source text and recorded outputs under your own access controls, because the fixture directory is a copy of your data by design. And if a paid key costs less than an evening of your time, buy the key and spend the evening on the parsing logic instead.

## Where this landed

Two days got me a batch job I can re-run for free, a failure table that shortens every future debugging session, and one durable habit: when output looks wrong, check what I measured before I check what the model said.

If you are wiring a similar batch job and want a free tier to be the cheap first pass rather than the thing you trust blindly, MonkeyCode's free model access and free server option is worth trying — just keep the replay mode, so your tests never depend on it staying available.

Comments

[Alex Shev]

The retry behavior is the interesting failure here: once a chunker breaks structural data, repeating the same strategy can amplify corruption rather than recover. JSON should usually be chunked at parsed-object boundaries, with a validation step before sending work onward. Capturing the failed fragment and the retry decision also makes it possible to distinguish transient errors from deterministic bad input.