Splitting the Job in Two
Klyro's diagnostic core runs on exactly two LLM calls per run, not one, and the split isn't cosmetic. An Analyst looks at the load-test results and figures out what's actually wrong. An Investigator takes that diagnosis and writes the code fix. Two agents, two prompts, two different jobs, because asking a single call to both spot a performance regression and safely edit source code tends to blur the line between noticing a problem and being trusted to touch a codebase. The Analyst runs on each provider's smaller, faster model since diagnosis is closer to pattern matching; the Investigator runs on the larger one, because proposing an actual patch deserves the model with more room to reason.
The Rate Limit Nobody Saw Coming
The provider story behind this pipeline isn't a straight line. Groq was the original plan. Then a Mistral key showed up with limits that looked better on paper, and the pipeline moved over. Then Mistral's account came back rate-limited to zero requests per minute, the kind of thing no dashboard warns you about in advance and no amount of retrying fixes on its own. The response wasn't to go looking for a third single provider to bet on. It was to stop betting on one provider at all: the pool now holds two Mistral keys and two Groq keys at once, spanning both providers, so a 429 against one provider's entire account still has somewhere else to go.
How the Rotation Actually Works
The mechanics live in LLMProvider, a small class that treats the pool as an ordered list of { apiKey, baseUrl, model } entries. Hit a 429 on whichever entry is active, and it rotates to the next one and retries the exact same prompt immediately, cycling through the whole pool before it ever falls back to a plain backoff-and-retry. That's deliberately narrow: the pool and its order are configured up front in SSM Parameter Store, not improvised mid-run. The one thing rotation never touches is schema repair. If a model returns JSON that fails validation, the retry goes back to the same pool entry with the error appended to the prompt, because that's the model getting something wrong, not the provider going down, and those two failure modes call for different fixes.
What the Investigator Is Not Allowed to Touch
Letting an LLM propose code changes only works if the blast radius is small and provable. The Investigator can write to exactly three files: demo-app/src/orders.js, demo-app/src/logger.js, and demo-app/config/logger.json. Any other path gets rejected before a single build step runs, no exceptions. Every patch also has to carry the original_sha256 of the file it targets, checked against the live file at apply time. If the hash doesn't match, the patch doesn't land. That single check rules out an entire class of problems: a stale patch written against a version of the file that's already moved on, or a patch quietly overwriting a change no one asked for.
Flexible Where It Counts, Rigid Everywhere Else
The pool rotation and the allowlist look like they're pulling in opposite directions, one bending to keep the pipeline running, the other refusing to bend at all. They're actually the same design instinct applied twice. Decide in advance exactly where flexibility is safe, wire it in deliberately, and then hold the line everywhere else without exception. A rate limit shouldn't stall a demo run. A model hallucinating a change to a file it was never handed shouldn't reach production, even in a sandbox. Neither rule got relaxed to solve the other's problem.
Top comments (1)
The Analyst/Investigator split is the right call — the instinct to separate "what's wrong" from "fix it" maps to how you'd structure a human review process too. Mixing diagnosis and code edit into one prompt creates a context contamination problem where the model's certainty about the diagnosis bleeds into overconfidence about the fix.
The provider pool story is genuinely useful. Mistral going to 0 req/min with no dashboard warning is the kind of production surprise that only shows up in real write-ups like this. The design response — stop betting on one provider — is the correct one. The LLMProvider rotation logic with { apiKey, baseUrl, model } tuples is clean: the pool structure is provider-agnostic enough to absorb future entries without rewiring the rotation.
The distinction between 429 errors and JSON validation failures getting different retry paths is exactly the right call. These are different failure modes with different causes: provider capacity vs model output quality. Conflating them with a single retry strategy would mask real patterns.
The SHA256 hash check on file patches is a sharp safety mechanism. It's solving a subtle problem — not just "is this an allowed file?" but "is this the version of the file I was reasoning about?" That class of stale-patch errors is easy to miss and hard to detect after the fact.
The framing at the end — flexible where it's safe, rigid everywhere else — is a useful mental model for agentic systems generally. At Black Label we think about the same tradeoff: where does the agent get to adapt, and where does it need hard stops?