Every Bug I Fixed Today Was Hiding Another One

#wemakedevs #aws #hackathon #buildinpublic

Building MindMap Debugger for AWS First Commit — Day 2

On Day 1, I got the first working version running — paste in an argument, and the tool reads through it and points out contradictions, powered by Groq's API. It worked. And in the process of testing it, I ran straight into a contradiction of my own, before I'd even finished building the thing meant to catch them.

Yesterday ended with a working extraction pipeline and a deceptively simple plan for today: wrap it in Strands Agents SDK, build the Cedar policy gate, wire it all together, done by evening.

That plan lasted about twenty minutes — turns out the tool wasn't the only thing finding contradictions today. My own assumptions kept getting caught out too.

Bug #1: The AI started thinking out loud inside my JSON

First task — swap the raw Groq client for Strands' Agent wrapper, since Build It's "use AWS-native tooling" requirement means the model call itself needs to route through Strands, not straight to Groq. The SDK was already installed. My extraction logic already worked. This should have been a clean drop-in replacement.

Instead, the model started returning this:

"We need to extract propositions. Two statements: 'SQLite is fast enough for our needs.' and 'the system must survive process restarts...' Could split second into two claims? ... Now check pairwise contradictions. P1: SQLite is fast enough for our needs. No contradiction. P2: System must survive process restarts. No contradiction..."

— followed, eventually, buried at the very end of several paragraphs of visible internal monologue, by the actual JSON I'd asked for.

What was actually happening: openai/gpt-oss-120b is a reasoning model. It always narrates its thinking process, and nothing in Strands' default parameters was telling it to suppress that at the API level. My JSON-extraction fallback could technically still dig the {...} block out of the noise, but it was one weird sentence away from breaking completely.

Fixing it took three attempts, and two of them were wrong in instructive ways:

Adding response_format: {"type": "json_object"} alone — no change. Still leaking.
Adding a top-level reasoning_format: "hidden" parameter — this didn't just fail quietly, it crashed outright: AsyncCompletions.create() got an unexpected keyword argument 'reasoning_format'. Strands' OpenAI-compatible wrapper validates params against a fixed list and rejects anything it doesn't recognize.
The actual fix — nest it inside extra_body instead, so Strands passes it through as a raw field straight to Groq's API rather than checking it against OpenAI's own parameter list:

params={
    "temperature": 0.2,
    "response_format": {"type": "json_object"},
    "extra_body": {
        "reasoning_format": "hidden",
    },
},

Clean JSON, first try, no rambling. First win of the day.

Bug #2: The contradiction that refused to be found

Fixing the leak didn't fix the actual reasoning underneath it. My test sentence — "SQLite is fast enough for our needs. But the system must survive process restarts, and in-memory data does not survive restarts." — contains a genuine contradiction. Using SQLite in-memory conflicts with surviving a restart. But nothing in the wording says so directly; you have to connect three separate facts to see it.

The model kept extracting all three propositions perfectly. It never once flagged the conflict.

I tried two fixes, and both taught me something by failing:

Adding a more explicit prompt rule, spelling out that a "solution" claim can contradict a "requirement" claim even with zero shared wording. Result: worse. The model went from finding one weak relation to finding none at all — the extra instruction diluted the one rule that actually mattered instead of reinforcing it.
Cranking reasoning_effort to "high", forcing more internal reasoning before answering. Result: no change whatsoever. Same weak relation, same missing contradiction.

This is the part that actually surprised me: I'd assumed harder reasoning settings would obviously help with harder reasoning problems. They didn't move the needle at all. So I stopped guessing and looked up whether this was a known thing — and it is. Multi-hop implicit contradictions, where the conflicting facts are spread across separate premises with no shared vocabulary connecting them, are a documented, genuinely hard case for large language models in general. It's not unique to this model, this SDK, or my prompt. It's a real limitation of how these models reason.

The decision that mattered more than any prompt tweak: stop fighting one adversarial sentence. Accept it as an honest, documented limitation instead of a bug to keep chasing, and pick clearer demo inputs where the model's actual strength — catching direct, explicit contradictions — gets to shine. Trying to force a model past its genuine reasoning limits, on a deadline, is a worse use of time than being honest about where those limits are.

Bug #3: Rebuilding a policy that no longer existed

Time to move to Cedar — the AWS-native policy engine at the heart of the Build It track's story. Except the actual rules I'd designed for it — which findings get shown, under what conditions — had lived entirely in a conversation that had already expired. No notes. Nothing saved. The design was just gone.

Rebuilding it meant re-deciding, from first principles, what the policy should actually reward. The real question wasn't technical — it was what earns trust in front of judges:

Direct contradictions: always surfaced, no matter the confidence score. Missing a genuine contradiction is worse than showing one that's slightly uncertain — and contradictions are the entire point of the tool.
Circular reasoning: only surfaced above 0.6 confidence. A shaky contradiction is still useful context for a reader. A shaky circularity claim just looks like the tool crying wolf. Being selective here signals judgment, not just pattern-matching.

That became the actual Cedar policy:

permit (
    principal == Service::"Detector",
    action == Action::"surface_finding",
    resource
)
when {
    resource.type == "direct_contradiction"
};

permit (
    principal == Service::"Detector",
    action == Action::"surface_finding",
    resource
)
when {
    resource.type == "circular" &&
    resource.confidence >= 0.6
};

With a Python enforcement layer (cedar_gate.py) mirroring the same logic, since standing up the full Cedar authorization engine was heavier setup than the timeline allowed — but the actual policy spec above is the real, canonical rule set, honestly documented as such rather than hidden.

First smoke test, using two contradictions (one strong, one deliberately weak) and two circular findings (one strong, one deliberately weak):

Three of four findings passed through. Both contradictions surfaced — even the weak one, exactly per policy. The strong circular finding passed. The weak one got correctly denied. Real, demonstrable policy logic — not "show everything the model said and hope it looks intentional."

Bug #4: The wall at the end of the day

Wiring apply_gate() into the pipeline was one import and one function call — the smallest change of the day. Then I ran the full pipeline against real input for the very first time: a longer, five-sentence argument about launching a product under conflicting pressure, instead of my short toy test sentence.

It broke immediately:

openai.APIError: Failed to validate JSON. Please adjust your prompt.
See 'failed_generation' for more details.

failed_generation, the field that's supposed to show you the broken output, came back completely empty. No malformed JSON to inspect. No partial text. Just a flat wall.

I raised max_tokens to rule out a truncation issue — no change, which at least told me it wasn't simply running out of room. I added debug logging to print the raw error body directly from the API response, ready to finally see what Groq was actually rejecting.

And that's where the day ended — mid-investigation, with the fix still unknown.

The twist: it wasn't the bug I thought it was

Picking the debugging back up, the actual cause turned out to have nothing to do with malformed JSON at all. It was a token budget death spiral: reasoning_effort: "high" combined with strict JSON mode was quietly consuming the model's entire output budget on hidden internal reasoning for this longer, more tangled input — leaving zero tokens left to actually write the answer. Not truncation in the way I'd assumed; raising max_tokens on its own hadn't touched it, because the problem wasn't the ceiling, it was what was eating the budget underneath it. The real fix was dropping reasoning effort to "medium" and raising the token ceiling together, at the same time.

Fixing that immediately uncovered a second, completely different bug hiding behind it: my circular-reasoning detector only ever looked for cycles built from one specific relation type. Real circular arguments, it turns out, don't always stay consistent — a claim can "depend on" another, which in turn "supports" the first, and that's just as circular as two matching relations, but my detector was blind to the mix. A one-line fix — treating both relation types as cycle-relevant — and circular reasoning started showing up for the first time all night.

Fixing that revealed a third bug: to make extraction more reliable (since a single pass sometimes missed things), I started running it multiple times and merging the results — a deliberate reliability upgrade, not a bug fix. But that upgrade introduced its own bug: the same circular reasoning chain occasionally got reported twice, because the underlying graph search could rediscover an identical cycle starting from two different entry points. Fixed with a dedup pass that collapses cycles representing the same set of claims into one.

Three real, distinct root causes, each one hiding behind the last, like debugging nesting dolls. None of them were the bug I originally thought I was chasing.

Where it stands now

Full pipeline, verified end-to-end: paste an argument in, get contradictions and circular reasoning chains out, filtered through real policy logic, rendered live in the browser. On one real test case, the system now reliably catches multiple genuine contradictions and several distinct circular reasoning chains — with zero duplicates.