DEV Community: Abhishek Kundagol

Beyond LLM JSON mode: decisions your code can branch on

Abhishek Kundagol — Wed, 21 Oct 2026 13:46:24 +0000

Your support pipeline asks a model which queue a ticket belongs to. It comes back with three sentences: "This appears to be a billing issue, as the customer mentions being charged twice. It could also relate to their account. Billing seems like the best fit." Now your code has to find the queue in that paragraph. So you reach for LLM JSON mode, and the paragraph turns into {"queue": "billing"}.

That is better. It is still not a decision your code can trust, and this post explains why, and what a typed decision looks like instead.

The problem with deciding in prose

When software gets its decisions back as text, three things go wrong.

You can't see how sure it is. "Billing seems like the best fit" reads the same whether the model was certain or guessing. A ticket that is clearly about a double charge and one that could be billing or account access look identical by the time they reach your if.

The words come first. When a model writes its explanation before it settles the answer, the answer tends to follow the text it already wrote, not the other way round.

Parsing is a guess. Your regex works until the model says "Billing" with a capital B, or "the billing team", or adds a second suggestion at the end.

What LLM JSON mode fixes, and what it doesn't

Structured outputs and JSON mode fix the last problem. The format is constrained, so the response parses and the field is where you expect it.

They don't fix the first two. The model is still writing text underneath, token by token, with no sense of how sure it is. The queue inside your JSON is a token in a string. {"queue": "billing"} carries no number that tells you whether to act on it or send it to a person.

That missing number is the part that matters in production. Routing a ticket wrongly is cheap to undo. Issuing a refund wrongly is not. Without a probability, both get the same treatment.

A typed decision instead

A typed decision is an answer your code can branch on directly: one of the options you defined, with a probability for every option. Here is the first request from the Wity quickstart, in Python:

import os, requests

r = requests.post(
    "https://api.wity.alphanimble.com/v1/systemone",
    headers={"Authorization": f"Bearer {os.environ['WITY_API_KEY']}"},
    json={
        "state": "Hi, I was charged twice for my subscription this month. Please refund the duplicate.",
        "questions": {
            "intent": {
                "type": "choice",
                "instructions": "What does the customer want?",
                "criteria": {
                    "billing": "A charge, refund, invoice or payment problem",
                    "technical_support": "Something in the product is broken",
                    "account_access": "Cannot log in or is locked out",
                    "cancellation": "Wants to cancel or downgrade",
                },
            }
        },
        "reasoning": "auto",
    },
    timeout=60,
)
r.raise_for_status()
intent = r.json()["answers"]["intent"]
print(intent["choice"], intent["probabilities"][intent["choice"]])  # billing 0.99

You send the state (the thing to decide about) and the question, with the options spelled out. Nothing asks the model to "answer in JSON", because the answer was never text. This is what comes back:

{
  "model": "wity-1",
  "answers": {
    "intent": {
      "type": "choice",
      "choice": "billing",
      "probabilities": {
        "billing": 0.9912,
        "technical_support": 0.0031,
        "account_access": 0.0019,
        "cancellation": 0.0038
      },
      "confidence": 0.93,
      "reasoning": { "mode": "auto", "thought": false, "reason": null, "forecast": false, "thought_tokens": null }
    }
  },
  "usage": { "input_tokens": 212, "output_tokens": 0 },
  "metadata": { "reasoning": "auto", "elapsed_ms": 94.1 }
}

Reading the answer

choice is the option to act on. It can only be one of the keys you put in criteria, so there is nothing to normalise.
probabilities covers every option you offered and sums to 1. You can read any of them, not only the winner.
confidence says how peaked the distribution is. The docs are clear that it describes the shape of the answer, not the chance of being right: use probabilities for anything that acts, and confidence for sorting a review queue.
reasoning.thought is false here. The question was clear, so Wity answered in one fast pass. On a close call it thinks first, and the answer comes back in the same shape.

The full distribution is where the value is. A single label tells you what was picked. The probabilities tell you how the evidence split, and whether a second option deserves a look.

Branching on probabilities

Once the answer is a number, the rule lives in your code, not in a prompt. This is the three-zone pattern from the docs on probabilities, for a question named route with a refunds_queue option:

ans = r.json()["answers"]["route"]
p = ans["probabilities"][ans["choice"]]

if ans["choice"] == "refunds_queue" and p >= 0.95:
    issue_refund()             # cheap to undo: act on your own
elif p >= 0.80:
    queue_for_one_click(ans)   # a person confirms with one click
else:
    send_to_human(ans)         # genuinely unsure: full review

Refunds get a stricter bar because they cost money. Other routes need 0.80 because a wrongly routed ticket is cheap to move. You set the lines from what a mistake costs, and you can move them without touching a prompt.

No model is perfect, and these numbers are not a promise about any single answer. That's the reason to have them: act on the clear calls and hand the close ones to a person.

Where text still belongs

Some actions need words: a city to type into a form, a one-line reply, a field value. For those, Wity's generate primitive writes short text in a shape you define, from the same state, after the decision is made. Output is capped at 100 tokens. If the answer is one of a set you could list, use choice instead, because a decision gives you probabilities and generated text doesn't.

The rule of thumb: decide first, in a type, and write text only for the part of the action that needs it.

The quickstart gets you to your first typed decision in about a minute, and every account starts with $5 of free credit to use within 48 hours of signing up.

System 1 vs System 2 AI, and the gap between them

Abhishek Kundagol — Tue, 20 Oct 2026 10:02:07 +0000

Psychologists describe two modes of thought. System 1 is fast and intuitive. System 2 is slow and deliberate. Most of what we decide in a day is System 1, and we switch to System 2 only when something feels off.

That split is a useful way to look at the models we put inside software. Ask "system 1 vs system 2 AI" and you get two families, each good at half the job. This post walks through what goes wrong at each extreme, and what a model that sits between them does instead.

System 2: models that think through everything

Reasoning models think through every question, even the easy ones, and answer in paragraphs you still have to parse.

Look at the decisions a real system makes in a day and most of them are easy. The language of a ticket. Whether a page has loaded. Which of five clearly different queues a message belongs to. A direct read gets these right, and thinking about them only adds seconds.

So a System 2 model costs you twice. You wait for deliberation the question never needed, and then you write code to dig the answer out of prose. On one call that's tolerable. On every event in a pipeline, it decides whether the model can sit inside a request at all.

System 1: models that answer at a glance

Fast decision models go the other way. They answer in a single glance. Easy calls are instant. Hard ones come back confidently wrong.

Take a delivery question:

"Ordered Friday. Delivery takes 9 days, weekends included." Which day does it arrive?

At a glance, Saturday and Sunday look about equally likely. Getting it right takes counting, and a model that can only glance has no way to slow down. It picks one, fast, and your code has nothing to tell it that this answer was a coin flip.

That's the System 1 failure: not that it's wrong sometimes (every model is), but that the hard questions get the same treatment as the easy ones.

The gap between them

Put the two side by side and the gap is clear:

System 2 is careful on every question, including the many that don't need it.
System 1 is fast on every question, including the few that do need care.

What you want is fast on the easy majority and careful on the close calls, decided per question, with the same kind of answer either way. That's what we mean by System 1.5.

What a System 1.5 model does

Wity answers most questions in one fast pass, in about a tenth of a second. When an answer is uncertain or unstable, it thinks first and then answers. Either way you get the same thing back: a typed decision with a probability for every option.

In auto mode the flow has three steps:

Direct read. Wity answers at once and checks whether that answer is robust: a clear winner that doesn't change when the options are presented differently.
Close call? Most questions pass, and the answer comes straight back. If not, it thinks.
Think, then stop. It reasons step by step and stops as soon as the answer has settled, instead of running a fixed long chain of thought.

There are three reasons it will think:

close_call: the top options are close. A ticket at 0.51 billing and 0.46 account access deserves a closer look.
order_sensitive: the answer changes with the order of the options. A sound answer shouldn't depend on that.
forecast: the question asks how likely something is, not which answer is right.

Back to the delivery question. Saturday and Sunday are close, so Wity counts: Friday plus seven is Friday again, then Saturday, then Sunday. It answers Sunday, and is now sure of it.

Most thoughts finish in a few hundred tokens, and thought answers typically take one to four seconds. The easy questions in the same request still come back at direct-read speed.

You can see when it thought

Every answer says whether it thought, and why:

"reasoning": {
  "mode": "auto",
  "thought": true,
  "reason": "close_call",
  "thought_tokens": 184
}

Log that next to each decision. A question that often thinks for close_call is telling you two of its options overlap. Sharper descriptions, or splitting it in two, make it both faster and more accurate.

Choosing the mode

One field sets it for the whole request:

{ "state": "…", "questions": { … }, "reasoning": "auto" }

auto for almost everything: fast where it can be, careful where it has to be.
off for the lowest, most predictable latency, like a step in an interactive UI. Close calls still show up as close probabilities, so your thresholds keep working.
always for batch jobs where every decision is expensive to get wrong and nobody is waiting, like overnight review of flagged transactions.

The output type is the same in every mode, so switching never changes your code. Thinking isn't billed either, so a thought answer costs the same as a direct one. The only reason to turn it off is time, not money.

Where this leaves you

No model is perfect, and a 1.5 model doesn't change that. What changes is where the effort goes: none on the questions that don't need it, and some on the ones that do. The probability on every answer still lets your code decide when to act and when to hand a decision to a person.

The reasoning docs cover the modes, the response fields and how to plan around latency.

What is Wity? The first System 1.5 model, explained.

Abhishek Kundagol — Mon, 19 Oct 2026 11:35:46 +0000

Software makes thousands of small decisions a day. Which team owns this ticket? Is this scan a receipt or an invoice? Has the page finished loading? Is the customer reporting an accident?

None of these needs an essay. Each needs an answer your code can act on, and some sense of how sure that answer is. Today those calls usually go to one of two kinds of model, and neither is built for the job. Wity is a System 1.5 model, and it sits between them. This post explains what it is from scratch: the idea, what an answer looks like, and where it fits.

Two kinds of model, two kinds of problem

Large language models think through every question, even the easy ones. Then they answer in paragraphs your code has to parse. You pay in seconds and tokens for reasoning that a clear-cut question never needed, and you still have to guess how confident the answer was.

Fast decision models answer in a single glance. Easy calls come back instantly. Hard ones come back instantly too, and confidently wrong, because the model has no way to slow down.

What "System 1.5" means

Psychologists describe two modes of thought. System 1 is fast and intuitive. System 2 is slow and deliberate. Most of what you decide in a day is System 1, and you switch to System 2 only when something feels off.

Wity works the same way. It answers most questions in one fast pass, in about a tenth of a second. When an answer is uncertain or unstable, it thinks first and then answers. Either way you get the same thing back: a typed decision with a probability for every option. That's why we call it a System 1.5 model. It sits between instinct and deliberation and picks per question.

What a decision looks like

You send a state (the thing to decide about: text, JSON, an image, or all three) and one or more questions with the options spelled out:

curl https://api.wity.alphanimble.com/v1/systemone \
  -H "Authorization: Bearer $WITY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "state": "Hi, I was charged twice for my subscription this month. Please refund the duplicate.",
    "questions": {
      "intent": {
        "type": "choice",
        "instructions": "What does the customer want?",
        "criteria": {
          "billing": "A charge, refund, invoice or payment problem",
          "technical_support": "Something in the product is broken",
          "account_access": "Cannot log in or is locked out",
          "cancellation": "Wants to cancel or downgrade"
        }
      }
    },
    "reasoning": "auto"
  }'

The answer comes back under the name you gave the question:

{
  "model": "wity-1",
  "answers": {
    "intent": {
      "type": "choice",
      "choice": "billing",
      "probabilities": {
        "billing": 0.9912,
        "technical_support": 0.0031,
        "account_access": 0.0019,
        "cancellation": 0.0038
      },
      "confidence": 0.93,
      "reasoning": { "mode": "auto", "thought": false, "reason": null, "forecast": false, "thought_tokens": null }
    }
  },
  "usage": { "input_tokens": 212, "output_tokens": 0 },
  "metadata": { "reasoning": "auto", "elapsed_ms": 94.1 }
}

There's no prose to parse. choice is the option to act on, and probabilities covers every option you offered. reasoning.thought is false here because the question was clear, so Wity answered directly.

Four primitives, every answer typed

Every question takes one of four shapes. You pick the shape, Wity fills it:

noul: a yes-or-no call. "Was the insured driver at fault?"
score: where on a scale something sits. "How bad is the damage: cosmetic, repairable or total loss?"
choice: one option from a set. "Which team owns this?"
generate: the short text an action needs, in a shape you define. "The policy number from the scan."

Because every decision carries its probabilities, your code decides how sure is sure enough:

intent = answers["intent"]
if intent["probabilities"][intent["choice"]] >= 0.85:
    route(ticket, intent["choice"])       # sure enough: act on its own
else:
    send_to_triage(ticket)                # close call: a person decides

Act on clear calls, hand close ones to a person, and set that line per question.

Five things that set it apart

Multimodal. Photos, screenshots and scanned pages go in the same request as your text.
128K-token context. A whole contract, a long email thread or a stack of scans fits in one state. No chunking.
generate. A primitive for the few places software genuinely needs words, written from the same state, right after the decision.
Auto-thinking. Instant on clear-cut questions, deliberate on shaky ones, decided per question.
On-prem. It can run inside your own network, with the same API as the cloud.

Fast and cheap enough to put everywhere

A decision model earns its place by being cheap enough to call on every event and fast enough to sit inside a request. We ran Wity and eight fast, low-cost models through the same 200 real-world cases. Wity's typical call took 109 ms, against 943 ms for the next fastest. 1,000 runs cost $3.29, against $17.95 for the next cheapest. That's our own benchmark, and the method is published alongside it.

Pricing is one number: $0.042 per 1M tokens, and failed calls are free. Thinking isn't billed, so a thought answer costs the same as a direct one.

No model is perfect. Use the probability on each decision to choose when to act and when to escalate, and validate generated text like any other input.

What's next

This is the first post in a series where we build up from here, one idea a day: System 1 vs System 2, why decisions shouldn't come back as paragraphs, when a model should think, each primitive in depth, and real builds with code.

If you want to try it now, the quickstart gets you to your first decision in about a minute. Every account starts with $5 of free credit to use within 48 hours of signing up, and the Playground runs real requests from your browser with no code at all.

The Two Ways AI Memory Breaks

Abhishek Kundagol — Thu, 20 Aug 2026 07:26:44 +0000

The interesting decision our memory system Memuron makes isn't "should I keep this."

It's "is this new, or is this something I already know, changed?"

Day 4. This one's about what we're building.

In the last post i said the hard part of the memory is deciding. I think most people hear that and picture a bouncer, something at the door turning away claims that don't make the cut.

That's the wrong picture. Throwing things away is one thing a memory system shouldn't do. Storage is cheap. The expensive question is structural.

Every new piece of information is either a genuinely new thing, or it's a change to something already in the record. Getting that wrong in either direction is what breaks systems.

Call every update a new memory and you get five near-identical claims about the same fact, drifting apart, all retrievable, none authoritative. Your agent picks one at random and sounds confident.

Call every new thing an update and you quietly overwrite distinct facts that happened to look similar. That one's worse, because it's silent.

So that judgement, new or changed, is the actual work. And it has to be made at write time, by something that can look at what's already known.

Two things we decided early. Nothing gets overwritten: a change appends, and the previous state stays in the record. And every one of those decisions gets stored with the reasoning behind it, so you can go back and see not just what the system concluded, but why it concluded it.

Neither is clever. Both are the kind of thing you can't add later.

Day 5: why nobody in your org chart owns any of this.

Your AI Memory Has No One in Charge

Abhishek Kundagol — Tue, 18 Aug 2026 05:15:51 +0000

Every time your AI stores something, it makes a decision.

Keep this or merge this with what we already know. Correct the old version or reject it since it is wrong.

In most systems today, nobody makes that decision. It's just an insert.

Day 3.

Think about how a good analyst handles new information. They don't just append it. They ask: does this contradict what I already have? Is this an update or a duplicate? Is the source reliable? Should this replace something?

That's the actual work of remembering. Not the writing down, the judging.

Now look at what most AI memory does. Embed the text. Insert the row and done. Every claim enters with equal standing. The correction doesn't overwrite the mistake, it just sits next to it. The duplicate becomes two votes for the same idea. The hallucination gets stores with exactly the same confidence as the verified fact.

Then six weeks later, retrieval pulls one of them out, and no one can tell you which one it was or why.

Writing is the easy half. Deciding is the whole problem.

Day 4: what happens when you put something in charge of that decision.

We at AlphaNimble building Memuron, a memory system for AI agents. This series is the thinking behind it, in the open.

Retrieval Is Not Memory

Abhishek Kundagol — Sat, 15 Aug 2026 06:10:57 +0000

"We have memory. We're using RAG."

You have retrieval. Those aren't the same thing, and the gap between them is where agents quietly go wrong.

Day 2.

RAG finds documents that look relevant to your question. That's it . That's the whole job. It's a very good search engine bolted to a very good writer.

But consider what it can't do.

Your customer changed their pricing tier in March. The old contract is still in the index. The new one is too. RAG doesn't know which one is true, it just knows both are relevant. It hands the model two answers and lets it guess.

Memory would know one of those facts replaced the other, and when.

That's the difference. Retrieval finds. Memory concludes.

Retrieval asks "what documents match?" Memory asks "what do I actually believe, what changed my mind, and when did that happen?"

One is lookup. The other is position.

This matters because a system that only retrieves can never be wrong and it can never be right either. It has no beliefs to correct. Every contradiction in your data is a contradiction it will faithfully pass along, forever, with total confidence.

Day 3: if memory means concluding things, then something has to decide what gets remembered. Right now, in most systems, nothing does.

We at AlphaNimble are building Memuron, a memory system for AI agents. This series is thinking behind it, in the open.

Context Windows Are Not Memory

Abhishek Kundagol — Fri, 14 Aug 2026 05:44:10 +0000

Everyone is racing to make AI smarter.
Almost nobody is asking what it's allowed to remember.

I'm starting a daily series on the least understood layer in AI. Day 1.

Here's the confusion at the center of it: we've started calling the context window "memory." It isn't.

A context window is a desk. You pile things on it, you work, and at the end of the session someone clears the desk. Bigger models just give you a bigger desk.

Memory is the filing cabinet. What survives the desk being cleared.

So picture hiring a brilliant analyst with no long-term memory. Every morning you re-brief them on the company, the customers, the decisions you already made together. They nod. They do genuinely excellent work. And by tomorrow, it's gone.

You'd never call that person a knowledge worker. You'd call it a very expensive Groundhog Day.

That is most "AI agents" running in production today.

The industry's answer has been to treat this as a storage problem. Bigger context. Another vector database. Stuff more in, hope the right thing comes out.

I think that's the wrong frame entirely. Memory isn't a storage problem.

It's a trust problem.

Tomorrow, Day 2: why RAG is not memory — and why confusing the two costs teams more than they realise.

—

We at Alphanimble building Memuron, a memory system for AI agents. This series is the thinking behind it, in the open. Every post is something I've had to figure out to build the thing.

AI #AIAgents #AIMemory

Your AI Doesn't Have Amnesia — It Has a Storage Problem

Abhishek Kundagol — Wed, 08 Jul 2026 05:35:03 +0000

I hit this wall over and over.

I'd have a long conversation with one LLM — Claude, ChatGPT, Gemini, whatever — and then want to continue the same thread in another. Maybe I started researching in ChatGPT and wanted to implement in Cursor. Maybe Claude had better reasoning for one part, but I needed a different tool for the next step.
The only workaround? Copy-paste the entire conversation. Every message. Every detail. Just to give the new model enough context to pick up where the last one left off.

It works, barely. But it's awful. You're manually shuttling context between tools that are supposed to be smart. And you're burning tokens on duplicated history — paying twice (or three times) for information you already established once.

That friction is where this problem really started for me. Not abstractly. Practically. Every time you switch AI tools, you start over — or you become the copy-paste middleware between siloed memories.

Every AI remembers you differently. None of them share.

ChatGPT has its own memory. Claude has connectors and conversation context. Cursor picks up rules and session context. Gemini holds a different slice of your preferences. Every custom agent, IDE plugin, and workflow tool stores what it learns separately.
Each platform is getting smarter. Each one remembers more. And none of them talk to each other.

You explain your tech stack in Claude, then again in Cursor. You set tone preferences in ChatGPT that your coding agent never sees. You research in one tool and reconstruct context from scratch in another.

It's not just inconvenient. It's a structural flaw: each AI owns its own memory, and there's no shared layer underneath.
And underneath most of these systems? Vector databases — similarity search over a bag of facts. Tell an agent "I like blue" on Monday and "actually, I prefer red" on Tuesday, and a plain vector store returns both. The agent gets a contradiction it can't resolve.

Every contradicted memory is a silent hallucination waiting to surface.

What we built: one memory graph, every agent

Memuron is a semantic memory layer for AI agents — not a vector bucket, but a managed knowledge graph that any tool can read from and write to.

Connect Cursor, Claude, ChatGPT, or your own agents via MCP or API, and they all work against the same durable memory. Start a conversation in one place. Continue in another. No copy-pasting. No re-explaining. No wasted tokens shuttling context you already established.

Memuron doesn't replace your AI tools. It gives them a shared brain.

How it works

The Guardian decides before it stores. Every write passes through an LLM pipeline that asks: is this a new fact, an update to something already known, or a link to a related memory? When preferences change, Memuron evolves the existing memory in place — it doesn't pile up contradictions.
A graph, not a pile of embeddings. Memories are typed nodes connected by semantic links. Agents can search, traverse, and reason over relationships — not just grab the nearest vector match.

One substrate, many clients. MCP-native integration means Cursor, Claude, and ChatGPT query and write to the same org-scoped graph. OAuth authentication — no API key wrangling in your editor.

What you get

Hybrid Graph Search — vector, lexical, and link-description matching fused together. Returns memories and the edge that made them relevant.
GraphFS — query your memory graph like a filesystem:
cwd /spaces/work ls | grep "auth" | semantic "rate limiting" | links
Spaces — separate work, personal, and project context, each with its own Guardian rules.
Document Ingestion — PDFs, DOCX, Markdown, images, and more, parsed into chunk nodes and linked into your graph. Resolve any chunk back to the original source.
URL & Folder Sync — fetch pages, sync local folders, keep stable identities so re-runs update instead of duplicate.
Context Assembly — prompt-ready blocks with citations, graph breadcrumbs, and hard token budgets.
Multi-Tenant Isolation — cryptographic tenant boundaries for B2B SaaS and team deployments.

- REST API + CLI — `uvx memuron` for scripts, CI, and custom integrations.

Memory that travels with you

The future of AI isn't ten separate brains that each forget what the others learned. It's one durable memory layer that every agent can tap into — smart enough to update itself, structured enough to query, and open enough to connect anywhere.
You shouldn't have to be the copy-paste layer between your AI tools. Memuron is that layer — so you can stop repeating yourself and start building on what you already know.
Memory that decides before it stores. One graph for every agent you use.