DEV Community: Salmon Joy

Structured outputs as application contracts

Salmon Joy — Wed, 07 Oct 2026 05:10:32 +0000

Your LLM Output Should Have a Contract, Not a Vibe

So there I was, an AI engineer at a fintech startup, trying to turn a GPT‑4 chat into a credit‑score suggestion engine. The user typed “I just got a promotion, how much should I loan?” and the model spat back a paragraph that looked like poetry: “Congrats! Maybe $5‑10k could work, but keep your debt‑to‑income ratio low…” I tried to yank the numbers out with a regex like /\$(\d+)-(\d+)k/. Spoiler: it broke the moment someone said “$5‑10 k” with a thin space or used “5‑10k” without the dollar sign. One mis‑typed hyphen and my downstream service crashed, mis‑classifying a safe applicant as risky. It felt like trying to catch a greased pig with a colander.

That’s when I discovered the power of structured outputs as application contracts. Instead of hoping the model “gives me what I need”, I told it exactly what shape the response must have – a JSON object that matches a JSON Schema and a typed TypeScript interface. The prompt became:

{
  "loan_range": { "min": int, "max": int },
  "confidence": "high" | "medium" | "low"
}

Now the model can’t accidentally sprinkle a poem in there; it either obeys or says “I’m sorry, I can’t comply”. That refusal is a feature: my code checks if (response.refusal) … and falls back to a safe default.

Why is this better than regex? Regex is a brittle detective that looks for patterns in free text. Anything outside the pattern – extra whitespace, emojis, a different phrasing – sends it into a dead end. A schema validator, on the other hand, parses a proper data structure and instantly tells you which fields are missing or of the wrong type. It’s like checking a passport instead of eyeballing a face.

In my edutech side‑project, we needed a list of quiz questions with “question”, “options”, and “answer”. By versioning the schema ("$schema": "http://json-schema.org/draft-07/schema#", "title": "QuizV2"), we could roll out a new field “explanation” without breaking older clients. Downstream safety checks – e.g., “no option string longer than 200 chars” – are baked into the schema, so the LLM can’t hand us a monster answer that blows up the UI.

A quick case study: after switching to structured outputs and enabling refusals, our fintech loan‑approval pipeline’s error rate dropped from 12% to under 1%. The only time we got a refusal was when the user asked for advice on illegal activity, and we gracefully returned a polite error message.

Bottom line: treat the LLM like a junior dev who follows a contract, not a poet who vibes with your mood. Define a JSON Schema, generate typed interfaces, validate, version, and let refusals be your safety net.

Got a funny regex story or a schema win? Drop a comment – I’d love to hear how you’re taming your LLMs!

If you are someone who loves to know the technical work and architecture design I have shared more details based on my experience on this here: https://github.com/SalmonJoy/My_guide_for_building_AI_systems/blob/main/Structured_outputs_as_application_contracts.md

Context engineering vs prompt engineering

Salmon Joy — Tue, 06 Oct 2026 13:17:36 +0000

Prompt Engineering Is Too Small a Mental Model, Think Context Architecture

I was knee‑deep in a fintech app that tried to answer user questions about credit‑score changes. My first version was a classic “prompt engineering” hack:

“You are a friendly finance bot, explain the score drop in plain English.”

It worked for the first few queries, then the answers got weird – sometimes the bot repeated the same disclaimer, other times it started spilling the policy docs verbatim. I was losing token budget fast and the model kept “remembering” the old disclaimer even after I removed it. I realized I was treating the whole prompt as a monolith, when in reality I was juggling five different pieces of context that should have their own lanes.

Enter context architecture

Think of an LLM conversation like a pizza kitchen.

System instructions – the chef’s recipe book; they stay on the board all night.
User state – the current order ticket (what the customer just asked, plus any previous choices).
Retrieved knowledge – the pantry shelf you pull ingredients from (a database lookup of the user’s credit history).
Tool results – the side‑dish the sous‑chef prepares (e.g., a risk‑score API call).
Policy – the health‑inspection sticker you slap on the box to make sure you never serve raw chicken.

The magic happens in precedence. The model first looks at system instructions, then layers user state, then fetched knowledge, then tool results, and finally policy overrides anything that would break compliance. If you drop a policy clause at the bottom of a huge prompt, it might never be seen because the model runs out of tokens before it gets there. That’s why you always put the “stable‑prefix” (system + policy) at the front – it’s cached, cheap, and guarantees the model never forgets the rules.

Token budget = pizza dough budget

You only have so many tokens per request; if you waste them on stale context (like a disclaimer you already sent), you run out of room for the juicy answer.

Contamination risk is when old user state leaks into a new conversation – imagine the kitchen forgetting to clear the ticket and adding yesterday’s pepperoni to today’s vegan pizza. To avoid that, we clear the user state each session or use a fresh context window.

Stable‑prefix caching

(OpenAI Prompt Caching) lets you store the system + policy block once and reuse it across millions of calls. In my edutech project, we cached a 120‑token prefix that described the “tutor bot” persona and compliance rules. The cache cut latency by 30 % and saved us $2 K a month on token spend.

Case study: Fintech fraud‑alert bot

Piece	Content
System	“You are a compliance‑aware fraud analyst.”
Policy	“Never reveal raw transaction IDs.”
User	“My card was declined twice.”
Retrieved	“Last 5 transactions: $23, $0, $0.”
Tool	“Risk engine returns confidence 0.92.”

The final prompt stayed under 350 tokens, the answer was spot‑on, and no policy leakage occurred.

So the next time you’re tempted to dump a giant prompt into the model, pause and ask yourself:

What belongs in system instructions?
What’s the user state?
What do I need to fetch?
What tool results am I adding?
What policy must sit at the very front?

Build that context architecture, and you’ll spend less time chasing bugs and more time building cool features.

What weird context mishap have you run into? Drop a comment and let’s swap stories!

If you are someone who loves to know the technical work and architecture design I have shared more details based on my experience on this here:

https://github.com/SalmonJoy/My_guide_for_building_AI_systems/blob/main/Context_engineering_vs_prompt_engineering.md

Model selection is an architecture decision

Salmon Joy — Tue, 06 Oct 2026 12:51:02 +0000

Stop Asking “Which LLM Is Best?” – Ask Which Model Fits This Workload

Last quarter I was elbow‑deep in a fintech product that had to turn noisy bank statements into clean expense reports. My first instinct was to grab the “biggest” LLM, fire‑off GPT‑4, Claude‑2, Gemini‑1.5 and hope it would magically understand every column, currency and weird abbreviation. After a weekend of latency checks, cost tallies and exploding error logs I realized the “best” model on every public leaderboard still missed our real KPI: cost per successful parsing.

The fix was to stop treating LLMs like shiny toys and start treating them as replaceable services. I listed the dimensions that actually mattered:

capability (financial jargon, tables)
reasoning (multi‑step extraction without hallucination)
latency (<300 ms for a smooth UI)
context length (statements can be 10 k tokens)
modality (sometimes we get a PDF image, so OCR + generation matters)
cost (every API call eats our margin)

Then I built a tiny benchmark that mirrors the exact workflow: feed a real statement, ask for JSON line items, measure success (JSON parses, totals match). I ran it against three services: OpenAI gpt‑3.5‑turbo‑16k, AWS Bedrock Claude‑3 Opus, and Google Gemini‑1.5‑Flash.

Model	Accuracy	Cost / 1k‑tok	Latency
gpt‑3.5‑turbo	92 %	$0.012	420 ms
Claude‑3 Opus	95 %	$0.018	280 ms
Gemini‑Flash	88 %	$0.006	150 ms

Cost‑per‑successful‑task = (cost per call) ÷ (success rate).

GPT‑3.5: $0.013
Claude‑Opus: $0.019
Gemini‑Flash: $0.007

Suddenly Gemini‑Flash became the clear winner for our budget‑tight pipeline, even though its raw accuracy looked lower on paper.

The trick is to version‑track the exact model you use (e.g., gpt-3.5-turbo-0613) and re‑run the benchmark whenever a new upgrade lands. In a later ed‑tech project I swapped Claude‑2 for Claude‑3‑Sonnet after a single test showed a 30 % latency drop and a 5 % reasoning bump, shaving $0.003 per student quiz from the cost‑per‑task metric.

So the next time someone asks “Which LLM is best?” answer with a checklist and a cost‑per‑successful‑task chart. It turns brag‑rights into real engineering decisions.

What’s the weirdest metric you’ve used to pick a model? Drop a comment, I’m curious about your war stories!

If you are someone who loves to know the technical work and architecture design I have shared details based on my experience on this here:

https://github.com/SalmonJoy/My_guide_for_building_AI_systems/blob/main/Model_selection_is_an_architecture_decision.md