DEV Community: Trida AI

Model Upgrades Are Deployment Events

FDE at Trida AI — Wed, 21 Oct 2026 01:34:08 +0000

Originally published at Trida AI.

Copy as markdown

Swapping a model takes a minute. You change one string in a config file, restart the service, and a different system starts answering your users. Nothing in your code changed. A good deal of your system's behaviour just did.

That gap is where migrations go wrong. Over months, a production system quietly grows around one model's habits. Prompts are tuned against its phrasing. Parsers expect its formatting quirks. Tool calls follow its rhythm, retry logic assumes its latency, cost forecasts assume its tokenizer. The replacement may top every public benchmark and still be worse at your specific job, in specific places, for reasons no leaderboard will show you.

So we treat a model change the way we treat any other deployment: an event with a before and an after, evidence gathered on both sides, and a way back. The model is a dependency. Upgrading it deserves at least the suspicion you would give a major-version bump of a library your product cannot run without. More, actually. A library does not rewrite its own behaviour between releases.

Pinning buys time. It does not buy permanence.

You have two ways to name a model, and each fails differently. Call an alias, the short undated name, and the provider can move it to a newer snapshot underneath you. Nothing errors. Your system just drifts, and the first symptom is a slow slide in output quality that nobody can date. Pin a dated snapshot instead and behaviour stays put, but you have scheduled your own migration: snapshots retire.

The clocks are public and they are not generous. On Friday, OpenAI shuts down a batch of legacy snapshots (GPT-4, GPT-4 Turbo, the first GPT-4o snapshot, o1, o3-mini and o4-mini among them), with replacements named on its deprecations page. The named replacements are GPT-5.6 models, Sol, Terra and Luna by tier, and a second wave is already scheduled behind this one: GPT-4o's later snapshots, GPT-4o mini, GPT-4.1 and GPT-4.1 mini retire on January 14, 2027. OpenAI commits to at least six months' notice for generally available models, three months for specialised variants, and possibly as little as two weeks for previews. Six months sounds comfortable until the work lands in the quarter when everything else ships too.

For generally available models, Anthropic's floor is shorter. Its model deprecations page promises customers with active deployments at least 60 days' notice before a publicly released model retires, and requests to a retired model simply fail. Claude Sonnet 4.5 was deprecated on September 30, 2026 and retires on November 30. Anthropic names Claude Sonnet 5.5 as the replacement. The same page carries a quieter trap: on Claude 4.7 and later, setting temperature, top_p or top_k to a non-default value returns a 400 error. Code that never changed models can still break on the day it is forced to move.

And if you reach the same model through Amazon Bedrock or Google Cloud, check that platform's schedule too. The provider's clock is not always your platform's clock.

A better model can still regress your workload

The evidence that upgrades hurt real prompts is not folklore. In a case study of toxicity detection on social media text, 58.8% of prompt and model combinations lost accuracy after an API update, and among those that dropped, 70.2% dropped by more than 5%. The update was newer. The workload was worse.

Slices are the operative word. An aggregate eval score can hold steady while one segment of your traffic falls off a cliff. If your evals only report the average, you will promote a regression and call it an upgrade.

Run the migration like a release

Start with an inventory, because the model ID never lives in one place. It sits in application config, in worker env vars, in scheduled jobs, in the eval harness, in the docs nobody updated. List every caller before you touch anything.

Then freeze a baseline. Run your eval set against the current model and keep the outputs, scored by slice. The slice is the unit that matters: document type, language, request shape, whatever partitions your traffic.

Replay the same set against the candidate. Compare slice by slice rather than by the total, and compare cost and latency alongside quality. A candidate that is two points better and three times slower is not obviously a win.

The migration gate: production traffic runs on the current pinned model while the candidate receives a shadow copy of the same requests; both sets of outputs are compared slice by slice in the eval replay, and only a candidate that passes the gate earns a 5% canary share of live traffic, with rollback to the pinned model available until its retirement date.

Only then let the candidate touch real work. Shadow first: it answers live requests, its outputs are logged and scored, nobody acts on them. Canary second: a small share of traffic, watched slice by slice. Promotion is a config change, and so is rollback, flipping back to the pinned ID. That is exactly why the whole sequence has to finish while the old model is still being served. A rollback plan with an expiry date is a deadline wearing a costume.

One migration, traced

Say the workload is invoice extraction: an agent reads incoming invoices and posts structured totals to an accounting system. The old snapshot retires in six weeks. Here is the shape of a migration that respects the clock.

One migration traced end to end: an invoice-extraction agent's baseline is frozen by document-type slice, the candidate model wins overall but drops on credit notes, a prompt fix targets that slice alone, the re-run and a 5% canary pass, and the candidate is promoted with two weeks to spare before the old snapshot retires.

The baseline is frozen by document type: standard invoices, credit notes, multi-page scans, invoices in a second language. The candidate scores two points higher overall and drops sharply on credit notes, where it returns the gross figure where the old model returned the net. Nothing in the aggregate score objected. The slice did.

The repair is narrow: an explicit instruction about credit notes, tested against that slice first, then the full set to confirm nothing else moved. The candidate shadows for several days of real traffic, takes a 5% canary share, and is promoted with two weeks to spare. Every step is ordinary release engineering. The only unusual part is remembering that a model needs it.

Where the gate leaks

An eval set measures the workload you had when you built it. Traffic drifts, new document types arrive, and a gate that passed in spring can certify a migration in autumn against a world that no longer exists. Refresh the set from recent production failures, or it becomes a museum.

Outputs are also non-deterministic, even at settings that should pin them down. A single replay run will show differences that are noise wearing the costume of signal. Repeat runs and thresholds, not exact-match comparisons, are what separate a regression from a coin flip.

And the rollback really does expire. Once the retirement date passes, the old behaviour is gone. Not deprecated, gone. Any problem you find afterwards gets fixed forward, on the new model, under time pressure. That is the argument for starting at the deprecation notice rather than the shutdown reminder. It is also the argument for not jumping on day one: a replacement that has been serving other people's traffic for a few months has a track record your evals can learn from. Use the notice window. It is the one part of this the providers actually give you for free.

Work with us

If your systems are pinned to snapshots with retirement dates on them, or worse, riding aliases nobody has scheduled, we run the migration like the release it is. Start at Trida AI's intake.

The Trida AI Academy: Courses, Skill Tests, and Blueprint Builds

FDE at Trida AI — Wed, 21 Oct 2026 01:28:35 +0000

Originally published at Trida AI.

Copy as markdown

Every company hiring forward deployed engineers runs into the same problem: the role is proven in production, but it is hired on paper. A resume lists technologies. A profile lists keywords. Nothing in the standard process shows whether an engineer can take a problem, build the system, and ship it. So hiring becomes a bet, and when the bet is wrong, it surfaces inside a customer's deployment.

Trida AI Academy is the proof system we built instead. At its core are three kinds of evidence: certification courses an engineer passes to earn a credential, skill tests that badge individual skills under time pressure, and blueprint builds — complete agents, built to a specification and reviewed by a person. Each one leaves evidence. All of it lands on a single profile. And that profile is what carries an engineer into Trida Network, where recruiters and companies see verified skill instead of claims. That is the flywheel: prove you can ship, be seen for it, get matched on it.

Proof one: certification

A certification course is the foundation. Engineers work through readings and hands-on labs, material built from real deployment work: production repositories, evaluation sets, the failure modes that only appear after launch. Every module ends with a quiz. There is no proctor and no cohort schedule; if a module does not clear, the engineer goes back, learns the part that was missed, and takes it again. Clear the core modules and an elective, and the credential issues automatically, carrying a unique ID that anyone can verify.

The credential is not attendance. It is a recorded pass, and the ID means a recruiter, a customer, or we ourselves can verify it.

Proof two: skill badges

Certifications establish breadth. Skill tests pin down the individual skills a deployment turns on. Each test is timed and scored, one skill at a time. Pass, and the badge goes on the engineer's profile.

The granularity matters. An engineer can be strong on retrieval and new to orchestration. A shelf of badges says precisely where the strength is, skill by skill, each one earned under time pressure rather than typed into a profile. For anyone evaluating an engineer, it reads like a map instead of a claim.

Proof three: blueprint builds

The third proof is a finished system. In Blueprint Builder, the engineer takes a specification for a real working agent in a real business vertical, written the way production systems are actually run, with a human holding the decisions that carry consequences. They build the agent end to end, publish it on GitHub, and submit it for review. A person reviews and certifies the build; it is not auto-graded, because what makes a build good is judgment, and judgment does not reduce to a test suite.

That matters for forward deployed work, because the job is never a slice of a system. It is the whole system, shipped.

Three kinds of proof: a certification course proves the foundation, skill tests prove individual skills under time pressure, and a blueprint build proves you can ship the whole system — reviewed, public, and verifiable.

The profile is the point

None of the three proofs lives in a drawer. The credential, the badges, and the reviewed build collect on one Academy profile: a record a third party can open and inspect. Verify the credential ID. Read the build on GitHub. See which skills carry badges. An engineer moving toward forward deployed roles does not have to argue that they can ship. The profile shows the evidence, in the open, before the first conversation.

Then the Network takes it from there

A verified profile is only useful if the right people see it. Trida Network matches Academy-certified engineers with companies, ranked by verified skill, not resume keywords, and introductions are reviewed by a person before they happen. Recruiters browsing the Network are looking at proof profiles: the credential, the badges, the public build. Companies can shortlist on evidence that was produced in the open, and engineers get seen for what they demonstrably can do.

The Trida flywheel: certifications, skill badges, and blueprint builds collect on one verified profile; the Network puts that profile in front of recruiters; and the Academy's material is built from real deployment work.

Why we built it this way

Trida AI is a deployment company, and the quality of a deployment is the quality of the engineers on it. "Trust us" is not a quality system. The Academy makes the standard inspectable: the courses, the tests, and the blueprints are public, and anyone can look at what they ask of an engineer. For companies, that means an engineer's skills can be evidenced before they ever touch a production system. For the market, it means the supply side of forward deployed talent gets built on proof instead of paper — and the loop is designed to compound: engineers prove themselves in the Academy, the Network matches them on that proof, and the Academy's material is built from real deployment work, so every round of the system is grounded in the work itself.

What proof does not settle

Proof covers skill. It does not measure fit, taste, or how an engineer works inside a particular team, and no credential settles those questions. That is why the blueprint review is a human judgment and why Network introductions are reviewed by a person. Evidence removes the cheapest doubt in hiring — whether the skill is there at all. The rest still takes people.

See the proof system

Start with the courses, where the foundation is earned and certified.

Test a single skill against the clock with a skill test, and put the badge on a profile.

Or go straight to the build itself: build a blueprint, in the open, reviewed by a person.

Sandboxing Agents That Run Code

FDE at Trida AI — Wed, 21 Oct 2026 01:26:35 +0000

Originally published at Trida AI.

Copy as markdown

Give an agent a shell and you have given it your shell.

Same home directory. Same SSH keys sitting in it. Same cloud credentials in the environment, same open network connection. The agent does not need to be malicious for this to go wrong. It needs to be obedient, following an instruction that arrived inside a web page, a package README, or a tool result you asked it to read. Prompt injection stops being a content problem the moment the reader can execute commands.

The usual first answer is permission prompts, a human approving each command before it runs. That decays in practice. Approving the fortieth command of a session is not a review, it's a reflex. Anthropic reports that sandboxing safely reduced permission prompts by 84% in its internal usage with Claude Code. Fewer prompts, applied where they matter, beat more prompts applied everywhere.

A prompt asks. A boundary refuses.

A sandbox (the isolated environment an agent's code runs in, in this post) is a boundary enforced below the agent, by the operating system or a hypervisor, not by the model's good behaviour. It is a short list of allowances. These directories may be read, these may be written. These network hosts may be reached. This much CPU and memory, for this long. Inside the list, the agent works freely and nobody approves anything. Outside it, a strict configuration has nothing to approve either: the call fails the way a call to a nonexistent file fails. Looser configurations soften the refusal into a notification and a chance to run the command outside the boundary instead, and that softness is the permission prompt creeping back in through the side door.

That placement is the whole argument. A system prompt that says "never read files outside the project" is advice the model can be talked out of by the next document it reads. An operating-system boundary cannot be talked out of anything. It also covers the part teams forget: the scripts, installers and subprocesses a command spawns. A boundary drawn around the parent has to hold for all of them, and the sandboxed Bash tool in Claude Code is built that way: its limits apply to every process a command starts, too.

Two walls, not one

The boundary needs a filesystem wall and a network wall, and each one is pointless alone. Filesystem isolation without network control still lets a script send anything it can read, anywhere. Network isolation without filesystem control still lets it read and alter the machine it lives on, including the configuration of the very proxy meant to contain it.

Together they close the loop. The filesystem wall says the working directory is the only writable place, and the paths that matter cannot be read at all. The network wall says there is no direct route out: traffic passes through a proxy running outside the sandbox, and the proxy checks each destination against an allowlist of domains.

The read half of that wall is the one teams leave open by accident. Claude Code's documentation is blunt about its own default: sandboxed commands can read most of the machine, credential files included, unless those paths are explicitly denied. A sandbox switched on with its read defaults untouched still lets a manipulated script read your keys; it just cannot send them anywhere while the network wall holds. Deny the credential paths explicitly. Do not let one wall cover for the other.

Credentials get the same treatment, and this is the part worth copying even if you copy nothing else. The real secret never enters the sandbox. Cloudflare's sandbox platform does this as secure credential injection: an egress proxy adds authentication to outgoing calls at the network layer, so the agent can use a service without ever holding its credential. If the code inside is compromised, there is no key inside to steal; the proxy authenticates only specific requests, to specific hosts.

The sandbox boundary: the agent and the code it runs sit inside an OS-enforced boundary with a writable working directory and no direct network route; all traffic passes through a proxy outside the boundary, which checks hosts against an allowlist and injects credentials; the host filesystem, SSH keys and cloud credentials remain outside, unreachable under the configured boundary.

Match the boundary to the tenant

Not every deployment needs the same strength of wall. Overspending here is its own failure: a sandbox so slow or so locked down that the agent cannot install a dependency will be switched off by Friday.

For a single developer's own agent on their own machine, an OS-level sandbox is usually the right weight: Linux bubblewrap and macOS Seatbelt primitives, with no container to spin up or manage. The threat is a manipulated agent reaching beyond the project, and the OS boundary answers it.

For shared infrastructure (many sessions, many users, code you did not write), you want a kernel between the workload and the host. An application kernel such as gVisor intercepts the container's system calls and services most of them in user space, limiting what reaches the host kernel directly, which is why Google Cloud offers it as GKE Sandbox for pods that need hardening. A step beyond that, a microVM gives each workload its own kernel behind a hypervisor. Firecracker microVMs were built at AWS for this multi-tenant case, and the project's own figures put boot time under 125 milliseconds, with under 5 MiB of memory overhead per microVM. Isolation priced in milliseconds.

Whichever layer you pick, three settings matter as much as the layer: a wall-clock limit, so a runaway loop dies on its own; resource caps, so one session cannot starve the host; and an ephemeral filesystem, so anything the run created dies with it unless deliberately exported.

One run, traced

Take a typical run, configured the way this post argues for, and watch where the walls meet the work. The task: clone a repository, run its test suite, fix the one failing test.

One run traced end to end: the agent clones through the credential proxy, edits and tests freely inside the workspace, an install script's attempt to read the SSH directory and contact an unlisted host are both refused by the boundary, and the finished diff leaves through the proxy while the sandbox is destroyed.

The agent's git client authenticates to the proxy with a scoped credential; the proxy verifies the repository and branch, attaches the real token, and forwards the request. The agent then edits files and runs the test suite inside the workspace — dozens of commands, zero permission prompts, because none of them can leave the boundary.

Mid-run, a dependency's install script does two things its author should not have asked for. It tries to read the SSH directory, and it tries to send a request to a host that is not on the allowlist. Inside the sandbox the first read is refused: that path was explicitly denied when the boundary was configured, so for this run it might as well not exist. The second request reaches the proxy and is refused, and the refusal is logged with the host named. The install continues without that step; the tests pass. The diff and the logs are the only things that leave; then the sandbox is destroyed, along with everything the install script wrote.

What a sandbox does not do

It does not judge intent inside the walls. An agent allowed to write to a repository can still write a bad change to that repository, and an agent allowed to call an API can still make a wrong call the API permits. Containment answers "how far can this reach?" It says nothing about "should it?" Reviews, evals and scoped permissions still carry that question.

Its coverage also has edges you have to read, not assume. In Claude Code, the documented boundary wraps shell commands; its file tools, MCP servers and hooks run outside it, following permission rules instead. A team that enables the sandbox and declares the agent contained has checked one door of several. Whatever runs the agent, the agent process itself included, needs to sit behind a boundary if the claim is going to hold.

Set the allowlist from a specification and it will be wrong in both directions: hosts the work needs will be missing, and hosts nobody needs will sit there because someone added them "just in case". Set it from observed traffic during a supervised run, then tighten. A proxy log that has never denied anything usually means the list is too wide to mean much.

Not every agent needs one, either. An agent with no shell, no code execution and a handful of scoped, read-only API tools has a small surface, and a sandbox around it buys little. Sandbox the execution. That is where the reach is.

Work with us

If your agents run code on machines that also hold production credentials, the useful question is not whether the model behaves — it is what a bad run can reach. Drawing that boundary, and setting the allowlist from your agents' real traffic, is deployment work we do. Tell us about it through Trida AI's intake.

Too Many Tools Hurt Agents

FDE at Trida AI — Wed, 21 Oct 2026 01:19:50 +0000

Originally published at Trida AI.

Copy as markdown

Connecting one more MCP server is the easiest way to make a working agent worse.

Nobody does it on purpose. The agent handles support tickets well, so someone adds the billing server. Then the CRM. Then the deployment tools, because the agent kept asking a human for things it could have looked up itself. Each addition is reasonable. The result isn't. The agent that was reliable in June is picking the wrong tool by September, and the post-mortem blames the model.

The model is usually innocent. What changed is the size of the choice it faces at the start of every request.

Every tool an agent might call has a definition: a name, a description, a parameter schema. In the standard setup, all of those definitions are loaded into the model's context before it reads a word of the request. Anthropic's worked example of a five-server setup (GitHub, Slack, Sentry, Grafana, Splunk) comes to 58 tools consuming roughly 55K tokens before the conversation starts, and Anthropic reports seeing definitions reach 134K tokens. That's a tax paid on every request, for tools the request will mostly never touch.

The tax is the visible part. The accuracy cost hides better. Anthropic notes that the most common failures in that setup are wrong tool selection and wrong parameters, especially between similar names. Its example pair is notification-send-user and notification-send-channel, two definitions that differ by one word and opposite intent. When Anthropic's Tool Search Tool loaded only the few tools a request actually needed instead of the whole catalogue, token use fell by about 85%, and accuracy on its internal MCP evaluations rose sharply: Opus 4 from 49% to 74%, Opus 4.5 from 79.5% to 88.1%. Same models. Smaller choice.

The catalogue is not the working set

Here's the distinction we think teams miss. A catalogue is everything the agent is allowed to reach, ever. A working set is what this request needs, usually three to five tools. Loading the catalogue to serve the working set confuses storage with attention. You wouldn't hand a new engineer the manual for every system in the building before their first ticket. You'd tell them where the manuals live.

So the fix is retrieval, applied to tools the way it's already applied to documents. A search step reads the request, ranks the catalogue, and loads a shortlist. The RAG-MCP study tested exactly this against a large pooled catalogue: with every definition in context, tool-selection accuracy sat at 13.62%; with retrieval narrowing the candidates first, it reached 43.13% accuracy, while prompt tokens fell by more than half. A 43% score is nothing to celebrate in absolute terms. The direction is the finding: the model didn't get smarter, the question got smaller.

A more recent shortlist study adds a useful wrinkle. On the BFCL benchmark, an adaptive shortlist that showed about seven tools per query, chosen per request, matched the coverage of showing fifty every time, 90.3% against 90.8%. The same study found the catch on ToolBench: a fixed shortlist of five found nothing at all on the hardest queries, where the right tool ranked sixth to twentieth, while the adaptive one, allowed to look deeper when the first pass was thin, still found 16.7% of them. Shortlists have to be able to widen. A retrieval step that can't miss loudly will miss silently.

It also means the raw count was never quite the villain. Fifty well-separated tools can be an easier choice than ten that overlap. What degrades selection is neighbours: tools whose descriptions promise nearly the same thing, so the model splits its confidence between them and picks by wording accidents. Count is just the easiest way to accumulate neighbours.

Deferred loading in practice: the request is searched against the full tool registry, a shortlist of three to five tools is loaded, and the model chooses from that shortlist plus the small resident core. The rest of the catalogue never enters context.

What we'd change first

Start by splitting the catalogue. A handful of tools get used on most turns. Those stay resident, loaded every time. Everything else is deferred: present in a registry the agent can search, absent from its context until a request actually calls for it. On most stacks this is configuration, not surgery.

Then consolidate. Anthropic's guidance on writing tools for agents is blunt that more tools don't always lead to better outcomes, and its remedy is to build the tool around the workflow instead of the API: one scheduling tool that checks availability and books, rather than separate tools for listing users, listing events and creating events that the model has to chain correctly every time. Every consolidated tool deletes several near-neighbours at once.

Descriptions get the same treatment. A good tool description reads like directions to a colleague: what it does, when to reach for it, and — the part everyone skips — when not to. "Use this to message one person. For a channel, use the channel tool." Two sentences like that do more for selection accuracy than a model upgrade, because most wrong-tool calls are really two descriptions that never agreed on the boundary between them.

Last, measure selection itself. Log which tool was chosen, which were on the shortlist, and which one the completed task actually needed. Wrong-tool rate is a metric you can watch fall as you defer, consolidate and rewrite. If it doesn't fall, the problem is somewhere else and you've learned that cheaply.

One request traced end to end: posting a deploy failure to the incident channel. Search over the deferred catalogue returns three candidates, the model picks the channel tool over its one-word neighbour, and the other fifty-five definitions never enter context.

Where this bends

A six-tool agent doesn’t have this problem, and adding a search step to it buys latency and a new failure point in exchange for nothing. Anthropic’s own guidance puts the payoff at ten-plus tools, or once definitions pass roughly ten thousand tokens. Below that, load everything and spend your effort on the descriptions.

Retrieval also moves the failure rather than deleting it. If the right tool isn’t in the shortlist, the model can’t call it, and it will often complete the task with the wrong tool rather than report the gap. That’s why the widening behaviour matters, and why the resident core should be chosen from logs of what your agent actually calls, not from what the architecture diagram says it should.

And none of this rescues bad tools. Defer a confusing pair and they still confuse, just later and in a smaller room. The shortlist is a way to stop paying for the whole catalogue on every request. The catalogue still has to deserve the agent’s trust: one tool per job, boundaries written down, names that don’t collide.

Work with us

If your agent’s tool list has grown server by server and its wrong-tool calls have grown with it, the catalogue is where we’d start: what stays resident, what gets deferred, and which tools should never have been separate. Bring us the list through Trida AI’s intake.

Structured Output Is Not a Guarantee

FDE at Trida AI — Wed, 21 Oct 2026 01:18:07 +0000

Originally published at Trida AI.

Copy as markdown

The invoice parses perfectly. Every field is there, the total is a number, the currency is one of the allowed values, the date is a date. Your pipeline writes it downstream without a murmur — and the total is wrong. The model transposed two digits, the schema never noticed, and nobody was ever going to notice, because the check everyone trusted had already passed.

This is the misunderstanding we see most often in production AI work right now. Structured output, schema-constrained generation, is a real advance, and teams keep reading its guarantee as bigger than it is. The guarantee covers the shape of the answer. It was never about the truth of the values inside it.

What the constraint actually buys

The mechanism matters here, because the mechanism is what makes the guarantee strong in one place and absent in another. When you send a JSON Schema with the request, the provider compiles that schema into a grammar and constrains generation against it: tokens that would break the schema are not available to the model as it writes. OpenAI describes Structured Outputs as ensuring the model will always adhere to your supplied JSON Schema, so a required key is not omitted and an enum value is not invented. Anthropic describes its implementation the same way: a guarantee delivered through constrained decoding.

That buys you something genuine. A whole class of plumbing failures disappears: no more prose wrapped around the JSON, no missing field three lines before a crash, no retry loop whose only job is begging for the same shape again. If your system consumes model output in code (and in production, something always does), this is the difference between a format you hope for and a format you can build on.

But notice what the grammar cannot see. It knows that total must be a number. It does not know what the invoice said. It knows status must be one of four values. It does not know which one the document in front of it supports. Shape is a property of the answer alone; truth is a relationship between the answer and the world, and the grammar only ever sees the answer.

The validation stack: constrained generation produces the shape, a schema check confirms it, semantic checks test the values against the source and the rules, and only then does output reach the downstream system. Failures go to one bounded repair attempt, then to a reject queue.

Three ways a valid answer is still wrong

The first is the plain wrong value: a transposed total, a date read from the wrong line of the document, a name taken from the signature block instead of the letterhead. Every field is the right type. The extraction is simply incorrect, and schema validation is structurally incapable of catching it.

The second is internal inconsistency. Line items that sum to 4,280 and a total field that says 4,820. A start date after the end date. A headcount of twelve and a list of nine names. A schema describes fields one at a time; many production errors live in the relationships between fields, which is exactly the territory a schema does not cover.

The third is subtler, and it is created by the constraint itself. In OpenAI's strict mode, every property in the schema is required — there is no honest way to say "the document doesn't state this." So when the source is silent, the model still has to put something in the field. A required field pointed at a document that doesn't contain the answer manufactures a plausible value. The tighter you make the schema, the more important it becomes to give the model an explicit way to say "not stated" (a null where the schema allows one, or a dedicated value in the enum), and to treat that answer as a routing signal, not a failure.

The guarantee has edges, and the vendors say so

Read the provider documentation closely and the guarantee is narrower than the marketing version. Anthropic's documentation is unusually candid about this. Some constraints in your schema — numeric bounds, string lengths — are not enforced during generation at all; the SDKs rewrite them into field descriptions, and validating SDK helpers check them against the original schema afterwards. A refusal or a response cut off at the token limit can return output that doesn't match the schema. Even enum casing is not guaranteed: a value can come back differing from the schema only in capitalization, with no error raised.

None of this is a reason to avoid structured output. It is a reason to keep your own validator downstream of it, permanently, even on the days it never fires. The bad days are the ones that matter.

Validation first, repair second, rejection always available

The production pattern we argue for has three layers, and the order is the point.

First, deterministic semantic checks — the ones a schema can't express. Totals against line items. Values against the source text: does this figure actually appear in the document? Codes and identifiers against the systems that own them: does this purchase order exist? These checks are ordinary code, cheap to run, and they catch the errors that actually cost money.

Second, bounded repair. When a semantic check fails, send the output back once — with the specific error attached, not a generic "try again." "Total does not match the line-item sum of 4,280; re-read the source" is an instruction a model can act on. One attempt, occasionally two. A repair loop with no bound is not robustness; it is a system that has decided to keep guessing until something passes, and something eventually will.

Third, a reject path that is a real destination: a queue a person works, with the source, the failed output, and the failed check all attached. The reject queue is also your best design feedback. A field that lands there every week is telling you the schema, the prompt, or the source document is wrong, and that signal is worth more than another retry.

One invoice through the stack: the model's schema-valid answer carries a transposed total, the line-item check catches it, a single repair with the error attached corrects it, and the verified record is what reaches the downstream system.

Where this leaves the schema

None of this argues against schemas. It argues for treating the schema as the first layer of a validation stack rather than as the stack. Design it deliberately: field order matters, because generation follows it — put the evidence a value rests on before the value where you can, and let the model work outward from the source instead of committing early and rationalizing backwards. Keep schemas inside the providers' complexity limits, and expect the first request against a new one to pay a grammar-compilation cost that later requests don't.

The honest summary is short. Structured output ended the era of parsing apologies: the almost-JSON, the missing key, the shape that changed between calls. What it did not end, and never claimed to if you read the documentation it ships with, is the need to check whether the answer is right. Shape is guaranteed. Truth is still your job.

Work with us

If your team's extraction or agent pipeline trusts schema-valid output a little more than it should — and most do, right up until the first wrong value reaches a system of record — that gap between shape and truth is exactly the kind of problem we work on. Tell us about it through Trida AI's intake.

Multi-Agent Systems in Production

FDE at Trida AI — Wed, 21 Oct 2026 01:11:22 +0000

Originally published at Trida AI.

Copy as markdown

Every vendor demo now seems to end the same way: one agent hands work to another, a third checks it, a fourth reports back. It looks like a team. It demo-animates like a team.

Production is less kind. The honest summary of what has actually been measured so far is narrower and, we think, more useful: multiple agents help a lot on one shape of problem, and actively hurt on another. The dividing line isn't the framework you pick or the model you run. It's how tightly the pieces of the work depend on each other.

Where it earns its keep

The strongest published result is a research system, and the shape of the task explains the result. Anthropic's multi-agent research system puts a lead agent in charge of planning and synthesis, with subagents searching in parallel, each in its own context window. On Anthropic's internal research evaluation it outperformed a single agent by 90.2%, and token usage alone explained most of the performance spread the team measured. Research rewards that shape: the question breaks into independent directions, each direction is mostly reading, and a subagent that goes down a dead end doesn't corrupt the others. Its findings come back as text for the lead to weigh.

Notice what the subagents don't do. They don't build on each other's half-finished work. They don't make decisions the others have to live with. They gather, compress and report, and the one agent that decides sees all of it.

That pattern generalises beyond research. Checking a claim against many sources. Reading a large document set where each document can be judged on its own. Triaging a queue of independent items. Fan-out works when the units are separable and the cost of one unit failing is a retry, not a rebuild.

Where it breaks

Split a piece of making in half and the failure shows up immediately. Cognition's widely argued essay, Don't Build Multi-Agents, makes the point with a small example: two subagents building halves of one game, each working reasonably, producing pieces built on conflicting assumptions that no final agent can cleanly merge. Their underlying claim is the one to hold onto: every action an agent takes carries decisions it never stated, and agents that can't see each other's decisions will eventually make incompatible ones.

This isn't only an argument from one team's experience. The MAST study built a failure taxonomy from more than 1,600 annotated traces across seven multi-agent frameworks, found failure rates of 41% to 86.7% across the systems it examined, and sorted the failures into three buckets: system design issues, misalignment between agents, and failures of verification. Read that list slowly. None of the three is "the model wasn't smart enough." They're coordination failures, the same class of problem that makes human projects fail, except that agents hold the misunderstanding silently and execute it at machine speed.

There's a second, quieter cost. More agents means more tokens, and not a little more. Anthropic reports its multi-agent system using roughly 15 times the tokens of a chat interaction, against about four times for a single agent. Fifteen times the spend buys a real gain on the right task. On the wrong one it buys you a faster, more expensive route to an answer one agent would have reached, plus a synthesis step where errors from several threads get averaged into something that reads as consensus.

The test we apply

A worked example — one claim checked end to end: the brief the lead writes, what a worker returns, and the check at the boundary before anything is merged.

Before a deployment goes multi-agent, we want four answers, and they have to be yes in the work itself, not in the pitch for it.

Can the task be split into pieces that don't need to see each other working? If piece B changes its mind when it learns what piece A decided mid-flight, you don't have two tasks. You have one task and a coordination problem wearing a costume.

Is the work mostly reading, searching and judging — rather than writing into shared state? Parallel readers are cheap to reconcile. Parallel writers are not.

Can each piece's output be checked on its own, before anything is merged? A verifier that only sees the combined result arrives too late; by then you can't tell which thread introduced the error. Verification belongs at the boundary of each subagent's work, where a bad output is still small.

Is the task valuable enough to pay roughly an order of magnitude more in tokens for a faster or broader answer? Sometimes it plainly is. Often the honest answer is that a single agent, given better tools and a well-built context, gets there for a fraction of the cost.

When those answers are yes, the architecture that holds up is boring in a good way: one lead agent that plans, delegates with written briefs that state the objective, the boundaries and the output format, and synthesises; workers that return findings rather than half-built artifacts; and a check on each return before it enters the lead's picture of the world. Anthropic's team found the briefs mattered more than expected: vague delegation was behind duplicated work and gaps in its early versions. That's familiar to anyone who has managed people. It turns out to be just as true here, and less forgiving, because an agent won't come back and ask what you meant.

The architecture that holds up — one lead agent plans and synthesises, workers search in parallel in their own context windows, and every return is checked before it merges.

What still isn't working

Real-time coordination is the open problem. Agents are poor at renegotiating a plan mid-task, at noticing that a sibling's output invalidates their own, at knowing when to stop. Frameworks will keep adding protocols for agent-to-agent communication, and some of that plumbing is useful. But a message bus doesn't fix a spec problem, and most of the failures in the MAST taxonomy are spec and coordination problems, not plumbing problems.

Our position is simple, and it will probably age: default to one agent. Add a second when the task's shape demands breadth that one context can't cover (not the demo, not the org chart in the slide), and keep the deciding in one place while you do it. The teams publishing the best results are, underneath the diagrams, doing exactly that.

Work with us

Splitting a workflow across agents is easy to demo and hard to keep honest in production. If you're deciding where a single agent ends and a team of them should begin, bring the workflow to Trida AI's intake and we'll work through the shape of the task with you.

Long-Horizon Agents Need a Map, Not a Bigger Context Window

FDE at Trida AI — Wed, 21 Oct 2026 01:09:38 +0000

Originally published at Trida AI.

Copy as markdown

An agent on a long task usually loses its way well before it runs out of context. The window still has room. The transcript holds everything it has done, in order, at full length. What it no longer holds, in any usable form, is where the work stands.

That distinction matters because the fix most teams reach for is size. Give the model a larger window, feed it more history, let it carry the whole job in its head. Size helps with one problem: information that was dropped entirely. Most long-horizon failures are a different problem. The information is present. It is buried under tool output, dead ends and superseded attempts, and the model has to reconstruct its own position from that pile on every step.

The trend line makes this urgent rather than academic. METR's measurements put the length of tasks agents can finish at 50% reliability as doubling roughly every seven months, and the same work found success falling below 10% on tasks that take a person more than about four hours. Tasks are getting longer faster than raw memory is getting more reliable. So the question for a deployment is not how much an agent can hold. It is how the agent knows where it is.

A transcript is not a map

A transcript records a journey the way a dashcam does: complete, sequential, and almost impossible to navigate by. Ask it a simple question mid-task (what have we already ruled out?) and the answer has to be mined from hundreds of turns, some of which contradict later turns.

Worse, a long transcript actively degrades the reasoning done over it. Chroma's Context Rot study tested 18 production models and found performance getting less reliable as input length grew, in uneven ways, even on simple retrieval and copying tasks. Similar but irrelevant material hurt more than plain filler. A long agent transcript is exactly that material at scale: near-miss attempts, stale intermediate results, and tool output that was useful for one step and noise ever after.

A map solves a different problem than memory does. It does not try to keep everything. It keeps position: the goal, what is done, what was decided and why, what is still open, and where the underlying detail lives if it is needed again. A driver crossing a continent does not memorise every mile driven. They carry a map, a current location, and a route — and they can re-orient in seconds after any detour.

Same journey, two records. The transcript keeps everything and answers nothing quickly. The map keeps position and points to the detail.

What goes on the map

Keep it small enough to read at a glance, and structured enough that updating it is a habit rather than a chore. In our view five entries earn their place.

The goal, stated once, in the terms the work will be judged by. Then current position: the last verified state of the work, not the last thing attempted. Decisions belong here too, with the reason attached, because a decision without its reason gets relitigated three hours later by an agent that cannot tell it was settled. Open ends come next: the questions still live, the branches not yet taken, the failures already ruled out. Finally, pointers. File paths, document links, query names: lightweight references the agent can follow to reload detail on demand, instead of carrying that detail in context the whole way.

That last entry does quiet, important work. Anthropic's writing on effective context engineering describes the same pattern from the other side: keep identifiers in context, load the underlying data just in time through tools, and treat context as a finite attention budget rather than a storage bin. Their long-horizon toolkit — summarising history when it grows unwieldy, keeping structured notes outside the window, and letting focused sub-agents return short distilled summaries instead of raw workings — is, in effect, map maintenance described three ways. We agree with the toolkit. Our argument is about priority: build the map first, deliberately, as a designed artifact. The rest follows from it.

A map also changes what a fresh start costs. When a run is compacted, restarted, or handed to another agent, the newcomer does not need the saga. It needs the map and a few pointers. Anthropic's later work on harnesses for long-running agents makes a parallel point in software work: durable progress files and a structured list of what is done and what is not let a new session orient itself and make incremental progress instead of declaring victory early or redoing finished work. Orientation, in both accounts, is a file — not a memory.

Keep the map honest

A map that is written once and never updated is worse than none, because the agent trusts it. So the working rule is simple: no step counts until the map reflects it, and verification is what updates position. An agent that has attempted a fix has moved its transcript forward. It has only moved its position when a check (a test, a query result, a human confirmation) says the fix holds.

That rule also gives long runs a natural rhythm. Work in increments small enough to verify. Record the outcome, including the failures, since a ruled-out branch written on the map never gets explored twice. When the map and the territory disagree, believe the territory, fix the map, and note the disagreement. It usually marks the spot where the task was misunderstood.

The loop that keeps a long run oriented: position changes only when a result is verified.

Where a map does not help

Short tasks do not need one. If a job finishes in a handful of steps inside one context, map-keeping is ceremony, and the transcript is the record. Maps also decay. On a fast-changing task the maintenance cost is real, and a stale map misleads with confidence, which is why the verification rule above is not optional decoration.

There is a harder limit underneath. A map fixes orientation; it does not fix judgment. An agent that misreads its goal, or verifies its work against the wrong check, will maintain an excellent map of the wrong journey. And summarising a long history into map form always discards something whose importance only shows up later. Every team that compacts context learns to tune what gets kept, starting generous and tightening with experience. The map is a discipline, not a guarantee.

Still, the direction is clear. Windows will keep growing, and long tasks will keep degrading inside them for the reasons the context-rot results describe. The teams whose agents finish four-hour, eight-hour, day-long work reliably will not be the ones who waited for a window big enough to hold the whole job. They will be the ones who taught their agents to carry a map, keep it current, and check their position against the territory as they go.

Work with us

If your team is stretching agents from ten-minute demos toward work that runs for hours, the bottleneck is probably orientation rather than model size — and it is fixable with structure, verification and honest state-keeping. Tell us where your long runs lose their way through Trida AI's intake, and we will map out what it takes to keep them on course.

The FDE Shortage Is a Proof Problem

FDE at Trida AI — Wed, 21 Oct 2026 01:02:54 +0000

Originally published at Trida AI.

Copy as markdown

Earlier this month, Business Insider reported that Asa Kalavade, Amazon Web Services' vice president for forward-deployed engineering, calls qualified FDE candidates "unicorns". AWS is not short of commitment. It announced a $1 billion forward-deployed engineering organization in June and plans to staff it with thousands of engineers. Forrester estimates that Amazon, Microsoft, OpenAI and Anthropic pledged roughly $9 billion to FDE ventures between May and July alone. The money is there. The people are the constraint.

Kalavade was specific about what AWS screens for. Three things: a product customer mindset, deep software expertise, and hands-on experience building with agents. Most engineers have one of the three, maybe two. The unicorn is the one carrying all of them at once.

So where does AWS find them? Mostly, it doesn't — yet. The first hires are seasoned operators, people who have already done deployment work elsewhere. That works as a seed. It does not scale into thousands, because the pool of finished FDEs is small and every hyperscaler is fishing in it at the same time. Universities are not refilling it either; per the same reporting, colleges are still building their first FDE training programs. An industry that needs thousands of a new kind of engineer has almost no way to certify one — and no shared proof that a given engineer can do the job.

Two of the three skills never show up on a resume

Look at AWS's list again. Software expertise is the easy third: work history shows it, a technical interview can probe it. The other two are different. A product mindset means the engineer walks into a customer's operation and finds the problem worth solving, not just the ticket they were handed. Agentic experience means they have actually shipped systems where models act — with the guardrails, evals and failure handling that production demands, not a demo.

Neither appears on a CV in any form a recruiter can check. Candidates claim both; interviews guess at both. This is why FDE hiring stays slow even with unlimited budget behind it. The bottleneck is not finding applicants. It is verifying the two skills that decide whether a deployment works, and a resume is the wrong instrument for that job.

Trida AI's route: engineers earn an Academy certification against the three skills the role demands, the credential makes that skill verifiable by anyone, and the Network matches certified engineers to companies by verified skill.

How Trida AI built the certification route

Trida AI was set up around this exact gap. We deploy forward-deployed engineers into enterprise clients, and we kept hitting the same wall as everyone else: skill that could not be checked before an engagement started. So we built the other side of it — a way for engineers to prove the skill, and for companies to hire on that proof.

The Trida AI Academy does not train anybody. Its certifications are free, self-paced and always open — no cohorts, no schedule — and they are earned, not attended: readings, hands-on labs, and quizzes the engineer passes themselves, on material built from real deployment work, production repos, eval sets and failure modes. Clear the bar and the credential issues with a unique ID that anyone can verify. What gets certified is the same three dimensions AWS screens for: production-grade software, agentic systems built and evaluated by hand, and business problems worked end to end instead of toy examples.

The Trida Network is where certified engineers meet the work. It matches Academy-certified forward-deployed engineers with companies that need them, ranked by real, verified skill rather than a resume keyword search — and when a role and a candidate line up, a person reviews the match before the introduction is made. An engineer who has demonstrated the three-part package does not need to call themselves a unicorn. The evidence is attached to the profile.

That gives a company two ways in: deploy with Trida AI's own engineers, or hire certified talent directly through the Network and run the deployment yourself. Either way, the starting point is the same — skill that has been tested before the engagement begins, not discovered during it.

The hyperscalers will solve their own hiring problem eventually; $9 billion buys a lot of persistence. But most companies do not need thousands of FDEs. They need two, or five, or a pod for one hard workflow — and they cannot wait for the talent market to sort itself out. For them, the question is whether proof of FDE skill exists outside any single vendor's hiring process. It does now. Building it was the point.

Work with us

If you need forward-deployed engineers and the job market is offering you unicorns, start with talent that is already certified and verified — tell us what you are trying to deploy through Trida AI's intake.

Insurance Cash Reconciliation Agent Blueprint

FDE at Trida AI — Wed, 21 Oct 2026 01:01:09 +0000

Originally published at Trida AI.

Copy as markdown

Every morning, the cash team at a specialty insurer faces the same pile: money landed in the bank overnight, and someone has to work out which policies it pays. The wire amounts rarely match the bills. One transfer covers dozens of policies, explained only by an advice written as prose. An insured has paid late, or short, or twice. Some of the money matches nothing anyone can see.

The errors in that pile are quiet ones: cash applied to the wrong policy looks exactly like cash applied to the right one. So this open blueprint, the Insurance Cash Reconciliation Agent, takes the working-out as its job. The matching arithmetic is checked in code. A model is used only where judgment is needed: reading the unstructured paper, proposing where unexplained money belongs. And nothing posts itself: a named controller approves every posting, and a day counts as reconciled only when every cent of it is accounted for.

One note before the machinery: the carrier in the repository is Northgate Specialty Insurance, and it is a reference prototype. The architecture and the code are the deliverable, and both are public.

Why the cash never equals the bill

Most premium arrives through brokers, and brokers remit net: they deduct their contracted commission and send the rest. A wire for less than the billed premium is usually not a short payment. It is the correct payment, and recognising it means knowing the contract behind it, and which of its rates was in force the day the cash arrived. Rates change mid-book; the change reprices nothing backwards.

That history also exposes a particular failure shape. When a broker deducts at a rate that is clean and plausible but not the contracted one, the wire settles nothing exactly, and the pattern in the numbers says why. The blueprint raises it as its own exception and drafts the query note for the broker: what was expected, what arrived, the difference. Drafted, never sent: sending is a person's act.

The day's flow. Matching is arithmetic, checked in code; the model reads and proposes; a person approves the posting.

Reading is the model's job. Matching is not.

An exact reference with an exact amount settles first in the fixed order of attempts. Then the netting-adjusted figure, premium minus the commission the contract allows. A reference whose amount differs is never force-fitted to the bill: the cash applies to what it genuinely covers, and the difference becomes a partial or overpaid exception with an aging clock. Duplicates are caught before any of it runs.

The model's work sits inside that order, in three places. It reads remittance advices, the emails and documents that explain a lump sum, and turns them into per-policy allocations, each required to cite the words in the document it came from. It proposes an allocation for a lump sum nobody explained, restricted to the paying broker's own open items, summing to the wire. And it drafts those broker query notes. Every reading is validated before it is worth anything: advice lines that do not sum to the wire are refused, allocations to cancelled policies are flagged rather than posted, and an unreadable document is held whole while the day carries on.

Installments get worked one by one, oldest first; a reference naming a specific installment settles that one first, and the record shows the choice. A free-text hint pointing elsewhere is information, not instruction: the cash applies oldest-first, and the difference is flagged.

Suspense is a place, not a bin

Cash that survives every attempt unexplained goes to suspense, carrying a reason code and the date it arrived. Auditors ask about this pile first, so it is treated as work in progress, never as a bin. Each item sits in an aging view, oldest first, owned by a named member of the team. Information often lands after the cash does, and a re-match pass works the open items against the books as they now stand; recoveries still arrive as proposals, in a pack, at the gate.

The hardest items have no payer to work from at all. For those, a resolution scan reads the whole book and builds candidates under a strict discipline: a candidate exists only when the arithmetic is exact, or when the wire's text names the policy, quoted verbatim. A model may rank them, and may add one only by quoting that text. Policies the evidence names but the cash cannot settle come back flagged, considered and unchoosable. The item stays open while all this stands proposed, because a hypothesis is not a claim on the cash. A controller picks one candidate, or none.

Two partial wires from the same payer can also settle one bill, when they price out exactly, each at the rate of its own day, so a pair can straddle a rate change. A pair that lands a cent short proves nothing, and neither does an exact sum across two different payers.

Some cash is never identified. After a governed age it can be proposed for a governed end: written off to income, or moved to an unclaimed-property review list, because it may be owed to someone the desk could not find. Either way it takes two approvers, and the assembler can be neither. Review-list money stays visible, with its full chain beside it.

The footing rule. A day is reconciled only when the three figures add up to what the bank says arrived.

A day counts only when it foots

The day's report is the control everything answers to: the bank total, split into what was applied, what sits in suspense, and what has not been worked yet. If they do not add up, to the cent, the report says so in words and refuses to call the day reconciled. On a book with more than one currency, the identity holds per currency first. Cash is never converted to hunt for a match, so a wire that arrives in the wrong currency waits in suspense in its own denomination, with its reason named. Conversions recorded for reporting use the rate in force on the day the cash arrived, and keep all three figures: the cash, the rate, the converted amount.

Money flows back out too. A policy cancelled mid-term earns a return premium, computed pro-rata in code and owed back to the insured, or to the broker who collected it. Cash that arrives later for that policy meets the amount owed before it meets anything else. Payments can also be taken back by the bank; a return un-applies the cash by replaying the original arithmetic exactly, and a foreign-currency return reverses at the rate recorded when the cash first arrived, even if the rate has since moved. Approved packs are never rewritten; they gain a flag, and the return stands as its own entry.

Month end gets the same control at a larger scale: the close pack rolls suspense from opening to closing, currency by currency, and foots or says that it has not. Each broker's own monthly statement is checked against the books line by line, and every disagreement is classified: agreed, commission differs, amount differs, missing on the books, missing on the statement. The model's proposals are graded too, by what the gate actually did with them, and each broker's paperwork is scored on the record.

What the checks prove, and what they don't

The blueprint is tested against planted cases built into its months: the pair of wires one cent short, the wire in the wrong currency quoting a bill's exact amount in the other one, the broker who applied next quarter's commission rate a month early. Passing those cases means the code still behaves as designed on the traps the builders thought of. That is a regression gate, not evidence about your book or your brokers.

The habit that gives the gate its value is unglamorous: when the desk meets a new failure shape, the case goes in first, and the behaviour changes second.

Honest limits

The book in the repository is Northgate's, chosen to exercise the shapes a real desk meets. Rates, thresholds and waiting periods are the blueprint's starting set; a deployment replaces them with the carrier's own contracts and policies first.

The built-in fallback reader handles only advices written in a tidy, predictable shape, and real remittance paper is rarely tidy. Reading free-form documents well is the connected model's work, and live verification against a real model is still pending; the blueprint would rather say that than grade its own homework. The boundary at the far end is deliberate too: the system never moves money and never writes to a general ledger. Its output is an approved posting file, an audit record, and reports that foot. A person still uploads the file.

Work with us

If your cash team starts its mornings with a pile of wires that do not match the bills, and a suspense balance nobody wants to explain to an auditor, start with Trida AI's intake: a few questions about your book and your brokers, then a conversation with the founders and engineers who would build it with you.

Get the Code on GitHub: https://github.com/tridaai/blueprint-insurance-cash-reconciliation

Text-to-SQL Against a Real Schema: Where the Demo Database Stops Telling the Truth

FDE at Trida AI — Wed, 21 Oct 2026 00:54:25 +0000

Originally published at Trida AI.

Copy as markdown

Twelve questions, three production models, each asked three times. On the clean database the scores ran from 5 to 11 of 12. On the same data in legacy dress they ran from 6 down to 1. The failures were the same failures, model after model.

We built the test twice. The small dataset is a sales book: seven customers, six products, eleven orders. The larger one is a wholesale business: 47 active customers, 20 products, 161 live orders and four deleted ones besides, three companies filed under two customer records each, two credit holds. Each dataset is stored two ways. The first copy is the demo version: names that say what they mean, dates in one format, statuses stored as words. The second holds exactly the same business data under the rules real databases accumulate: abbreviated names, undocumented integer status codes, dates in three or four formats, deleted orders still in the table, product attributes in a key-value table, credit holds written only in a notes field.

Then we asked Anthropic's Claude Sonnet 4.5, Opus 4.5 and Haiku 4.5 the same twelve questions against each copy, three runs apiece: 432 answers in all. Same prompt, the schema definition as the only context, no retries. Gold answers were written first, and the gold queries return identical results on both copies of each dataset, so any gap is the schema's doing, not the data's. A question counted as correct only when the executed result matched the gold result exactly.

The test rig: one dataset, two schemas, the same twelve questions, scored against gold answers.

The clean schema failed in exactly one way

Every easy question passed on the clean schema: counts, totals, a category lookup. The six misses, three medium and three hard, share a single cause. The model had to guess the literal values stored in the status columns, and it guessed the way a person writes: 'Completed' with a capital C, 'partially paid' where the stored value is just 'partial'. The database holds 'completed', lowercase.

Those guesses returned nothing. Not wrong numbers: empty result sets and null totals, because a filter that matches no rows produces silence, and a sum over silence is null. The joins were right. The aggregations were right. Revenue by category came back with the correct shape and zero rows inside it.

There was no consistency to lean on, either. The same model wrote 'completed' correctly in one question and 'Completed' in another. A schema definition tells a model which columns exist. It says nothing about the values inside them, and in this run the values were the whole game.

The legacy schema failed in four different ways

Four misses trace to a deleted order nobody filtered out. One order in the legacy table carries a deleted flag, and it is worth 1,400. The model ignored the flag four times: a total inflated by exactly 1,400, a count of 10 completed orders instead of 9, quarterly revenue of 5,160 instead of 3,760, and a revenue-by-category table where Electronics alone absorbed the phantom 1,400. Each answer was precisely, confidently wrong.

One miss was an attribute that was not a column. Asked which products are made in Germany, the model found the attribute table and joined it, then filtered on the attribute name 'country'. The stored name is 'origin_country'. Zero rows.

One miss was a fact that lives in a sentence. Asked for the completed-order value of customers who are not on credit hold, the model returned 4,050 instead of 3,450. On the legacy schema the credit hold exists only as text in a notes field, three words, 'CREDIT HOLD', attached to one customer. No column carries it, and the model never looked at the notes. A system reading only the schema cannot know which customer is on hold, because the schema never says. That is not a prompting problem.

One miss was not SQL at all. Asked how much invoiced money is still outstanding, the model replied with a paragraph of reasoning and a half-finished code block. No executable query. One answer in twenty-four came back unusable as an answer.

Not everything broke. The model converted epoch timestamps into a 2024 date range without trouble. Faced with two customer rows for the same company on the legacy schema, it summed across both and got the duplicate question right, a question the clean copy failed on a capital letter. The reasoning was rarely the problem. The misses were facts about the data that the schema doesn't state.

Three models, two datasets, the same traps

The worked example above is Sonnet on the small dataset. The full scores (a range where the three runs disagreed): small clean schema — Sonnet 5–6, Opus 10, Haiku 9; small legacy — Sonnet 5, Opus 6, Haiku 3. On the larger dataset the clean scores held up (Sonnet 6, Opus 11, Haiku 8) and the legacy scores fell through the floor: Sonnet 2, Opus 3, Haiku 1.

The traps did not care which model walked into them. Guessing stored values sank all three; the deleted flag was the biggest killer, 84 of the 156 legacy misses across both datasets. The attribute-name guess caught Sonnet and Opus on both datasets. Haiku failed the same questions two other ways: prose on the small dataset, and on the larger one a query that went looking for Germany in the customer table. The credit-hold question beat all three models on both datasets. And 21 of the 432 answers came back as paragraphs rather than a query at all, usually on the hardest questions — all from Sonnet and Haiku; Opus always returned a query.

A bigger model helped on the clean schema. On the legacy schema it barely mattered: the best legacy score in these runs was 6, and nobody beat 3 on the larger dataset. The information the models needed was not in the prompt. It was not in the schema either.

What we would do about each one

Publish the values alongside the columns. A data dictionary carrying the actual stored values of status-type columns, or a few sample rows, belongs in the model's context next to the schema definition. Six clean-schema misses say the values are the context that matters.

Make deletion a default rather than a detail. Serve the model a view with deleted rows already excluded, or pin the rule to the schema where no query can miss it. A flag every query must remember is a flag some query will forget — we watched it happen four times in twelve questions.

Treat attribute tables as first-class. The attribute names belong in the dictionary too. Given 'country' or 'origin_country' to guess, the model will guess the obvious one and get silence.

Promote notes to data. If a fact decides money, a credit hold, a do-not-ship, it needs a column. Text fields are where facts go to be missed.

Execute before you trust. Run every generated query and inspect what comes back. An empty set or a null total, on a question that should have an answer, is a detectable smell, and it would have caught the silent clean-schema failures. The inflated totals are harder: they sail through any check that only looks for emptiness, which is why the dictionary and the defaults come first.

Then we applied the fixes

So we applied the fixes a schema and a prompt can carry, and ran everything again: dictionary in the prompt, deleted rows excluded by default, credit holds as a column. Another 432 answers. Small legacy: Sonnet 5 → 8, Opus 6 → 10, Haiku 3 → 10. Larger legacy: Sonnet 2 → 9, Opus 3 → 10, Haiku 1 → 10. Clean schemas: every model reached 11 or 12. Across all 432 fixed answers, not one miss fell into the four failure modes above.

What remained was different in kind. Duplicated customers still got double-counted on a signup question — nothing in these fixes merges two records for one company. One model added the gross and net amount columns together. A few answers still came back as prose, or with the right rows carrying an extra column. Smaller problems, and honest ones: identity and output discipline, not facts the schema hides.

The limits of this test

Two synthetic datasets, both ours, the larger one still small next to a real warehouse. Three models from one provider, three runs each, one prompt per pass, no tools. Exact-match scoring is strict: a right answer carrying an extra column counted as a miss, and prose counted as a miss even when a query was buried in it. The legacy transformations were ours, documented and deliberate, and probably kinder than plenty of real schemas, where codes go undocumented for a decade and three systems disagree about what a customer is. What transfers is the shape of the failures: they clustered where the schema stops describing the data, and production schemas stop much earlier than demo ones.

Work with us

If your team is putting natural-language questions over a database that has been accumulating decisions for a decade, the demo is the easy part. The real work is the dictionary, the defaults and the checks around the model, and it starts before the first question is asked. Tell us about the database through Trida AI's intake, and we will talk through what it takes to make its answers worth trusting.

The First Two Weeks Decide

FDE at Trida AI — Wed, 21 Oct 2026 00:52:42 +0000

Originally published at Trida AI.

Copy as markdown

TL;DR: The embed phase is the opening stretch of a deployment, when a forward deployed engineer (FDE) works inside the customer's operation before anything is built. Its job is to replace assumptions with facts: what the problem actually is, what the data actually holds, who can actually say yes. Five heuristics below. Skip them and the build still happens. It just builds the wrong thing, on schedule.

The brief is almost never the problem.

A brief describes the problem as it looks from a meeting room: the queue is too long, the review step is too slow, the team wants a copilot. Sit with the people doing the work and a different picture shows up within days. The queue is long because a third of the items arrive incomplete. The slow step is slow because two teams disagree about what a finished item looks like. Nobody wrote that down, because to the people living it, it isn't a problem statement. It's Tuesday.

That gap, between the brief and Tuesday, is what the first two weeks are for. Everything after is execution. These are the heuristics we hold the embed phase to, stated as principles, because they're the part of a deployment that decides whether the rest deserves to happen.

Five heuristics for the embed phase

Work on the problem in the room, not the proxy in the brief. The forward deployed engineer role was built on exactly this distinction: embed with the customer and solve their problem, not a stand-in for it. A proxy problem is recognisable. It names a tool (“we need a chatbot”) instead of an outcome, and it survives contact with nobody. So the first heuristic is rude but necessary: keep asking who hurts, at what step, measured how, until the answer names a workflow and a person rather than a technology.
Ask for access on day one. Data, systems, the users who’ll live with the result, and the security reviewer who’ll eventually judge it. Access requested in week six arrives as a delay; access requested on day one arrives as a plan. If an account, a data extract or a review meeting can’t be scheduled early, that isn’t admin noise. It’s the first honest signal about how the deployment will actually run, and it belongs in the open while there’s still time to act on it.
Read the data before designing around it. Sample records lie by omission in a specific way: they’re the clean ones. The real corpus has the duplicate accounts, the scanned attachment nobody can search, the field that changed meaning two years ago and kept its name. An architecture sketched before anyone has looked at a few hundred unfiltered records is fiction with diagrams. Look first. Some designs die in that reading, and they deserve to.

What the embed phase has to surface — the real problem, the real data, real access and a real owner, before the first workflow is chosen.

Pick a first workflow that’s small enough to ship and real enough to matter. Both halves count. A first workflow with no real users proves nothing except that a demo works. One that tries to absorb an entire department proves nothing either, slower. The test we use: named users, a volume you can count, and a before-and-after someone can check. If it succeeds, it earns the right to grow. If it fails, it fails small, in weeks, with the lesson still cheap.
Name who can say no. Every deployment has more vetoes than the kickoff admits: a security review, a data owner, a team lead whose workflow changes, the users themselves, who veto by ignoring. The embed phase should surface all of them and agree, in plain language, what “in production” means for this workflow: who’s accountable for it, what good output looks like, and what happens when the system is unsure. A deployment, in our vocabulary, means a system running in production (real users, real data, an owner for failures), not a pilot that impressed a room. That definition gets agreed here, or argued about later at a much higher price.

Where this breaks

Honesty about the limits, because there are two real ones.

Discovery can become theatre. Two weeks of interviews and sticky notes that end in a report nobody acts on is worse than no discovery at all, because it spends the customer’s patience and calls it diligence. The embed phase only earns its time if it ends in decisions: a chosen workflow, an access plan, an owner, a definition of done. If it can’t produce those, say so and stop, rather than drifting into a build to justify the calendar.

And heuristics can’t fix an organisation that won’t open the door. If users won’t give an hour, if data access is refused, if no one will own the outcome, the right move is to pause the engagement, not to build a polished demo against sample data and hope. A demo built in the dark doesn’t de-risk anything. It just defers the argument to go-live, where the same people say no with more at stake.

There’s a quieter failure too. Embed with one loud user and you’ll build for them. The person who talks most in discovery is rarely the median case, so the workflow you choose has to be checked against the people who’ll use it quietly, at volume, when nobody from the pod is watching.

What it decides

Four statements you should be able to make when the embed phase ends — the problem, the data, the first workflow and the owner.

By the end of the embed phase you should be able to say four things without hedging: the problem, in the words of the people who have it; the state of the data, from having read it; the first workflow, with its users and its measure; and the owner, by name. If you can, the build that follows has a chance, because it’s aimed at something real. If you can’t, building anyway isn’t momentum. It’s a guess with a budget.

Work with us

If a deployment you’re planning still describes its first workflow in slide language, the honest next step is two weeks of finding out what’s actually true, before anyone writes code. Bring us that starting point through Trida AI’s intake and we’ll help you turn the brief into a problem worth building against.

Data Residency in AI Deployment

FDE at Trida AI — Tue, 20 Oct 2026 16:00:47 +0000

Originally published at Trida AI.

A deployment can keep every record in the right country and still fail on residency. That sounds like a contradiction until you follow the data instead of the database.

Source data is only the first stop. Once a workflow starts, copies multiply quietly: passages are embedded and stored in a vector index, prompts travel to wherever the model actually runs, traces capture the text someone will later need to debug, and backups replicate all of it somewhere else. A security review that only asks where the originals live has checked one stop on a five-stop route.

Residency is a route, not an address

Start with inference, because it is the stop people forget. A model call is a data transfer. If your records sit in a European region but the model endpoint routes requests elsewhere for capacity or price, residency broke in transit while every dashboard still showed the database at home. This is why routing modes matter as much as storage regions. AWS built its AWS European Sovereign Cloud around exactly this point: content stays in the region selected, and Bedrock inference there runs in-region rather than through global cross-region routing. You may never need a sovereign cloud. You do need to know which mode your deployment is in, and to be able to say so in one sentence.

Retrieval is the second quiet copy. Embeddings feel abstract, so teams treat the vector store as plumbing. It is a derived copy of your documents, often including the text chunks themselves, and it inherits the same residency and access rules as the source. Same for the eval set, which is built from real examples because synthetic ones do not measure much. Real examples mean regulated data in a test file, usually with weaker controls than the system it tests.

Then there are the records the system keeps about itself. Logs and traces are where prompts go to be readable. They are also the first thing an incident review asks for, so you cannot simply switch them off. The workable answer is deliberate: traces live inside the same boundary as the data, access to them is named rather than general, and retention is a number someone chose, not a default nobody remembers setting.

Five stops on the route. Residency has to hold at every stop, not only where the originals sit.

The vendor questions belong on this route too. What a platform does with inputs and outputs is a documented position, not a matter of trust. Amazon Bedrock, for example, states that customer inputs and outputs are not shared with model providers and are not used to improve the base models, and offers private connectivity from a customer's VPC. Whatever stack you choose, get the equivalent answers in writing for training use, retention, and who operates the service. If a provider cannot answer plainly, that is an answer.

Start the review before the architecture hardens

Most security reviews stall deployments for a boring reason: they arrive at the end, when every answer has already been built. Is the agent allowed to read that share? Which keys encrypt the backups, and who holds them? How long do traces survive? Asked in week one, these are design choices, cheap to make well. Asked after the build, the same questions become rebuilds, and the review gets blamed for a delay the schedule created.

So ask for the questionnaire first. A good review also improves the system in ways nobody files under security. Deciding which data the workflow may touch forces the scope conversation teams keep postponing. Deciding who can read the logs forces an honest answer about how much sensitive text the system will handle. Security teams, in our experience of reading their questionnaires, are often the first people to ask what the system is actually for. That question deserves an early audience.

Four controls answer the bulk of most reviews, and none of them is exotic. The agent gets its own identity, scoped to the systems and actions its task needs, never a borrowed human login. Retrieval checks the requesting user's permissions before documents reach the model, so the system cannot summarise what the person asking was never allowed to read. Inference runs in-region or over private routes, with any cross-region behaviour off or explicitly disclosed. And the records stay inside the boundary, under a retention limit with a named owner.

Four controls that answer most reviews. None is exotic; the work is deciding them before the build, not after.

Governance is who can change the answer

Residency settings and guardrails (the in-line checks that constrain what a system may read, send and do) describe a deployment on the day it ships. Governance decides whether it still looks like that in six months. The NIST AI Risk Management Framework puts this well in structure if not in so many words: its Govern function wraps Map, Measure and Manage, because the standing decisions about accountability are what keep the other three honest over time.

In practice, governance for a deployment is a short list of named permissions. Who can add a new data source to the system's reach. Who can widen what the agent is allowed to do without asking. Who approves a model swap, knowing a new model can mean a new region, a new retention position, a new subprocessor list. Who can grant themselves access to the traces. If those four questions have names attached, the review's findings stay fixed after they are fixed. If they have a committee attached, they quietly unfix themselves.

Honesty about limits, then. Residency is not security: data can sit in exactly the right place and still be exposed by an over-permissioned agent or a prompt injection that talks the system into fetching what it should not. Keeping everything in-region or on premises narrows your model choices and can add cost and latency, and for some workloads that trade is wrong. And some data should not enter an AI workflow at all, in any region, under any control set. A review that ends in "not this workflow" has done its job, not failed it.

The decision to take from this is sequencing, not caution. Draw the route, start the review while the answers are still cheap, and name who can change them. Deployments that do this tend to find the security review is the shortest part of the project. Deployments that treat it as a final exam keep retaking it.

Work with us

If your AI project is waiting on a security review it hasn't started yet, begin earlier this time: tell us where your data has to live and what it may never touch, through Trida AI's intake, and we will design the deployment to that boundary from the first sketch.