DEV Community: RLM Labs

Non-deterministic is a design choice

RLM Labs — Wed, 21 Oct 2026 09:35:55 +0000

The most successful AI framework company defines its own field as a problem without an end. You should not accept the definition.

LangChain, announcing its Series B in October 2025, described the work the company sells tooling for:

"the iterative process of refining non-deterministic LLM systems into reliable experiences"

Read the sentence as an engineer. The system is non-deterministic, and the job is to keep refining the system toward reliability. No finish line appears in the sentence. The product is the treadmill.

I do not dispute the first half. A language model is non-deterministic. Even with a fixed seed and an identical request, GPU arithmetic does not promise identical output twice. Anyone telling you otherwise is selling you something.

I dispute the conclusion everybody draws from the first half: since the model is non-deterministic, the whole system must be. The conclusion does not follow. You choose where determinism lives.

Two of the largest companies in the industry already made this choice, for actions. AWS writes about its agent policy layer:

"consistent, deterministic enforcement that remains reliable regardless of how the agent is implemented."

Apple, in a WWDC26 session on securing agentic features in apps, told developers where to start:

"we should try to focus on deterministic mitigations as a baseline because their security guarantees are easier to audit and reason about."

Easier to audit and reason about. Keep the phrase. A probabilistic check gives you a probability. A deterministic check gives you a rule you read, test and argue with.

Both companies apply the principle to what a model does. Neither applies the principle to what a model claims. The answer at the end of a run still gets judged, if anyone judges the answer at all, by another probabilistic system.

Merlin moves the line. The model stays non-deterministic. The model plans, reads, writes and proposes an answer, and two runs of the same request take different paths. After the proposal, deterministic rules get the last word. A model still gives an opinion on the answer, and rules run before and after the opinion and override the opinion. A rule counts open tasks. A rule checks whether a cited line exists in a file the run read. A rule compares the bytes Merlin verified with the bytes Merlin committed. Each rule gives the same verdict every time on the same record, and the rules carry tests pinning those verdicts.

The result is a system whose path varies and whose standard does not. You will not get the same run twice. You will get the same refusal twice, for the same reason, on the same evidence.

The objection I hear most: deterministic rules are rigid, and rigid rules block good work. Sometimes they do, and we publish those cases in this series too. A rigid rule blocking good work shows up in a log with a name and a line number, and you fix the rule. A probabilistic judge waving through bad work shows up nowhere. I will take the failure I see over the failure I never find.

Non-deterministic models are a fact. Non-deterministic systems are a decision. You get to make the decision differently.

The limiting factor isn't capability

RLM Labs — Tue, 20 Oct 2026 05:10:39 +0000

Models write good code now. Ask anyone shipping with them. The problem moved, and most of the industry is still arguing about the old one.

Reuben Brooks put the shift in two sentences in May 2026:

"Existing models can already write almost all of your code."
"The limiting factor is whether you can know that they did what you wanted."

OpenAI reached the same place from inside their own team. In February 2026 they described five months of building a product with zero hand-written code. Here is what slowed them down at the start:

"Early progress was slower than we expected, not because Codex was incapable, but because the environment was underspecified."

And in October, Google Cloud AI Research opened the Harness² paper with a line you should read twice:

"Even a strong model can perform substantially differently under different harnesses"

Three groups, three different starting points, one conclusion. The model is not the bottleneck. The system around the model is.

I agree, and I bet the project on the idea before the industry had a word for the idea. On 31 December 2025, three MIT researchers published a paper on recursive language models: keep a long input outside the model's context and let the model work through the input in pieces, from a programming environment. Eighteen days later, on 18 January 2026, I made the first commit to Merlin. The design document lists what the system does not do. The first non-goal reads:

"Modify or retrain any LLM"

No fine-tuning. No custom model. Every improvement had to come from the structure around a model somebody else trained. The industry started calling this harness engineering in February 2026. We were already six weeks into building one.

Here is where I part ways with most of the harness work I read. The OpenAI team answered the problem with more autonomy: agents reviewing agents, humans optional in review, and as few blocking merge gates as possible. In their own words:

"Humans may review pull requests, but aren't required to."

For their goal, shipping speed on an internal product, the trade makes sense.

For mine, the trade runs backwards. If the hard part is knowing whether the work was done, then every layer you add without a check makes the hard part harder. A second model reviewing the first model adds an opinion. A rule reading the record adds a fact.

So the question I care about is narrower than how capable the model is. I want the system to show you, with evidence you check yourself, what the model did and did not do. Capability improves every quarter without any help from me. Evidence improves only when somebody builds for evidence.

Every post after this one comes back to the same split. The model proposes. Something outside the model decides what the system has earned the right to claim.

Every guardrail points at the wrong thing

RLM Labs — Mon, 19 Oct 2026 08:21:58 +0000

The big platforms ship real controls for AI. Read where each control sits. Every one of them guards what the AI does. None of them guards what the AI tells you.

AWS describes the policy layer of its AgentCore platform this way:

"Every agent action through Amazon Bedrock AgentCore Gateway is intercepted and evaluated at the boundary outside of agent's code - ensuring consistent, deterministic enforcement that remains reliable regardless of how the agent is implemented."

Deterministic. Outside the agent's code. Good engineering. Now look at the object: every agent action.

Google introduced its Agent Gateway in April 2026:

"Agent Gateway provides air traffic control to ensure interactions between agents and data is secure."

Anthropic's Agent SDK lists its permission system like this:

"Control which tools run automatically, which need approval"

AWS's open-source Strands framework now groups its controls under a heading:

"Gate tool calls with interventions"

Four vendors. Four checkpoints. All four sit on the action path: the tool call, the data access, the side effect. The newest AWS feature, temporal policies, goes further and reasons over a whole session, for example blocking a money transfer until an approval exists. Still an action.

Here is what none of those controls inspect: the sentence at the end. The summary telling you the migration finished. The report saying the audit found no issues. The final answer saying all 25 tasks are done. You read the answer and you act on the answer. On these platforms, nothing deterministic stands between the model and the claim.

You might say the grounding vendors cover this. Some do, partly. Contextual AI ships a model built to refuse when its documents do not support an answer. Cleanlab and Galileo score responses for errors. Every one of those sits beside an orchestrator built by someone else, and most of them grade with another model. I cover the trouble with a model grading a model in a later post.

We put the gate on the claim. Here is a real example from our own logs, 1 August 2026. A Merlin run finished 24 of 25 tasks and wrote 11 files. The model proposed a final answer saying the job was done. A six-line rule rejected the answer:

route              REJECT_UNSUPPORTED_CLAIMS
source             deterministic_guard_guard_3c
reason             full-success final is forbidden while task_summary
                   has pending or in-progress tasks

The rule did not read the prose. The rule counted open tasks, found one, and refused to let a claim of full success exist. The 11 files still shipped, with a manifest of what changed. Only the false sentence died.

Action gates matter. Keep them. A model with production credentials needs a fence around the tools. But the fence answers a different question. The fence answers "was the model allowed to do this." You also need an answer to "is the model telling the truth about what got done." Right now, on the platforms quoted above, you build the second answer yourself.

The test I use: pick any guardrail in your stack and ask what object the guardrail inspects. If the object is a tool call, the guardrail protects your systems. If the object is the final claim, the guardrail protects you.

Nobody will vouch for the answer

RLM Labs — Sun, 18 Oct 2026 20:34:26 +0000

Microsoft says so in the overview page for its Agent Framework, the product meant to help you build on its models:

"You are responsible for carefully reviewing and testing applications you build using Microsoft Agent Framework in the context of your specific use cases, and making all appropriate decisions and customizations. This includes implementing your own responsible AI mitigations such as metaprompt, content filters, or other safety systems, and ensuring your applications meet appropriate quality, reliability, security, and trustworthiness standards."

The Allen Institute says so on the model card for OLMo, one of the most open models anyone has shipped:

"many statements from OLMo or any LLM are often inaccurate, so facts should be verified."

LangChain, the most widely used framework for building on these models, defined its whole category in its Series B announcement as:

"the iterative process of refining non-deterministic LLM systems into reliable experiences"

Read the word iterative. Reliability is something you work toward, release after release. Nobody promises you arrive.

Then, on 9 October 2026, Google Cloud AI Research published Harness², a framework for improving the system around a model without touching the model. Good work. The paper reports gains on three professional benchmarks. In the limitations section, the authors write this about the signals their method uses to decide whether a change helped:

"They do not guarantee correctness or establish that an adopted change will transfer to a different task."

I respect the honesty. I also want you to notice the pattern. The model vendor, the framework vendor, the platform vendor and now the research lab building self-improving systems all draw the same line. Each one improves the odds. None of them stands behind the answer.

The check falls to you. Every page quoted above hands the check to the person reading the output.

I spent thirty years as the person flown in when a distributed system misbehaved in production. You learn one thing fast in the job: the system from the demo is not the system in front of you, and the only thing you trust is what you checked yourself. The log line. The row count. The diff.

AI moved the same problem up a level. The output reads well. The output reads well whether or not the output is right. And the people selling you the output have written the rule into their own documentation: checking is your job.

We built Merlin on the opposite assumption. If nobody vouches for the answer, the system producing the answer has to carry the evidence for the answer, and has to refuse to claim anything the evidence does not support. Not as a feature you switch on. As the rule the whole system runs under.

The rest of this series shows what goes wrong, including in our own system, and what we built instead. Start with the question these quotes leave you holding. Stop asking whether the model is good. Ask who checks the answer.