DEV Community: Shrijith Venkatramana

Lean for Developers: Turn Program Requirements into Theorems

Shrijith Venkatramana — Fri, 16 Oct 2026 19:59:46 +0000

Hello, I'm Shrijith Venkatramana, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.

A test suite can pass while a program still violates an invariant nobody thought to test.

A counter exceeds its limit. A retry handler executes a payment twice. A pagination function silently skips records. A resource allocator assigns more capacity than the system has available.

These bugs often survive because tests examine particular executions while requirements describe what must hold across many possible executions.

What if you could express a requirement as a mathematical proposition, implement the behavior, and ask a computer to verify that your implementation satisfies that proposition?

That is the idea behind Lean, a programming language and interactive theorem prover. Lean lets developers define functions, express their properties mathematically, and construct machine-checked proofs about those properties.

You do not need to formalize an entire application to use it. Start with one function, one invariant, and one proof.

Tests check examples; specifications define obligations

Consider a function that increments a counter without allowing it to exceed a configured limit.

An ordinary test suite might cover these cases:

The counter increases when it is below the limit.
The counter stays at the limit when it reaches the limit.
The counter does not exceed the limit when given an unexpectedly large input.

These are useful tests. But suppose the function accepts integers in a large domain. How many inputs should you test? Ten? A thousand? A million?

For a function over unbounded mathematical integers, exhaustive enumeration is impossible.

Property-based testing improves coverage by generating many inputs and checking general properties. Formal verification takes a different approach: it attempts to establish a property for every input in the mathematical domain described by the specification.

The distinction is between testing selected executions and proving a proposition about all executions represented by a model.

This approach has a long history. In 1969, computer scientist C. A. R. Hoare published An Axiomatic Basis for Computer Programming, helping establish a systematic foundation for reasoning about program correctness.

One of the central ideas associated with this work is the Hoare triple:

{P} C {Q}

Here, P is a precondition, C is a command, and Q is a postcondition.

Read it as: if P holds before executing C, then a terminating execution of C establishes Q.

For our counter, we might write:

{current < limit} increment(current) {result = current + 1}

This describes the increment behavior when the counter is below the limit. It does not specify the other cases, such as what happens when the counter has already reached its limit.

A complete specification must cover those cases as well.

Lean provides a language in which we can represent these requirements precisely and have a proof checker verify the resulting arguments.

Translate the requirement into a mathematical contract

Let's begin with three requirements:

The result must never exceed the limit.
When the current counter is below the limit, increment it by one.
When the current counter is at or above the limit, return the limit.

We can express the implementation in Lean:

def boundedInc (limit current : Nat) : Nat :=
  if current < limit then current + 1 else limit

A few details matter here.

Nat is Lean's type of natural numbers: 0, 1, 2, and so on. The function receives two natural numbers and returns another natural number.

The expression if current < limit then ... else ... selects one of two branches, depending on the comparison.

Now let's specify the required behavior independently of the implementation.

def BoundedIncSpec (limit current result : Nat) : Prop :=
  result <= limit ∧
  (current < limit -> result = current + 1) ∧
  (limit <= current -> result = limit)

This definition introduces three obligations.

The expression result <= limit says that the result must stay within the limit.

The symbol ∧ means logical AND: both propositions must hold.

The expression P -> Q means that if proposition P holds, proposition Q must follow.

Finally, Prop is Lean's type of propositions. The definition BoundedIncSpec describes a proposition about the relationship between the inputs and the result.

Notice that result is an independent parameter. We have described what constitutes a correct result without assuming how the result is calculated.

This separation matters. The same specification could be satisfied by different implementations: a conditional expression, a library function, or a more complicated implementation optimized for a particular environment.

A specification describes the obligation. An implementation supplies the mechanism.

One subtlety deserves attention: our specification is deliberately tailored to natural numbers and the stated requirements. It does not say that the function must preserve information about how far an input exceeded the limit. That is intentional. Formal specification forces us to distinguish essential requirements from incidental implementation details.

Prove that the implementation satisfies the contract

We now have a function and a specification. Can Lean establish that the function satisfies the specification for every possible pair of natural numbers?

Here is a complete example for a Lean project using Mathlib, Lean's mathematical library.

import Mathlib

def boundedInc (limit current : Nat) : Nat :=
  if current < limit then current + 1 else limit

def BoundedIncSpec (limit current result : Nat) : Prop :=
  result <= limit ∧
  (current < limit -> result = current + 1) ∧
  (limit <= current -> result = limit)

theorem boundedInc_le_limit (limit current : Nat) :
    boundedInc limit current <= limit := by
  unfold boundedInc
  split_ifs <;> omega

theorem boundedInc_eq_succ (limit current : Nat)
    (h : current < limit) :
    boundedInc limit current = current + 1 := by
  simp [boundedInc, h]

theorem boundedInc_eq_limit (limit current : Nat)
    (h : limit <= current) :
    boundedInc limit current = limit := by
  have h' : ¬ current < limit := by omega
  simp [boundedInc, h']

theorem boundedInc_correct (limit current : Nat) :
    BoundedIncSpec limit current (boundedInc limit current) := by
  unfold BoundedIncSpec
  constructor
  · exact boundedInc_le_limit limit current
  constructor
  · intro h
    exact boundedInc_eq_succ limit current h
  · intro h
    exact boundedInc_eq_limit limit current h

The main result is the final theorem:

theorem boundedInc_correct (limit current : Nat) :
    BoundedIncSpec limit current (boundedInc limit current)

It says that for every natural-number limit and every natural-number counter value, the result of boundedInc satisfies the specification.

There is no finite test set here. The theorem covers the entire declared input domain.

How does the proof work?

First, boundedInc_le_limit establishes that the result never exceeds the limit. The unfold tactic exposes the function definition. split_ifs separates the two branches of the conditional, and omega solves the resulting natural-number arithmetic obligations.

Next, boundedInc_eq_succ proves the increment behavior under its precondition. The simp tactic simplifies the conditional using the supplied hypothesis.

Then, boundedInc_eq_limit proves the other branch. Given that the limit is less than or equal to the current value, omega establishes that the condition current < limit cannot hold.

Finally, boundedInc_correct combines these three results. The constructor tactic decomposes the conjunction into its component obligations. Each obligation is discharged by one of the preceding theorems.

The keyword by introduces a proof constructed using tactics. Lean checks that the resulting proof establishes the stated proposition.

The tactics are convenient ways of constructing proofs; they are not substitutes for proof checking. Lean's trusted kernel checks the resulting proof terms.

The distinction is fundamental: the machine is not merely accepting a developer's assertion that the function is correct. It checks a formal argument that the implementation satisfies the proposition.

Understand exactly what the proof establishes

It is tempting to interpret a verified function as a guarantee that the corresponding production feature cannot fail. That conclusion would be too broad.

Our theorem proves a property of a mathematical function over natural numbers. It does not prove that a production implementation using a 32-bit integer behaves identically.

A 32-bit unsigned integer has a finite representable range. Depending on the language and implementation, arithmetic near the boundary may wrap, trap, or follow other specified behavior. Lean's Nat does not impose that machine-level limit.

To verify the production version, we would need a model that represents the relevant integer bounds and behavior, then establish the corresponding properties of that model and the implementation.

The same distinction applies to external systems. A proof about an authorization function does not automatically prove that a database transaction commits atomically, that a remote identity provider returns trustworthy data, or that a network retry cannot duplicate an external side effect.

Those are separate obligations.

A useful verification project therefore makes its assumptions and boundaries explicit.

What does the function assume about its inputs? Which properties are established by the proof? Which properties depend on the runtime, compiler, database, or external services?

These questions are not peripheral details. They determine what assurance the proof actually provides.

Lean's architecture helps with this distinction. The language combines programming and theorem proving within a type-theoretic framework, and its kernel checks proof terms produced with the help of tactics.

That design allows developers to use automation without treating the automation itself as the ultimate authority on correctness.

There is another limitation worth understanding: a formally verified implementation can satisfy an incorrect specification perfectly.

Suppose the business requirement says a customer must never withdraw more than their available balance. A developer mistakenly specifies that a withdrawal must never exceed the original balance at account creation. Lean might help prove the latter property, but it would not establish that the business requirement had been captured correctly.

Formal verification reduces ambiguity in the specification and provides evidence about its implementation. Human judgment remains necessary to determine whether the specification describes the system we actually want.

Move from function contracts to state invariants

Many important software properties concern not just the output of a function, but how a system's state changes over time.

Consider inventory management.

An application tracks how many units of a product are available and how many have been reserved. When a customer places an order, the system must transfer units from available inventory to reserved inventory.

Two requirements are immediate:

A reservation must not consume more units than are available.
A successful reservation must conserve the total number of tracked units.

We can represent the state as a structure:

structure Inventory where
  available : Nat
  reserved : Nat

The first field counts available units; the second counts reserved units.

Next, define the reservation operation. If sufficient inventory exists, it returns the updated state. Otherwise, it fails.

def reserve (i : Inventory) (qty : Nat) : Option Inventory :=
  if qty <= i.available then
    some {
      available := i.available - qty
      reserved := i.reserved + qty
    }
  else
    none

Option Inventory represents either a successful result (some updatedInventory) or failure (none).

The operation subtracts the requested quantity from the available inventory and adds it to the reserved inventory. It performs no update when the requested quantity is too large.

Now we can express a useful invariant: whenever the operation succeeds, the sum of available and reserved inventory remains unchanged.

theorem reserve_preserves_total
    (i : Inventory) (qty : Nat) (next : Inventory)
    (h : reserve i qty = some next) :
    next.available + next.reserved =
      i.available + i.reserved := by
  unfold reserve at h
  split_ifs at h with hEnough
  · injection h with hNext
    cases hNext
    simp
    omega
  · simp at h

The theorem does not merely test a reservation of three units or ten units. It covers every natural-number inventory state and every requested quantity. It says that if the operation returns a successful result, the total tracked inventory is conserved.

The key assumption is that the result of reserve equals some next. In the successful branch, Lean can use that equality to identify the returned state with the state constructed by the implementation. The arithmetic proof then follows from the condition that the requested quantity did not exceed the available inventory.

This example demonstrates why it is useful to think in terms of invariants.

A state invariant is a property that should remain true across permitted state transitions. Examples include:

The sum of reserved and available resources remains constant.
A financial balance never becomes negative.
A completed order cannot return to the pending state.
A user cannot gain permissions through an unauthorized transition.
A distributed protocol never reaches a forbidden combination of states, under its formal assumptions.

In production, inventory correctness involves additional concerns such as database transactions, concurrent reservations, inventory adjustments, and retries. The theorem above proves only a property of the modeled operation. It does not establish that a real database-backed application implements that operation atomically.

Even so, the proof provides a precise foundation around which to build the rest of the verification effort.

This is a useful pattern for developers: isolate the state transition, write down what must remain true, and prove that each permitted transition preserves the invariant. For a system with many transitions, the proof structure can follow the design of the system itself.

Integrate specification into the development workflow

A proof is most useful when it is part of the regular engineering process rather than an isolated research exercise.

One workable process looks like this:

:::mermaid
flowchart TD
A["Write the requirement"] --> B["Define the model and contract"]
B --> C["Implement the function"]
C --> D["Prove the contract"]
D --> E{"Does Lean accept the proof?"}
E -->|No| F["Review the code, proof, and specification"]
F --> D
E -->|Yes| G["Review assumptions and system boundaries"]
G --> H["Run tests and verification in CI"]
:::

Start with a pure function: a function whose output depends only on its inputs and that has no external side effects. Pure functions are easier to specify because their behavior can be described without modeling a database, network connection, clock, or mutable environment.

For an existing application, identify a calculation or state transition and separate it from the surrounding I/O where practical. Define its inputs, outputs, and invariants in Lean. Prove the desired properties and keep those definitions and proofs alongside the implementation.

Then add the relevant build and verification commands to continuous integration. If a code change makes a theorem unprovable, the verification build should fail.

This is valuable because it turns an invariant into an explicit engineering constraint. Instead of relying on every future developer to remember why a particular condition matters, the team maintains a checked proposition that must continue to hold.

Formalization can also improve code review. A reviewer can examine the specification independently of the implementation and ask whether it captures the intended behavior. A second review can focus on the proof and its assumptions.

The separation makes two different questions visible:

Are we implementing the right requirement?

Does the implementation satisfy that requirement?

Those questions are related, but they are not interchangeable.

Testing remains part of this workflow. Unit tests exercise concrete examples, integration tests examine interactions with dependencies, and operational monitoring reveals behavior in the deployed environment. Lean contributes another form of evidence: a proof about the modeled properties of the code.

A practical adoption strategy is to begin with one small component, integrate its proof into the build, and expand only when the approach proves useful to the team.

Start with one invariant

The most effective first Lean project is probably not a complete application or an attempt to verify every line of an existing codebase.

Choose a function with a meaningful invariant: a bounded counter, a pagination helper, a resource allocator, a validation routine, or a state transition. State the requirement precisely, encode the implementation, and prove that the implementation satisfies it.

The broader idea has been around since the foundations of program logic. Lean makes this style of reasoning available in a modern programming environment, bringing together executable definitions, mathematical propositions, proof automation, and machine-checked proofs.

The shift is from documenting what code is supposed to do to encoding a requirement that a machine can check against a formal model of the implementation.

The scope of the guarantee depends on the quality of the specification and the boundary of the model. That makes choosing the right invariant an important engineering task in its own right.

Which invariant in your codebase would be most valuable to turn into a machine-checked theorem?

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

⭐ Star it on GitHub:

HexmosTech / LiveReview

Risk-Aware AI Code Review for Business-Critical Systems

LiveReview: Risk-Aware AI Code Review for Business-Critical Systems

LiveReview is built for engineering teams that ship often, but also care about keeping production stable for customers. It approves some 40-50% of low-risk changes automatically, and it gives your team the tools to handle the rest:

Understand risky changes faster.

Fix problems in those changes, with human attention directed to the work that needs it.

Enforce your engineering standards.

Learn from production outcomes, such as incidents, rollbacks, and queue wait times.

Improve the review process over time and make it more efficient.

Connect to the major engineering tools that your team already uses.

Auto-Approve: Ship the Safe 50% Without a Human

Auto-Approve Upto 50% of Your Pull Requests, Safely. Risk scoring decides which MRs/PRs are safe enough to ship without a human, so your team only spends review time on risky and critical changes.
demo.mp4
LiveReview auto-approving a low-risk change while…

View on GitHub

Click below to try LiveReview with your codebase:

How Meta Reduced Code Review Load While Maintaining Production Reliability

Shrijith Venkatramana — Sun, 11 Oct 2026 18:44:20 +0000

AI coding agents have created a peculiar problem for software engineering: writing code is getting cheaper, but establishing that the code is safe to ship still requires scarce human attention.

At Meta, this problem became measurable. Over one year, significant lines of code per human-landed diff increased by 105.9%, while diffs per developer per month increased by 51%. More than 80% of the increase in diff volume was attributed to agentic AI.

Meanwhile, the share of diffs reviewed within 24 hours was declining. Some large engineering groups had thousands of pending reviews.

Meta's response was to automate a specific class of code reviews rather than attempt to automate all of them.

The system, called RADAR (Risk Aware Diff Auto Review), combines machine-learned risk scores, LLM-based code review, deterministic checks, and conservative eligibility policies. Its central idea is simple: let low-risk changes proceed through automated verification, and reserve human attention for changes that need human judgment.

In their 2026 paper, Automating Low-Risk Code Review at Meta: RADAR, Risk Calibration, and Review Efficiency, Chris Adams, Nachiappan Nagappan, Peter Rigby, and their colleagues report results from more than 535,000 RADAR-reviewed diffs.

The reported outcomes are worth understanding in detail: the system reduced review latency while exhibiting substantially lower revert and production-incident rates than non-RADAR changes.

Here is how it works, why the design matters, and what engineering teams can learn from it.

Code generation scaled faster than code review

Code review has always served several purposes: finding defects, maintaining consistency, sharing knowledge, and making engineers accountable for changes.

But these benefits come with an operational cost. Every change entering a review queue competes for attention from someone who must understand its context, inspect its implementation, and decide whether it is safe to merge.

AI agents change the economics of this process.

Consider a developer who previously produced five diffs per day and now produces ten with coding agents. If each diff still requires the same human review effort, review demand doubles even if the developer's working hours remain constant.

At Meta, the pressure was larger than a simple increase in the number of changes. Significant lines of code per human-landed diff had increased by 105.9%, too. Larger changes can demand more review effort, although lines of code alone are an imperfect measure of complexity.

The result was a capacity mismatch:

Code production became cheaper and faster.
Human review capacity remained constrained by attention and calendar time.
Review queues grew, and timely reviews became less common.

This is a queueing problem as much as a tooling problem. If changes arrive faster than reviewers can process them, the queue grows. Adding more code-generation capacity can make the bottleneck worse.

Meta's approach was to change which changes consume human review capacity.

Routine formatting changes, dead-code removal, mechanical refactoring, and similar low-risk work can often be verified through a combination of deterministic checks and automated semantic inspection. Changes involving security, business logic, or substantial architectural decisions deserve more scrutiny.

The objective is to allocate review effort according to risk rather than treat every diff as equally deserving of human attention.

RADAR is a funnel, not an LLM with merge permissions

RADAR does not simply ask an LLM whether a patch looks safe and then merge whatever it approves.

It uses multiple stages, with different rules depending on who or what produced the change.

Meta already had RACER, its GenAI-powered code-editing and refactoring system. Developers can delegate tasks such as dead-code removal, framework migrations, complexity reduction, lint fixes, and test generation to RACER. The system generates diffs in a sandbox, runs verification, and submits changes for review.

RADAR adds an automated path for qualifying changes to progress through review and landing.

The pipeline combines three important kinds of evidence:

Static eligibility and safety checks. These enforce hard constraints involving the source of a change, its scope, review requirements, and restricted areas of the codebase.

Diff Risk Score (DRS). This machine-learning model estimates the risk associated with a change, primarily its likelihood of contributing to a production incident. Its purpose is to identify changes that should be excluded from automated approval.

Automated Code Review (ACR). An LLM-based review agent examines the actual code changes, looking for safe patterns and signals of defects or risk. It can distinguish a large mechanical refactor from a change that modifies important behavior in ways that require human judgment.

All applicable gates must pass.

A failure at any stage sends the change to the appropriate human-review path. Certain deterministic codemods have a separate, more permissive route after the transformation itself has been approved.

The distinction matters. Static rules can enforce known constraints but cannot understand every semantic consequence of a change. An LLM can interpret code behavior but should not be the only safety mechanism. A risk model can prioritize changes but cannot prove that a particular patch is correct.

RADAR combines these mechanisms instead of asking one component to solve the entire problem.

Risk calibration determines how much work gets automated

The most consequential design decision is the risk threshold.

DRS expresses a change's risk as a percentile. A threshold of P5 admits only the lowest-risk 5% of changes under the relevant scoring distribution. P20 admits the lowest-risk 20%, and P50 admits the lowest-risk 50%.

These are eligibility thresholds, not guarantees that a given percentage of all submitted diffs will be approved. Changes must still satisfy the remaining requirements.

A stricter threshold reduces the number of changes eligible for automation. A more permissive threshold increases potential coverage, but also exposes the system to more risk.

Meta's paper reports an operational policy change from P25 to P50. After calibration, the RADAR approval rate reached 60.31%, while the RADAR Verification pass rate was 26.31%.

Why can the approval rate exceed the risk-score percentile?

Because the two numbers measure different things. P50 describes the risk-score eligibility boundary. The 60.31% figure measures the approval rate among the relevant eligible diffs, according to the paper's metric definition. They are not interchangeable measures of overall codebase coverage.

A simple hypothetical illustrates the underlying economics.

Suppose a team receives 10,000 diffs each month, and each human review consumes 20 minutes of reviewer time.

If 30% of those diffs can safely bypass human review, the system eliminates 3,000 manual reviews:

3,000 diffs x 20 minutes = 60,000 minutes

60,000 / 60 = 1,000 reviewer-hours

That is approximately 125 eight-hour workdays of reviewer capacity per month.

This is an illustrative calculation, not Meta's measured labor savings. It assumes that every qualifying diff would otherwise receive a 20-minute review and that automated review does not create equivalent additional human work elsewhere.

The broader point is that even partial automation can have a meaningful operational effect when applied to a high-volume workflow.

Risk calibration makes this a controllable trade-off. An organization can begin with a conservative threshold, observe production outcomes, and expand automation as evidence accumulates.

Reliability comes from layered controls and operational discipline

The system's safety design goes beyond scoring individual diffs.

For AI-generated changes, RADAR uses the ACE (AI Commit Eligibility) pipeline. Eligible changes must pass safety checks, satisfy the applicable DRS threshold, and pass automated code review.

RACER runbooks receive additional controls because different automated tasks have different operational histories.

Each runbook must demonstrate an acceptable record over a 60-day lookback window, including zero production incidents and low revert and human-rejection rates. The system also imposes per-runbook daily limits, configurable risk thresholds, and denylisting for runbooks or sensitive use cases that should not be allowed to land automatically.

Established runbooks can receive more permissive thresholds and higher volume limits. The paper describes limits reaching 2,000 diffs per day for high-volume use cases.

This is a useful distinction between model confidence and operational confidence.

A model may judge an individual change to be low-risk. The organization must also establish that the process generating that change is trustworthy, that its historical performance supports the decision, and that its volume cannot overwhelm downstream systems.

The human-authored path has its own safeguards. RADAR Verification can allow eligible changes to ship with deferred review. RADAR Approval applies stricter criteria to determine whether that review can be waived entirely. Authors retain control over whether to use the automated path or request human review.

For automated bot changes, the paper also describes a configurable landing delay, during which a human can reject a pending change.

These controls make the system operationally governable. Teams can expand or pause automation, restrict particular sources, and adjust risk policies without treating the LLM as an unquestionable authority.

The resulting architecture is closer to a risk-managed production pipeline than a conventional code-review chatbot.

The production numbers: more throughput, fewer incidents

The paper reports the following outcomes for RADAR.

Metric	Reported result
Diffs reviewed by RADAR	535,290
Diffs landed by RADAR	331,720
Peak daily review throughput	25,000+ diffs
RADAR approval rate after calibration	60.31%
RADAR Verification pass rate	26.31%
Revert rate relative to non-RADAR diffs	1/3
Production-incident rate relative to non-RADAR diffs	1/50

The safety figures deserve careful interpretation.

RADAR-reviewed changes had approximately one-third the revert rate of non-RADAR changes and one-fiftieth the production-incident rate. The paper reports statistically significant differences for these comparisons.

These results are consistent with the system selecting changes that are safer to automate. They do not establish that automated review itself caused the lower incident rate.

Selection is central to the design: RADAR deliberately accepts only changes meeting eligibility requirements and risk criteria. The RADAR population therefore differs from the non-RADAR population before the review process even begins.

That is a feature of the system, but it complicates causal interpretation.

The authors also note that production incidents were manually reviewed by domain experts, and none of the incidents in the reported analysis were judged to have been detected by a human reviewer. That observation is useful context, although it does not establish that human review would never have caught any of those defects.

The right conclusion is that RADAR demonstrated substantial production-scale automation with strong observed safety outcomes. It would be premature to conclude that LLM-based review universally outperforms human review.

Why review latency fell: a queueing and economic perspective

The paper reports two distinct efficiency outcomes for RADAR-handled changes compared with human-reviewed diffs:

Median time to close was reduced by over 330%, as reported by the authors.
Median diff review wall time was reduced by 35%.

The first metric measures elapsed time from publication to closure. The second measures time spent waiting for and undergoing review.

These measures capture different bottlenecks. A diff may wait for a reviewer, undergo several rounds of feedback, and then wait again. Automating eligible changes removes much of their dependence on reviewer availability.

There is a terminology caveat worth noting: a conventional percentage reduction cannot exceed 100%. The paper's reported “over 330% reduction” in time to close should therefore be interpreted using the authors' reported comparison rather than as a literal reduction in elapsed time. The 35% reduction in review wall time is more straightforward to interpret.

Consider the queueing dynamics. Let:

lambda = the rate at which diffs arrive.
mu = the rate at which the review system can process them.

When arrival demand approaches or exceeds effective review capacity, waiting times can rise sharply. Increasing reviewer capacity helps, but so does removing eligible changes from the human queue.

RADAR effectively reduces the arrival rate seen by human reviewers:

lambda_human = lambda_total x (1 - p)

Here, p is the fraction of incoming diffs that are safely handled without human review. This is a simplified model: it assumes the automated fraction is stable and ignores differences in review effort between changes.

If 40% of changes qualify, the human queue receives only 60% of the original volume, assuming all else remains constant.

That does not mean total engineering cost falls by exactly 40%. Automation introduces compute, model-inference, validation, and operational costs. Some rejected diffs still need human review. Teams must also monitor incidents and maintain the eligibility policies.

The economic value comes from reallocating expensive human attention. Reviewers can spend more time on architecture, security, business logic, and difficult debugging, rather than routine changes that are amenable to automated verification.

In that sense, RADAR is a form of capacity engineering: it changes the demand placed on a constrained resource rather than relying exclusively on increasing that resource's supply.

What engineering teams can learn from Meta's approach

The paper's findings suggest several practical principles for organizations adopting coding agents.

Start with a narrow automation envelope. Identify recurring changes with clear correctness criteria: mechanical migrations, dead-code removal, formatting, and similarly constrained tasks. Do not begin by allowing unrestricted automated merges.

Separate risk prediction from semantic review. A risk score, static checks, and an LLM-based review agent answer different questions. Combining them provides more useful control than treating any one signal as sufficient.

Evaluate automation sources independently. A proven deterministic transformation and an experimental coding agent should not inherit the same permissions merely because both produce diffs. Track each workflow's incident history, reverts, rejection rates, and volume.

Make risk thresholds operational controls. Start conservatively. Increase coverage only when the measured outcomes justify it, and preserve the ability to disable a source or return changes to human review.

Measure reliability and throughput together. Approval rates and time savings alone can reward an unsafe system. Monitor production incidents, reverts, validation failures, and the share of changes that actually reach production.

Treat the results as organization-specific. Meta operates a large monorepo with extensive testing, established deployment controls, and mature review infrastructure. A smaller organization with weaker tests or less reliable rollback mechanisms should expect different outcomes.

There is also an important limit to the evidence. The paper uses production telemetry, observational comparisons, and a difference-in-differences-style analysis of efficiency outcomes rather than randomized assignment. The authors acknowledge that the findings may not generalize directly to other repositories and engineering environments.

That makes the study valuable as an account of a production system and its measured outcomes, rather than a universal benchmark for the quality of AI code review.

Conclusion: automate the routine, preserve attention for the consequential

Meta's RADAR illustrates a different way to think about AI-assisted software engineering.

The challenge is no longer just generating correct code. It is maintaining a review process that can keep pace with the volume of changes that coding agents produce.

RADAR addresses this by making automated review conditional on risk, source history, eligibility rules, semantic analysis, and deterministic validation. Its production deployment processed more than half a million diffs, landed more than 331,000, and exhibited substantially lower revert and production-incident rates than non-RADAR changes.

The deeper engineering lesson is that reliable automation depends on controlling where it operates, how its permissions expand, and what happens when a change fails a safety gate.

As AI agents become more productive, human review may become more valuable precisely because it can be concentrated on the changes where judgment matters most.

Discussion question: If you were introducing risk-aware code review in your organization, which changes would you allow to land without human review first—and what production evidence would you require before expanding that scope?

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

⭐ Star it on GitHub:

HexmosTech / LiveReview

Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview: Risk-Aware AI Code Review for Business-Critical Systems

LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.
blast-radius-demo.mp4
LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.

The exact math, not a black box Visualize blast radius at a glance Every factor that feeds the score

How does Blast Radius scoring work? (a more technical explanation)
Here's the goal:

A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.

A 300-line UI change in one file, fully covered by tests…

View on GitHub

Click below to try LiveReview with your codebase:

How DeepWiki Works: Turning a Codebase into a Searchable Mental Model

Shrijith Venkatramana — Tue, 06 Oct 2026 19:47:14 +0000

An unfamiliar codebase does not feel difficult because it contains 100,000 lines of code.

It feels difficult because you do not know which 500 lines matter.

That is the problem DeepWiki is really solving.

When Cognition launched DeepWiki in May 2025, it described it as the public version of its internal wiki product. The basic interaction was simple: take a GitHub URL, replace github.com with deepwiki.com, and get a generated wiki you can ask questions about. Cognition said more than 50,000 public repositories had already been indexed, including projects such as the Model Context Protocol and LangChain.

The interesting part is what has to happen between the URL and the answer.

DeepWiki cannot simply throw an entire repository into an LLM context window and say "explain this."

Instead, it builds several representations of the codebase:

                 Git repository
                       |
              clone / scan / filter
                       |
          +------------+------------+
          |                         |
    structural view            semantic index
    file tree + README          chunks + embeddings
          |                         |
          v                         v
   wiki structure                 FAISS
          |                         |
          +------------+------------+
                       |
                retrieved context
                       |
                       v
                     LLM
                       |
          +------------+------------+
          |                         |
      wiki pages                  answers
      + diagrams               + citations

The open DeepWiki-Open implementation gives us a useful way to inspect these mechanics. It should not be treated as Cognition's proprietary production source code; rather, it is an inspectable implementation of the same general product architecture.

Code understanding is an attention problem

There is a long history behind this idea.

In a 2012 observational study of 28 professional developers across seven companies, developers were found to use recurring comprehension strategies, often tried to avoid program comprehension as a task in itself, and frequently preferred face-to-face communication to documentation. The important point is that understanding software competes with the actual maintenance task the developer is trying to perform.

Suppose I ask:

How does authentication work in this repository?

There may be relevant code in:

src/auth/
api/middleware/
api/routes/
models/user.py
config/security.py
tests/auth/

The hard part is selecting the evidence.

A human engineer does this by building a mental model:

Request
  -> middleware
  -> token validation
  -> user lookup
  -> authorization
  -> handler

Only after this map exists does it become easy to dive into individual functions.

DeepWiki's architecture reflects the same decomposition.

There is a global representation of the repository that answers:

What exists, and how should I organize it?

And there is a retrieval representation that answers:

Which pieces are relevant to this particular question?

That distinction is the key to understanding the system.

DeepWiki builds two maps of the repository

Think of the first representation as a documentation outline and the second as a semantic search engine.

The documentation outline is hierarchical:

Repository
  |
  +-- Architecture
  |     +-- Request lifecycle
  |     +-- Data layer
  |     +-- Authentication
  |
  +-- Core modules
  |     +-- API
  |     +-- Services
  |     +-- Workers
  |
  +-- Deployment
        +-- Docker
        +-- CI/CD

The semantic index is much flatter:

chunk_001 -> src/api/router.py
chunk_002 -> src/auth/token.py
chunk_003 -> src/services/user.py
chunk_004 -> tests/auth/test_token.py
...

Each chunk gets converted into a vector.

This is important because the two structures optimize different tasks.

A hierarchy gives you coverage and navigation. A vector index gives you fast semantic lookup.

Cognition later pushed this idea another step by exposing DeepWiki through an MCP server. The official server provides programmatic operations such as ask_question, read_wiki_contents, and read_wiki_structure. In other words, the wiki is becoming an interface for software agents as well as a web page for humans.

This is a useful mental model:

DeepWiki is not primarily generating documentation. It is building an externalized representation of a codebase that humans and agents can query.

That is a more powerful way to think about it.

Before the LLM writes, the code gets indexed

Now we can look at the first genuinely technical stage.

The open implementation starts by getting the repository locally. For remote repositories it can use Git and supports shallow cloning, which reduces unnecessary history and transfer cost.

It then walks the repository, filters files and directories, and turns source files into documents.

One practical constraint immediately appears: tokenization.

The implementation tracks token counts and keeps embedding chunks within a configured maximum, with MAX_EMBEDDING_TOKENS defaulting to 8192.

Why does this matter?

Consider a repository with 100,000 lines of code.

A rough estimate might be:

100,000 lines
x 5 to 10 tokens / line
-----------------------
500,000 to 1,000,000 tokens

Sending the whole repository to an LLM for every question would be wasteful even for very large-context models.

Instead, suppose the repository becomes 1,000 chunks.

Each chunk is embedded:

chunk -> embedding model -> vector

If the embedding dimension is 1,536, then storing the raw float32 vectors takes roughly:

1,000 vectors
x 1,536 dimensions
x 4 bytes
----------------
~6 MB

That is tiny compared with the original source tree.

The vectors are therefore an inexpensive reusable index.

The implementation also keeps metadata with each chunk, including things such as:

file_path
is_code
token_count
line information

The line tracking is important because retrieval is only useful if you can get back from:

"this chunk looks relevant"

to:

src/auth/token.py, lines 84-121

This is one of the places where the system stops looking like a simple "LLM wrapper" and starts looking like a conventional search system.

The historical connection here is semantic code search.

In 2019, a few academics published CodeSearchNet, a corpus containing about six million functions across six programming languages, together with expert relevance judgments for natural-language queries. They framed the central challenge exactly this way: natural language and programming-language expressions describe the same concepts using very different vocabulary.

DeepWiki sits on the same basic bridge:

"where are users authenticated?"
              |
              v
        query embedding
              |
              v
       nearest code chunks
              |
              v
        language model

DeepWiki plans the wiki before generating the wiki

This is perhaps the most important architectural decision.

A naive system might do:

repository -> LLM -> giant documentation

DeepWiki-Open instead performs an intermediate planning step.

It first reads the repository structure and README, then asks the LLM to produce a machine-readable wiki structure.

Conceptually:

<wiki_structure>
  <section title="Architecture">
    <page title="Request Lifecycle"
          importance="high"
          files="api/main.py,api/routers/..."/>
    <page title="Data Layer"
          importance="high"
          files="api/repository.py,..."/>
  </section>

  <section title="Deployment">
    ...
  </section>
</wiki_structure>

The actual implementation uses an XML structure containing sections and pages. A WikiPage carries information such as its title, importance, related pages, and the source file paths that should provide its context.

This is a form of coarse-to-fine generation:

repository
    |
    v
global structure
    |
    +---- page A -> relevant files -> generation
    |
    +---- page B -> relevant files -> generation
    |
    +---- page C -> relevant files -> generation

The advantage is that each individual generation problem becomes much smaller.

Suppose the model decides there should be 25 pages.

Rather than asking one model call to understand and document the entire repository, you can make roughly 25 focused generation problems.

The page prompt can then say, in effect:

You are writing the "Authentication" page.

Relevant files:
  api/auth.py
  middleware/security.py
  models/user.py

Explain:
  - how authentication starts
  - how credentials are validated
  - where identity is stored
  - important failure paths

That is a much more constrained task.

The open implementation goes further. Page prompts ask the model to identify the source files used for the explanation and impose formatting requirements for diagrams. The generated links are subsequently normalized into repository-specific URLs.

And there is a nice engineering detail hiding underneath all of this: the parser assumes the model can fail.

The structure parser strips formatting artifacts and has regex fallbacks when strict XML parsing fails, including cases where the model output gets truncated.

Questions are answered by retrieval (rather than memory)

Once the repository has been indexed, answering a question becomes a classic RAG pipeline.

The open implementation's flow is roughly:

user query
    |
    v
query embedding
    |
    v
FAISS top-k retrieval
    |
    v
relevant code chunks
    |
    +---- conversation history
    |
    v
prompt builder
    |
    v
LLM
    |
    v
answer

The mathematics is straightforward.

Let the query embedding be:

q = [q1, q2, ..., qd]

and a code chunk embedding be:

x = [x1, x2, ..., xd]

A common similarity measure is cosine similarity:

cos(q, x) = (q dot x) / (||q|| ||x||)

The intuition is simply:

Do the query and this code chunk point in a similar direction in semantic space?

For example, a developer asks:

Where is the GitHub access token used?

A lexical search might miss a function called:

authenticate_repository(...)

because the words "access token" do not appear in the function name.

A semantic embedding can put concepts such as:

access token
OAuth credential
repository authentication
private clone

near one another.

Suppose there are 5,000 chunks and the embeddings have 1,536 dimensions.

A flat exact scan is roughly:

5,000 x 1,536
~= 7.7 million dimension comparisons

That is quite manageable in optimized native code. For much larger indexes, approximate nearest-neighbor structures can reduce the search work further.

The academic foundation for this style of architecture was laid out explicitly by Patrick Lewis and colleagues in the 2020 RAG paper. Their formulation combines a model's learned parametric memory with an external non-parametric memory represented by a dense vector index. The language model does the reasoning and generation; retrieval supplies the changing, inspectable evidence.

DeepWiki-Open follows this basic pattern, although its implementation is an application architecture rather than a direct reproduction of the RAG research model.

There is also a more expensive mode.

Its Deep Research path runs multiple iterations, with separate stages for planning, intermediate updates and final synthesis. In the inspected implementation, the loop can traverse the repository repeatedly, up to five iterations.

So the modes form something like:

Fast
  query -> retrieve -> answer

Deep Research
  query
    -> plan
    -> retrieve
    -> inspect
    -> refine
    -> retrieve again
    -> synthesize

Codemap
  query
    -> retrieve
    -> generate skeleton
    -> enrich sections
    -> ground citations

Codemap is particularly interesting.

The implementation first creates a skeleton, then enriches it with prose and diagrams. It does not simply trust the line numbers produced by the LLM. Instead, it can read the actual file, search for the cited snippet, and recover the real line range.

That creates a three-step chain:

LLM says:
  "this claim comes from foo.py"

        |
        v

system reads foo.py

        |
        v

system locates the snippet

        |
        v

citation points to actual lines

This is a small but important shift.

The model is allowed to propose evidence.

The deterministic program decides whether that evidence actually exists.

That principle generalizes far beyond documentation systems.

The hard part at scale is orchestration

Once you view DeepWiki as an indexing and generation system, the operational design starts to make sense.

Wiki generation is too slow to treat as a normal HTTP request.

The open implementation therefore has a TaskRegistry and WikiTask state machine:

PENDING
   |
INDEXING
   |
DETERMINING_STRUCTURE
   |
GENERATING
   |
COMPLETED / FAILED

The frontend receives progress through Server-Sent Events.

This sounds like UI plumbing, but it is actually part of the product architecture.

A repository wiki might require:

clone
+ scan
+ embedding
+ structure generation
+ N page generations
+ post-processing

That can take long enough that a synchronous request would be the wrong abstraction.

The implementation also places explicit limits around concurrency.

The global wiki-task concurrency defaults to roughly half the available CPU cores, while the RAG layer separately uses a semaphore with a default concurrency of four.

That gives us a simple throughput model.

Suppose:

P = 24 pages
C = 8 concurrent page generations
t = 20 seconds average per page

Ignoring model-provider throttling and uneven page sizes:

T ~= ceil(P / C) x t
  ~= 3 x 20
  ~= 60 seconds

Serial execution would be:

24 x 20 = 480 seconds

So concurrency changes the economics of documentation generation dramatically.

But concurrency also creates failure modes.

The test suite in the open implementation contains two revealing cases:

test_submit_joins_active_task
test_page_failure_yields_placeholder_but_completes

The first prevents two requests for the same repository from kicking off duplicate work.

The second says that if one page fails, the entire wiki should still finish.

That is exactly the sort of detail you expect in a real asynchronous system. The expensive operation is the whole repository, so a single failed page should degrade one output rather than invalidate the entire job.

Caching is the other half of this economics.

The open implementation stores generated wikis in a local cache keyed by repository information such as host, owner, repository and language.

That creates a simple business equation:

cost per repository
  =
  clone cost
  + embedding cost
  + structure-generation cost
  + page-generation cost

The first two are largely preprocessing costs.

The expensive recurring component is usually generation:

generation cost
  ~ sum over pages of
      (retrieved input tokens + output tokens)
      x model price

Imagine:

25 pages
x 6 retrieved chunks/page
x 700 tokens/chunk
-------------------
105,000 retrieved input tokens

That is only an illustrative calculation. Actual context sizes, prompts, retries and outputs can move the number substantially.

The key architectural consequence is independent of the exact price:

You want to pay the expensive model cost once, then reuse the resulting representations many times.

That is why a wiki can make sense as a persistent artifact rather than regenerating an explanation on every question.

Cognition made this economics explicit in April 2026 when it said higher-quality DeepWiki generation modes would be usage-priced because these workflows consume meaningful compute, while the existing baseline generation experience would remain free.

The deeper idea is that DeepWiki is doing something closer to compilation than chat.

A compiler spends compute up front to create an intermediate representation so subsequent operations become cheaper.

DeepWiki spends compute up front to create:

repository
   -> structural representation
   -> semantic representation
   -> generated documentation
   -> cached knowledge

Then later questions can operate on those artifacts instead of repeatedly starting from raw source.

Conclusion: the wiki is really a cache for software understanding

The easiest way to misunderstand DeepWiki is to think of it as:

"An LLM that writes documentation."

The more useful mental model is:

"A system that compiles a codebase into representations optimized for human and machine understanding."

The pieces line up:

Git repository
    |
    +--> file tree + README
    |        |
    |        +--> wiki structure
    |                 |
    |                 +--> generated pages
    |
    +--> chunks
             |
             +--> embeddings
                      |
                      +--> semantic retrieval
                               |
                               +--> RAG answers
                               +--> deep research
                               +--> codemaps
                               +--> citations

And this explains why the architecture has so many seemingly mundane components.

The embeddings matter because the repository is too large to read every time.

The wiki structure matters because search alone does not give you a coherent global model.

The task registry matters because generation takes time.

The cache matters because generation costs money.

The citation grounding matters because an LLM's claim that "this is line 84" is not evidence that line 84 actually contains the claim.

The parser fallbacks matter because models sometimes emit malformed structured output.

The whole system is therefore a combination of old ideas and new capabilities: information retrieval, software indexing, asynchronous job processing, caching, and program comprehension, with an LLM sitting in the middle as the component that turns selected code into explanations.

That may also be the larger pattern for AI software engineering.

The winning systems may be less about giving a model more context and more about building better representations of the context before the model ever sees it.

What would you add to DeepWiki's representation of a codebase before trusting it with a 500,000-line monorepo: dependency graphs, Git history, runtime traces, tests, production telemetry, or something else?

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

⭐ Star it on GitHub:

HexmosTech / LiveReview

Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview: Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.
blast-radius-demo.mp4
LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.

The exact math, not a black box Visualize blast radius at a glance Every factor that feeds the score

How does Blast Radius scoring work? (a more technical explanation)
Here's the goal:

A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.

A 300-line UI change in one file, fully covered by…

View on GitHub

Click below to try LiveReview with your codebase:

Blender for Developers: 15 Concepts Explained Simply

Shrijith Venkatramana — Sat, 03 Oct 2026 19:49:10 +0000

You already understand objects, trees, transforms, interpolation, pipelines, coordinate systems, and data structures.

Then you open Blender.

Suddenly there are meshes, vertices, normals, modifiers, F-curves, armatures, constraints, materials, world space, local space, UVs, nodes, render engines, and approximately 700 buttons.

The underlying ideas are much smaller than the interface makes them appear.

A useful way to think about Blender is:

Blender is a system for describing a scene as data, transforming that data over time, and finally projecting it into pixels.

That is much closer to a software system than to a drawing program.

This article picks 15 concepts from the Blender vocabulary that give you the most leverage for making small animations. The terminology follows the Blender 5.2 LTS glossary.

I will use a simple running example: imagine a cube-shaped owl mascot that jumps into frame, looks at the camera, hops onto a platform, and points at some text.

1. Start With the Right Mental Model

1. Object

An Object is Blender's container for something that exists in the scene: a mesh, curve, camera, light, armature, and so on.

For a developer, think of it roughly as an entity with:

object
    transform
    data
    relationships
    modifiers
    animation

A cube in Blender isn't fundamentally "a cube."

It is an object whose underlying data happens to be a cube-shaped mesh.

That distinction becomes important very quickly.

Suppose your owl has two identical eyeballs.

You can have:

Eye_Left  ----\
               -> same mesh data
Eye_Right ----/

The two objects can have different positions and rotations while sharing the same underlying geometry.

This is one reason Blender can represent a complicated scene without duplicating everything.

2. Data-blocks

Blender calls its fundamental named pieces of reusable file data data-blocks.

Meshes, materials, armatures, node trees and animation actions are examples. The glossary explicitly describes data-blocks as the base units of data in a .blend file.

This is much easier to understand if you think like a programmer.

Imagine:

Object A -> Mesh #17
Object B -> Mesh #17
Object C -> Mesh #23

The objects are instances/references to data.

This explains many Blender behaviors that otherwise seem strange:

changing an object's transform does not necessarily modify its mesh
two objects can share geometry
materials can be reused
animation data is stored separately
linked data can be reused across files

Blender itself grew out of an in-house production tool at Ton Roosendaal's Dutch animation studio NeoGeo. The first Blender source files date to January 2, 1994. The original motivation was partly practical: make a 3D production system where repeated changes from clients were easier to manage.

That history explains something about Blender's architecture: it was built around manipulating a production scene, rather than merely drawing individual pictures.

3. Transform + Local Space + World Space

Every object has a transform:

location
rotation
scale

Conceptually, you are applying a transformation to geometry.

A simplified matrix view is:

p_world = M * p_local

where:

p_local = point in object's coordinate system
M       = object's transformation matrix
p_world = point in world coordinates

So if your owl's eye is at:

local position = (0.2, 0.0, 0.1)

and the owl itself moves across the scene, the eye can move with it without changing its local coordinates.

This is exactly the same reason hierarchical transforms exist in graphics engines.

A common developer mistake is to ask:

"Why is this object at (2, 3, 5) when I set its position to (0, 0, 0)?"

Often the answer is that (0,0,0) is being interpreted in a different coordinate system.

Blender distinguishes world space from local space in roughly the same way a game engine does.

One more reason to understand transforms: rotation.

Euler rotations are intuitive:

rotate X
rotate Y
rotate Z

but the order matters. In certain orientations, two rotation axes can line up and you get gimbal lock.

Quaternions exist partly because representing and interpolating 3D rotations directly with three sequential Euler angles has these problems.

You don't need quaternion mathematics to make your first animation.

You do need to know that rotation representation is a mathematical choice rather than merely a UI preference.

2. Geometry Is Data, Not a Picture

4. Mesh

A mesh is the classic polygonal representation of a 3D shape:

vertices
edges
faces

The glossary defines a mesh exactly this way.

Think of it as a graph embedded in 3D.

A cube might contain:

8 vertices
12 edges
6 faces

The faces are what give the object a surface.

This leads naturally to the next concept.

5. Topology

Topology is the arrangement of those vertices, edges and faces.

Geometry tells you things such as:

"this vertex is here"

Topology tells you:

"this vertex is connected to these vertices"

That distinction becomes extremely important when deforming objects.

Suppose your owl is just a hard cube.

Topology barely matters.

Now suppose you want to turn the cube into a character whose face bends when it speaks.

Suddenly topology matters enormously.

A rough debugging invariant for a closed genus-0 mesh is Euler's formula:

V - E + F = 2

For a cube:

V = 8
E = 12
F = 6

8 - 12 + 6 = 2

This doesn't tell you whether a model is aesthetically good.

It tells you that the connectivity has certain mathematical properties.

A torus, for example, has a different topology:

V - E + F = 0

This is one reason graphics programmers sometimes find topology easier to understand than artists do initially: it is literally a graph-structure problem.

6. Normals

A normal is a unit vector perpendicular to a surface.

Why should you care about a vector sticking out of a polygon?

Because lighting cares.

For a simplified diffuse lighting model:

brightness ~ max(0, N dot L)

where:

N = surface normal
L = direction toward the light

If the surface faces the light:

N dot L ≈ 1

If it is perpendicular:

N dot L ≈ 0

If it points away:

N dot L < 0

and the contribution is clamped to zero.

So a huge portion of what your eye interprets as "shape" is actually a consequence of geometry plus normals plus lighting.

This also explains why a model can look completely broken after a seemingly harmless edit: sometimes the geometry is fine but the normals are inconsistent.

3. Modifiers: Treat Geometry Like a Pipeline

7. Modifiers

A modifier is a non-destructive operation applied on top of existing data.

This is one of Blender's most useful ideas.

Instead of:

mesh -> permanently modify mesh

you can have:

base mesh
   |
   v
bevel
   |
   v
subdivision
   |
   v
other modifier
   |
   v
final geometry

This is almost exactly how a compiler pipeline or image-processing pipeline feels.

You preserve a relatively simple source representation and derive progressively richer representations from it.

That has a major operational benefit:

Keep the source representation cheap to edit.

8. Bevel

A bevel rounds or chamfers an edge.

This sounds trivial.

It is one of the highest-return techniques in product animation.

Compare:

sharp cube

with:

slightly rounded cube

The second catches highlights differently and therefore looks much more like a manufactured object.

This matters because the real world almost never contains perfectly sharp macroscopic edges.

A two-minute bevel can therefore produce more perceptual improvement than hours of adding geometric detail elsewhere.

For your cube owl, you might start with:

cube
+ bevel
+ material
+ light

and already have something usable.

9. Subdivision Surface

Subdivision Surface takes a relatively low-poly mesh and generates a smoother higher-poly surface.

The important intuition is:

You are specifying a coarse control structure and asking an algorithm to construct a smoother surface from it.

This goes back to Ed Catmull and Jim Clark's 1978 work on recursively generated B-spline surfaces for arbitrary topological meshes. Their paper is one of the foundations of the subdivision-surface approach used throughout computer graphics.

The cost is straightforward.

A subdivision step roughly turns:

1 quad -> 4 quads

So if you start with 10,000 faces:

10,000
   ->
40,000
   ->
160,000
   ->
640,000

Three levels means roughly 64x as many faces as the starting mesh.

This is a recurring graphics lesson:

A small-looking visual operation can create a huge amount of underlying computation.

For simple product animation, you usually want the lowest-resolution base mesh that gives you enough control, and let modifiers generate detail later.

4. The Camera Turns 3D Into 2D

10. Shading

Shading is the process of determining how surfaces appear under lighting.

Developers sometimes approach materials backwards:

"What color should this object be?"

A better question is:

"How should this surface interact with light?"

Two objects can have the same base color but look completely different because one is:

smooth + glossy

and another is:

rough + diffuse

The glossary distinguishes diffuse light, specular light, roughness and related material concepts.

This is why lighting is part of modeling.

A perfectly modeled object under poor lighting can look worse than a simple object under good lighting.

11. Camera Projection

A camera decides how the 3D world becomes a 2D image.

There are two important projections:

perspective
orthographic

In perspective projection, objects farther away look smaller.

A simplified relationship is:

x_screen ~ f * x / z
y_screen ~ f * y / z

So if:

z doubles

then the projected size is approximately:

half

This is why a road converges toward the horizon and why a face looks larger when a camera gets close.

In orthographic projection, distance does not produce the same scale change. The mapping is conceptually closer to:

x_screen ~ s * x
y_screen ~ s * y

The Blender glossary defines perspective projection by rays converging through an observer point, while orthographic projection uses parallel viewing lines.

For a technical product video, orthographic projection can sometimes give the clean "diagram" feeling.

Perspective gives the familiar photographic feeling.

12. Ray Tracing

Ray tracing asks a simple question:

If light travels through this scene, what happens when it hits things?

A ray may:

hit -> reflect
hit -> refract
hit -> absorb

and continue.

That is the conceptual basis of ray-traced rendering.

The naive algorithm is expensive.

If you have:

R rays
N triangles

then checking every ray against every triangle gives roughly:

R * N

intersection tests.

A scene with:

1,000,000 triangles

and billions of rays would obviously be unpleasant.

This is why structures such as a BVH, or Bounding Volume Hierarchy, exist. Instead of testing a ray against every triangle immediately, the renderer first tests hierarchical bounding volumes and eliminates large portions of the scene.

This is a familiar software-engineering pattern:

brute force
    ->
spatial index
    ->
avoid work

Databases have indexes.

Compilers have intermediate representations.

Graphics renderers have acceleration structures such as BVHs.

5. Animation Is a Function of Time

Here is the biggest conceptual jump for a programmer coming into Blender.

Animation is not:

frame 1
frame 2
frame 3
frame 4
...

It is:

property = f(time)

Blender's animation system is built around exactly this idea.

13. Keyframes

A keyframe specifies a property at a particular frame.

For example:

frame 1   -> location.z = 0
frame 20  -> location.z = 3

You do not normally specify frames 2 through 19.

You specify the important states.

This idea has a direct historical connection to traditional animation. Senior animators created important poses, while other artists filled in the intermediate frames.

John Lasseter's 1987 paper, "Principles of Traditional Animation Applied to 3D Computer Animation," described how these traditional animation ideas transfer into 3D computer animation. He discussed principles such as squash and stretch, timing and anticipation using examples including Luxo Jr..

So when you place two keyframes and let Blender fill in the motion, you are using a very old animation production idea implemented as a mathematical system.

14. F-Curves and Interpolation

Blender stores the changing value of an animated property in an F-Curve.

Think:

time -> property value

For example:

frame 0   -> x = 0
frame 25  -> x = 10

The system computes the values between them.

The Blender manual describes F-Curves as curves representing animated property values, with time on one axis and the property's value on the other.

This is where animation becomes mathematically interesting.

A Bézier curve can be written as:

B(t) =
    (1-t)^3 P0
  + 3(1-t)^2 t P1
  + 3(1-t)t^2 P2
  + t^3 P3

0 <= t <= 1

You don't normally calculate this yourself.

The important intuition is that:

P0, P3 = where the motion starts and ends
P1, P2 = influence the shape of the motion

That means you are effectively programming velocity and acceleration without explicitly writing a physics simulation.

Consider your owl jumping.

You might specify:

frame 1  -> z = 0
frame 12 -> z = 2
frame 24 -> z = 0

A linear interpolation produces something mechanical.

A carefully shaped F-Curve can make the motion:

slow
  ->
fast
  ->
slow

which your visual system interprets as intentional movement.

The mathematics is simple.

The difficult part is choosing the function.

That is why good animation is closer to control-system design than to drawing.

6. Constraints and Relationships: Build the System Once

15. Parenting and Constraints

Parenting creates a relationship in which one object affects another. A child inherits relevant transformations from its parent.

Suppose your owl has:

Owl
 |
 +-- Eye_Left
 +-- Eye_Right
 +-- Beak
 +-- Wing_Left
 +-- Wing_Right

If you move the owl, everything moves with it.

You don't need six independent animations.

This is another familiar programming pattern:

parent transform
      |
      +--> child transform

Then there are constraints.

A constraint says, roughly:

"derive this object's behavior from some other thing."

For example:

Camera
   |
   +-- look at --> Owl

or:

Eye
   |
   +-- track --> Target

The important mental shift is:

Keyframes describe values. Constraints describe relationships.

That distinction becomes increasingly valuable as a scene grows.

Imagine manually animating a camera to follow the owl.

You might keyframe:

frame 1
frame 10
frame 20
frame 30
...

But if the owl's path changes, the camera animation becomes invalid.

A constraint instead captures:

camera orientation = function(target position)

Now the owl can move differently and the camera can continue following it.

This is essentially dependency management.

Blender's constraint system even distinguishes between world, local, pose and other spaces, because the mathematical frame in which a relationship is evaluated changes the result.

What about Armatures and IK?

If you move from a cube mascot to an articulated character, you eventually encounter armatures, bones, and inverse kinematics.

An armature is a skeleton-like structure made of bones used to control how a character or object moves.

With forward kinematics:

shoulder
   ->
elbow
   ->
hand

you move the parent and the children follow.

With inverse kinematics:

hand target
    ->
elbow position
    ->
shoulder rotation

you specify the result and the system solves backward through the chain.

For a simple product animation, you may never need IK.

That is an important productivity lesson:

Learn the concepts your animation actually requires.

A cube owl pointing at a screen can often be built with objects, transforms, keyframes and constraints.

You don't need to build a Pixar-style character rig merely because Blender has one.

7. The Production Loop: Pixels Have a Price

Once you understand the preceding concepts, Blender becomes much easier to reason about as a pipeline:

scene data
   |
   v
geometry
   |
   v
modifiers
   |
   v
camera
   |
   v
shading + lighting
   |
   v
render
   |
   v
image

Animation adds another dimension:

time
 |
 +----> transforms
 |
 +----> materials
 |
 +----> camera
 |
 +----> constraints

The final result is still just a sequence of images.

And that has a very simple operational consequence.

Suppose you make a 10-second video at 30 FPS:

10 * 30 = 300 frames

At 1920 x 1080:

1920 * 1080 = 2,073,600 pixels/frame

Therefore the entire video contains approximately:

2,073,600 * 300
= 622,080,000 pixels

So your "10-second animation" is actually around 622 million pixel outputs before accounting for multiple samples, reflections, transparency, shadows, motion blur, depth of field and other rendering work.

That is why render economics matter.

A rough mental model is:

total render time
    ~
number of frames
*
time per frame

If a frame takes 20 seconds:

300 * 20 sec
= 6,000 sec
= 100 minutes

Cutting frame time from 20 seconds to 10 seconds saves roughly 50 minutes.

But reducing the frame count from 300 to 150 does the same thing.

This is why production graphics has a lot in common with systems engineering.

You are constantly trading:

quality
geometry
samples
resolution
memory
time

against one another.

And this leads to a useful rule for small animations:

Optimize the pipeline, not individual frames.

A badly structured scene that takes 40 seconds per frame is a 200-minute problem.

A well-structured scene that takes 5 seconds per frame is a 25-minute problem.

The 15 Concepts in One Mental Model

You can compress almost the entire article into this:

1.  Object
        |
2.  Data-block
        |
3.  Transform / coordinate space
        |
4.  Mesh
        |
5.  Topology
        |
6.  Normal
        |
7.  Modifier
        |
8.  Bevel
        |
9.  Subdivision
        |
10. Shading
        |
11. Camera projection
        |
12. Ray tracing
        |
13. Keyframes
        |
14. F-Curves / interpolation
        |
15. Parenting / constraints

And then remember the fundamental developer abstraction:

scene = data + relationships + functions(time) + renderer

Once you see Blender this way, the interface becomes much less mysterious.

You aren't learning thousands of buttons.

You are learning a fairly small set of abstractions that happen to have a very large UI wrapped around them.

Blender itself is a good historical example of why these abstractions matter. It began as a production tool inside NeoGeo, became a commercial product through NaN, and after NaN collapsed in 2002, Ton Roosendaal's "Free Blender" campaign raised enough money to buy back the rights and release the software as GPL open source on October 13, 2002.

The interesting part is that much of the software's complexity exists to let you describe increasingly complicated scenes without manually specifying every consequence.

That is exactly the kind of abstraction developers should appreciate.

The practical takeaway

For a first useful animation, learn these operations:

add object
move object
rotate object
scale object
bevel object
apply material
place camera
place light
add keyframes
edit F-curves
parent objects
add a constraint
render

You can make a remarkable amount of professional-looking motion before touching fluids, particles, simulations, sculpting, UV workflows, character rigs or procedural geometry.

That is probably the right place to start.

What Blender concept took you the longest to understand, even though it turned out to be conceptually simple?

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

⭐ Star it on GitHub:

HexmosTech / LiveReview

Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview: Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.
blast-radius-demo.mp4
LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.

The exact math, not a black box Visualize blast radius at a glance Every factor that feeds the score

How does Blast Radius scoring work? (a more technical explanation)
Here's the goal:

A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.

A 300-line UI change in one file, fully covered by…

View on GitHub

Click below to try LiveReview with your codebase:

How chDB Turns Ephemeral Agent State into Durable Memory

Shrijith Venkatramana — Wed, 30 Sep 2026 19:44:34 +0000

Agent memory sounds like an LLM problem until you actually deploy an agent.

A useful agent needs to retain things such as:

project conventions
previous decisions
tool results
failed approaches
user preferences
investigation history

Now put that agent inside a temporary CI runner or Lambda or sandbox.

Where does all of that state live?

A file works well until you need the agent to move to another machine. A remote database solves portability, but introduces a network dependency for every read and write.

chDB Durable occupies the space between the two.

It keeps the database embedded and local, while using object storage as the durable backing layer.

This becomes common as agents move between laptops, CI runners, sandboxes, containers, and short-lived cloud machines. A local agent can accumulate weeks of useful state: project conventions, user preferences, tool failures, corrected assumptions, previous decisions, and the evidence behind those decisions.

A SQLite file can preserve some of this.

A remote Postgres or ClickHouse server can preserve much more.

But there is a useful middle ground:

Keep the database local for fast reads, while making its state durable in object storage.

That is the idea behind the new chDB Durable Layer.

1. Agent memory is becoming a database problem

A useful way to think about an agent is as a loop:

observe
  -> reason
  -> act
  -> observe
  -> reason
  -> act
  -> ...

Memory sits across this loop.

The simplest implementation is often a list of messages:

messages = [
    "User prefers uv over Poetry",
    "The checkout service runs in eu-west-1",
    "Do not modify migrations during deployment",
]

Eventually that becomes insufficient.

Suppose an agent originally learns:

Deploy checkout-api to eu-west-1.

Later, someone changes the infrastructure:

Deploy checkout-api to eu-central-1.

What should the agent store?

Just the new answer?

Or also:

old belief
new belief
who changed it
when it changed
why it changed
what evidence supported it
which agent used the old belief

This is where the problem starts looking much more like a database than a prompt.

The research literature has been moving in this direction for several years. MemGPT, for example, explicitly treats an LLM agent as a system with multiple memory tiers and a mechanism for moving information between them. The underlying idea is familiar from operating systems: fast working memory is valuable, but larger persistent state must exist somewhere else.

Agent memory is therefore partly a data placement problem.

The interesting question becomes:

Where should the agent's durable state live?

2. The SQLite-to-server spectrum has a missing middle

There are several obvious answers.

Local SQLite

For a small agent, SQLite is excellent.

It is embedded, transactional, mature, and requires no server.

An agent can checkpoint its state:

agent process
    |
    v
SQLite file

The problem appears when memory becomes historical and analytical.

You start wanting queries such as:

SELECT project, count()
FROM tool_events
GROUP BY project;

Or:

SELECT *
FROM memory_history
WHERE memory_id = 'deployment-region'
ORDER BY version;

Or:

SELECT model, sum(tokens), sum(cost)
FROM events
WHERE created_at >= today() - 7
GROUP BY model;

That is a different workload from "save my current checkpoint."

SQLite plus replication

You can replicate SQLite's WAL to object storage.

That solves the single-disk durability problem.

But now there is another component in the deployment:

agent
  |
SQLite
  |
replication process
  |
object storage

There is more operational machinery, and the underlying database is still fundamentally a transactional row store.

Remote database

The other solution is to put memory into Postgres, pgvector, ClickHouse, or a managed memory platform.

This solves portability and shared access.

The architecture becomes:

agent
   |
network
   |
database server

Now every recall is potentially a remote operation.

For a conventional application, this is often perfectly reasonable.

For an agent, the distinction matters because one user request can generate many sequential tool calls.

If an agent makes 10 data-access operations and each introduces 50 ms of network latency:

10 * 50 ms = 500 ms

Half a second appears before counting model inference.

And that is the optimistic case. Retries, connection failures, authentication, network jitter, and service saturation add another dimension.

Embedded analytical database

Now consider chDB.

It puts the ClickHouse query engine inside the application process.

agent process
    |
   chDB
    |
local database

This gives the agent SQL, columnar storage, analytical queries, and local execution without requiring a database server.

But one problem remains:

The database is still attached to the machine.

This was the problem the chDB Durable Layer was designed to address.

3. The idea: local working copy, durable object

The architecture is simple enough to draw on a whiteboard:

                 object storage
              authoritative state
                     |
             flush / checkpoint
                     |
                     v
+---------------------------------------+
|              Agent process            |
|                                       |
|      chDB + local MergeTree           |
|                                       |
|      recall -> local SQL query        |
|      writes -> local WAL buffer       |
|                                       |
+---------------------------------------+

The crucial distinction is between the working copy and the authoritative copy.

The agent queries its local chDB database.

Object storage holds the durable state.

When the agent needs to publish changes, it calls:

obj.flush()

When it wants to create a new base snapshot and shorten future recovery:

obj.checkpoint()

The result is an interesting deployment shape:

No database server
No persistent volume requirement
No replication sidecar
No network call for every recall

The durable object has an identity.

For example:

s3://my-agent-state/agent-memory
└── acme-checkout-api

The namespace is the storage root.

The object is one complete chDB database.

That object can later be opened on another machine.

This is particularly useful for agents because their compute environment is increasingly disposable.

Imagine:

Laptop
   |
   | develop
   v
CI runner
   |
   | test
   v
sandbox
   |
   | reproduce bug
   v
another machine

The runtime can disappear.

The agent's accumulated state does not have to.

4. What actually happens during durability?

The interesting engineering is underneath the small API.

There are four concepts to keep in your head.

`execute()`: mutate locally

An operation such as:

obj.execute(
    "INSERT INTO memories VALUES (...)"
)

changes the local working database.

The mutation enters the WAL buffer.

It has not necessarily become durable yet.

`flush()`: make a promise real

obj.flush()

is the durability boundary.

When it returns, the writes covered by that flush have reached object storage.

That gives application code an important semantic rule:

execute()
    = "I changed my local memory."

flush()
    = "I promise this memory survived the machine."

For an agent API, this distinction matters.

Consider a tool:

remember("The deployment region is eu-central-1")

Returning success immediately after execute() means the tool may claim to have remembered something that disappears when the sandbox disappears.

Returning success only after flush() gives the tool an actual durability guarantee.

`checkpoint()`: create a new base

Between checkpoints, the durable state consists conceptually of:

base snapshot
+
WAL segments

Recovery therefore looks like:

restore base
+
replay committed WAL
=
current database

As the WAL grows, recovery has more work to do.

A checkpoint folds the current state into a new base.

So there is a tradeoff:

frequent checkpoint
    -> more transfer work
    -> shorter recovery

infrequent checkpoint
    -> less transfer work
    -> more WAL replay

This is a familiar database systems tradeoff appearing inside an agent architecture.

`head.json`: who owns the database?

There is another problem.

Suppose two workers open:

acme-checkout-api

at the same time.

Both could believe:

"I am the writer."

That is dangerous.

Durable uses a head record with conditional writes to establish lease and fencing semantics.

At a high level:

read current head = version 17

writer A:
    CAS(17 -> 18)  succeeds

writer B:
    CAS(17 -> 18)  fails

Only one process becomes the current writer.

This gives Durable a deliberate constraint:

one writer per object.

That is a useful property for a per-user or per-project agent brain.

It is a poor fit for a team database in which ten independent processes continuously update the same object.

5. Store memory as history, not as one giant document

This is where using an analytical database becomes interesting.

A naive memory system might have:

memory.json

A more useful design has several relations:

memories
memory_history
raw_transcripts
events

For example:

CREATE TABLE memory_history
(
    memory_id String,
    version UInt32,
    op LowCardinality(String),
    content String CODEC(ZSTD(3)),
    edited_by String,
    edited_at DateTime64(3, 'UTC'),
    prev_id String,
    note String
)
ENGINE = MergeTree
ORDER BY (memory_id, version, edited_at);

Now a revision does not destroy the original fact.

Instead:

version 1:
Deploy checkout-api to eu-west-1.

version 2:
Deploy checkout-api to eu-central-1.

You can query the current state.

You can also query the history.

You can even ask:

Why did the agent believe this?

Which evidence produced the belief?

When did the belief change?

How often was this memory recalled?

Did those recalls actually help?

That last class of questions is important.

A memory system should eventually become capable of analyzing its own memory.

That is naturally expressed with SQL.

Raw evidence and actual memory are different

One especially useful distinction in the ClickMem design is between evidence and memory.

A transcript may contain:

User:
"Maybe the service is in eu-west-1?"

Tool:
"Deployment failed."

Agent:
"Let's inspect the infrastructure."

None of that necessarily deserves promotion into long-term memory.

Instead:

raw_transcripts
    = searchable evidence

memories
    = curated beliefs

This reduces a common failure mode of memory systems:

every conversation
    -> memory
    -> future behavior

A transient mistake then becomes persistent behavior.

The alternative is:

conversation
    -> evidence
    -> evaluation
    -> memory

That resembles the observation/retrieval/reflection pattern explored in early generative-agent research, but gives the resulting state a conventional data model.

6. The math, storage economics, and operational tradeoffs

There are two pieces of mathematics worth understanding.

Semantic recall

Suppose every memory has an embedding vector m, and the current task has query vector q.

A common semantic score is cosine similarity:

score(q, m) = (q dot m) / (||q|| ||m||)

The intuition is simple:

same direction
    -> high similarity

different direction
    -> low similarity

A memory query can therefore look like:

1. Filter project
2. Filter active memories
3. Generate semantic candidates
4. Rank candidates
5. Return top K

For a brute-force scan with:

N memories
D embedding dimensions

the rough amount of dot-product work is:

N * D

For example:

N = 100,000 memories
D = 1,536

N * D
= 153,600,000

That is roughly 154 million multiply-add terms for a full scan.

Whether that is acceptable depends on hardware, query frequency, and filtering. The useful observation is that memory retrieval is a data-processing problem, so the choice between prefiltering, vector indexes, approximate nearest-neighbor methods, and brute-force scans becomes a database engineering decision.

chDB already provides the local analytical machinery around the embedding column; the durability layer does not need to know whether your recall logic is lexical, vector-based, temporal, or some combination.

Storage is often cheaper than the memory architecture makes it look

ClickHouse published an instructive experiment using a 1.45 GB Claude Code transcript sample containing 213,721 rows.

Loaded into MergeTree:

ZSTD(3): 521 MiB
LZ4:     991 MiB

So the ZSTD representation is roughly:

521 / 1,383 ~= 0.38

or about a 62% reduction relative to the original binary size.

There was another useful observation:

The screenshots represented only about 0.8% of rows, but roughly two-thirds of the bytes.

That leads to a practical rule:

textual history -> compress in the database

large binary objects
    -> keep outside the table
    -> store object keys in the database

Otherwise your screenshots, audio, or video become part of every checkpoint.

A rough storage-cost calculation

Suppose an agent eventually accumulates:

0.5 GiB compressed durable state

Using an illustrative S3 Standard rate of:

$0.023 / GB-month

the raw storage component is approximately:

0.5 * 0.023
= $0.0115/month

about one cent per month.

That is not a complete cloud bill. PUT/GET requests, transfer, replication, and the exact storage class and region all matter.

But the calculation exposes something useful:

For small per-agent memory stores, storage bytes are unlikely to be the expensive part.

The architectural costs are more likely to come from:

inference
network calls
database operations
operational complexity
recovery behavior

This is why a one-bucket architecture can be attractive.

Instead of paying continuously for:

database server
connection pool
database operations
persistent volumes
replication sidecars

you retain a local process and pay for durable state only at explicit boundaries.

The tradeoff is that you now own the lifecycle:

IAM
bucket policies
encryption
object naming
retention
flush policy
checkpoint policy

That is a much smaller operational surface than operating a database cluster, but it is still an engineering responsibility.

7. When should developers use Durable?

The cleanest decision rule is based on the shape of the workload.

Use plain local chDB when:

the state is disposable
or
the state never leaves one machine

Use a transactional store when:

you mainly need checkpoints
and
you do little historical analysis

Use a server database when:

many writers
many clients
centralized governance
large working sets
continuous shared access

The Durable Layer fits the middle:

one logical owner
+
local analytical queries
+
append-heavy history
+
portable durable state

A useful deployment might look like:

                   S3 / GCS / Azure
                         |
                 durable agent object
                         |
          +--------------+--------------+
          |                             |
       laptop                         CI runner
          |                             |
       chDB                          chDB
          |                             |
      local SQL                     local SQL
          |                             |
       recall                         recall

The database moves.

The query engine stays embedded.

The service layer disappears.

And the boundary between "memory" and "data engineering" becomes much thinner.

There is also a nice historical symmetry here.

ClickHouse itself began at Yandex under Alexey Milovidov and colleagues as an attempt to generate real-time analytical reports from continuously arriving, non-aggregated data. Its early production system eventually became the engine behind Yandex.Metrica.

Years later, the same broad philosophy is useful for agents:

keep the raw history
store it efficiently
query it later
derive answers from the history

The difference is that the data is now an agent's experiences rather than web analytics.

That may be one of the more useful ways to think about agent memory.

Memory is a database that happens to influence future decisions.

And once you view it that way, persistence, provenance, revision history, compression, query latency, leases, checkpoints, and recovery stop looking like infrastructure details. They become part of the agent's cognitive architecture.

The chDB Durable Layer is interesting because it gives developers a missing deployment primitive:

embedded analytical state
+
portable durability
+
object storage
+
no database server

The result is a local agent brain that can survive the machine that created it.

What do you think the right primitive for long-lived agent memory should be: a vector store, a transactional database, or an analytical database with durable local state?

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

⭐ Star it on GitHub:

HexmosTech / LiveReview

Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview: Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.
blast-radius-demo.mp4
LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.

The exact math, not a black box Visualize blast radius at a glance Every factor that feeds the score

How does Blast Radius scoring work? (a more technical explanation)
Here's the goal:

A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.

A 300-line UI change in one file, fully covered by…

View on GitHub

Click below to try LiveReview with your codebase:

OpenShell: Building a Security Boundary Around AI Agents

Shrijith Venkatramana — Tue, 29 Sep 2026 20:26:45 +0000

An AI coding agent with shell access has crossed a boundary that a chatbot never crossed.

A chatbot can suggest:

rm -rf /important/directory

An agent can actually run it.

It can read your repository, inspect environment variables, install packages, call external APIs, modify configuration, create files, launch subprocesses and keep doing all of this after you have stopped watching.

That changes the security problem.

The old question was:

“Can I trust this program?”

The more useful question for agents becomes:

“What happens when this program cannot be trusted for the next ten minutes?”

That is the problem NVIDIA OpenShell is trying to solve.

OpenShell is a runtime for autonomous AI agents. The important architectural idea is simple:

Put the security boundary outside the agent.

The agent gets a sandbox. The sandbox gets a policy. The operating system and runtime enforce that policy.

This article looks at why that distinction matters, how OpenShell works, what its policy model looks like, and where the economics and engineering trade-offs appear.

At the time of writing, the latest OpenShell release is 0.1.1, while NVIDIA's documentation still describes the software as alpha. So think of this as an architectural tour of an emerging runtime, not a production certification.

1. An AI agent is starting to look like a foreign process on your machine

Traditional software usually has a relatively stable authority model.

You install a compiler. The compiler can read your source tree.

You run a test suite. The test runner can access whatever the test process can access.

You start a server. The server has the permissions you gave its Unix user.

With an AI agent, the software is producing the next action dynamically.

A coding agent might do this:

read issue
    ↓
inspect repository
    ↓
search documentation
    ↓
install dependency
    ↓
run tests
    ↓
discover failure
    ↓
modify code
    ↓
run shell command
    ↓
call GitHub API
    ↓
spawn another process
    ↓
try another approach

The problem is not that any individual operation is unusual.

The problem is composition.

An agent can combine five individually reasonable permissions into a dangerous capability.

For example:

read ~/.config
+
read environment variables
+
execute Python
+
internet access
+
GitHub token
=
potential credential exfiltration

The model does not need to be malicious for this to matter.

A prompt injection buried inside a README can tell the agent to inspect a local file. A package installation step can execute arbitrary code. A tool response can contain instructions that compete with the original task.

This is exactly the sort of environment studied in AgentDojo, which evaluated tool-using agents against adversarial content. The benchmark contained 97 realistic tasks and 629 security test cases involving things such as email, banking and travel workflows.

The important lesson is architectural:

Prompt-level instructions are not a sufficient security boundary for a process that can act on the world.

That is why OpenShell is interesting.

It does not try to make the model perfectly trustworthy.

It assumes the opposite.

2. Sandboxing is an old operating-systems idea

OpenShell may sound like an AI-specific security invention, but its central idea is much older than neural networks.

In 1975, Jerome Saltzer and Michael Schroeder published The Protection of Information in Computer Systems. Among the principles they discussed were least privilege and fail-safe defaults.

The intuition is straightforward:

Give a program only the authority required for its job, and make access unavailable unless it has been explicitly granted.

The same philosophy appears in the browser.

When Google Chrome shipped in 2008, Adam Barth, Collin Jackson, Charlie Reis and the Chrome team described a design in which rendering happened in restricted processes while a more privileged browser process mediated access to the outside world.

The reason was practical.

If an attacker found a vulnerability in the renderer, arbitrary code execution inside the renderer should still have limited access to the machine.

That distinction is worth remembering:

Compromised process
        ≠
Compromised machine

OpenShell is applying a similar idea to agents.

You can think of it as:

AI agent
    |
    | requests
    v
sandbox boundary
    |
    +---- filesystem policy
    +---- process policy
    +---- network policy
    +---- credential policy
    |
    v
host / enterprise infrastructure

The agent is allowed to be wrong.

The security boundary is supposed to remain correct anyway.

This is a very different philosophy from:

System prompt:
"Never access secrets."

That is an instruction.

A sandbox policy is an enforcement mechanism.

3. OpenShell's mental model: the browser tab for agents

NVIDIA describes OpenShell using a useful analogy: the browser tab.

A browser tab is not allowed to behave like an arbitrary native process with unrestricted access to your computer.

It operates inside a bounded environment.

OpenShell tries to give an agent the same relationship with the host.

You can launch an agent such as Claude Code inside a sandbox:

openshell sandbox create -- claude

The architecture then separates several responsibilities.

There is a gateway, which acts as the control plane.

There is a sandbox, where the workload executes.

There is a supervisor, which lives on the trusted side of the workload boundary and mediates privileged operations.

The agent itself becomes a child process inside this environment.

Conceptually:

                     CONTROL PLANE
                          |
                       Gateway
                          |
                 policy / providers
                          |
              +-----------+-----------+
              |                       |
        Supervisor                Supervisor
              |                       |
        Sandbox A                 Sandbox B
              |                       |
        Claude Code              Codex
              |                       |
          processes              processes

The distinction between the supervisor and the agent is important.

The agent should not control the mechanism that controls the agent.

That is analogous to putting authentication checks outside the application instead of asking the application to certify that its own requests are safe.

This is one of the strongest parts of the design.

If the agent modifies its own source code, installs a new package, spawns a child process or becomes confused by prompt injection, those actions still encounter the external enforcement boundary.

OpenShell currently exposes several runtime backends, including Docker, Podman, Kubernetes and VM-backed execution. The exact isolation machinery varies, but the policy model is intended to remain stable.

4. The interesting part is the policy: an agent gets a capability graph

A useful way to think about an OpenShell policy is as a graph.

Let:

F = filesystem permissions
N = network destinations
P = executable/process identities
S = credential bindings

The agent's actual capability set is roughly a subset of:

F × N × P × S

This is not a formal security metric. It is a useful mental model.

Imagine an agent has:

3 executable identities
4 network endpoints
2 HTTP operation classes

That gives at most:

3 × 4 × 2 = 24

meaningful executable-to-endpoint-to-operation combinations to reason about.

With unrestricted outbound networking, the number of possible destinations becomes effectively unbounded from the policy's point of view.

That is the value of least privilege: reduce the graph before you have to reason about its behavior.

A simplified policy can look like this:

version: 1

filesystem_policy:
  include_workdir: true
  read_only:
    - /usr
    - /lib
    - /etc
  read_write:
    - /tmp

landlock:
  compatibility: best_effort

network_policies:
  github_api:
    name: github-api-readonly

    endpoints:
      - host: api.github.com
        port: 443
        protocol: rest
        enforcement: enforce
        access: read-only

    binaries:
      - path: /usr/bin/curl
      - path: /usr/bin/gh

This says something considerably stronger than:

"The agent should probably only use GitHub."

It says:

These binaries
    ↓
may connect to
    ↓
this endpoint
    ↓
using
    ↓
these permitted operations

OpenShell can go further and constrain HTTP methods and paths.

For example:

network_policies:

  github_repository_api:

    endpoints:
      - host: api.github.com
        port: 443
        protocol: rest
        enforcement: enforce

        rules:
          - allow:
              method: "*"
              path: "/repos/acme/project/**"

        deny_rules:
          - method: "*"
            path: "/repos/acme/project/hooks"

          - method: "*"
            path: "/repos/acme/project/hooks/**"

    binaries:
      - path: /usr/bin/gh

Now you are describing something closer to an application firewall:

binary
+
host
+
port
+
protocol
+
HTTP method
+
path

This matters because an agent does not merely need “Internet access.”

It often needs one very specific network action.

For example:

GET /repos/acme/project/issues

is a very different capability from:

POST /repos/acme/project/hooks

The second might enable an entirely different class of side effect.

This is where agent security starts looking less like prompt engineering and more like operating-system protection.

5. Under the hood, OpenShell combines several old security mechanisms

The interesting technical detail is that OpenShell does not depend on a single sandbox primitive.

It layers multiple controls.

Filesystem: Landlock

OpenShell uses Linux Landlock to constrain filesystem access.

Conceptually:

read:
    /usr
    /lib
    /etc

read/write:
    /sandbox
    /tmp

everything else:
    deny

This is implemented at the kernel level.

That matters because the agent cannot simply decide:

"Ignore the policy."

and turn it off from inside the process.

The current implementation distinguishes between static filesystem/process controls and dynamic network controls. Filesystem restrictions are established when the sandbox starts, while network policy can be changed while the sandbox is running.

Processes: reduced privilege

The workload runs as an unprivileged identity with reduced capabilities.

OpenShell's architecture explicitly aims to avoid giving the agent Linux capabilities inside the workload.

The point is defense in depth.

Suppose the agent contains a vulnerability.

Suppose the agent executes malicious code.

Suppose a dependency behaves badly.

The resulting process still has a restricted operating-system authority.

Network: deny first

This may be the most important practical control.

Outbound networking starts from a deny-by-default posture.

Connections are mediated through the OpenShell network path, where policy can inspect the destination, calling executable and, for supported protocols, higher-level HTTP operations.

That allows a policy like:

/usr/bin/gh
    -> api.github.com:443
    -> GET
    -> /repos/acme/project/**

while denying:

/usr/bin/gh
    -> random-attacker.example

and potentially:

/usr/bin/gh
    -> api.github.com
    -> POST /hooks

The current architecture also tracks executable identity using the executable's observed path and SHA-256 digest. That closes an interesting hole:

allowed binary
    ↓
replace binary
    ↓
same path

A path alone is weaker than:

path + executable identity

because otherwise an allowed executable could potentially be replaced by another program.

Credentials: give the agent access without handing it the secret

This is particularly relevant for coding agents.

Suppose your agent needs a GitHub token.

The naive approach is:

export GITHUB_TOKEN=...
claude

Now the process has the secret.

Any code it executes may potentially inspect it.

OpenShell instead has a provider model in which credentials can be stored outside the sandbox and bound to authorized endpoints.

Conceptually:

Agent
  |
  | "call GitHub"
  v
OpenShell policy
  |
  | allowed?
  v
credential binding
  |
  v
GitHub

The agent can get the capability without necessarily getting the raw credential as ordinary application data.

This is a much better abstraction for autonomous software.

The question becomes:

“Which service may this workload access?”

rather than:

“How do I smuggle the password into this process?”

6. The operational challenge is allowing autonomy without turning policy into babysitting

There is an uncomfortable tension here.

A very restrictive agent may be safe but useless.

Imagine an agent starts working and discovers it needs:

registry.npmjs.org

but the network is denied.

You have two choices.

The old approach is:

developer notices failure
    ↓
opens policy file
    ↓
adds endpoint
    ↓
restarts things
    ↓
agent continues

That becomes expensive very quickly.

OpenShell has a policy advisor workflow that turns a denied request into a proposed policy change.

The important property is that the agent can propose the permission, but the proposal does not automatically become policy.

Conceptually:

agent request
    ↓
DENY
    ↓
agent explains:
"I need registry.npmjs.org"
    ↓
policy proposal
    ↓
validation / review
    ↓
APPROVE
    ↓
live policy update
    ↓
retry

This preserves a useful separation:

agent proposes
human authorizes
runtime enforces

OpenShell also includes a policy prover. The current implementation models policy, credentials and binary capabilities and uses formal reasoning to identify changes such as newly reachable credentialed destinations.

That introduces an interesting idea:

security policy itself becomes something you can analyze mechanically.

You can think of the prover as asking questions such as:

Did this change introduce a new credentialed path?

Did this change introduce a new capability?

Did this change make a previously inaccessible endpoint reachable?

This is much more tractable than trying to prove that an LLM will never make a bad decision.

A small economics calculation

Suppose an agent generates:

20 permission requests/day

and each one takes:

20 seconds

of human attention.

Then:

20 × 20 sec = 400 sec
             ≈ 6.7 min/day

For one agent, that is manageable.

Now suppose you operate 100 agents:

100 × 20 × 20 sec
= 40,000 sec/day
≈ 11.1 hours/day

The exact numbers are hypothetical, but the operational principle is real:

the cost of agent security depends heavily on how often humans must intervene.

A policy system therefore has to optimize for two things at once:

minimal authority
+
minimal interruption

That is why live policy updates, narrow proposals, deterministic checks and audit logs are more interesting than simply putting the agent in Docker and stopping there.

Auditability matters too

OpenShell records structured security events for things such as:

allowed network access
denied network access
policy changes
security findings

That gives operations teams a way to answer:

What did the agent try to do?

What did the runtime allow?

What did the runtime deny?

Which policy allowed it?

Which executable made the request?

For autonomous systems, this starts looking like observability for authority rather than merely observability for performance.

A distributed system has traces for:

request A -> service B -> database C

An agent runtime needs something similar for:

agent -> binary -> endpoint -> credential -> action

7. What OpenShell does not solve

The sandbox boundary is powerful precisely because it has a clear scope.

It also has clear limits.

Suppose you allow:

agent -> api.github.com

The sandbox cannot infer whether the agent should create a repository release.

That is an application-level authorization problem.

Similarly, if you allow:

agent -> your production API

then the sandbox cannot magically determine whether:

DELETE /customers/123

was semantically appropriate.

There are at least four different layers of security:

1. Model behavior
2. Agent/application behavior
3. Runtime containment
4. Infrastructure authorization

OpenShell mainly strengthens layer 3 and connects it to parts of layers 2 and 4.

It does not make the model trustworthy.

It does not eliminate prompt injection.

It does not eliminate supply-chain attacks.

It does not guarantee that an allowed API call is logically correct.

It does not make a kernel vulnerability disappear.

And a sandbox is only as good as the isolation boundary beneath it.

This is another lesson from decades of systems security: containment reduces the consequences of failure; it does not make failure impossible.

The browser analogy is useful again.

Chrome's sandbox did not make browsers bug-free.

It changed the consequence of a renderer compromise.

That is the right way to think about an agent sandbox.

The agent may still go wrong.

The question is whether “the agent went wrong” means:

bad code in /tmp

or:

production database deleted
AWS credentials exfiltrated
private repositories copied

Those are very different failure domains.

The larger idea

AI agents are slowly turning software from:

programs that execute instructions

into:

programs that continuously choose instructions

Once that happens, the operating system becomes relevant again.

The shell becomes relevant.

Permissions become relevant.

Process identity becomes relevant.

Credential boundaries become relevant.

Audit logs become relevant.

Least privilege becomes relevant.

In other words, many of the ideas that operating-systems and security engineers have worked on for decades suddenly become central to AI engineering.

OpenShell is interesting because it treats the agent as a potentially untrusted process and asks a very old systems question:

What authority does this process actually need?

That is a much more scalable question than asking the model to behave itself.

For a developer, the mental shift is simple:

Before:

AI agent
   |
   +---- full machine
   +---- full filesystem
   +---- full network
   +---- credentials

After:

AI agent
   |
   v
OpenShell sandbox
   |
   +---- specific files
   +---- specific binaries
   +---- specific endpoints
   +---- specific API operations
   +---- specific credentials

The real promise of agentic software may therefore depend less on making agents perfectly aligned and more on making them containable when they are wrong.

And that is an old engineering trick.

We learned it from browsers.

We learned it from operating systems.

We are now applying it to machines that can reason about what to do next.

Conclusion

The deepest idea behind OpenShell is not Docker, Landlock, seccomp or YAML.

It is the decision to move trust outside the agent.

Once an autonomous system can execute arbitrary code, persistent memory, external tools and network calls, “please behave” stops being a sufficient security architecture.

A better model is:

Give the agent enough authority to be useful.

Put the authority behind a boundary it cannot rewrite.

Make that boundary explicit.

Make changes reviewable.

Make violations observable.

That is the basic architecture required when software starts acting more like an autonomous operator than a traditional program.

The interesting question for developers is no longer whether agents will get shell access.

They already are.

The interesting question is: what is the smallest world you are willing to let an agent operate in?

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

⭐ Star it on GitHub:

HexmosTech / LiveReview

Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview: Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.
blast-radius-demo.mp4
LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.

The exact math, not a black box Visualize blast radius at a glance Every factor that feeds the score

How does Blast Radius scoring work? (a more technical explanation)
Here's the goal:

A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.

A 300-line UI change in one file, fully covered by…

View on GitHub

Click below to try LiveReview with your codebase:

Gisting: Compressing LLM Agent Context

Shrijith Venkatramana — Mon, 28 Sep 2026 20:10:17 +0000

An AI agent can remember too much.

That sounds like a strange problem. More context should mean more information, so more context should mean a better agent.

In practice, the opposite can happen.

A coding agent may start with a system prompt, repository conventions, an issue, a few files, tool outputs, test failures, and then accumulate dozens of turns of its own reasoning. By turn 30, the agent may be carrying 30,000 tokens of history to answer a question that depends on 2,000 of them.

The engineering problem is therefore:

How do we preserve the useful state of a long context while throwing away most of the tokens?

One answer is gisting.

Gisting treats a long context as something that can be compressed into a much smaller representation and then reused.

The idea has an interesting history. In 2023, Jesse Mu, Xiang Lisa Li, and Noah Goodman published Learning to Compress Prompts with Gist Tokens at NeurIPS. Their motivating example was almost mundane: a fixed instruction prompt gets encoded again and again, even though its meaning has not changed. Their solution was to train the model itself to turn the prompt into a few internal "gist" tokens.

For agents, this idea becomes much more interesting.

1. The real problem is context, not memory

Consider a coding agent working on this task:

Fix the authentication bug in the payments service.

The agent might accumulate:

System instructions                         1,500 tokens
Repository instructions                     2,000
Issue description                             800
Relevant source files                       4,500
Tool output                                 3,000
Turn 1                                       1,200
Turn 2                                       1,800
Turn 3                                       2,100
Turn 4                                       2,400
...
Turn 15                                      2,000

Eventually:

Total context ~= 25,000 tokens

Yet much of that history is redundant.

The agent does not need to remember:

"I opened auth.py."

"I then searched for validate_token."

"Then I inspected line 84."

"Then I ran pytest."

What it really needs is something closer to:

Relevant state:
- Authentication uses JWT tokens.
- Token validation happens in auth.py:84.
- The failure occurs when expired tokens are refreshed.
- The refresh path bypasses validate_token().
- tests/test_auth.py reproduces the bug.
- A patch was attempted but failed because refresh_token() expects
  a normalized subject claim.

The first representation is history.

The second is state.

Gisting is the attempt to move from the first representation toward the second.

This is a useful way to think about agent context:

Raw history
    |
    v
Compression
    |
    v
Compact representation
    |
    v
Future reasoning

The goal is not to preserve every word.

The goal is to preserve the information required for future computation.

2. What gisting actually is

There are two ideas that are often mixed together.

Text summarization

A normal agent might ask an LLM:

Summarize the previous 20 turns in 800 tokens.

The result is ordinary natural language:

The user is debugging JWT refresh behavior...

This can be passed to almost any LLM.

Learned gist tokens

The original gisting paper does something different.

Suppose the original prompt is:

t = [t1, t2, ..., tn]

The model inserts a small number of special tokens:

t1 t2 ... tn <G1> <G2> ... <Gk>

The Transformer is trained with an attention mask that forces later tokens to access the earlier prompt only through the gist tokens.

Conceptually:

Original prompt
      |
      v
[G1] [G2] [G3] [G4]
      |
      v
Future input

The gist tokens therefore become a bottleneck.

The model has to learn:

"What information from this prompt must survive so that I can perform the task later?"

The important detail is that these are internal model representations, not miniature English summaries.

The paper trained LLaMA-7B and FLAN-T5-XXL models and reported prompt compression of up to 26x, with reductions in FLOPs and storage while retaining similar output quality on its evaluations.

This is closer to learned memory than to summarization.

3. Why the attention mask matters

This is the clever part of the original method.

Without any restriction, the model can cheat.

Imagine:

Prompt:
"Translate the following text into French."

Gist tokens:
<G1> <G2>

Input:
"The cat is sleeping."

If the later input can still directly attend to the original prompt, the model has no reason to encode anything into <G1> and <G2>.

So gisting changes the attention pattern.

Normally, later tokens can attend to previous tokens:

Prompt ----------------------> Future tokens
       \                     /
        \                   /
         Gist tokens ------>

With gist masking:

Prompt ---> Gist tokens ---> Future tokens
                 ^
                 |
        compressed information

The future part cannot directly look back at the original prompt.

That creates a training pressure:

If the future needs information,
the information must pass through the gist.

This is essentially a learned information bottleneck.

The authors point out that the masking change is very small at the implementation level: the model architecture does not need to become a new neural architecture. The attention pattern is changed so the model learns the compression behavior during instruction tuning.

That distinction matters.

Gisting is not:

summary = model(prompt)

It is:

gist = learned_compressor(prompt)
output = model(gist, new_input)

where gist is represented inside the Transformer's activation space.

4. The economics are simple

The reason engineers should care about this is mathematical.

For a standard Transformer, self-attention over n input tokens has roughly:

O(n^2)

attention complexity during the prefill phase.

During generation, cached keys and values change the picture. Each newly generated token still needs to attend across the existing context, so the attention work is roughly proportional to:

O(n)

per generated token.

That means an agent with a huge context pays repeatedly for the same historical information.

Suppose:

Original context = 12,000 tokens
Compressed context = 1,000 tokens
Future generation = 300 tokens
Number of turns = 20

Ignoring constants and other model costs, the context-attention work associated with generation is approximately:

Full history:

12,000 * 300 * 20
= 72,000,000

Compressed:

1,000 * 300 * 20
= 6,000,000

So the difference is:

66,000,000

context-token attention interactions.

That is a 12x reduction in the context length seen by every generated token.

There is a catch: compression itself costs compute.

So gisting only makes economic sense when the compressed representation is reused enough times.

Let:

Ccompress = cost of creating the gist

Cfull = cost of processing the full context per future turn

Cgist = cost of processing the compressed context per future turn

R = number of future reuses

Gisting pays off when:

Ccompress + R * Cgist < R * Cfull

or:

R > Ccompress / (Cfull - Cgist)

This gives an important systems principle:

Compression is an amortization problem.

A 20-token system prompt used once does not need elaborate compression.

A 20,000-token agent history reused for another 40 turns is a different engineering problem.

The same logic applies to token billing.

Suppose compression reduces each future request by 11,000 input tokens and the agent makes 20 requests:

11,000 * 20
= 220,000 input tokens saved

At an input price of P dollars per million tokens:

Savings = 0.22 * P dollars

The exact dollar value depends on the model and pricing, but the scaling argument does not.

5. Where this becomes useful for agents

An agent has several different kinds of context.

They should not all be compressed in the same way.

Stable instructions

Examples:

Coding conventions
Security policies
Tool usage rules
Output format
Repository architecture rules

These are excellent candidates for caching or learned compression because they are reused repeatedly.

Episodic history

Examples:

Previous tool calls
Previous failed approaches
Old intermediate reasoning
Past conversation turns

This is where context gisting becomes useful.

The agent can periodically replace:

Turn 1
Turn 2
Turn 3
...
Turn 18

with:

Agent state as of turn 18

Exact evidence

This category is different:

Database rows
Source-code lines
Exception messages
API responses
Financial values
Configuration values

Compressing these aggressively can be dangerous.

An agent may need the exact string:

"expected audience=payments-api"

A summary such as:

"The JWT audience configuration was incorrect."

is semantically close but operationally weaker.

This suggests a useful architecture:

             Agent Context
                  |
       +----------+----------+
       |          |          |
    Stable      Episodic    Exact
    context      state      evidence
       |          |          |
    Cache       Gist/       Retrieve
                 summary     on demand

In other words, gisting should usually reduce the working context, not become the only copy of the information.

For a coding agent, a sensible system might look like:

                    +----------------+
                    | Agent          |
                    +-------+--------+
                            |
             +--------------+--------------+
             |              |              |
             v              v              v
        System rules    Agent gist     Retrieved code
             |              |              |
             +--------------+--------------+
                            |
                            v
                       LLM context

The gist remembers the trajectory.

Retrieval restores exact facts.

The current user request provides the immediate objective.

6. The compression bottleneck can destroy information

There is a fundamental problem with any lossy context compression:

The compressor does not know what the future question will be.

Suppose an agent summarizes a debugging session:

We fixed the OAuth callback bug.

That sounds useful.

Three turns later the user asks:

Which exact redirect URI caused the failure?

The answer may already have disappeared from the compressed representation.

This is why context compression behaves differently from ordinary text summarization.

A summary can preserve semantic meaning while destroying future utility.

Recent evaluations of gist-token compression found several failure patterns, including information being lost at segment boundaries, information that is locally surprising being dropped, and information being degraded across multiple compression stages. The 2025 ACL study also found that gist-based compression can perform well on some retrieval and long-document tasks while struggling on tasks requiring precise recall.

This leads to an important engineering rule:

Do not ask:

"What is important?"

Ask:

"What information might future computation require?"

Those are different questions.

For an agent, one practical response is to preserve certain classes of information explicitly:

Goals
Decisions
Constraints
Open questions
Identifiers
Errors
Exact values
Files changed
Commands executed
Artifacts produced

Then allow everything else to enter the compressed state.

A useful pattern is:

Gist:
    high-level state and reasoning

Retrieval store:
    exact historical evidence

Current context:
    current task and retrieved evidence

That is much safer than:

100,000 tokens
       |
       v
800-token summary
       |
       v
delete everything

7. Gisting changes the architecture of an agent

Once context is treated as a compressible state, an agent starts to look less like a chatbot and more like a state machine.

Instead of:

history = history + new_turn

you can think in terms of:

state_(t+1) = compress(state_t, observation_t)

and later:

action_t = model(state_t, current_observation_t)

This is a much more scalable abstraction.

For example:

             ┌──────────────┐
             │ Current task │
             └──────┬───────┘
                    |
                    v
             ┌──────────────┐
             │ Agent model  │
             └──────┬───────┘
                    |
                    v
             ┌──────────────┐
             │ Tool actions │
             └──────┬───────┘
                    |
                    v
             ┌──────────────┐
             │ Observations │
             └──────┬───────┘
                    |
                    v
             ┌──────────────┐
             │   Gisting    │
             └──────┬───────┘
                    |
                    v
             ┌──────────────┐
             │ Compact state│
             └──────────────┘
                    |
                    └──────> next turn

This is the deeper idea behind gisting.

The agent does not need to carry its entire past forward.

It needs to carry forward a representation of its past that remains useful for future actions.

That is a very different mental model from simply "increase the context window."

There is also an important practical limitation: the original learned gist-token approach produces model-specific internal representations. An application using a closed LLM API generally cannot take those hidden activations and feed them back later as arbitrary input. For API-based systems, natural-language compression methods such as prompt compression are therefore often easier to deploy, while true gist-token methods are more natural when you control the model and inference stack. Work on long-context compression has also shown that gist-style methods can degrade as the context becomes longer, so compression ratio alone is not a sufficient metric.

The interesting engineering question is therefore not:

"How large should the context window be?"

It is:

"What is the smallest representation of the agent's past that preserves the information needed for future computation?"

That is the problem gisting puts on the table.

And once agents become long-running systems, that problem starts to look less like prompt engineering and more like designing a memory hierarchy.

Conclusion

LLM agents are often built around an implicit assumption:

More history = more intelligence

A better model may be:

More relevant state = more intelligence

Gisting is one attempt to make that transition.

It treats context as a representation that can be compressed, cached, and reused rather than as an ever-growing transcript.

The next generation of agents may therefore have something resembling:

working memory
episodic memory
compressed state
retrievable evidence

rather than one giant conversation buffer.

That raises an interesting question for anyone building agents:

What information should an agent be allowed to forget?

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

⭐ Star it on GitHub:

HexmosTech / LiveReview

Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview: Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.
blast-radius-demo.mp4
LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.

The exact math, not a black box Visualize blast radius at a glance Every factor that feeds the score

How does Blast Radius scoring work? (a more technical explanation)
Here's the goal:

A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.

A 300-line UI change in one file, fully covered by…

View on GitHub

Click below to try LiveReview with your codebase:

Hard Negative Mining: Teaching an LLM What "Almost Right" Looks Like

Shrijith Venkatramana — Sun, 27 Sep 2026 19:48:07 +0000

A model learns surprisingly little from examples that are obviously wrong.

Suppose you are training a retrieval model with:

Query: "How do I rotate an AWS IAM access key?"

And your negative example is:

"How do I resize a Kubernetes pod?"

The model can separate these easily. One is about IAM credentials; the other is about Kubernetes.

Now give it:

"How do I create an AWS IAM access key?"

That is a much more interesting negative.

It uses the same vocabulary. It concerns the same object. It may even appear in the right neighborhood of the embedding space.

But it answers a different question.

This is the basic idea behind hard negative mining: instead of teaching a model merely what is wrong, deliberately show it things that are almost right and force it to learn the distinction.

The technique became important in computer vision with work such as FaceNet, and later became a major component of dense text retrieval. The interesting part for LLM developers is that the same idea now appears in RAG, semantic search, reranking, embedding training, preference data, and other systems where the model must distinguish between closely related alternatives. (CV Foundation)

1. Easy negatives are cheap—and often useless

Imagine you have a query q, a relevant document d+, and an irrelevant document d-.

A contrastive training objective wants:

similarity(q, d+) > similarity(q, d-)

An easy negative might already be far away:

similarity(q, d+) = 0.82
similarity(q, d-) = 0.12

There is little ambiguity.

The model already knows that the two things are different.

A hard negative could look like:

similarity(q, d+) = 0.82
similarity(q, d-) = 0.76

Now the model has a real problem.

Why does this matter?

Because retrieval systems rarely fail by retrieving something completely unrelated.

They fail by retrieving:

the wrong version of a library
the wrong API endpoint
a document about installation instead of configuration
a description of creating something instead of deleting it
a closely related bug with a different root cause

In production, the enemy is usually semantic confusion, not semantic ignorance.

Hard negative mining turns that confusion into training data.

2. The idea predates LLMs

The underlying idea is older than language models.

In 2015, Florian Schroff, Dmitry Kalenichenko and James Philbin published FaceNet. They trained a neural network to map faces into an embedding space where images of the same person were close together and different people were far apart.

The important engineering problem was choosing which examples to train on.

Suppose you have:

anchor   = Alice's face
positive = another image of Alice
negative = Bob's face

A completely different-looking Bob is not particularly useful.

The interesting negative is a Bob who looks sufficiently similar to Alice to make the distinction difficult.

FaceNet therefore used online triplet mining, progressively selecting difficult examples during training. The paper describes this as a way of increasing the difficulty of the triplets as the network improves. (CV Foundation)

A few years later, the same logic became important in text retrieval.

Karpukhin and colleagues' 2020 Dense Passage Retrieval work showed that a relatively simple dual-encoder could produce effective dense retrieval for open-domain question answering. But there was a problem: what you use as a negative during training determines what distinctions the retriever learns. (ACL Anthology)

Then came ANCE from Microsoft researchers including Lee Xiong and Chenyan Xiong.

Their observation was particularly important:

The negatives used during training often did not resemble the documents the model would actually confuse during inference.

Their solution was to use an approximate-nearest-neighbor index to retrieve difficult negatives using the model itself, and periodically refresh that index as the model changed. In other words, the model was used to discover the mistakes that the next version of the model needed to learn from. (arXiv)

That creates an interesting feedback loop:

train model
    |
    v
model retrieves difficult documents
    |
    v
use those as negatives
    |
    v
train model again
    |
    v
model discovers new difficult documents

Hard negative mining is therefore partly a data-generation strategy and partly an optimization strategy.

3. Where hard negatives appear in an LLM system

You do not usually "hard-negative mine an LLM" in the same sense that you mine negatives for a classifier.

The technique is most useful when an LLM system has to rank, retrieve, compare, or distinguish things.

Consider a typical RAG pipeline:

user query
    |
    v
embedding model
    |
    v
vector database
    |
    v
top 20 passages
    |
    v
reranker
    |
    v
top 5 passages
    |
    v
LLM

There are several places where hard negatives matter.

Embedding models

You train:

query -> relevant passage

and contrast it against:

query -> similar but irrelevant passage

This teaches the embedding space to preserve the distinctions your application actually cares about.

Rerankers

Suppose the retriever returns:

1. PostgreSQL deadlock troubleshooting
2. PostgreSQL transaction rollback
3. PostgreSQL isolation levels
4. PostgreSQL connection pooling

A reranker needs to understand why #1 answers the query while #3 is merely related.

That makes #2 and #3 substantially more useful negatives than an article about Kubernetes.

Recommendation and semantic matching

The same pattern appears in:

query -> document
user -> product
question -> answer
bug report -> solution
code change -> relevant documentation

Whenever the challenge is distinguishing nearby alternatives, hard negatives are valuable.

This is also why systems such as RocketQA explicitly incorporated denoised hard negatives alongside cross-batch negatives and data augmentation when training dense passage retrievers. (ACL Anthology)

4. How do you actually mine a hard negative?

The simplest method is almost embarrassingly straightforward.

Take your current model and ask:

"What documents does this model think are relevant?"

Suppose your training example is:

q  = "How do I configure PostgreSQL connection pooling?"
d+ = "PgBouncer configuration guide"

Run the query through your retriever.

It produces:

1. PgBouncer configuration guide       0.91
2. PostgreSQL connection pooling       0.89
3. PostgreSQL connection management    0.87
4. PostgreSQL authentication           0.71
5. PostgreSQL installation             0.62

After removing the known positive, candidates #2 and #3 are excellent places to look for hard negatives.

A basic mining procedure is:

for each (q, d+):

    candidates = retrieve(q, corpus, top_k=100)

    candidates = remove(candidates, d+)

    hard_negatives = top candidates

But there is a major trap.

Hard does not mean wrong

A document can be highly similar because it is actually relevant.

That gives you a false negative.

For example:

Query:
"How do I rotate an AWS access key?"

Positive:
"Rotating IAM access keys"

Candidate:
"IAM access key security best practices"

You might classify the candidate as irrelevant simply because it does not contain the exact answer.

But it may still be legitimately useful.

Training the model to push it away could damage the retrieval space.

This is one of the reasons RocketQA introduced denoising for hard negatives rather than simply taking the nearest documents blindly. Later work on dense retrieval has continued to treat false-negative removal as an important part of hard-negative construction. (ACL Anthology)

A practical production pipeline therefore looks more like:

BM25 / embedding retrieval
          |
          v
    100 candidates
          |
          v
remove known positives
          |
          v
cross-encoder / LLM judge
          |
          v
remove probable false negatives
          |
          v
     hard negatives

This is considerably more useful than simply selecting the nearest vectors.

5. The math: why hard negatives produce more gradient

Consider a simple contrastive objective.

For query q, positive d+, and negatives d1 ... dn:

L = -log(
      exp(sim(q,d+) / T)
      --------------------------------
      exp(sim(q,d+) / T) +
      sum_i exp(sim(q,di) / T)
    )

where:

sim(q,d) = similarity between query and document
T        = temperature

You do not need to memorize the equation.

Think of it as a competition.

The model gets rewarded when the positive takes most of the probability mass.

Suppose:

positive = 0.90
negative A = 0.20
negative B = 0.10
negative C = 0.05

The positive is already winning comfortably.

Now consider:

positive = 0.90
negative A = 0.87
negative B = 0.25
negative C = 0.10

Negative A is creating a much larger training signal.

The model must figure out:

"What feature distinguishes these two things?"

That is exactly the behavior we want.

There is a useful geometric interpretation.

Imagine the embedding space as a map:

                       unrelated
                          *
                    *
              *             *

        positive *
                  \

                   * hard negative

Easy negatives teach the model about broad regions.

Hard negatives teach it about the decision boundary.

That distinction becomes especially important as the model gets better.

Early in training:

easy negatives -> useful

Later:

easy negatives -> mostly redundant
hard negatives -> increasingly valuable

This is the same basic intuition behind ANCE's dynamic mining: use the evolving model to find examples near its current retrieval boundary. (arXiv)

6. The operational economics are a little counterintuitive

Hard negative mining costs compute.

Suppose you have:

1,000,000 training queries

and you want:

20 hard negatives / query

You now have roughly:

20,000,000 candidate relationships

The expensive part is often not training itself.

It is finding and validating those negatives.

There are several ways to control the cost.

Start with cheap retrieval

Use BM25 or an existing embedding model to retrieve:

top 100

Then only send a small candidate set to a more expensive judge.

You do not need an LLM to examine one million documents per query.

Mine periodically

You also rarely need to regenerate the negatives after every optimization step.

For example:

train
 -> mine
 -> train
 -> mine
 -> train

rather than:

train
 -> mine
 -> train
 -> mine
 -> train
 -> mine

ANCE is an interesting example of this engineering compromise: it uses an asynchronously refreshed approximate-nearest-neighbor index instead of rebuilding everything synchronously after every parameter update. (arXiv)

Use a teacher to manufacture useful mistakes

Suppose an expensive cross-encoder can score a candidate at:

0.97 relevant
0.84 relevant
0.31 relevant
0.02 relevant

The useful region may be around:

0.4 - 0.8

The 0.02 example teaches almost nothing.

The 0.97 example may actually be another positive.

The middle examples are where the information density is.

So the goal is not:

"Find the hardest possible negative."

It is:

Find negatives that are difficult enough to produce useful learning without becoming mislabeled positives.

That is a much better engineering objective.

7. A practical recipe for developers

Suppose you are building a retrieval system for a company's internal engineering documentation.

You have:

10,000 queries
50,000 documents

Start with ordinary positive examples:

(query, relevant_document)

Then:

Step 1: Retrieve candidates

For every query:

top 50 using BM25
top 50 using embeddings

Union them.

Step 2: Remove obvious positives

Remove the labeled document and documents known to answer the same question.

Step 3: Score candidates

Use a cross-encoder or sufficiently capable LLM judge.

Ask something like:

Given the query and document,
does this document directly answer the query?

Return:
RELEVANT
IRRELEVANT

Step 4: Keep the confusing ones

Construct something like:

positive:
"How to rotate PostgreSQL credentials"

hard negative:
"How to rotate PostgreSQL TLS certificates"

hard negative:
"How to reset PostgreSQL passwords"

easy negative:
"How to install PostgreSQL"

Step 5: Train

Use the positive and selected negatives in your contrastive objective.

Step 6: Re-mine

After the model improves, ask it again what it gets confused by.

The old hard negatives may become easy.

New hard negatives will emerge.

That gives you a virtuous cycle:

better model
    ->
better mistakes
    ->
better training data
    ->
better model

And that is the deeper lesson of hard negative mining.

You are not merely collecting more examples.

You are collecting the examples that expose the current limits of the model.

For LLM systems, that can be more valuable than adding another 10 million random training examples.

The interesting question is therefore:

When building your next RAG or retrieval system, would you rather collect more data—or deliberately collect the mistakes your current model is already making?

References

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

⭐ Star it on GitHub:

HexmosTech / LiveReview

Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview: Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.
blast-radius-demo.mp4
LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.

The exact math, not a black box Visualize blast radius at a glance Every factor that feeds the score

How does Blast Radius scoring work? (a more technical explanation)
Here's the goal:

A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.

A 300-line UI change in one file, fully covered by…

View on GitHub

Click below to try LiveReview with your codebase:

Using MTP to Accelerate RL Training of LLMs

Shrijith Venkatramana — Sat, 26 Sep 2026 20:27:10 +0000

A modern reasoning LLM may spend most of its training time doing something that looks almost trivial:

Generate token.

Generate token.

...

Then score the trajectory.

Then update the model.

The optimizer is doing serious work, the GPU cluster is expensive, and the environment may be running thousands of parallel tasks.

Yet the rollout stage can still reduce to a long chain of sequential token-generation steps.

This is where Multi-Token Prediction (MTP) becomes interesting.

MTP is usually introduced as a training objective: instead of predicting only the next token, train the model to predict several future tokens.

But there is a second interpretation that matters for reinforcement learning:

MTP can turn the model into its own small draft model.

That means a reasoning model can propose several tokens cheaply, while the full model verifies them in parallel.

For RL, where generating trajectories is often the dominant cost, this changes the economics of the training loop.

The recent MiMo-V2-Flash technical report from Xiaomi's LLM-Core team makes this connection explicit, describing MTP as a way to accelerate RL rollouts, improve utilization with small batches, and reduce the cost of long-tail trajectories. Their final model uses three lightweight MTP layers and reports an acceptance length of up to 3.6 tokens and decoding speedups of up to 2.6x in their evaluation setup.

Let's unpack how this works.

1. The RL training loop has an awkward bottleneck

Consider a simplified RL setup for mathematical reasoning.

You give the model:

Solve:

If 3x + 7 = 22, what is x?

The model generates:

We need to isolate x.
3x = 22 - 7
3x = 15
x = 5

A verifier checks the answer.

If the answer is correct, the trajectory gets a positive reward.

The trainer then updates the policy.

At scale, the loop looks roughly like this:

prompt
  |
  v
rollout / generation
  |
  v
environment / verifier
  |
  v
reward
  |
  v
policy update
  |
  +-------> next rollout

The problem is that rollout generation is autoregressive.

To produce token t+1, the model needs token t.

So generating 1,000 tokens is approximately 1,000 sequential decoding steps.

You can add more prompts to a batch, but there is a limit. Reasoning trajectories are highly variable in length.

One sample might finish in 200 tokens.

Another might continue for 8,000.

Eventually you get the classic distributed-systems problem:

GPU 1:  done
GPU 2:  done
GPU 3:  still generating...
GPU 4:  still generating...
GPU 5:  done

       ^ idle GPUs ^

The MiMo-V2-Flash report explicitly calls out this long-tail behavior. In small-batch RL, trajectories can approach batch size 1, leaving substantial GPU capacity unused.

This creates an uncomfortable situation:

You may have enough aggregate compute.

What you lack is enough parallel work at each moment.

2. MTP: make the model predict several tokens at once

The usual language-model objective is next-token prediction.

Given:

x1, x2, x3, ..., xt

the model predicts:

P(x[t+1] | x[1:t])

With MTP, the model also learns to predict future tokens:

P(x[t+1] | x[1:t])
P(x[t+2] | x[1:t])
P(x[t+3] | x[1:t])
...

The important architectural idea is that these predictions can share the expensive transformer trunk.

A useful mental model is:

                 +--> token t+1
hidden state ----+--> token t+2
                 +--> token t+3
                 +--> token t+4

Instead of running four completely independent copies of the model, you reuse the expensive computation that produced the hidden state.

This idea was studied systematically by Fabian Gloeckle and colleagues in 2024. They trained models with multiple prediction heads and found both training-quality improvements and substantial inference acceleration. Their 13B models showed higher HumanEval and MBPP performance than comparable next-token models, while models trained for 4-token prediction were reported to run up to 3x faster at inference in their experiments.

This gives MTP two possible roles:

MTP during training
    |
    +--> richer training objective

MTP during inference
    |
    +--> draft several future tokens

The second role is what becomes useful for RL.

3. MTP becomes speculative decoding

The connection becomes clearer if we look at speculative decoding.

In 2022, Yaniv Leviathan, Matan Kalman, and Yossi Matias introduced speculative decoding as a way to accelerate autoregressive generation.

The basic trick is simple:

Instead of asking the expensive model for one token:

big model -> token 1
big model -> token 2
big model -> token 3
big model -> token 4

use a cheap model to propose several:

small model -> token 1, token 2, token 3, token 4

Then ask the big model to verify them together.

If the proposed tokens are correct, several autoregressive steps collapse into one expensive verification pass.

The key observation in the original work was that difficult generation contains local sequences that are comparatively easy to predict. Speculative decoding exploits that redundancy. The authors demonstrated 2x-3x acceleration on T5-XXL while preserving the output distribution.

Now replace the separate draft model with MTP heads attached to the same model.

You get:

                    +--> draft token 1
                    +--> draft token 2
main model state ---+--> draft token 3
                    +--> draft token 4
                              |
                              v
                       main model verifies
                              |
                              v
                         accepted tokens

This is attractive because the draft model is already inside the model.

You do not need:

large policy model
        +
separate small draft model
        +
synchronization
        +
two model deployments

You can instead have:

large policy model
        +
lightweight MTP module

That is the architecture Xiaomi uses in MiMo-V2-Flash.

Their MTP modules deliberately use a lightweight dense FFN and sliding-window attention rather than duplicating the expensive MoE/global-attention machinery of the main network. Each MTP block is about 0.33B parameters, while the full MiMo-V2-Flash model has 309B total parameters and 15B active parameters per token.

The engineering principle is important:

The draft computation must be cheap enough that verification remains worthwhile.

4. Why this matters more for RL than ordinary serving

For normal inference, speculative decoding saves user-facing latency.

For RL, it can reduce training time because rollout is part of the optimization loop itself.

Suppose one training iteration requires:

100,000 trajectories

and the average trajectory is:

2,000 generated tokens

Then the rollout stage produces roughly:

100,000 * 2,000
= 200,000,000 generated tokens

That is 200 million autoregressive token decisions.

Now suppose the effective accepted length from MTP is 3 tokens.

Very roughly, instead of requiring one main-model decoding step for every token, you might need something closer to:

200,000,000 / 3
~ 66,700,000

main verification steps.

This is not a 3x end-to-end RL speedup. That would be too simplistic.

Each verification step is more expensive than a normal one-token decode.

There is also rejected draft work, MTP overhead, scheduling overhead, synchronization, environment latency, and reward computation.

But it shows the basic source of leverage:

RL compute
    |
    +--> policy forward/backward
    |
    +--> rollout generation  <---- MTP attacks this
    |
    +--> environment
    |
    +--> reward computation

If rollout is 60% of wall-clock time and MTP somehow halves rollout time, the theoretical end-to-end improvement is approximately:

old time = 0.60 + 0.40
new time = 0.30 + 0.40
speedup  = 1.00 / 0.70
         ~ 1.43x

That is a much more realistic way to think about the economics.

Accelerating one stage by 2x does not mean the entire training system becomes 2x faster.

5. The math: acceptance length is the real lever

The central quantity is the number of draft tokens accepted by the main model.

Call it:

A = average accepted tokens per verification cycle

Without MTP:

A ~= 1

With a good MTP system:

A > 1

Suppose a trajectory requires T output tokens.

A crude approximation is:

verification_steps ~= T / A

So the relative reduction in sequential verification work is approximately:

reduction ~= 1 - 1/A

For:

A = 2

you eliminate roughly:

1 - 1/2 = 50%

of sequential steps.

For:

A = 3

you eliminate roughly:

1 - 1/3 = 67%

For:

A = 3.6

the rough figure is:

1 - 1/3.6
~ 72%

Again, this is not wall-clock speedup. It is a way of understanding the mechanical source of the gain.

The interesting part is that acceptance is related to uncertainty.

Suppose the model is completing:

"The capital of France is ..."

The next few tokens are easy.

An MTP head has a good chance of producing:

Paris

and perhaps the surrounding punctuation or explanation.

Now compare this with a difficult reasoning step where many continuations are plausible.

The MTP predictions diverge.

Acceptance falls.

The MiMo-V2-Flash experiments found a strong inverse relationship between next-token cross-entropy and MTP acceptance length. Their reported fit had R^2 = 0.995 across the evaluated datasets. Lower-uncertainty contexts produced acceptance lengths around 3.6 tokens, while more uncertain tasks had shorter accepted sequences.

This creates an important systems rule:

MTP performance is workload-dependent.

A benchmark with easy repetitive code may benefit much more than a benchmark dominated by uncertain reasoning branches.

And RL can change the model's entropy during training.

That matters.

A 2026 study called Bebop specifically investigates this issue and reports that MTP acceptance can degrade during RL as model entropy changes. The authors propose rejection-sampling and training approaches designed to preserve acceptance during RL, reporting up to 1.8x end-to-end acceleration in their asynchronous RL experiments.

So the real optimization target is not:

"Add MTP."

It is:

maximize useful accepted tokens
while
keeping draft cost low
and
preserving RL training behavior

6. What an RL engineer actually has to build

A production MTP-enabled RL system is more than adding three linear layers.

The rollout service now has roughly this structure:

Prompt
  |
  v
Main model forward
  |
  +--> MTP draft heads
  |       |
  |       +--> token 1
  |       +--> token 2
  |       +--> token 3
  |
  v
Main model verification
  |
  +--> accepted prefix
  |
  v
Continue generation

There are several engineering details that matter.

Keep the MTP module lightweight

If the draft module costs nearly as much as the main model, you have recreated the problem.

MiMo-V2-Flash therefore uses dense lightweight FFNs and local attention for its MTP blocks.

Measure acceptance by workload

Do not report only one global acceptance number.

Track something like:

acceptance_length
  by task type
  by trajectory length
  by generation temperature
  by training step
  by model checkpoint

A system that starts at:

A = 3.5

and falls to:

A = 1.4

after several million RL updates has a different economics profile.

Watch the long tail

This is particularly important in RL.

Suppose 64 sequences are generating concurrently.

If 63 finish quickly and one trajectory runs 20x longer, your effective utilization can collapse.

MTP attacks this by reducing the number of sequential decoding operations in that long trajectory.

The MiMo-V2-Flash infrastructure combines MTP with sequence-level scheduling, partial rollout, load balancing, and asynchronous reward computation. This is a useful clue about how the authors approached the problem operationally: MTP is one component in a larger rollout system rather than a magic optimization in isolation.

Profile memory, not just FLOPs

Autoregressive decoding is frequently memory-bound.

Each token requires repeatedly using model weights and reading/writing KV-cache state.

MTP increases token-level parallelism. Several candidate tokens can be considered together, improving arithmetic intensity and making better use of the accelerator.

That is why a roofline-style analysis is more useful than simply counting theoretical FLOPs.

The MiMo report explicitly notes that speedup depends on batch size, acceptance length, computation/I/O balance, and kernel efficiency. In their measured setup, three MTP layers produced roughly 1.8x-2.7x speedup across the tested batch sizes and acceptance lengths from 2.8 to 3.8.

7. The economics: think in tokens per dollar, not tokens per second

Consider a hypothetical RL cluster costing:

$100,000 per day

Suppose the training pipeline spends:

60% rollout
40% everything else

So:

rollout cost = $60,000/day
other cost   = $40,000/day

Imagine an MTP deployment reduces rollout wall-clock cost by 40%.

Then:

new rollout cost = $36,000/day

total cost = $36,000 + $40,000
           = $76,000/day

You have reduced total compute expenditure by:

1 - 76,000 / 100,000
= 24%

That is before considering whether you use the freed capacity to run more trajectories.

And this is where RL becomes interesting economically.

You have two choices after making rollout cheaper:

same training budget
       |
       +--> finish training faster
       |
       +--> generate more trajectories
       |
       +--> run more experiments
       |
       +--> increase environment diversity

For research teams, the last three may matter more than the raw speedup.

A 25% reduction in training cost can effectively become more than a 25% increase in experimentation capacity if the system was previously compute-constrained.

The broader lesson is that MTP is an example of a recurring systems principle in AI:

When a workload is sequential, look for structure that lets you trade dependency depth for parallel work.

Normal batching increases parallelism across requests.

MTP increases parallelism inside a request.

That distinction is especially useful when RL trajectories become long, irregular, and difficult to batch efficiently.

Conclusion

MTP started as a modification to the language-model training objective.

Then speculative decoding gave it another interpretation:

Predict the future cheaply.
Verify it with the expensive model.

For RL, that interpretation is particularly valuable because the system repeatedly generates trajectories before it can learn from them.

The architecture therefore becomes:

LLM
 |
 +--> MTP drafts several tokens
 |
 +--> main model verifies them
 |
 +--> environment scores trajectory
 |
 +--> optimizer updates policy
 |
 +--> repeat

The interesting metric is no longer just model throughput.

It is:

useful accepted tokens / unit time

And the interesting economic question is:

How much more RL learning can I buy
with the same accelerator budget?

The research direction is still evolving. Recent work suggests that MTP acceptance itself can change during RL, which means the draft mechanism may need to co-evolve with the policy rather than remain a static inference optimization.

That makes MTP more than a decoding trick.

It is a way of changing the computational shape of RL.

What other parts of the LLM RL stack do you think are fundamentally sequential today, but could be turned into parallel work?

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

⭐ Star it on GitHub:

HexmosTech / LiveReview

Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview: Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.
blast-radius-demo.mp4
LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.

The exact math, not a black box Visualize blast radius at a glance Every factor that feeds the score

How does Blast Radius scoring work? (a more technical explanation)
Here's the goal:

A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.

A 300-line UI change in one file, fully covered by…

View on GitHub

Click below to try LiveReview with your codebase:

Multi-Teacher On-Policy Distillation: How One LLM Can Learn From Several Expert Models

Shrijith Venkatramana — Fri, 25 Sep 2026 19:31:40 +0000

The usual way to improve an LLM is to make the model itself better.

Train it on more data. Run more reinforcement learning. Increase the context. Increase the parameter count.

But there is another possibility:

Build several models that become very good at different things, then teach a single student to absorb those capabilities.

That sounds like ordinary knowledge distillation.

The interesting part is what happens when the teachers are specialists, and the student is allowed to make its own mistakes while learning from them.

This is the idea behind Multi-Teacher On-Policy Distillation (MOPD), described in Xiaomi's 2026 MiMo-V2-Flash technical report.

The basic pattern is:

                    +-------------------+
                    |   Math Teacher    |
                    +---------+---------+
                              |
                    +---------v---------+
                    |                   |
                    |      Student      |----> final model
                    |                   |
                    +---------^---------+
                              |
                    +---------+---------+
                    |                   |
          +---------+---------+   +-----+---------+
          | Coding Teacher   |   | Agent Teacher |
          +------------------+   +---------------+

The student generates its own trajectory.

A domain-appropriate teacher looks at those exact tokens and provides dense feedback.

Over many iterations, the student becomes a single model that contains capabilities that previously lived in separate experts.

MiMo-V2-Flash uses this as the third stage of its post-training pipeline, after SFT and domain-specific RL. Its report describes specialist teachers for coding, search, tool use, mathematics, reasoning, and safety. (arXiv)

1. The problem: capability specialization creates a new integration problem

Imagine you have five excellent models:

Teacher A -> mathematics
Teacher B -> coding
Teacher C -> web search
Teacher D -> tool use
Teacher E -> general reasoning

Each model has undergone its own optimization process.

The math teacher has seen enormous amounts of reward pressure around mathematical reasoning.

The coding teacher has spent its RL budget on writing and executing code.

The search teacher has learned how to explore a web environment.

Eventually, you have several models that are locally excellent but globally fragmented.

How do you get one production model with all five capabilities?

There are several obvious approaches.

Parameter merging

Average or otherwise combine the weights.

The problem is that neural networks do not give you a clean "math capability vector" and "coding capability vector" that can simply be added together.

Sequential training

Train on mathematics, then coding, then search.

Now you risk the familiar continual-learning problem: the later optimization changes parameters that were useful for earlier capabilities.

Offline distillation

Ask each teacher to generate a large dataset, then fine-tune the student on those examples.

This works, but the student mostly learns from trajectories that the teachers considered worth generating.

The important question is:

What happens when the student behaves differently from the teacher?

That question leads directly to on-policy distillation.

2. From Hinton's distillation to learning from your own mistakes

The history is useful here.

In 2015, Geoffrey Hinton, Oriol Vinyals, and Jeff Dean described knowledge distillation as a way to compress an ensemble of models into a single deployable model.

The motivation was practical. An ensemble could be more accurate, but serving many large networks for every prediction was expensive. Their idea was to have the large ensemble teach one smaller model. (arXiv)

The basic idea was:

large ensemble
      |
      v
soft predictions
      |
      v
small student

For autoregressive language models, however, another problem appears.

Suppose the teacher produces:

The function is convex because its second derivative is positive...

and the student produces:

The function is convex because its derivative is increasing...

Those trajectories have different prefixes.

The probability distribution the student sees at token 20 depends on what the student generated at tokens 1-19.

So training on teacher-generated text gives you one distribution of states, while inference gives you another.

In 2024, Rishabh Agarwal and colleagues at Google DeepMind formalized this issue in Generalized Knowledge Distillation. Their on-policy approach has the student generate its own outputs, then asks the teacher to evaluate those student-generated trajectories. The point is to train on the states the student will actually visit. (ML Anthology)

That distinction is easiest to see like this:

OFF-POLICY

teacher -> generates trajectory -> student learns trajectory


ON-POLICY

student -> generates trajectory
              |
              v
         teacher evaluates it
              |
              v
         student learns from it

For an autoregressive model, that difference is substantial.

The teacher is no longer saying:

"Here is the answer I would have written."

It is effectively saying:

"Given the exact state you reached, here is how I would evaluate the next decision."

That is much closer to interactive coaching.

3. MOPD adds multiple specialist teachers

MOPD takes the on-policy idea and turns it into a multi-expert system.

The MiMo-V2-Flash pipeline is roughly:

Stage 1
Base model
   |
   v
SFT student
   |
   v
Stage 2
+-------------------------------+
|                               |
| math RL        -> math teacher|
| coding RL      -> code teacher|
| search RL      -> search teacher
| tool-use RL    -> agent teacher
| reasoning RL   -> reasoning teacher
|                               |
+-------------------------------+
   |
   v
Stage 3
MOPD
   |
   v
single general-purpose student

A prompt is associated with an appropriate domain teacher.

For example:

"Prove this number theory result"
            |
            v
      math teacher


"Fix this failing Python test"
            |
            v
      coding teacher


"Find the answer using web search"
            |
            v
      search teacher

The critical part is that the student still generates the response.

Suppose the student is trying to solve a coding problem.

It generates:

Step 1: inspect repository
Step 2: open foo.py
Step 3: modify function
Step 4: run tests
Step 5: ...

The coding teacher evaluates the probabilities it would assign to those student-generated tokens.

The teacher does not necessarily need to generate a competing four-thousand-token answer.

Instead, it supplies token-level information about the student's trajectory.

This makes MOPD very different from simply asking five teachers for five answers and throwing all of them into an SFT dataset.

The MiMo report explicitly describes MOPD as an on-policy RL process rather than parameter merging or static expert-generated datasets. (arXiv)

4. The math: turn teacher probabilities into a reward

Here is the core idea without getting buried in notation.

Let:

pi_student(y_t | x, y_<t)

be the probability the student assigns to the token it actually generated.

Let:

pi_teacher(y_t | x, y_<t)

be the probability the specialist teacher assigns to that same token, given the same prefix.

Now define the token-level signal:

A_t = log( pi_teacher / pi_student )

Everything is evaluated at the actual student token and its actual prefix.

This has a very intuitive interpretation.

Case 1: teacher likes the token more than the student does

Suppose:

student probability = 0.10
teacher probability = 0.50

Then:

A_t = log(0.50 / 0.10)
    = log(5)
    ≈ +1.61

That is a positive learning signal.

The student should increase the probability of making this decision in similar states.

Case 2: teacher dislikes the token

Suppose:

student probability = 0.40
teacher probability = 0.05

Then:

A_t = log(0.05 / 0.40)
    = log(0.125)
    ≈ -2.08

Now the signal is negative.

The student should reduce the probability of making that choice.

Case 3: they agree

Suppose:

student probability = 0.20
teacher probability = 0.20

Then:

A_t = log(1) = 0

There is no corrective pressure.

So the teacher is providing a dense, token-level advantage signal.

The underlying objective can be understood as minimizing the reverse KL divergence:

D_KL(student || teacher)

with the expectation taken over tokens sampled from the student.

The MiMo formulation then turns this into a policy-gradient-style surrogate objective. It also uses importance sampling when the training policy and sampling policy differ, and discards tokens whose probability ratios fall outside a specified range. (arXiv)

That last engineering detail matters.

Your rollout worker may use a slightly different inference configuration from the policy being updated.

If:

mu = sampling policy
pi = current training policy

then a correction factor of roughly

pi(y_t) / mu(y_t)

is needed.

But enormous ratios make the estimator unstable, so MOPD clips the usable region and drops sufficiently discrepant tokens. (arXiv)

This is one reason MOPD should be thought of as an RL system with a specialized reward, rather than simply "cross-entropy against a teacher."

5. Why this can combine capabilities without generating an enormous dataset

There is an important computational distinction.

Imagine a student generates 4,000 tokens.

With ordinary teacher-generated distillation, the teacher might need to autoregressively generate a 4,000-token answer.

That is 4,000 sequential decoding steps.

With on-policy token-level distillation, the student already generated the trajectory.

The teacher can evaluate the student trajectory with a forward pass over the sequence and produce logits for the relevant positions.

Conceptually:

Teacher generation:

t1 -> t2 -> t3 -> t4 -> ... -> t4000
 ^     ^     ^     ^
serial decoding


Teacher evaluation:

[t1 t2 t3 t4 ... t4000]
          |
          v
     logits for all
     causal positions

The exact implementation still has substantial transformer compute, especially with long contexts, but the workload is much more amenable to batching than autoregressive teacher generation.

That changes the economics.

Suppose:

1,000,000 training prompts
4,000 student tokens / prompt
1 selected teacher / prompt

Then the teacher processes approximately:

1,000,000 * 4,000
= 4 billion teacher-scored tokens

The important variable is selected teachers per trajectory, not the number of teachers in your library.

Seven specialist teachers do not automatically mean seven teacher evaluations for every token.

You can instead have:

7 teachers
   |
routing
   |
1 teacher per trajectory

The fixed cost is building seven good teachers.

The variable cost is scoring student trajectories with the relevant teacher.

That is an interesting economic trade:

Specialist RL
     |
     v
expensive capability acquisition
     |
     v
reusable teacher
     |
     v
many student models
     |
     v
amortized capability transfer

A company building multiple downstream models could reuse the same expert pool.

That is where multi-teacher distillation starts looking less like a training trick and more like an organizational architecture for model development.

6. The hard part is the infrastructure, not the equation

It is tempting to read the MOPD equation and conclude that implementation is just:

loss = log(student_prob / teacher_prob)

It is not.

A production implementation has at least these components:

                +----------------------+
                | Prompt / environment |
                +----------+-----------+
                           |
                           v
                  +----------------+
                  | Student rollout|
                  +--------+-------+
                           |
                           v
               +-----------------------+
               | Select domain teacher |
               +-----------+-----------+
                           |
                           v
               +-----------------------+
               | Teacher forward pass  |
               | on student trajectory |
               +-----------+-----------+
                           |
                           v
               +-----------------------+
               | Token-level advantage |
               +-----------+-----------+
                           |
                 +---------+---------+
                 |                   |
                 v                   v
             MOPD signal        ORM signal
                 |                   |
                 +---------+---------+
                           |
                           v
                    policy update

MiMo-V2-Flash is a useful concrete case because its authors describe the surrounding machinery.

Their RL/MOPD infrastructure uses SGLang for inference and Megatron-LM for training, with FP8 training and inference. They also describe rollout-routing replay, data scheduling, partial rollouts, prefix caching, and load balancing. (arXiv)

For coding agents, the scale gets even more concrete.

The report describes training across roughly 120,000 interactive environments and a Kubernetes-based environment setup involving more than 10,000 concurrent pods. (arXiv)

That tells you something important about MOPD:

the loss function is the easy part.

The difficult parts are:

rollout throughput
teacher serving
policy/rollout synchronization
importance sampling
long trajectories
GPU utilization
reward latency
environment orchestration

This is also why the distinction between an ordinary SFT pipeline and an on-policy pipeline matters operationally.

In SFT:

dataset -> GPU -> gradient update

In MOPD:

prompt
  -> rollout
  -> environment interaction
  -> teacher inference
  -> reward computation
  -> advantage construction
  -> policy update
  -> next rollout

Your training system has become a distributed control loop.

7. What MOPD changes about LLM post-training

The deeper idea is not "use several teachers."

It is:

separate capability acquisition from capability integration.

A specialist can be optimized very aggressively for one domain.

Then a general student can learn from many such specialists while remaining the entity that generates its own trajectories.

That gives you a useful architecture:

                    specialist RL
                         |
       +-----------------+------------------+
       |                 |                  |
       v                 v                  v
    math expert      coding expert      agent expert
       |                 |                  |
       +-----------------+------------------+
                         |
                         v
                 on-policy student
                         |
                         v
                  unified model

There is also an interesting feedback loop.

MiMo's authors describe a process in which a distilled student can itself become the starting point for another round of specialized RL, producing stronger teachers for a later student. (arXiv)

So you can imagine:

Student v1
   |
   v
specialized RL
   |
   v
Teachers v2
   |
   v
MOPD
   |
   v
Student v2
   |
   v
specialized RL
   |
   ...

That starts to resemble an internal capability-production system.

But the benchmark results are also a useful warning against treating this as automatic capability addition.

In the MiMo experiments, MOPD improved or matched the strongest teacher on several listed benchmarks, including AIME 2025, HMMT, LiveCodeBench, and tau²-Bench. But there were also regressions: for example, BrowseComp fell 6.3 points relative to the listed best teacher, and Arena-Hard Creative Writing fell 3.9 points. SWE-Bench Verified was 73.4 after MOPD versus 74.2 for the best teacher in that comparison. (arXiv)

So the realistic interpretation is:

MOPD provides a mechanism for capability transfer.

It does not guarantee perfect capability preservation.

And that is probably the most useful way for developers to think about it.

Knowledge distillation began with the problem of making ensembles cheaper.

On-policy distillation added the idea of teaching students on states they actually visit.

MOPD pushes that one step further:

many specialized policies
          |
          v
student-generated trajectories
          |
          v
token-level expert feedback
          |
          v
one deployable policy

The interesting engineering question is therefore no longer just:

"How do I make a bigger model?"

It becomes:

"How cheaply can I create specialized intelligence, and how efficiently can I transfer it into one model?"

That is a very different post-training economics.

What do you think is the harder scaling problem for the next generation of LLMs: training stronger specialist teachers, or building the infrastructure to distill them into a single general model?

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

⭐ Star it on GitHub:

HexmosTech / LiveReview

Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview: Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.
blast-radius-demo.mp4
LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.

The exact math, not a black box Visualize blast radius at a glance Every factor that feeds the score

How does Blast Radius scoring work? (a more technical explanation)
Here's the goal:

A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.

A 300-line UI change in one file, fully covered by…

View on GitHub

Click below to try LiveReview with your codebase:

DeepSeek's Attention Stack: How DSA, CSA, HCA, and mHC Fit Together

Shrijith Venkatramana — Thu, 24 Sep 2026 19:20:30 +0000

At 1 million tokens, the difficult part of an LLM is no longer simply having a large memory.

It is deciding what part of that memory is worth reading.

DeepSeek's recent architecture work can be understood as a sequence of answers to that problem:

MLA compresses the representation stored for each token.
DSA learns which parts of that compressed history matter.
CSA combines compression and learned sparse selection.
HCA compresses history even more aggressively and reads it densely.
mHC solves a different problem: keeping information flowing stably through many Transformer layers.

These are easy to confuse because they all appear in the same family of models, but they operate at different levels.

The useful mental model is:

                         DeepSeek-V4

        ┌──────────────────────────────────────────┐
        │              Residual stream             │
        │                    │                     │
        │                   mHC                    │
        │                    │                     │
        │      ┌─────────────┴─────────────┐      │
        │      │                           │      │
        │     CSA                         HCA     │
        │      │                           │      │
        │ compress + select           compress only│
        │      │                           │      │
        │     DSA                      dense attn  │
        │      │                           │      │
        │      └─────────── attention ────┘      │
        └──────────────────────────────────────────┘

The interesting part is that DeepSeek did not discover one magic attention mechanism.

It built a hierarchy of information access.

1. The problem DeepSeek was actually trying to solve

The original Transformer attention mechanism has a simple rule:

For every query, compare it with every previous token.

For a sequence of length L, that gives roughly:

attention work ~ O(L^2)

At 4K tokens, this is manageable.

At 128K, it becomes expensive.

At 1M tokens, blindly comparing everything with everything becomes a systems problem.

This matters even more for reasoning models and agents.

A normal chatbot might generate 500 tokens after reading a 5K-token prompt.

An agent can repeatedly:

read repository
→ call tool
→ inspect result
→ reason
→ call another tool
→ inspect another result
→ continue

The context keeps accumulating.

The model therefore needs something closer to a working memory hierarchy.

DeepSeek approached this problem incrementally.

In December 2025, the DeepSeek-V3.2 paper introduced DeepSeek Sparse Attention (DSA). The interesting detail is that V3.2 did not redesign the entire model. It started from the previous checkpoint and introduced DSA through continued training.

Then, on December 31, 2025, Zhenda Xie and colleagues published the mHC work, attacking a different issue: information propagation through increasingly sophisticated residual connections.

A few months later, DeepSeek-V4 combined these ideas with two new attention forms, CSA and HCA, and pushed the context length to one million tokens.

That sequence is useful because it reveals what each mechanism is actually responsible for.

2. DSA: the model learns where to look

Start with ordinary attention.

For one query q_t, we calculate something like:

score(t, s) = q_t · k_s

for every preceding token s.

Then we use those scores to form the weighted sum of the values.

DSA inserts a cheap indexer before the expensive attention operation.

Conceptually:

query
  │
  ▼
lightweight indexer
  │
  ├── score token 17
  ├── score token 18
  ├── score token 19
  ├── ...
  └── score token 100000
          │
          ▼
       Top-k
          │
          ▼
 expensive attention

Instead of doing expensive attention over the entire history, the model first asks:

Which locations appear relevant to this query?

Then it performs the real attention operation only on those locations.

The DSA paper expresses this as:

I(t,s) = sum_j w(t,j) * ReLU(q(t,j) · k(s,j))

followed by:

S_t = TopK(I(t,:))

and the normal attention operation is performed only on S_t.

The important distinction is that DSA does not merely make attention smaller.

It makes the selection content-dependent.

For one query:

"Where did we define the database schema?"

the indexer might favor a handful of earlier locations containing schema definitions.

For another query:

"What did the user say about authentication?"

a different subset becomes relevant.

This makes DSA closer to a learned retrieval mechanism than to a fixed sparse pattern.

A concrete training trick

There is a nice engineering story in the V3.2 paper.

DeepSeek did not immediately turn on sparse attention.

First, it trained the lightweight indexer while keeping normal dense attention active.

The dense model supplied a target attention distribution, and the indexer was trained using KL divergence:

L_indexer = KL(
    dense_attention_distribution
    ||
    softmax(indexer_scores)
)

They used only 1,000 warm-up steps, covering about 2.1 billion tokens.

Only after the indexer had learned something approximating the dense model's attention pattern did they turn on actual top-k selection.

Then they trained the model for another 15,000 steps over about 943.7 billion tokens, selecting 2,048 KV tokens per query.

That is a useful lesson for developers:

sparsity can be trained as a routing problem before it becomes a computational constraint.

3. CSA: compress first, then ask DSA where to look

Now we can understand CSA.

CSA stands for Compressed Sparse Attention.

It essentially says:

DSA is useful, but do not even make the indexer search over every original token.

Suppose:

L = 1,000,000 tokens
m = 4

CSA compresses the sequence dimension by approximately 4x.

So instead of:

1,000,000 KV entries

we get roughly:

250,000 compressed KV entries

The compression itself is learned.

Very roughly, for a group of tokens:

C_comp = sum_j S_j * C_j

where the S_j weights are normalized and learned from the hidden states.

Now DSA operates on this compressed sequence.

For DeepSeek-V4-Pro:

m = 4
top-k = 1024

So the rough information flow is:

1,000,000 original tokens
          │
          ▼
   compression / 4
          │
          ▼
250,000 compressed entries
          │
          ▼
       DSA TopK
          │
          ▼
  1,024 entries
          │
          ▼
    core attention

That is the central relationship between DSA and CSA.

DSA is the selection mechanism.

CSA changes the objects being selected.

DSA alone:

tokens → score → TopK → attention

CSA:

tokens → compress → score compressed entries → TopK → attention

This produces an important back-of-the-envelope calculation.

With a million-token context:

dense attention:
1,000,000 KV candidates

CSA core attention:
1,024 KV candidates

Ignoring all other costs:

1,000,000 / 1,024 ≈ 977

So the expensive core attention sees almost three orders of magnitude fewer candidates.

But this does not mean the entire attention layer becomes 977x cheaper.

The lightning indexer still has to inspect the compressed history:

~250,000 compressed candidates

and the compression itself has a cost.

The actual architecture is therefore a trade:

cheap broad search
        +
expensive narrow attention

That pattern is familiar from information retrieval systems.

You do not run the most expensive ranking model against ten million documents.

You first retrieve a candidate set.

4. HCA: what happens when even 4:1 compression is unnecessary?

CSA gives us:

moderate compression
+
sparse selection

But DeepSeek asks another question:

What if we compress history so aggressively that we no longer need sparse selection?

That is HCA: Heavily Compressed Attention.

For HCA, DeepSeek uses:

m' = 128

So:

1,000,000 tokens / 128
≈ 7,812 compressed entries

Now dense attention over 7,812 entries is much more manageable.

There is no DSA-style TopK step.

The architecture is simply:

1,000,000 tokens
       │
       ▼
   compress / 128
       │
       ▼
~7,812 compressed entries
       │
       ▼
   dense attention

This reveals something important about CSA versus HCA.

They are not competing mechanisms.

They represent two different resolutions of memory.

Think of a map.

When navigating a city, you might need:

nearby streets        → detailed
nearby neighborhoods  → moderately detailed
distant cities        → coarse

CSA is the middle layer.

HCA is the far-away layer.

DeepSeek-V4 interleaves the two.

For V4-Pro, the paper specifies:

CSA compression:       4:1
HCA compression:     128:1
CSA top-k:            1024
sliding window:        128 tokens

There is also a local sliding-window branch containing recent uncompressed KV entries.

That last detail matters.

Compression is excellent for long-range information, but language has very strong local dependencies.

The token immediately before the current token matters.

A variable name five tokens ago matters.

The beginning of the current sentence matters.

So DeepSeek effectively gives the model:

recent context:
    individual tokens

medium-range context:
    compressed blocks + learned TopK

long-range context:
    heavily compressed blocks

This is much closer to a multi-resolution memory system than to conventional attention.

5. Where does mHC fit? It is solving a different problem.

This is where the terminology becomes confusing.

mHC has almost nothing to do with deciding which token to attend to.

It changes the residual connection between Transformer layers.

The conventional Transformer update looks roughly like:

x_(l+1) = x_l + F_l(x_l)

The identity path is extremely important.

It gives information and gradients a clean route through hundreds of layers.

Hyper-Connections generalize this idea by maintaining multiple residual streams.

Instead of one stream:

we might have:

x1
x2
x3
x4

and allow the layer to mix them.

DeepSeek's mHC uses a formulation of the form:

X_(l+1) = B_l X_l + C_l F_l(A_l X_l)

Here:

X_l     = multiple residual streams
A_l     = mixes them into the layer input
F_l     = the actual Transformer block
B_l     = residual-stream mixing
C_l     = writes the layer output back

The problem is that unrestricted matrices B_l can make signal propagation unstable when many layers are stacked.

mHC constrains B_l to be doubly stochastic.

That means:

B >= 0

each row sums to 1
each column sums to 1

The set of these matrices is the Birkhoff polytope.

Why is this useful?

Imagine each residual stream as carrying some quantity of information.

A doubly stochastic transformation behaves like a conservative redistribution:

stream 1 ─────┐
stream 2 ──┐  │
stream 3 ──┼──┼──> redistributed streams
stream 4 ──┘  │
              │

It can mix information between streams, but it cannot arbitrarily amplify the whole residual transformation.

DeepSeek's stated mathematical property is that:

||B||_2 <= 1

so the residual mapping is non-expansive.

This is why mHC belongs to the stability side of the architecture.

CSA/HCA answer:

What should this layer read?

mHC answers:

How should information survive and mix as it passes through the stack?

That distinction is worth remembering.

6. The connection between them is architectural, not mathematical

The cleanest way to understand DeepSeek-V4 is to separate the system into three axes.

Axis 1: What information is available?

Controlled by KV compression.

MLA → CSA/HCA

The model stores a much smaller representation of history.

Axis 2: What information gets expensive attention?

Controlled by sparse routing.

DSA → CSA

The model uses a cheap indexer to identify a small subset.

Axis 3: How does information propagate between layers?

Controlled by mHC.

mHC → residual stream

The model gets a richer residual topology while constraining the residual mixing for stability.

Put differently:

             INFORMATION ACCESS

                    CSA
                  /     \
             compression  DSA
                  \       /
                   attention


             INFORMATION PROPAGATION

                    mHC
                     │
              residual streams
                     │
              Transformer blocks

The important insight is that mHC is orthogonal to DSA.

You could imagine a model with DSA and ordinary residual connections.

You could imagine mHC combined with dense attention.

DeepSeek combines them because the engineering constraints are interconnected, but the mechanisms attack different bottlenecks.

This also explains why the V4 paper describes them separately in its architecture section.

7. The economics of this design: memory becomes an architectural resource

The million-token context claim is ultimately an economics claim as much as a modeling claim.

At one million tokens, KV cache is expensive.

Suppose a conventional attention system stores a large KV tensor for every layer and every token.

Multiply:

tokens
× layers
× KV dimensions
× bytes per element
× concurrent requests

and the memory requirement grows rapidly.

Then there is the bandwidth required to read those KV entries during decoding.

This is why the V4 paper reports a particularly useful comparison.

At a 1M-token context:

V4-Pro:
~27% of the V3.2 single-token inference FLOPs
~10% of the V3.2 KV-cache size

And V4-Flash goes further:

~10% of V3.2 FLOPs
~7% of V3.2 KV cache

The model therefore spends a large architectural budget deciding what not to compute.

There is an equally important operational story.

DeepSeek did not merely invent the equations and hope GPU kernels would follow.

They changed the serving system around the architecture:

compressed KV layout
+ sparse-attention kernels
+ sliding-window state
+ on-disk KV caching
+ contextual parallelism
+ fused mHC kernels

For example, mHC increases activation memory and pipeline communication relative to ordinary residual connections. DeepSeek reports using fused kernels, selective recomputation, and pipeline overlap to constrain the wall-time overhead to about 6.7% of the overlapped pipeline stage.

That is a useful lesson for anyone building LLM infrastructure:

an architectural optimization is only real when the hardware sees it.

A paper can turn:

O(L^2)

into something much smaller on paper and still lose in practice to:

kernel launch overhead
memory movement
unfavorable gathers
communication
poor GPU occupancy

DeepSeek's V4 work is interesting precisely because the algorithm, training procedure, kernels, cache format, and distributed runtime were designed together.

A developer's mental model

You do not need to remember every equation.

Remember this pipeline:

                         1M-token history
                                │
                                ▼
                     compress the KV history
                                │
              ┌─────────────────┴────────────────┐
              │                                  │
             CSA                                HCA
              │                                  │
       4:1 compression                    128:1 compression
              │                                  │
       DSA TopK selection                  dense attention
              │                                  │
              └──────────────┬───────────────────┘
                             │
                    local 128-token window
                             │
                             ▼
                      attention output
                             │
                             ▼
                           mHC
                             │
                             ▼
                    next Transformer layer

So the one-line summary is:

DSA = learned retrieval
CSA = compression + learned retrieval
HCA = extreme compression + dense retrieval
mHC = stable multi-stream residual transport

And there is a broader architectural idea underneath all four:

A million-token context does not require treating a million tokens as equally expensive pieces of memory.

DeepSeek's approach is to represent history at different resolutions and spend computation where the query actually needs detail.

That is a very general idea.

It applies beyond attention to databases, retrieval systems, caches, agent memory, and even compiler architectures: store broadly, index cheaply, compute precisely where needed.

The interesting question for the next generation of LLMs is therefore less "How do we make context windows bigger?" and more:

How many different resolutions of memory should a model have, and how should it learn when to move between them?

What do you think is the more important direction from here: better learned retrieval like DSA, more aggressive hierarchical compression like HCA, or changing the Transformer's information-flow topology like mHC?

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

⭐ Star it on GitHub:

HexmosTech / LiveReview

Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview: Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.
blast-radius-demo.mp4
LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.

The exact math, not a black box Visualize blast radius at a glance Every factor that feeds the score

How does Blast Radius scoring work? (a more technical explanation)
Here's the goal:

A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.

A 300-line UI change in one file, fully covered by…

View on GitHub

Click below to try LiveReview with your codebase:

Muon: What Happens When an LLM Optimizer Treats a Weight Matrix Like a Matrix

Shrijith Venkatramana — Wed, 23 Sep 2026 19:50:25 +0000

Most LLM developers know the AdamW update by heart:

take the gradient, keep moving averages, normalize the update, change the weights.

But there is a question hiding underneath all of this:

What exactly is a 4096 x 4096 weight matrix?

AdamW mostly treats it as 16 million scalar coordinates.

Muon treats it as a matrix.

That distinction is the whole story.

Muon is an optimizer for the hidden matrix parameters of neural networks. Its core operation takes the momentum update, looks at its singular directions, throws away the singular-value magnitudes, and keeps the directions. It then approximates this operation cheaply with a few Newton-Schulz iterations.

The result is an optimizer that has produced faster training in small-model competitions, scaled to multi-billion-parameter language models, and is now part of the mainstream PyTorch optimization stack.

The interesting part is not merely that "Muon beats AdamW."

The interesting part is why someone thought to optimize a neural network matrix as a matrix in the first place.

1. The story starts with a matrix

Consider a transformer linear layer:

y = W x

where W might be a 4096 x 4096 matrix.

During backpropagation we obtain a gradient:

G = dL/dW

AdamW maintains statistics for each scalar element of G.

Very roughly:

m_t = beta1 * m_(t-1) + (1-beta1) * G_t

v_t = beta2 * v_(t-1) + (1-beta2) * G_t^2

update = m_t / sqrt(v_t)

The important detail is G_t^2: the second-moment estimate is elementwise.

This gives Adam a coordinate-wise view of the parameter space.

Muon starts from a different observation:

W is not merely a bag of numbers. It represents a linear transformation.

For a matrix, one of the natural ways to understand that transformation is through its singular value decomposition:

G = U Sigma V^T

Here:

U and V = directions
Sigma  = magnitudes along those directions

So there are two different questions:

Which directions does the gradient want to move?
How large should each of those directions be?

Muon makes a very particular choice:

preserve the directions, but approximately equalize their magnitudes.

That is what orthogonalizing the update means.

The idea appeared publicly in October 2024, when Keller Jordan and collaborators were working on the NanoGPT speedrunning competition. On October 15, 2024, a Muon-based run set a new training-speed record, improving the previous result by about 35%. The project then became a collaboration involving people such as Jeremy Bernstein, Laker Newhouse, Yuchen Jin, Vlado Boza, Jiacheng You, and Franz Cesista. Jordan's account is unusually concrete about the engineering: Boza found that treating Q, K, and V separately worked better; Jin pushed experiments to larger models and supplied much of the H100 compute; Bernstein, You, and Cesista reduced the cost of the matrix orthogonalization itself.

That history matters because Muon did not emerge from a giant benchmark suite first.

It emerged from people trying to make a tiny training program go faster.

2. The key idea: keep the directions, flatten the singular values

Suppose an update matrix has the form

G = U diag(20, 3, 0.2) V^T

The gradient has three principal matrix directions.

One direction has magnitude 20.

Another has magnitude 3.

Another has magnitude 0.2.

Muon constructs an approximation of

U diag(1, 1, 1) V^T

up to the appropriate scaling for the matrix shape.

The singular vectors remain.

The singular values disappear.

For a square matrix, this produces an ordinary orthogonal matrix. For a rectangular matrix, it produces a semi-orthogonal matrix.

Another way to write the operation is:

Ortho(G) ~= U V^T

This has a useful geometric interpretation.

Imagine that your update is saying:

"Move strongly in direction A,
 somewhat in direction B,
 and barely at all in direction C."

Muon says:

"Those are the important directions.
Let's give them roughly equal opportunity to affect the layer."

The original Muon write-up notes that transformer updates often have high condition numbers: a few directions can dominate the update while other directions have much smaller singular values. The authors proposed that orthogonalization may help those lower-magnitude directions contribute more. That explanation is an empirical hypothesis rather than the complete theoretical justification for Muon.

This also explains why Muon is fundamentally different from simply changing Adam's hyperparameters.

Adam changes how each coordinate is scaled.

Muon changes the matrix geometry of the update.

There is an analogy to dimensionality reduction, but in reverse.

PCA asks:

Which directions contain most of the variation?

Muon asks:

What if the update contains a few dominant directions,
but I want the matrix update to retain all of its principal directions
at comparable scale?

3. But an SVD every training step would be ridiculous

There is an obvious problem.

If we literally want:

G = U Sigma V^T

G -> U V^T

we could compute an SVD.

For a huge transformer, doing a full SVD for every large weight matrix at every optimizer step would be an unattractive idea.

Muon's practical insight is:

we do not need to compute the SVD explicitly.

Instead, use Newton-Schulz iteration.

Start by normalizing the matrix:

X = G / ||G||_F

Then repeatedly apply a matrix polynomial.

The production Muon implementation uses:

X_next = a X
          + b (X X^T) X
          + c (X X^T)^2 X

with coefficients approximately:

a =  3.4445
b = -4.7750
c =  2.0315

and typically five iterations.

Why does this work?

Take the SVD:

X = U Sigma V^T

The polynomial operation preserves the singular vectors:

p(X) = U p(Sigma) V^T

So instead of manipulating the whole matrix conceptually, we can think about what the polynomial does to each singular value.

The quintic mapping is:

p(s) = a s + b s^3 + c s^5

Repeatedly apply it.

The goal is for the singular values to converge toward 1.

So:

U Sigma V^T
        |
        v
U p(Sigma) V^T
        |
        v
U p(p(Sigma)) V^T
        |
        v
U I V^T

The fascinating implementation detail is that we never explicitly calculate U, Sigma, or V.

We only perform matrix multiplications.

This is where numerical linear algebra meets GPU engineering.

The early Muon work considered several ways of doing the orthogonalization. SVD was too slow. Other Newton-style methods had numerical problems in lower precision. Newton-Schulz could be run efficiently in bfloat16, which made it much more suitable for modern accelerators. The coefficients themselves were tuned experimentally; Jordan describes researchers using Desmos to explore polynomial shapes during the NanoGPT speedrun.

That is a useful general lesson for ML engineers:

an algorithm that is mathematically expensive may become practical when you find a formulation that maps onto the hardware's favorite operations.

4. The deeper idea: Muon is really about choosing a geometry

There is a deeper way to understand all of this.

Suppose a linear layer is:

y = W x

and we change the weights by:

W -> W + dW

The resulting change in the output is:

dy = dW x

So we can ask:

How large should a weight update be if I care about controlling the change it causes to the layer's output?

Suppose we measure vectors using RMS:

||x||_RMS = sqrt((1/d) sum_i x_i^2)

Then a matrix has an operator norm describing its maximum RMS-to-RMS amplification:

||W||_(RMS->RMS)

Now imagine the optimization problem:

minimize      <G, dW>

subject to    ||dW||_(RMS->RMS) <= eta

In plain English:

Choose the update that gives the largest first-order
decrease in loss, while limiting how much the layer
can change its outputs.

The solution involves the orthogonalized gradient:

dW ~= -eta * scale * U V^T

So the U V^T operation is not merely an arbitrary trick.

It arises from asking what "the biggest useful update" means under a matrix norm that is tied to the behavior of a linear layer.

This is part of Jeremy Bernstein and Laker Newhouse's broader work on modular duality. Their 2025 ICML paper develops a framework in which different neural-network modules can be assigned different geometries, with GPU-friendly dualization procedures for layers such as Linear and Conv2D. Newton-Schulz appears naturally in that construction.

This perspective also connects Muon to Shampoo.

Without its accumulation mechanism, the Shampoo update can be algebraically reduced to an orthogonalized gradient:

G
 |
 v
U Sigma V^T
 |
 v
U V^T

So Muon can be viewed as a particularly cheap, momentum-based way of getting this matrix-aware behavior.

That is one reason the optimizer is intellectually interesting.

It is less about inventing another collection of moving averages and more about asking:

What metric should a neural-network layer use for optimization?

5. Getting from a 2024 speedrun to real LLM training

The early results were promising, but there was a serious problem:

Would this thing actually scale?

Moonshot AI addressed that question in the 2025 paper Muon is Scalable for LLM Training.

They identified two practical issues that mattered at larger scale:

1. Weight decay
2. Correct scaling of the Muon update

The second point is particularly important.

Muon produces an orthogonalized matrix whose RMS behavior depends on the dimensions of the matrix.

A 1024 x 1024 matrix and a 8192 x 8192 matrix cannot simply receive the identical raw update scale and be expected to behave identically.

Moonshot introduced an update scaling rule designed to make Muon's update RMS comparable to AdamW's. Their experiments reported roughly 2x computational efficiency at compute-optimal training, with comparable performance reached using roughly 52% of the training FLOPs of the AdamW counterparts in their scaling experiments.

They also trained Moonlight, a 3B/16B mixture-of-experts model, on 5.7 trillion tokens using Muon.

This is where the distinction between "interesting optimizer paper" and "useful engineering technique" becomes important.

A 2x efficiency result is economically meaningful only when the comparison is properly controlled.

For example, if a training run costs:

$1,000,000

and the compute requirement genuinely falls by 48%, the idealized savings are:

$1,000,000 * 0.48 = $480,000

But actual GPU spend is not a pure FLOP meter.

You also have:

GPU utilization
communication
checkpointing
data loading
optimizer implementation
network topology
engineering time
failed runs

So "48% fewer FLOPs" should be read as an opportunity for lower cost, rather than as a promise of a 48% lower cloud bill.

There is also a useful memory difference.

Adam-like optimizers commonly maintain two moment tensors:

m
v

Muon's core optimizer state contains one momentum buffer.

Ignoring parameter replicas, master weights, sharding, and datatype choices:

AdamW optimizer state: 2 x parameter bytes
Muon optimizer state:  1 x parameter bytes

At 100B parameters, if those state tensors were stored in fp32:

100B * 4 bytes = 400 GB

AdamW moments:
2 * 400 GB = 800 GB

Muon momentum:
1 * 400 GB = 400 GB

That difference becomes relevant when optimizer state is one of the constraints determining how many GPUs a training job needs.

The computational cost of Newton-Schulz is also less frightening than the name suggests.

For an n x m matrix, with m <= n, the Muon write-up derives an extra cost of roughly:

6 T n m^2

FLOPs for T Newton-Schulz steps.

The corresponding forward-plus-backward cost for the linear layer scales roughly like:

6 n m B

where B is the number of tokens processed by the layer in the batch.

The ratio is therefore approximately:

overhead ~= T m / B

Take a hypothetical training setup:

model width m = 4096
tokens per batch B = 4,000,000
Newton-Schulz steps T = 5

Then:

overhead ~= 5 * 4096 / 4,000,000
         ~= 0.00512
         ~= 0.51%

That is the operational trick.

The expensive-looking matrix computation is amortized across millions of tokens.

For the actual NanoGPT speedrun configuration discussed by Jordan, the corresponding estimate was about 0.7%.

As of 2026, the idea has also moved into mainstream infrastructure. Current PyTorch documentation exposes torch.optim.Muon, including different update-scaling modes, and the DeepSpeed team added Muon support in June 2026.

6. What does a developer actually do with Muon?

The first mistake would be:

"Replace AdamW everywhere with Muon."

That is not how the original method is intended to be used.

Muon is primarily for 2D hidden weight matrices.

Parameters such as:

embeddings
biases
LayerNorm parameters
other 1D parameters
input layers
output heads

remain on a conventional optimizer such as AdamW.

The original experiments also found that Q, K, and V were better treated as separate matrices rather than as one fused QKV matrix.

Conceptually, your optimizer setup looks like:

hidden Linear weights  -> Muon
embeddings              -> AdamW
normalization           -> AdamW
biases                  -> AdamW
LM head                 -> AdamW

With current PyTorch, a schematic setup looks like this:

muon_opt = torch.optim.Muon(
    muon_params,
    lr=3e-4,
    weight_decay=0.01,
    momentum=0.95,
    nesterov=True,
    adjust_lr_fn="match_rms_adamw",
)

adamw_opt = torch.optim.AdamW(
    adamw_params,
    lr=3e-4,
    weight_decay=0.01,
)

The important part is not the exact numbers above.

The important part is constructing muon_params deliberately rather than doing:

[p for p in model.parameters() if p.ndim == 2]

because a tensor being 2D does not automatically mean that its optimization geometry should be Muon's.

Also, do not blindly copy early Muon examples that use a fixed lr=0.02.

There have been several generations of scaling conventions. Keller Jordan's original implementation, Moonshot's RMS-matching variant, and Bernstein's theoretical scaling rule use different parameter-shape-dependent factors. Current PyTorch exposes all three approaches.

For a developer evaluating Muon, I would therefore treat the optimizer as part of the training configuration, not as a one-line AdamW replacement.

A reasonable experiment is:

same architecture
same data
same tokens
same batch size
same hardware
same evaluation checkpoints

compare:
    AdamW
    Muon + AdamW hybrid

Then measure:

validation loss vs tokens
validation loss vs FLOPs
validation loss vs GPU-hours
peak memory
step time

The last three matter because an optimizer can improve sample efficiency while making each step slower, or reduce FLOPs without reducing wall-clock time on a communication-bound cluster.

7. The larger lesson

Muon is interesting because it exposes a broader idea that is easy to miss when working with large neural networks:

the optimizer contains assumptions about what a parameter means.

Adam implicitly says:

parameters are coordinates
gradients are coordinates
normalize coordinates independently

Muon says:

these parameters form matrices
matrices represent linear operators
linear operators have meaningful singular directions
optimize those operators using a matrix-aware geometry

That shift is bigger than the particular Newton-Schulz polynomial.

The Newton-Schulz iteration is the implementation technique.

The deeper idea is choosing a geometry that matches the structure of the object being optimized.

That may also explain why optimizer research sometimes looks disconnected from ordinary software engineering. An optimizer sounds like a small implementation detail:

optimizer.step()

Yet changing that one line changes the effective geometry of a trillion-parameter computation.

Muon began as an October 2024 experiment in a community speedrun, accumulated contributions from researchers and engineers working on the math and GPU implementation, scaled into the Moonlight training run, and subsequently became available in major training infrastructure. The interesting question now is less "Is Muon the replacement for AdamW?" and more:

How many other parts of deep-learning systems are still being optimized using mathematical abstractions chosen for convenience rather than for the structure of the object itself?

What other neural-network components do you think deserve their own optimization geometry?

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

⭐ Star it on GitHub:

HexmosTech / LiveReview

Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview: Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.
blast-radius-demo.mp4
LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.

The exact math, not a black box Visualize blast radius at a glance Every factor that feeds the score

How does Blast Radius scoring work? (a more technical explanation)
Here's the goal:

A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.

A 300-line UI change in one file, fully covered by…

View on GitHub

Click below to try LiveReview with your codebase:

DEV Community: Shrijith Venkatramana

Lean for Developers: Turn Program Requirements into Theorems

Tests check examples; specifications define obligations

Translate the requirement into a mathematical contract

Prove that the implementation satisfies the contract

Understand exactly what the proof establishes

Move from function contracts to state invariants

Integrate specification into the development workflow

Start with one invariant

HexmosTech / LiveReview

Risk-Aware AI Code Review for Business-Critical Systems

LiveReview: Risk-Aware AI Code Review for Business-Critical Systems

Auto-Approve: Ship the Safe 50% Without a Human

How Meta Reduced Code Review Load While Maintaining Production Reliability

Code generation scaled faster than code review

RADAR is a funnel, not an LLM with merge permissions

Risk calibration determines how much work gets automated

Reliability comes from layered controls and operational discipline

The production numbers: more throughput, fewer incidents

Why review latency fell: a queueing and economic perspective

What engineering teams can learn from Meta's approach

Conclusion: automate the routine, preserve attention for the consequential

HexmosTech / LiveReview

Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview: Risk-Aware AI Code Review for Business-Critical Systems

How DeepWiki Works: Turning a Codebase into a Searchable Mental Model

Code understanding is an attention problem

DeepWiki builds two maps of the repository

Before the LLM writes, the code gets indexed

DeepWiki plans the wiki before generating the wiki

Questions are answered by retrieval (rather than memory)

The hard part at scale is orchestration

Conclusion: the wiki is really a cache for software understanding

HexmosTech / LiveReview

Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview: Blast-Radius Aware AI Code Review for Business-Critical Systems

Blender for Developers: 15 Concepts Explained Simply

1. Start With the Right Mental Model

1. Object

2. Data-blocks

3. Transform + Local Space + World Space

2. Geometry Is Data, Not a Picture

4. Mesh

5. Topology

6. Normals

3. Modifiers: Treat Geometry Like a Pipeline

7. Modifiers

8. Bevel

9. Subdivision Surface

4. The Camera Turns 3D Into 2D

10. Shading

11. Camera Projection

12. Ray Tracing

5. Animation Is a Function of Time

13. Keyframes

14. F-Curves and Interpolation

6. Constraints and Relationships: Build the System Once

15. Parenting and Constraints

What about Armatures and IK?

7. The Production Loop: Pixels Have a Price

The 15 Concepts in One Mental Model

The practical takeaway

HexmosTech / LiveReview

Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview: Blast-Radius Aware AI Code Review for Business-Critical Systems

How chDB Turns Ephemeral Agent State into Durable Memory

1. Agent memory is becoming a database problem

2. The SQLite-to-server spectrum has a missing middle

Local SQLite

SQLite plus replication

Remote database

Embedded analytical database

3. The idea: local working copy, durable object

4. What actually happens during durability?

execute(): mutate locally

flush(): make a promise real

checkpoint(): create a new base

head.json: who owns the database?

5. Store memory as history, not as one giant document

Raw evidence and actual memory are different

`execute()`: mutate locally

`flush()`: make a promise real

`checkpoint()`: create a new base

`head.json`: who owns the database?