DEV Community: Shrijith Venkatramana

FlashAttention Explained: The Optimization That Made Modern LLMs Practical

Shrijith Venkatramana — Thu, 11 Jun 2026 17:32:44 +0000

Hello, I'm Shrijith Venkatramana. I'm building git-lrc, an AI code reviewer that runs on every commit. Star Us to help devs discover the project. Do give it a try and share your feedback for improving the product.

Large language models keep getting bigger.

Context windows have grown from a few thousand tokens to hundreds of thousands, and some models now advertise context lengths measured in millions of tokens.

Yet for years, one part of the Transformer threatened to become the bottleneck:

Attention.

Not because it required too much math.

Because it moved too much data.

FlashAttention is one of the most important optimizations in modern AI infrastructure because it attacks exactly that problem. It doesn't change the Transformer architecture, approximate attention, or introduce a new model design. Instead, it rethinks how attention is executed on GPUs.

The result is dramatically lower memory usage, faster training, faster inference, and practical long-context models.

Let's see how it works.

1. The Real Cost of Attention

Every Transformer layer computes attention using three matrices:

Query (Q)
Key (K)
Value (V)

The standard attention equation is:

Attention(Q,K,V) = softmax(QK^T / sqrt(d))V

Conceptually, every token compares itself against every other token.

For a sequence of N tokens, the attention score matrix contains:

N x N

elements.

A sequence length of 16,384 tokens requires:

16,384 x 16,384 = 268 million

attention scores.

The first instinct is usually:

"That's a lot of computation."

But on modern GPUs, the bigger problem is often memory traffic.

2. GPUs Are Often Memory-Bound, Not Compute-Bound

Modern GPUs can perform enormous numbers of floating-point operations per second.

What they cannot do nearly as efficiently is move massive amounts of data between memory levels.

A simplified GPU memory hierarchy looks like:

HBM (GPU memory)
    |
L2 Cache
    |
Shared Memory
    |
Registers

The farther away the data is from the compute units, the more expensive it is to access.

Traditional attention repeatedly writes and reads the full attention matrix from HBM.

The workflow looks roughly like this:

Load Q
Load K

Compute QK^T

Write attention matrix to memory

Read attention matrix

Apply softmax

Write result

Read result

Multiply by V

Write output

A huge amount of time is spent moving data rather than performing useful computation.

This is the problem FlashAttention solves.

3. The Core Insight: Never Materialize the Attention Matrix

The key observation behind FlashAttention is surprisingly simple:

You don't actually need to store the full attention matrix.

Traditional attention explicitly creates:

QK^T

and writes it to memory.

FlashAttention never does.

Instead, it processes attention in small blocks.

Conceptually:

Q Block 1 x K Block 1
Q Block 1 x K Block 2
Q Block 1 x K Block 3
...

Each block is loaded, processed, contributes to the final result, and is discarded.

The gigantic attention matrix never exists in memory.

This immediately removes one of the largest memory bottlenecks in Transformer execution.

4. Why This Is Harder Than It Sounds

At first glance, block-wise processing seems impossible.

The softmax operation requires information from an entire row.

softmax(x_i) = exp(x_i) / sum(exp(x_j))

To compute a single probability, you need a denominator that depends on all scores.

If only one block is visible at a time, how can softmax be computed correctly?

This is where FlashAttention becomes clever.

Instead of storing all scores, it maintains running statistics while processing blocks.

These include:

Running maximum
Running normalization term
Running output accumulator

As each block arrives, these values are updated.

When all blocks have been processed, the result is mathematically identical to standard attention.

Not approximate.

Not close.

Exactly identical.

5. Online Softmax: The Trick That Makes It Work

The breakthrough behind FlashAttention is often called Online Softmax.

Imagine attention scores arriving in chunks:

Block A
Block B
Block C

A naive implementation would need every score before computing softmax.

Online Softmax instead maintains enough information to update the result incrementally.

The algorithm keeps track of:

Current maximum score
Current normalization factor
Current weighted output

When a new block arrives:

Update maximum

Rescale previous values

Accumulate new contributions

This allows FlashAttention to process attention as a stream rather than as a giant matrix.

The memory savings are enormous.

More importantly, the final output is identical to what standard attention would have produced.

6. Tiling: Making GPUs Happy

FlashAttention is often described as an IO-aware algorithm.

IO-aware means the algorithm is designed around memory movement rather than purely around arithmetic complexity.

The implementation uses tiling.

Instead of operating on huge matrices, FlashAttention loads small chunks into fast on-chip memory:

Load tile into shared memory

Compute attention

Update softmax statistics

Accumulate output

Discard tile

Load next tile

Because shared memory is dramatically faster than HBM, the GPU spends much more time performing useful work.

A useful mental model is:

Traditional Attention:

Memory -> Compute -> Memory -> Compute -> Memory

FlashAttention:

Memory -> Compute -> Compute -> Compute

The computation remains approximately O(N^2), but memory traffic is dramatically reduced.

That's where most of the speedup comes from.

7. FlashAttention-2, FlashAttention-3, and Production Systems

The original FlashAttention paper introduced the core idea.

FlashAttention-2 improved:

GPU utilization
Parallelism
Work partitioning
Training throughput

The result was significantly higher performance on modern accelerators.

FlashAttention-3 pushed things further for newer NVIDIA Hopper GPUs and introduced support for modern low-precision formats such as FP8.

Today, FlashAttention is used throughout the AI ecosystem.

You'll find it in:

PyTorch
Hugging Face Transformers
vLLM
TensorRT-LLM
Many open-weight LLMs

For many developers, enabling FlashAttention can be as simple as selecting an optimized attention backend.

The model architecture remains unchanged.

The performance characteristics improve dramatically.

What FlashAttention Does Not Fix

FlashAttention is powerful, but it doesn't magically eliminate all scaling problems.

The computation is still approximately:

O(N^2)

with respect to sequence length.

Very long contexts still require substantial compute.

FlashAttention primarily reduces memory traffic and memory footprint.

It makes attention much more efficient, but it does not change the fundamental quadratic interaction pattern of standard attention.

That's why researchers continue exploring alternatives such as:

Sliding-window attention
Linear attention
State-space models
Hybrid architectures

These approaches attempt to address the computational scaling problem itself rather than the memory movement problem.

Conclusion

FlashAttention is one of the rare breakthroughs that became foundational almost immediately.

It didn't replace Transformers.

It didn't invent a new attention mechanism.

It didn't require retraining models.

Instead, it recognized that modern GPUs spend an enormous amount of time moving data around and redesigned attention to minimize that movement.

By treating memory access as the bottleneck rather than arithmetic, FlashAttention transformed attention from a memory-heavy operation into a much more hardware-efficient one.

Many of today's long-context LLMs would be significantly slower, more expensive, or simply impractical without it.

As AI systems continue to scale, FlashAttention serves as an important reminder that sometimes the biggest breakthroughs don't come from changing the algorithm itself. They come from understanding how that algorithm interacts with real hardware.

What do you think has been the most important infrastructure breakthrough for LLMs so far: FlashAttention, quantization, KV caching, speculative decoding, or something else entirely?

*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

HexmosTech / git-lrc

Free, Micro AI Code Reviews That Run on Commit

git-lrc

Free, Micro AI Code Reviews That Run on Commit

AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.

See It In Action

See git-lrc catch serious security issues such as leaked credentials, expensive cloud operations, and sensitive material in log statements

git-lrc-intro-60s.mp4

Why

🤖 AI agents silently break things. Code removed. Logic changed. Edge cases gone. You won't notice until production.
🔍 Catch it before it ships. AI-powered inline comments show you exactly what changed and what looks wrong.
…

View on GitHub

Mixture of Experts (MoE) Explained Simply: How Modern AI Models Get Bigger Without Getting Slower

Shrijith Venkatramana — Wed, 10 Jun 2026 18:01:16 +0000

Large language models keep getting larger.

Hundreds of billions of parameters. Trillions of parameters. Yet somehow, many of these models remain surprisingly fast and affordable to run.

How?

The trick is that most modern frontier models don't use all of their parameters for every token.

Instead, they use a technique called Mixture of Experts (MoE).

Think of it like replacing a single giant software service with a fleet of specialized microservices. Rather than every request hitting every service, a router decides which specialists should handle a particular request.

That's the core idea behind MoE.

Let's break down how it works, why it matters, and what challenges engineers face when running MoE models in production.

1. The Scaling Problem

Traditional transformer models are dense.

When a token enters a transformer layer:

Attention runs.
The feed-forward network (MLP) runs.
Every parameter in that layer participates in computation.

If you double the model size, you roughly double the compute cost.

This creates a painful tradeoff:

More parameters → better quality
More parameters → slower inference and more expensive training

Researchers wanted a way to increase model capacity without increasing computation proportionally.

MoE emerged as one of the most successful solutions. Instead of activating every parameter, MoE activates only a small subset for each token.

2. The Restaurant Analogy

Imagine a restaurant with eight specialists:

Pizza chef
Sushi chef
Pastry chef
Grill chef
Salad chef
Soup chef
Pasta chef
Dessert chef

When a customer orders pizza, there's no reason for all eight chefs to work on the order.

The restaurant manager simply routes the request to the relevant specialists.

MoE applies the same idea.

Instead of one large neural network handling every token:

Multiple expert networks exist
A router chooses which experts should process each token
Only selected experts perform computation

The result is a model that can contain many more parameters than are actually used for any individual inference step.

3. What Actually Changes Inside a Transformer?

One surprising fact about MoE:

Most of the transformer remains unchanged.

Typically:

Attention layers stay dense
Embeddings stay dense
Normalization layers stay dense

The feed-forward network (MLP) is replaced by a collection of experts.

A standard transformer block looks roughly like:

Input
  ↓
Attention
  ↓
Feed Forward Network
  ↓
Output

An MoE block becomes:

Input
  ↓
Attention
  ↓
Router
  ↓
Selected Experts
  ↓
Combine Outputs
  ↓
Output

Each expert is often just another feed-forward network.

Instead of one MLP, you may have:

Expert 1
Expert 2
Expert 3
...
Expert 64

The router decides which ones should handle each token.

4. How Routing Works

The router is usually a lightweight neural network.

For each token it produces scores:

Token: "database"

Expert 1: 0.05
Expert 2: 0.61
Expert 3: 0.09
Expert 4: 0.25

The model then selects the top experts.

Top-2 Routing

Historically, many MoE systems used Top-2 routing:

Selected:
Expert 2
Expert 4

Both experts process the token.

Their outputs are combined using the router probabilities as weights.

Switch Routing

Later, Google's Switch Transformer simplified this further.

Instead of selecting two experts:

Selected:
Expert 2

Only one expert runs.

This significantly reduces communication and inference overhead while preserving much of the benefit.

5. Why MoE Models Are So Efficient

Let's compare two hypothetical models.

Dense Model

100B parameters
100B active per token

MoE Model

8 experts × 100B parameters
= 800B total parameters

Only 2 experts active
= 200B active parameters

The MoE model can have dramatically larger capacity while activating only a fraction of its weights during inference.

This is often called conditional computation.

Different inputs trigger different computation paths.

The model effectively says:

"Not every problem requires every part of my brain."

This is one reason MoE architectures became attractive for large-scale LLMs. They allow parameter counts to grow much faster than inference cost.

6. The Hidden Engineering Challenges

The basic idea sounds simple.

Production systems quickly reveal the hard parts.

Challenge 1: Expert Collapse

Suppose the router learns:

90% of tokens → Expert 7

Now Expert 7 receives almost all training.

Other experts receive little data and become useless.

Researchers combat this with load-balancing losses that encourage more even utilization.

Challenge 2: Distributed Communication

Imagine:

GPU 1 → Experts 1-8
GPU 2 → Experts 9-16
GPU 3 → Experts 17-24

A batch of tokens may need experts spread across multiple machines.

Now inference becomes a networking problem.

Token activations must be shuffled between devices before expert computation can occur.

In many MoE deployments, communication becomes a significant bottleneck.

Challenge 3: Load Imbalance

Real traffic isn't uniform.

Some experts become hot.

Others remain mostly idle.

This creates GPU utilization problems similar to uneven request distribution in distributed systems.

Modern routing approaches focus heavily on balancing expert workloads.

Challenge 4: Token Dropping

Experts often have limited capacity.

If too many tokens are routed to one expert:

Capacity: 1000 tokens
Incoming: 1500 tokens

Some tokens may need to be rerouted or dropped.

Managing these overflow situations becomes part of production MoE serving infrastructure.

7. What MoE Looks Like in Production

For developers building AI systems, the practical implications are interesting.

Memory Footprint

An MoE model may advertise:

600B parameters

But only a fraction are active for any token.

Compute cost may resemble a much smaller dense model.

Inference Isn't Automatically Cheaper

Many developers assume:

Fewer active parameters
=
Lower latency

Not always.

Routing overhead, expert communication, and distributed synchronization can erase part of the theoretical gain.

Serving MoE efficiently often requires specialized inference stacks.

Observability Matters

Production teams increasingly monitor:

Expert utilization
Router entropy
Token distribution
Expert hot spots
Cross-device traffic

An overloaded expert can become the AI equivalent of a hot database shard.

Routing Becomes Product Behavior

Recent research suggests routing patterns can become task-specific.

Different prompt categories often activate different expert combinations, meaning the routing system itself becomes part of the model's learned intelligence.

Conclusion

Mixture of Experts is one of the most important ideas behind modern large-scale AI systems.

Instead of making every token pass through every parameter, MoE introduces specialization:

Experts perform different computations
Routers choose which experts to use
Only a small subset activates per token

The result is a model that can grow dramatically in total capacity while keeping computation relatively manageable.

For software engineers, MoE feels surprisingly familiar.

It's essentially:

Service routing
Load balancing
Resource scheduling
Distributed systems

...implemented inside a neural network.

As AI systems continue scaling, understanding MoE is becoming as important as understanding transformers themselves.

Question: If you were designing an MoE model, would you optimize primarily for maximum model quality, or for predictable production latency and infrastructure simplicity? Why?

*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

HexmosTech / git-lrc

Free, Micro AI Code Reviews That Run on Commit

git-lrc

Free, Micro AI Code Reviews That Run on Commit

AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.

See It In Action

See git-lrc catch serious security issues such as leaked credentials, expensive cloud operations, and sensitive material in log statements

git-lrc-intro-60s.mp4

Why

🤖 AI agents silently break things. Code removed. Logic changed. Edge cases gone. You won't notice until production.
🔍 Catch it before it ships. AI-powered inline comments show you exactly what changed and what looks wrong.
…

View on GitHub

Speculative Decoding: How LLMs Generate Tokens Faster Without Changing the Answer

Shrijith Venkatramana — Tue, 09 Jun 2026 17:01:40 +0000

Large Language Models keep getting smarter.

But there's a problem: users don't experience intelligence directly. They experience latency.

If a model takes 30 seconds to write an answer instead of 3 seconds, most users won't care that it scored higher on some benchmark.

This creates an interesting engineering challenge:

How do we make LLMs generate text faster without making them worse?

One of the most important techniques to emerge in recent years is speculative decoding.

The idea sounds almost absurd at first:

What if a small model could guess what a large model is about to say, and the large model simply verifies those guesses?

Surprisingly, that's exactly what happens.

Let's see how it works.

The Fundamental Bottleneck of LLM Inference

To understand speculative decoding, we first need to understand why LLMs are slow.

Imagine a model is generating:

The capital of France is Paris.

The model doesn't generate the entire sentence at once.

Instead it generates one token at a time:

The
The capital
The capital of
The capital of France
...

Each new token requires another forward pass through the model.

For a large model with hundreds of billions of parameters, every token is expensive.

This means generation is inherently sequential:

Token 1 → Token 2 → Token 3 → Token 4

You can't generate token 4 until you know token 3.

This sequential nature becomes one of the biggest sources of inference latency.

The Intuition: Let a Smaller Model Predict Ahead

Suppose you have:

A large model (expensive)
A small model (cheap)

The small model is usually less accurate.

But it's often correct about obvious next tokens.

For example:

Prompt:

The capital of France is

Small model prediction:

Paris

Large model prediction:

Paris

Both agree.

Now imagine the small model predicts several tokens:

Paris, which is

Instead of asking the large model for each token individually, we ask it to verify the entire sequence in one pass.

If the predictions are correct, we've effectively skipped multiple expensive decoding steps.

This is the core idea behind speculative decoding.

A Simple Example

Let's say our draft model predicts:

The weather today is sunny and warm.

Tokenized:

sunny
and
warm
.

The large model then evaluates these proposed tokens.

Possible outcome:

Token	Draft Model	Large Model
sunny	✓	✓
and	✓	✓
warm	✓	✓
.	✓	✓

Everything matches.

The large model accepts all four tokens.

Instead of generating four separate tokens sequentially, we've effectively generated four tokens in a single verification step.

That's a major latency reduction.

What Happens When They Disagree?

This is where the algorithm becomes interesting.

Suppose the draft model predicts:

The weather today is rainy and cold.

The large model evaluates the proposal.

rainy   ✓
and     ✓
cold    ✗

The large model agrees until "cold".

At that point:

Accepted tokens are kept.
Incorrect tokens are discarded.
Generation resumes from the first disagreement.

Result:

The weather today is rainy and pleasant.

Only part of the speculation was useful.

But even partial acceptance can significantly improve throughput.

Why This Doesn't Change Model Quality

A common misconception is:

"Aren't we replacing the big model with a smaller model?"

No.

The large model remains the source of truth.

The draft model merely proposes candidates.

The verification process guarantees that the final output follows the same probability distribution as standard decoding.

Conceptually:

Normal Decoding
---------------
Large Model → Token

Speculative Decoding
--------------------
Small Model → Proposed Tokens
Large Model → Verify Tokens
Accepted Tokens → Output

The final answer is still determined by the large model.

The user gets the same quality, but faster.

A Simplified Algorithm

At a high level:

while not finished:

    proposed = small_model.generate(k_tokens)

    verification = large_model.evaluate(proposed)

    accepted = longest_matching_prefix(
        proposed,
        verification
    )

    output.extend(accepted)

    if mismatch:
        output.append(
            large_model.next_token()
        )

In practice, the real algorithm is more sophisticated because it must preserve exact sampling behavior.

But this captures the overall workflow.

Why It Works So Well

Speculative decoding exploits an observation about language:

Most tokens are predictable.

Consider:

Once upon a

Most models will predict:

time

Similarly:

Thank you for your

Likely:

help

Large models spend a surprising amount of compute confirming obvious continuations.

A smaller model can often predict these easy regions accurately.

The large model only needs to intervene when things become ambiguous.

This creates a useful division of labor:

Component	Job
Small Model	Predict likely tokens
Large Model	Verify and correct
User	Receives faster output

Modern Variants

Research and production systems have extended the original idea in several directions.

Self-Speculative Decoding

Instead of using two separate models:

Early layers generate drafts
Full model verifies

This avoids maintaining a second model entirely.

Multi-Token Prediction

Some architectures are trained to predict multiple future tokens directly.

Instead of:

Predict token N+1

They predict:

N+1
N+2
N+3
...

This increases opportunities for speculative execution.

Tree-Based Speculation

Rather than proposing a single sequence:

A → B → C

The draft model proposes multiple branches:

      B1
    /
A --
    \
      B2

The verifier can then select among several possible continuations.

These approaches push throughput even further.

Where You'll Encounter It

Many developers use speculative decoding without realizing it.

Modern inference systems frequently employ variants of:

Server-side LLM inference platforms
High-throughput API providers
Optimized open-source inference engines
Enterprise deployment stacks

Whenever you see a large model streaming unusually quickly, there's a decent chance some form of speculative execution is happening behind the scenes.

It's becoming one of the standard techniques for making frontier models economically viable at scale.

Final Thoughts

Speculative decoding is a beautiful example of an engineering idea that sounds counterintuitive but turns out to be remarkably effective.

Instead of trying to make large models inherently faster, it asks a different question:

What if most of the work they're doing is already predictable?

By letting a smaller model make educated guesses and allowing the larger model to verify them, we can reduce latency dramatically while preserving output quality.

As LLM deployment scales to millions of users and billions of generated tokens, techniques like speculative decoding are likely to matter just as much as advances in model architecture itself.

Question: If you were deploying a large LLM in production, would you prefer investing in a better model, a faster model, or inference optimizations like speculative decoding? Why?

*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

HexmosTech / git-lrc

Free, Micro AI Code Reviews That Run on Commit

git-lrc

Free, Micro AI Code Reviews That Run on Commit

AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.

See It In Action

See git-lrc catch serious security issues such as leaked credentials, expensive cloud operations, and sensitive material in log statements

git-lrc-intro-60s.mp4

Why

🤖 AI agents silently break things. Code removed. Logic changed. Edge cases gone. You won't notice until production.
🔍 Catch it before it ships. AI-powered inline comments show you exactly what changed and what looks wrong.
…

View on GitHub

The Top Golang Mocking Libraries in 2026: A Practical Comparison

Shrijith Venkatramana — Mon, 08 Jun 2026 18:34:47 +0000

A few years ago, choosing a Go mocking framework was mostly a matter of personal preference.

Today, things are different.

Most Go developers have at least one AI coding assistant generating tests alongside them. Some teams even generate the majority of their unit tests automatically. Yet one area remains surprisingly messy: mocks.

Ask an LLM to write a test for the same interface and you'll often get completely different results depending on whether your project uses GoMock, Mockery, MockIO, Minimock, Moq, or hand-written test doubles.

The problem isn't that the models are bad.

The problem is that mocking libraries represent very different philosophies:

Strict vs flexible
Generated vs runtime-created
DSL-heavy vs idiomatic Go
Feature-rich vs minimalist

In this article we'll compare the most popular Go mocking libraries in 2026, examine their strengths and weaknesses, and discuss which one may be the best fit for your project.

What Makes a Good Mocking Library?

Before comparing tools, it's worth defining what matters.

A good mocking library should ideally provide:

Easy mock generation
Clear test failures
Minimal boilerplate
Strong refactoring support
Good IDE experience
Readable tests
Reliable call verification

Different libraries optimize for different parts of this list.

That's why there is no universally correct answer.

1. GoMock: The Enterprise Workhorse

GoMock remains one of the most widely used mocking frameworks in the Go ecosystem.

Originally created by Google and now actively maintained by Uber, it has become the standard choice for many large organizations.

Its philosophy is straightforward: define expectations explicitly and verify them rigorously.

Example

func TestUserService(t *testing.T) {
    ctrl := gomock.NewController(t)

    repo := NewMockUserRepository(ctrl)

    repo.EXPECT().
        GetUser(gomock.Any(), "123").
        Return("John", nil)

    result, _ := service.GetUser("123")

    assert.Equal(t, "John", result)
}

What It Does Well

Excellent matcher support
Strong verification guarantees
Call ordering support
Mature ecosystem
Well understood across large teams

Drawbacks

Requires code generation
Can become verbose
DSL feels heavy in simple tests
Generated files add maintenance overhead

Best Fit

Large codebases where consistency and strictness matter more than simplicity.

2. Testify + Mockery: The Safe Default

If you started a new Go project today and asked ten developers which mocking stack to use, this would probably be the most common answer.

Testify provides assertions and mocking support while Mockery generates mocks from interfaces.

The combination has become the default choice for many teams.

Example

func TestUserService(t *testing.T) {
    repo := mocks.NewUserRepository(t)

    repo.EXPECT().
        GetUser(mock.Anything, "123").
        Return("John", nil).
        Once()

    result, _ := service.GetUser("123")

    assert.Equal(t, "John", result)
}

What It Does Well

Familiar API
Large community
Excellent assertion integration
Good balance between flexibility and verification
Easy onboarding for new developers

Drawbacks

Less strict than GoMock
Generated mocks can grow large
Expectations are easier to misconfigure

Best Fit

Most application teams.

If you're unsure what to choose, this is usually the safest answer.

3. MockIO: The Most Interesting Newcomer

MockIO takes a different approach.

Unlike traditional Go mocking frameworks, it supports runtime-created mocks and offers a modern matcher system inspired by frameworks from other languages.

For developers tired of constantly regenerating mocks, this is immediately appealing.

Example

func TestUserService(t *testing.T) {
    ctrl := mock.NewMockController(
        t,
        mockopts.StrictVerify(),
    )

    repo := mock.Mock[UserRepository](ctrl)

    mock.WhenDouble(
        repo.GetUser(
            mock.AnyContext(),
            mock.Equal("123"),
        ),
    ).ThenReturn("John", nil)

    result, _ := service.GetUser("123")

    assert.Equal(t, "John", result)
}

What It Does Well

Runtime mocks
Rich matcher support
Powerful argument capture
Less dependency on generated code
Modern API design

Drawbacks

Smaller ecosystem
Depends on compiler internals and unsafe features
Less proven in very large codebases

Best Fit

Developers looking for a modern alternative to traditional code-generation workflows.

4. Minimock: Fast and Strict

Minimock focuses on simplicity and performance.

It generates lightweight mocks and automatically verifies expectations when tests finish.

The result is a relatively small API surface with strong guarantees.

Example

func TestUserService(t *testing.T) {
    ctrl := minimock.NewController(t)

    repo := NewUserRepositoryMock(ctrl)

    repo.GetUserMock.
        When(minimock.AnyContext, "123").
        Then("John", nil)

    result, _ := service.GetUser("123")

    assert.Equal(t, "John", result)
}

What It Does Well

Fast execution
Strict verification
Clean generated code
Automatic cleanup integration

Drawbacks

Smaller community
Fewer advanced capabilities
Less flexibility than GoMock

Best Fit

Teams that value strict tests and fast feedback cycles.

5. Moq: The Go-Like Option

Moq has a philosophy that many Go developers appreciate:

Don't build a framework if ordinary Go code can do the job.

Instead of constructing a large expectation DSL, Moq generates structs whose behavior is implemented through functions.

Example

func TestUserService(t *testing.T) {
    repo := UserRepositoryMock{
        GetUserFunc: func(
            ctx context.Context,
            id string,
        ) (string, error) {
            return "John", nil
        },
    }

    result, _ := service.GetUser("123")

    assert.Equal(t, "John", result)
}

What It Does Well

Extremely simple
Minimal abstraction
Highly readable tests
Easy to debug
Feels like ordinary Go

Drawbacks

Limited matcher support
Manual verification is sometimes necessary
Less suitable for highly complex interaction testing

Best Fit

Developers who prefer explicit code over frameworks.

The Bigger Trend: Fewer Mocks, More Fakes

One of the most interesting testing trends in 2026 is that many experienced Go teams are using fewer mocks than they did a few years ago.

Instead of mocking every dependency, they're increasingly creating lightweight in-memory implementations.

For example:

type FakeUserRepo struct {
    users map[string]User
}

func (r *FakeUserRepo) GetUser(
    ctx context.Context,
    id string,
) (User, error) {
    return r.users[id], nil
}

Compared to mocks, fakes often provide:

Better readability
More realistic behavior
Easier maintenance
Reduced brittleness
Better AI-generated tests

Mocks remain valuable for external boundaries:

Payment providers
Email services
Message queues
LLM providers
Third-party APIs

But many teams no longer mock every interface by default.

Which One Should You Choose?

If you're starting a new project today:

Choose GoMock if

You want maximum verification and are working in a large organization.

Choose Testify + Mockery if

You want the safest and most widely adopted option.

Choose MockIO if

You want modern runtime mocking and fewer code-generation steps.

Choose Minimock if

You prioritize speed and strictness.

Choose Moq if

You believe tests should look as much like ordinary Go as possible.

Final Thoughts

The most important shift in Go testing isn't a new mocking framework.

It's that maintainability has become more important than capability.

In 2026, every major mocking library can mock interfaces effectively. The real differentiator is what your tests look like six months later when someone else has to understand them.

The best mocking framework is rarely the one with the longest feature list.

It's the one your team can read, trust, and maintain.

And increasingly, it's the one that both humans and AI assistants can work with comfortably.

What does your team use today: a mocking framework, hand-written fakes, or a mix of both? Have your testing practices changed since AI coding assistants became part of your workflow?

*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

HexmosTech / git-lrc

Free, Micro AI Code Reviews That Run on Commit

git-lrc

Free, Micro AI Code Reviews That Run on Commit

AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.

See It In Action

See git-lrc catch serious security issues such as leaked credentials, expensive cloud operations, and sensitive material in log statements

git-lrc-intro-60s.mp4

Why

🤖 AI agents silently break things. Code removed. Logic changed. Edge cases gone. You won't notice until production.
🔍 Catch it before it ships. AI-powered inline comments show you exactly what changed and what looks wrong.
…

View on GitHub

What's Happening With AI Token Prices, and What It Means for Your Budget

Shrijith Venkatramana — Sun, 07 Jun 2026 15:36:48 +0000

If you prefer watching a video version of this article, check out:

Every few months, we seem to hear two completely different stories about AI pricing.

On one side, AI labs and industry commentators tell us that AI is getting dramatically cheaper. If current trends continue, they argue, we'll eventually stop thinking much about token costs at all because intelligence will be abundant and inexpensive.

On the other side, developers are staring at real AI bills. Teams building agents are running into rate limits. Companies are spending meaningful amounts of money on inference. For many organizations, token costs are still a very real constraint.

So which story is true?

The answer is: both.

To understand why, it helps to look beyond the headlines and examine what's actually happening to the economics of AI inference.

From Software Eating the World to AI Eating Software

Over the last few decades, software transformed almost every industry.

Healthcare, finance, education, transportation, government—software touched all of them. Marc Andreessen famously described this trend as "software eating the world."

Now we may be entering the next phase.

Increasingly, AI agents are starting to eat software itself. Instead of clicking through interfaces and workflows, users can simply tell an AI what they want done and let it handle the work.

That's why token economics matter.

The cost of tokens ultimately determines the cost of the products, agents, and applications being built on top of modern AI models.

Why Token Economics Matters

Take AI-powered code review tools as an example.

At LiveReview, we compete with products like CodeRabbit and other AI-assisted code review platforms. One of the most important questions in that market is simple:

If a customer spends one dollar, how much value do they get?

Can we provide more useful reviews? Better analysis? More actionable feedback?

Those questions are directly tied to token costs because token costs sit underneath almost every AI product.

One of the reasons we're optimistic about the future is that customers should be able to buy more intelligence for the same amount of money over time.

In practical terms, that means the same budget should eventually buy more reviews, more analysis, more automation, and better outcomes.

To understand why, it's worth revisiting a familiar idea: Moore's Law.

Moore's Law and the Economics of Compute

Moore's Law is one of the most important trends in the history of technology.

Originally proposed by Intel co-founder Gordon Moore, it observed that the amount of compute available per dollar roughly doubled every two years.

If one dollar bought 100 units of compute today, then in two years it might buy 200 units. Two years after that, 400 units. Then 800.

That steady increase in compute-per-dollar helped drive the software revolution.

But what's interesting is that parts of the AI industry now appear to be moving even faster than traditional Moore's Law.

The Emergence of Tiered Super-Moore's Law

Recent research has introduced the idea of "Tiered Super-Moore's Law."

The basic observation is straightforward.

New AI capabilities usually appear first in expensive frontier models. Then competition, optimization, infrastructure improvements, and engineering innovations rapidly drive down the cost of delivering those same capabilities.

You can think of the market as having three broad tiers.

Frontier Models

These are the most capable models available.

Think GPT, Claude Opus, Gemini Pro, and other flagship systems operating at the cutting edge.

These models are typically priced above $5 per million input tokens.

Mid-Tier Models

These models offer strong performance at much lower prices.

Pricing generally falls between $0.50 and $5 per million tokens.

Economy Models

These are highly optimized models designed for large-scale production workloads.

Pricing is often below $0.50 per million tokens, making them attractive for repetitive or high-volume tasks.

The key insight is that capabilities don't stay in the frontier tier forever.

A capability that first appears in an expensive model eventually works its way down into cheaper and cheaper models.

AI Prices Are Falling Remarkably Fast

According to the research, frontier and mid-tier model prices have often fallen by 10× to 30× per year.

That's an astonishing rate of change.

Imagine a task that costs $100 today.

If prices fall by 10× over the next year, that same task costs about $10.

Another year of similar improvement brings the cost down to roughly $1.

That's not a small efficiency gain. It's a dramatic reduction in the cost of intelligence.

The pattern is becoming increasingly familiar: new capabilities show up first in expensive models, and then the industry gets very good at making those capabilities cheaper.

How Frontier Capabilities Become Affordable

The process is surprisingly predictable.

A frontier model demonstrates a new capability. Then researchers and engineers figure out how to reproduce similar results more efficiently.

Some of the techniques used include:

Quantization — Reduces the numerical precision of model weights and computations to lower memory usage and speed up inference.
Distillation — Trains a smaller model to mimic the behavior of a larger model, preserving much of its capability at a lower cost.
Mixture-of-Experts architectures — Activates only a subset of specialized model components for each task, reducing compute requirements.
Flash Attention — An optimized attention algorithm that improves speed and memory efficiency when processing long contexts.
Speculative decoding — Uses a smaller model to predict likely outputs ahead of time, allowing the larger model to generate responses faster.
KV-cache optimization — Reuses previously computed attention keys and values so the model does not need to recompute them for every token.
Prompt caching — Stores and reuses computations for repeated prompts, reducing latency and inference costs.
Parameter-efficient fine-tuning — Adapts models to new tasks by updating only a small subset of parameters instead of retraining the entire model.
Model routing — Directs requests to the most appropriate model for a given task, balancing cost, speed, and quality.

Every one of these innovations reduces the amount of compute needed to achieve a given result.

Over time, capabilities that were once expensive become available to almost everyone.

That's one of the reasons AI pricing behaves differently from traditional software pricing. The thing being sold—the intelligence itself—is constantly becoming cheaper to produce.

The Economy Tier Effect

What's particularly interesting is that even economy models continue getting cheaper.

The rate of improvement is slower than at the frontier, but it's still meaningful.

Suppose a task costs $100 today using economy-tier models.

If costs fall roughly 2× per year:

Year 1: $50
Year 2: $25
Year 3: $12.50
Year 4: $6
Year 5: Approximately $3

Five years later, a workload that once cost $100 might cost only a few dollars.

Even if model quality stayed exactly the same, the cost of accessing that quality would continue to decline.

Buying More Intelligence Per Dollar

Another way to think about all of this is that you're buying more intelligence for the same amount of money.

Of course, intelligence isn't a perfectly measurable unit.

But the practical effect is easy to see.

For the same budget, organizations can often:

Process more data
Analyze more context
Run more agents
Generate more outputs
Perform more reviews
Execute more reasoning steps

In other words, the amount of useful cognitive work available per dollar keeps increasing.

What Customers Should Demand

This has important implications for both buyers and builders of AI products.

Customers shouldn't expect AI products to stay static.

If intelligence is getting cheaper, then products should become more valuable over time.

That value might show up as:

More reviews per month
Better analysis
Larger context windows
Improved security scanning
More comprehensive monitoring
Higher reliability
Better user experiences

As the cost of intelligence falls, customers should benefit from those gains.

What This Means for LiveReview

For us at LiveReview, the goal is pretty simple.

As AI economics improve, we want those improvements to show up in the product.

That means:

More code reviews per dollar
More analysis per dollar
Better answers to developer questions
Increased security coverage
Improved production stability
Greater overall effectiveness

The objective isn't simply to capture efficiency gains internally.

The objective is to turn those gains into more value for customers.

Over time, customers should expect to get substantially more from the same subscription spend.

Why This Trend Is Likely to Continue

Many of the techniques responsible for previous cost reductions have already delivered enormous gains.

But there is still plenty of room for improvement.

Future advances may include:

Better model compression techniques, allowing models to deliver similar performance while using fewer resources.
More sophisticated routing systems that direct tasks to the most efficient model for the job instead of always using the most expensive option.
Improved reasoning architectures that can solve complex problems with fewer computational steps.
More efficient inference algorithms that reduce the amount of compute required to generate high-quality outputs.
Hardware improvements, including new generations of AI chips designed specifically for large-scale inference workloads.
Lower energy consumption, which can significantly reduce the operational costs of running AI systems.
Better datacenter utilization, helping providers get more value from existing infrastructure and pass some of those savings on to customers.

Each layer of optimization pushes the cost of intelligence lower.

Conclusion

The conversation around AI costs often sounds contradictory because people are looking at different parts of the same trend.

Yes, AI can still be expensive today.

Yes, developers still hit rate limits and budget constraints.

But the broader pattern is clear: the cost of intelligence is falling rapidly.

New capabilities appear first in expensive frontier models. Then optimization, distillation, compression, competition, and infrastructure improvements make those capabilities dramatically cheaper.

The result is that every year, a dollar tends to buy more intelligence than it did before.

For companies building AI products, the challenge is straightforward: take those falling costs and turn them into better outcomes for customers.

The companies that do that best are likely to be some of the biggest winners in the next phase of AI.

References

Tiered Super-Moore's Law: Price Evolution, Production Frontiers, and Market Competition in Large Language Model Inference Services (arxiv)
The Price of Progress: Price Performance and the Future of AI (arxiv)

*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

HexmosTech / git-lrc

Free, Micro AI Code Reviews That Run on Commit

git-lrc

Free, Micro AI Code Reviews That Run on Commit

AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.

See It In Action

See git-lrc catch serious security issues such as leaked credentials, expensive cloud operations, and sensitive material in log statements

git-lrc-intro-60s.mp4

Why

🤖 AI agents silently break things. Code removed. Logic changed. Edge cases gone. You won't notice until production.
🔍 Catch it before it ships. AI-powered inline comments show you exactly what changed and what looks wrong.
…

View on GitHub

Getting Started with Genkit in Go: Building Production-Ready AI Applications Without Reinventing the Wheel

Shrijith Venkatramana — Sat, 06 Jun 2026 18:18:42 +0000

Large Language Models have made it surprisingly easy to generate text.

Building a reliable AI application, however, is a completely different problem.

Once you move beyond a simple "send prompt, get response" demo, you quickly encounter real-world concerns:

Prompt management
Structured outputs
Multi-step workflows
Tool calling
Observability
Evaluation
Model switching
Production debugging

Many teams end up creating custom frameworks around OpenAI, Anthropic, Gemini, or local models just to manage these concerns.

This is where Genkit comes in.

Originally developed by Google, Genkit provides a framework for building AI-powered applications with a focus on workflows, tooling, observability, evaluation, and production readiness.

While most examples online focus on Node.js, Genkit now has growing support for Go, making it an interesting option for backend engineers who want AI capabilities without introducing an entirely separate application stack.

In this article we'll build practical examples and explore how Genkit helps structure real-world AI systems.

Why Genkit Exists

Most AI applications evolve like this:

Phase 1:

response := callLLM(prompt)

Everything seems simple.

Phase 2:

You need:

Retry logic
Prompt versioning
JSON outputs
Tool integrations
Tracing
Metrics
Human review workflows

Now your codebase starts accumulating AI-specific infrastructure.

Genkit attempts to provide these building blocks from day one.

Think of it as:

"Spring Boot for AI workflows" rather than "an LLM SDK."

Installing Genkit for Go

Create a new project:

mkdir genkit-demo
cd genkit-demo

go mod init github.com/example/genkit-demo

Install Genkit:

go get github.com/firebase/genkit/go/ai

Depending on your provider, you'll also install provider plugins.

For Gemini:

go get github.com/firebase/genkit/go/plugins/googleai

Your First AI Call

Let's start with a simple generation.

package main

import (
    "context"
    "fmt"

    "github.com/firebase/genkit/go/ai"
    "github.com/firebase/genkit/go/genkit"
    "github.com/firebase/genkit/go/plugins/googleai"
)

func main() {
    ctx := context.Background()

    g, err := genkit.Init(ctx,
        genkit.WithPlugins(
            &googleai.GoogleAI{
                APIKey: "YOUR_API_KEY",
            },
        ),
    )

    if err != nil {
        panic(err)
    }

    resp, err := g.Generate(ctx, ai.GenerateRequest{
        Model: "googleai/gemini-2.5-flash",
        Prompt: "Explain vector databases in one paragraph.",
    })

    if err != nil {
        panic(err)
    }

    fmt.Println(resp.Text())
}

This resembles a normal LLM call, but Genkit's value becomes more apparent when applications grow beyond this stage.

Structured Outputs: Stop Parsing AI Text

One of the most common mistakes in AI systems is asking models to return text and then parsing it manually.

Instead of:

Name: John
Score: 87
Risk: Medium

Use schemas.

Imagine a customer-support ticket classifier.

type TicketClassification struct {
    Category string `json:"category"`
    Priority string `json:"priority"`
    Summary  string `json:"summary"`
}

Prompt:

Classify this support ticket.

Return JSON matching the schema.

Now downstream services can safely consume the result.

Real-world uses:

Lead qualification
Risk analysis
Invoice extraction
Customer support routing
Contract review

Structured outputs dramatically reduce prompt fragility.

Building Multi-Step AI Workflows

Most production AI systems involve multiple steps.

Example:

Customer email arrives.

Workflow:

Summarize email
Detect sentiment
Extract action items
Generate response draft
Send for human review

Without a framework:

Controller
 ├─ LLM Call #1
 ├─ LLM Call #2
 ├─ LLM Call #3
 └─ LLM Call #4

Logic becomes difficult to maintain.

With Genkit, you can model the workflow as a flow.

summaryFlow := genkit.DefineFlow(
    g,
    "summarizeCustomerEmail",
    func(ctx context.Context, email string) (string, error) {

        result, err := g.Generate(ctx, ai.GenerateRequest{
            Model: "googleai/gemini-2.5-flash",
            Prompt: "Summarize:\n\n" + email,
        })

        if err != nil {
            return "", err
        }

        return result.Text(), nil
    },
)

Flows become reusable application components rather than scattered LLM calls.

Tool Calling: Let the Model Use Your Systems

A common misconception is that AI models should know everything.

In reality:

Models should reason.

Systems should provide facts.

Imagine an order-tracking assistant.

Instead of teaching the model about orders:

Order #78291
Status: Shipped
Carrier: FedEx
ETA: Tomorrow

Expose a tool.

func GetOrderStatus(orderID string) string {
    return "Shipped"
}

The model decides:

I need order information.
Call tool.
Read result.
Answer user.

This pattern enables:

Database lookups
CRM access
Internal APIs
Inventory systems
Knowledge bases

Many enterprise AI systems are essentially:

LLM + Tools

rather than

LLM + More Prompting

Observability: The Feature Most Teams Discover Too Late

Suppose users report:

"The AI gave a terrible answer."

Without tracing, you're blind.

Questions immediately arise:

Which prompt was used?
Which model answered?
What context was supplied?
Which tool calls executed?
How much did it cost?

Genkit includes observability capabilities that make debugging AI workflows significantly easier.

Traditional debugging:

Error at line 87

AI debugging:

Prompt
→ Context
→ Tool Calls
→ Model Output
→ Final Result

This is often the difference between a manageable production system and weeks of confusion.

Real Example: AI-Powered Incident Summaries

Imagine you're running a platform team.

Every incident generates:

Slack messages
Alerts
Logs
Jira tickets

Engineers spend time creating incident reports.

A Genkit workflow could:

Collect incident data
Summarize timeline
Identify root cause indicators
Draft postmortem
Suggest follow-up actions

Pseudo-flow:

Alerts
   ↓
Summarization
   ↓
Root Cause Analysis
   ↓
Draft Postmortem
   ↓
Engineer Review

This is exactly the type of repeatable, multi-step process where Genkit shines.

Model Portability Matters More Than Most Teams Expect

Early-stage teams often assume they'll stay with one model forever.

Reality:

Pricing changes
New models appear
Performance shifts
Compliance requirements emerge

Today's choice:

Gemini

Six months later:

Anthropic

Twelve months later:

Local model

Frameworks that separate application logic from model providers reduce migration pain.

Genkit encourages this separation.

Your workflow logic remains relatively stable while models evolve underneath.

Common Mistakes When Adopting Genkit

1. Treating It Like Another SDK

Genkit is most valuable when you embrace workflows, tools, schemas, and evaluation.

Using it only for text generation leaves much of its value unused.

2. Over-Automating

Not every process should become autonomous.

Many successful systems use:

AI → Human Review → Action

rather than

AI → Action

3. Ignoring Evaluations

A workflow that works today may degrade after:

Prompt changes
Model upgrades
Data changes

Evaluation should be treated as seriously as unit testing.

Final Thoughts

The AI ecosystem currently has no shortage of model providers.

What many teams actually need is better infrastructure around those models.

Genkit addresses a practical gap between simple API calls and production-grade AI systems. It provides a structured way to build workflows, integrate tools, monitor behavior, and evolve applications as models change.

For Go developers, that's particularly valuable because it allows AI capabilities to live inside existing backend services rather than forcing a separate JavaScript stack.

The interesting question is no longer:

"Which model should I use?"

It's increasingly:

"How do I build a system that can survive five generations of models?"

Frameworks like Genkit are one possible answer.

If you were building an AI-powered product today, which capability would you invest in first:

better models, better prompts, better tools, or better workflows?

And more importantly, which of those do you think will still be a competitive advantage three years from now?

*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

HexmosTech / git-lrc

Free, Micro AI Code Reviews That Run on Commit

git-lrc

Free, Micro AI Code Reviews That Run on Commit

AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.

See It In Action

See git-lrc catch serious security issues such as leaked credentials, expensive cloud operations, and sensitive material in log statements

git-lrc-intro-60s.mp4

Why

🤖 AI agents silently break things. Code removed. Logic changed. Edge cases gone. You won't notice until production.
🔍 Catch it before it ships. AI-powered inline comments show you exactly what changed and what looks wrong.
🔁 Build a…

View on GitHub

Steering Vectors: The Hidden Control Knobs Inside Large Language Models

Shrijith Venkatramana — Thu, 04 Jun 2026 18:52:02 +0000

What if you could change how an AI thinks without retraining it?

Not by rewriting prompts. Not by fine-tuning billions of parameters. Not by collecting another mountain of training data.

Instead, imagine finding a direction inside the model's internal representation space and nudging the model a little in that direction.

A small push.

A different behavior.

This idea sits at the heart of one of the most fascinating areas of modern AI interpretability: steering vectors.

Steering vectors suggest that many behaviors we care about—careful reasoning, honesty, coding style, security awareness, verbosity, and more—may already exist inside a model. The challenge is learning how to activate them.

Let's explore what steering vectors are, how they're created, and why they might become one of the most practical tools for controlling AI systems.

1. What Exactly Is a Steering Vector?

Large language models process information through layers of high-dimensional activations.

At any point during generation, the model's internal state can be represented as a vector containing thousands of numbers.

Researchers discovered something surprising:

Different behaviors often correspond to different regions of this activation space.

For example:

Writing Python code
Solving math problems
Speaking French
Explaining concepts carefully
Producing insecure code

Each tends to produce distinctive activation patterns.

A steering vector is essentially the difference between two activation patterns.

Suppose we gather examples where the model is:

Careful
Methodical
Thorough

and compare them to examples where it is:

Rushed
Superficial
Incomplete

The average difference between these internal states becomes a steering vector.

At inference time, we can add that vector back into the model's activations:

new_activation = activation + α × steering_vector

where α controls the steering strength.

Conceptually, it's like moving the model's internal state toward a desired behavior.

2. Why Steering Vectors Matter

Traditionally, changing model behavior meant:

More training
More data
More compute
More cost

Steering vectors challenge that assumption.

They suggest that many capabilities already exist inside the model and merely need to be activated.

This has an important implication:

The model may know more than it appears to know.

The behavior is already present, but not always dominant.

Instead of teaching the model something new, steering often means amplifying a latent behavior that already exists.

This is one reason steering vectors have attracted significant attention from interpretability researchers.

They provide a glimpse into how concepts may be organized internally.

3. The Most Useful Coding Applications

For software engineering, steering vectors could be particularly valuable.

Careful Code Review

Imagine building a vector from examples of excellent code reviews versus weak reviews.

When applied, the model might become more likely to:

Identify edge cases
Spot race conditions
Notice missing validation
Highlight maintainability concerns

without changing the prompt itself.

Security-Oriented Coding

A vector could be constructed from secure versus insecure implementations.

The model may become more likely to:

Validate inputs
Sanitize outputs
Handle failures explicitly
Avoid common vulnerabilities

Better Refactoring

Some code is technically correct but difficult to maintain.

A refactoring-oriented steering vector could encourage:

Clearer abstractions
Better naming
Simpler control flow
Reduced complexity

Thinking Before Coding

Perhaps the most interesting possibility is steering toward analysis before implementation.

Many coding assistants jump directly into code generation.

A steering vector could encourage the model to spend more effort evaluating requirements, assumptions, and tradeoffs before writing the first line of code.

4. How Researchers Create Steering Vectors

The simplest approach is surprisingly straightforward.

First, collect two sets of examples.

Positive Examples

Examples that exhibit the target behavior.

For example:

High-quality code reviews
Secure implementations
Careful reasoning traces

Negative Examples

Examples lacking that behavior.

For example:

Superficial reviews
Insecure implementations
Rushed solutions

Run both datasets through the model.
Capture activations from a chosen layer.
Compute the average activation for each group.
Subtract one average from the other.

The resulting difference vector becomes the steering vector.

Researchers often call this a contrastive activation difference.

More advanced approaches use:

Linear probes
PCA
Sparse Autoencoders (SAEs)
Contrastive learning techniques

to identify cleaner and more interpretable directions.

5. How Do You Evaluate a Steering Vector?

Creating a steering vector is easy.

Proving it works is much harder.

A common mistake is assuming a behavior improved simply because the output changed.

Researchers typically evaluate steering vectors by running controlled benchmarks.

For example:

Generate a set of coding tasks.
Run the baseline model.
Run the steered model.
Compare measurable outcomes.

Metrics might include:

Bugs discovered
Security issues identified
Test coverage quality
Correctness
False positive rates

Human review is equally important.

Many steering vectors initially appear useful but primarily increase verbosity.

Longer answers often look smarter, even when they aren't.

A good evaluation distinguishes genuine capability improvements from stylistic changes.

6. Where Steering Vectors Are Heading Next

The most exciting research is moving beyond single dense vectors.

A common criticism of steering vectors is that they often blend multiple concepts together.

A "careful reasoning" vector might simultaneously influence:

Length
Formality
Confidence
Attention to detail

Recent interpretability work attempts to break these behaviors into smaller, more precise features.

Instead of steering toward a broad concept like "good coding," future systems may activate specific internal features such as:

Checking edge cases
Searching for counterexamples
Validating assumptions
Looking for security risks

The long-term vision is not merely controlling outputs.

It is understanding and controlling the internal computations that generate those outputs.

If successful, steering could become one of the most practical bridges between interpretability research and real-world AI systems.

Final Thoughts

Steering vectors reveal something profound about large language models.

Many behaviors that appear mysterious from the outside may correspond to surprisingly simple geometric directions on the inside.

We are still far from fully understanding these representations.

But the idea that a model's behavior can be altered by moving through activation space—without retraining and sometimes without even changing the prompt—offers a fascinating glimpse into how intelligence may be organized inside neural networks.

And perhaps more importantly, it suggests that the future of AI control might involve understanding the model's internal world rather than merely observing its outputs.

Question: If you could build a steering vector for your coding assistant today, what behavior would you choose: deeper reasoning, stronger security awareness, better code reviews, more maintainable code, or something else entirely?

*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

HexmosTech / git-lrc

Free, Micro AI Code Reviews That Run on Commit

git-lrc

Free, Micro AI Code Reviews That Run on Commit

AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.

See It In Action

See git-lrc catch serious security issues such as leaked credentials, expensive cloud operations, and sensitive material in log statements

git-lrc-intro-60s.mp4

Why

🤖 AI agents silently break things. Code removed. Logic changed. Edge cases gone. You won't notice until production.
🔍 Catch it before it ships. AI-powered inline comments show you exactly what changed and what looks wrong.
🔁 Build a…

View on GitHub

Use claude-lrc Reviews to Make Your Prod More Stable

Shrijith Venkatramana — Wed, 03 Jun 2026 19:24:38 +0000

With tools like Claude, code generation is now fast and relatively inexpensive. For decades, writing code was expensive and reviewing it was relatively cheap, and most engineering processes were built around that reality.

A handful of senior engineers could review the output of many developers because code generation itself was the constraint.

Today, a single engineer can generate migrations, APIs, tests, infrastructure code, documentation, and refactors in a fraction of the time it previously required.

AI has made code generation abundant.

Most engineering processes are still adapting.

Code generation has accelerated dramatically.

Scrutiny has become the scarce resource.

The New Failure Mode

The biggest risk with AI-generated code comes from assumptions that never receive enough scrutiny.

Humans have always written imperfect code.

Production stability usually comes from somebody asking difficult questions.

Claude-generated code often looks reasonable.

It compiles.

It passes tests.

It follows conventions.

Production failures often emerge from assumptions that escaped scrutiny.

A retry loop that amplifies load.
A missing authorization check.
A concurrency edge case.
A migration that works in staging but struggles in production.
A cache invalidation strategy that quietly fails under real traffic.

As AI increases code output, the amount of code requiring scrutiny grows as well.

The same reviewers, staff engineers, security engineers, and technical leads are expected to evaluate far more than before.

Review quality declines when reviews become rushed, and delivery slows when reviews become overloaded.

Neither outcome improves software quality.

Centralized Review Breaks Down. So: Decentralize It!

A small group of senior engineers can no longer carry the entire review burden.

That model fit an environment where code generation was naturally constrained.

Today, Claude and other AI tools allow every developer to produce significantly more code than before.

The path forward involves spreading scrutiny throughout the development process.

More developers need access to the kinds of questions experienced reviewers ask.

More assumptions need examination before code reaches a formal review queue.

More scrutiny needs to happen across the team instead of flowing through a handful of overloaded people.

Review must scale.

And scaling review means decentralizing it.

Human reviewers remain essential.

Their expertise has greater impact when scrutiny becomes as natural to each individual as brushing their teeth daily.

Review Must Become as Ubiquitous and Obvious as Brushing One's Teeth

The cheapest and most effective review happens before code gets committed.

Developers benefit from challenging assumptions while they still have context and while fixes still take minutes.

Instead of waiting for formal review, they can ask:

What risks am I missing?
What assumptions am I making?
What would a skeptical reviewer question?
What reliability issues exist here?
What security concerns should I investigate further?

These are exactly the kinds of questions developers can ask Claude while they're still building.

The earlier these questions are asked, the easier they are to answer.

Broader review throughout development creates more opportunities to catch issues early.

Frequent scrutiny while context is fresh helps teams move faster with greater confidence.

Review as Engineering Hygiene

Review is most valuable when it becomes a routine habit rather than a special event.

Nobody waits until they have a cavity before brushing their teeth.

The value comes from small preventative actions performed consistently.

Software quality works the same way.

A five-minute review before a commit can prevent hours of debugging after deployment.

A quick challenge to an assumption can eliminate the need for an incident retrospective.

The best production incident is the one that never happens.

That requires making review part of everyday engineering work.

Where claude-lrc Fits

claude-lrc makes continuous review practical by helping developers challenge assumptions while they work.

The goal is continuous scrutiny that fits naturally into development.

The goal is review that developers can perform regularly.

Inside Claude Code.

Using natural language.

Using slash commands.

Using workflows they already use every day.

Alongside code generation, claude-lrc helps developers examine the code they create.

Throughout development, claude-lrc helps distribute review across many small moments of scrutiny.

Developers can challenge assumptions earlier.

Teams can surface risks before formal review.

Senior engineers can spend more time on high-value judgment and architectural decisions.

Small reviews.

Frequent reviews.

Just like running tests, review should become a normal part of building software.

The Next Engineering Habit

Continuous review is becoming the next standard engineering practice.

Version control became standard.

Testing became standard.

Continuous integration became standard.

Observability became standard.

AI-assisted development is creating demand for another standard practice.

Claude can help generate more code than ever before.

Delivering reliable software still depends on careful evaluation and sound judgment.

Human attention remains a finite resource.

The teams that succeed will scale scrutiny alongside generation.

That means making review continuous.

And making it available to everyone, not just reviewers.

We already treat testing as a continuous activity.

Should code review become continuous too?

The Most Surprising Idea in AI Image Generation: Sculpting Meaning from Noise

Shrijith Venkatramana — Tue, 02 Jun 2026 17:08:22 +0000

When most developers first encounter AI image generation, they imagine something like a digital artist.

You type:

A red Ferrari on a mountain road at sunset

and somewhere inside the model, it must be drawing wheels, painting reflections, and composing a scene.

But that's not what happens.

In many modern image generation systems, the process begins with something much stranger: pure static. Random noise. The visual equivalent of a detuned television.

The AI doesn't start with a Ferrari.

It starts with chaos.

Then, step by step, it sculpts that chaos into an image that matches your prompt.

Once you understand this idea, many things about modern AI suddenly make more sense.

The Counterintuitive Discovery: Destroying Images Is Easy

Imagine you have a photograph of a Ferrari.

Now add a tiny amount of random noise.

Then add a little more.

Then more.

Eventually, the image becomes indistinguishable from static.

Ferrari
   ↓
Slightly Noisy Ferrari
   ↓
Very Noisy Ferrari
   ↓
Static

This process is trivial. Anyone can write it in a few lines of code.

What Jascha Sohl-Dickstein and his collaborators realized in 2015 was that while the forward process is easy, perhaps the reverse process could be learned.

In other words:

If we know how to destroy structure, can a neural network learn how to rebuild it?

That simple question eventually became the foundation of diffusion models.

The Sculptor Analogy That Finally Made It Click for Me

Many explanations describe diffusion as "removing noise."

Technically correct.

But I think there's a better mental model.

Imagine a sculptor standing beside a random lump of clay.

The prompt doesn't create the clay.

The prompt tells the sculptor what to carve.

Random Clay
      +
"Ferrari"
      ↓
Sculpting
      ↓
Ferrari

The same thing happens in diffusion models.

The initial noise is usually independent of the prompt.

Instead, the prompt influences every refinement step afterward.

The model repeatedly asks:

If this image is supposed to become a Ferrari, what should I change next?

Over time, rough shapes emerge.

Then wheels.

Then reflections.

Then details.

The image isn't retrieved from memory. It's progressively constructed.

The Training Loop Is Shockingly Small

One of the most surprising things about diffusion models is how simple the core training loop is.

At a high level:

for image, caption in dataset:

    t = random_timestep()

    noise = randn_like(image)

    noisy_image = add_noise(image, noise, t)

    predicted_noise = model(
        noisy_image,
        caption,
        t
    )

    loss = mse(predicted_noise, noise)

    loss.backward()

That's essentially the whole idea.

The model is not directly learning:

Draw a Ferrari.

It is learning:

Given a noisy image and a caption, predict what noise was added.

That sounds almost too simple.

Yet the behavior that emerges is remarkable.

Why Generation Is Just Training in Reverse

During inference, we start from pure noise.

Then we repeatedly ask the model what noise should be removed.

Conceptually:

image = random_noise()

for t in reversed(range(T)):

    predicted_noise = model(
        image,
        prompt,
        t
    )

    image = remove_noise(
        image,
        predicted_noise,
        t
    )

return image

The process looks like this:

Static
   ↓
Less Static
   ↓
Rough Shapes
   ↓
Car-Like Shapes
   ↓
Ferrari

The model performs the same skill it learned during training.

The only difference is where the noisy image came from.

During training, the noise came from a real photograph.

During inference, it comes from randomness itself.

Does the Noise Secretly Contain the Ferrari?

This is one of the most common misconceptions.

No.

The starting noise doesn't secretly contain a hidden Ferrari.

It is genuinely random.

However, the noise acts as a seed.

Consider two different random starting points:

Noise A → Ferrari A
Noise B → Ferrari B

Same prompt.

Different image.

Different camera angle.

Different lighting.

Different details.

The prompt answers:

What should this image become?

The random seed answers:

Which version of that thing?

In practice, both matter.

The prompt provides direction.

The noise provides variation.

Where Do Transformers Enter the Picture?

Many developers assume diffusion models replaced transformers.

Not exactly.

In most modern text-to-image systems, they work together.

A simplified architecture looks like this:

Prompt
   ↓
Transformer
   ↓
Text Embedding
   ↓
Diffusion Model
   ↓
Image

The transformer's job is understanding language.

It learns relationships such as:

Ferrari is a car
Red modifies Ferrari
Mountain road describes the scene

The diffusion model then uses that understanding while denoising.

At every step, the image generation process is guided by the prompt representation produced by the transformer.

One model understands meaning.

The other turns that meaning into pixels.

The Unusual Origin Story: A Physics Idea That Changed AI

What makes diffusion particularly interesting is where it came from.

Jascha Sohl-Dickstein's 2015 paper wasn't framed primarily as a computer vision breakthrough.

It drew heavily from ideas in statistical physics and nonequilibrium thermodynamics.

The original insight was not:

How do we draw images?

It was closer to:

How do complex probability distributions evolve into simple ones?

And then:

Can we learn the reverse transformation?

That shift in perspective is what makes the idea feel so elegant.

Many breakthroughs happen when someone enters a field carrying mental models from another discipline.

Diffusion models are a great example.

They treat image generation not as drawing, but as reversing a physical process.

Final Thoughts

The next time an image model generates a stunning scene from a short prompt, it's worth remembering what happened under the hood.

The system didn't start with a sketch.

It didn't search a database for the closest image.

It started with randomness.

Then, hundreds of times in succession, it made tiny corrections guided by your prompt.

A Ferrari emerged from static.

A castle emerged from noise.

Meaning emerged from chaos.

And perhaps that's the most surprising part of all.

If you had been working on image generation in 2014, would you have tried to teach a model how to draw images—or would it ever have occurred to you to teach it how to remove noise instead?

Building Truly Cross-Platform Claude Code Hooks with Go, Bash, PowerShell, WSL, and Git-Bash

Shrijith Venkatramana — Sun, 31 May 2026 16:57:59 +0000

Claude Code hooks are powerful. They let you intercept tool execution, enforce policies, run validations, collect telemetry, or integrate external systems before and after Claude performs actions.

Unfortunately, the moment you try to distribute hooks to real developers, you run into a problem:

Some developers use Linux
Some use macOS
Some use Windows PowerShell
Some use Git-Bash
Some use WSL
Some use combinations of all of the above

A simple shell script quickly turns into a compatibility nightmare.

After experimenting with several approaches, I arrived at a surprisingly effective pattern:

Use thin platform-specific wrappers whose only job is downloading and launching a Go binary. Put all real logic inside the Go executable.

This gives you the convenience of native hooks while keeping the implementation portable, testable, and maintainable.

Let's walk through the architecture.

The Cross-Platform Hook Problem

Suppose you build a hook that validates commands before Claude executes them.

The naive implementation might look like this:

#!/usr/bin/env bash

python validate.py

Looks fine until:

Python isn't installed
The user runs PowerShell
The user runs Git-Bash
The user runs WSL
Path handling differs
Quoting rules differ

Now you have:

validate.sh
validate.ps1
validate.py
requirements.txt

And eventually:

works-on-my-machine/

The problem isn't Claude.

The problem is that shells are operating-system specific.

The Better Architecture

Instead, think of the hook as a bootstrapper.

Claude Hook
     |
     v
Thin Wrapper
     |
     v
Download Go Binary (if needed)
     |
     v
Execute Go Binary
     |
     v
Actual Hook Logic

The wrapper becomes extremely small.

The Go executable contains:

Policy checks
Configuration loading
JSON parsing
API calls
Logging
Cross-platform filesystem access
Everything else

Once the binary exists locally, future hook invocations bypass installation entirely.

Bootstrapping on First Run

The wrapper checks whether the executable exists.

If not:

Detect platform
Download correct binary
Make executable if needed
Run binary

Example release layout:

releases/
├── hook-linux-amd64
├── hook-linux-arm64
├── hook-darwin-amd64
├── hook-darwin-arm64
├── hook-windows-amd64.exe
└── hook-windows-arm64.exe

A GitHub Releases page works perfectly for hosting.

Example Bash wrapper:

#!/usr/bin/env bash

set -e

HOOK_DIR="$HOME/.claude-hooks"
BIN="$HOOK_DIR/hook"

mkdir -p "$HOOK_DIR"

if [ ! -f "$BIN" ]; then
    curl -L \
      https://example.com/hook-linux-amd64 \
      -o "$BIN"

    chmod +x "$BIN"
fi

exec "$BIN" "$@"

The wrapper is tiny and almost never changes.

Supporting PowerShell

Windows users deserve first-class support.

PowerShell wrapper:

$HookDir = "$env:USERPROFILE\.claude-hooks"
$Binary = "$HookDir\hook.exe"

New-Item `
  -ItemType Directory `
  -Force `
  -Path $HookDir | Out-Null

if (!(Test-Path $Binary)) {

    Invoke-WebRequest `
        -Uri "https://example.com/hook-windows-amd64.exe" `
        -OutFile $Binary
}

& $Binary $args
exit $LASTEXITCODE

The important detail is that PowerShell's quoting rules differ significantly from Bash.

By moving all logic into Go, you avoid maintaining duplicate implementations.

Handling WSL and Git-Bash

This is where things become interesting.

Many Windows developers don't actually run PowerShell.

They run:

WSL Ubuntu
Git-Bash
MSYS2
Cygwin

Each environment reports itself differently.

A good Go bootstrapper can detect them.

Example:

func detectEnvironment() string {

    if runtime.GOOS != "windows" {
        return "native"
    }

    if os.Getenv("WSL_DISTRO_NAME") != "" {
        return "wsl"
    }

    if os.Getenv("MSYSTEM") != "" {
        return "git-bash"
    }

    return "powershell"
}

You can then adjust behavior.

For example:

switch detectEnvironment() {

case "wsl":
    // Linux paths

case "git-bash":
    // Mixed Windows/POSIX paths

case "powershell":
    // Native Windows paths
}

This is dramatically easier than maintaining separate shell implementations.

Cross-Platform Techniques in Go

The Go standard library already solves most portability issues.

Use filepath

Avoid hardcoded separators.

Bad:

path := home + "/config/settings.json"

Good:

path := filepath.Join(
    home,
    "config",
    "settings.json",
)

Use os.UserHomeDir

Avoid platform assumptions.

home, err := os.UserHomeDir()

Works on:

Linux
macOS
Windows
WSL

Use os.Executable

Finding your own binary location:

exe, err := os.Executable()

Useful when loading bundled resources.

Detect Operating System

switch runtime.GOOS {

case "windows":
    // Windows

case "linux":
    // Linux

case "darwin":
    // macOS
}

Detect Architecture

fmt.Println(runtime.GOARCH)

Possible values:

amd64
arm64
386

Useful for selecting downloads.

Full Example Downloader

A minimal self-updating launcher:

package main

import (
    "io"
    "net/http"
    "os"
)

func download(url string, dest string) error {

    resp, err := http.Get(url)
    if err != nil {
        return err
    }
    defer resp.Body.Close()

    out, err := os.Create(dest)
    if err != nil {
        return err
    }
    defer out.Close()

    _, err = io.Copy(out, resp.Body)

    return err
}

Combined with:

runtime.GOOS
runtime.GOARCH

you can dynamically fetch the correct binary.

Why This Pattern Scales Better

The biggest benefit isn't portability.

It's maintainability.

Without this pattern:

hook.sh
hook.ps1
hook.py
hook.js

With this pattern:

hook.sh      (tiny)
hook.ps1     (tiny)

hook-go/
    all logic

The wrappers rarely change.

The Go binary evolves independently.

Testing becomes easier.

Distribution becomes easier.

Versioning becomes easier.

And most importantly, you stop fighting shell differences.

Final Thoughts

Many engineering teams start by writing Claude hooks as shell scripts because it feels fast.

That works for one machine.

The moment multiple operating systems enter the picture, the complexity grows rapidly.

A small bootstrap wrapper plus a Go executable gives you a surprisingly robust deployment model:

Bash support
PowerShell support
Linux support
macOS support
Windows support
WSL support
Git-Bash support
Single implementation of business logic

The shell becomes a launcher.

Go becomes the platform.

That's usually the point where hook maintenance stops being a headache.

How are you handling cross-platform automation today—shell scripts, Node.js, Python, or compiled binaries? I'd be interested to hear which approach has held up best as your team and environments grew.

*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

HexmosTech / git-lrc

Free, Micro AI Code Reviews That Run on Commit

git-lrc

Free, Micro AI Code Reviews That Run on Commit

AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.

See It In Action

See git-lrc catch serious security issues such as leaked credentials, expensive cloud operations, and sensitive material in log statements

git-lrc-intro-60s.mp4

Why

🤖 AI agents silently break things. Code removed. Logic changed. Edge cases gone. You won't notice until production.
🔍 Catch it before it ships. AI-powered inline comments show you exactly what changed and what looks wrong.
🔁 Build a…

View on GitHub

Lean4 Might Be the Missing Piece in AI: Why Theorem Provers Are Suddenly Everywhere

Shrijith Venkatramana — Sat, 30 May 2026 17:23:37 +0000

Most discussions about AI focus on larger models, larger datasets, and larger GPUs.

But there is an uncomfortable reality that every engineer building production AI systems eventually runs into:

LLMs can produce convincing answers, but they cannot guarantee correctness.

Ask an LLM to write code, reason about a distributed system, derive a mathematical formula, or analyze a security protocol. The result might be brilliant. It might also be subtly wrong.

The problem isn't intelligence.

The problem is verification.

That is why a relatively obscure technology from the world of formal methods is suddenly attracting attention:

Lean4.

A theorem prover originally designed for mathematicians is increasingly being viewed as a way to build AI systems that can not only generate answers, but actually prove that those answers are correct.

Let's look at what Lean4 is, how it works, and why some researchers believe theorem provers may become a critical layer in future AI systems.

The Fundamental Problem: LLMs Don't Know What's True

Large language models operate by predicting likely sequences of tokens.

That sounds obvious, but the implications are important.

When ChatGPT generates a response, it isn't checking whether a statement is true.

It is generating text that statistically resembles text associated with the prompt.

Consider a simple coding example:

def is_sorted(arr):
    return all(arr[i] < arr[i+1]
               for i in range(len(arr)-1))

Looks reasonable.

But there is a subtle bug.

[1,1,2,3]

is sorted, yet the function returns False because it uses < instead of <=.

Many tests might pass.

A code reviewer might miss it.

An LLM might confidently explain why the implementation is correct.

None of these establish correctness.

Testing can show the presence of bugs.

It cannot prove the absence of bugs.

That distinction is what theorem proving is about.

What Exactly Is Lean4?

Lean4 is two things:

A programming language
A theorem prover

The theorem prover part is the interesting piece.

Instead of writing code and then testing it, you describe properties that must always hold.

Lean then requires a mathematical proof that those properties are true.

For example, consider a simple theorem:

For every natural number n, n + 0 = n

In Lean this becomes something that must be formally proven.

The system does not accept hand-wavy reasoning.

Every logical step must be justified.

If any step is invalid, the proof fails.

This is fundamentally different from traditional software validation.

Traditional testing:

Input A -> Pass
Input B -> Pass
Input C -> Pass

Formal proof:

For all valid inputs:
    Property P always holds

The theorem checker verifies the proof mechanically.

No intuition.

No assumptions.

No trust.

Only proof.

Why Lean Feels Different From Traditional Formal Methods

Formal verification has existed for decades.

Historically it suffered from two problems:

Tools were difficult to use
Formalization was extremely expensive

Lean changes the equation in several ways.

First, it is designed as a practical programming language.

Second, it has a large ecosystem called Mathlib containing thousands of formally verified definitions and theorems.

Instead of proving everything from scratch, developers can build on existing verified foundations.

For example:

Natural numbers
Integers
Groups
Rings
Calculus
Probability
Linear algebra

Much of this already exists inside the ecosystem.

This makes Lean feel closer to software engineering than traditional theorem proving systems.

You are often composing verified building blocks rather than creating everything from first principles.

The AI + Lean Workflow Is What Makes This Interesting

The most exciting development is not Lean itself.

It's the combination of Lean and LLMs.

Think about the typical AI workflow today:

Prompt
    ↓
LLM
    ↓
Answer

Now compare that with an emerging architecture:

Prompt
    ↓
LLM
    ↓
Candidate Solution
    ↓
Lean
    ↓
Verification
    ↓
Accepted / Rejected

The LLM becomes a generator.

Lean becomes a verifier.

This separation is powerful.

Humans already work this way.

A mathematician may invent a proof.

A journal referee verifies it.

An engineer may write code.

Tests verify it.

An architect proposes a design.

Structural calculations verify it.

The same pattern can apply to AI systems.

Generation and verification become separate concerns.

A Concrete Example: Finding Bugs Automatically

Imagine an LLM generating a sorting algorithm.

The desired property is:

For any list L:

sort(L) returns:
    1. A permutation of L
    2. Elements in non-decreasing order

An LLM might generate:

def sort(xs):
    return sorted(set(xs))

At first glance it appears to work.

But duplicates disappear.

[1,1,2]

becomes:

[1,2]

The algorithm violates the permutation property.

A theorem prover can catch this immediately.

The interesting part is that verification is not based on finding a counterexample through testing.

The proof obligation itself fails.

The algorithm cannot be proven correct.

This is fundamentally stronger than conventional testing approaches.

Why This Matters Beyond Mathematics

Many people hear "theorem prover" and assume this is only useful for mathematicians.

That is increasingly false.

Formal verification is already used in areas such as:

Compilers

The famous CompCert compiler demonstrates that compiler correctness can be formally proven.

Cryptography

Security protocols often rely on formal proofs.

A tiny mistake can compromise billions of dollars.

Aerospace

Flight control systems require exceptionally high confidence.

Finance

Smart contracts and trading infrastructure can benefit from machine-checked guarantees.

AI Agents

Agents increasingly perform actions instead of merely generating text.

As autonomy increases, verification becomes more valuable.

The more expensive a mistake becomes, the more attractive formal guarantees become.

The Bigger Picture: Probabilistic Intelligence + Deterministic Verification

There is a tendency to think of theorem provers and LLMs as competing technologies.

They're not.

In many ways they complement each other.

LLMs are excellent at:

Search
Exploration
Creativity
Pattern matching
Generating candidate solutions

Theorem provers are excellent at:

Verification
Correctness
Logical consistency
Mathematical guarantees

One generates.

The other validates.

A useful analogy is software development itself.

We don't replace programmers with compilers.

We use compilers to verify what programmers produce.

Future AI systems may look similar:

LLM = Generator

Theorem Prover = Verifier

The combination is potentially far more powerful than either component alone.

Final Thoughts

For years the AI industry has largely optimized for capability.

Can the model write code?

Can it solve math problems?

Can it reason?

Those are important questions.

But another question is becoming increasingly important:

How do we know the answer is actually correct?

Theorem provers such as Lean4 offer one possible answer.

They provide a mechanism for transforming "the model thinks this is right" into "this has been formally verified."

Whether Lean itself becomes dominant remains to be seen.

But the broader idea—combining probabilistic generation with formal verification—feels less like a niche research direction and more like a plausible next step in the evolution of AI systems.

What do you think?

Will theorem provers become a standard component of future AI stacks, or will they remain specialized tools used only in high-assurance domains?

*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

HexmosTech / git-lrc

Free, Micro AI Code Reviews That Run on Commit

git-lrc

Free, Micro AI Code Reviews That Run on Commit

AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.

See It In Action

See git-lrc catch serious security issues such as leaked credentials, expensive cloud operations, and sensitive material in log statements

git-lrc-intro-60s.mp4

Why

🤖 AI agents silently break things. Code removed. Logic changed. Edge cases gone. You won't notice until production.
🔍 Catch it before it ships. AI-powered inline comments show you exactly what changed and what looks wrong.
🔁 Build a…

View on GitHub

I Pointed Chrome's Prompt API at a 1.25 Million Character Memoir, and It Got Interesting Fast

Shrijith Venkatramana — Fri, 29 May 2026 18:36:12 +0000

A straightforward engineering question: what happens when you feed a long book to an on-device language model in Chrome and start adjusting the parameters?

To explore this, I built a small experiment called Gemini Nano Book Lab: a Chrome extension sidepanel that uses Chrome’s built-in Prompt API to answer questions about Richard Wagner’s My Life, while also exposing some of the underlying mechanics.

The response is only part of it. The experiment also captures:

Model download behavior
Retrieval cost
Time to first token
Context window pressure
Effects of different chunking strategies
Places where the API works well, and where its limits become obvious

If you’re an engineer interested in systems that have rough edges—and therefore teach you something—this is a useful area to explore.

What the Prompt API Is

Chrome’s Prompt API is part of the browser’s built-in AI features. Instead of sending prompts to a cloud endpoint, a web app or extension can request an on-device language model session and prompt it locally.

Resources:

Core capabilities:

Local inference
Streaming results
Availability check before session creation
Context usage measurement
Events like contextoverflow
(In some environments) sampling parameters like temperature and top-k

This makes it more than a simple text box—it becomes an environment for experimentation.

Why a Long Book?

Long inputs expose the interesting problems. Short prompts hide a lot; a paragraph‑long demo can make any model look magical. A long corpus forces concrete decisions:

What chunk size works well?
Should chunks overlap?
How many chunks should you retrieve?
What latency comes from retrieval vs. prompting?
How much of the context window is spent just staging evidence?
When does naive retrieval return technically relevant but semantically weak passages?

For the first version, I used Project Gutenberg’s plain text of Richard Wagner’s My Life:

Project Gutenberg source text

That gave a corpus of about 219,572 words and 1,251,663 characters in the run shown below.

The Setup

The demo is a Chrome extension sidepanel rather than a normal web app. This was a deliberate choice. Extensions provide a more reliable built‑in AI surface in Chrome, and they allow a compact benchmark UI where controls, streamed output, and telemetry live side by side.

The extension has three tasks:

Load and normalize the Wagner text.
Chunk it and retrieve relevant excerpts for a user question.
Create a Prompt API session, run the prompt, and record timings and context usage.

Benchmark Design

The benchmark starts simple. I didn’t begin with embeddings, vector databases, or sophisticated semantic retrieval. I wanted a baseline that is easy to reason about.

The first‑version controls are:

question text
chunk size (characters)
chunk overlap (characters)
number of retrieved chunks
streaming on/off
temperature and top‑k (when exposed)

This provides enough surface to see the tradeoffs without making the experiment too complex.

Prompt API Availability and Session Creation

The first question isn’t “What should I prompt?” but “Is the model available here?”

Here’s the availability and session setup wrapper:

function getPromptApi(): PromptApi | null {
    const maybePromptApi = (globalThis as typeof globalThis & {
        LanguageModel?: PromptApi
    }).LanguageModel
    return maybePromptApi ?? null
}

export async function inspectPromptApi(): Promise<PromptApiCapabilities> {
    const promptApi = getPromptApi()

    if (!promptApi) {
        return {
            supported: false,
            availability: 'unavailable',
            statusMessage:
                'LanguageModel is unavailable in this browser context. Use a recent Chrome build with the Prompt API enabled.',
            defaultTemperature: null,
            maxTemperature: null,
            defaultTopK: null,
            maxTopK: null,
        }
    }

    const availability = await promptApi.availability({
        expectedInputs: [{ type: 'text', languages: ['en'] }],
        expectedOutputs: [{ type: 'text', languages: ['en'] }],
    })

    return {
        supported: true,
        availability,
        statusMessage:
            availability === 'available'
                ? 'Prompt API ready.'
                : 'Model can be downloaded or is unavailable on this device.',
        defaultTemperature: null,
        maxTemperature: null,
        defaultTopK: null,
        maxTopK: null,
    }
}

This may not look exciting, but it matters. One early lesson with built‑in AI is that availability is part of your product surface. Hardware support, model download state, and browser support determine whether your app works at all.

Chunking a Large Corpus

After loading the book, I split it into overlapping chunks. The code tries to respect paragraph and sentence boundaries rather than slicing blindly at exactly N characters.

export function buildChunks(
    text: string,
    chunkSize: number,
    overlap: number,
): CorpusChunk[] {
    const safeChunkSize = Math.max(600, chunkSize)
    const safeOverlap = clampOverlap(safeChunkSize, overlap)
    const chunks: CorpusChunk[] = []

    let startOffset = 0
    let index = 0

    while (startOffset < text.length) {
        const desiredEnd = Math.min(text.length, startOffset + safeChunkSize)
        const endOffset =
            desiredEnd === text.length
                ? text.length
                : findBoundary(text, startOffset, desiredEnd)

        const textSlice = text.slice(startOffset, endOffset).trim()

        if (textSlice) {
            index += 1
            chunks.push({
                id: `chunk-${String(index).padStart(3, '0')}`,
                index,
                text: textSlice,
                startOffset,
                endOffset,
            })
        }

        if (endOffset >= text.length) {
            break
        }

        startOffset = Math.max(endOffset - safeOverlap, startOffset + 1)
    }

    return chunks
}

This decision changes the system’s behavior. Small chunks improve precision but can break context apart. Large chunks preserve narrative structure but use more context budget. Overlap helps with boundaries but increases repeated text and token pressure. Engineering often comes down to choosing which kind of trade‑off you can accept.

Cheap Retrieval on Purpose

The first retriever is lexical, not semantic. That keeps the failure modes visible. If retrieval is too smart too early, you skip an educational stage.

export function rankChunks(
    chunks: CorpusChunk[],
    query: string,
    maxChunks: number,
): RankedChunk[] {
    const queryTokens = tokenize(query)

    return chunks
        .map((chunk) => {
            const { score, matchedTerms } = scoreChunk(chunk, queryTokens, query)
            return {
                ...chunk,
                score,
                matchedTerms,
            }
        })
        .filter((chunk) => chunk.score > 0)
        .sort((left, right) => right.score - left.score)
        .slice(0, maxChunks)
}

This retriever scores term overlap between the question and chunk text. It is fast, explainable, and flawed—exactly what I wanted for a baseline.

Measuring More Than the Final Answer

The benchmark records more than whether the model answered correctly. It measures:

corpus load time
chunking time
retrieval time
session setup time
context measurement time
prompt time
time to first streamed chunk
context usage before and after the run

This is the core flow:

const corpus = await loadWagnerCorpus()
const chunks = buildChunks(corpus.text, config.chunkSize, config.chunkOverlap)
const selectedChunks = rankChunks(chunks, query, config.retrievedChunks)

const session = await createPromptSession({
    config,
    onDownloadProgress(progress) {
        downloadProgress.push(progress)
    },
})

const estimatedInputUsage = await measureContextUsage(session, input)

const { text, firstChunkMs } = await executePrompt({
    session,
    input,
    streaming: config.streaming,
    signal,
    onChunk: callbacks.onChunk,
})

At this point the demo becomes less a “chatbot” and more an instrument panel.

What the Results Show

In the run shown in the screenshots, the app reported approximately:

corpus size: 219,572 words / 1,251,663 characters
chunk count: 477
retrieval time: 8.7 ms
session setup: 7.5 ms
prompt time: 17,396 ms
first chunk: 7,226 ms
total: 32,766 ms
context usage: 3417 / 9216

Several observations stand out.

1. Retrieval is essentially free compared to inference

Lexical retrieval took 8.7 ms. That is tiny compared to the 17.4 second prompt time. For early‑stage RAG in the browser, this suggests a useful lesson: before over‑optimizing retrieval, understand your inference costs. In this setup, retrieval is not the bottleneck. Prompting is.

2. Time to first token matters more than people think

The first chunk arrived after about 7.2 seconds. That number changes the perceived feel of the product. If the first token arrives quickly, the experience feels responsive. If it takes several seconds, users may wonder if it has hung or if they asked too much. A good benchmark should capture that moment, not just the final duration.

3. The context window is generous but not infinite

The run used about 3417 units of a 9216 context window. That sounds comfortable, but long‑form exploration can consume budget quickly. If you increase chunk size, overlap, or retrieved chunk count, the window fills with evidence before the model answers. That’s why the demo exposes chunk controls prominently.

4. Total time tells a bigger story than prompt time alone

The total was about 32.8 seconds—notably higher than prompt time alone. That gap hides real product behavior: corpus loading, chunking, preparation work, model readiness, UI update overhead, and one‑time costs that don’t appear if you only look at prompt(). For engineers, this is an important shift: users experience the whole pipeline, not just the API call.

The Limits Show Up Quickly

The Prompt API is interesting not because it’s limitless, but because its limits are visible and teach you something. Here are the main ones I encountered.

Long context is still a budgeting problem

You cannot stuff an entire million‑character book into a prompt. Even when the corpus lives locally, context remains scarce. That pushes you toward retrieval, chunking, and prompt construction strategies sooner than you might expect.

Lexical retrieval is fast but semantically clumsy

The retrieved excerpts screenshot shows this clearly. Some selected chunks are relevant to the query “How does Wagner describe his early artistic ambitions?” But some are relevant mostly because they contain overlapping words like “early”, “artistic”, or “ambitions”, not because they are the best narrative evidence. That is a useful failure mode—it shows why better retrieval becomes necessary.

Availability is not guaranteed

The Prompt API is not a universal browser primitive yet. It depends on Chrome support, device capability, model management, and the environment. Every serious app needs a plan for unsupported devices, first‑time model download, delayed readiness, and the possibility that the model is unavailable or removed.

Streaming helps, but latency is still real

Streaming makes the wait feel more humane after generation starts, but it does not remove the wait before generation starts. A slow first‑token experience remains an issue.

Browser AI is not the same as exact system telemetry

In the current version, I can measure prompt timing and context usage cleanly. What I cannot claim cleanly is exact model memory consumption, the way I might with a dedicated server‑side runtime. Some metrics are authoritative; some are approximate. Good benchmarking should label the difference honestly.

What I Like About This API

Even with those limits, building on a browser‑native AI surface has clear benefits. You ask the browser what is available. You create a session. You stream output. You inspect context pressure. You see download progress. You can build a real experiment around that.

For an engineer, that means you can learn about product design, retrieval systems, latency, UI feedback, and model constraints all within one project.

Next Steps

Obvious and useful extensions:

Compare lexical retrieval against something semantic.
Run the same question across several chunk sizes side by side.
Test how structured output changes context usage.
Make failure cases first‑class, not hidden.
Add exportable benchmark traces so results can be compared over time.

This becomes less about whether the model answered, and more about why this configuration behaved the way it did.

Final Thought

The Prompt API made me think less about “AI features” and more about systems behavior under constraints. That is why this experiment felt worth building. The model answered a question about Wagner—fine. But the more interesting outcome was watching the browser become a measurable inference environment with its own quirks, bottlenecks, and product tradeoffs.

If you are early in your engineering journey, this is the kind of project I would recommend: one that looks like a demo from a distance, but up close turns into a lesson about architecture. And that is usually where the real learning starts.

*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

HexmosTech / git-lrc

Free, Micro AI Code Reviews That Run on Commit

git-lrc

Free, Micro AI Code Reviews That Run on Commit

AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.

See It In Action

See git-lrc catch serious security issues such as leaked credentials, expensive cloud operations, and sensitive material in log statements

git-lrc-intro-60s.mp4

Why

🤖 AI agents silently break things. Code removed. Logic changed. Edge cases gone. You won't notice until production.
🔍 Catch it before it ships. AI-powered inline comments show you exactly what changed and what looks wrong.
🔁 Build a…

View on GitHub