DEV Community: eleonorarocchi

Setting Up a Local AI Coding Agent with Ollama and Aider (part 3)

eleonorarocchi — Fri, 17 Jul 2026 20:14:00 +0000

Dashboard

After collecting the metrics in ollama_usage.jsonl, I wanted a way to visualize them.

I used two tools:

Streamlit → quick, operational view
Grafana   → more stable dashboards and long-term monitoring

Both read the same data, but in different ways.

The final architecture looks like this:

Aider → proxy on 11435 → ollama_usage.jsonl
                            ├─ Streamlit
                            └─ exporter → Prometheus → Grafana

Streamlit

Streamlit was the fastest way to build a local user interface.

Inside WSL:

python3 -m venv ~/.venvs/aider-dashboard
source ~/.venvs/aider-dashboard/bin/activate
pip install streamlit pandas streamlit-autorefresh

Then I created the dashboard:

cat > ~/aider_usage_dashboard.py <<'PY'
import json
from pathlib import Path

import pandas as pd
import streamlit as st
from streamlit_autorefresh import st_autorefresh

LOG_PATH = Path.home() / "ollama_usage.jsonl"

st.set_page_config(
    page_title="Aider / Ollama usage dashboard",
    layout="wide",
)

st_autorefresh(interval=5000, key="aider_dashboard_refresh")

st.title("Aider / Ollama usage dashboard")
st.caption("Local statistics: tokens, response times, and generation speed.")

if not LOG_PATH.exists():
    st.warning(f"No file found: {LOG_PATH}")
    st.stop()

rows = []
with LOG_PATH.open("r", encoding="utf-8") as f:
    for line in f:
        line = line.strip()
        if not line:
            continue
        try:
            rows.append(json.loads(line))
        except json.JSONDecodeError:
            pass

if not rows:
    st.warning("The file exists, but it does not contain any valid metrics yet.")
    st.stop()

df = pd.DataFrame(rows)

if "ts" in df.columns:
    df["ts"] = pd.to_datetime(df["ts"], errors="coerce")
    df = df.sort_values("ts")
    df["run"] = range(1, len(df) + 1)
else:
    df["run"] = range(1, len(df) + 1)

numeric_cols = [
    "prompt_tokens",
    "response_tokens",
    "total_tokens",
    "wall_time_s",
    "ollama_total_s",
    "load_s",
    "prompt_eval_s",
    "generation_s",
    "generation_tokens_per_s",
]

for col in numeric_cols:
    if col in df.columns:
        df[col] = pd.to_numeric(df[col], errors="coerce")

total_calls = len(df)
total_tokens = int(df["total_tokens"].sum()) if "total_tokens" in df.columns else 0
avg_time = df["wall_time_s"].mean() if "wall_time_s" in df.columns else None
avg_tps = df["generation_tokens_per_s"].mean() if "generation_tokens_per_s" in df.columns else None

c1, c2, c3, c4 = st.columns(4)

c1.metric("Recorded responses", total_calls)
c2.metric("Total tokens", f"{total_tokens:,}")
c3.metric("Average response time", f"{avg_time:.1f}s" if pd.notna(avg_time) else "n/a")
c4.metric("Average tokens/s", f"{avg_tps:.2f}" if pd.notna(avg_tps) else "n/a")

st.divider()

left, right = st.columns(2)

with left:
    st.subheader("Tokens per response")
    token_cols = [c for c in ["prompt_tokens", "response_tokens", "total_tokens"] if c in df.columns]
    if token_cols:
        st.line_chart(df.set_index("run")[token_cols])

with right:
    st.subheader("Response times")
    time_cols = [c for c in ["wall_time_s", "ollama_total_s", "generation_s", "prompt_eval_s"] if c in df.columns]
    if time_cols:
        st.line_chart(df.set_index("run")[time_cols])

left, right = st.columns(2)

with left:
    st.subheader("Generation speed")
    if "generation_tokens_per_s" in df.columns:
        st.line_chart(df.set_index("run")[["generation_tokens_per_s"]])

with right:
    st.subheader("Input vs output tokens")
    cols = [c for c in ["prompt_tokens", "response_tokens"] if c in df.columns]
    if cols:
        st.bar_chart(df.set_index("run")[cols])

st.divider()

st.subheader("Latest responses")
show_cols = [
    c for c in [
        "ts",
        "model",
        "prompt_tokens",
        "response_tokens",
        "total_tokens",
        "wall_time_s",
        "ollama_total_s",
        "generation_s",
        "generation_tokens_per_s",
        "done_reason",
    ]
    if c in df.columns
]

st.dataframe(
    df[show_cols].tail(50).sort_index(ascending=False),
    use_container_width=True,
)

st.caption(f"Reading data from: {LOG_PATH}")
PY

To start it:

source ~/.venvs/aider-dashboard/bin/activate
streamlit run ~/aider_usage_dashboard.py --server.address 127.0.0.1 --server.port 8501

Then open:

http://localhost:8501

I used this as a quick operational dashboard. Of course, this was only the starting point; I later customized it with all the metrics I actually needed.

Grafana

Grafana, a widely used data visualization and analytics tool, requires a few additional components:

exporter → Prometheus → Grafana

Docker Desktop was already installed on my Windows PC, but I first had to enable WSL integration from Docker Desktop:

Settings → Resources → WSL Integration

Then, inside WSL, I fixed the Docker permissions:

sudo groupadd docker 2>/dev/null || true
sudo usermod -aG docker $USER
newgrp docker

To verify the setup:

docker version
docker compose version
docker ps

Monitoring Directory

mkdir -p ~/monitoring/llm-grafana
cd ~/monitoring/llm-grafana

mkdir -p prometheus
mkdir -p grafana/provisioning/datasources
mkdir -p ollama-jsonl-exporter

Prometheus Exporter

The exporter reads the JSONL file and exposes Prometheus metrics on port 9108.

cat > ollama-jsonl-exporter/Dockerfile <<'DOCKER'
FROM python:3.12-slim
WORKDIR /app
COPY exporter.py /app/exporter.py
ENV LOG_PATH=/data/ollama_usage.jsonl
ENV PORT=9108
EXPOSE 9108
CMD ["python", "/app/exporter.py"]
DOCKER

The exporter.py file is the component that converts the JSONL data into Prometheus metrics.

Prometheus

Prometheus is the monitoring engine. It is responsible for collecting and storing metrics such as CPU usage, memory consumption, and application-specific measurements.

cat > prometheus/prometheus.yml <<'YAML'
global:
  scrape_interval: 5s
  evaluation_interval: 5s

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["prometheus:9090"]

  - job_name: "aider-ollama"
    static_configs:
      - targets: ["ollama-jsonl-exporter:9108"]
YAML

Grafana Data Source

cat > grafana/provisioning/datasources/prometheus.yml <<'YAML'
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
YAML

Docker Compose

The working configuration mounts the entire prometheus directory rather than the individual configuration file.

cat > docker-compose.yml <<'YAML'
services:
  ollama-jsonl-exporter:
    build:
      context: ./ollama-jsonl-exporter
    container_name: ollama-jsonl-exporter
    restart: unless-stopped
    volumes:
      - ${HOME}/ollama_usage.jsonl:/data/ollama_usage.jsonl:ro
    ports:
      - "127.0.0.1:9108:9108"

  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus-llm
    restart: unless-stopped
    depends_on:
      - ollama-jsonl-exporter
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.path=/prometheus"
      - "--storage.tsdb.retention.time=30d"
    volumes:
      - ./prometheus:/etc/prometheus:ro
      - prometheus_data:/prometheus
    ports:
      - "127.0.0.1:9090:9090"

  grafana:
    image: grafana/grafana:latest
    container_name: grafana-llm
    restart: unless-stopped
    depends_on:
      - prometheus
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=admin
      - GF_USERS_ALLOW_SIGN_UP=false
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning:ro
    ports:
      - "127.0.0.1:3001:3000"

volumes:
  prometheus_data:
  grafana_data:
YAML

To start the stack:

cd ~/monitoring/llm-grafana
docker compose up -d --build

To verify it:

docker compose ps
curl http://127.0.0.1:9108/metrics | head -n 20
curl http://127.0.0.1:9090/-/ready

Grafana is available at:

http://localhost:3001

On the first login, you need to configure a user account.

Useful Grafana Queries

In the Grafana panels, I used Code mode instead of the visual query builder.

Recorded responses:

sum(aider_ollama_calls_total)

Total tokens:

sum(aider_ollama_tokens_total)

Latest response time:

aider_ollama_last_wall_time_seconds

Tokens per second for the latest response:

aider_ollama_last_generation_tokens_per_second

For KPIs, I used Stat panels, while for trends I used Time series panels.

Daily Workflow

Terminal 1:

cd ~
python3 ~/ollama_usage_proxy.py

Terminal 2:

cd ~/repos/my-project
aider14stats

Terminal 3:

cd ~/monitoring/llm-grafana
docker compose up -d

Terminal 4:

source ~/.venvs/aider-dashboard/bin/activate
streamlit run ~/aider_usage_dashboard.py --server.address 127.0.0.1 --server.port 8501

With this setup, I ended up with two complementary views:

Streamlit → operational debugging
Grafana   → stable monitoring

This is still an experimental setup. For a more mature environment, I would probably consolidate everything into a single tool.

Setting Up a Local AI Coding Agent with Ollama and Aider (part 2)

eleonorarocchi — Tue, 14 Jul 2026 20:48:00 +0000

Metrics

Once I had Aider working with a local Ollama instance, there was still one thing missing: understanding what was actually happening under the hood.

I wanted to see:

how many tokens were being processed
how long each response took
how fast text generation was
whether one model was slower than another
whether the context window was growing too large

Aider alone didn't provide enough visibility. However, Ollama exposes several useful metrics in the final response returned by its API.

So I decided to add a small local proxy.

Architecture

The request flow became:

Aider
  ↓
Local proxy on port 11435
  ↓
Ollama on Windows (11434)
  ↓
qwen2.5-coder:14b

The proxy performs two simple tasks:

forwards every request to Ollama
stores the execution statistics in a JSONL file

The resulting log file is:

~/ollama_usage.jsonl

Creating the Proxy

Inside WSL:

cat > ~/ollama_usage_proxy.py <<'PY'
#!/usr/bin/env python3
import json
import time
import urllib.request
from datetime import datetime, timezone
from http.server import ThreadingHTTPServer, BaseHTTPRequestHandler

UPSTREAM = "http://127.0.0.1:11434"
LOG_FILE = "/home/xxxxx/ollama_usage.jsonl"

def ns_to_s(value):
    if isinstance(value, (int, float)):
        return value / 1_000_000_000
    return None

def safe_div(a, b):
    if not a or not b:
        return None
    return a / b

class OllamaProxy(BaseHTTPRequestHandler):
    protocol_version = "HTTP/1.0"

    def log_message(self, format, *args):
        return

    def do_GET(self):
        self.forward()

    def do_POST(self):
        self.forward()

    def forward(self):
        length = int(self.headers.get("Content-Length", "0") or "0")
        body = self.rfile.read(length) if length else None

        url = UPSTREAM + self.path
        headers = {
            k: v for k, v in self.headers.items()
            if k.lower() not in {
                "host",
                "content-length",
                "connection",
                "accept-encoding",
            }
        }

        request = urllib.request.Request(
            url,
            data=body,
            headers=headers,
            method=self.command,
        )

        started = time.perf_counter()
        final_obj = None

        try:
            with urllib.request.urlopen(request, timeout=None) as response:
                self.send_response(response.status)
                self.send_header(
                    "Content-Type",
                    response.headers.get("Content-Type", "application/json"),
                )
                self.end_headers()

                for line in response:
                    self.wfile.write(line)
                    self.wfile.flush()

                    try:
                        obj = json.loads(line.decode("utf-8"))
                        if obj.get("done") is True:
                            final_obj = obj
                    except Exception:
                        pass

        except Exception as exc:
            self.send_error(502, f"Proxy error: {exc}")
            return

        wall_time_s = time.perf_counter() - started

        if not final_obj:
            return

        prompt_tokens = final_obj.get("prompt_eval_count")
        response_tokens = final_obj.get("eval_count")

        prompt_eval_s = ns_to_s(final_obj.get("prompt_eval_duration"))
        eval_s = ns_to_s(final_obj.get("eval_duration"))
        total_s = ns_to_s(final_obj.get("total_duration"))
        load_s = ns_to_s(final_obj.get("load_duration"))

        row = {
            "ts": datetime.now(timezone.utc).isoformat(),
            "endpoint": self.path,
            "method": self.command,
            "model": final_obj.get("model"),
            "done_reason": final_obj.get("done_reason"),
            "prompt_tokens": prompt_tokens,
            "response_tokens": response_tokens,
            "total_tokens": (prompt_tokens or 0) + (response_tokens or 0),
            "wall_time_s": round(wall_time_s, 3),
            "ollama_total_s": round(total_s, 3) if total_s is not None else None,
            "load_s": round(load_s, 3) if load_s is not None else None,
            "prompt_eval_s": round(prompt_eval_s, 3) if prompt_eval_s is not None else None,
            "generation_s": round(eval_s, 3) if eval_s is not None else None,
            "prompt_tokens_per_s": round(safe_div(prompt_tokens, prompt_eval_s), 2)
            if safe_div(prompt_tokens, prompt_eval_s) else None,
            "generation_tokens_per_s": round(safe_div(response_tokens, eval_s), 2)
            if safe_div(response_tokens, eval_s) else None,
        }

        with open(LOG_FILE, "a", encoding="utf-8") as f:
            f.write(json.dumps(row, ensure_ascii=False) + "\n")

if __name__ == "__main__":
    server = ThreadingHTTPServer(("127.0.0.1", 11435), OllamaProxy)
    print("Ollama usage proxy listening on http://127.0.0.1:11435")
    print(f"Forwarding requests to {UPSTREAM}")
    print(f"Writing statistics to {LOG_FILE}")
    server.serve_forever()
PY

Then make it executable:

chmod +x ~/ollama_usage_proxy.py

Starting the Proxy

In a WSL terminal:

cd ~
python3 ~/ollama_usage_proxy.py

Leave this terminal running while using Aider.

Running Aider Through the Proxy

In a second WSL terminal:

cd ~/repos/my-project
OLLAMA_API_BASE=http://127.0.0.1:11435 aider --model ollama_chat/qwen2.5-coder:14b --no-show-model-warnings

The distinction is important:

11434 = Direct connection to Ollama (no metrics)
11435 = Proxy enabled (metrics collected)

Inspecting the JSONL Log

After Aider generates a response:

tail -n 5 ~/ollama_usage.jsonl

To make the output easier to read, I installed jq, a powerful open-source command-line tool for parsing, filtering, formatting, and manipulating JSON data:

sudo apt install -y jq

Then:

tail -n 5 ~/ollama_usage.jsonl | jq .

A typical log entry looks like this:

{
  "model": "qwen2.5-coder:14b",
  "prompt_tokens": 1280,
  "response_tokens": 310,
  "total_tokens": 1590,
  "wall_time_s": 42.3,
  "generation_s": 37.8,
  "generation_tokens_per_s": 8.2
}

Creating an Alias

To avoid accidentally connecting Aider to the wrong port, I created a shell alias:

echo 'alias aider14stats="OLLAMA_API_BASE=http://127.0.0.1:11435 aider --model ollama_chat/qwen2.5-coder:14b --no-show-model-warnings"' >> ~/.bashrc
source ~/.bashrc

From that point on, starting Aider was as simple as:

cd ~/repos/my-project
aider14stats

What I Learned

Logging should be part of the design from the very beginning.

Without metrics, a local model may simply feel slow. With proper measurements, I can understand:

how much input I'm sending
how much output I'm receiving
how long each request takes
how many tokens per second are being generated
whether the model was already loaded into memory

Having access to this information completely changes the way I can evaluate and compare locally hosted LLMs.

Setting Up a Local AI Coding Agent with Ollama and Aider

eleonorarocchi — Sat, 11 Jul 2026 13:41:12 +0000

Over the past few months, I've experimented with several workflows for using LLMs in software development. In this article, I describe the local setup I've validated on my own PC: Ollama running on Windows, Aider inside Ubuntu on WSL2, and a code-focused model running entirely on the local machine.

The machine I used for my first experiment runs Windows 11 with an Intel Core i7-1355U CPU, 48 GiB of RAM, and no dedicated NVIDIA GPU. It's generally a capable machine, but not particularly well suited for AI inference. Because it lacks a dedicated GPU, I had to run the model in CPU-only mode.

The goal of this setup was to create an environment where both the source code and the data remain entirely on the local machine—far from the cloud-based frontier model approach.

From an architectural standpoint, I decided to keep Ollama on Windows because I also use the installed models with other Windows applications, and I wanted to avoid maintaining duplicate model installations. Aider, on the other hand, runs inside WSL, where installation and configuration are simpler and the Linux command-line environment is more convenient.

Ollama remains exposed on the local port 11434. Aider, running inside Ubuntu WSL2, connects to Ollama through the OLLAMA_API_BASE environment variable.

This approach avoids duplicating models inside WSL while keeping development repositories in the Linux filesystem under ~/repos, where Git and other command-line tools work much more naturally.

Local Setup

Since this machine is not an AI workstation, the solution had to be realistic. That meant avoiding massive GPU-hosted models, cloud APIs, and overly complex automation.

The final architecture looks like this:

Windows
 ├─ Ollama
 │   └─ qwen2.5-coder:14b
 │
 └─ WSL Ubuntu
     ├─ Aider
     ├─ Git
     └─ Development repositories

Keeping Ollama on Windows while running Aider inside WSL was a natural choice. I already use Ollama with several Windows applications, so duplicating the models inside WSL made little sense. At the same time, I much prefer working with the Linux command line for software development.

Installing Ollama on Windows

I installed Ollama on Windows and downloaded the primary coding model:

ollama pull qwen2.5-coder:14b

I also installed the smaller model for quicker testing:

ollama pull qwen2.5-coder:7b

To verify the installation:

ollama list

The WSL-to-Windows Connectivity Issue

Initially, Aider running inside WSL couldn't communicate with Ollama running on Windows.

To make it work, I had to configure WSL to use mirrored networking.

From PowerShell:

notepad $env:USERPROFILE\.wslconfig

Then I added the following configuration:

[wsl2]
networkingMode=mirrored
dnsTunneling=true
firewall=true
autoProxy=true

Finally, I restarted WSL:

wsl --shutdown

After reopening Ubuntu, I tested the connection:

curl http://127.0.0.1:11434/api/tags

The response confirmed that WSL could successfully reach the Ollama server running on Windows at 127.0.0.1:11434.

Installing Aider in WSL

Inside Ubuntu/WSL:

sudo apt update
sudo apt install -y git python3 python3-pip python3-venv pipx curl
python3 -m pipx ensurepath
exec $SHELL -l
pipx install aider-chat

To verify the installation:

aider --version

I also configured Git so I could version-control my experiments:

git config --global user.name "xxxx"
git config --global user.email "xxxxxxx@xxxxx.xx"

Creating the First Repository

I created a simple test project:

mkdir -p ~/repos
cd ~/repos
mkdir my-project
cd my-project
git init

Then I launched Aider:

cd ~/repos/my-project
OLLAMA_API_BASE=http://127.0.0.1:11434 aider --model ollama_chat/qwen2.5-coder:14b --no-show-model-warnings

When Aider displayed:

Aider v0.86.2
Model: ollama_chat/qwen2.5-coder:14b
Git repo: .git

I finally knew the basic setup was working correctly.

Opening the Repository from Windows Explorer

For convenience, I also wanted to access the repository directly from Windows Explorer.

From inside WSL:

explorer.exe .

This command opens the current WSL directory directly in Windows File Explorer.

What's Next?

Once I had Aider working with a local Ollama instance, there was still one thing I wanted to understand better: what was happening under the hood. How many tokens were being processed? How long did inference take? What kind of throughput could I expect?

We'll cover all of that in the next installment. 😉

If AI writes code, what is our job now?

eleonorarocchi — Wed, 01 Jul 2026 10:38:16 +0000

Anthropic published a very interesting article on recursive self-improvement: the possibility that AI systems will increasingly contribute to building better future versions of themselves.

Article: https://www.anthropic.com/institute/recursive-self-improvement

It is a huge topic, almost science fiction: AI developing AI, increasingly fast improvement cycles, systems capable of automating growing parts of research and development.

But reading the article as a developer, I think the most interesting question is not:

Will AI replace us?

The more useful question today is another one:

If AI can write, test, refactor, and review code better and better, what is the real role of humans in software development?

For years, we have measured a developer's work partly by their ability to produce code: building features, fixing bugs, optimizing parts of a system, handling refactoring, and doing reviews.

Today, however, an increasingly large part of these activities can be accelerated by AI tools.

This means that the value of the developer is shifting from "doing" to "deciding well what should be done."

One of the most important points in Anthropic's article is that AI becomes very strong when the goal is clear, especially on well-defined tasks.

But are we, as developers, still truly understanding what is being produced?

Because if AI generates code that works, but we can no longer explain it, maintain it, or assess its risks, then we have not really increased our productivity. We have only pushed technical debt a little further down the road.

The bottleneck

In many teams, the historical problem was producing enough code. But with today's AI tools, the bottleneck shifts: it becomes the human ability to review, validate, and understand.

If a tool can generate in a few minutes what used to take hours, the critical point becomes how quickly we can understand, verify, and maintain it.

This is where the developer of the near future is defined: someone who owns judgment skills that cannot be delegated to AI.

And so my provocation is this:

What if part of the low productivity of those teams does not depend on the tools, but on the fact that they keep working as if we were still in 2015?

Today we have tools that can accelerate coding, debugging, testing, review, documentation, and learning. But if we use them with old mindsets, rigid processes, and poorly collaborative habits, the gain remains limited.

Generator-Evaluator Loops for AI Agents

eleonorarocchi — Fri, 22 May 2026 05:26:00 +0000

TL;DR

Separating the generator from the evaluator improves quality and reduces premature self-validation.
The loop works best when feedback is explicit and based on clear rubrics, especially for subjective or complex tasks.
It is useful when the task has high value; for simple or easily testable tasks, it can become overengineering.

How to Separate Production and Evaluation in Tasks Without Ground Truth

If self-evaluation is a recurring failure mode in AI agents, the most natural solution is to separate the role of producing from the role of evaluating.

This is the principle behind generator-evaluator loops: one agent produces an output, another evaluates it according to explicit criteria, feedback is sent back to the generator, and the system iterates until the result reaches an acceptable threshold.

The pattern is simple to describe, but powerful, especially in domains where stable ground truth does not exist: design, writing, UX, naming, strategy, documentation, software architecture, complex refactoring.

In these cases, however, it is not enough to ask the model to "do better." You need to build an environment where improvement is driven by separate, structured, and repeatable critique.

From a Single Agent to an Agentic System

A single agent tends to merge three different roles: it plans the work, produces the output, and finally evaluates the result.

This fusion is convenient but fragile. While it may work for short tasks, in longer ones it often leads to premature convergence, because the same agent chooses a direction, develops it, then positively self-evaluates and closes the task.

To create a more robust system, these roles should be separated into at least three distinct agents:

the planner, which breaks the request into manageable parts and defines priorities, constraints, and work sequence;
the generator, which produces the artifact (e.g. code, interface, document, proposal, strategy, ...);
the evaluator, which judges the output without having participated in its generation.

This separation almost resembles a miniature organizational structure, and in fact, similarly, it helps reduce coupling between decision-making, production, and approval. As in a company, each role has a different objective, and this difference introduces useful friction.

The Analogy with GANs

The most immediate parallel is with Generative Adversarial Networks (GANs).

In a classic GAN, there are two neural networks: a generator and a discriminator. The generator produces synthetic data, for example images, while the discriminator receives both real and generated data and attempts to distinguish between them. In this way, the generator improves by trying to produce outputs plausible enough to fool the discriminator, while the discriminator improves by becoming better at detecting artificial outputs.

This idea has been applied across many domains: image generation, face synthesis, super-resolution, computer vision, synthetic data generation, image-to-image translation, music, and video. Examples include models such as StyleGAN, CycleGAN, TextGAN, and MuseGAN, which show how the generative-adversarial principle can be adapted to different forms of content.

In our case-AI agents-the analogy is useful because it captures an architectural intuition: a generative system improves when exposed to separate judgment.

There are, however, important aspects that should not be confused with the comparison, because a generator-evaluator loop is not a GAN in the technical sense.

In real GANs, there is a mathematical loss function: the discriminator is trained on real data, while the generator receives an optimization signal through gradients.

In AI agents, by contrast, feedback is linguistic and heuristic: the evaluator does not directly update the generator's weights, nor does it necessarily have access to a dataset of real examples.

When an evaluator judges a landing page, a piece of writing, or a strategy, it is not distinguishing "true" from "false" in the GAN sense. It is estimating how closely the output aligns with a set of preferences, criteria, and conventions.

Making Subjectivity Evaluatable

The core challenge is turning vague judgments into observable criteria.

Saying "this UI is ugly" does not help an agent improve-just as saying the same thing to a human designer would not.

Saying "the visual hierarchy does not sufficiently distinguish between the primary action and secondary content" is much more useful.

An effective evaluator needs what are commonly called evaluation rubrics.

In frontend design, for example, a rubric might include four dimensions:

Design quality, which measures whether the result feels like a coherent system or merely an assembly of components. It evaluates visual identity, creative direction, color usage, typography, rhythm, and layout.
Originality, which measures whether intentional choices are present or whether the output looks derived from templates, library defaults, and generic patterns (this includes recognizing AI slop: predictable gradients, white cards, interchangeable hero sections, stock icons, personality-less compositions).
Craft, which measures execution: spacing, contrast, alignment, typographic hierarchy, color consistency, and attention to detail.
Functionality, which measures usability: whether users understand what to do, find the primary actions, navigate without ambiguity, and complete intended tasks.

These criteria do not all need to carry the same weight. In many cases, models are already reasonably strong in craft and functionality; they produce orderly layouts, readable text, and understandable structures. The recurring issue is often the lack of originality and direction.

For this reason, an evaluator focused on real quality should penalize not only errors, but also blandness.

The same principle can be applied to other domains.

For writing, a rubric might evaluate thesis strength, argumentative structure, informational density, rhythm, specificity, voice, redundancy, and the ability to anticipate objections.

An AI-generated text may be correct, fluent, and well formatted, yet still fail to say anything memorable. In that case, a useful evaluator should be able to distinguish between superficial clarity and real argumentative value.

For strategy, a rubric might evaluate diagnostic quality, explicit assumptions, trade-offs, feasibility, prioritization, risks, dependencies, metrics, and contextual alignment.

And what about code? Beyond tests, a rubric should assess simplicity, maintainability, consistency with the codebase, error handling, extensibility, technical debt introduced, readability, and impact on existing abstractions.

The Operational Loop

A typical generator-evaluator loop follows a simple sequence:

the system receives a specification;
the planner breaks it into subtasks and defines success criteria;
the generator produces a first version of the output;
the evaluator inspects the artifact using explicit criteria and, when possible, real tools such as browsers, test runners, screenshots, parsers, linters, metrics, documentation, or access to the codebase;
the evaluator assigns scores, identifies issues, and produces actionable feedback;
the generator receives the feedback and iterates.

The cycle continues until the output exceeds a threshold, the budget is exhausted, or the system determines that human intervention is required.

The core of the loop is actionable feedback, because the evaluator must not only detect failure, but also make it correctable.

A generator-evaluator loop is fundamentally a matter of harness engineering, because the evaluator must have access to the right tools.

That said, an evaluator does not automatically improve simply because it is separated from the generator. For it to work, it must be calibrated using explicit criteria, scoring scales, potentially few-shot examples, thresholds, and similar mechanisms.

When to Use This Pattern

A generator-evaluator loop has a cost: more model calls, more tokens, more latency, more orchestration, and more complexity. As a result, it is impossible to apply it everywhere.

As a rule of thumb, if a reliable automated test exists, it is usually better to use it.

And if the task is simple or low-value, a complex loop may genuinely become overengineering.

Why AI Agents can’t judge themselves

eleonorarocchi — Fri, 15 May 2026 05:19:00 +0000

TL;DR

AI agents tend to overestimate the quality of their own outputs when there is no external verification criterion. In subjective tasks (design, writing, UX, naming, strategy), simply asking the model to "reflect" is not enough: it often remains trapped in the same trajectory that produced the first plausible solution, leading to weak critiques and superficial improvements.
Achieving real quality requires designing the runtime around the model: tests, rubrics, separate evaluators, external tools, and generator-evaluator loops that introduce critical distance between the system that produces the output and the one that approves it.

Why Internal Feedback Is Not Enough in Subjective Tasks

Sometimes, when you ask a model to evaluate a response it previously generated, it will rate it as good even when it clearly is not. Buggy code gets labeled "production-ready"; a generic layout is described as "modern and coherent"; a technically correct but flat piece of writing is called "clear, incisive, and well-structured."

This behavior becomes especially evident when the task lacks binary verification.

If the agent has to write a function and there is a reliable test suite available, the system has access to an external oracle: the test either passes or fails.

But as soon as the task moves into domains such as design, writing, naming, UX, strategy, or product architecture, quality can no longer be reduced to an assert.

This is where the self-evaluation problem in AI agents emerges: the same system that produces the output struggles to judge it with enough critical distance. And, if we think about it, humans often behave the same way.

The point, however, is not that LLMs have ego, self-esteem, or a desire to appear competent. Saying that agents "self-promote" is a useful shortcut, but technically inaccurate. A model is not trying to convince us that its output is good. More often, it simply remains inside the same probabilistic trajectory that generated the artifact in the first place.

If we ask:

Generate a landing page for a SaaS product.

And immediately afterward:

Evaluate the quality of the landing page.

we have not really created two distinct processes. We have asked the same model to continue reasoning within the same semantic space, with the same context, assumptions, and implicit orientation toward completing the task.

The result is often an evaluation that is overly generous, poorly discriminative, and not very useful for driving meaningful improvement.

Tasks With an Oracle and Tasks Without One

We can distinguish between two classes of tasks.

The first includes tasks with an external oracle. These are cases where quality can be verified relatively objectively: an automated test, a query returning an expected result, a formal constraint, a compiler, syntactic validation, or a numerical measurement.

In software engineering, many tasks fall at least partially into this category. Code can be evaluated through unit tests, integration tests, type checkers, linters, benchmarks, and static analysis. These tools do not fully capture software quality, but they provide strong signals. If an agent produces code that does not compile or breaks a test suite, the system does not need to "guess" that something is wrong: it knows.

The second class includes tasks without a clear oracle. Here, quality is subjective, multidimensional, or context-dependent. A UI can be technically correct but visually uninspired. A text can be grammatically flawless but lack a real thesis. A strategy can be well formatted yet impossible to execute. A naming proposal can be understandable but forgettable.

In these cases, the problem is not just verifying whether the output is correct, but determining whether it is actually good.

And unfortunately, "good" does not mean one single, clearly identifiable thing.

In design, it may mean visual coherence, originality, hierarchy, usability, or identity; in writing, clarity, density, rhythm, argumentative strength, or voice; in strategy, accurate diagnosis, explicit trade-offs, feasibility, specificity, and contextual alignment.

When an external oracle is missing, the agent tends to rely on its own linguistic evaluation. And that is exactly where the system becomes fragile.

The Failure Mode: Premature Convergence

The most common failure mode is not catastrophic error, but premature convergence.

The agent produces a plausible solution, refines it superficially, and declares it sufficient. The result is not necessarily wrong. Often, it is worse: mediocre but defensible.

This "plausible mediocrity" is difficult to detect because it contains many superficial signals of quality.

An AI-generated landing page will probably include a hero section, a CTA, a feature grid, a few cards, a responsive layout, a pleasant color palette, and tidy copy. A strategy document will include neatly titled sections, bullet points, frameworks, and recommendations. A refactoring will contain cleaner names and a few extra abstractions.

But all of this can still remain generic.

The agent tends to improve what it has already produced instead of questioning whether the direction itself is correct. It polishes the first solution instead of challenging it. It adds local coherence, not global quality.

This is where self-evaluation fails: not because the model cannot recognize any errors, but because it often does not apply criticism strong enough to break away from the first acceptable solution.

The Limits of Reflective Prompting

One of the earliest responses to this problem was reflective prompting: asking the model to critique its own output, identify issues, propose improvements, and iterate.

This approach works to some extent because it can eliminate obvious errors, improve clarity, fix inconsistencies, and add missing details. However, its main limitation is that the critique remains inside the same process that generated the output.

Prompts such as:

Reflect on your work and improve it.

or:

Identify any problems in the previous response.

often produce generic feedback:

"It could be more specific";
"Clarity could be improved";
"I would add more concrete examples";
"The structure is already solid, but it could be refined."

These observations are true, but weak, and they rarely lead to a substantial change in direction.

For simple tasks this may be enough, but for high-value tasks it often is not.

Why Runtime Matters

This problem contributed to the rise of harness engineering: designing the runtime around the model.

As I described in previous articles about harnesses, the core idea is that the performance of an agentic system depends not only on the model itself, but also on the operational environment in which the model works. What matters is how the prompt is constructed, which tools are available, how context is managed, how intermediate states are stored, how tests are executed, how feedback is orchestrated, when the system decides to iterate, and when it decides to stop.

In modern agentic systems, the model is just one component. Final behavior emerges from the interaction between the model, tools, memory, context, schedulers, evaluators, acceptance criteria, and retry mechanisms.

This shift in perspective is fundamental. If the model struggles to evaluate itself, the solution is not necessarily to wait for a better model. It is to design a runtime that makes the evaluation process less fragile.

In coding, this may mean running tests, reading errors, applying patches, and retrying. In design, it may mean generating screenshots, navigating the interface, and verifying interactive states. In writing, it may mean using editorial rubrics, comparing versions, and evaluating density and redundancy. In strategy, it may mean making assumptions explicit and testing alternative scenarios.

The runtime introduces signals that the model alone does not reliably produce.

Critical Distance as an Architectural Requirement

The self-evaluation problem can be summarized like this: generation and evaluation are too close to each other.

Critical distance can be introduced in many ways. Sometimes changing the prompt, role, or critique format is enough; other times it requires a different model, a different temperature, a stricter rubric, few-shot examples, external tools, or a separate agent.

The principle remains the same: the system must create a separation between the entity that produces and the entity that approves.

This separation does not guarantee perfect evaluation, but it reduces the risk that the agent settles for the first plausible solution.

This naturally leads to the generator-evaluator pattern: one agent produces, another evaluates, feedback returns to the first, and the cycle continues until the output surpasses a threshold.

It is not always necessary: for simple tasks it can become overengineering.

But for subjective, long, or high-value tasks, it becomes one of the most useful patterns in agent engineering.

How Stripe, Shopify, and Airbnb Build AI Harnesses

eleonorarocchi — Sat, 09 May 2026 07:04:00 +0000

TL;DR

There is no single model of harness engineering.
OpenAI builds repository-centered harnesses, Anthropic focuses on agent cognitive continuity, while companies like Stripe, Shopify, and Airbnb develop vertical harnesses built around compliance, context, and action verification.
Harness engineering is becoming a domain-specific discipline, shaped by the type of risk each company needs to control.

Why There Is No Single Harness: Stripe, Shopify, Airbnb, and the Industrial Fragmentation of Agent Engineering

After observing OpenAI's repository harness and Anthropic's runtime harness (if you haven't done so already, read my articles: OpenAI and the New Cognitive Architecture of Software Repositories e Anthropic and the Runtime Harness for Persistent Agents), one might expect the industry to be converging toward a fairly clear formula: define memory, tools, feedback loops, constraints, and let the agents work.

In reality, the opposite is happening: the more public big-tech case studies become, the more it becomes clear that the word harness is starting to cover profoundly different architectures.

I find this to be the most interesting signal of the sector's maturation, because it means we are no longer witnessing the birth of a standard, but rather the emergence of multiple implementation paradigms.

The comparative analyses published about companies like Stripe, Shopify, and Airbnb demonstrate this very clearly.

The Point Is Not Model Capability. It's the Cost of Failure.

As long as we talk about coding agents in the abstract, there is a tendency to imagine that the problem is singular: making the model more reliable.

However, in industrial environments, reliability is not a neutral category; it depends on what the company considers tolerable or intolerable. That is where the divergence begins.

Stripe: The Harness as a Compliance Boundary

In the financial domain, the problem is not only producing a correct modification, but producing a modification that does not violate policies, introduce vulnerabilities, alter critical transactional flows, and remains fully auditable.

In this context, the harness tends to become an approval gate, with automated validation, side-effect simulation, and above all, compliance controls.

The agent does not operate in an open environment, but inside a risk-clearing chamber.

The harness is primarily a containment boundary.

Shopify: Harnesses for Context Distribution

Shopify's problem is almost the opposite: the commerce domain is hyper-fragmented, with different themes, plugins, merchant logics, and unpredictable customizations.

The primary risk, beyond causing damage, is producing something generically correct but locally useless.

For this reason, the harness must excel at contextual retrieval, access to internal documentation, merchant-state simulation, and precise distribution of relevant information.

The model must not only be safe, but operate with an accurate understanding of the merchant's specific context.

Airbnb: Harnesses as Perceptual Verifiability

In customer-facing and UI-heavy workflows, the problem changes once again, because an agent can propose a technically reasonable modification while still breaking selectors, navigation, UX flows, or intermediate states.

In cases like Airbnb, the harness emphasizes browser instrumentation, screenshot verification, replayability, and control over executed actions.

The core question becomes: does the action actually produce the intended effect in the user environment?

The harness therefore becomes a perceptual surface.

From Best Practices to a Domain-Specific Discipline

What these cases show is that a harness is not a universal checklist of components.

A harness is a response to the failure modes that each organization considers economically most dangerous:

for OpenAI, the risk is codebase entropy;
for Anthropic, it is cognitive drift;
for Stripe, regulatory side effects;
for Shopify, the loss of situational context;
for Airbnb, the non-verifiability of actions.

Same word, completely different problems.

The Real Maturity of the Industry Is This Fragmentation

We often interpret fragmentation as a lack of standards. But I would argue the opposite can also be true: when a discipline is young, everyone uses the same generic formulas; as it matures, specialized architectures begin to emerge.

That is perhaps exactly what is happening with harness engineering.

Just like choosing the best TypeScript framework, we are now entering the phase where the real question becomes: which harness architecture is most coherent with the type of risk my agent cannot afford to tolerate?

Anthropic and the Runtime Harness for Persistent Agents

eleonorarocchi — Fri, 01 May 2026 05:13:00 +0000

TL;DR

Anthropic shows that the real challenge for AI agents is not starting a task, but staying coherent throughout long executions.
Avoiding cognitive drift requires a runtime harness built on external memory, checkpoints, and continuous re-anchoring.
The next frontier is not autonomy alone: it is cognitive continuity.

Anthropic and the Runtime Harness: the Real Problem with Agents Is Not Acting, but Not Getting Lost While They Act

If the OpenAI case showed how a repository can be rethought to become readable for agents, the contribution published by Anthropic in Harness design for long-running application development opens an even more delicate question: what happens when the challenge is no longer how to start a task well, but how to keep it alive for hours?

Because this is where many agentic systems truly begin to break.

Not at the first tool call, nor at the first planning step, but perhaps at the twentieth minute-when context starts to thin out, micro-errors begin to accumulate, and the agent keeps acting while preserving only the illusion of coherence.

In its article Harness design for long-running application development, Anthropic puts its finger exactly on this point: the frontier of agent engineering is not simply autonomy, but the persistence of autonomy over time.

The Most Underestimated Failure Mode: Cognitive Drift

Many agents appear to work well as long as we observe them on short tasks:

generating a component;
fixing a function;
calling two or three tools.

But when the task stretches across dozens of files, multiple review phases, intermediate validations, and distributed dependencies, a phenomenon begins that is very familiar to those who use them in real settings: the agent continues to produce output, but progressively loses the center of its own intention.

Anthropic treats this as a structural problem, not as a simple "model limitation": and this is precisely where the runtime harness emerges.

From the Context Window to External Cognition

The starting point is almost brutal: the context window, by itself, is too fragile a memory to sustain long-running tasks.

Even with very large contextual windows, the model suffers from imperfect compression, unstable salience, priority loss, and partial retrieval of goals.

For this reason, Anthropic builds around Claude an external procedural memory composed of persistent scratchpads, task files, execution summaries, serialized checkpoints, and continuously updated state notes.

In practice, the model is no longer forced to "remember everything", because it can reread what it has already established.

This makes an enormous difference.

The Harness as a System of Continuous Re-Anchoring

In the classical paradigm, we tell the agent: continue.

In the Anthropic paradigm, instead, we tell it: stop, reread where you are, summarize what you are doing, update your state, then continue.

This creates a re-anchoring cycle.

The agent is periodically brought back to the goal, to the progress already completed, to the constraints still open, and to the errors that have emerged.

It is a form of "artificial continuity".

Cognition is not allowed to flow in a monolithic way; it is broken apart, recorded, and reconsolidated.

Multi-Agent Evaluation: Thinking Is Not Enough, You Need to Be Critiqued

Another interesting aspect of Anthropic's work is the use of generator/evaluator structures: one agent produces, and a second agent evaluates quality, coherence, usability, and adherence to requirements.

The result is not simply "more review".

It is something subtler: verification stops being a final phase and becomes part of cognitive continuity itself.

In this way, each evaluation prevents the primary agent from drifting too far away from the correct trajectory.

The Runtime Harness Is Not Meant to Make the Agent Act Better: It Is Meant to Make It Think Longer

This is perhaps the most important point: while OpenAI builds above all a structural harness, Anthropic builds above all a temporal harness.

The problem it is solving is no longer "how do I get Claude to generate good code?", but "how do I prevent Claude from losing the thread while it continues generating it?".

It sounds like a nuance, but it completely changes the design, because here:

the memory is an external artifact,
planning is serialized,
review is recurrent,
the task is continuously re-anchored.

So this is not only orchestration-it is assisted cognitive continuity.

Conclusion

If OpenAI's repository harness teaches us that an agent needs to live inside a readable codebase, Anthropic reminds us that this is not enough.

An agent may have perfect tools, perfect documentation, perfect constraints and still get lost if it is allowed to run for too long without an external memory that keeps it coherent.

And this is where the runtime harness changes the game: it does not merely build an environment in which the agent can act; it builds an environment in which the agent can continue to know why it is acting.

In agent engineering, this may be the difference between episodic automation and real autonomy.

OpenAI and the New Cognitive Architecture of Software Repositories

eleonorarocchi — Tue, 28 Apr 2026 05:36:00 +0000

TL;DR

OpenAI's latest harness engineering report suggests something deeper than "agents can write a lot of code."
It suggests that the real bottleneck in agentic software is no longer just the model, but the repository itself.
Once agents become primary executors, codebases must stop being designed only for human maintainers and start becoming semantically navigable computational environments.

OpenAI and the Birth of the Repository Harness: When Code Must Become Readable to Agents

Over the past few months, the concept of harness engineering has become one of the most frequently discussed categories in AI engineering, especially as companies have started confronting a very simple problem: an agent may be brilliant in isolated executions, but without an environment intentionally designed around it, it quickly begins to generate entropy.

As I discussed in my previous article,Harness Engineering: The Most Important Part of AI Agents harnesses represent the truly critical layer of an agentic system, and this infrastructure must evolve significantly when moving from prototype to production.
The case recently published by OpenAI, however, adds an even more important piece to the puzzle: it suggests that the first object we need to learn how to design for agents may not be the model itself, but the repository.

The Number Everyone Quoted — and the One That Actually Matters

In the report Harness engineering: leveraging Codex in an agent-first world, OpenAI explains that it built a functional internal beta with roughly one million lines of code generated entirely by Codex, zero manually written lines, and more than 1,500 pull requests handled by an extremely small team.

It is an impressive figure, and naturally it made headlines.
But stopping at the quantity means missing the central point.

The real message of the report is something else:

productivity did not increase because Codex "writes code very fast";
it increased because engineers stopped treating the repository as a simple container of files and started treating it as an environment computable by agents.

In other words, OpenAI did not simply use a coding agent inside a codebase: it transformed the codebase into something an agent can read, interpret, and correct reliably.

From Human Codebase to Agent-Readable Codebase

There are at least four very clear signals of this transformation.

1. Repository Knowledge Becomes the System of Record

OpenAI insists on one precise point: the repository must contain the operational truth.

This means:

versioned internal documentation;
architectural maps;
decision histories;
files such as AGENTS.md that function as a semantic entry point for agents.

This is not about adding "more documentation," but about ensuring that the repository becomes machine-queryable memory, not merely something readable by humans.

The agent should not have to infer structure from scattered code; it should be able to interrogate that structure directly.

2. CI Stops Being Just Quality Assurance and Becomes a Runtime Training Mechanism

Linting, formatting, boundary checks, import policies, automated verification: in a traditional pipeline these serve to maintain order, while in a repository harness they serve something more: they become deterministic feedback loops that continuously teach the agent which behaviors are allowed and which are not.

The agent makes a mistake, CI blocks the execution, the log returns the reason, the task is iterated again: quality control stops being post-production and becomes part of the execution-time reasoning process.

3. Observability Is Designed for the Agent Too

OpenAI explains that it invested heavily in structured logs, diagnostic traces, verifiable outputs, and inspection tools.

This is because an agent that cannot properly read its own failures is forced to regenerate blindly; conversely, an agent with access to semantically dense error information can perform self-debugging.

Observability, therefore, is no longer just a developer dashboard: it becomes a cognitive surface.

4. Developers Stop Being Authors of Code and Become Authors of Constraints

This is perhaps the most interesting point in the entire OpenAI article: human work does not disappear, it shifts.

Less time spent on:

direct implementation;
manual fixes;
tactical coding.

More time spent on:

designing repository structure;
defining architectural boundaries;
building feedback loops;
cleaning entropy.

The engineer writes fewer and fewer features, and more and more conditions of intelligibility.

The Repository Harness as the New Unit of Design

If we look closely, the OpenAI case suggests a strong thesis: the first mature industrial harness is not simply a wrapper around the model; it is a codebase deliberately made readable to agents.

And this is an important distinction.

For years we assumed that the agent problem was primarily about improving:

prompting;
reasoning;
tool use.

OpenAI shows that there is an upstream layer beyond all of that:

a mediocre agent inside an agent-readable repository can still produce usable work;
a highly capable agent inside an opaque repository will still produce entropy.

The bottleneck is not only the model, but increasingly the computability of the environment.

Conclusion

Perhaps OpenAI's most interesting contribution to the harness engineering debate is not having shown that software can be built with agents.

It is having shown that, to do it seriously, we need to accept one uncomfortable fact:

it is no longer enough for code to be maintainable by humans;
it must become navigable, verifiable, and semantically readable by agents.

And this radically shifts the work of engineering.

We are no longer designing only applications — we are (perhaps finally) beginning to design repositories that can be inhabited by non-deterministic intelligences.

Building a Harness: From Prototype to Production

eleonorarocchi — Fri, 24 Apr 2026 05:12:00 +0000

TL;DR

An agent doesn’t truly work because of the model, but because of the harness controlling it.
Moving from demo to production requires handling errors, state, memory, and observability.
A well-designed harness reduces model unpredictability and shifts complexity into code, making the system reliable and usable in real-world scenarios.

In article Harness Engineering: The Most Important Part of AI Agents we saw a fundamental point: the problem with agents isn't (only) the model, but the system around it.

But what does it really mean to build that system?

The moment everything breaks

There's a fairly universal phase: you've implemented a demo, it works well, the model responds, uses a tool, maybe even completes multi-step tasks, and everything looks promising.

Then you try to use it in a real-world context, and the problems emerge:

invalid outputs
incorrect API calls
infinite loops
loss of context

It's not that the model got worse—the system's complexity increased without having a harness solid enough to manage it.

The harness as a control system

It becomes clear that the harness isn't just a "container"—it's more like a control system designed to guide the model along a precise path, reducing its freedom when necessary and allowing it when useful.

This is a delicate balance: too much control means loss of flexibility; too little control means loss of reliability.

And this is where the real design work begins.

Error handling becomes the main case

In traditional software, errors are edge cases. Anyone with experience in agent-based systems knows that errors are the norm.

The key idea, however, is that a well-designed harness does not assume everything will go well—quite the opposite.

It therefore introduces mechanisms such as:

validating outputs before using them
retrying when something goes wrong
falling back to alternative paths
controlled interruption of loops

This is what makes the system usable.

State and memory: the invisible problem

Another issue that emerges very early is state management: an agent without memory is little more than a stateless function—but adding memory introduces complexity:

what to store
for how long
how to update the state
what happens when it becomes inconsistent

These decisions must be made when structuring the harness.

And it's precisely here that many subtle bugs tend to arise.

Observability: knowing what's happening

When something goes wrong (and sooner or later it will), the important question is:

"Can I understand what happened?"

Without logging and tracing, working with agents becomes almost impossible.

Because you need to see:

every step of the reasoning
every tool call
every output transformation

And not just for debugging, but to evolve the system.

Moving complexity to the right place

An interesting aspect is that, as you improve the harness, the system becomes more predictable—even without changing the model.

This happens because complexity is being moved out of an "opaque" component (the model) and into code that can actually be controlled.

It's a shift in strategy:

less blind trust in the model
more explicit control in the system

Which, ultimately, is software engineering.

In fact, we can say that building agents today is much closer to traditional software engineering than it might seem.

There are flows, states, error handling, integrations, observability…

The only difference is that instead of deterministic functions, there's a probabilistic model.

The harness is what holds everything together—and that's what makes the difference between something that only works in a demo and something that truly works in production.

Harness Engineering: The Most Important Part of AI Agents

eleonorarocchi — Tue, 21 Apr 2026 05:25:00 +0000

TL;DR

LLMs don’t become agents because they’re more intelligent, but because we place them inside a system that makes them usable.
That system - which handles context, tools, errors, and flows - is the harness.
If an agent doesn’t work, the problem is most likely not the model, but everything you’ve built around it.

The agent isn’t broken. Your harness is.

In recent years, we’ve seen an impressive acceleration in the world of language models. Every month (every day!) something more powerful, more efficient, more “intelligent” comes out. And inevitably, the conversation always focuses there: which model to use, how many parameters it has, how well it performs on benchmarks.

But when you try to build something real, something interesting happens: the model stops being the main problem.

When you move from a demo to a system that actually has to work (with real users, messy data, unpredictable edge cases), you realize that the LLM alone isn’t enough. Not because it isn’t powerful enough, but because it isn’t designed to be reliable.

This is where what’s called harness engineering comes into play.

It’s not about the model, it’s about the system

There’s a concept that comes up often lately: agent = model + harness.

It sounds like a simplification, but it’s actually a very accurate description of what happens in practice.

The model generates text. The harness decides what that text means, what to do with it, when to trust it, and when not to.

It’s a subtle distinction, but it completely changes the way you design a system.

Because the moment you start building an agent, you are implicitly also building a way to manage context, call external tools, verify that the output makes sense, and recover when something goes wrong.

And none of that lives inside the model. It lives around the model.

The “strange” behavior of LLMs

Anyone who has worked even a little with these systems has already seen the problem.

Same prompt, same input, two different outputs.
Or: it works perfectly for ten requests, and then fails on something trivial.

That’s not a bug. It’s the nature of the model.

LLMs are not deterministic systems designed to be 100% reliable. They are excellent at generalizing, less so at guaranteeing consistency.

And this is where the developer’s role changes.

You’re no longer writing code that does things.
You’re building a system that manages an unreliable component.

And that system is the harness.

From prompt engineering to system design

For a long time, we treated these problems as an extension of prompt engineering.

“Let’s write a better prompt.”

That works, up to a point.

Then you start adding automatic retries, structured parsing, output validation, memory between steps.

And without realizing it, you’re no longer working on a prompt — you’re designing a system.

This is probably the most important transition: moving from thinking in terms of input/output to thinking in terms of flows, states, and controls.

The harness as a translator between model and reality

A useful way to think about the harness is as a translation layer.

On one side, the model: operating in natural language, probabilistic, flexible.

On the other side, the real world: APIs that break, incomplete data, rigid formats, irreversible actions.

The harness sits in between and acts as a mediator.

It takes something “soft” (the model’s text) and turns it into something “hard” (concrete actions).
And it also does the reverse: it takes structured signals and makes them usable for the model.

Why two agents using the same model behave differently

Let’s say we have two applications using the exact same LLM and getting completely different results.

At first, it seems strange.

But looking closer, you realize the difference isn’t in the model. It’s in everything around it:

How context is managed
When tools are called
What happens when something fails

In other words: the harness.

Local LLM with Google Gemma: On-Device Inference Between Theory and Practice

eleonorarocchi — Fri, 17 Apr 2026 06:30:00 +0000

TL;DR

Running an LLM locally on a smartphone is now possible—and it’s not even that exotic anymore. The interesting part is no longer whether it can be done, but how it’s done and what trade-offs actually emerge: model format, runtime, performance, and distribution.

To understand this better, I built a small Flutter app that performs on-device inference using LiteRT-LM and a Gemma 4 E2B model.

The Starting Point

Anyone working with LLMs already knows: local inference isn’t new. Between quantization, smaller models, and optimized runtimes, running models directly on devices is a real path.

So the interesting question today is no longer “can it be done?”, but rather: what does this integration actually look like when you bring it to mobile?

To answer that, I chose a deliberately simple setup: a Flutter app, a textarea, a button, and a response generated locally by the model. No backend, no API, no remote calls. Just the app and the model.

Why LiteRT-LM

It’s worth pausing here, because the runtime significantly changes the kind of work you’re doing.

LiteRT-LM is not the only option for on-device inference. In the mobile local-model landscape, alternatives like llama.cpp (with GGUF models, widely used for quantized LLMs), ONNX Runtime (more focused on cross-platform portability), and ExecuTorch (the mobile runtime from the PyTorch ecosystem, still maturing) offer different approaches depending on the model type and target hardware.

The main advantage of LiteRT-LM, however, is its native integration with the Android ecosystem and direct support for hardware delegates like the device’s GPU and NPU, making it the most straightforward choice for on-device inference without dealing with format conversions or external dependencies.

That said, there is a trade-off: the approach is less flexible than others. You can’t just use “any” model on the fly—you either use models already prepared for LiteRT or handle the conversion yourself.

Why Gemma 4 E2B

For the model, I used this variant:

https://huggingface.co/litert-community/gemma-4-E2B-it-litert-lm

The choice is not random. The Gemma 4 family includes different variants designed to balance capability and computational requirements. The E2B version is interesting because it sits at a sensible middle ground: it’s not the largest model in the family (far from it), but it’s capable enough to produce useful output while still being compact enough to make sense on a smartphone.

In other words: it’s a practical choice—not because it’s “the best ever,” but because it represents the kind of compromise that makes sense when constraints include not just output quality, but also memory, loading time, and inference speed.

The First Thing You Notice: Size

The file you download from Hugging Face weighs about 2.4 GB.

That’s not automatically a deal-breaker. Today, app stores and distribution systems offer various strategies for handling large assets: dynamic downloads, splits, additional modules, local caching...

Still, it’s important to be aware of this when thinking about production, because you’ll definitely need to reason concretely about how to package and distribute your app.

For a simple experiment like this, the easiest approach is to include the model in the app assets and then copy it to the local filesystem on first launch.

If you’re wondering why the model needs to be copied to the local filesystem, the reason is simple: LiteRT-LM (and many ML runtimes in general) require a file path on disk because they need direct access to the model file. During inference, the runtime constantly jumps between different parts of the model and accesses specific blocks (layers, weights, cache), often reusing data or working in parallel. This requires fast random access. Also, the model is not fully loaded into memory but memory-mapped as needed. None of this is feasible with a stream from assets, which only provides sequential access.

A Step-by-Step Guide

1. Create the Flutter project

From the terminal:

flutter create edge_llm_app
cd edge_llm_app
flutter run

At this point, you’ll see the classic default Flutter app with the counter.

2. Add LiteRT-LM to the Android project

This step adds the Android runtime required to run the model on-device.

Open the file:

android/app/build.gradle.kts

If there’s no dependencies block, you can add one at the end of the file. Inside it, insert:

dependencies {
    implementation("com.google.ai.edge.litertlm:litertlm-android:latest.release")
}

3. Enable the native library for GPU backend

To use the GPU (and other accelerators) for general-purpose computation—not graphics—you use OpenCL. In this case, it’s needed to run heavy computations like those of language models. Of course, this only works if the device supports it.

Open the file:

android/app/src/main/AndroidManifest.xml

Find the <application> tag and add this line inside it:

<uses-native-library
    android:name="libOpenCL.so"
    android:required="false" />

This allows the app to use OpenCL if the device supports it.

4. Download the model

Download the .litertlm file from:

https://huggingface.co/litert-community/gemma-4-E2B-it-litert-lm

In the Files and versions tab, you’ll find the model file. For simplicity, you can rename it:

gemma.litertlm

5. Copy the model into the right folder

Create the assets folder if it doesn’t exist:

android/app/src/main/assets

Then place the downloaded file inside:

android/app/src/main/assets/gemma.litertlm

6. Create the Flutter ↔ Android bridge

In the Flutter project, create this file:

lib/llm_service.dart

And paste this code:

import 'package:flutter/services.dart';

class LlmService {
  static const _channel = MethodChannel('llm');

  static Future<void> init() async {
    await _channel.invokeMethod('init');
  }

  static Future<String> ask(String prompt) async {
    final result = await _channel.invokeMethod('ask', {
      'prompt': prompt,
    });
    return result;
  }
}

This file is the bridge between the Flutter UI and the native Android code that will actually run the model.

7. Modify `MainActivity.kt`

Open:

android/app/src/main/kotlin/com/example/edge_llm_app/MainActivity.kt

(The exact path may vary slightly depending on your package name.)

Replace the content with a version that:

initializes the engine
copies the model from assets
exposes two methods to Flutter: init and ask

For example:

// (code unchanged)

This is the core of the integration. The model is copied from assets to the filesystem, the runtime is initialized, and the prompt is passed to the model.

8. Replace the default Flutter UI

Open:

lib/main.dart

Replace its content with something simple but usable, for example:

// (code unchanged)

At this point, you have a minimal UI that’s sufficient to test inference.

9. Run the app on your phone

Now you can run:

flutter run

This is where you see the difference compared to an API call.

When you press “Send,” the phone does the work. The UI may freeze for a few seconds, then the response arrives (the UI can definitely be improved, but that’s not the goal here).

From the logs, you can clearly see the different phases of inference: prefill, generation, output.

And most importantly: everything happens locally!

What This Exercise Really Shows

In the end, the interesting point is not proving that you can run an LLM on a phone. That’s already established.

The real insight is understanding what kind of integration you are building.

LiteRT-LM simplifies execution on mobile but requires you to accept a specific ecosystem. Gemma 4 E2B makes sense because it sits in a realistic range for this type of use. And the model size is not so much an absolute deal-breaker as it is an architectural variable you need to manage.

The biggest difference, however, is conceptual: when working with APIs, AI is an external service. Here, it becomes part of the application itself. You start reasoning in terms of filesystem, memory, initialization time, hardware, and acceleration.

You’re no longer just making a request.

You’re executing something locally.

And that’s the most interesting paradigm shift of all.