DEV Community: eleonorarocchi

How I Use Codex to Build Applications Without Losing Control

eleonorarocchi — Fri, 31 Jul 2026 05:19:00 +0000

In 2026, there is a great deal of discussion around Agentic Coding.

Every week, a new framework, orchestrator, or multi-agent system appears. To anyone unfamiliar with these concepts, it may seem as though developing with AI requires an infrastructure that is almost more complex than the application itself.

This perspective can discourage both beginners and more experienced developers who are still hesitant about using AI.

A few days ago, I watched Cole Medin's video, My COMPLETE Agentic Coding Workflow to Build Anything (No Fluff or Overengineering), in which he gives a very practical demonstration of how he organizes development with Claude Code.

As I watched it, I realized that the real value of the video was not Claude, but the method. I therefore decided to run an experiment: apply the same principles to Codex, adapting the structure to its conventions.

This article describes that port and what I learned during the process.

The AI Layer in the Original Workflow

In the original workflow, Cole defines an AI Layer as the set of resources stored in the repository that help the coding agent understand:

what it needs to build;
how it should build it;
which workflows it should follow;
which information it should consult depending on the task.

This layer includes the PRD, global rules, reference documentation, commands, and any relevant Skills.

One particularly important detail is the distinction between commands and Skills.

In Cole's project, the main workflows are Markdown files stored in the .claude/commands/ directory:

.claude/
└── commands/
    ├── commit.md
    ├── create-prd.md
    ├── create-rules.md
    ├── execute.md
    ├── init-project.md
    ├── plan-feature.md
    └── prime.md

Cole explains that, in Claude Code, commands and Skills have become technically very similar concepts. However, he continues to distinguish between them from an operational perspective.

A command is a workflow that the user explicitly chooses to start:

/prime
/create-prd
/plan-feature
/execute
/commit

A Skill, on the other hand, represents a capability or a set of instructions that the agent can use when the context requires it.

Commands and Skills in Codex

Cole's distinction is particularly useful for the port because slash commands and Skills are also different concepts in Codex.

Slash commands are actions that can be invoked from the interface using /. Many are built-in commands that change the mode or state of the session, although the list may also include enabled Skills and custom prompts.

Skills, by contrast, are reusable workflows organized into directories containing a SKILL.md file. At a minimum, this file specifies the Skill's name, description, and operating instructions.

Codex initially loads only the Skill's metadata and reads the full instructions when it decides to use it. This mechanism prevents unnecessary use of the context window.

Porting the Workflow to Codex

To bring the methodology to Codex, I converted the Claude Code Markdown commands into Codex Skills while preserving their semantics as workflows explicitly started by the user.

The resulting structure looks like this:

.agents/
└── skills/
    ├── agent-browser/
    │   └── SKILL.md
    ├── commit/
    │   └── SKILL.md
    ├── create-prd/
    │   └── SKILL.md
    ├── create-rules/
    │   └── SKILL.md
    ├── execute/
    │   └── SKILL.md
    ├── init-project/
    │   └── SKILL.md
    ├── plan-feature/
    │   └── SKILL.md
    └── prime/
        └── SKILL.md

To preserve the semantics of user-initiated workflows, it is possible to disable implicit invocation for Skills that we do not want Codex to select autonomously.

For example:

prime/
├── SKILL.md
└── agents/
    └── openai.yaml

Inside openai.yaml:

policy:
  allow_implicit_invocation: false

This prevents Codex from implicitly activating prime, execute, or commit based on the prompt. The Skills must be invoked explicitly, for example:

$create-prd
$create-rules
$prime

From this point onward, whenever I refer to $create-prd, $create-rules, $prime, $plan-feature, $execute, and $commit, I mean the custom Skills created for my port, not features preinstalled in Codex.

From `CLAUDE-template.md` to `AGENTS.md`

Converting the commands was not the only necessary change.

The original repository included a CLAUDE-template.md file, which served as the basis for the global instructions provided to Claude. In the port, I transformed it into an AGENTS.md file at the root of the project.

AGENTS.md is the native mechanism through which Codex receives persistent instructions and context. At the beginning of a session, Codex builds an instruction chain: it starts at the repository root and moves down to the current working directory, loading any AGENTS.md or AGENTS.override.md files it encounters along the way. Instructions defined deeper in the directory tree can supplement or replace more general ones.

Persistent instructions and Skills therefore have different responsibilities:

AGENTS.md defines how Codex should work within the repository;
Skills describe scoped workflows such as prime, create-prd, plan-feature, execute, and commit.

Specialized documentation can live in separate files, provided that AGENTS.md, a Skill, or a plan clearly specifies when it should be consulted.

A Practical Experiment: A Todo List

Once I had finished reclassifying the files, I still needed to verify whether the method actually worked with Codex.

To focus on the process rather than product complexity, I deliberately chose a simple application: a personal todo list.

I launched Codex in the folder containing the base files and gave it a brief similar to this:

I want to create a simple todo list application that allows me to add items, view the list, delete items, and mark them as completed.

I do not need authentication. I would like to use Next.js and deploy the application to Vercel for personal use.

I would like your recommendations regarding the architecture. I would also like a subagent to research best practices for this type of dashboard.

Once the research is complete, ask me all the questions necessary to clarify even the smallest product details.

Codex began asking me questions until the main open points had been clarified.

The goal of this phase is not merely to describe what we expect, but to eliminate as many assumptions as possible. Once the requirements were sufficiently clear, I invoked the following Skill:

$create-prd

From the Brief to the PRD

The resulting PRD.md file represents what we want to build. In my case, it contains:

the product objective;
primary users;
use cases;
MVP features;
out-of-scope features;
the technology stack;
architectural constraints;
non-functional requirements;
general success criteria;
implementation phases.

The division into phases is particularly important. It makes little sense to ask Codex to implement the entire product in a single session.

Each phase should be small enough to be planned, implemented, and validated separately. This makes the process easier to verify, while the PRD effectively also becomes a roadmap.

Adding Project Rules

As mentioned earlier, the repository already contained a base AGENTS.md file derived from the previous CLAUDE-template.md. This file includes the initial structure and general guidelines that I want to preserve across every project.

After defining what needed to be built, I invoked:

$create-rules

In my workflow, this Skill did not create AGENTS.md from scratch. It analyzed the PRD and the repository, then expanded the existing file with project-specific information, including:

the technology stack;
development, build, and test commands;
the directory structure;
conventions;
architectural decisions.

It is important for the final result to remain concise enough. AGENTS.md should contain instructions that are generally applicable to the repository. Specialized documentation can live in separate files, provided that AGENTS.md, a Skill, or a plan clearly indicates when it should be consulted.

The First Context Reset: `$prime`

Once the initial project definition was complete, I opened a new session and invoked:

$prime

This is the first intentional context reset. The new session does not inherit the long initial conversation. Instead, it reconstructs the project through PRD.md, AGENTS.md, the repository, and the Git history.

When I open the new session, Codex has already loaded the applicable instructions contained in AGENTS.md. The $prime Skill then completes its mental model by:

reading the PRD;
exploring the repository;
identifying the entry points;
checking the Git status;
reviewing recent commits;
determining which phases have already been completed;
identifying the next objective.

At the end, Codex provides a summary of its understanding of the project.

This phase should be reviewed carefully. If the agent's mental model is incorrect, it is best to intervene before moving on to implementation.

The PIV Cycle

At this point, we reach the core of the PIV cycle:

Plan → Implement → Validate

Each phase of the PRD goes through these three steps.

Plan: Turning a Phase into a Plan

I selected the first phase identified by Codex and invoked:

$plan-feature

This phase does not produce any code yet. Its purpose is to turn a PRD feature into a detailed technical plan.

Codex analyzes the repository, consults the relevant documentation, and defines:

the objective of the feature;
the current state of the project;
architectural decisions;
the files to create or modify;
the tasks to perform;
edge cases;
the testing and validation strategy;
the criteria required for the work to be considered complete.

In my case, $plan-feature produced a new persistent file:

.agents/
└── plans/
    └── implement-todo-dashboard-mvp.md

This detail is important: the plan does not remain confined to the conversation but becomes an artifact stored in the repository.

The document must be complete enough to be executed by a new Codex session that did not participate in the discussion during which the plan was prepared.

Before proceeding, I reviewed the file to verify that it:

was consistent with the PRD;
did not expand the scope;
contained sufficiently clear tasks;
defined genuinely verifiable completion criteria.

Implement: Executing the Plan in a New Context

Once the plan had been approved, I opened a new session dedicated to execution.

This is the second intentional context reset. The new session does not need to know every question that was asked, every rejected alternative, or the intermediate reasoning that occurred during planning. It only needs the decisions consolidated in the plan and the persistent context stored in the repository.

I then invoked:

$execute .agents/plans/implement-todo-dashboard-mvp.md

During this phase, Codex followed the tasks defined in the document, created the necessary files, implemented the todo list dashboard, and added the planned checks.

The purpose of $execute is to carry out an already approved plan. If an important unforeseen decision emerges during the work, the agent stops and asks for clarification.

Validate: Verifying the Result

Once implementation is complete, the third phase of the PIV cycle begins: validation.

Codex runs the checks defined in the plan, for example:

npm run typecheck
npm run lint
npm run build
npm run test

For a web application, however, these checks are not sufficient. The actual behavior of the interface must also be verified.

In my case, the main scenarios were:

adding a new item;
displaying it in the list;
marking it as completed;
deleting it;
reloading the page;
verifying that the state was persisted correctly.

As in Cole's example, the agent-browser Skill can be used to automate these flows and test the application as a user would.

Automated verification does not eliminate the need for human verification. At the end of the process, it is still important to personally test the application's main flows.

Git as Long-Term Memory

After verifying that the feature worked correctly, I invoked:

$commit

The Skill standardizes the commit message, describing what was implemented and how it was validated.

The Git history therefore takes on an additional role: it does not merely preserve code changes but becomes a form of long-term memory for Codex.

During a future execution of $prime, the agent can review recent commits to understand how the project has evolved and which parts of the PRD have already been addressed.

With this methodology, the repository contains several layers of memory:

PRD.md          → what we want to build
AGENTS.md       → how Codex should work
.agents/plans/  → how to implement individual features
Git history     → what was actually changed
Tests           → what must continue to work

For the next feature, a new context is opened, $prime is run again, and another PIV cycle begins.

Conclusion: A Lightweight Method or a Complete Framework?

At this point, a comparison with more structured frameworks such as the BMAD Method is inevitable.

BMAD aims to cover the entire development lifecycle: from initial analysis and product definition to architecture, the breakdown into epics and stories, implementation, and review. To achieve this, it provides specialized agents, guided workflows, intermediate artifacts, and modules dedicated to specific areas.

The method presented by Cole, and adapted by me for Codex, starts from the opposite choice: use the smallest possible number of components and fully understand the role of each one.

A PRD.md, an AGENTS.md, a few persistent plans, and a small number of Skills make up almost the entire system. A small structure is easier to customize and evolve alongside the project.

To be clear, this does not mean that the workflow is inherently better than BMAD. It simply offers a different trade-off.

A complete framework may be useful when a standardized process, clearly defined roles, numerous project artifacts, and greater consistency across different people or initiatives are required.

The lighter approach may be more suitable when the goal is to get started quickly, maintain direct control over the process, and modify each component according to the needs of a specific repository.

Naturally, simplicity comes at a cost: there is no external framework deciding which documents to produce, which checks to apply, or how to handle every possible scenario. The responsibility for evolving the Skills, AGENTS.md, tests, and rules remains with the developer.

Before starting a new project with a coding agent, the question should therefore be:

How much structure does this project actually need?

For my experiment, I did not need to adopt the complete process of a framework such as BMAD, even though I knew that BMAD also offered lighter workflows for more limited tasks. I wanted a process that was structured enough to avoid vibe coding, but simple enough to be understood, modified, and controlled directly.

And this may be the most important takeaway: using a coding agent without losing control does not necessarily require more automation. Sometimes, all it takes is a small number of clear artifacts, explicit steps, and the discipline not to skip planning and validation.

Repo

If you're interested in using the ready-made scaffolding for Codex, you can find it here: https://github.com/eleonorarocchi/codex-agentic-starter
.. and please give it a star ;-)

Setting Up a Local AI Coding Agent with Ollama and Aider (part 4)

eleonorarocchi — Tue, 21 Jul 2026 19:26:00 +0000

Limitations

Once the setup with Aider, Ollama, Grafana, and Streamlit was working, I decided to push it a little further.

The idea was to test a newer model based on a different approach from traditional autoregressive models: DiffusionGemma.

The model I wanted to try was:

google/diffusiongemma-26B-A4B-it

The naming is interesting:

26B = total number of parameters
A4B = approximately 4B active parameters

This is a Mixture-of-Experts model, but that does not mean it runs like a 4B model.

When running it locally, you still have to handle a very large model.

Why I Did Not Use It with Aider

My main workflow was:

Aider → Ollama → local model

However, DiffusionGemma was not supported by Ollama in my setup. I therefore changed the goal: instead of using it with Aider, I tested it directly with the dedicated llama.cpp runner.

I did not expect it to become part of my daily workflow, but I still wanted to run a technical experiment.

Installing the Hugging Face CLI

Inside WSL, I installed the Hugging Face CLI using its dedicated installer:

cd ~
curl -LsSf https://hf.co/cli/install.sh | bash
exec $SHELL -l

Then I verified the installation:

hf --help

Downloading the GGUF Model

I downloaded the Q4_K_M quantization:

mkdir -p ~/models/diffusiongemma

hf download unsloth/diffusiongemma-26B-A4B-it-GGUF \
  --local-dir ~/models/diffusiongemma \
  --include "*Q4_K_M*"

Then I checked that the file was present:

find ~/models/diffusiongemma -name "*.gguf" -type f

The download completed successfully, but the file is large: approximately 16 GB.

Building the Dedicated Runner

To test the model without Ollama, I used the DiffusionGemma branch of llama.cpp.

First, I installed the required dependencies:

sudo apt update
sudo apt install -y git cmake build-essential

Then I cloned the repository and checked out the relevant branch:

mkdir -p ~/src
cd ~/src

git clone https://github.com/ggml-org/llama.cpp llama.cpp-diffusiongemma
cd llama.cpp-diffusiongemma

git fetch origin pull/24423/head:diffusiongemma
git checkout diffusiongemma

For a CPU-only build:

cmake -B build -DGGML_NATIVE=ON
cmake --build build -j $(nproc) --config Release --target llama-diffusion-cli

To verify the build:

ls -l build/bin/llama-diffusion-cli

Running the Model

I stored the model path in a variable:

MODEL=$(find ~/models/diffusiongemma -name "*Q4_K_M*.gguf" -type f | head -n 1)
echo "$MODEL"

Then I launched the runner:

cd ~/src/llama.cpp-diffusiongemma

./build/bin/llama-diffusion-cli \
  -m "$MODEL" \
  -ngl 0 \
  -cnv \
  -n 256

The command ran successfully, but on the machine I was using, performance was far too slow for practical use.

I also tried a lower output limit:

./build/bin/llama-diffusion-cli \
  -m "$MODEL" \
  -ngl 0 \
  -cnv \
  -n 32

And another variant with an explicit thread count:

./build/bin/llama-diffusion-cli \
  -m "$MODEL" \
  -ngl 0 \
  -cnv \
  -n 64 \
  -t 10

The Bottleneck

My machine has:

Intel Core i7-1355U
48 GB of RAM
Intel Iris Xe
no dedicated GPU

The available RAM is enough to load many quantized models, but that does not make them fast enough for interactive use.

The real bottlenecks are:

mobile CPU
memory bandwidth
lack of dedicated VRAM
the higher computational cost of the diffusion architecture

The fact that a model is described as 26B-A4B does not mean it is equivalent to a dense 4B model.

It only means that a subset of the parameters is activated for each token. The full model is still large, and when running it locally, you still pay the cost in memory usage, I/O, and runtime complexity.

Setting Up a Local AI Coding Agent with Ollama and Aider (part 3)

eleonorarocchi — Fri, 17 Jul 2026 20:14:00 +0000

Dashboard

After collecting the metrics in ollama_usage.jsonl, I wanted a way to visualize them.

I used two tools:

Streamlit → quick, operational view
Grafana   → more stable dashboards and long-term monitoring

Both read the same data, but in different ways.

The final architecture looks like this:

Aider → proxy on 11435 → ollama_usage.jsonl
                            ├─ Streamlit
                            └─ exporter → Prometheus → Grafana

Streamlit

Streamlit was the fastest way to build a local user interface.

Inside WSL:

python3 -m venv ~/.venvs/aider-dashboard
source ~/.venvs/aider-dashboard/bin/activate
pip install streamlit pandas streamlit-autorefresh

Then I created the dashboard:

cat > ~/aider_usage_dashboard.py <<'PY'
import json
from pathlib import Path

import pandas as pd
import streamlit as st
from streamlit_autorefresh import st_autorefresh

LOG_PATH = Path.home() / "ollama_usage.jsonl"

st.set_page_config(
    page_title="Aider / Ollama usage dashboard",
    layout="wide",
)

st_autorefresh(interval=5000, key="aider_dashboard_refresh")

st.title("Aider / Ollama usage dashboard")
st.caption("Local statistics: tokens, response times, and generation speed.")

if not LOG_PATH.exists():
    st.warning(f"No file found: {LOG_PATH}")
    st.stop()

rows = []
with LOG_PATH.open("r", encoding="utf-8") as f:
    for line in f:
        line = line.strip()
        if not line:
            continue
        try:
            rows.append(json.loads(line))
        except json.JSONDecodeError:
            pass

if not rows:
    st.warning("The file exists, but it does not contain any valid metrics yet.")
    st.stop()

df = pd.DataFrame(rows)

if "ts" in df.columns:
    df["ts"] = pd.to_datetime(df["ts"], errors="coerce")
    df = df.sort_values("ts")
    df["run"] = range(1, len(df) + 1)
else:
    df["run"] = range(1, len(df) + 1)

numeric_cols = [
    "prompt_tokens",
    "response_tokens",
    "total_tokens",
    "wall_time_s",
    "ollama_total_s",
    "load_s",
    "prompt_eval_s",
    "generation_s",
    "generation_tokens_per_s",
]

for col in numeric_cols:
    if col in df.columns:
        df[col] = pd.to_numeric(df[col], errors="coerce")

total_calls = len(df)
total_tokens = int(df["total_tokens"].sum()) if "total_tokens" in df.columns else 0
avg_time = df["wall_time_s"].mean() if "wall_time_s" in df.columns else None
avg_tps = df["generation_tokens_per_s"].mean() if "generation_tokens_per_s" in df.columns else None

c1, c2, c3, c4 = st.columns(4)

c1.metric("Recorded responses", total_calls)
c2.metric("Total tokens", f"{total_tokens:,}")
c3.metric("Average response time", f"{avg_time:.1f}s" if pd.notna(avg_time) else "n/a")
c4.metric("Average tokens/s", f"{avg_tps:.2f}" if pd.notna(avg_tps) else "n/a")

st.divider()

left, right = st.columns(2)

with left:
    st.subheader("Tokens per response")
    token_cols = [c for c in ["prompt_tokens", "response_tokens", "total_tokens"] if c in df.columns]
    if token_cols:
        st.line_chart(df.set_index("run")[token_cols])

with right:
    st.subheader("Response times")
    time_cols = [c for c in ["wall_time_s", "ollama_total_s", "generation_s", "prompt_eval_s"] if c in df.columns]
    if time_cols:
        st.line_chart(df.set_index("run")[time_cols])

left, right = st.columns(2)

with left:
    st.subheader("Generation speed")
    if "generation_tokens_per_s" in df.columns:
        st.line_chart(df.set_index("run")[["generation_tokens_per_s"]])

with right:
    st.subheader("Input vs output tokens")
    cols = [c for c in ["prompt_tokens", "response_tokens"] if c in df.columns]
    if cols:
        st.bar_chart(df.set_index("run")[cols])

st.divider()

st.subheader("Latest responses")
show_cols = [
    c for c in [
        "ts",
        "model",
        "prompt_tokens",
        "response_tokens",
        "total_tokens",
        "wall_time_s",
        "ollama_total_s",
        "generation_s",
        "generation_tokens_per_s",
        "done_reason",
    ]
    if c in df.columns
]

st.dataframe(
    df[show_cols].tail(50).sort_index(ascending=False),
    use_container_width=True,
)

st.caption(f"Reading data from: {LOG_PATH}")
PY

To start it:

source ~/.venvs/aider-dashboard/bin/activate
streamlit run ~/aider_usage_dashboard.py --server.address 127.0.0.1 --server.port 8501

Then open:

http://localhost:8501

I used this as a quick operational dashboard. Of course, this was only the starting point; I later customized it with all the metrics I actually needed.

Grafana

Grafana, a widely used data visualization and analytics tool, requires a few additional components:

exporter → Prometheus → Grafana

Docker Desktop was already installed on my Windows PC, but I first had to enable WSL integration from Docker Desktop:

Settings → Resources → WSL Integration

Then, inside WSL, I fixed the Docker permissions:

sudo groupadd docker 2>/dev/null || true
sudo usermod -aG docker $USER
newgrp docker

To verify the setup:

docker version
docker compose version
docker ps

Monitoring Directory

mkdir -p ~/monitoring/llm-grafana
cd ~/monitoring/llm-grafana

mkdir -p prometheus
mkdir -p grafana/provisioning/datasources
mkdir -p ollama-jsonl-exporter

Prometheus Exporter

The exporter reads the JSONL file and exposes Prometheus metrics on port 9108.

cat > ollama-jsonl-exporter/Dockerfile <<'DOCKER'
FROM python:3.12-slim
WORKDIR /app
COPY exporter.py /app/exporter.py
ENV LOG_PATH=/data/ollama_usage.jsonl
ENV PORT=9108
EXPOSE 9108
CMD ["python", "/app/exporter.py"]
DOCKER

The exporter.py file is the component that converts the JSONL data into Prometheus metrics.

Prometheus

Prometheus is the monitoring engine. It is responsible for collecting and storing metrics such as CPU usage, memory consumption, and application-specific measurements.

cat > prometheus/prometheus.yml <<'YAML'
global:
  scrape_interval: 5s
  evaluation_interval: 5s

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["prometheus:9090"]

  - job_name: "aider-ollama"
    static_configs:
      - targets: ["ollama-jsonl-exporter:9108"]
YAML

Grafana Data Source

cat > grafana/provisioning/datasources/prometheus.yml <<'YAML'
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
YAML

Docker Compose

The working configuration mounts the entire prometheus directory rather than the individual configuration file.

cat > docker-compose.yml <<'YAML'
services:
  ollama-jsonl-exporter:
    build:
      context: ./ollama-jsonl-exporter
    container_name: ollama-jsonl-exporter
    restart: unless-stopped
    volumes:
      - ${HOME}/ollama_usage.jsonl:/data/ollama_usage.jsonl:ro
    ports:
      - "127.0.0.1:9108:9108"

  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus-llm
    restart: unless-stopped
    depends_on:
      - ollama-jsonl-exporter
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.path=/prometheus"
      - "--storage.tsdb.retention.time=30d"
    volumes:
      - ./prometheus:/etc/prometheus:ro
      - prometheus_data:/prometheus
    ports:
      - "127.0.0.1:9090:9090"

  grafana:
    image: grafana/grafana:latest
    container_name: grafana-llm
    restart: unless-stopped
    depends_on:
      - prometheus
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=admin
      - GF_USERS_ALLOW_SIGN_UP=false
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning:ro
    ports:
      - "127.0.0.1:3001:3000"

volumes:
  prometheus_data:
  grafana_data:
YAML

To start the stack:

cd ~/monitoring/llm-grafana
docker compose up -d --build

To verify it:

docker compose ps
curl http://127.0.0.1:9108/metrics | head -n 20
curl http://127.0.0.1:9090/-/ready

Grafana is available at:

http://localhost:3001

On the first login, you need to configure a user account.

Useful Grafana Queries

In the Grafana panels, I used Code mode instead of the visual query builder.

Recorded responses:

sum(aider_ollama_calls_total)

Total tokens:

sum(aider_ollama_tokens_total)

Latest response time:

aider_ollama_last_wall_time_seconds

Tokens per second for the latest response:

aider_ollama_last_generation_tokens_per_second

For KPIs, I used Stat panels, while for trends I used Time series panels.

Daily Workflow

Terminal 1:

cd ~
python3 ~/ollama_usage_proxy.py

Terminal 2:

cd ~/repos/my-project
aider14stats

Terminal 3:

cd ~/monitoring/llm-grafana
docker compose up -d

Terminal 4:

source ~/.venvs/aider-dashboard/bin/activate
streamlit run ~/aider_usage_dashboard.py --server.address 127.0.0.1 --server.port 8501

With this setup, I ended up with two complementary views:

Streamlit → operational debugging
Grafana   → stable monitoring

This is still an experimental setup. For a more mature environment, I would probably consolidate everything into a single tool.

Setting Up a Local AI Coding Agent with Ollama and Aider (part 2)

eleonorarocchi — Tue, 14 Jul 2026 20:48:00 +0000

Metrics

Once I had Aider working with a local Ollama instance, there was still one thing missing: understanding what was actually happening under the hood.

I wanted to see:

how many tokens were being processed
how long each response took
how fast text generation was
whether one model was slower than another
whether the context window was growing too large

Aider alone didn't provide enough visibility. However, Ollama exposes several useful metrics in the final response returned by its API.

So I decided to add a small local proxy.

Architecture

The request flow became:

Aider
  ↓
Local proxy on port 11435
  ↓
Ollama on Windows (11434)
  ↓
qwen2.5-coder:14b

The proxy performs two simple tasks:

forwards every request to Ollama
stores the execution statistics in a JSONL file

The resulting log file is:

~/ollama_usage.jsonl

Creating the Proxy

Inside WSL:

cat > ~/ollama_usage_proxy.py <<'PY'
#!/usr/bin/env python3
import json
import time
import urllib.request
from datetime import datetime, timezone
from http.server import ThreadingHTTPServer, BaseHTTPRequestHandler

UPSTREAM = "http://127.0.0.1:11434"
LOG_FILE = "/home/xxxxx/ollama_usage.jsonl"

def ns_to_s(value):
    if isinstance(value, (int, float)):
        return value / 1_000_000_000
    return None

def safe_div(a, b):
    if not a or not b:
        return None
    return a / b

class OllamaProxy(BaseHTTPRequestHandler):
    protocol_version = "HTTP/1.0"

    def log_message(self, format, *args):
        return

    def do_GET(self):
        self.forward()

    def do_POST(self):
        self.forward()

    def forward(self):
        length = int(self.headers.get("Content-Length", "0") or "0")
        body = self.rfile.read(length) if length else None

        url = UPSTREAM + self.path
        headers = {
            k: v for k, v in self.headers.items()
            if k.lower() not in {
                "host",
                "content-length",
                "connection",
                "accept-encoding",
            }
        }

        request = urllib.request.Request(
            url,
            data=body,
            headers=headers,
            method=self.command,
        )

        started = time.perf_counter()
        final_obj = None

        try:
            with urllib.request.urlopen(request, timeout=None) as response:
                self.send_response(response.status)
                self.send_header(
                    "Content-Type",
                    response.headers.get("Content-Type", "application/json"),
                )
                self.end_headers()

                for line in response:
                    self.wfile.write(line)
                    self.wfile.flush()

                    try:
                        obj = json.loads(line.decode("utf-8"))
                        if obj.get("done") is True:
                            final_obj = obj
                    except Exception:
                        pass

        except Exception as exc:
            self.send_error(502, f"Proxy error: {exc}")
            return

        wall_time_s = time.perf_counter() - started

        if not final_obj:
            return

        prompt_tokens = final_obj.get("prompt_eval_count")
        response_tokens = final_obj.get("eval_count")

        prompt_eval_s = ns_to_s(final_obj.get("prompt_eval_duration"))
        eval_s = ns_to_s(final_obj.get("eval_duration"))
        total_s = ns_to_s(final_obj.get("total_duration"))
        load_s = ns_to_s(final_obj.get("load_duration"))

        row = {
            "ts": datetime.now(timezone.utc).isoformat(),
            "endpoint": self.path,
            "method": self.command,
            "model": final_obj.get("model"),
            "done_reason": final_obj.get("done_reason"),
            "prompt_tokens": prompt_tokens,
            "response_tokens": response_tokens,
            "total_tokens": (prompt_tokens or 0) + (response_tokens or 0),
            "wall_time_s": round(wall_time_s, 3),
            "ollama_total_s": round(total_s, 3) if total_s is not None else None,
            "load_s": round(load_s, 3) if load_s is not None else None,
            "prompt_eval_s": round(prompt_eval_s, 3) if prompt_eval_s is not None else None,
            "generation_s": round(eval_s, 3) if eval_s is not None else None,
            "prompt_tokens_per_s": round(safe_div(prompt_tokens, prompt_eval_s), 2)
            if safe_div(prompt_tokens, prompt_eval_s) else None,
            "generation_tokens_per_s": round(safe_div(response_tokens, eval_s), 2)
            if safe_div(response_tokens, eval_s) else None,
        }

        with open(LOG_FILE, "a", encoding="utf-8") as f:
            f.write(json.dumps(row, ensure_ascii=False) + "\n")

if __name__ == "__main__":
    server = ThreadingHTTPServer(("127.0.0.1", 11435), OllamaProxy)
    print("Ollama usage proxy listening on http://127.0.0.1:11435")
    print(f"Forwarding requests to {UPSTREAM}")
    print(f"Writing statistics to {LOG_FILE}")
    server.serve_forever()
PY

Then make it executable:

chmod +x ~/ollama_usage_proxy.py

Starting the Proxy

In a WSL terminal:

cd ~
python3 ~/ollama_usage_proxy.py

Leave this terminal running while using Aider.

Running Aider Through the Proxy

In a second WSL terminal:

cd ~/repos/my-project
OLLAMA_API_BASE=http://127.0.0.1:11435 aider --model ollama_chat/qwen2.5-coder:14b --no-show-model-warnings

The distinction is important:

11434 = Direct connection to Ollama (no metrics)
11435 = Proxy enabled (metrics collected)

Inspecting the JSONL Log

After Aider generates a response:

tail -n 5 ~/ollama_usage.jsonl

To make the output easier to read, I installed jq, a powerful open-source command-line tool for parsing, filtering, formatting, and manipulating JSON data:

sudo apt install -y jq

Then:

tail -n 5 ~/ollama_usage.jsonl | jq .

A typical log entry looks like this:

{
  "model": "qwen2.5-coder:14b",
  "prompt_tokens": 1280,
  "response_tokens": 310,
  "total_tokens": 1590,
  "wall_time_s": 42.3,
  "generation_s": 37.8,
  "generation_tokens_per_s": 8.2
}

Creating an Alias

To avoid accidentally connecting Aider to the wrong port, I created a shell alias:

echo 'alias aider14stats="OLLAMA_API_BASE=http://127.0.0.1:11435 aider --model ollama_chat/qwen2.5-coder:14b --no-show-model-warnings"' >> ~/.bashrc
source ~/.bashrc

From that point on, starting Aider was as simple as:

cd ~/repos/my-project
aider14stats

What I Learned

Logging should be part of the design from the very beginning.

Without metrics, a local model may simply feel slow. With proper measurements, I can understand:

how much input I'm sending
how much output I'm receiving
how long each request takes
how many tokens per second are being generated
whether the model was already loaded into memory

Having access to this information completely changes the way I can evaluate and compare locally hosted LLMs.

Setting Up a Local AI Coding Agent with Ollama and Aider

eleonorarocchi — Sat, 11 Jul 2026 13:41:12 +0000

Over the past few months, I've experimented with several workflows for using LLMs in software development. In this article, I describe the local setup I've validated on my own PC: Ollama running on Windows, Aider inside Ubuntu on WSL2, and a code-focused model running entirely on the local machine.

The machine I used for my first experiment runs Windows 11 with an Intel Core i7-1355U CPU, 48 GiB of RAM, and no dedicated NVIDIA GPU. It's generally a capable machine, but not particularly well suited for AI inference. Because it lacks a dedicated GPU, I had to run the model in CPU-only mode.

The goal of this setup was to create an environment where both the source code and the data remain entirely on the local machine—far from the cloud-based frontier model approach.

From an architectural standpoint, I decided to keep Ollama on Windows because I also use the installed models with other Windows applications, and I wanted to avoid maintaining duplicate model installations. Aider, on the other hand, runs inside WSL, where installation and configuration are simpler and the Linux command-line environment is more convenient.

Ollama remains exposed on the local port 11434. Aider, running inside Ubuntu WSL2, connects to Ollama through the OLLAMA_API_BASE environment variable.

This approach avoids duplicating models inside WSL while keeping development repositories in the Linux filesystem under ~/repos, where Git and other command-line tools work much more naturally.

Local Setup

Since this machine is not an AI workstation, the solution had to be realistic. That meant avoiding massive GPU-hosted models, cloud APIs, and overly complex automation.

The final architecture looks like this:

Windows
 ├─ Ollama
 │   └─ qwen2.5-coder:14b
 │
 └─ WSL Ubuntu
     ├─ Aider
     ├─ Git
     └─ Development repositories

Keeping Ollama on Windows while running Aider inside WSL was a natural choice. I already use Ollama with several Windows applications, so duplicating the models inside WSL made little sense. At the same time, I much prefer working with the Linux command line for software development.

Installing Ollama on Windows

I installed Ollama on Windows and downloaded the primary coding model:

ollama pull qwen2.5-coder:14b

I also installed the smaller model for quicker testing:

ollama pull qwen2.5-coder:7b

To verify the installation:

ollama list

The WSL-to-Windows Connectivity Issue

Initially, Aider running inside WSL couldn't communicate with Ollama running on Windows.

To make it work, I had to configure WSL to use mirrored networking.

From PowerShell:

notepad $env:USERPROFILE\.wslconfig

Then I added the following configuration:

[wsl2]
networkingMode=mirrored
dnsTunneling=true
firewall=true
autoProxy=true

Finally, I restarted WSL:

wsl --shutdown

After reopening Ubuntu, I tested the connection:

curl http://127.0.0.1:11434/api/tags

The response confirmed that WSL could successfully reach the Ollama server running on Windows at 127.0.0.1:11434.

Installing Aider in WSL

Inside Ubuntu/WSL:

sudo apt update
sudo apt install -y git python3 python3-pip python3-venv pipx curl
python3 -m pipx ensurepath
exec $SHELL -l
pipx install aider-chat

To verify the installation:

aider --version

I also configured Git so I could version-control my experiments:

git config --global user.name "xxxx"
git config --global user.email "xxxxxxx@xxxxx.xx"

Creating the First Repository

I created a simple test project:

mkdir -p ~/repos
cd ~/repos
mkdir my-project
cd my-project
git init

Then I launched Aider:

cd ~/repos/my-project
OLLAMA_API_BASE=http://127.0.0.1:11434 aider --model ollama_chat/qwen2.5-coder:14b --no-show-model-warnings

When Aider displayed:

Aider v0.86.2
Model: ollama_chat/qwen2.5-coder:14b
Git repo: .git

I finally knew the basic setup was working correctly.

Opening the Repository from Windows Explorer

For convenience, I also wanted to access the repository directly from Windows Explorer.

From inside WSL:

explorer.exe .

This command opens the current WSL directory directly in Windows File Explorer.

What's Next?

Once I had Aider working with a local Ollama instance, there was still one thing I wanted to understand better: what was happening under the hood. How many tokens were being processed? How long did inference take? What kind of throughput could I expect?

We'll cover all of that in the next installment. 😉

If AI writes code, what is our job now?

eleonorarocchi — Wed, 01 Jul 2026 10:38:16 +0000

Anthropic published a very interesting article on recursive self-improvement: the possibility that AI systems will increasingly contribute to building better future versions of themselves.

Article: https://www.anthropic.com/institute/recursive-self-improvement

It is a huge topic, almost science fiction: AI developing AI, increasingly fast improvement cycles, systems capable of automating growing parts of research and development.

But reading the article as a developer, I think the most interesting question is not:

Will AI replace us?

The more useful question today is another one:

If AI can write, test, refactor, and review code better and better, what is the real role of humans in software development?

For years, we have measured a developer's work partly by their ability to produce code: building features, fixing bugs, optimizing parts of a system, handling refactoring, and doing reviews.

Today, however, an increasingly large part of these activities can be accelerated by AI tools.

This means that the value of the developer is shifting from "doing" to "deciding well what should be done."

One of the most important points in Anthropic's article is that AI becomes very strong when the goal is clear, especially on well-defined tasks.

But are we, as developers, still truly understanding what is being produced?

Because if AI generates code that works, but we can no longer explain it, maintain it, or assess its risks, then we have not really increased our productivity. We have only pushed technical debt a little further down the road.

The bottleneck

In many teams, the historical problem was producing enough code. But with today's AI tools, the bottleneck shifts: it becomes the human ability to review, validate, and understand.

If a tool can generate in a few minutes what used to take hours, the critical point becomes how quickly we can understand, verify, and maintain it.

This is where the developer of the near future is defined: someone who owns judgment skills that cannot be delegated to AI.

And so my provocation is this:

What if part of the low productivity of those teams does not depend on the tools, but on the fact that they keep working as if we were still in 2015?

Today we have tools that can accelerate coding, debugging, testing, review, documentation, and learning. But if we use them with old mindsets, rigid processes, and poorly collaborative habits, the gain remains limited.

Generator-Evaluator Loops for AI Agents

eleonorarocchi — Fri, 22 May 2026 05:26:00 +0000

TL;DR

Separating the generator from the evaluator improves quality and reduces premature self-validation.
The loop works best when feedback is explicit and based on clear rubrics, especially for subjective or complex tasks.
It is useful when the task has high value; for simple or easily testable tasks, it can become overengineering.

How to Separate Production and Evaluation in Tasks Without Ground Truth

If self-evaluation is a recurring failure mode in AI agents, the most natural solution is to separate the role of producing from the role of evaluating.

This is the principle behind generator-evaluator loops: one agent produces an output, another evaluates it according to explicit criteria, feedback is sent back to the generator, and the system iterates until the result reaches an acceptable threshold.

The pattern is simple to describe, but powerful, especially in domains where stable ground truth does not exist: design, writing, UX, naming, strategy, documentation, software architecture, complex refactoring.

In these cases, however, it is not enough to ask the model to "do better." You need to build an environment where improvement is driven by separate, structured, and repeatable critique.

From a Single Agent to an Agentic System

A single agent tends to merge three different roles: it plans the work, produces the output, and finally evaluates the result.

This fusion is convenient but fragile. While it may work for short tasks, in longer ones it often leads to premature convergence, because the same agent chooses a direction, develops it, then positively self-evaluates and closes the task.

To create a more robust system, these roles should be separated into at least three distinct agents:

the planner, which breaks the request into manageable parts and defines priorities, constraints, and work sequence;
the generator, which produces the artifact (e.g. code, interface, document, proposal, strategy, ...);
the evaluator, which judges the output without having participated in its generation.

This separation almost resembles a miniature organizational structure, and in fact, similarly, it helps reduce coupling between decision-making, production, and approval. As in a company, each role has a different objective, and this difference introduces useful friction.

The Analogy with GANs

The most immediate parallel is with Generative Adversarial Networks (GANs).

In a classic GAN, there are two neural networks: a generator and a discriminator. The generator produces synthetic data, for example images, while the discriminator receives both real and generated data and attempts to distinguish between them. In this way, the generator improves by trying to produce outputs plausible enough to fool the discriminator, while the discriminator improves by becoming better at detecting artificial outputs.

This idea has been applied across many domains: image generation, face synthesis, super-resolution, computer vision, synthetic data generation, image-to-image translation, music, and video. Examples include models such as StyleGAN, CycleGAN, TextGAN, and MuseGAN, which show how the generative-adversarial principle can be adapted to different forms of content.

In our case-AI agents-the analogy is useful because it captures an architectural intuition: a generative system improves when exposed to separate judgment.

There are, however, important aspects that should not be confused with the comparison, because a generator-evaluator loop is not a GAN in the technical sense.

In real GANs, there is a mathematical loss function: the discriminator is trained on real data, while the generator receives an optimization signal through gradients.

In AI agents, by contrast, feedback is linguistic and heuristic: the evaluator does not directly update the generator's weights, nor does it necessarily have access to a dataset of real examples.

When an evaluator judges a landing page, a piece of writing, or a strategy, it is not distinguishing "true" from "false" in the GAN sense. It is estimating how closely the output aligns with a set of preferences, criteria, and conventions.

Making Subjectivity Evaluatable

The core challenge is turning vague judgments into observable criteria.

Saying "this UI is ugly" does not help an agent improve-just as saying the same thing to a human designer would not.

Saying "the visual hierarchy does not sufficiently distinguish between the primary action and secondary content" is much more useful.

An effective evaluator needs what are commonly called evaluation rubrics.

In frontend design, for example, a rubric might include four dimensions:

Design quality, which measures whether the result feels like a coherent system or merely an assembly of components. It evaluates visual identity, creative direction, color usage, typography, rhythm, and layout.
Originality, which measures whether intentional choices are present or whether the output looks derived from templates, library defaults, and generic patterns (this includes recognizing AI slop: predictable gradients, white cards, interchangeable hero sections, stock icons, personality-less compositions).
Craft, which measures execution: spacing, contrast, alignment, typographic hierarchy, color consistency, and attention to detail.
Functionality, which measures usability: whether users understand what to do, find the primary actions, navigate without ambiguity, and complete intended tasks.

These criteria do not all need to carry the same weight. In many cases, models are already reasonably strong in craft and functionality; they produce orderly layouts, readable text, and understandable structures. The recurring issue is often the lack of originality and direction.

For this reason, an evaluator focused on real quality should penalize not only errors, but also blandness.

The same principle can be applied to other domains.

For writing, a rubric might evaluate thesis strength, argumentative structure, informational density, rhythm, specificity, voice, redundancy, and the ability to anticipate objections.

An AI-generated text may be correct, fluent, and well formatted, yet still fail to say anything memorable. In that case, a useful evaluator should be able to distinguish between superficial clarity and real argumentative value.

For strategy, a rubric might evaluate diagnostic quality, explicit assumptions, trade-offs, feasibility, prioritization, risks, dependencies, metrics, and contextual alignment.

And what about code? Beyond tests, a rubric should assess simplicity, maintainability, consistency with the codebase, error handling, extensibility, technical debt introduced, readability, and impact on existing abstractions.

The Operational Loop

A typical generator-evaluator loop follows a simple sequence:

the system receives a specification;
the planner breaks it into subtasks and defines success criteria;
the generator produces a first version of the output;
the evaluator inspects the artifact using explicit criteria and, when possible, real tools such as browsers, test runners, screenshots, parsers, linters, metrics, documentation, or access to the codebase;
the evaluator assigns scores, identifies issues, and produces actionable feedback;
the generator receives the feedback and iterates.

The cycle continues until the output exceeds a threshold, the budget is exhausted, or the system determines that human intervention is required.

The core of the loop is actionable feedback, because the evaluator must not only detect failure, but also make it correctable.

A generator-evaluator loop is fundamentally a matter of harness engineering, because the evaluator must have access to the right tools.

That said, an evaluator does not automatically improve simply because it is separated from the generator. For it to work, it must be calibrated using explicit criteria, scoring scales, potentially few-shot examples, thresholds, and similar mechanisms.

When to Use This Pattern

A generator-evaluator loop has a cost: more model calls, more tokens, more latency, more orchestration, and more complexity. As a result, it is impossible to apply it everywhere.

As a rule of thumb, if a reliable automated test exists, it is usually better to use it.

And if the task is simple or low-value, a complex loop may genuinely become overengineering.

Why AI Agents can’t judge themselves

eleonorarocchi — Fri, 15 May 2026 05:19:00 +0000

TL;DR

AI agents tend to overestimate the quality of their own outputs when there is no external verification criterion. In subjective tasks (design, writing, UX, naming, strategy), simply asking the model to "reflect" is not enough: it often remains trapped in the same trajectory that produced the first plausible solution, leading to weak critiques and superficial improvements.
Achieving real quality requires designing the runtime around the model: tests, rubrics, separate evaluators, external tools, and generator-evaluator loops that introduce critical distance between the system that produces the output and the one that approves it.

Why Internal Feedback Is Not Enough in Subjective Tasks

Sometimes, when you ask a model to evaluate a response it previously generated, it will rate it as good even when it clearly is not. Buggy code gets labeled "production-ready"; a generic layout is described as "modern and coherent"; a technically correct but flat piece of writing is called "clear, incisive, and well-structured."

This behavior becomes especially evident when the task lacks binary verification.

If the agent has to write a function and there is a reliable test suite available, the system has access to an external oracle: the test either passes or fails.

But as soon as the task moves into domains such as design, writing, naming, UX, strategy, or product architecture, quality can no longer be reduced to an assert.

This is where the self-evaluation problem in AI agents emerges: the same system that produces the output struggles to judge it with enough critical distance. And, if we think about it, humans often behave the same way.

The point, however, is not that LLMs have ego, self-esteem, or a desire to appear competent. Saying that agents "self-promote" is a useful shortcut, but technically inaccurate. A model is not trying to convince us that its output is good. More often, it simply remains inside the same probabilistic trajectory that generated the artifact in the first place.

If we ask:

Generate a landing page for a SaaS product.

And immediately afterward:

Evaluate the quality of the landing page.

we have not really created two distinct processes. We have asked the same model to continue reasoning within the same semantic space, with the same context, assumptions, and implicit orientation toward completing the task.

The result is often an evaluation that is overly generous, poorly discriminative, and not very useful for driving meaningful improvement.

Tasks With an Oracle and Tasks Without One

We can distinguish between two classes of tasks.

The first includes tasks with an external oracle. These are cases where quality can be verified relatively objectively: an automated test, a query returning an expected result, a formal constraint, a compiler, syntactic validation, or a numerical measurement.

In software engineering, many tasks fall at least partially into this category. Code can be evaluated through unit tests, integration tests, type checkers, linters, benchmarks, and static analysis. These tools do not fully capture software quality, but they provide strong signals. If an agent produces code that does not compile or breaks a test suite, the system does not need to "guess" that something is wrong: it knows.

The second class includes tasks without a clear oracle. Here, quality is subjective, multidimensional, or context-dependent. A UI can be technically correct but visually uninspired. A text can be grammatically flawless but lack a real thesis. A strategy can be well formatted yet impossible to execute. A naming proposal can be understandable but forgettable.

In these cases, the problem is not just verifying whether the output is correct, but determining whether it is actually good.

And unfortunately, "good" does not mean one single, clearly identifiable thing.

In design, it may mean visual coherence, originality, hierarchy, usability, or identity; in writing, clarity, density, rhythm, argumentative strength, or voice; in strategy, accurate diagnosis, explicit trade-offs, feasibility, specificity, and contextual alignment.

When an external oracle is missing, the agent tends to rely on its own linguistic evaluation. And that is exactly where the system becomes fragile.

The Failure Mode: Premature Convergence

The most common failure mode is not catastrophic error, but premature convergence.

The agent produces a plausible solution, refines it superficially, and declares it sufficient. The result is not necessarily wrong. Often, it is worse: mediocre but defensible.

This "plausible mediocrity" is difficult to detect because it contains many superficial signals of quality.

An AI-generated landing page will probably include a hero section, a CTA, a feature grid, a few cards, a responsive layout, a pleasant color palette, and tidy copy. A strategy document will include neatly titled sections, bullet points, frameworks, and recommendations. A refactoring will contain cleaner names and a few extra abstractions.

But all of this can still remain generic.

The agent tends to improve what it has already produced instead of questioning whether the direction itself is correct. It polishes the first solution instead of challenging it. It adds local coherence, not global quality.

This is where self-evaluation fails: not because the model cannot recognize any errors, but because it often does not apply criticism strong enough to break away from the first acceptable solution.

The Limits of Reflective Prompting

One of the earliest responses to this problem was reflective prompting: asking the model to critique its own output, identify issues, propose improvements, and iterate.

This approach works to some extent because it can eliminate obvious errors, improve clarity, fix inconsistencies, and add missing details. However, its main limitation is that the critique remains inside the same process that generated the output.

Prompts such as:

Reflect on your work and improve it.

or:

Identify any problems in the previous response.

often produce generic feedback:

"It could be more specific";
"Clarity could be improved";
"I would add more concrete examples";
"The structure is already solid, but it could be refined."

These observations are true, but weak, and they rarely lead to a substantial change in direction.

For simple tasks this may be enough, but for high-value tasks it often is not.

Why Runtime Matters

This problem contributed to the rise of harness engineering: designing the runtime around the model.

As I described in previous articles about harnesses, the core idea is that the performance of an agentic system depends not only on the model itself, but also on the operational environment in which the model works. What matters is how the prompt is constructed, which tools are available, how context is managed, how intermediate states are stored, how tests are executed, how feedback is orchestrated, when the system decides to iterate, and when it decides to stop.

In modern agentic systems, the model is just one component. Final behavior emerges from the interaction between the model, tools, memory, context, schedulers, evaluators, acceptance criteria, and retry mechanisms.

This shift in perspective is fundamental. If the model struggles to evaluate itself, the solution is not necessarily to wait for a better model. It is to design a runtime that makes the evaluation process less fragile.

In coding, this may mean running tests, reading errors, applying patches, and retrying. In design, it may mean generating screenshots, navigating the interface, and verifying interactive states. In writing, it may mean using editorial rubrics, comparing versions, and evaluating density and redundancy. In strategy, it may mean making assumptions explicit and testing alternative scenarios.

The runtime introduces signals that the model alone does not reliably produce.

Critical Distance as an Architectural Requirement

The self-evaluation problem can be summarized like this: generation and evaluation are too close to each other.

Critical distance can be introduced in many ways. Sometimes changing the prompt, role, or critique format is enough; other times it requires a different model, a different temperature, a stricter rubric, few-shot examples, external tools, or a separate agent.

The principle remains the same: the system must create a separation between the entity that produces and the entity that approves.

This separation does not guarantee perfect evaluation, but it reduces the risk that the agent settles for the first plausible solution.

This naturally leads to the generator-evaluator pattern: one agent produces, another evaluates, feedback returns to the first, and the cycle continues until the output surpasses a threshold.

It is not always necessary: for simple tasks it can become overengineering.

But for subjective, long, or high-value tasks, it becomes one of the most useful patterns in agent engineering.

How Stripe, Shopify, and Airbnb Build AI Harnesses

eleonorarocchi — Sat, 09 May 2026 07:04:00 +0000

TL;DR

There is no single model of harness engineering.
OpenAI builds repository-centered harnesses, Anthropic focuses on agent cognitive continuity, while companies like Stripe, Shopify, and Airbnb develop vertical harnesses built around compliance, context, and action verification.
Harness engineering is becoming a domain-specific discipline, shaped by the type of risk each company needs to control.

Why There Is No Single Harness: Stripe, Shopify, Airbnb, and the Industrial Fragmentation of Agent Engineering

After observing OpenAI's repository harness and Anthropic's runtime harness (if you haven't done so already, read my articles: OpenAI and the New Cognitive Architecture of Software Repositories e Anthropic and the Runtime Harness for Persistent Agents), one might expect the industry to be converging toward a fairly clear formula: define memory, tools, feedback loops, constraints, and let the agents work.

In reality, the opposite is happening: the more public big-tech case studies become, the more it becomes clear that the word harness is starting to cover profoundly different architectures.

I find this to be the most interesting signal of the sector's maturation, because it means we are no longer witnessing the birth of a standard, but rather the emergence of multiple implementation paradigms.

The comparative analyses published about companies like Stripe, Shopify, and Airbnb demonstrate this very clearly.

The Point Is Not Model Capability. It's the Cost of Failure.

As long as we talk about coding agents in the abstract, there is a tendency to imagine that the problem is singular: making the model more reliable.

However, in industrial environments, reliability is not a neutral category; it depends on what the company considers tolerable or intolerable. That is where the divergence begins.

Stripe: The Harness as a Compliance Boundary

In the financial domain, the problem is not only producing a correct modification, but producing a modification that does not violate policies, introduce vulnerabilities, alter critical transactional flows, and remains fully auditable.

In this context, the harness tends to become an approval gate, with automated validation, side-effect simulation, and above all, compliance controls.

The agent does not operate in an open environment, but inside a risk-clearing chamber.

The harness is primarily a containment boundary.

Shopify: Harnesses for Context Distribution

Shopify's problem is almost the opposite: the commerce domain is hyper-fragmented, with different themes, plugins, merchant logics, and unpredictable customizations.

The primary risk, beyond causing damage, is producing something generically correct but locally useless.

For this reason, the harness must excel at contextual retrieval, access to internal documentation, merchant-state simulation, and precise distribution of relevant information.

The model must not only be safe, but operate with an accurate understanding of the merchant's specific context.

Airbnb: Harnesses as Perceptual Verifiability

In customer-facing and UI-heavy workflows, the problem changes once again, because an agent can propose a technically reasonable modification while still breaking selectors, navigation, UX flows, or intermediate states.

In cases like Airbnb, the harness emphasizes browser instrumentation, screenshot verification, replayability, and control over executed actions.

The core question becomes: does the action actually produce the intended effect in the user environment?

The harness therefore becomes a perceptual surface.

From Best Practices to a Domain-Specific Discipline

What these cases show is that a harness is not a universal checklist of components.

A harness is a response to the failure modes that each organization considers economically most dangerous:

for OpenAI, the risk is codebase entropy;
for Anthropic, it is cognitive drift;
for Stripe, regulatory side effects;
for Shopify, the loss of situational context;
for Airbnb, the non-verifiability of actions.

Same word, completely different problems.

The Real Maturity of the Industry Is This Fragmentation

We often interpret fragmentation as a lack of standards. But I would argue the opposite can also be true: when a discipline is young, everyone uses the same generic formulas; as it matures, specialized architectures begin to emerge.

That is perhaps exactly what is happening with harness engineering.

Just like choosing the best TypeScript framework, we are now entering the phase where the real question becomes: which harness architecture is most coherent with the type of risk my agent cannot afford to tolerate?

Anthropic and the Runtime Harness for Persistent Agents

eleonorarocchi — Fri, 01 May 2026 05:13:00 +0000

TL;DR

Anthropic shows that the real challenge for AI agents is not starting a task, but staying coherent throughout long executions.
Avoiding cognitive drift requires a runtime harness built on external memory, checkpoints, and continuous re-anchoring.
The next frontier is not autonomy alone: it is cognitive continuity.

Anthropic and the Runtime Harness: the Real Problem with Agents Is Not Acting, but Not Getting Lost While They Act

If the OpenAI case showed how a repository can be rethought to become readable for agents, the contribution published by Anthropic in Harness design for long-running application development opens an even more delicate question: what happens when the challenge is no longer how to start a task well, but how to keep it alive for hours?

Because this is where many agentic systems truly begin to break.

Not at the first tool call, nor at the first planning step, but perhaps at the twentieth minute-when context starts to thin out, micro-errors begin to accumulate, and the agent keeps acting while preserving only the illusion of coherence.

In its article Harness design for long-running application development, Anthropic puts its finger exactly on this point: the frontier of agent engineering is not simply autonomy, but the persistence of autonomy over time.

The Most Underestimated Failure Mode: Cognitive Drift

Many agents appear to work well as long as we observe them on short tasks:

generating a component;
fixing a function;
calling two or three tools.

But when the task stretches across dozens of files, multiple review phases, intermediate validations, and distributed dependencies, a phenomenon begins that is very familiar to those who use them in real settings: the agent continues to produce output, but progressively loses the center of its own intention.

Anthropic treats this as a structural problem, not as a simple "model limitation": and this is precisely where the runtime harness emerges.

From the Context Window to External Cognition

The starting point is almost brutal: the context window, by itself, is too fragile a memory to sustain long-running tasks.

Even with very large contextual windows, the model suffers from imperfect compression, unstable salience, priority loss, and partial retrieval of goals.

For this reason, Anthropic builds around Claude an external procedural memory composed of persistent scratchpads, task files, execution summaries, serialized checkpoints, and continuously updated state notes.

In practice, the model is no longer forced to "remember everything", because it can reread what it has already established.

This makes an enormous difference.

The Harness as a System of Continuous Re-Anchoring

In the classical paradigm, we tell the agent: continue.

In the Anthropic paradigm, instead, we tell it: stop, reread where you are, summarize what you are doing, update your state, then continue.

This creates a re-anchoring cycle.

The agent is periodically brought back to the goal, to the progress already completed, to the constraints still open, and to the errors that have emerged.

It is a form of "artificial continuity".

Cognition is not allowed to flow in a monolithic way; it is broken apart, recorded, and reconsolidated.

Multi-Agent Evaluation: Thinking Is Not Enough, You Need to Be Critiqued

Another interesting aspect of Anthropic's work is the use of generator/evaluator structures: one agent produces, and a second agent evaluates quality, coherence, usability, and adherence to requirements.

The result is not simply "more review".

It is something subtler: verification stops being a final phase and becomes part of cognitive continuity itself.

In this way, each evaluation prevents the primary agent from drifting too far away from the correct trajectory.

The Runtime Harness Is Not Meant to Make the Agent Act Better: It Is Meant to Make It Think Longer

This is perhaps the most important point: while OpenAI builds above all a structural harness, Anthropic builds above all a temporal harness.

The problem it is solving is no longer "how do I get Claude to generate good code?", but "how do I prevent Claude from losing the thread while it continues generating it?".

It sounds like a nuance, but it completely changes the design, because here:

the memory is an external artifact,
planning is serialized,
review is recurrent,
the task is continuously re-anchored.

So this is not only orchestration-it is assisted cognitive continuity.

Conclusion

If OpenAI's repository harness teaches us that an agent needs to live inside a readable codebase, Anthropic reminds us that this is not enough.

An agent may have perfect tools, perfect documentation, perfect constraints and still get lost if it is allowed to run for too long without an external memory that keeps it coherent.

And this is where the runtime harness changes the game: it does not merely build an environment in which the agent can act; it builds an environment in which the agent can continue to know why it is acting.

In agent engineering, this may be the difference between episodic automation and real autonomy.

OpenAI and the New Cognitive Architecture of Software Repositories

eleonorarocchi — Tue, 28 Apr 2026 05:36:00 +0000

TL;DR

OpenAI's latest harness engineering report suggests something deeper than "agents can write a lot of code."
It suggests that the real bottleneck in agentic software is no longer just the model, but the repository itself.
Once agents become primary executors, codebases must stop being designed only for human maintainers and start becoming semantically navigable computational environments.

OpenAI and the Birth of the Repository Harness: When Code Must Become Readable to Agents

Over the past few months, the concept of harness engineering has become one of the most frequently discussed categories in AI engineering, especially as companies have started confronting a very simple problem: an agent may be brilliant in isolated executions, but without an environment intentionally designed around it, it quickly begins to generate entropy.

As I discussed in my previous article,Harness Engineering: The Most Important Part of AI Agents harnesses represent the truly critical layer of an agentic system, and this infrastructure must evolve significantly when moving from prototype to production.
The case recently published by OpenAI, however, adds an even more important piece to the puzzle: it suggests that the first object we need to learn how to design for agents may not be the model itself, but the repository.

The Number Everyone Quoted — and the One That Actually Matters

In the report Harness engineering: leveraging Codex in an agent-first world, OpenAI explains that it built a functional internal beta with roughly one million lines of code generated entirely by Codex, zero manually written lines, and more than 1,500 pull requests handled by an extremely small team.

It is an impressive figure, and naturally it made headlines.
But stopping at the quantity means missing the central point.

The real message of the report is something else:

productivity did not increase because Codex "writes code very fast";
it increased because engineers stopped treating the repository as a simple container of files and started treating it as an environment computable by agents.

In other words, OpenAI did not simply use a coding agent inside a codebase: it transformed the codebase into something an agent can read, interpret, and correct reliably.

From Human Codebase to Agent-Readable Codebase

There are at least four very clear signals of this transformation.

1. Repository Knowledge Becomes the System of Record

OpenAI insists on one precise point: the repository must contain the operational truth.

This means:

versioned internal documentation;
architectural maps;
decision histories;
files such as AGENTS.md that function as a semantic entry point for agents.

This is not about adding "more documentation," but about ensuring that the repository becomes machine-queryable memory, not merely something readable by humans.

The agent should not have to infer structure from scattered code; it should be able to interrogate that structure directly.

2. CI Stops Being Just Quality Assurance and Becomes a Runtime Training Mechanism

Linting, formatting, boundary checks, import policies, automated verification: in a traditional pipeline these serve to maintain order, while in a repository harness they serve something more: they become deterministic feedback loops that continuously teach the agent which behaviors are allowed and which are not.

The agent makes a mistake, CI blocks the execution, the log returns the reason, the task is iterated again: quality control stops being post-production and becomes part of the execution-time reasoning process.

3. Observability Is Designed for the Agent Too

OpenAI explains that it invested heavily in structured logs, diagnostic traces, verifiable outputs, and inspection tools.

This is because an agent that cannot properly read its own failures is forced to regenerate blindly; conversely, an agent with access to semantically dense error information can perform self-debugging.

Observability, therefore, is no longer just a developer dashboard: it becomes a cognitive surface.

4. Developers Stop Being Authors of Code and Become Authors of Constraints

This is perhaps the most interesting point in the entire OpenAI article: human work does not disappear, it shifts.

Less time spent on:

direct implementation;
manual fixes;
tactical coding.

More time spent on:

designing repository structure;
defining architectural boundaries;
building feedback loops;
cleaning entropy.

The engineer writes fewer and fewer features, and more and more conditions of intelligibility.

The Repository Harness as the New Unit of Design

If we look closely, the OpenAI case suggests a strong thesis: the first mature industrial harness is not simply a wrapper around the model; it is a codebase deliberately made readable to agents.

And this is an important distinction.

For years we assumed that the agent problem was primarily about improving:

prompting;
reasoning;
tool use.

OpenAI shows that there is an upstream layer beyond all of that:

a mediocre agent inside an agent-readable repository can still produce usable work;
a highly capable agent inside an opaque repository will still produce entropy.

The bottleneck is not only the model, but increasingly the computability of the environment.

Conclusion

Perhaps OpenAI's most interesting contribution to the harness engineering debate is not having shown that software can be built with agents.

It is having shown that, to do it seriously, we need to accept one uncomfortable fact:

it is no longer enough for code to be maintainable by humans;
it must become navigable, verifiable, and semantically readable by agents.

And this radically shifts the work of engineering.

We are no longer designing only applications — we are (perhaps finally) beginning to design repositories that can be inhabited by non-deterministic intelligences.

Building a Harness: From Prototype to Production

eleonorarocchi — Fri, 24 Apr 2026 05:12:00 +0000

TL;DR

An agent doesn’t truly work because of the model, but because of the harness controlling it.
Moving from demo to production requires handling errors, state, memory, and observability.
A well-designed harness reduces model unpredictability and shifts complexity into code, making the system reliable and usable in real-world scenarios.

In article Harness Engineering: The Most Important Part of AI Agents we saw a fundamental point: the problem with agents isn't (only) the model, but the system around it.

But what does it really mean to build that system?

The moment everything breaks

There's a fairly universal phase: you've implemented a demo, it works well, the model responds, uses a tool, maybe even completes multi-step tasks, and everything looks promising.

Then you try to use it in a real-world context, and the problems emerge:

invalid outputs
incorrect API calls
infinite loops
loss of context

It's not that the model got worse—the system's complexity increased without having a harness solid enough to manage it.

The harness as a control system

It becomes clear that the harness isn't just a "container"—it's more like a control system designed to guide the model along a precise path, reducing its freedom when necessary and allowing it when useful.

This is a delicate balance: too much control means loss of flexibility; too little control means loss of reliability.

And this is where the real design work begins.

Error handling becomes the main case

In traditional software, errors are edge cases. Anyone with experience in agent-based systems knows that errors are the norm.

The key idea, however, is that a well-designed harness does not assume everything will go well—quite the opposite.

It therefore introduces mechanisms such as:

validating outputs before using them
retrying when something goes wrong
falling back to alternative paths
controlled interruption of loops

This is what makes the system usable.

State and memory: the invisible problem

Another issue that emerges very early is state management: an agent without memory is little more than a stateless function—but adding memory introduces complexity:

what to store
for how long
how to update the state
what happens when it becomes inconsistent

These decisions must be made when structuring the harness.

And it's precisely here that many subtle bugs tend to arise.

Observability: knowing what's happening

When something goes wrong (and sooner or later it will), the important question is:

"Can I understand what happened?"

Without logging and tracing, working with agents becomes almost impossible.

Because you need to see:

every step of the reasoning
every tool call
every output transformation

And not just for debugging, but to evolve the system.

Moving complexity to the right place

An interesting aspect is that, as you improve the harness, the system becomes more predictable—even without changing the model.

This happens because complexity is being moved out of an "opaque" component (the model) and into code that can actually be controlled.

It's a shift in strategy:

less blind trust in the model
more explicit control in the system

Which, ultimately, is software engineering.

In fact, we can say that building agents today is much closer to traditional software engineering than it might seem.

There are flows, states, error handling, integrations, observability…

The only difference is that instead of deterministic functions, there's a probabilistic model.

The harness is what holds everything together—and that's what makes the difference between something that only works in a demo and something that truly works in production.

DEV Community: eleonorarocchi

How I Use Codex to Build Applications Without Losing Control

The AI Layer in the Original Workflow

Commands and Skills in Codex

Porting the Workflow to Codex

From CLAUDE-template.md to AGENTS.md

A Practical Experiment: A Todo List

From the Brief to the PRD

Adding Project Rules

The First Context Reset: $prime

The PIV Cycle

Plan: Turning a Phase into a Plan

Implement: Executing the Plan in a New Context

Validate: Verifying the Result

Git as Long-Term Memory

Conclusion: A Lightweight Method or a Complete Framework?

Repo

Setting Up a Local AI Coding Agent with Ollama and Aider (part 4)

Limitations

Why I Did Not Use It with Aider

Installing the Hugging Face CLI

Downloading the GGUF Model

Building the Dedicated Runner

Running the Model

The Bottleneck

Setting Up a Local AI Coding Agent with Ollama and Aider (part 3)

Dashboard

Streamlit

Grafana

Monitoring Directory

Prometheus Exporter

Prometheus

Grafana Data Source

Docker Compose

Useful Grafana Queries

Daily Workflow

Setting Up a Local AI Coding Agent with Ollama and Aider (part 2)

Metrics

Architecture

Creating the Proxy

Starting the Proxy

Running Aider Through the Proxy

Inspecting the JSONL Log

Creating an Alias

What I Learned

Setting Up a Local AI Coding Agent with Ollama and Aider

Local Setup

Installing Ollama on Windows

The WSL-to-Windows Connectivity Issue

Installing Aider in WSL

Creating the First Repository

Opening the Repository from Windows Explorer

What's Next?

If AI writes code, what is our job now?

The bottleneck

Generator-Evaluator Loops for AI Agents

TL;DR

How to Separate Production and Evaluation in Tasks Without Ground Truth

From a Single Agent to an Agentic System

The Analogy with GANs

Making Subjectivity Evaluatable

The Operational Loop

When to Use This Pattern

Why AI Agents can’t judge themselves

TL;DR

Why Internal Feedback Is Not Enough in Subjective Tasks

Tasks With an Oracle and Tasks Without One

The Failure Mode: Premature Convergence

The Limits of Reflective Prompting

Why Runtime Matters

Critical Distance as an Architectural Requirement

How Stripe, Shopify, and Airbnb Build AI Harnesses

TL;DR

Why There Is No Single Harness: Stripe, Shopify, Airbnb, and the Industrial Fragmentation of Agent Engineering

The Point Is Not Model Capability. It's the Cost of Failure.

Stripe: The Harness as a Compliance Boundary

Shopify: Harnesses for Context Distribution

Airbnb: Harnesses as Perceptual Verifiability

From Best Practices to a Domain-Specific Discipline

The Real Maturity of the Industry Is This Fragmentation

From `CLAUDE-template.md` to `AGENTS.md`

The First Context Reset: `$prime`