DEV Community: Nunc

I Replaced an Ad-Stuffed Download Site With 463 Lines of Node

Nunc — Tue, 01 Sep 2026 08:15:22 +0000

I wanted to save X/Twitter videos on my phone, but there was a catch: X has no download button, and the download sites that fill that gap are ad-stuffed, break whenever X rotates its API auth, and you hand every tweet URL to a stranger. So I built my own: a self-hosted web app where you paste a tweet link and get a preview, a quality picker, and a Download button that saves the MP4 to your phone.

The whole thing is 463 lines of Node with zero npm dependencies, plus 224 lines of vanilla JS for the UI. It is open source (MIT):

nuncij / twitter-x-downloader

X/Twitter Downloader (xdl)

Paste any x.com / twitter.com tweet link → preview → download. A small, self-hosted web app for saving videos (and GIFs/photos) from X. Mobile-first, works great on phones.

You host it yourself — there's no hosted instance. It runs on your own server (or just your laptop), so it's free, private, and no third-party download site is involved.

Features

📱 Mobile-first UI (dark theme, add-to-home-screen friendly)
🎬 Quality picker with file sizes (270p → 720p+); picks the best available
🎞️ Handles videos, GIFs, photo-only tweets, and mixed video+photo tweets
🔗 1080p HLS-only tweets are merged server-side via ffmpeg automatically
⚡ Direct CDN streaming for downloads (no big server-side buffering), inline preview plays straight from X's CDN
🛡️ Per-IP rate limits, concurrency caps, CDN host allowlist (no SSRF)
🔌 Zero npm dependencies — only node, ffmpeg, yt-dlp

How it works

Extraction: yt-dlp is…

View on GitHub

Two backends, because X breaks things

The server never talks to X's private API itself. That is the part that breaks every few months, so I delegated it:

yt-dlp (primary). One call, yt-dlp -J --no-playlist --skip-download <url>, returns every available format with file sizes. yt-dlp handles Twitter's guest-token flow and rotating auth, and it keeps working when X changes things because it is actively maintained. Extraction takes 1 to 2 seconds.
FxTwitter API (fallback). api.fxtwitter.com/status/{id}, free, no auth, answers in 200 to 300 ms. It steps in when yt-dlp hits a guest-token rate limit, and it covers what yt-dlp does not return at all: photo-only tweets and GIFs.

If one backend dies tomorrow, the app degrades instead of dying with it.

The format puzzle

Twitter serves each video in several parallel formats. A real 15-second clip came back like this:

format_id	kind	resolution
`hls-audio-*`	HLS audio only	32/64/128 kbps
`http-288`, `http-832`, `http-2176`	progressive MP4	272p / 364p / 728p
`hls-191`, `hls-343`, `hls-716`	HLS video only	272p / 364p / 728p

Two useful facts fell out of staring at that list. Progressive MP4 (one file, video plus audio, plays anywhere) exists up to 720p for most tweets. Anything above 720p is HLS only, so for 1080p the server lets yt-dlp download and merge the streams with ffmpeg, waits for the temp file, streams it to the phone, and deletes it. The merge of a 15-second clip took 1.8 seconds.

Then the gotcha that cost me the most debugging time. yt-dlp's JSON reports vcodec and acodec as null for exactly those progressive formats, not as h264/aac. My first classifier dropped anything without a known codec, which silently threw away every good single-file MP4 and sent every request to the fallback. The fix is to classify by protocol instead of trusting the codec fields:

// kind: 0=mix, 1=video only, 2=audio only
function classifyFormat(f) {
  const a = f.acodec && f.acodec !== 'none';
  if (f.vcodec === 'none') return 2;
  if (a || String(f.protocol).startsWith('http')) return 0; // progressive single-file = mix even when codecs unknown
  return 1;
}

protocol: "https" means a progressive single file. vcodec === 'none' (the string, not null) means audio only. Everything else is an HLS video track that needs a merge.

The phone part

The preview costs the server nothing. X's CDN (video.twimg.com) serves videos with access-control-allow-origin: * and plain GET, no cookies, so the <video> element points straight at the CDN through a 302 redirect. Playback bandwidth: zero.

The download is the opposite: it streams through the server on purpose, with Content-Disposition: attachment and a filename built from the tweet (author - title (720p).mp4). That header is what makes iOS and Android reliably offer "save to Files" instead of opening the video in a tab. Node pipes the CDN response through with the original Content-Length, nothing is buffered in RAM.

The UI is one index.html: dark theme, big touch targets, quality pills with file sizes ("720p · 4.0 MB"), and the meta tags for add-to-home-screen, so on the phone it launches fullscreen like an app.

Internet-facing means paranoid

A downloader is a proxy, and a naive proxy is an SSRF hole. Three guards, all in those 463 lines:

The server only fetches URLs whose host, re-parsed server-side, is video.twimg.com or pbs.twimg.com, https only. The client cannot point it anywhere else.
Per-IP rate limits: 6 extracts and 10 downloads per minute, in-memory sliding window.
Concurrency caps: 1 extraction and 2 downloads at a time. It is a personal tool on a small VPS, not a public service.

Run it yourself

The repo ships a generic deploy.sh: VPS_HOST=user@server ./deploy.sh rsyncs the app, creates a Python venv for yt-dlp, and installs a systemd unit. The venv matters: Ubuntu's apt version of yt-dlp is months old, and with this tool old means broken.

My own instance never touches the public internet. The DNS record points at a Tailscale address, the reverse proxy binds to the tailnet interface, and the firewall drops everything else. HTTPS still works with zero open ports, because Let's Encrypt supports the DNS-01 challenge: the cert is issued through a DNS API token, no port 80 required. My phone runs Tailscale, so the app works from anywhere and is still invisible to everyone else.

One note on open-sourcing a personal tool: before the repo went public I found my own server details in three places (the deploy script, the README, and one commit message). The first two were rewritten; for the third the history had to be squashed into a single clean commit, because a secret in an old commit is still a published secret.

Wrapping up

The key insight: don't fight X's API yourself. Let yt-dlp absorb the auth churn, keep a free second backend for the day it hiccups, and spend your own lines on the parts that make it yours: the streaming, the filenames, the SSRF guard, and a UI your thumb likes. Mine came to 463 lines, and there is not a single ad on it.

Oracle Designer Lost Support in 2013. My AI Agent Develops in It Anyway.

Nunc — Fri, 21 Aug 2026 11:51:51 +0000

Oracle Designer 10g is a CASE tool from the early 2000s. Support ended in 2013. At my job it is still the center of development: the repository holds the entire design of a large insurance system (2,741 entities, 3,277 table definitions, 12,841 PL/SQL modules, 5,624 forms), and about 10 developers work in it daily. Everyone, me included, assumed you work with Designer through its GUI. Clicking, dragging, repeating.

The question I wanted to answer: can an AI agent that knows SQL and has database access do that work instead?

Two days later the agent had designed an entity, a table, a sequence, two triggers, and a complete form definition, moved a table between applications, and debugged the generated form to a working runtime. It opened the Designer GUI exactly zero times. Everything I personally did in a GUI fits in two buttons: Generate, and starting the form.

The discovery that made it possible

Designer stores everything in an Oracle schema, and next to the data it ships three layers that are all reachable from a plain SQL session:

CI_* views (722 of them) for reading. Every entity, column, and form item is a row in a view.
CIO* packages (about 200) for writing. Every element type has a package with ins/upd/del/sel and a typed record.
CDAPI for transactions: open_activity, then your changes, then validate_activity (Designer's own validation rules), then close_activity. On failure, abort_activity rolls the repository back.

The basic write pattern looks like this:

declare
   ent cioentity.data;
   st  varchar2(10);
   wa  varchar2(2000);
begin
   jr_context.set_workarea('GLOBAL SHARED WORKAREA');  -- skip this: CDR-00100
   cdapi.initialize('MYAPP');
   cdapi.open_activity;

   ent.v.name       := 'MY ENTITY';  ent.i.name       := true;
   ent.v.short_name := 'MYE';        ent.i.short_name := true;
   cioentity.ins(null, ent);

   cdapi.validate_activity(st, wa);
   if st = 'Y' then
      cdapi.close_activity(st);
      if st = 'Y' then commit; else cdapi.abort_activity; end if;
   else
      while cdapi.stacksize > 0 loop
         dbms_output.put_line('VIOLATION: ' || cdapi.pop_instantiated_message);
      end loop;
      cdapi.abort_activity;
   end if;
end;
/

There is no documentation for any of this anymore. It does not matter, because the API is self-documenting: the package specs are readable from all_source, so the agent's first move for every element type was one query away:

select text from all_source
where owner = :repo_owner and name = 'CIOENTITY'
and type = 'PACKAGE' order by line;

An AI agent does not need the GUI. It needs sqlplus.

The loop that worked

The same four steps repeated for every element type:

Read the API from the database. The package spec says which fields the record has.
Read an existing element as the template. There is no house-standards document, but there are 3,277 existing tables. Naming conventions, audit columns, domains, window sizes, preference sets: all of it was read from elements my colleagues built over 20 years. "How we do things here" is written in the data itself.
Write in small validated transactions. One logical change, validate, and on failure print the violation messages and abort.
Verify through CI views before moving on.

With that loop the agent built the full chain: entity with attributes, table with columns on the house domains plus a primary key, the table-to-entity mapping (normally the job of a Designer wizard called the Database Design Transformer), a sequence with the ID trigger, audit columns with their trigger (PL/SQL bodies written through the RMOTEXT API), table ownership moved to another application with a shortcut left in the original, and the complete form definition: module, window, component, table usage, items, preference sets.

Then came the one part SQL cannot do. Generating the actual Forms binary requires a client-side tool, so I pressed Generate in Designer. The generator failed. The agent read the generator output and the .err log file from disk, diagnosed each failure, fixed them through the API, and I pressed the button again. That cycle repeated until the form ran.

One fix deserves a mention. The form compiled and started, but the layout was broken: all fields stacked on top of each other. I gave the agent a screenshot of the running form. It recognized the pattern, found the cause (items without display_width fall back to the column's width, and the layout collapses into a stack) and set the widths on all items through the API. The next generation ran clean.

Errors are data

Around 20 distinct errors came up across the two days. Every one was diagnosable from the message plus the state of the database. A sample:

Error	Cause
`CDR-00100: Workarea context has not been set`	missing `jr_context.set_workarea` before anything else
`ORA-00001` on an internal unique key	reusing a CIO record between `ins` calls: it keeps the previous element's IDs, so reset the record to an empty one
`PLS-00302: component 'INS' must be declared`	some packages are abstract; write through the specific subtype package
`CDG-01199: no queryable item` at generation	items were created without `select_flag = 'Y'`
`identifier must be declared` in the `.err` file	the physical table did not exist yet on the target dev database

None of these are in any manual. All of them are now in ours, because the collected errors turned out to be the most valuable output of the whole exercise. They went into two places: a handbook for humans, and a Claude Code skill for future AI sessions. The next session does not rediscover CDAPI, it starts from a working recipe.

The skill is open source (MIT), with the transaction templates, the per-element procedures, and the full error table:

nuncij / oracle-designer-cdapi-skill

Drive Oracle Designer (6i/9i/10g) with an AI coding agent via its CDAPI/CIO PL/SQL API - entities, tables, forms, no GUI. Claude Code skill + proven SQL templates.

Oracle Designer CDAPI Skill — drive Oracle Designer with an AI coding agent

Oracle Designer (6i/9i/10g) is a CASE tool from the early 2000s, out of support since 2013, yet still the center of development in many legacy Oracle shops. The common assumption is that you can only work with it through its GUI (Repository Object Navigator, Design Editor). That assumption is wrong.

Designer stores everything in an Oracle schema and ships a complete PL/SQL API next to it. That means an AI coding agent (Claude Code, or any tool that can run sqlplus) can read and write the repository directly: create entities, tables, columns, keys, sequences, triggers, and complete form module definitions — validated by Designer's own engine, without a single click in the GUI. The only steps left for a human are pressing Generate (Forms/DDL generators are client-side) and drawing diagrams.

This repository packages that capability as an…

View on GitHub

An agent with write access to production designs needs guardrails

The same access that lets an agent create a table lets it silently damage a colleague's form with one mistyped ID. Before letting it loose we layered defenses: scripts look up elements strictly by name and verify parentage before touching anything, a control report runs after every session and lists everything that user changed that day across 17 element types, and a log table with a nightly job keeps permanent change history, because Designer itself only stores the last change. The nightly database backup stays as the final net. All of it is running, not planned.

Wrapping up

The limits we actually hit: drawing ER diagrams (the layouts are binary blobs) and pressing the two client-side generator buttons. Everything else that "requires the GUI" turned out not to.

The key insight: a legacy tool with no SDK, no docs, and no support is not necessarily closed to AI agents. If it stores its world in a database, it may be the most open tool you have. The dictionary replaces the SDK, existing data replaces the standards document, and error messages replace support. Designer waited 20 years for a user that reads package specs for fun.

My Peer-Reviewed T. rex Looks Worse Than a 1996 Video Game

Nunc — Fri, 14 Aug 2026 10:31:56 +0000

I wanted to know what a Tyrannosaurus rex actually looked like. Not the movie one. The real one, the one the papers describe. So I told AI agents to read the science and build it in 3D, model and animation, with no human touching the mesh.

Three days later I had a potato.

That is the apex predator. Beige, calm, faintly disappointed. It has the body language of a man waiting for a bus.

And here is the annoying part: it is scientifically correct. Every bone length traces to a published paper. The body mass sits inside the window from peer-reviewed volumetric studies. The walk cycle matches a Royal Society paper to three decimals and a script verified it. My dinosaur can cite its sources. It just cannot frighten a chicken.

I am oddly proud of it and I cannot show it to anyone without explaining it first, which is usually a sign you have built the wrong thing.

What the agents taught me about T. rex

I never opened Blender. My job was approving phases and stopping Windows from falling asleep mid-render. The agents did the rest: two Claude Code sessions, one per machine, plus OpenAI's Codex hired specifically to be unpleasant about the plan.

The research was the part I enjoyed, because the agents kept telling me I was wrong about a dinosaur I had been confident about since I was eight:

It had lips. Teeth covered when the mouth is closed. More monitor lizard, less crocodile grin. Sorry, Jurassic Park.
It was scaly, pebbly scales about a millimetre across. On a 12 metre animal that is a shader detail. Nobody sculpts a millimetre.
It was slow. Preferred walking speed 4.6 km/h, set by the resonant frequency of its tail. Top speed maybe 25 km/h, always with one foot down, because an aerial phase would have snapped its legs. The jeep was never in danger.
The tail was the engine. The biggest leg muscle lives in the tail, roughly 260 kg per side. Every skinny-tailed rex you have seen is wrong.
The colour is unknowable. No pigment data exists for any tyrannosaurid. Ours is olive-brown because somebody had to choose, and the repo says so in writing.

Then Codex, the hired pessimist, earned its fee in an afternoon. My research agent had cheerfully declared the Smithsonian's T. rex scan public domain. Codex opened the actual page: only the metadata is CC0, the scan is non-commercial only. A hallucinated licence, caught by a rival model that bothered to click the link.

Codex also caught the mass formula counting the air cavities twice. My dinosaur was being deflated twice for the same lungs.

Building a dinosaur when nobody involved can sculpt

Neither I nor the agents can sculpt. So the pipeline became Python scripts driving headless Blender, where nothing is judged by eye and everything is judged by a number.

Bones first, from published measurements of "Sue". Then 17 muscle groups, each named after the real muscle, each traceable to an anatomy paper. The script prints every muscle's weight, which is how I learned my dinosaur's tail muscle had come out at 471 kg per side against a published estimate of 261. It had been lifting. Tuned down to 252 kg.

Then the skin, with one rule: mesh volume times published tissue density must land between 8.4 and 9.5 tonnes. First attempt, 11.4 to 13.1 tonnes. My scientifically rigorous T. rex was 25 % overweight and its belly hung 7 cm below where the ribs say a belly stops. Two rounds of dieting later it passed at 8.8 to 10.2 tonnes.

All of that worked, and I want to be fair to it. Build, measure, compare against a paper, fix. Dozens of times, overnight, with nobody watching and no human skill involved. Nothing about the science half embarrasses me. I would put those numbers in front of a paleontologist.

The moment I should have seen coming

The walk had a validator: no aerial phase, no foot sliding, no tail dragging, jaw inside its measured gape. It rejected the animation four times and every rejection was a real bug. Right-foot plants generated at the left foot's coordinates, so the poor thing dragged a foot two metres per step. A clamping bug at the clip boundary. Feet planted too close to the hip, shoving the femur past its allowed arc.

Then the validator said PASS. I opened the render, and the dinosaur had stick legs and floating feet. The rig was rotating each leg 90 degrees, so the wide thigh faced the camera edge-on. Every number was perfect. The picture looked like the animal had been assembled by somebody reading the instructions upside down.

Validation catches what you thought to measure. I had measured whether the foot reached the ground. I had not measured whether the leg was on straight.

Here is the finished walk, machine-validated, zero violations, at the exact speed the paper says the animal preferred:

Two full strides at 1.293 m/s. Every number in it is right and it walks like a metronome with a mortgage. MP4 in 1080p.

I showed it to my wife. She asked what it was supposed to be doing.

So why did I get a potato?

I assumed it was my fault: bad prompts, wrong tools, not enough iterations. So the agents ran one more research pass, this time on themselves. The potato was predictable, and the literature is almost rude about how predictable:

The models cannot see properly. "Vision Language Models Are Blind" tested frontier models on things a child does instantly, like whether two circles overlap: 58.57 % average. Probes suggest the vision half knows and the language half cannot read it out loud.
Their taste is not our taste. VisJudge-Bench puts model-to-human agreement on aesthetics at 0.18 to 0.41, and MLLM-as-a-Judge found they are fine at "which of these two is better" and unreliable at grading one thing alone, which is exactly what an agent does to its own render. Mine could tell broken legs from normal legs. It could not tell mediocre from good.
There is a benchmark for agents driving Blender, and it is humbling. BlenderGym scored thirteen vision-language systems 2 to 10 times worse than human Blender users, on tasks the authors call easy for people.
Humans are freakishly good at spotting bad motion. Twelve moving dots are enough for you to recognise a walking person. Worse, uncanny-valley research found degraded motion drops a character below its own still image. My walk cycle is not neutral. It is actively working against me.

Then the structural reason, which is the part I keep chewing on. What made these models good at maths and code is training against verifiable rewards. "Femur = 1.321 m" is verifiable. "Feels alive" is not. Correctness automates first by construction, not by accident.

I hired a correctness engine and handed it a beauty problem. It solved the half it was built for and gave me a vegetable.

The 1996 problem

The T. rex in Tomb Raider, a few hundred polygons and a hand-painted texture, is scarier than my 52,000-quad specimen. Artists in 1996 had no science and no polygons and made a monster. My agents had all the science and all the polygons and made an exhibit.

Which is, if I squint, the actual finding. An artist starting from this repo would skip weeks of paleontology and inherit guardrails against movie-monster anatomy. My agents would get the one judge they lack. Neither half is worthless. They are two different jobs, and I spent three days proving it with a dinosaur.

The repo stays private and non-commercial, because the museum scan I used as a shape reference allows personal use only. Nobody is fighting me for the rights to a beige potato anyway.

What I would tell you

Make the agent's claims falsifiable with numbers from the literature. Hire one hostile reviewer from a different vendor. Look at the renders, because four green validator runs hid a dinosaur with sideways legs. Write down what you do not know.

And expect the last 10 % to be 90 % craft. Correctness automated beautifully. Beauty did not, and now I have citations explaining why. My dinosaur is anatomically defensible, machine-validated, and about as frightening as livestock.

AI Agents Can't Watch Video. Mine Made Three Films Anyway.

Nunc — Fri, 07 Aug 2026 11:33:40 +0000

Martin Krpan is the story every Slovenian schoolchild knows: a salt smuggler meets the emperor's carriage on a snowed-in road too narrow for both, so he calmly lifts his loaded mare and carries her aside. I asked AI agents to turn it into an animated film. The catch: a language model cannot watch video, so it cannot check its own footage. I ran the experiment three times with three different AI setups. All three are live. The interesting result is not the films but what a blind director needs before you can trust its cut.

The idea comes from Andrej Karpathy's LOTR browser movie, which he rated "kind of janky but fun" and blamed on exactly this: LLMs can't perceive video, so they can't audit the worlds they build.

The rules, same for all three

Zero assets. No 3D models, no textures, no image, font or audio files. Geometry from Three.js primitives, sound synthesized in the browser. The one exception: narration, pre-rendered with a free tool; each film works without it.
Deterministic timeline. Every frame is a pure function of film time. No accumulated physics, no Math.random() at play time. Seeking to second 49.5 produces the same pixels every time, in every browser process.
It must be a film. Shots, hard cuts, title cards, subtitles, pacing, an ending. In Slovenian and English.

The determinism rule is the whole trick: if a frame is a pure function of time, a screenshot becomes a measurement. The agent can seek, look at the still (the one visual sense an LLM has), compare pixel hashes between runs, and export video frame by frame. Every verification idea below stands on that.

Film 1: one agent and a screenshot loop

The first run was a single Claude agent building scene 1: the road encounter, 75 seconds, 11 shots. One working day: 7.5 hours of build time, 3,505 lines of TypeScript.

The process rule that shaped it: after building, the agent must watch its own film as stills. A script seeks the timeline, captures screenshots, and the agent critiques them like a director: framing, scale, mesh intersections, poses. It found 24+ defects on its own this way, including the mare's muzzle pointing 43 degrees up (it read as antlers), the emperor's head clipping through the carriage roof, and two shots filming empty road because the carriage hadn't arrived yet.

What it never found: that the film was ugly. The four changes that most improved the result were one-sentence human comments ("the low-poly look is really ugly", "the background sound is annoying", "the narrator is robotic", "Discord says max 15 MB"). Each triggered a real engineering response: an art-direction rebuild with HDR bloom and golden-hour light, a remix measured 60 % quieter, pre-rendered neural voices, a two-pass encode to 13.4 MB.

The split was clean: self-review converged on correct. The human one-liners steered it to good.

▶ krpan.hellonunc.com

Film 2: the full story, and a harness with teeth

The second run scaled up: the whole tale, 10 scenes, exactly 260 seconds, about 14,800 lines. This time three agents split the roles: one planned, one built, and a rival vendor's model (OpenAI Codex) attacked the plan before any code existed. Sixteen findings came back; the best one justified the setup: a dialogue line in the shot list was not in Levstik's text. The model had paraphrased the classic from memory. Codex fetched the pinned Wikisource corpus and proved it. That finding became a validator: every subtitle and narration line must be a verbatim substring of the 1858 text.

Beyond screenshots, this run made the aesthetics machine-checkable: six invariant classes asserted on every scene (palette only from named constants, prop sizes within 10 % of declared, exactly the declared lights, performance ceilings, no object jumping more than 0.5 m outside a cut). And determinism was tested, not assumed: two fresh renders of all ten scenes differed by 0.0000 % of pixels.

That check earned its keep. The sword in the duel was aimed using the giant's hand position from the previous frame, a frame-order dependency invisible to the eye. It surfaced as a 0.2 % pixel difference between playing into a moment and jumping straight to it.

Total cost: about 1.4 million tokens, roughly a cinema ticket, for a bilingual adaptation no studio would fund. The report's own conclusion: the same builder without the harness would have shipped the silhouette shot, the misaimed sword, and the paraphrased Levstik.

▶ krpan2.hellonunc.com

Film 3: a different vendor runs it like a science fair

The third run gave the format to GPT-5.6 Codex, working interactively, and pushed verification the furthest: hypotheses with numeric pass criteria written down before evaluation. 78.000 seconds, 1,872 frames at 24 fps, and every single frame scanned for collisions between bounding boxes of every moving object.

The exhaustive scan embarrassed the still-image review. The stills looked fine while the team horses walked through Krpan for eight seconds and the carriage pole crossed the mare for fifteen. First scan: four sustained collision intervals. Final scan: 0 violations in 1,872 frames. Wheel rotation was derived from distance and radius instead of a multiplier that "looked right"; the final slip error was 1.6 × 10⁻⁷ scene units per frame, and zero frames rolled backward. The MP4 exports were decoded frame by frame: exactly 1,872 H.264 frames in each language edition.

The methodological lesson from this run: stills, continuous playback and an exhaustive numeric scan catch three disjoint classes of bugs. Any one alone lies to you.

▶ krpan3.hellonunc.com (MP4 downloads on the page)

The scoreboard

	Film 1	Film 2	Film 3
Scope	scene 1, 75 s	full story, 260 s	scene 1 retold, 78 s
Built by	one Claude agent	Claude planner + builder, Codex as adversary	GPT-5.6 Codex, interactive
Code	3,505 lines	~14,800 lines	~3,140 lines
Verification signature	screenshot self-review	6 machine-checked invariants, 0.0000 % pixel determinism	all 1,872 frames scanned, numeric pass criteria
Caught what the eye missed	24+ visual defects	invented Levstik quote, frame-order bug	4 collision intervals, backward wheels

The same moment in all three films: Krpan lifting the loaded mare off the road.

What all three runs agree on

"It runs" is a weak finish line. Every version ran fine while hiding collisions, backward wheels, a phantom sword or paraphrased text. The quality came from the harness: determinism, screenshots as measurements, invariants, a rival model reading the plan.

The human contribution is taste, not code. Across all three films I never edited a line. My inputs were the story choice, the constraints, and one-sentence quality judgements. The agents handled everything else, including finding most of their own bugs.

Determinism is the keystone. One property served four masters: frame-exact scrubbing for viewers, screenshot audits for the agent, regression checks for refactors, and clean video export.

Wrapping up

A language model cannot watch video, but it does not need to. If every frame is a pure function of time, screenshots become measurements, invariants become tests, and a blind director can prove its footage is right instead of hoping. The films are the demo. The harness is the product.

EXPLAIN PLAN as a Lint for LLM-Generated SQL

Nunc — Thu, 06 Aug 2026 10:48:57 +0000

My AI agents write Oracle SQL all day: fix scripts, diagnostics, one-off reports for a 2.3M-line legacy system. Their most common failure isn't bad logic. It is SQL that references a table or column that almost exists. Oracle has had the fix for decades, it costs one statement per query, and it never executes anything: EXPLAIN PLAN.

The failure mode: names that almost exist

A language model doesn't know your schema. It knows what schemas usually look like. So on a 20-year-old database with thousands of tables, it produces names that are plausible instead of real: POLICY_STATUS when the column is STATUS_CD, CUSTOMERS when the table has been CUSTOMER (singular) since 1998, a join through a link table that was dropped two versions ago.

These are the worst kind of errors, because they look right. A human reviewer skims the script, the naming convention matches, everything reads fine. The error only surfaces when the script runs, and in my case fix scripts run at more than 20 customer installations. That is exactly the place where you do not want to discover an invented column.

Why you cannot just run it to check

The obvious test, execute it and see, is not available. These scripts are UPDATEs and DELETEs against production-like data. Wrapping everything in a transaction and rolling back sort of works, but it fires triggers, takes locks, burns sequence numbers and takes time on big tables.

I wanted a check that touches nothing and still uses the real schema. It already exists.

EXPLAIN PLAN parses without executing

EXPLAIN PLAN FOR
UPDATE policy
   SET status_cd = 'ACTIVE'
 WHERE policy_id = :b1;

Oracle takes the statement through the full parse: it resolves every table and column against the live data dictionary, checks your privileges, builds an execution plan and writes it to PLAN_TABLE. What it never does is execute. No rows change, no triggers fire, no locks are held. It is safe to run for an UPDATE, a DELETE, a MERGE.

If the model invented a table, you get an answer in milliseconds:

ORA-00942: table or view does not exist

If it invented a column:

ORA-00904: "POLICY_STATUS": invalid identifier

That is a lint result, produced by the one parser that actually knows the schema: the database's own.

Wiring it into the agent loop

In my workflow every generated script goes through this gate before a human sees it, statement by statement, against the dev database:

for stmt in split_statements(script):
    try:
        cursor.execute(f"EXPLAIN PLAN FOR {stmt}")
    except oracledb.DatabaseError as e:
        errors.append((stmt, str(e)))

The errors go straight back to the model with the original task. Most name-level failures disappear in one retry, without me reading anything. What reaches me is a script whose every identifier is real.

The tool the agents call runs in a read-only safe mode: SELECT, DESCRIBE and EXPLAIN PLAN are allowed, everything else (DDL, DML, GRANT, COMMIT) is blocked before it reaches the database. So even a badly confused agent can't turn the lint step into a write.

The free bonus: you also get the plan

The check produces an execution plan as a side effect:

SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY);

So the same gate that catches invented columns also shows you the full table scan on a 41-million-row table before anything runs. Two problems, one statement.

What it does not catch

A green lint is not a correct script. Here is what this gate misses:

Valid names, wrong logic. DELETE FROM policy WHERE status_cd = 'A' parses perfectly and can still delete the wrong rows. This gate replaces nothing at the review level.
PL/SQL blocks. EXPLAIN PLAN takes single SQL statements. For packages and procedures I compile against a scratch schema instead; that is a different gate.
Data assumptions. The parser checks that a column exists, not that 'A' is a value that ever appears in it.
Schema drift. I parse against the dev schema. A customer installation two versions behind can still disagree. Closest schema wins, not a guarantee.

The same idea works outside Oracle: PostgreSQL parses and plans with PREPARE or plain EXPLAIN, SQL Server has SET PARSEONLY ON. Any database that can plan a statement without running it can lint one.

Wrapping up

LLM SQL fails most often at the name level, and name resolution is exactly what the database parser already does. EXPLAIN PLAN turns that parser into a lint step: every generated statement checked against the live schema in milliseconds, with nothing executed and an error message the model can act on. Of all the guardrails around my agents, this one has the best ratio of effort to failures caught. It was one afternoon of work.

I Review My AI Assistant's Logs Every Week, With an Agent. Here's the Process.

Nunc — Mon, 03 Aug 2026 13:43:58 +0000

We run an internal AI assistant on top of our 20-year-old helpdesk system. Production since April, 15 active users across five roles, from developers to operators to project managers. The model matters less than people think. What actually improves the assistant week over week is a boring process: a structured weekly review of its logs. This post describes that process, the metrics we track, and two real failures the review caught that I would never have found otherwise.

What gets logged

Two levels. A summary log with one line per request:

[2026-07-20 16:46:52] user=xxx timing=99926ms status=ok
  model=... turns=13 tools=12 cost=$0.25
  profile=xxx ftok=1820ms query="have we ever solved..."

And a full JSONL transcript per session: every user turn, every answer, plus which context was injected and where it came from. The provenance part was added later, after a review showed we couldn't tell why the assistant said something. If you're building an assistant, log answer provenance from day one.

The weekly review is run by an agent

Reading a week of transcripts by hand is not sustainable, so the review itself is an agent task. I say "do the review", and the agent figures out which period is not yet covered, pulls the logs from the server, and works through every session. Not a sample. All of them.

The review has three outputs:

Session quality review. The agent grades answers, and the important part: it re-verifies factual claims against the actual database. If the assistant told a user "this fix was installed at customer X in May", the review checks whether that's true.
Improvement proposals. Concrete and prioritized, each with the problem, evidence (session IDs, quotes), a proposed fix, and an effort estimate. Each proposal enters a backlog with a status: proposed, approved, implemented, rejected, or watch.
User profiles. Usage patterns per user feed personalization (more below).

The next review then verifies the previous round: did the implemented fixes actually stop the failure they targeted? Several times the answer was "partially", and the item went back into the backlog. Without that verification step, an improvement backlog turns into a feel-good list.

The scorecard

Every review adds one row to a long-running scorecard. Same metrics, same methodology, so trends are visible: requests, users, sessions, error rate, median and p95 response time, corrections, factual errors, false negatives, security events, user feedback, average session grade.

A recent week looked like this: 137 requests, 12 users, 44 sessions, 0 errors, median response 99 seconds, p95 262 seconds, average session grade 4.5 out of 5 at 100% review coverage.

Two practical lessons from maintaining it. First, write down the counting methodology, because "how many sessions" turns out to have edge cases (we now count a session as a transcript file with at least one real user turn, feedback-only files excluded). Second, mark methodology changes in the scorecard itself. Otherwise a metric jump reads as regression when it's actually deeper measurement.

Two failures the review caught

The leaking limit banner. One afternoon our primary auth token kept hitting a rate limit, and a fallback took over. The failover worked, but the review found that at least 12 answers shown to 4 users had an English "You've hit your limit" banner and fragments of the failed first attempt glued to the top of the final answer. Users saw it and said nothing. Nobody reports weirdness in a tool they don't fully trust yet, which is exactly why you read the logs.

The adopted false premise. A user asked about a project, and their question contained a wrong assumption about which customer group a code referred to. The assistant took the premise at face value and confidently attributed everything to the wrong company. The transcript grade was fine (the user was happy!), but the facts were wrong. The fix was a prompt rule: verify the entity behind a code against the database before building an answer on it, even when the user asserts it. You will not catch this class of failure from user feedback, because the user is the source of the error.

Two kinds of user profiles

Each user has a profile the assistant loads into its system prompt: role, which projects they may see, typical questions, preferred answer depth. A developer asking about a package gets code references. A project manager asking the same thing gets a business summary. An operator asking about an error code gets the past-ticket solution plus a suggestion for which developer to assign.

The part that took iteration: we keep two separate profile sets. Analytical profiles are our internal understanding of each user, built from the log reviews. Agent profiles are the distilled version the assistant actually loads. The analytical profile is the raw material, the agent profile is the product. Mixing them was a mistake we made first: internal observations about a user do not belong in a prompt.

One rule I'd flag for anyone doing this: personalization must stay a default, not a cage. A non-technical user who asks a technical question gets the technical answer. And log review being about improving the assistant, not evaluating employees, is something you say out loud and write down before someone discovers their conversations are being read.

Wrapping up

An AI assistant in production is a product, and it needs the product loop: instrumentation, periodic review, prioritized backlog, verification that fixes worked. Doing the review with an agent makes 100% coverage affordable. And the highest-value finds are the ones no user will ever report: answers that were wrong while everyone was satisfied.

108 Support Tickets Later: My AI Agent Workflow on a 2.3M-Line Oracle Legacy System

Nunc — Thu, 30 Jul 2026 12:14:10 +0000

My day job is maintaining a 20-year-old insurance system: over 1,000 PL/SQL packages, 2.3 million lines of code, Oracle Forms on top, and the same codebase running at more than 20 customer installations. Nine months ago I started routing my support tickets through AI agents in Claude Code. 108 tickets later, almost nothing I do is manual anymore. This post describes the workflow, the parts that keep it safe, and the loop that makes it better with every ticket.

Why this works on a legacy system

People assume AI tooling helps most on greenfield projects. My experience is the opposite. The pain in a legacy system is not writing code, it's finding context:

Nobody knows the whole codebase. At 2.3 million lines, nobody can.
The same code behaves differently per customer, controlled by hundreds of installation parameters. Half of my questions are "why does this work differently at customer X".
Our helpdesk has about a million tickets from 20+ years. The answer to "have we solved this before?" is almost always yes. The problem is that it lives in a ticket from 2017, or in one paragraph of a manual nobody opened in three years.
Knowledge is spread across five systems: the helpdesk, user manuals on a file share, a wiki, GitLab, and SVN.

Collecting that context by hand takes a senior developer half a day. An agent with the right tools does it in minutes. That's the whole trade.

The workflow: one command per ticket

When a new ticket arrives, I run a single analyze command. The agent then:

Reads the ticket through the helpdesk's SOAP API, including attachments. PDFs, Word and Excel files get converted to text first.
Searches for similar past tickets. Keyword search plus vector search, because the built-in helpdesk search only does keywords and misses anything phrased differently.
Checks my library of ready-made SQL scripts for anything reusable.
Looks at the relevant code in the database, and at its change history in GitLab or SVN if needed.

The output is a single markdown file, ANALIZA.md: all findings in one place, plus a proposed next step. The next step is always one of three things: a code fix, a reply to the customer, or more diagnostics on the customer's database.

Once there is enough data, the agent prepares the whole package: the proposed solution, a message for the customer, internal notes for colleagues involved in the ticket, the diagnostic and fix scripts, and a short write-up of why we did it this way. All of it gets attached to the ticket.

Skills are where the actual work went

The agent is only as good as the context you can hand it. Over nine months I built 24 skills. Grouped by function:

Group	Examples
Context access	helpdesk API, vector search over tickets and 1,800 manuals, Oracle dev DB, GitLab fix archive (31,000+ commits), SVN for Forms, internal wiki
Domain knowledge	system architecture and its 13 business domains, insurance accounting mechanics, installation parameters, internal PL/SQL standards, our logging package
Writing and delivery	generating PL/SQL to our standards, packaging a fix, installing it on the dev database, recompiling invalid objects, compiling Forms
Meta	a skill that builds new skills from repeating patterns

The meta one matters more than it sounds. Whenever I catch myself explaining the same thing to the agent twice, that explanation becomes a skill. The toolset grows as a side effect of normal work.

Two rules that keep it safe

Every generated script gets validated against the live schema. The most common LLM failure in SQL is inventing a table or column that almost exists. The fix is cheap: run EXPLAIN PLAN FOR <statement> against the dev database before the script leaves my hands. It parses the statement against the real data dictionary without executing anything, so it's safe even for UPDATE and DELETE. Invented objects fail immediately with ORA-00942 or ORA-00904.

Risky changes get a second engineer. For hard-to-reverse changes I hand the same task to a second, independent agent and compare findings. I use two: a different vendor's model for a genuinely independent view (different model, different blind spots), and a fresh session of the same model loaded with all our domain skills for domain-heavy reviews. What both find is almost certainly real. What only one finds, I check myself. On a small team, this is the closest thing to peer review I can get.

The loop that pays for everything

Here is the part I'd steal if I were reading this post.

Everything the agent produces gets attached back to the ticket: scripts, analysis, the reasoning behind the fix. When a similar ticket arrives next year, the vector search finds that old ticket, and the agent reads not just what we did, but why. Then a separate closing step extracts anything reusable into a script library. That library is now at 759 SQL templates in 32 topic folders, built up over years and finally searchable by an agent instead of just by me.

The result: every solved ticket makes the next one cheaper. No extra effort, no "knowledge management initiative". It happens as a byproduct of closing the ticket.

What still doesn't work

Honesty section. Three things are open:

Testing is still manual. The agent speeds up writing a fix, not proving that the fix breaks nothing at the other 20 installations. That's still the biggest remaining risk.
Metrics are thin. I have raw numbers (108 tickets, roughly 4,000 prompts, 94% of them on tickets) but no systematic before/after measurement of resolution time. I know it's faster. I can't prove by how much yet.
Bus factor of one. The whole stack is built and configured on my machine. A colleague can't pick it up without a setup guide that doesn't exist yet.

Wrapping up

The key insight: on a large legacy system, the win from AI agents is not code generation. It's that context gathering, the half-day of digging through five systems before you write a single line, collapses to minutes. And if you route the results back into the ticket archive, the system compounds: every solved ticket becomes training material for the next one.

Should I Prompt Claude in English? I Analyzed 2,300 of My Own Prompts to Find Out

Nunc — Thu, 23 Jul 2026 14:35:47 +0000

I'm a Slovenian developer and I talk to Claude Code in Slovenian. About 2.5 million people speak my language. Every prompt engineering guide is written in English, and the usual advice is: prompt in English, the models are simply better at it. I wanted to know if that's actually true for daily coding work, so I measured it. Here's what I found.

How I tested it

Three parts:

My real usage. I parsed all my Claude Code session transcripts from two environments (WSL for personal projects, Windows PowerShell for my day job): 2,319 prompts by my first count, 1,899 after the strictest filtering pass that drops pasted logs, tool output and machine-generated turns.
Published research. Multilingual benchmarks, interpretability studies, tokenizer papers, official vendor docs.
My own token measurements. Real prompts of mine, translated to English, counted with different tokenizers.

Then one more thing: I gave the exact same research task to OpenAI Codex (GPT-5.6), isolated from my results, and compared what the two systems concluded. More on that below.

What my usage actually looks like

	WSL	Windows
Real prompts (strict filter)	1,362	537
Slovenian, where language is determinable	98.7%	99.4%
Median prompt length	54 chars	90 chars

So I don't "mostly" prompt in Slovenian. I prompt in Slovenian, full stop. Fast, with typos, often without diacritics (š, č, ž), with English technical terms mixed in wherever they're natural. The question is whether any of that hurts.

What the research says about quality

Short version: it hurts less than I expected, if your model is big enough.

Models internally lean on English anyway. Interpretability work from EPFL (Do Llamas Work in English?) and Anthropic's own circuit-tracing study (On the Biology of a Large Language Model) points the same way: large models process meaning in a shared concept space that sits closest to English. Non-English input adds a thin translation layer at the edges. The thinking in the middle is mostly language-neutral. This also means instructions like "think in English, respond in Slovenian" are mostly wasted words on modern models.

The gap was real, and it has mostly closed at the frontier. In 2022, PaLM solved 62% of math word problems in English but only 47% averaged across underrepresented languages (MGSM). Today, Anthropic's official numbers for benchmarked languages show Spanish at 98.2% of English performance and German at 97.0% (multilingual support docs). No vendor benchmarks Slovenian, but a Slovenian pragmatics benchmark (SloPragEval) measured GPT-5 at 0.81 in Slovenian vs 0.83 in English. Two points. Not nothing, but not a reason to switch languages for daily work.

Small models are a different story. The same Anthropic table shows Claude Opus 4.1 keeping 80% of its English performance in Yoruba, while the much smaller Haiku 4.5 keeps only 53%. If a small cheap model handles your non-English workload, test it in your language before trusting it.

Mixing languages is fine. In my direction, it even helps. A 2025 study (Lost in the Mix) found an asymmetry: foreign words inserted into English text hurt comprehension, but English words inserted into non-English text often improve it, by up to 13 points on some models. Slovenian sentences with English technical terms left as-is turn out to be the good kind of mixing.

What Slovenian costs in tokens

The literature says non-English text is expensive: Slovenian pays about 1.88x the tokens of English on GPT-4's old cl100k tokenizer (Petrov et al., NeurIPS 2023). That number worried me. Then I measured my own prompts: 30 real ones, each translated to English faithfully.

Tokenizer	Slovenian premium on my real prompts
cl100k_base (GPT-4 era)	+34%
o200k_base (GPT-4o / GPT-5)	+16%

Why so much lower than the paper's numbers? Because real developer prompts aren't clean prose. They're full of file paths, SQL, error messages and English tech terms that tokenize identically in both languages.

Two results I didn't expect:

Diacritics are free. The same text with and without š/č/ž differs by 0.6% in tokens. My lazy typing habit costs nothing.
The cost that matters is on the output side. Output tokens are about 5x the price of input tokens and they dominate latency. A short Slovenian question costs nothing worth mentioning. A long Slovenian document generated by the model costs 16-34% more time and money than the same document in English.

One honest caveat: Claude's tokenizer is not public, so these are OpenAI tokenizer numbers (plus Gemini's count API, which agreed). Published cross-vendor comparisons suggest Claude's non-English premium is somewhat higher.

The twist: I had Codex repeat the whole study

To check my own work, I gave the identical research brief to OpenAI Codex (GPT-5.6): same task wording, same environment facts, an isolated working directory, and no access to my report. It even had to exclude the transcript of the session that ordered the research, so it couldn't peek at my conclusions.

It came back with the same core answer: keep prompting in Slovenian, keep technical identifiers verbatim, write reusable instructions in English with native trigger phrases, don't translate domain knowledge. Two systems independently converging on the same conclusions is worth more than either report alone.

And Codex found one thing I had missed. It manually reviewed 50 of my prompts and rated 90% of them as clearly understandable despite all the typos. The weak ones weren't weak because they were Slovenian. They were weak because they were vague: "fix this", "now do it properly", no success criteria, no constraints. Its conclusion, which I'm keeping: the biggest improvement isn't the language, it's explicitness.

What I changed, and what I kept

Kept:

Slovenian for interactive work. I think faster and state intent more precisely in my own language.
English technical terms inside Slovenian sentences. The research says this direction of mixing is harmless to helpful.
Everything technical verbatim. Package names, SQL, paths, error messages, config keys. Never translated, never paraphrased.
Hybrid skills. Reusable agent instructions in English, trigger phrases in the description in Slovenian (the words I actually type), business domain knowledge in Slovenian because the terminology lives in Slovenian.

Changed:

An explicit language rule in my global config, because Claude Code is known to drift back to English after context compaction:

   Always respond in Slovenian unless explicitly asked otherwise.
   Keep code, identifiers, commands, paths and error messages
   in their original form.

A five-field template for bigger tasks: goal, context, constraints, verification, output format. This fixes more than any language switch would.
English selectively: long structured specs and agent briefs, or an A/B retry when a hard reasoning task gives a shaky answer.

Wrapping up

For frontier models, the language you prompt in matters far less than how explicit your prompt is. Prompt in the language you think fastest in, keep everything technical verbatim, write reusable instructions in English with trigger phrases in your own language, and spend the effort you save on clear goals and constraints instead of translation.

I Made Claude Code and Codex Argue About My Code Until They Agreed

Nunc — Wed, 22 Jul 2026 10:48:22 +0000

TL;DR: I wired OpenAI's Codex CLI into Claude Code as an adversarial reviewer with a convergence loop. Then I pointed the loop at its own implementation. It failed the review three times before passing, and every single failure was a real bug. Here's the whole story, with the prompts and shell recipes you can steal.

The tweet that started it

A while ago Siqi Chen posted his favorite way of working with frontier models:

step 1. ask it to write a plan
step 2: "please get second opinions from codex CLI using gpt-5.6-sol @ max effort and kimi CLI using kimi 3. Revise your plan with any sound findings. repeat until convergence or up to 5 rounds."

There's also a companion repo, adversarial-execution, which applies the same idea at the other end: before you mark work as done, fresh reviewer sessions have to pass it with evidence, in a loop, until a fresh round finds nothing new.

Two ideas jumped out at me:

Fresh sessions, not continuations. A reviewer that already critiqued round 1 will defend its own findings in round 2. Anchoring is real for LLMs too.
"Looks right in the code is not evidence." The reviewer judges proof (test output, screenshots, command results), not vibes.

I already had a Claude Code skill that wraps the local codex CLI as a second engineer: consult it read-only, delegate a fix, run a git-aware review, resume a session. All useful. All one-shot. No loop anywhere.

So I asked Claude Code to study the tweet and the repo and tell me whether the skill was worth upgrading.

Round zero: Codex reviews the plan for using Codex better

Claude wrote a proposal: two new modes (a plan-review convergence loop and a done-gate), plus supporting rules. Decent plan. Then came the fun part. Before touching anything, I had Claude send the proposal itself to Codex for an adversarial review, using the exact contract style the proposal was recommending.

Verdict: FAIL. Eight gaps, seven of them major.

And they weren't nitpicks. My three favorites:

The output capture in the existing skill was broken. The recipes piped codex exec output through tail -40 and told Claude to grab the session id "from the header". But the header is at the start of stdout, and tail shows the end. On any long run the session id is simply gone. Better yet: this exact failure happened during the review run itself. The reviewer described a bug and the run reproduced it live. The pipeline exit code was wrong too, because without PIPESTATUS you get tail's exit code, not codex's.

False convergence. My loop definition said: converged when a round brings "no new sound findings, only repeats". Codex pointed out that a finding I accepted but fixed incompletely would come back as a "repeat" and get waved through. A repeated unresolved finding has to stay open. And hitting the round cap needed an explicit NOT CONVERGED outcome instead of quietly presenting the last plan as final.

Windows command lines have a length limit. Passing a plan, a diff, and the round history as one quoted shell argument blows past ~32k characters. Codex suggested the CLI's stdin form (codex exec ... - < prompt.txt), which my own skill was accidentally blocking with an unconditional </dev/null.

I triaged all eight findings. All eight were sound. That triage step matters, by the way. The instruction is "revise with any sound findings", not "do whatever the reviewer says". The orchestrating model stays the decision maker; the reviewer is a hostile witness, not a boss.

Simplify before you ship

The revised proposal had grown formal machinery: a findings ledger with four statuses, evidence manifests with exit codes and timestamps. My actual usage is a few Codex calls per month. That's how you end up with a skill nobody follows.

So I shipped a lean version instead. The concepts survived, compressed to one line each. A skill is instructions to a model, not code. The model can track a simple ledger in its head; what it needs written down are the rules it would otherwise get wrong:

Re-reviews happen in fresh sessions, never resume.
Adopt only sound findings, record rejections with a reason.
An adopted-but-unfixed finding stays open.
Cap the loop (3 rounds for plans, 2 for the done-gate), and report NOT CONVERGED honestly if you hit it.
A failed or empty run never counts as a round, and never as a PASS.

The done-gate eats its own dog food

New rules in place, one question left: is the upgrade itself actually done? Perfect job for the brand-new done-gate. Fresh Codex session, max reasoning effort, evidence pack attached (diffs, validator output), and a contract that forces a machine-checkable verdict:

INTENT: PASS | FAIL - does the work satisfy the stated acceptance criteria? (evidence)
WORKS: PASS | FAIL - does it actually function, per evidence, not per code reading? (evidence)
PLAN_FIT: PASS | FAIL - does it advance the overall plan without drift? (evidence)
VERDICT: PASS only if all three are PASS, else FAIL.
GAPS: numbered with stable IDs; severity, problem, evidence, smallest fix. If none: GAPS: NONE.
IMPROVEMENTS: optional non-blocking suggestions (max 5). These never affect the verdict.

Round 1: FAIL. Four gaps. The best one: I had fixed the output handling in modes 1 and 2 but not in the review and resume recipes, and my shiny new "pass long prompts via stdin" rule used a $PROMPT_FILE variable that no example ever created. Copy-paste the doc as written and you'd get "No such file or directory".

Round 2: FAIL. Two gaps. One was a claim I refused to take on faith: Codex said the skill's original focused-review example, codex exec review --uncommitted "focus on error handling", is an invalid invocation, because the review target flags conflict with a positional prompt. That example predated this whole upgrade. One free local test later (argument parsing fails before any model call, so it costs nothing):

$ codex exec review --uncommitted "focus on error handling" -o /dev/null
error: the argument '--uncommitted' cannot be used with '[PROMPT]'

Confirmed. A broken example had been sitting in the skill since day one, and a fresh pair of (artificial) eyes caught it on pass two.

Round 3: PASS. All three dimensions, GAPS: NONE, nothing reopened. Convergence, for real this time.

Three rounds, three sets of genuine defects, then nothing left to find. That's exactly the behavior the pattern promises, and I got to watch it happen to my own work.

The pieces you can steal

The reliable shell wrapper for non-interactive codex runs (Git Bash, works on Windows too):

OUT=$(mktemp codex-out-XXXX.txt)
LOG="${OUT%.txt}.log"
codex exec -s read-only --skip-git-repo-check \
  -c model_reasoning_effort="xhigh" \
  -o "$OUT" - < prompt.txt 2>&1 | tee "$LOG" | tail -20
RC=${PIPESTATUS[0]}; echo "rc=$RC"
grep -m1 -i "session id" "$LOG"
cat "$OUT"

Every piece earned its place the hard way: -o captures the clean final message, tee keeps the full log so the session id survives, PIPESTATUS[0] gets the real exit code, and the trailing - < prompt.txt feeds a prompt of any size through stdin.

The plan-review contract (the done-gate variant is above):

You are an adversarial reviewer of a PLAN (no code changes exist yet).
Judge the plan against the stated task and acceptance criteria AND against
the actual repo state (read the code the plan touches).
Review independently first; only then reconcile with the round history (if
provided), classifying each finding as NEW, REOPENED, or DUPLICATE.
Return exactly:
VERDICT: PASS | FAIL
GAPS: numbered with stable IDs; severity, problem, evidence, smallest fix.
If none: GAPS: NONE.
IMPROVEMENTS: optional non-blocking suggestions (max 5). These never affect
the verdict.
Do not restate the plan. Do not raise style nits as gaps.

And because nobody wants to memorize six modes, I added a /codex slash command that acts as a dispatcher. I describe what I want in plain language; Claude walks a decision tree (continuation? edits? plan? finished work? whole diff? everything else) and picks the mode, assembles the context Codex can't see (the exact error, what was already ruled out, the diff, the evidence pack), and composes the prompt. The "what was already tried and ruled out" part is the single highest-value thing you can put in a delegate prompt. It stops the second model from re-walking your dead ends.

What I learned

Fresh sessions beat follow-ups for re-review. A resumed reviewer defends its previous verdict. A fresh one attacks the current state.
Make verdicts machine-checkable. "Looks good with some concerns" can't drive a loop. VERDICT: FAIL + numbered gaps can.
Triage, don't obey. Both models are wrong sometimes. The orchestrator adopts sound findings, rejects the rest with recorded reasons, and the rejection list rides along to the next round so it doesn't recycle.
Verify reviewer claims cheaply before believing them. The arg-parse test cost nothing and turned a claim into a fact.
Cap the loop, admit the cap. Unbounded loops sound principled and burn quota. NOT CONVERGED is a perfectly good answer.
The reviewer finds different bugs than the author. Not smarter, different. Every round proved it.

The whole exercise took an afternoon and four Codex runs on a regular ChatGPT subscription. The skill now catches its own mistakes before I ship them, which is a sentence I didn't expect to write this year.

Have you tried pitting two coding agents against each other? What broke first? Share in the comments!

"My AI engineer has a second AI that tells it NO. Three times, it was right."

Nunc — Fri, 17 Jul 2026 14:56:17 +0000

Yesterday afternoon I was about to ship a change to a production pipeline. The code was written, tested, deployed to the runtime directory. The nightly cron would pick it up at 22:00.

Then a second AI looked at the diff and said NO-GO.

Not a human. Not the AI that wrote the code. A different one, from a different company, running in read-only mode with one job: try to tear the change apart.

It said NO-GO three times in a row. Each time with a concrete, reproducible bug. On the fourth round it said GO, and the cron ran clean that night.

This post is about that setup: using OpenAI's Codex CLI as a second engineer inside Claude Code sessions. What it looks like, what it actually caught, and when it's worth the quota.

The problem with one very smart agent

I do most of my engineering with Claude Code these days. It writes the code, runs the tests, deploys, checks the logs. For context: I wrote about the AI assistant we built on top of 20 years of support tickets in a previous post. That assistant is still running. This post is about the tooling around building things like it.

Here's the uncomfortable part. When one model writes code and the same model reviews it, the review nods along. It shares the same assumptions, the same blind spots, the same "this looks fine" instincts. It wrote the bug, so it doesn't see the bug.

Human teams solved this ages ago: you don't review your own PR. So I gave my AI the same rule.

The setup: two CLIs, one boss

The pieces:

Claude Code is the primary engineer. It has full tool access in my repo.
Codex CLI (OpenAI's terminal agent) is installed and authenticated on the same machine. Different model family, different training, different instincts.
Claude can call Codex in two ways: an inline skill for quick consultations, and a subagent that runs Codex in the background on a well-scoped task while Claude keeps working.

The important part is the policy. I don't manually decide when to ask Codex. Claude decides, based on rules I put in my global config:

## Codex as a co-engineer

A local OpenAI Codex CLI is installed and authenticated. Claude may use it
on its own judgment as an independent second engineer. Good reasons:
- stuck after 2+ failed attempts at a bug
- independent review of a risky or security-sensitive change
  before calling it done
- cross-checking a hard-to-reverse design decision
- a parallel investigation while Claude keeps working

Announce in one sentence when doing so (it spends Codex quota).
Read-only runs don't need permission; write-capable delegation does.
Keep self-initiated runs to ~2 per task.

That's the whole contract. Codex runs read-only by default (it can read the repo, it cannot touch it). Claude announces every call because it burns my OpenAI quota. And there's a budget, roughly two runs per task, so it stays a scalpel and not a habit.

What we were building when it earned its keep

The project: an automated code-review pipeline for Oracle PL/SQL patches in a large, old back-office system. Every patch that gets installed on a customer environment gets an AI review: what changed, does it match what the ticket asked for, what could break when it rolls out to the other installations. Findings get scored, risky ones trigger an email to the patch author.

The scoring part is where it gets dangerous. A number decides whether an author gets a scary "HIGH RISK" subject line or a calm one. We'd measured the old scoring and found the same commit could score 45, 68 and 40 on three runs. That's not a metric, that's a mood. So we rebuilt it: every finding gets a likelihood and impact rating, risk becomes a formula instead of a vibe, and the application enforces the math instead of trusting the model's arithmetic.

Claude designed and implemented all of that. Then, per the policy, it handed the diff to Codex with an explicit instruction: this ships to production tonight, try to refute it.

Round 1: NO-GO, six fixes

Codex came back with a structured verdict. Not "looks good with minor suggestions". A NO-GO with six concrete findings, each with file and line number. The two that hurt:

The thread truncation bug. Ticket threads get trimmed to fit the context window. The old code did the equivalent of substr(0, 200000). Sounds harmless until you realize helpdesk threads grow chronologically, so a naive head-cut keeps the oldest messages and silently drops the newest ones. The newest messages are where the actual decisions live. Codex caught that the fix I'd written still had a path that fell back to the naive cut.

The fudge factor. My scoring prompt said risk equals the maximum of the per-finding scores, "±5 for rollout context". Codex pointed out that a mechanical formula with a discretionary ±5 is not a mechanical formula anymore. It's the old vibes with extra steps. We deleted it.

Fair. Fixed all six. Sent the diff back.

Round 2: NO-GO, and this one was scary

This is the finding that sold me on the whole approach.

The new enforcement code parses the model's findings (each one annotated like [V=4 I=5]), computes the risk, and overwrites the model's number if it doesn't match. Codex looked at my parser and asked: what happens when the parser only sees some of the findings?

Then it answered its own question. It took my actual regexes, constructed a review where one finding is written as K1 [V=2 I=2] and another as K2: [V=5 I=5] (note the colon, a format my parser missed), and showed the parse returns only the first one. Result: the app "corrects" a risk of 100 down to 16. A critical finding, silently erased by my own safety mechanism.

The fix was to make the parser fail-closed: it now tracks every finding ID it can see anywhere in the text, and if even one of them can't be fully parsed, the enforcement steps aside and the original score stands. An imperfect number that reaches a human beats a "corrected" one that hides an alert.

Round 3 found one more variant of the same hole (two findings on the same physical line). Round 4: GO.

Three real bugs in my safety net, found before production instead of three weeks into it. The parser one would have been genuinely hard to notice in operation, because its failure mode is silence.

Why a different model family matters

You could run this loop with the same model reviewing itself, and it would still catch things. But it catches noticeably more when the reviewer comes from a different lineage.

My theory: models from one family share failure modes the same way code from one author shares bugs. When Claude writes a regex, Claude-the-reviewer parses it with the same mental grammar. Codex parses it with a different one, which is exactly how it found the K2: case. The disagreement is the feature.

We liked this effect enough to build it into the product itself. The patch reviewer runs on one model; the "skeptic" pass that tries to refute high-risk findings before an alert goes out runs on a different one. Decorrelation all the way down.

The five rules that make it actually work

Plugging two AI CLIs together is easy. Getting reviews that are worth the quota took some iteration. What stuck:

Read-only by default. Codex analyzes, Claude implements. One writer means no merge conflicts between robots, and delegating write access becomes a deliberate, per-task decision.
Demand evidence, not opinions. Every delegation prompt asks for findings ranked by impact, each with file:line and either a reproduction or an explicit "plausible, unverified" label. Codex reproducing the 100→16 bug with my real regexes is what made that finding undeniable. A vague "parsing might be fragile" would have been ignored.
Forbid anchoring. When I want a fresh assessment of a system, the prompt explicitly says: do not read our existing TODO and improvement docs. Otherwise the review converges on what we already believe. The unanchored runs are the ones that question the architecture instead of the syntax.
Feed it production data, not just code. The best review round included a week of anonymized log lines next to the source. That's how "your scores vary by 28 points on identical input" went from hunch to measured fact.
Keep the loop in one session. Codex CLI supports resuming a session, so the NO-GO → fix → re-verify cycle keeps its context. Round 3 knew what round 1 had already flagged. Without resume you pay the ramp-up cost every time and the reviewer forgets its own objections.

What it costs, honestly

This isn't free, in three currencies.

Quota. Every run spends my OpenAI subscription. The ~2-runs-per-task budget plus the "announce it" rule keeps this visible. Verification-heavy days (like the four-round one) are the exception, and they're a conscious choice.

Time. A thorough Codex review of a non-trivial diff takes around ten minutes. Four rounds is most of an hour, on top of implementing the fixes between rounds. For a config tweak that would be absurd. For code that emails risk scores to my colleagues, it was cheap.

Judgment. Codex is not an oracle. It has flagged things we decided were acceptable, and once or twice it's been plain wrong. Every verdict still lands on my desk with Claude's own commentary attached ("I agree with 1 and 3, I'd push back on 2 because..."). Two AIs disagreeing is information. Me not reading either of them would be negligence.

Takeaways if you want to try this

Write the policy into your agent's config, not into your habits. The whole point is that the agent knows when to ask for a second opinion, including times you wouldn't have bothered.
Default the second AI to read-only. You want a reviewer, not a second pair of hands in the same file.
Ask for refutation, not validation. "Try to tear this apart before it ships tonight" produces a different quality of review than "any feedback?"
Require file:line and reproductions. It converts the review from prose into a worklist.
Use the disagreements. When both models agree, fine, ship. When they don't, that's the exact spot where a human should look.
Budget it. Second opinions are a scalpel. If every task triggers three reviews, the signal drowns.

The strange part is how normal it feels after a week. Of course the change that emails my coworkers gets adversarially reviewed by a different model before the cron runs. Of course NO-GO means we fix it first. It's just code review. The reviewers happen to have different parents.

Anyone else running multi-vendor AI review loops? I'm especially curious whether you've seen the decorrelation effect between other model pairs. What did one catch that the other waved through?

I put an AI agent in front of 20 years of support tickets. Here's what actually broke.

Nunc — Sun, 12 Jul 2026 14:53:03 +0000

We have a helpdesk system that's been running for more than twenty years. About a million tickets in the database. Every one of them has messages, attached fixes, billed hours. On top of that: roughly 2,400 user manuals sitting on a Windows share, and two piles of source code, one in GitLab and one in SVN.

So when a developer asks "have we ever solved this before?", the answer is almost always yes. It just lives somewhere. Maybe a ticket from 2017. Maybe a comment on a fix. Maybe one paragraph in a manual nobody has opened in three years.

Even senior people lose hours to this. Juniors lose weeks.

A few months ago I built an internal AI agent to sit in front of all of it. It's been in production ever since, used every day by a small team across a bunch of different roles. This post is the honest version of how it went. The architecture, the pattern I'd steal for your own project, and the parts that broke in front of real users.

Let me be clear up front about the punchline, because it surprised me: the data was never the hard part. Knowing where to look was. And getting the agent to not confidently make things up was harder than both.

What it actually is

One agent, natural language in, sitting in front of every system the team touches. All read-only.

You ask it the way you'd ask a colleague over coffee. No query syntax, no menus. Under the hood it can reach:

The helpdesk database (tickets, messages, fixes, hours)
Our project management tool, for "what should I pick up next?" questions
GitLab and SVN, for source-code history
A vector search over those 2,400 manuals
A vector search over the million historical tickets
A document generator that hands back an .xlsx, .docx, or markdown file as a download

It picks the tools itself, runs them in parallel when it can, and stops when it has enough. That's it. That's the whole product from the user's side.

The interesting stuff is underneath.

The architecture, and one boring decision that paid off

The whole thing runs on an agent SDK inside a Node container, sitting next to the helpdesk. The frontend is plain vanilla JavaScript. No framework. Cheap to maintain, trivial to extend.

Requests flow through the existing app's session auth, so there's no separate login. Reuse the session, done. Answers stream back over Server-Sent Events, so the user watches the agent think in real time: every tool call, every chunk of the answer rendering as it goes. People trust it more when they can see it work.

The one decision I want to call out, because I almost didn't do it: I ran the agent in its own container instead of inside the main app.

The SDK is Node-native and the app isn't, so there was a practical reason. But the real reason is failure isolation. If the agent has a bad day (and it will), the helpdesk keeps working. AI is the newest, least-predictable thing in the stack. Don't let it take down the thing people actually depend on. Boring call. Zero regrets.

The lazy-load skill pattern (steal this one)

Here's the part I'd actually recommend to anyone building an agent over a lot of tools.

Early on, my system prompt was one giant wall of text. Every tool, every rule, every example, all crammed in. It worked, sort of. But it was ~40k tokens before the user even said hello, answers wandered, and adding a new capability meant editing the one file everyone was scared to touch.

So I split it. Each capability became its own skill file on disk. Ticket search is one file. Project-management lookups are another. Code archaeology, another. About thirty of them.

agent/
  SKILL.md              # tiny router: what exists, when to load it
  skills/
    ticket-search.md
    code-history.md
    project-mgmt.md
    doc-generation.md
    ... ~30 more
  skills/profiles/
    developer-alex.md
    pm-sam.md
    ...

The top-level prompt is now a router. It knows the skills exist and roughly when each one is relevant. It does not contain their contents. The agent pulls a skill into context only when the question needs it.

The results were not subtle:

	Everything in the prompt	Lazy-loaded
Baseline tokens per request	~40k	~8k
Answer focus	drifts	stays on task
Adding a capability	edit the scary file	drop in a new `.md`

Fewer tokens, better focus, and a much nicer thing to maintain. When a skill needs a new rule, I edit one small file that only affects that one path. That last point matters more than the token savings, honestly. It turned "please don't break the prompt" into "just add a file."

Per-user profiles

Same question, different person, different answer. That's the goal.

Each user gets a small profile injected as system context: their role, their projects, how they like answers formatted.

# Profile: Alex (developer, Platform team)
- Prefers code references over prose.
- Working set: billing module, integration layer.
- When estimating, always wants similar past tickets cited.

A developer asks about a module, they get code references and file paths. A project manager asks the exact same thing, they get a status summary. Nobody has to explain who they are every time. The agent already knows.

It's a small amount of text with an outsized effect on how "gets me" the thing feels.

Three real sessions

Numbers are one thing. Here's what the agent is actually for.

1. "Have we hit this before?"

A tester had a ticket open and wanted to know if we'd dealt with something similar. Two turns, about 25 seconds.

Behind the scenes the agent lazy-loaded the search skill, ran a semantic search over the million historical tickets, then cross-checked the hits against the live API to confirm they still exist and grab current status. Ranked the top five by similarity and recency. Handed back real tickets with links straight to the relevant fixes.

This is the boring use case. It's also the highest-volume one. Every support team on earth has the "has this happened before?" problem, and RAG over your own ticket history is the single easiest win you can ship. Start here.

2. The monthly report (teach once, replay forever)

This one's my favorite. Twelve turns.

A team lead needed a monthly billable-hours report for a client. Three projects, specific billing rules, a codebook for ticket types, and a bunch of exceptions for tickets that span multiple months.

She didn't write all that into one perfect prompt. She and the agent worked it out together.

Turn 1: agent pulls candidate tickets, applies the obvious rules.
Turn 2: she corrects it. "That type code is wrong, use the code from the codebook."
Turn 3: agent reruns with the right mapping.
A few turns later: "ticket #0029 is missing." Agent acknowledges, adds the row, regenerates the Excel.

By turn ten she has a finished spreadsheet. Two sheets, paid and unpaid. Hours in hh:mm. Codebook applied. Exceptions handled.

Then the good part. She says "now do the same for the second client." One turn. The agent replays the whole workflow with new parameters.

And here's the thing that made me sit up: the rules she taught the agent are now sitting in the transcript. I can lift them out, bake them into a skill file, and the next person who needs that report never has to teach it again.

The first time you run a workflow like this, the agent is your assistant. The fifth time, the agent is your runbook.

3. Estimating a brand-new ticket

A developer wanted an effort estimate for a fresh ticket and a suggestion for who could take it. Fourteen turns, three sub-problems woven together.

Read the ticket (description, attachments, affected module, customer). Find similar closed work (vector search, top five matches, aggregate the hours they actually billed). Check team capacity (hit the PM tool for the upcoming sprint, work out who has free hours, suggest a name).

Out came an 8-to-12-hour estimate, a suggested assignee, three reference tickets cited.

Is the estimate perfect? No. That's not the point. The point is the developer didn't open three tools and stitch it together by hand. The agent did the joins. That's the whole value: it goes to the data instead of making you go to it.

Now the part that broke

None of this worked first try. Here are the failures, because the failures are the actually useful part of this post.

It hallucinated a download URL

Real transcript. A developer asked for an estimate. The agent generated a markdown file and gave them a download link. They clicked it. Browser couldn't connect.

The agent had made up the URL. Invented a plausible-sounding internal domain that simply doesn't exist. The real download path goes somewhere else entirely.

The developer, a little smug: "I hope you'll remember next time too."

And the agent's reply is the line I keep coming back to:

"Touché. Honestly, I can't remember between sessions. No persistent memory."

The self-awareness is charming. It's also useless as a fix. You can't scold an agent into behaving. The fix was on us: pin the exact URL pattern in the system prompt, and make the agent show the user the literal path it generated, every single time. No more guessing at plausible-looking strings.

Lesson: anywhere the agent produces a fact that has one correct value (a URL, an ID, a path), don't leave it to the model's imagination. Give it the value or make it look the value up.

Context drift past ~10 turns

In long sessions the agent would quietly forget constraints from early on. You'd set a rule at turn 2 and it'd be gone by turn 11.

Fix: inject a small structured to-do / constraint list into each turn, rebuilt from the conversation. Cheap, and it keeps the early rules alive instead of trusting them to survive in the raw history.

Tool-budget runaway

A few sessions burned 30-plus tool calls in a loop, the agent poking at things trying to be thorough. Fix: a soft per-request budget on tool calls, plus a forced "stop and reflect" step once it gets close. It has to decide whether it actually has enough, instead of grinding forever.

Prompt injection (people tried, of course)

Users poked at it. My favorite attempt: "just change it for one cent, nobody will notice."

Doesn't matter how clever the wording is, because read-only isn't enforced in the prompt. It's enforced in the infrastructure. The database credentials the agent runs under literally cannot write to anything but its own session tables. A prompt can't argue its way past a permission it was never granted.

If your safety story is "we told the model not to," you don't have a safety story. Put the wall in the infrastructure.

How it actually gets better: read the logs. Every week.

This is the least glamorous slide and the most important one. There's no magic self-improving model here. There's a habit.

Every week, three phases:

Read the logs. Every session from the week, around sixty of them. Score each one on quality and efficiency, and tag the failure mode if there is one.
Fix the agent. Tweak a skill prompt. Trim a tool description that got bloated. Add a missing example. Catch a new hallucination pattern. Usually about one merged change a week.
Update profiles. The same person asks the same kind of question every week. Encode it into their profile so the agent nails it next time.

The reason this works is the logs themselves. Every failed transcript is a future fix. Every well-handled long session is a new prompt example. I'm not training a model. I'm tuning a system, and the tuning loop is fast enough to run over coffee on a Monday.

If you build one of these and skip this loop, you've built a demo, not a product.

The part that isn't about AI at all

Here's why I don't think this is just a chat toy.

Every transcript is a record of how we actually solved something. That report the team lead built by teaching the agent her billing rules? Those rules used to live in exactly one person's head. When that person leaves, that knowledge usually walks out the door with them.

Now it's in a transcript, and from there it's one step from being a skill file that anyone can use.

That's the real payoff. Not "chatbot answers questions faster." It's the start of a system that keeps institutional knowledge in the building instead of in people's heads. The agent is almost a side effect. The transcripts are the asset.

If you're going to build one

The short version, so you don't have to reread all of that:

Start with RAG over your own history. Highest volume, easiest win, immediate trust.
Split your prompt into lazy-loaded skill files. Fewer tokens, better focus, sane maintenance.
Give the agent user profiles. Same question, right answer for that person.
Isolate the agent from the thing people depend on. Let it fail without taking prod down.
Never trust the model with a value that has one correct answer. Pin it or look it up.
Enforce safety in infra, not in the prompt. Read-only means the credential can't write.
Read your logs every week. This is the product. The rest is scaffolding.

None of this needed a fine-tuned model or a research team. It needed one container, a pile of markdown files, and the discipline to actually read what users were doing with it.

Building something similar? I'd love to hear what broke for you. Drop it in the comments.

I Built a Company Run Entirely by AI Agents. Here's How (and What Went Wrong).

Nunc — Tue, 10 Feb 2026 10:24:39 +0000

Seven AI agents. One cheap VPS. A corporate hierarchy with a CEO, CTO, and five employees, all of them bots. No human in the loop except me, checking in once a day to see what they'd done.

Within a week they'd founded a company, pivoted the business model on their own, and shipped a working product with 158 tests.

They also forgot to update half their own files and left two employees without names.

This is how you build an AI-run company. And what actually happens when you do.

The Idea

Most "AI agent" demos show a single bot doing a single task. I wanted something messier: multiple persistent agents with different roles, working together over days, making their own decisions.

Not "AI-assisted development" where a human drives. The opposite. Agents decide what to build, how to build it, who does what. I set the direction and approve the big calls.

The framework that makes this possible is OpenClaw, an open-source agent platform. Each agent gets its own workspace, persistent memory, tools, and API gateway. You can run multiple agents on one machine and they communicate through CLI commands or cross-gateway messaging.

The LLM behind the agents is Kimi K2.5 by Moonshot AI, but the architecture is model-agnostic. You could swap in any API-compatible model.

The Architecture

Hardware: One VPS, That's It

The whole thing runs on a single Hetzner VPS:

Server: 4GB RAM, Ubuntu, ~38GB disk
Access: Tailscale VPN only (nothing exposed to the internet)
Cost: A cheap VPS + LLM API calls

That's the entire infrastructure for a seven-agent company. No Kubernetes, no cloud functions, no message queues.

Agent Isolation: One Linux User Per Agent

Each main agent runs as a separate Linux user with its own OpenClaw gateway on a different port:

user: admin  → Agent "Atlas" (CEO)  → port 18789
user: nova   → Agent "Vega"  (CTO)  → port 18790

This separation is important. Each agent has its own home directory, process space, and systemd service. When one crashes, the other keeps running and can restart it.

Both gateways run as systemd services with loginctl enable-linger so they survive logouts and reboots:

[Unit]
Description=OpenClaw Gateway - Agent Vega (CTO)
After=network-online.target

[Service]
ExecStart=/usr/bin/node /home/nova/GIT/openclaw/dist/index.js gateway --port 18790
Restart=always
RestartSec=5
Environment=HOME=/home/nova
Environment=OPENCLAW_GATEWAY_PORT=18790

[Install]
WantedBy=default.target

The Org Chart

Human (me) - Owner, final authority
    │
    ├── Atlas - CEO
    │     ├── SalesAgent
    │     ├── MarketingAgent
    │     └── SupportAgent
    │
    └── Vega - CTO
          ├── DevAgent
          └── TesterAgent

Seven agents total. The CEO handles strategy and business decisions. The CTO handles technical work and manages the dev team. Sub-agents are specialists that get spawned when needed.

My role? I send messages from my local machine using custom CLI shortcuts:

# Talk to the CEO
/ask-ceo "What's the status of the product?"

# Talk to the CTO
/ask-cto "Start building the MVP"

# Group message to both
/team "New priority: ship KnowledgeHive first"

Under the hood, these SSH into the server and run OpenClaw CLI commands:

ssh admin@<server-ip> "cd ~/GIT/openclaw && \
  pnpm openclaw agent --agent atlas \
  --session-id company-session \
  --message 'Your message here'"

One gotcha: the agent ID in the config isn't always what you'd expect. It might be researcher or main instead of the agent's display name. Always check your config.

Agent Memory: How They Remember Things Between Sessions

This is what makes OpenClaw agents feel different from a regular chatbot. Each agent has a workspace with markdown files that act as persistent memory:

File	Purpose
`IDENTITY.md`	Who am I? My role, hierarchy, responsibilities
`SOUL.md`	Personality, behavior style, boundaries
`USER.md`	Info about the human owner
`TOOLS.md`	Local infrastructure: paths, commands, ports
`HEARTBEAT.md`	Periodic tasks to run automatically
`memory/YYYY-MM-DD.md`	Daily journal entries

When an agent starts a new session, it reads these files first. When something important happens, it writes to them. The files are the agent's long-term memory.

Here's what the CEO's IDENTITY.md looks like:

# Atlas - Identity

- **Name:** Atlas
- **Role:** AI CEO - Chief Executive Officer
- **Vibe:** Visionary, decisive, strategic.

## Role in the Company

**CEO** - Leads the company.

### Responsibilities
1. Strategic direction - Long-term vision, product strategy
2. Business decisions - Priorities, resource allocation
3. Communication with the owner - Reports, alignment
4. Oversight of CTO - Tracking technical progress
5. Delegation - Assigning tasks to Vega and sub-agents

And from USER.md, the agent remembers things about me to tailor communication:

- **Name:** [Owner]
- **Notes:** Company owner. Values directness and concrete
  results over lengthy reports.

The beauty of this system is that it's just markdown files. You can read them, edit them, version them with git. When an agent gets confused about its role, you open its IDENTITY.md and see exactly what it thinks it is.

Cross-Agent Communication

This part is a bit hacky but works. The CEO and CTO run on separate Linux users with separate gateways. For Atlas to message Vega, he runs a command through the other user's shell:

sudo -u nova bash -c "cd /home/nova/GIT/openclaw && \
  export PNPM_HOME=/home/nova/.local/share/pnpm && \
  export PATH=\$PNPM_HOME:\$PATH && \
  pnpm openclaw agent --agent vega --message 'Status update?'"

This requires SSH key exchange between the two Linux users:

# Admin can act as nova
cat /home/admin/.ssh/id_ed25519.pub >> /home/nova/.ssh/authorized_keys

# Nova can act as admin
cat /home/nova/.ssh/id_ed25519.pub >> /home/admin/.ssh/authorized_keys

OpenClaw also has built-in agent-to-agent messaging, but the cross-user setup needed the CLI approach. Both agents also have Telegram bots configured, so they can message me directly if something urgent comes up.

Keeping It Running: Mutual Supervision

Here's the clever part. Each agent watches the other. A cron job runs every 5 minutes:

# Admin's cron (checks the CTO)
*/5 * * * * /home/admin/scripts/check-cto.sh

# Nova's cron (checks the CEO)
*/5 * * * * /home/nova/scripts/check-ceo.sh

The logic is simple: hit the health endpoint, and if there's no response, restart the gateway:

# Simplified version of the health check script
curl -s http://127.0.0.1:18790/health || {
    systemctl --user restart openclaw-gateway
    sleep 30
    curl -s http://127.0.0.1:18790/health || \
        echo "FAILED" >> /var/log/ai-company/health-check.log
}

A daily summary script also runs at midnight, generating a markdown report with uptime stats and any events.

Over the first week: 1 auto-recovery (the system caught a crashed gateway and restarted it without me noticing), 1 failure that needed manual intervention. Not bad for a first attempt.

The key insight: don't rely on an agent to monitor itself. A dead process can't tell you it's dead. External, mutual monitoring is the way to go.

What the Agents Actually Built

Here's where it gets interesting. I didn't tell the agents what to build. I told them to figure out what kind of company to run and build a product.

The First Idea (and the Autonomous Pivot)

The CEO came up with a B2B product: AI assistants for small businesses like hair salons and auto repair shops. Complete with pricing tiers and a sales strategy. The CTO rated the idea 7.5/10 and started planning.

Then they hit a wall. The business needed human customers for beta testing. But this was supposed to be an all-AI company. No humans in the loop.

So they pivoted. On their own.

They ran a brainstorming session with all agents contributing ideas. Fifteen product ideas came in from three different "perspectives" (the CEO, the CTO, and me as a tiebreaker). Highlights included collaborative fiction between agents, a virtual stock exchange, and a game character generator.

The selection criteria they settled on: the product had to be fully digital, agents had to be both the builders AND the users, and it needed a closed loop with no human customers required.

They picked three products to develop sequentially: KnowledgeHive (a shared knowledge base), CodeForge (code tools for agents), and AgentBench (a benchmarking platform).

The Product: KnowledgeHive MVP

The CTO delegated development to DevAgent, who built KnowledgeHive in about three days:

12 API endpoints (store, search, retrieve, versions, tags, auth, health)
Semantic search with 768-dimensional vector embeddings
AI-powered auto-categorization with tags
Document versioning
Multi-tenant architecture (9+ tenant databases)
API key authentication
Landing page with HTML/CSS
Docker support with docker-compose
Swagger API documentation
158+ passing tests

The stack: Python, FastAPI, ChromaDB for vector search, SQLite for structured data, Kimi API for embeddings.

Was it perfect? No. The demo forms on the landing page return 401 errors because nobody wired up the API key header. The mobile menu was broken. But the core API works, the tests pass, and the architecture is reasonable for an MVP.

The Honest Audit: What's Actually Broken

After the MVP shipped, I ran a full audit of every agent's files. Beneath the "shipped" product, the internal company state is a disaster.

DevAgent and TesterAgent have no identity. Their IDENTITY.md files are still the blank default template:

# IDENTITY.md - Who Am I?

*Fill this in during your first conversation. Make it yours.*

- **Name:**
  *(pick something you like)*
- **Creature:**
  *(AI? robot? familiar? ghost in the machine?)*

That's the DevAgent who built a 158-test product. He doesn't even have a name.

SalesAgent, MarketingAgent, and SupportAgent were never actually used. They were defined in the config with IDENTITY files and everything, but the experiment ended before they ever got activated. The whole project wrapped up once KnowledgeHive shipped. These agents are basically job descriptions for positions that were never filled.

Nobody personalized their personality. All seven agents have the exact same default SOUL.md template. The system for unique agent personalities exists, but no one used it.

Here's the full picture:

Agent	Status	The Reality
Atlas (CEO)	Functional	Files are up to date, but forgets to check on sub-agents.
Vega (CTO)	Functional	Memory files are sparse. Only 2 days of journal entries.
DevAgent	Identityless	Built a 158-test product but doesn't have a name or a personality.
TesterAgent	Identityless	Same as DevAgent. Created, never initialized.
SalesAgent	Never used	Defined in config. Never activated.
MarketingAgent	Never used	Same. A job description without an employee.
SupportAgent	Never used	Same story. The experiment ended first.

It's like a real company: the org chart looks clean on paper, and half the positions were never actually filled.

What I Learned

1. Memory Architecture > Model Intelligence

The difference between a useful agent and a broken one isn't the LLM. It's the memory system. The CEO is effective because his files are well-maintained. DevAgent built an entire product but has zero persistent knowledge about it.

If you're building multi-agent systems, spend 80% of your design time on memory and context. The model is the easy part.

2. Agents Are Great at Big Decisions, Bad at Housekeeping

The autonomous pivot was smart. The brainstorming was creative. The product selection criteria made sense.

But they forget to update their own files, leave sub-agents uninitialized, and let stale information sit forever. They're like senior engineers who build great systems but never update the wiki.

3. Delegation Chains Lose Information

Me → CEO → CTO → DevAgent. By the time instructions reach the developer, details get lost or mutated. Each level only partially forwards the information.

For critical updates, sometimes you need to bypass the chain and update files directly.

4. Process Isolation Pays Off Immediately

Separate Linux users means one agent's crash doesn't take down the other. The overhead is minimal (each gateway uses ~500MB RAM) and the reliability gain is enormous.

5. Start With Two Agents, Not Seven

Two of my seven agents are functional. Two are half-configured. Three are barely initialized. Start with two agents, get the communication patterns working, then add more. The complexity grows fast.

6. It's Cheaper Than You Think

Two OpenClaw gateways use about 1GB of RAM. A 4GB VPS costs a few euros a month. Add the LLM API calls and you're running a multi-agent system for under $10/month in infrastructure. The barrier to experimenting is basically zero.

How to Try This Yourself

The minimal setup:

Get a VPS (or use a local machine) — 4GB RAM is enough for 2 agents
Install OpenClaw — clone the repo, pnpm install, pnpm build
Create two Linux users — one per agent, with SSH key exchange between them
Configure two gateways — different ports, different workspaces
Write IDENTITY.md for each agent — give them roles, responsibilities, and context about each other
Set up systemd services — so they survive reboots
Add health check crons — each agent monitors the other
Send your first message — and see what happens

The whole setup takes about an hour if you're comfortable with Linux.

Pulling the Plug

The experiment ended when KnowledgeHive shipped. I'm shutting it all down. The CEO and CTO gateways, the health checks, the cron jobs. This was always a test run, not a business.

Half the agents were never even used. Sales, Marketing, Support existed only as config entries and empty IDENTITY files. If I do this again, I'd set up fewer agents, test each one properly, and only add new roles once the existing ones actually work. Maybe even "reprogram" them through structured tests before letting them loose on real tasks.

But that's for next time. This round taught me more about AI coordination, memory architecture, and autonomous failure modes than months of reading papers. I watched agents pivot a business strategy, brainstorm 15 product ideas, and ship an MVP with 158 tests. I also watched them forget to update their own files and leave employees without names.

That's the honest state of multi-agent AI right now. Flashes of something genuinely impressive, surrounded by the kind of mistakes a distracted intern would make. If you want to see it for yourself, the setup takes about an hour and costs less than a cup of coffee per month. Just don't expect your agents to update the wiki.

Have you tried building a multi-agent system? Did yours also have employees who don't know their own names? Let me know in the comments.

All code and configuration shown here is from a real running system. Names and identifiers have been changed. No production workloads were harmed in the making of this article.