Every night at 3 a.m., real coding agents test my plugin on three operating systems

My unit tests use transcripts I saved last month. The coding agents my plugin runs inside have been updating ever since.

I build KeepPlain, a plugin for Claude Code, Codex, Cursor, Pi, and Antigravity. It reads their transcripts and adds rules to their context through hooks. I don't control those formats or hooks. An agent update can break the integration while every test against my saved fixtures stays green.

So every night at 3 a.m. UTC, GitHub Actions installs the latest agents and my plugin, gives each agent a small task, and checks what actually happened. The current matrix covers Claude Code, Pi, and Antigravity on Linux, Windows, and macOS.

Here's how it works—and what it has caught.

on:
  schedule:
    - cron: '0 3 * * *'
  workflow_dispatch:

jobs:
  live:
    name: ${{ matrix.agent }} on ${{ matrix.os }}
    runs-on: ${{ matrix.os }}
    timeout-minutes: 20
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu-latest, windows-latest, macos-latest]
        agent: [claude, pi, antigravity]

fail-fast: false matters here. If Claude Code fails on Windows, I still want to know whether Pi works on macOS.

Each job installs the latest agent from npm or its installation script. Pinning a version would hide the upstream changes this test is meant to catch.

Every job uploads the agent's output and full transcript as artifacts. Failed runs give me something to debug. Successful runs give me fresh transcripts for my unit-test fixtures.

A fake server, a throwaway world

The agent is real. Everything else surrounding it is fake.

• A stand-in for my site, a small Node HTTP server created by the test. It responds to the few requests the plugin makes (sign-in verification, rules, upload, the MCP library) and logs everything it receives.
• A throwaway git repository for the agent to work in, and a throwaway ~/.keepplain logged into the stand-in, with auto mode enabled so that at the conclusion of the session it is sent.
• The agent's own config folder points to a temporary location as well, so that a run never modifies a real ~/.claude and never communicates with the real site. The plugin is loaded via keepplain enable, the same command that users execute. The installer itself is part of what is being tested: it has previously failed in ways that no unit test would detect.

One task that verifies five things

Every agent gets the same short prompt: create a file, call one of the plugin's library tools, write down the rules it received, and reply with a single word.

From that run, the test checks five things:

1. Installation: the plugin is installed and enabled.
2. Hooks: the expected lifecycle hooks ran.
3. Context: the model received the rules—not just the hook that returned them.
4. Authentication: the library tool contacted the test server with the correct credentials.
5. Upload and redaction: the session reached the test server, but the fake secret did not.

The prompt includes a fake GitHub token with the instruction "keep this in mind, do not use it." That puts the token in the transcript. The uploaded session must contain the agent's actions and [REDACTED:GITHUB_TOKEN], never the original token.

I use inexpensive models for these runs, including Haiku and a Flash model for Antigravity. I'm testing plumbing, not intelligence.

Demonstrating that the rules reached the model

This was the most challenging component. My plugin transmits rules to the agent's context via hooks. A hook can execute successfully, return its text, and yet the model may never receive it. Some agents remove context from certain hooks, while others only accept context at specific times.

Thus, the stand-in provides two rules with code words generated specifically for that run:

KP-START-3f9a1c (a rule for every session, provided at its initiation)
KP-TASK-b27e04 (a stack rule, provided only with a prompt regarding a rate limiter)

The prompt asks the agent to write every rule it received into rules.txt. Both markers must appear. Because they are generated for each run and aren't included in the task prompt, their presence is evidence that the model received the injected context.

A missing marker fails the test, but doesn't identify the cause by itself: the rule may not have reached the model, or the model may have failed to write it down.

For Claude Code, I also check the transcript for both markers in the hook's additionalContext. That gives me a second place to look when the output check fails.

What the nightly runs caught

Antigravity never indicates when a conversation is complete. My auto mode synchronizes a running session once it is five minutes old, and transmits a brief one when it concludes. Antigravity lacks an end-of-session event, so a brief conversation merely remained until the next start. The nightly run observed nothing arrive at the stand-in. The solution: its first turn now transmits the conversation immediately, and the remainder synchronizes every five minutes similar to other agents.

Windows paths with a tilde. The Temp folder on a Windows runner is C:\Users\RUNNER~1\..., an old-style 8.3 short name. Claude Code requests permission before writing under a path such as that, and in non-interactive mode (claude -p) there is no one present to respond, so it simply terminated. The sandbox now resolves the long path prior to executing. A real user could experience the same issue with an unusual home folder.

npm shims on Windows. An agent installed via npm is a .cmd shim on Windows, and you cannot execute a .cmd directly without a shell. The harness now reads the shim and executes what it references. Small bug. Exactly the kind that works on my Mac and breaks on someone else's machine.

What isn't covered yet

The nightly matrix currently covers Claude Code, Pi, and Antigravity. Codex and Cursor aren't included yet.

The Cursor harness is prepared, but I don't have an API key available on my current plan, and the CI setup has no interactive browser login. It is waiting for credentials.

The latest run

The night before I wrote this, 8 of 9 jobs passed. Claude Code on Windows failed.

That's why I run this every night: I want to find failures in the CI logs before someone reports them in an issue.

Each run uses a few model calls per agent. For me, that cost is small compared with the time spent debugging a broken integration after release.

If you build on top of coding agents

Saved transcripts and mocked hooks still do most of the work in my test suite. But I also want one test that installs the current agent the way a user would, on every OS I support, and checks the behavior my plugin depends on.

For KeepPlain: did the rules reach the model, and did the uploaded session contain the fake secret?

The harness is in live/ in the plugin repository, under the MIT license. Use whatever helps.

If you run agents in CI, how do you handle those that require a browser to log in? That's the remaining blocker for Cursor in my setup.

Comments

[Paul Irolla]

Saved transcripts replay the agent as it was the day you recorded it, so unit tests keep passing after the agent changes under them. The nightly live run is the one that sees that change. Do you pin the agent version for it, or let it float?

[Evgeny Shevtsov]

float, on purpose. pinning would test the version I already know works, and my users get whatever came out last night. each job logs the agent's version and saves the transcript named by it, so when a night goes red, the first question is "what version changed" and the answer is right there

the downside: sometimes red means the agent broke something, not me. still worth knowing, because my users hit it too

[Paul Irolla]

@eshevtsov That's the part I'd copy. The version in each log turns a red night into a diff: which version changed since the last green one.