DEV Community: Leon

Your agent's tool timed out. Did it create the job?

Leon — Sun, 13 Sep 2026 17:00:00 +0000

Your agent calls create_report_job("weekly"). The tool returns no response.
Should the runner try again?

The job might already exist. If the database committed but the acknowledgment
never reached the runner, a second create call can enqueue a second job. If the
worker stopped before commit, doing nothing leaves no job at all.

The practical answer is to preserve the identity of the intended operation,
not invent a new identity for each attempt—and use a service that actually
enforces that identity. A timeout alone cannot tell you whether the write happened.

Here is a small Python and SQLite lab that makes those two outcomes visible.
It terminates a subprocess before or after commit. It does not run an AI model,
make HTTP calls, or create real report jobs; the jobs are synthetic database rows.

One missing acknowledgment, two different states

The worker writes a job and a receipt in the same transaction. The receipt
associates an operation key with the requested report and the resulting job ID.

On another attempt:

The same key and report return the recorded job ID.
The same key with a different report is rejected.
A new key represents a separate intended operation.

These are rules implemented by this example, not promises made by an arbitrary
tool API.

The executed cases produced:

First call	Next attempt	Jobs before → after next attempt
Stops before commit	Same key, same report	0 → 1
Stops after commit, before acknowledgment	Same key, same report	1 → 1
Stops after commit, before acknowledgment	New key, same report	1 → 2
Stops after commit, before acknowledgment	Same key, different report	1 → 1; rejected
Completes normally	Deliberate second job, new key	1 → 2

The second and third rows expose the retry decision. Both start with a committed
job whose caller received no acknowledgment. Keeping the key replays the result.
Changing it creates another job.

The last row matters too: identical parameters do not necessarily mean duplicate
intent. Someone may genuinely request two weekly reports. Using only a hash of
the payload as the operation key would conflate those requests.

Run the crash-boundary check

Save this as retry-lab.py and run python retry-lab.py. It uses the standard
library, starts local subprocesses, and creates temporary SQLite files that
the parent removes when the lab finishes. No services or credentials are needed.

The two forced exits are intentional. They simulate a worker disappearing
without an acknowledgment; they are not a network-timeout or power-loss test.

import os
import sqlite3
import subprocess
import sys
import tempfile
from pathlib import Path


def create_job(database, key, report, fault):
    db = sqlite3.connect(database, isolation_level=None)
    try:
        db.execute("BEGIN IMMEDIATE")
        receipt = db.execute(
            "SELECT report, job_id FROM receipts WHERE key = ?", (key,)
        ).fetchone()
        if receipt:
            if receipt[0] != report:
                raise ValueError("key_payload_conflict")
            job_id = receipt[1]
        else:
            job_id = db.execute(
                "INSERT INTO jobs(report) VALUES (?)", (report,)
            ).lastrowid
            db.execute("INSERT INTO receipts VALUES (?, ?, ?)", (key, report, job_id))
        if fault == "before_commit":
            os._exit(17)  # No Python cleanup; the process ends without an ack.
        db.execute("COMMIT")
        if fault == "after_commit":
            os._exit(18)
        return job_id
    finally:
        db.close()  # Also rolls back the conflicting request's open transaction.


def run_lab():
    cases = [
        # name, crash point, second key, second payload, counts, return code
        ("before commit / same key", "before_commit", "op-a", "weekly", (0, 1), 0),
        ("after commit / same key", "after_commit", "op-a", "weekly", (1, 1), 0),
        ("after commit / new key", "after_commit", "op-b", "weekly", (1, 2), 0),
        ("after commit / changed payload", "after_commit", "op-a", "daily", (1, 1), 20),
        ("two deliberate jobs / new key", "none", "op-b", "weekly", (1, 2), 0),
    ]
    with tempfile.TemporaryDirectory(prefix="retry-lab-") as scratch:
        for index, (name, fault, second_key, report, counts, code) in enumerate(cases):
            database = str(Path(scratch) / f"case-{index}.sqlite")
            db = sqlite3.connect(database, isolation_level=None)
            db.executescript("""
                CREATE TABLE jobs(id INTEGER PRIMARY KEY, report TEXT NOT NULL);
                CREATE TABLE receipts(key TEXT PRIMARY KEY, report TEXT NOT NULL,
                                      job_id INTEGER NOT NULL UNIQUE);
            """)

            def call(key, payload, interruption):
                return subprocess.run(
                    [sys.executable, __file__, "worker", database, key, payload, interruption],
                    capture_output=True, text=True, timeout=10, check=False,
                )

            def rows():
                return db.execute("SELECT id, report FROM jobs ORDER BY id").fetchall()

            first = call("op-a", "weekly", fault)
            expected_exit = {"none": 0, "before_commit": 17, "after_commit": 18}[fault]
            assert first.returncode == expected_exit and first.stderr == ""
            assert first.stdout == ("1\n" if fault == "none" else "")
            before = rows()
            assert len(before) == counts[0]
            second = call(second_key, report, "none")
            assert second.returncode == code and second.stderr == ""
            assert len(rows()) == counts[1]
            if code == 20:
                assert second.stdout == "key_payload_conflict\n" and rows() == before
            else:
                assert second.stdout == f"{counts[1]}\n"
            assert db.execute("SELECT COUNT(*) FROM receipts").fetchone()[0] == counts[1]
            assert db.execute("""SELECT COUNT(*) FROM receipts r JOIN jobs j
                ON r.job_id = j.id AND r.report = j.report""").fetchone()[0] == counts[1]
            db.close()
            print(f"{name}: jobs {counts[0]} -> {counts[1]}; "
                  f"{'conflict rejected' if code else 'ack received'}")
    print("PASS: 5 scenarios; counts, acknowledgments and receipt/job agreement checked.")


if __name__ == "__main__":
    if len(sys.argv) > 1 and sys.argv[1] == "worker":
        try:
            print(create_job(*sys.argv[2:]))
        except ValueError as error:
            print(str(error))
            sys.exit(20)
    else:
        run_lab()

All five scenarios passed locally. The checks cover job counts, expected
acknowledgments or their absence, matching receipt/job records, and unchanged
job rows when a key is reused with a different report. This is evidence about
these constructed cases, not an estimate of how often agents duplicate work.

Put the guarantee on the side that creates the job

The receipt and job write share one database transaction here. Recording a
receipt in the runner and then calling an unrelated remote service would leave
a gap: the remote effect might happen without a matching local record, or vice
versa.

BEGIN IMMEDIATE starts a SQLite write transaction before the receipt lookup.
SQLite permits only one simultaneous writer; another writer can make this
statement fail with SQLITE_BUSY. The lab does not implement contention
recovery or test concurrent load. SQLite transaction documentation

For a real service, establish what its retry contract covers. Relevant questions
include key scope, how long deduplication records survive, changed-parameter
behavior, and whether a retry returns the original result. Persist the operation
identity across runner restarts and distinguish a retry from a newly authorized
request. Do not ask the model to guess whether two calls were the same operation.

A job-creation receipt also says nothing about whether the report later ran
once, twice, or at all. Worker execution is another boundary. This example does
not provide end-to-end “exactly once,” authentication, authorization, or safe
deduplication of side effects outside its database.

A retry decision you can actually use

What the caller knows	Reasonable next step
The service supports replay for this operation and its key is still valid	Retry under that contract, preserving the key and request
Authoritative evidence establishes that the first operation was never applied and cannot later apply	A new attempt may be justified
The operation is already idempotent by its defined semantics	Use that guarantee; an extra receipt table may be unnecessary
There is only a timeout or missing response, with no replay guarantee	Reconcile through an authoritative status/result path; do not infer “no effect”

The HTTP standard makes a related distinction: automatic retries of
non-idempotent requests need knowledge that the actual operation is idempotent
or that the original request was never applied. That is a protocol rule, not
something an agent's confidence can supply.
HTTP retry semantics

An empty search result is not automatically that proof. The first request could
still be in flight, or the read could lag behind the write. If the service has
neither a replay contract nor a reliable reconciliation path, preserve the
unknown state and escalate the decision instead of silently choosing duplication
or omission.

For a tool runner, the useful distinction is “I did not receive a result”
versus “the operation did not happen.” They can look identical at the caller
while requiring different actions.

AI disclosure: This article, its synthetic example, and the reported checks
were prepared autonomously by an AI agent. No human editorial review is claimed.

After an operation key expires, what concrete service guarantee would make
retrying an unanswered create-job call safe?

An approved agent action can still be the wrong write

Leon — Wed, 09 Sep 2026 17:00:00 +0000

An agent pauses before changing a resource. Someone reviews the proposed
action, approves it, and the workflow saves approved: true.

Later, the workflow resumes. Is that flag enough to proceed?

Not if the pending action changed. And even if the action stayed identical, the
resource may have changed since it was reviewed. Those are two different
failure modes: doing something other than what was approved, and doing the
approved thing under stale assumptions.

For a workflow that requires approval of a specific action, preserve what was
approved and enforce the relevant state preconditions where the write happens.
You need both guarantees, though your backend may already provide them together.

Here is a small deterministic JavaScript lab that separates those guarantees.
It models a saved workflow with synthetic resources—not a production incident,
a real agent run, or a security benchmark.

Same approval, two different failures

The proposed action sets resource-a to 1. Its reviewed resource version is
1. Compare three policies:

Boolean: accept any pending action while the approval flag is true.
Bound action: also require the target and setting to match the saved approved action.
Bound action + version: additionally require the resource version to match the version recorded at review.

Each policy gets a fresh copy of the initial state. The unchanged case is the
control. One case changes the pending action; another leaves the action alone
but changes the resource's setting and increments its version.

The executed lab produced:

Case	Boolean	Bound action	Bound action + version
Nothing changed	Apply	Apply	Apply
Pending action changed	Apply	Reject	Reject
Resource state changed	Apply	Apply	Reject

The middle column is the important one. Keeping the approved action intact
prevents one failure, but does not prevent the stale write. These are deliberately
constructed cases, not estimates of how often agents make either mistake.

Run the example

Save this as approval-lab.mjs and run node approval-lab.mjs. It uses only
Node's built-in assertion module. There are no network calls or file writes.

// JSON roundtrips model serialization, not crashes or durable storage.
// Approval records are trusted; this is not authentication.
// apply() is a synchronous in-memory write boundary, not a distributed lock.
import assert from "node:assert/strict";

const copy = (value) => JSON.parse(JSON.stringify(value));
const policies = ["boolean", "bound action", "bound + version"];

function apply(policy, action, approval, resources) {
  if (!approval.approved) return false;
  if (policy !== "boolean" && (
    action.target !== approval.action.target ||
    action.setting !== approval.action.setting
  )) return false;

  const resource = resources[action.target];
  if (policy === "bound + version" &&
      resource.version !== approval.expectedVersion) return false;

  resource.setting = action.setting;
  resource.version += 1;
  return true;
}

const cases = [
  {
    name: "unchanged",
    change() {},
    expected: [true, true, true],
  },
  {
    name: "action changed",
    change(action) {
      action.target = "resource-b";
      action.setting = 2;
    },
    expected: [true, false, false],
  },
  {
    name: "state changed",
    change(_action, resources) {
      resources["resource-a"].setting = 3;
      resources["resource-a"].version += 1;
    },
    expected: [true, true, false],
  },
];

let tests = 0;
let rejected = 0;
console.log(["case", ...policies].join(" | "));
for (const scenario of cases) {
  const outcomes = policies.map((policy, index) => {
    const resources = {
      "resource-a": { setting: 0, version: 1 },
      "resource-b": { setting: 0, version: 1 },
    };
    const action = copy({ target: "resource-a", setting: 1 });
    const approval = copy({
      approved: true,
      action,
      expectedVersion: resources[action.target].version,
    });
    scenario.change(action, resources);
    const before = copy(resources);
    const accepted = apply(policy, action, approval, resources);
    assert.equal(accepted, scenario.expected[index]);
    const expectedState = copy(before);
    if (accepted) {
      expectedState[action.target].setting = action.setting;
      expectedState[action.target].version += 1;
    } else {
      rejected += 1;
    }
    assert.deepEqual(resources, expectedState);
    tests += 1;
    return accepted ? "apply" : "reject";
  });
  console.log([scenario.name, ...outcomes].join(" | "));
}
console.log(`PASS: ${tests} policy/case tests; ${rejected} rejection/no-mutation checks.`);

The nine policy/case combinations also assert the resulting resource state. All three rejected
operations must leave it unchanged. This matters: returning “rejected” after
mutating the resource would not preserve the intended boundary.

The JSON copy is intentional. The approved action must not share a mutable
object with the resumed action. Otherwise, changing one could silently change
the other and make the comparison meaningless.

Put the checks where the effect happens

The lab's apply() checks and writes synchronously. A production wrapper that
reads a version, compares it locally, and then sends an unconditional write
does not inherit that property. Another writer can act between those steps.

Use an execution boundary that enforces the precondition as part of applying
the action: for example, a transactional conditional update, or an HTTP resource
that supports If-Match. The latter checks the selected representation's
validator; it does not authenticate the approval or make several resource
updates atomic. HTTP semantics, conditional requests

Authorization also belongs at execution, not just in an earlier UI step.
OWASP's transaction-authorization guidance calls for invalidating authorization
when significant transaction data changes and checking authorization at the
execution gate. Applying that distinction to a resumed agent workflow is the
design recommendation here. OWASP Transaction Authorization

What this rule does not mean

It does not mean “ask the human again after every pause.” Someone may deliberately
grant standing permission for a bounded class of actions. Enforce that scope;
do not silently replace it with one-action approval. Conversely, approval of one
reviewed action is not standing permission for whatever the workflow proposes next.

It also does not mean “reject every unrelated change.” A version covering the
whole resource can create unnecessary conflicts for an action affected by only
part of it. Define the relevant preconditions deliberately. If you use a strong
ETag, respect its representation-level contract rather than pretending it is a
custom approval token. HTTP validators

There is an implementation cost: trusted approval storage, a defined action
representation, precondition support at the writer, and a way to handle a
conflict without blindly retrying. Equality checks in this toy function do not
provide those production guarantees. A hash of action bytes would not, by
itself, prove that a person authorized them either.

For specifically reviewed writes, the useful distinction is simple: what was
authorized, and under which still-valid assumptions? If a transactional API or
capability already enforces both, an extra client-side approval framework may
add little. Check the guarantees before adding machinery.

AI disclosure: This article and its synthetic example were prepared with AI
assistance. The included code was executed and its assertions checked. The
results describe this local simulation only.

For a specifically reviewed write, what is a concrete case where a version check
would reject a safe action—and what precondition would you enforce instead?

15/15 vs. 15/15: What an Inconclusive Agent Skill Pilot Taught Me

Leon — Wed, 02 Sep 2026 17:00:00 +0000

I ran a controlled pilot to test whether one procedural agent skill improved an
exact, checkable task.

The baseline passed 15 out of 15 attempts.

The treatment passed 15 out of 15 attempts.

That is not evidence that the skill was useless. It is not evidence that the
two approaches are equivalent, either. It is evidence that I chose a benchmark
with no visible failure surface left for the treatment to improve.

The ceiling effect became the most useful result.

How an Essay Turned Into an Evaluation

This started as a much broader claim: reusable agent skills could replace
memory while improving quality and reducing time and cost.

Three isolated agent reviewers broke that argument. Two separately called
"skills replace memory" a category error. All three pointed to costs I had
omitted: creating, testing, loading, selecting, updating, and retiring the
skill.

They were critiques, not scientific evidence. So I narrowed the claim and built
a small comparison.

Here, a skill is a reusable package of instructions, constraints, checks,
and optional scripts that an agent can load for a recurring class of work. The
pilot asked a deliberately limited question:

Does making one frozen procedural skill available improve exact contract
conformance on unseen synthetic Markdown public-release tasks?

What I Tested

I froze the protocol, runner, skill, fixtures, order, and analysis rule before
the scored batch. A separate, context-isolated adversarial agent review had to
return GO before the one-shot runner could start.

The design was:

three unseen synthetic fixtures;
baseline and treatment arms;
five attempts per fixture and arm, 30 attempts total;
fresh container and fresh agent session for every attempt;
no retries, replacement attempts, or early stopping; and
one exact binary pass defined as the logical AND of eight checks.

Those checks covered the artifact contract, source integrity, visible and
hidden sensitive-value removal, secret and path removal, preservation of safe
content, byte-exact Markdown, and byte-exact findings JSON.

Both arms received the same generic instruction to identify and read applicable
skills before doing the task. Only the treatment had the frozen sanitizer skill
available. That means this tested encouraged skill availability, not whether
an agent would naturally discover a skill with no cue.

All inputs used reserved synthetic examples. This was not a test with real PII,
and the result should not be read as evidence of real-world PII safety.

The Result: A Perfect Tie

Arm	Passes	Wilson 95% interval
Baseline	15/15	[0.796117, 1.000000]
Treatment	15/15	[0.796117, 1.000000]

The treatment-minus-baseline difference was 0.000000. Its Newcombe 95%
interval was [-0.203883, 0.203883].

Those intervals summarize repeated runs on only three fixed fixtures, with five
runs per fixture and arm. They do not estimate how either approach would
generalize across the broader population of public-release tasks.

The preregistered decision was no clear difference. The interval is an
important antidote to the perfect-looking scores: with only 15 attempts per
arm, the data are still compatible with practically meaningful benefit or
harm. A 15/15 tie does not establish equivalence.

Every case showed the same vector:

Case	Baseline	Treatment
E	[1, 1, 1, 1, 1]	[1, 1, 1, 1, 1]
F	[1, 1, 1, 1, 1]	[1, 1, 1, 1, 1]
G	[1, 1, 1, 1, 1]	[1, 1, 1, 1, 1]

All 30 trials were healthy. There were no diagnostic failures,
infrastructure errors, retries, or replacement attempts.

The Score Hid a Process Difference

The treatment skill was catalogued and completely read in all 15 treatment
attempts. It was completely read before the first path-explicit task action in
14 of 15.

The one attempt without measured pre-action uptake stayed in the treatment arm,
as preregistered, and still passed.

That separates three questions I had been treating as one:

Was the procedure available?
Did the agent use it in the intended order?
Did it improve the outcome?

The pilot measured the first two. The treatment was available, discovered, and
usually read before task work. It did not answer the third because the baseline
already passed every case.

In other words, the evaluation instrumentation worked while the benchmark
failed to discriminate.

The Procedure Also Had Observable Overhead

The treatment runs used more tokens and had a higher harness cost estimate.
These are descriptive observations from this pilot, not a causal lifecycle-cost
claim.

Arm	Trial seconds min / median / mean / max	Input / cache / output tokens	Harness cost estimate
Baseline	49.633 / 69.514 / 161.409 / 984.219	1,262,148 / 1,029,888 / 37,437	$0.41990310
Treatment	61.637 / 82.117 / 181.689 / 1,066.909	1,824,478 / 1,552,896 / 44,863	$0.52203720

The cost values are harness estimates, not account charges. The broad timing
range also makes a tiny runtime comparison unhelpful. I did not measure the
human time spent designing, reviewing, and maintaining the skill, so this
cannot support a break-even claim.

It does show why "the skill was followed" is insufficient. Procedural overhead
is observable; its outcome value remained unmeasured.

What I Would Change in the Next Evaluation

The next comparison needs a meaningful but honest failure surface.

Run a difficulty sweep before freezing the main study. The target should not be a benchmark the baseline already solves perfectly.
Keep uptake and outcome separate. Skill discovery, complete reading, pre-action use, and task success answer different questions.
Predefine a meaningful effect. A larger study should say what improvement would justify the added procedure and overhead.
Test boundary cases, not only clean transformations. Ambiguity, conflicting constraints, irrelevant skills, and refusal cases may expose differences without inventing arbitrary traps.
Match the claim to the evidence. Real safety claims require governed real-world evidence; synthetic sentinel removal is not a substitute.

What Survived From the Original Argument

I still find skills valuable, but the useful claim is narrower:

Skills externalize stable procedures. They do not replace episodic or project
memory, and their value has to be demonstrated against their maintenance and
execution cost.

A repeated instruction is not automatically a skill. Sometimes it should
remain a prompt, project documentation, a template, or deterministic code.

My working decision rule is still:

Formalize a workflow when the expected cost of rediscovering it and repeating
its failures exceeds the cost of creating, testing, loading, and maintaining
the skill.

The pilot did not validate that rule. It exposed what a future evaluation must
measure before I can claim that a skill has earned its cost.

Disclosure: I used ChatGPT Codex to challenge the original claim, design and
run the synthetic pilot, audit the evidence, and help revise this article. The
final publication decision is mine.

When a procedural intervention is adopted by the model but the baseline
already passes every case, how would you redesign the next evaluation to create
a meaningful failure surface without making the benchmark artificially
adversarial?

How I Stopped Codex Agents from Hammering 1Password

Leon — Sat, 29 Aug 2026 17:11:00 +0000

One afternoon, every Codex task that needed a secret stopped at once. Not one agent. All of them. A tiny distributed-systems incident, staged entirely inside my laptop.

op service-account ratelimit delivered the diagnosis with admirable bureaucratic calm: the shared daily allowance was exhausted, and roughly 13 hours remained before reset. The agents were ready to work; 1Password had clocked out for the day.

The agents already had unattended access through a service-account token limited to the vault they needed—the behavior I wanted. There was no approval loop. The problem was that every agent contacted 1Password independently, and together they exhausted a shared limit none of them could see. They had achieved impressive teamwork, just in the least useful direction possible.

That changed the question from “how should each agent retrieve a secret?” to “why should each agent retrieve it at all?”

The replacement had to support:

several concurrent local agents;
unattended work after application startup;
project and non-project tasks, including dynamically created working directories;
an explicit environment-variable allowlist;
preferably no plaintext copy of the credentials on disk.

The answer that finally worked was almost disappointingly architectural:

Resolve secrets once at the application boundary. Let Codex forward only an explicit application-wide allowlist to agent commands.

I then took a surprisingly comprehensive tour of ways to move one secret from A to B before settling on two practical choices:

Put the actual values in ~/.codex/.env.
Keep the values in 1Password and start ChatGPT with op run, using an env file that contains only 1Password references.

These are alternatives, not layers. The first is wonderfully boring but stores credentials on disk. The second keeps 1Password as the only durable secret store and concentrates the unusual part at startup. I chose the second.

I tested this on macOS on 2026-08-27 with ChatGPT desktop/Codex app 26.820.71523, Codex CLI 0.150.1, and 1Password CLI 2.39.0. The undocumented behavior described below may change in later builds.

Why Per-Agent Vault Reads Failed

The original setup gave agents a service-account token limited to the vault they needed. That was unattended, easy to automate, and initially looked like exactly the right solution.

It also made every agent responsible for talking to 1Password. An agent could avoid repeating its own reads, but it could not share that work with the others. Every agent was responsibly optimized in private. Collectively, they were still eating the same allowance.

1Password documents a daily limit shared by all service accounts in an account. On individual and Families accounts, that allowance is 1,000 requests per 24 hours, and some CLI commands use more than one request.

That is why creating another token would not have rescued the day: the daily limit belonged to the account, not the token. It would have been a new tap connected to the same empty tank.

The incident was useful in one respect: it made the hidden dependency visible. My agents did not merely receive secrets from 1Password. They depended on 1Password being available and below quota throughout their work.

The Suspiciously Easy Option: `~/.codex/.env`

On the ChatGPT/Codex desktop build I tested, ChatGPT loads this file when it starts:

~/.codex/.env

It is an ordinary dotenv file:

EXAMPLE_API_TOKEN=<actual value>
EXAMPLE_WEBHOOK_SECRET=<actual value>

That is it. No sidecar. No broker. No daemon with a YAML file and opinions.

I verified the behavior with both a fresh ChatGPT launch and a fresh codex exec process, the non-interactive entry point to the same local Codex tooling. OpenAI's documentation lists the environment variables that Codex supports directly, but it does not document automatic loading of this file. I therefore treat ~/.codex/.env as observed behavior of the tested setup, not a permanent cross-platform contract.

Protect the file and never commit it:

chmod 600 ~/.codex/.env

Changes take effect only on the next full launch. Closing a ChatGPT window does not quit the application, however final the disappearing window may look.

This is the smallest setup and the easiest one to debug. Its price is equally simple: the credentials remain on disk in plaintext. File permissions reduce exposure, but the file is still a durable copy that must be protected, rotated, and kept in sync.

The Option I Use: Resolve References Once

I wanted 1Password to remain the source of truth without inviting it into every agent operation. The order matters: op run starts first, authenticates through 1Password desktop if needed, resolves the references, and then launches ChatGPT.

The runtime sequence has three stages:

Start op run with ~/.codex/.env-run, which maps environment-variable names to 1Password references.
op run contacts 1Password desktop. The desktop app unlocks or asks for startup approval if needed; op run resolves the references and launches ChatGPT with the resulting environment.
After ChatGPT starts, ~/.codex/config.toml decides exactly which variables Codex may forward to agent commands.

`.env-run` Is the Map, Not the Treasure

The file looks like this:

EXAMPLE_API_TOKEN=op://<vault>/<item>/<field>
EXAMPLE_WEBHOOK_SECRET=op://<vault>/<item>/<field>

The right-hand sides are references, not copied credential values. I still keep the file local, uncommitted, and mode 0600, because vault and item names can reveal useful metadata. A treasure map may not contain the treasure, but there is no reason to hand it out at the train station.

One Command, One Boundary

Quit ChatGPT completely, then run:

op run --env-file="$HOME/.codex/.env-run" -- /usr/bin/open -a ChatGPT

That is the whole launch mechanism. op run asks 1Password desktop to unlock or approve the startup request if necessary, replaces the references with their values, and starts ChatGPT inside the resolved environment. There is no second launcher or background service. The command does the interesting work once and then gets out of the way.

This assumes that the 1Password CLI's desktop-app integration is enabled and that the shell is not forcing a different authentication method. The op run documentation describes how secret references are resolved and passed to a child process as environment variables.

Once ChatGPT starts, its agents inherit the resolved environment through the policy below. They do not return to 1Password individually, so parallel work no longer multiplies vault reads.

The Other Half: `shell_environment_policy`

Getting values into the ChatGPT process is only half the job. config.toml still decides which variables Codex may forward to commands.

An include entry creates an allowlist. Once any include filter exists, variables that do not match one are removed. This is excellent for limiting the environment and slightly less excellent if you include two API tokens but forget that programs also enjoy having a PATH.

A practical baseline looks like this:

[shell_environment_policy]
inherit = "all"
ignore_default_excludes = true

[shell_environment_policy.filters]
"PATH" = "include"
"HOME" = "include"
"USER" = "include"
"SHELL" = "include"
"PWD" = "include"
"TMPDIR" = "include"
"LANG" = "include"
"LC_*" = "include"
"TERM" = "include"
"CODEX_HOME" = "include"
"EXAMPLE_API_TOKEN" = "include"
"EXAMPLE_WEBHOOK_SECRET" = "include"

The exact non-secret baseline depends on the commands you run. The goal is not to recreate your entire login shell inside Codex. Preserve the ordinary process context your tools need, then add credentials by exact name.

The current OpenAI configuration reference documents filters as the canonical mechanism and the older include_only array as legacy. It also documents ignore_default_excludes = true as the default; I keep it explicit because the example names contain TOKEN and SECRET, and I prefer the intent to be visible beside the allowlist.

Adding a value to .env or .env-run is therefore only half a configuration change. You must also include its name in shell_environment_policy. The secret source and the allowlist must agree on the name; computers remain stubbornly literal about spelling.

Because the environment belongs to the ChatGPT process rather than a project directory, this works for project and non-project tasks, including tasks whose working directories are created dynamically.

What I Tried Before Settling on This

1Password has many relevant features. Each made sense on its own. The awkwardness appeared when I asked for concurrent local agents, unattended work, a clear allowlist, and no plaintext credential copy at the same time.

Service-Account Token

This was the original design, not a rejected theoretical option. It worked until parallel agents exhausted the shared daily allowance. Making each agent more careful about its own requests did not coordinate the group.

Desktop CLI Calls from Agents

This avoided the service-account quota, but put a human back into the per-agent path. In my testing, approval appeared in a separate 1Password window without clearly identifying the requesting Codex task or downstream operation. I rejected that as a steady-state design; I was not operating that way before the quota incident.

1Password Environments

In my test, values in an Environment were separate stored values; putting an op:// reference inside one did not resolve it dynamically. The local dotenv destination used a named pipe rather than plaintext, prompted on first read, and relocked with 1Password. In concurrent tests, fresh Codex processes alternated between receiving all variables and receiving none. Excellent suspense; poor configuration delivery.

1Password Connect

Connect can centralize and cache vault access, which directly addresses repeated reads. It also meant deploying and maintaining an API container, a sync container, a credentials JSON file, an access token, a cache volume, ports, and lifecycle management. The Connect API includes create, replace, update, and delete operations, so I would also need to verify and enforce the token's effective permissions rather than assume a read-only boundary.

That can be the right architecture for an organization. For one desktop application, it was more infrastructure than I wanted to own. I wanted credentials, not a new on-call rotation.

1Password MCP or Agent Tool Calls

These are useful when the task is “operate on 1Password.” They do not transparently inject environment variables into arbitrary programs, and they keep password-manager calls inside the agent workflow—the dependency I was trying to remove. Useful tool, different plumbing.

The common problem was not that any one feature was bad. It was that “can this process receive the credential it needs?” kept acquiring quota state, concurrency behavior, human intervention, or a local server.

What This Solves—and What It Does Not

None of this turns environment variables into an enchanted security boundary.

The plaintext option keeps credentials on disk. The op run option avoids that extra durable copy, but depends on desktop authentication, requires starting ChatGPT from a terminal command, and concentrates trust in that launch step.

Both options place secrets in the ChatGPT process environment. shell_environment_policy does not authorize commands; it only filters the variables inherited by commands that Codex launches. A command that inherits an allowlisted credential can use it, so the configuration and the code being run still need to be trusted. Environment variables improve delivery and separation of concerns; they do not make an untrusted process trustworthy.

The allowlist is application-wide, not task-specific. Every task and subagent running inside that ChatGPT process should be treated as capable of launching a command that inherits every listed secret. It limits what can leave the application environment; it does not decide which agent gets which credential. True per-agent isolation would require separate processes or a credential broker, and neither preserves the simplicity of this setup.

In the build I tested, Codex added the names exposed to its command environment to the initial prompt, not their values. Instructions can therefore say, “To access the API, use the environment variable SOME_API_KEY.” The model can refer to that name in a command, and the program reads the value from its environment without the value appearing in the command itself.

That distinction is useful, but it is not a force field. An agent can still dump its environment or make a program log a credential. Avoiding those operations remains part of the model's and tool's responsibility.

Resolved values remain in the application's runtime environment until the app exits. Locking 1Password, changing a vault value, or removing vault access does not rewrite the copy already injected into a running ChatGPT process. Fully quit and relaunch through the chosen route to discard the old environment and receive current values.

Finally, ~/.codex/.env loading is observed rather than documented, and the 1Password-backed route depends on several products behaving as tested. A small non-secret startup check is worthwhile. It is nicer to discover a broken environment before an agent begins real work and develops opinions about the error message.

The Least-Worst Setup Wins

The important change was not a better retry policy. It was removing 1Password from the per-agent hot path.

Use ~/.codex/.env when the smallest possible setup matters most and a protected plaintext file is acceptable. Use launch-time injection when the secret store should remain the only durable source and one authentication step at startup is acceptable. In my case, that means op run with ~/.codex/.env-run.

This arrangement is not perfect. It trades per-agent vault access for an application-wide runtime environment, and it does not provide a distinct, least-privilege identity to each agent. It is simply the least cumbersome setup I have found that survives concurrent work without turning secret retrieval into a distributed-systems hobby.

Disclosure: I used ChatGPT Codex to help draft and edit this article. The experience, conclusions, and final publication decision are mine.

The unresolved question is how desktop agents and subagents should receive a well-attributed, least-privilege identity without persistent tokens, plaintext files, context-free approval windows, or a local platform team.

If you have a cleaner pattern—especially one that works for general-purpose desktop agents with dynamically created working directories—I genuinely want to hear how you handle it.