DEV Community: Leon

IDE-Style Code Navigation and Refactoring Don’t Automatically Improve Your Coding Agent

Leon — Tue, 06 Oct 2026 14:36:13 +0000

The appeal of a drop-in upgrade for a coding agent is easy to understand: install a tool, give the agent IDE-style code navigation and refactoring, and expect better results. I wanted to know whether that actually helped the same agent finish tasks faster, with fewer mistakes or less token usage.

I tested one recommendation on ten refactorings in HTTPX. Each task had two attempts with the agent's ordinary tools and two with added navigation and refactoring tools: 40 completed runs.

Both workflows passed every acceptance check. With the added tools and explicit usage guidance, average agent time increased from 89.8 seconds to 111.4 seconds—about 24% longer. Average total reported tokens increased by about 40%.

For this prescribed workflow in fresh sessions, the added tools increased time and token usage without improving acceptance. That is why I want reproducible evidence and tested best practices before treating an extra tool as a drop-in improvement.

The Idea Behind the Recommendation

The idea made sense: “go to definition” and “find references” help when working with unfamiliar code. Searching for a method name can return unrelated methods with the same name; finding references to a specific symbol can give a more precise answer.

The recommended tool was Serena, which exposes these capabilities to agents through MCP, a protocol for connecting tools to AI clients. An agent can retrieve a function, inspect its references, or edit a particular symbol.

One route is a language server, which supplies code analysis for editor features through the Language Server Protocol (LSP). A native IDE integration is another: the JetBrains backend uses an IDE's own analysis and refactoring capabilities.

This experiment used the language-server backend with Pyright, a Python analysis tool.

The practical question was whether adding those capabilities improved the same agent's finished work after accounting for the effort of using them.

Why I Tested the Whole Workflow

Successful navigation and refactoring show capabilities. A productivity claim also needs evidence about completed work. For example, Serena's agent-written evaluations compare operation chains, call counts, and payload sizes on its JetBrains backend. Those reports do not establish gains across repeated development tasks with independent acceptance checks.

Its methodology raises a confirmation-bias concern. The agent chooses examples and judges its own work. Negative findings are requested, yet the prompt used to review the methodology assumes capable agents use the tools correctly, excludes misuse and failure modes, and explicitly says: “Do not question the validity of this assumption.”

I would not treat reports built on that protected assumption as proof of a drop-in improvement. For any tool, I want repeated complete-task comparisons, independent checks, and tested guidance for real workflows.

Ten Refactorings in HTTPX

I was looking for an existing, medium-sized Python project on GitHub: real code spread across files, enough variety for ten distinct refactorings, and a substantial test suite to check that behavior stayed intact. It also needed to be manageable for repeated runs.

HTTPX, a Python HTTP client, was an arbitrary choice that fit that brief. The selected snapshot had about 8,800 lines of production Python. Every attempt used the same fixed commit.

The ten tasks included renaming classes and methods, moving functions while preserving old imports, extracting shared implementations and modules, inlining a private helper, and simplifying control flow.

One task renamed Headers.get_list while leaving QueryParams.get_list unchanged. It also required preserving the old method as a compatibility alias. Another moved a file-length helper into a new module while preserving stream position and keeping the old import bound to the same function. These gave the checks something more specific to verify than whether the code still ran.

Every task had two attempts per condition, starting from fresh Docker copies of the same source. Runs were sequential, with the order reversed for the second repetition. The configuration was:

Component	Tested configuration
Agent	Codex CLI 0.160.0
Model	GPT-6.1 Sol, Medium reasoning
Evaluation runner	Harbor 0.23.0
Serena	1.7.0, LSP backend
Python analysis	Pyright 1.1.403
Per-run resources	2 CPUs, 4 GiB memory, Linux ARM64 Docker on a Mac

The ordinary workflow used shell commands, text search, file reads, and edits. The guided workflow kept those tools and added the tool server with explicit instructions: read its guidance, locate the target symbol, inspect references before editing, and use at least one applicable symbol-editing operation.

This compares a prescribed workflow using the added tools with the agent's ordinary workflow. The guidance is part of the intervention. It does not isolate the effect of LSP itself or measure how an agent chooses among optional tools.

Acceptance came from a separate offline verifier that checked change scope, requested structure, behavior, tests, type checking, and linting. Before the runs, it accepted ten known-good patches and rejected ten deliberately flawed ones. No human repaired the measured patches.

The full reproduction bundle is not public yet, so independent reproduction remains a limitation of this report.

What the 40 Runs Showed

Measurement	Ordinary tools	Guided tools	Change
Accepted attempts	20/20	20/20	Equal
Mean agent time	89.8 s	111.4 s	+24%
Mean total reported tokens	147,247	205,444	+40%
Mean uncached input tokens	20,508	30,047	+47%
Mean completed tool actions	9.6	20.4	+112%

For someone expecting a drop-in upgrade, this is a worse result on the measured efficiency criteria: the same accepted work took more time and tokens, despite explicit instructions to use the new tools.

All twenty attempts with the added tools followed the required workflow. Five of the 239 server calls failed.

Token figures are cumulative usage across model calls and include cached context. Billing was not measured.

Agent time includes MCP startup, tool use, and tests run by the agent. It excludes container and agent setup, and the independent verifier. Each attempt started a fresh tool server and language-server process, so these are results for fresh sessions rather than a server kept running throughout a working day.

When I compared matching attempts at the same task, the guided workflow was faster in 5 of 20 comparisons. The table below averages each task's two attempts per workflow:

Refactoring	Ordinary tools	Guided tools
Class rename across files	80.4 s	92.3 s
Rename one of two same-named methods	76.9 s	82.5 s
Function move with compatibility	104.4 s	133.6 s
Extract two classes into a module	80.8 s	122.1 s
Extract related helpers into a module	109.9 s	173.0 s
Extract a duplicate implementation	73.2 s	107.0 s
Extract a shared encoder	77.9 s	88.2 s
Extract a branch with edge cases	104.8 s	100.2 s
Inline a private helper	94.7 s	121.7 s
Simplify local control flow	95.2 s	93.7 s

The predeclared target was at least a 15% speed improvement without lower acceptance. It was not met.

Two repetitions per task give a limited view of run-to-run variation. These are descriptive results for one agent and model on ten curated tasks. Other coding agents, models, and Python projects may behave differently.

What Would Convince Me?

For any tool presented as a coding-agent upgrade, I want the authors to define "better," measure it, and turn the findings into tested guidance. People looking for a drop-in solution should not have to discover every trade-off themselves.

For a claim about speed, reliability, or token efficiency, I would look for:

The same tasks and starting code with and without the tool, with agent, model, backend, and versions recorded.
Repeated attempts, independent checks of the resulting code, and clear treatment of failures.
Complete-task time and token usage, including the work of using the tool.
Per-run records and runnable scripts, so someone else can repeat the comparison and inspect where the gains came from.
Tested best practices: which tasks benefit, when ordinary tools work better, and which agent instructions, backend, and setup produced the measured gains.

A tool may help on one task and add overhead on another. Publish both outcomes and the conditions behind them: repository size, model, backend, server lifetime, and instructions. If selective use is recommended, test that policy too.

Until that evidence is inspectable and reproducible, I do not find broad productivity claims trustworthy enough to base my workflow on.

Can They Actually Show an Improvement?

This experiment measured guided use on ten tasks with fresh tool servers. A larger repository, another language, a server that stays running, or a native IDE backend might produce a different result; those possibilities remain untested here. With both workflows passing every check, this comparison also cannot establish differences in maintainability or defects outside the tested cases.

For this workflow, the extra capabilities produced a slower result with more token usage. That is the risk of treating new tooling as an automatic upgrade: without reproducible evidence and tested best practices, you can make the same agent's workflow worse.

My question for the authors of coding-agent tools is straightforward: can you demonstrate an improvement in complete tasks, publish enough of the experiment for someone else to verify it, and show users when and how to get that benefit?

How has AI actually made your life better?

Leon — Sat, 03 Oct 2026 21:16:16 +0000

We spend a lot of time discussing new AI tools, comparing models, building things, and exploring what’s possible. I do this too.

But I want to hear more about what happens after the experimentation. What problem has AI helped you solve, and what difference did that make in your life?

I’ll start with some examples from my own experience.

Meals and Cooking

I use AI to plan meals around the ingredients I have, track what’s been used, and work out preparation instructions. It also orders groceries for the following week. It helps me organize batch cooking sessions that can cover multiple meals across several days.

Having the meal plan, grocery order, and preparation instructions fit together means I can focus on making the food, with much less figuring out what to eat, what to buy, and how to prepare it.

I also display the recipes on an e-reader that I can refer to while cooking.

The e-reader I use to follow recipes while cooking. Image edited with AI.

Keeping Up with the News

AI collects news from sources I’m interested in and filters out content I consider toxic, polarizing, or irrelevant. It can also turn that selection into a short morning audio briefing.

I used to finish reading the news pissed off. Being able to stay informed without carrying that frustration into the rest of my day has made a real difference.

Shopping

I use AI to find products, check prices and availability, look for discounts at checkout, place orders, and track deliveries.

Buying something takes much less of my attention. I can explain what I’m looking for, review the options, and have AI help handle the purchase and follow-up. I spend less time searching across stores, trying discount codes, and checking shipping updates myself.

Communications

I use AI to help manage conversations across email and messaging apps: contacting service providers, arranging appointments, understanding replies, drafting and sending messages, and following up. It brings together previous exchanges and relevant documents so we can continue a conversation with the right context.

I can ask where things stand and review a proposed response without having to reread old messages to reconstruct the situation. Much less of the correspondence and follow-through depends on me remembering who I was waiting for and keeping track of everything myself.

Work Management

I work with agents to organize and prioritize tasks, work out the next steps, and carry them out.

With agents helping plan and execute the work, more of my attention can go toward setting the direction, making decisions, and reviewing progress. The organizing, prioritizing, and follow-through no longer depend so much on me personally managing each step.

Monthly Finances and Paperwork

AI helps me work through my monthly financial routine: checking records, preparing transfers, distributing money across expense accounts and savings, and organizing invoices and documents. It also handles my tax reporting. I still review and approve the financial actions.

This used to take a few hours of my time every month. My part is now down to minutes—reviewing the results and approving the financial actions.

What About You?

Tell me about one problem AI has helped you solve. What was it like before, what do you do now, and what difference has it made?

It could be a recurring chore, a one-time problem, or something you couldn’t do before. Small improvements count too.

If you built something, I’d love to hear what it actually does for you now that you’re using it.

This post was written with AI assistance. The cover illustration was generated with AI.

How I Stopped Codex Agents from Hammering 1Password

Leon — Sat, 29 Aug 2026 17:11:00 +0000

One afternoon, every Codex task that needed a secret stopped at once. Not one agent. All of them. A tiny distributed-systems incident, staged entirely inside my laptop.

op service-account ratelimit delivered the diagnosis with admirable bureaucratic calm: the shared daily allowance was exhausted, and roughly 13 hours remained before reset. The agents were ready to work; 1Password had clocked out for the day.

The agents already had unattended access through a service-account token limited to the vault they needed—the behavior I wanted. There was no approval loop. The problem was that every agent contacted 1Password independently, and together they exhausted a shared limit none of them could see. They had achieved impressive teamwork, just in the least useful direction possible.

That changed the question from “how should each agent retrieve a secret?” to “why should each agent retrieve it at all?”

The replacement had to support:

several concurrent local agents;
unattended work after application startup;
project and non-project tasks, including dynamically created working directories;
an explicit environment-variable allowlist;
preferably no plaintext copy of the credentials on disk.

The answer that finally worked was almost disappointingly architectural:

Resolve secrets once at the application boundary. Let Codex forward only an explicit application-wide allowlist to agent commands.

I then took a surprisingly comprehensive tour of ways to move one secret from A to B before settling on two practical choices:

Put the actual values in ~/.codex/.env.
Keep the values in 1Password and start ChatGPT with op run, using an env file that contains only 1Password references.

These are alternatives, not layers. The first is wonderfully boring but stores credentials on disk. The second keeps 1Password as the only durable secret store and concentrates the unusual part at startup. I chose the second.

I tested this on macOS on 2026-08-27 with ChatGPT desktop/Codex app 26.820.71523, Codex CLI 0.150.1, and 1Password CLI 2.39.0. The undocumented behavior described below may change in later builds.

Why Per-Agent Vault Reads Failed

The original setup gave agents a service-account token limited to the vault they needed. That was unattended, easy to automate, and initially looked like exactly the right solution.

It also made every agent responsible for talking to 1Password. An agent could avoid repeating its own reads, but it could not share that work with the others. Every agent was responsibly optimized in private. Collectively, they were still eating the same allowance.

1Password documents a daily limit shared by all service accounts in an account. On individual and Families accounts, that allowance is 1,000 requests per 24 hours, and some CLI commands use more than one request.

That is why creating another token would not have rescued the day: the daily limit belonged to the account, not the token. It would have been a new tap connected to the same empty tank.

The incident was useful in one respect: it made the hidden dependency visible. My agents did not merely receive secrets from 1Password. They depended on 1Password being available and below quota throughout their work.

The Suspiciously Easy Option: `~/.codex/.env`

On the ChatGPT/Codex desktop build I tested, ChatGPT loads this file when it starts:

~/.codex/.env

It is an ordinary dotenv file:

EXAMPLE_API_TOKEN=<actual value>
EXAMPLE_WEBHOOK_SECRET=<actual value>

That is it. No sidecar. No broker. No daemon with a YAML file and opinions.

I verified the behavior with both a fresh ChatGPT launch and a fresh codex exec process, the non-interactive entry point to the same local Codex tooling. OpenAI's documentation lists the environment variables that Codex supports directly, but it does not document automatic loading of this file. I therefore treat ~/.codex/.env as observed behavior of the tested setup, not a permanent cross-platform contract.

Protect the file and never commit it:

chmod 600 ~/.codex/.env

Changes take effect only on the next full launch. Closing a ChatGPT window does not quit the application, however final the disappearing window may look.

This is the smallest setup and the easiest one to debug. Its price is equally simple: the credentials remain on disk in plaintext. File permissions reduce exposure, but the file is still a durable copy that must be protected, rotated, and kept in sync.

The Option I Use: Resolve References Once

I wanted 1Password to remain the source of truth without inviting it into every agent operation. The order matters: op run starts first, authenticates through 1Password desktop if needed, resolves the references, and then launches ChatGPT.

The runtime sequence has three stages:

Start op run with ~/.codex/.env-run, which maps environment-variable names to 1Password references.
op run contacts 1Password desktop. The desktop app unlocks or asks for startup approval if needed; op run resolves the references and launches ChatGPT with the resulting environment.
After ChatGPT starts, ~/.codex/config.toml decides exactly which variables Codex may forward to agent commands.

`.env-run` Is the Map, Not the Treasure

The file looks like this:

EXAMPLE_API_TOKEN=op://<vault>/<item>/<field>
EXAMPLE_WEBHOOK_SECRET=op://<vault>/<item>/<field>

The right-hand sides are references, not copied credential values. I still keep the file local, uncommitted, and mode 0600, because vault and item names can reveal useful metadata. A treasure map may not contain the treasure, but there is no reason to hand it out at the train station.

One Command, One Boundary

Quit ChatGPT completely, then run:

op run --env-file="$HOME/.codex/.env-run" -- /usr/bin/open -a ChatGPT

That is the whole launch mechanism. op run asks 1Password desktop to unlock or approve the startup request if necessary, replaces the references with their values, and starts ChatGPT inside the resolved environment. There is no second launcher or background service. The command does the interesting work once and then gets out of the way.

This assumes that the 1Password CLI's desktop-app integration is enabled and that the shell is not forcing a different authentication method. The op run documentation describes how secret references are resolved and passed to a child process as environment variables.

Once ChatGPT starts, its agents inherit the resolved environment through the policy below. They do not return to 1Password individually, so parallel work no longer multiplies vault reads.

The Other Half: `shell_environment_policy`

Getting values into the ChatGPT process is only half the job. config.toml still decides which variables Codex may forward to commands.

An include entry creates an allowlist. Once any include filter exists, variables that do not match one are removed. This is excellent for limiting the environment and slightly less excellent if you include two API tokens but forget that programs also enjoy having a PATH.

A practical baseline looks like this:

[shell_environment_policy]
inherit = "all"
ignore_default_excludes = true

[shell_environment_policy.filters]
"PATH" = "include"
"HOME" = "include"
"USER" = "include"
"SHELL" = "include"
"PWD" = "include"
"TMPDIR" = "include"
"LANG" = "include"
"LC_*" = "include"
"TERM" = "include"
"CODEX_HOME" = "include"
"EXAMPLE_API_TOKEN" = "include"
"EXAMPLE_WEBHOOK_SECRET" = "include"

The exact non-secret baseline depends on the commands you run. The goal is not to recreate your entire login shell inside Codex. Preserve the ordinary process context your tools need, then add credentials by exact name.

The current OpenAI configuration reference documents filters as the canonical mechanism and the older include_only array as legacy. It also documents ignore_default_excludes = true as the default; I keep it explicit because the example names contain TOKEN and SECRET, and I prefer the intent to be visible beside the allowlist.

Adding a value to .env or .env-run is therefore only half a configuration change. You must also include its name in shell_environment_policy. The secret source and the allowlist must agree on the name; computers remain stubbornly literal about spelling.

Because the environment belongs to the ChatGPT process rather than a project directory, this works for project and non-project tasks, including tasks whose working directories are created dynamically.

What I Tried Before Settling on This

1Password has many relevant features. Each made sense on its own. The awkwardness appeared when I asked for concurrent local agents, unattended work, a clear allowlist, and no plaintext credential copy at the same time.

Service-Account Token

This was the original design, not a rejected theoretical option. It worked until parallel agents exhausted the shared daily allowance. Making each agent more careful about its own requests did not coordinate the group.

Desktop CLI Calls from Agents

This avoided the service-account quota, but put a human back into the per-agent path. In my testing, approval appeared in a separate 1Password window without clearly identifying the requesting Codex task or downstream operation. I rejected that as a steady-state design; I was not operating that way before the quota incident.

1Password Environments

In my test, values in an Environment were separate stored values; putting an op:// reference inside one did not resolve it dynamically. The local dotenv destination used a named pipe rather than plaintext, prompted on first read, and relocked with 1Password. In concurrent tests, fresh Codex processes alternated between receiving all variables and receiving none. Excellent suspense; poor configuration delivery.

1Password Connect

Connect can centralize and cache vault access, which directly addresses repeated reads. It also meant deploying and maintaining an API container, a sync container, a credentials JSON file, an access token, a cache volume, ports, and lifecycle management. The Connect API includes create, replace, update, and delete operations, so I would also need to verify and enforce the token's effective permissions rather than assume a read-only boundary.

That can be the right architecture for an organization. For one desktop application, it was more infrastructure than I wanted to own. I wanted credentials, not a new on-call rotation.

1Password MCP or Agent Tool Calls

These are useful when the task is “operate on 1Password.” They do not transparently inject environment variables into arbitrary programs, and they keep password-manager calls inside the agent workflow—the dependency I was trying to remove. Useful tool, different plumbing.

The common problem was not that any one feature was bad. It was that “can this process receive the credential it needs?” kept acquiring quota state, concurrency behavior, human intervention, or a local server.

What This Solves—and What It Does Not

None of this turns environment variables into an enchanted security boundary.

The plaintext option keeps credentials on disk. The op run option avoids that extra durable copy, but depends on desktop authentication, requires starting ChatGPT from a terminal command, and concentrates trust in that launch step.

Both options place secrets in the ChatGPT process environment. shell_environment_policy does not authorize commands; it only filters the variables inherited by commands that Codex launches. A command that inherits an allowlisted credential can use it, so the configuration and the code being run still need to be trusted. Environment variables improve delivery and separation of concerns; they do not make an untrusted process trustworthy.

The allowlist is application-wide, not task-specific. Every task and subagent running inside that ChatGPT process should be treated as capable of launching a command that inherits every listed secret. It limits what can leave the application environment; it does not decide which agent gets which credential. True per-agent isolation would require separate processes or a credential broker, and neither preserves the simplicity of this setup.

In the build I tested, Codex added the names exposed to its command environment to the initial prompt, not their values. Instructions can therefore say, “To access the API, use the environment variable SOME_API_KEY.” The model can refer to that name in a command, and the program reads the value from its environment without the value appearing in the command itself.

That distinction is useful, but it is not a force field. An agent can still dump its environment or make a program log a credential. Avoiding those operations remains part of the model's and tool's responsibility.

Resolved values remain in the application's runtime environment until the app exits. Locking 1Password, changing a vault value, or removing vault access does not rewrite the copy already injected into a running ChatGPT process. Fully quit and relaunch through the chosen route to discard the old environment and receive current values.

Finally, ~/.codex/.env loading is observed rather than documented, and the 1Password-backed route depends on several products behaving as tested. A small non-secret startup check is worthwhile. It is nicer to discover a broken environment before an agent begins real work and develops opinions about the error message.

The Least-Worst Setup Wins

The important change was not a better retry policy. It was removing 1Password from the per-agent hot path.

Use ~/.codex/.env when the smallest possible setup matters most and a protected plaintext file is acceptable. Use launch-time injection when the secret store should remain the only durable source and one authentication step at startup is acceptable. In my case, that means op run with ~/.codex/.env-run.

This arrangement is not perfect. It trades per-agent vault access for an application-wide runtime environment, and it does not provide a distinct, least-privilege identity to each agent. It is simply the least cumbersome setup I have found that survives concurrent work without turning secret retrieval into a distributed-systems hobby.

Disclosure: I used ChatGPT Codex to help draft and edit this article. The experience, conclusions, and final publication decision are mine.

The unresolved question is how desktop agents and subagents should receive a well-attributed, least-privilege identity without persistent tokens, plaintext files, context-free approval windows, or a local platform team.

If you have a cleaner pattern—especially one that works for general-purpose desktop agents with dynamically created working directories—I genuinely want to hear how you handle it.