Built All Tools Proposed by AI and Measured Again

!

This article is reprinted from I Started Using Claude Code.

Measuring a box connected to all the tools I made with a stopwatch and a scale

In the previous article, when I gave the same task to Codex using my custom AIShell and regular Codex, the token usage decreased by 25.86%. I thought it seemed effective. However, since it was only the result of 3 tasks with 9 trials, I couldn't be certain.

This time, I increased the features and expanded the benchmark to measure again. To start with the results, the token usage decreased by 52.44%. That's an improvement from last time, more than I expected.

Built All the Convenient Tools

When increasing features, I asked Claude. Based on my development environment, what tools would be convenient. I got many suggestions.

At first, Claude suggested implementing just a few and seeing how it goes. But I'm mostly a hobbyist solo developer, running AI behind my main job, so I don't need to worry about the risks of the development results. I decided to build all of them.

Current AIShell Features

At the time of the previous article, there were 5 features.

• Return the current state of the working folder and changes since the last time all together
• Read only the necessary amount from multiple files
• Search a specified range
• Execute builds and tests, return important diagnostics
• Read the necessary range from saved full output

List all the features added this time.

• Wait for changes: Wait until AIShell detects file changes or build completion. While waiting, the AI doesn't need to issue check commands.
• Run in background and view when needed: Keep builds and tests running in the background while the AI proceeds with the next task. Retrieve intermediate progress and results when needed.
• Cancel running processes: Cancel a background process midway and finish it without leaving anything behind.
• Batch change multiple files: Apply changes to multiple files at once. Before applying, check if the files assumed as the basis for the edit have changed; if they have, stop.
• Show impact range of changes: Return which source files and which tests are affected by changing this file. Also trace impacts from build dependency records. For dependencies that cannot be traced, report them as unknown.
• Persist workspace memory: Even after restarting AIShell, remember the previous state and return changes that occurred while it was stopped as a diff.
• Compare branches and worktrees: Return differences between branches or changes occurring in a worktree all together.
• Understand detailed Git state: Report states with mixed staged and unstaged changes, or file renames, while preserving the identity of changes.
• Remember project configuration: Remember and return what the build command and test command are for this project. If configuration files like package.json change, relearn.
• Execute multiple searches in batch: Accept multiple search queries at once and return results with priority given to changed files and tests.
• Semantic search: In addition to textual matching, trace connections between functions and variables to search. Immediately after editing, treat old analysis results from before the edit as outdated.
• Format diagnostics: Format diagnostics from builds and tests into a fixed structure per format. If the format is broken, report it as broken.
• Reuse verification results: When repeating the same verification, use the previous result if the input hasn't changed. If the input has changed, re-execute.
• Suggest tests to run: Based on the changes, suggest which tests should be run now. You can either stop at the suggestion or wait for approval before executing.
• Search and compare past execution results: Search across multiple saved run outputs. Compare two runs and show the warnings that appeared and disappeared.
• Recovery operations: Check AIShell's own status and open the management app. These two operations remain available when settings are insufficient or when the management app has stopped AI operations.

Among these, four—Wait for changes, Run in background and view, Batch change, and Impact range—were added as new tools. The recovery operations also became two tools: status check and opening the management app. The remaining features were incorporated into the existing five tools. The number of tools visible to the AI has increased from 5 to 11.

We'll look at how effective each one was by task later.

Increased Benchmarks to 32 Tasks

I created 32 tasks based on scenarios that occur in actual development. Tasks include waiting for build completion, applying changes to multiple files, and searching for causes from past execution results. Each task was given to regular Codex and AIShell three times. Success was determined mechanically based on conditions set for each task.

Token Decreased by 52.44%

The overall results are as follows.

Measurement item	Regular Codex	AIShell	Change
Tasks with 3 successes (out of 32)	18	26	+8
Tokens per successful task	1,763,452	838,768	52.44% decrease
Median time	51.2 s	42.0 s	17.8% decrease
95th percentile time	238.1 s	106.4 s	55.3% decrease

The tokens per successful task is the total including those used in failed trials divided by the number of tasks with 3 successes. This is an improvement from last time's 25.86% decrease, and the number of successful tasks also increased.

Token reduction rate by task. 70% reduction in understanding project configuration again and waiting scenarios, 30% increase in broken diagnostic judgment

From here, we'll look at what was built for each feature and what happened in which tasks. I'll also write about features that didn't work as is.

Results of the 4 New Tools

Wait for Changes

Regular Codex keeps sending check commands while waiting, so all those checks become tokens and time. AIShell notifies when a change occurs.

Regular Codex repeats check commands, AIShell requests once and waits for notification

This was the feature with the biggest difference among the 32 tasks. In tasks that wait for external edits, tokens decreased by 75–78% and time was reduced by over 90%. Even in tasks that detect when change notifications have stopped and switch to re-investigation, tokens decreased by 68%.

Run in Background and View When Needed

This feature had two tasks. One task extracts the first failure from a running build and reports the exit code when finished. The other task cancels a running process and finishes it without leaving anything behind. Regular Codex failed all 6 times. AIShell succeeded 5 out of 6 times. Before considering efficiency, these were the kinds of tasks that couldn't be accomplished through a shell.

Batch Change Multiple Files

Before applying, it checks whether the files assumed as the basis for editing have changed; if they have, it stops.

There were three tasks. Regular Codex succeeded 5 out of 9 times, while AIShell succeeded all 9 times. Comparing tasks where both succeeded, tokens decreased by 33–40%.

Show Impact Range of Changes

In tasks that trace direct dependencies, Regular Codex succeeded 0 out of 3 times, while AIShell succeeded all 3 times. In tasks that show impacts from build dependency records, it was 1 vs 3, with a 49% token reduction.

There was also a task where AIShell failed. It was a task to report "I don't know beyond this point" when untraceable dependencies are mixed in. Regular Codex succeeded 2 times, while AIShell failed all 3 times. There are still gaps in how unknown parts are reported.

Results of Extensions to Existing Tools

Persisting Workspace Folder Memory

AIShell remembers the previous state even after a restart and returns changes that occurred while it was stopped as diffs. For restoration after restart, tokens decreased by 44%. For detecting edits during downtime, tokens decreased by 32%.

Git State Awareness

Features for comparing branches and worktrees were added as tools. For understanding states with mixed file renames, tokens decreased by 63%. For understanding worktree changes, tokens decreased by 51%. In the task of accurately reporting states where staged and unstaged changes are mixed, Regular Codex failed all 3 times, while AIShell succeeded all 3 times.

Project Configuration Awareness

This feature remembers what the build and test commands for this project are and returns them. It re-learns when package.json changes. For re-learning after configuration changes, tokens decreased by 77%. For scenarios where the remembered content could be used directly, tokens decreased by 46%.

Search and Read

In tasks that stream multiple search queries at once, tokens decreased by 54%. In search tasks that trace connections between functions and variables—including the requirement not to return stale analysis results right after edits—Regular Codex succeeded 0 out of 6 times, while AIShell succeeded 5 times.

There was also a failure here. In the task of reading only as much as fits within a specified amount and returning the rest as a continuation, both failed all 3 times.

Around Build and Test Execution

In the task of formatting diagnostic output from execution results into a fixed structure, success was 1 vs 2, with tokens 42% lower. In the task of proposing which tests should be run based on changes and then executing them after waiting for permission, Regular Codex succeeded 0 out of 6 times, while AIShell succeeded 5 times.

Ineffective tasks also cluster here. In the task of deliberately diagnosing corrupted diagnostic data as broken, AIShell used 30% more tokens than Regular Codex. The feature that reuses previous results when the same check is repeated showed almost no token difference. In scenarios where the information is already available locally, the overhead of going through AIShell remains.

Search Past Execution Results

AIShell saves the complete output of executions. For searching for errors across multiple executions, tokens decreased by 48%. For comparing two execution results and listing which warnings appeared and disappeared, tokens decreased by 46%. Regular Codex starts by running the command again to recreate the output, which accounts for this difference.

Continuing the Question from Last Time

The previous article started with the question, "With AI, do we even need a shell and terminal?"

The results this time show that a connection bypassing the shell used fewer tokens, less time, and succeeded in more tasks. At least within the scope of this benchmark, AI produces better results without using a shell.

Next, I will verify whether the same effects appear in everyday development.