anicca

Posted on Jul 25

Coding Assistant Tests: Japanese 28% Shorter, English 3% Longer

#ai #codex #benchmarking #productivity

Coding Assistant Tests: Japanese 28% Shorter, English 3% Longer

The result

I ran eight software engineering tasks in Japanese and English under four reply conditions, three trials each.
Genshijin cut Japanese output by 27.5% against normal replies, but made English output 2.5% longer.
When provider-reported input and output usage were priced together, Genshijin cost a median $0.01016 more in Japanese and $0.013105 more in English per paired task.
My recommendation is narrow: try it for long Japanese explanations, but start with a simple concise instruction for short questions or English-heavy work.

Codex is OpenAI's software development assistant. Genshijin is an added instruction that asks for shorter replies; it does not replace the model. An API call counts the input and output in tokens, the small text units used for billing. This article is for developers who use that kind of assistant in Japanese or English and care about either reading volume or token-priced usage.

I kept pasting the same kind of question into Codex and watching the same thing happen. The answer was correct. The preamble was longer than the answer needed to be. A few sentences about context, a few extra bullets, then another paragraph that said almost the same thing.

Genshijin says it can remove much of that waste. Its README says the skill cuts token usage by about 75%. I did not want to turn that headline into a bill-saving claim, so I ran the same tasks 192 times.

The Japanese result was real but smaller. The English result went the other way. Once input usage was included, neither language produced a lower median API-equivalent cost with Genshijin.

What Genshijin actually changes

Genshijin does not replace the model. It is a style skill. Its instructions tell the model to remove Japanese honorifics, cushioning phrases, hedging, and redundant particles while keeping the technical content.

Caveman is the English comparison. It is also a system-prompt skill for shorter replies. Caveman's Honest Numbers page reports a 65% average output reduction, zero input reduction, and about 1,000 to 1,500 extra input tokens per turn from the injected skill rules. It also says short coding questions can become net-negative.

That distinction matters. A skill can make the answer shorter while making the request larger. The README's 75% figure describes output behavior. It does not mean a Codex subscription bill will fall by 75%.

The 192-run test

The task set covered inline object props in React, token-expiry boundaries in authentication middleware, PostgreSQL connection-pool limits, rebase versus merge, callback to async/await conversion, monolith versus microservices, a small PR security review, and a PostgreSQL lost-update race condition.

I wrote separate native Japanese and English prompts with the same meaning. I compared four conditions:

Condition	Extra instruction
normal	No style instruction
terse	"Answer concisely."
caveman-full	`$caveman full`
genshijin-normal	`$genshijin normal`

Every task and condition ran three times. I randomized the condition order with a fixed seed. Each cell used a fresh codex exec --ephemeral --json session with gpt-5.6-sol, reasoning effort low, and a read-only sandbox. The prompt prohibited tools and asked for only the final answer.

Was that fair? It was controlled for a within-benchmark comparison: the same eight task meanings, three repetitions, fixed model settings, fresh sessions, and randomized condition order were used for every condition. It was not a universal estimate. Eight tasks cannot represent every coding workload, and native Japanese and English prompts can still differ in wording.

For every completed turn I stored the returned usage record (turn.completed usage), including input_tokens, cached_input_tokens, cache_write_input_tokens, output_tokens, and reasoning_output_tokens. All 192 cells had a final answer and usage object. None were excluded for a tool event.

I used the current OpenAI pricing page for gpt-5.6-sol Standard short-context: $5.00 per million input tokens, $0.50 for cached input, $6.25 for cache writes, and $30.00 for output. These figures are API-equivalent estimates, not my Codex subscription bill.

Output got shorter in Japanese only

The median output counts were:

Language	normal	terse	caveman	genshijin
Japanese	708.5	281.5	636.5	483.5
English	363.5	217.5	430.5	397.0

The paired median, meaning the median difference between matching tasks and trials, was 59.3% shorter for terse in Japanese and 39.0% shorter in English. Genshijin reduced Japanese by 27.5%, but increased English output by 2.5%. Caveman reduced Japanese by 13.5% and increased English output by 26.1%.

That is why the concise control matters. In Japanese, a plain instruction to be concise beat Genshijin on output length. In English, neither skill beat the normal median in this task set.

The test cannot prove why the languages diverged. One plausible explanation is that Genshijin explicitly targets Japanese honorifics, cushioning, and redundant particles, while English has fewer of those patterns to remove. That remains a hypothesis because this design did not isolate language-specific style effects from prompt wording.

Two direct answers

Why did Japanese get shorter while English got longer? The answer is a hypothesis, not a causal finding: Genshijin explicitly targets Japanese honorifics and cushioning, while English offers fewer of those patterns to remove. The measured result is consistent with that explanation, but this test did not isolate the mechanism.

Were technical accuracy and important warnings preserved? I do not know for all 192 cells. I visually checked the rebase warning and the expiry comparison in two representative outputs; I did not score correctness for every answer.

The representative example was machine-selected

Picking the best-looking before and after would be easy and misleading. I selected the Genshijin trial whose reduction was closest to the language median.

For Japanese, the selected task was rebase versus merge, trial 1. The normal answer used 531 output tokens. Genshijin used 375, a 29.4% reduction.

The normal answer explained that shared branches should use merge, personal branches can use rebase, and --force-with-lease is safer than --force. Genshijin kept the same decision and warning in fewer words. The win came from compression, not from removing the important caution.

For English, the selected task was the token-expiry boundary bug, trial 3. The normal answer used 99 tokens and Genshijin used 100. Both showed now >= expiresAt; Genshijin added one token's worth of explanation.

That example does not prove that Genshijin always expands English. It is simply the cell closest to the measured English median. Every output and the selection proof are saved in the run artifact.

Input usage changes the verdict

Japanese output got shorter, but the combined usage estimate did not get cheaper.

Language	Condition	Median input tokens	Median API-equivalent cost
Japanese	normal	24,230	$0.05599
Japanese	genshijin	26,854	$0.083215
English	normal	24,219.5	$0.0296105
English	genshijin	26,826	$0.0424305

The paired median cost difference was +$0.01016 in Japanese and +$0.013105 in English. It is a paired statistic, so it will not equal the simple subtraction of the rounded table medians.

The median added uncached input was 2,875.5 tokens in Japanese and 2,609 in English. At $30 per million output tokens, that input overhead needs about 479 saved output tokens in Japanese and 435 in English to break even under an uncached-input calculation. The measured output saving was 196.5 Japanese tokens and negative 9 English tokens.

I did not pretend cached input was always zero. The provider-reported cache split varied across ephemeral sessions. That makes a fixed claim such as "this skill always adds exactly 1,200 tokens" unjustified here. The defensible claim is narrower: under these conditions, the saved output did not offset the recorded usage and pricing, and the paired cost went up.

What the test says about quality

All 192 outputs were non-empty, contained a language signal, and avoided a deterministic marker that the model had run a tool. Those are artifact checks, not a blind quality pass.

The selected examples preserved the dangerous part of the rebase answer and the correct comparison operator for token expiry. That is an observation from the saved outputs, not proof of 100% technical accuracy. A full quality claim needs an independent judge that hides the condition names and scores correctness, completeness, code preservation, safety, and readability. Token counts cannot stand in for that judge.

So I cannot conclude that technical accuracy and warnings were preserved across all 192 cells. The evidence supports only those two representative checks.

Who should use it

I would try Genshijin for long Japanese design explanations, code reviews, and research notes. It did make Japanese output shorter, and its rules target the kinds of padding Japanese model replies often contain.

I would not call that a cost saving by default. In this gpt-5.6-sol API-equivalent calculation, the median cost increased in both languages.

For short questions, start with "Answer concisely." The terse condition cut Japanese output to 281.5 tokens and English output to 217.5, and its paired cost fell in both languages. It also adds less machinery to the request.

For English-heavy work, do not import the Japanese result by assumption. Genshijin increased English output by 2.5% in this eight-task set. Caveman increased it by 26.1%. Your own language mix, prompt length, and cache state matter.

If you use a subscription, an API-equivalent cost estimate is not the same as a lower monthly bill. I did not measure reading time here, so I keep that possible benefit separate from token-price savings.

My verdict

The README's approximately 75% figure is not a universal Codex discount. In this 192-run test, Genshijin cut Japanese output by 27.5%, increased English output by 2.5%, and increased the paired API-equivalent cost in both languages.

Use it selectively for long Japanese replies, then measure the work you actually do. For short questions, English-heavy sessions, or token-priced calls, try a simple concise instruction first. The boring answer is the useful one here: measure before adding a skill.