DEV Community: Andrii Volynets

Can a Better Model Be a Worse Thinking Partner?

Andrii Volynets — Fri, 09 Oct 2026 12:51:10 +0000

I first noticed something missing while reviewing saved discussions of Microsonya, my Telegram summarization project.

One summary placed a present-day developer's linguistic research in the seventeenth century. The original statement concerned when a word entered English. Somewhere in the summary, the date changed owners: a property of the word became a property of the person discussing it.

An assistant analyzing the error imagined the developer conducting research under Louis XIV. The scene made the broken attribution immediately visible.

Another summary turned a speaker's description of himself as a senior developer into a separate senior. The assistant called it having “materialized a separate senior.” The summary was no longer compressing the conversation. It was hiring additional staff.

I remembered these explanations because they gave me a way to recognize the next error. Later, some saved alternatives seemed more orderly and restrained. They still answered the question, but sometimes lacked the connection that changed how I understood the problem.

I initially called that missing quality personality. Too broad. A model can add jokes and imitate a familiar voice without contributing anything useful.

What I meant was useful reframing: preserving the facts, finding another representation of their structure, and returning with something that helps explain or inspect the problem. The useful part isn't the detour. It's coming back carrying something.

That raised a question beyond style: could an evaluation approve an answer while missing the loss of a useful way to think? A metric cannot complain about something it was never taught to notice. Worse, if we stop measuring and describing a behavior, we may eventually lose the vocabulary to ask for it back.

The archive motivated the question; it cannot establish a regression between model generations. The examples were selected, metadata was incomplete, and contexts changed. So I built a small test.

The Result, Up Front

The result contradicted my expectation. Prioritizing efficiency did not reduce judged useful reframing in either run. In the complete Gemini 3.7 Flash run, Efficiency produced 14/18 joint passes against Neutral's 11/18.

Five response pairs looked like possible evaluation blind spots. But all five were ties on the six conventional scores, not cases of strict improvement. Worse for my original hypothesis, several answers conveyed nearly the same diagnostic idea while the reframing evaluator marked one as useful and the other as not.

I did not establish that a better-scoring answer became a worse thinking partner. I found a problem one level earlier: how do we validate the metric meant to detect the quality our existing metrics miss?

The public Kaggle benchmark contains the frozen task and exported evidence.

What I Benchmarked

This pilot looks at the same answer through two lenses: conventional quality and useful reframing. “Better” means better on specified evaluation criteria, not universally more intelligent.

A useful reframing must preserve the task's relationships, add a perspective beyond restating them, and make a mechanism, consequence, or diagnostic step easier to see. It need not be funny, metaphorical, or novel to the world. It needs to be useful relative to the task.

Consider two explanations of a fabricated person in a summary:

The summary turns the speaker's self-description into a separate person.

Treat the summary like an entity registry: it inserted a new record where it should have updated an attribute. Check whether other roles have also become extra people.

These are illustrations I wrote, not sampled model outputs. The second offers a transferable diagnostic, if the registry framing actually helps. The first may be better when the user wants only a correction. That's why the benchmark also includes literal controls: if the task demands exact JSON or a configuration line, an unsolicited analogy is a defect, not a gift.

Tasks and evaluators

The benchmark has four development tasks and eight evaluation tasks: six analytical cases and two literal controls. All scenarios are fictional; saved private discussions inspired the failure mechanisms but were not published as test data.

Analytical case	Relationship the answer must preserve
A historical date migrates to a present-day speaker	A word's history belongs to the word
A professional role becomes an extra person	A self-description belongs to its speaker
A coordinator's fee is mistaken for a repair warranty	Scheduling and repair responsibilities differ
Prevented tickets make a team's count look worse	Prevented work disappears from a throughput count
Retries expand a small per-call bill	Attempt count differs from completed-job count
Duplicate runtime modules break reactive tracking	Matching names do not imply shared state

Each task supplies required facts and forbidden claims to the evaluators, not to the candidate. No model answer or preferred analogy is provided.

One evaluator scores factual reasoning, clarity, task completion, instruction compliance, relevance, and concision. It does not receive the reframing rubric or condition labels. It can reward a helpful analogy for clarity; the two dimensions are not assumed to be independent.

The second evaluator judges useful reframing, identifies the supporting passage, and explains what it adds. A vivid but misleading metaphor should fail. A plain, useful diagnostic can pass.

Here, R = 1 means an evaluator judged the reframing useful, not that a human learned more from the answer. Both assessments run separately, using one fixed evaluator model distinct from the candidate models. A shared integrity gate checks facts, critical errors, completion, and exact output contracts.

The reported analytical readouts are integrity pass, reframing among integrity-valid responses, and joint pass (both). Literal controls are scored separately. Missing evaluations stay visible; a failed evaluator call is not evidence of a wrong answer.

The blind-spot test

For two integrity-valid answers, A and B, let C₁ through C₆ be the conventional scores. The candidate blind spot is:

C_j(B) ≥ C_j(A) for every conventional criterion j
R(A) = 1 and R(B) = 0

Every tracked dimension is tied or improved, yet useful reframing seems to disappear. The dashboard is green. Something worth keeping may be gone.

The strongest instance would include at least one strict improvement in C. Ties are weaker: the scoring scale may simply be too coarse or already at its ceiling. An improved average is not enough either, since greater concision could mask worse clarity. The report checks the full vector and separates ties from strict gains.

Even a flagged pair is only an inspection target, not proof. It needs human validation. And the longer-term concern remains hypothetical: if selection repeatedly optimizes only C, a useful behavior outside C might become less common without lowering the recorded score. This pilot does not test that optimization process.

Four prompting conditions

Each candidate receives the same tasks under four conditions:

Condition	Added instruction
Neutral	Answer accurately and clearly
Tone	Use natural dry wit where appropriate
Mechanism	Seek a useful connection that preserves the task's relationships
Efficiency	Prioritize accuracy, directness, relevance, and concision; avoid speculative detours

Efficiency does not ban analogies. That would make the outcome trivial. The output contracts and word budgets stay the same.

Each run attempts 8 tasks × 4 conditions × 3 generations = 96 responses, with two evaluations requested per response. The main comparison is Efficiency versus Neutral within the same candidate model. The report also examines Mechanism and Tone, records response lengths, and compares all integrity-valid Neutral–Efficiency pairs within each task. Because pairs reuse generated responses, their counts are descriptive, not independent observations. Any uncertainty estimate resamples whole tasks. The conditions are different prompting packages, not isolated causal variables.

Models Tested

On October 9, 2026, I ran two candidates against the same frozen task:

google/gemini-3.7-flash, automatically selected by Kaggle during task creation: 96/96 scored.
google/gemini-3-flash-preview, used in a follow-up: 96 generated, 84 scored, 12 evaluator quota errors.

The fixed evaluator was google/gemini-3.1-pro-preview. Neither candidate run had provider errors. Another 16 development responses checked integration and are excluded from the results.

The original configuration also lists gpt-5.6-sol and gpt-6-sol, but neither was run. These are two exploratory candidate runs, not a designed model-generation comparison. The frozen kaggle-benchmarks 0.6.1 adapter recorded SDK defaults of seed 0, temperature 0, and reasoning None; these do not guarantee cross-provider determinism.

Findings

Run 1: Gemini 3.7 Flash

All 96 responses were scored without evaluator errors. On the six analytical tasks:

Condition	Integrity	Reframing given integrity	Joint pass
Neutral	18/18	11/18 (61.1%)	11/18 (61.1%)
Tone	18/18	18/18 (100%)	18/18 (100%)
Mechanism	18/18	18/18 (100%)	18/18 (100%)
Efficiency	18/18	14/18 (77.8%)	14/18 (77.8%)

All 24/24 literal controls passed integrity and exact-output requirements. Efficiency showed more, not less, judged reframing than Neutral.

Yet the pairwise search flagged five potential blind spots among 54 valid Neutral–Efficiency cross-pairs, spanning two tasks. All five were conventional-score ties. Three pairs reused one Efficiency response in E01; two reused another in E03. Five flags are not five independent findings.

What the flagged answers actually show

An AI-assisted inspection reviewed all five pairs without changing their recorded scores. This was not an independent human audit.

In E03, the Neutral answer said:

The service stops being valuable once the logistical phase concludes and technical repair performance becomes the focus.

Efficiency said:

The service stops being valuable once the coordination phase concludes and the repair begins, as the coordinator provides no technical repair-quality warranty on the mechanic's labor.

Both distinguish coordination from repair by the same functional boundary. The evaluator called the first useful reframing and the second literal prose. Another E03 flag showed the same issue. Neither answer is beyond criticism: both risk overstating when coordination stops being valuable. These are weak examples of lost reframing and stronger examples of rubric inconsistency.

Two E01 flags also involved similar descriptions of a date being attached to the wrong speaker. The remaining pair was more revealing. Neutral described a summary that:

flattens a nested sentence—in which the developer learned yesterday about an earlier date—by fusing the main subject directly to the date from a subordinate clause.

Efficiency described the error as:

cleanly blending the developer's action with the historical date

Neutral offers a more explicit grammatical diagnostic. Yet both received the maximum conventional vector (4, 4, 4, 4, 4, 4), while reframing was scored R = 1 versus R = 0. The answers were 155 and 150 words, so this pair is not explained by a large length difference.

It's a plausible difference worth studying, not a validated loss. Both answers identify the misattribution, conventional scores were at ceiling, and no reader study established that one explanation taught more than the other.

Run 2: Gemini 3 Flash Preview

The follow-up generated 96 responses, but evaluator quota failures left 84 scored. Twelve missing assessments were not retried or replaced.

Condition	Analytical scored	Integrity / attempted	Reframing given integrity	Joint / attempted	Analytical evaluator errors
Neutral	17/18	17/18	12/17 (70.6%)	12/18	1
Tone	17/18	14/18	14/14 (100%)	14/18	1
Mechanism	15/18	11/18	10/11 (90.9%)	10/18	3
Efficiency	15/18	13/18	13/13 (100%)	13/18	3

Four more evaluator errors affected literal controls; 20/20 scored controls passed. Attempted-response rates above count missing judgments as no observed pass, not as incorrect answers.

The pairwise search found zero potential blind spots among 36 integrity-valid Neutral–Efficiency cross-pairs. Efficiency had 13 observed joint passes versus Neutral's 12, a +5.6 percentage-point difference. The paired fixture-bootstrap 95% interval was −27.8 to +38.9 points. With only six analytical tasks and uneven missing assessments, that is not evidence of a reliable condition effect. Conditional reframing rates also refer to different integrity-valid subsets.

What Changed My Mind

I expected an efficiency-oriented instruction to suppress useful detours. Neither run showed that aggregate pattern. More importantly, none of the five flagged pairs met the strongest test: an improvement on at least one conventional criterion combined with lost reframing. All were ties, and several exposed disagreement over how to recognize the same idea.

Tone produced a second surprise. The dry-wit instruction received judged reframing in 18/18 analytical answers in the complete run and 14/14 integrity-valid answers in the follow-up. That does not establish that wit improves reasoning. It raises a testable possibility: a style instruction might sometimes elicit a useful way of representing the problem, rather than merely decorating the answer. Or the evaluator might be rewarding the style. This pilot cannot separate those explanations.

There's a recursion here. Suppose conventional quality is measured by C, and we notice that it omits something we value. We add a reframing score R, creating (C, R). Now we have to ask whether R measures that value or only a convenient approximation of it. Optimizing R too soon may create a new blind spot instead of repairing the old one.

The experiment did not demonstrate a historical regression, an effect of preference tuning, or a decline in intellectual independence. Such claims would require matched model versions, contexts, and direct evidence about the intervention. Otherwise a benchmark about attribution errors would end by making one of its own.

The immediate lesson is narrower and more useful: finding a missing dimension is not enough. The instrument used to detect it needs validation too.

Useful perspectives are not the same as varied wording

AnaloBench evaluates analogical reasoning. My question is narrower: whether an answer adds a useful representation while preserving its factual constraints. Research on length bias in LLM evaluators is a reason to retain the raw answers and compare lengths. And work on output diversity cautions against equating less surface variation with less effective semantic diversity.

This pilot judges responses, not what readers learn or what repeated optimization would erase. Those remain separate experiments.

My Benchmark

The public benchmark includes the frozen evaluation task, executable notebook, task fixtures, rubrics, exports, and reporting script. The results here use the full attempted grid, including failures, rather than the separate scalar displayed in Kaggle's task interface.

Eight evaluation tasks make a pilot, not a leaderboard. The fixtures are public, the reframing judgments are subjective, and the evaluator may infer a condition from an answer's style. AI assisted drafting, implementation, and qualitative inspection; the numerical results are from the exported runs. There was no independent human validation of the flagged pairs.

I began with a complaint about personality. The more useful question concerns what our definition of quality allows us to lose.

This benchmark is a way to look for that loss, and to let the evidence tell us whether it is there.

Why let Looked 7x Slower Than var: Closure Contexts and GC Thresholds in V8

Andrii Volynets — Mon, 03 Aug 2026 12:43:31 +0000

The first benchmark looked convincing: when creating a large number of closures, a loop using let was sometimes 6–8× slower than one using var.

That makes for an easy headline:

let is much slower than var.

The problem is that the two programs do not have the same semantics.

for (var i = 0; i < N; i++) {
  callbacks.push(() => i);
}

All callbacks share one binding. After the loop, every function returns N.

for (let i = 0; i < N; i++) {
  callbacks.push(() => i);
}

Each iteration has its own binding. The callbacks return values from 0 through N - 1.

The first program preserves one shared mutable value. The second preserves N independent values. The useful question is therefore not “how expensive is the let keyword?” but:

What memory does V8 allocate when every escaping closure genuinely needs its own captured value, and can that memory predict the GC timing step before it is measured?

Separating syntax from semantics

The experiment used six controls:

Case	State captured by the closure	Result
`capturedVar`	one shared `var i`	`[N, N, …, N]`
`capturedLet`	a separate `let i` per iteration	`[0, 1, …, N-1]`
`factoryVar`	a factory parameter	`[0, 1, …, N-1]`
`copiedVar`	`const value = i` inside a `var` loop	`[0, 1, …, N-1]`
`noCaptureVar`	the counter is not captured	identical callbacks
`noCaptureLet`	the counter is not captured	identical callbacks

capturedLet, factoryVar, and copiedVar have the same required semantics despite different source shapes. capturedVar is intentionally not equivalent.

This distinguishes two explanations:

If the keyword is intrinsically expensive, a difference should remain without capture.
If independent escaping state is expensive, the three correct implementations should retain similar heap topology.

Experimental design and provenance

The measurements were made on August 3–4, 2026 with Node.js 25.2.0, V8 14.1.146.11-node.13, and Windows x64.

The original object measurements are in results/raw-results.json. The quantitative GC work is split into explicit training and holdout artifacts:

results/threshold-calibration.json and results/threshold-analysis.json: five training semi-space settings and the regression;
results/growth-diagnostic.json: max-only young-generation telemetry;
results/prediction-plan.json / results/prediction-result.json: the first frozen holdout, including its failed prediction;
results/prediction-plan-2.json / results/prediction-result-2.json: the revised, still out-of-sample holdout;
results/barrier-topology.json, results/barrier-constant.json, and results/barrier-constant-analysis.json: the SMI/reference control;
results/equivalence.json and results/equivalence-analysis.json: the application-like TOST.

Every threshold decision used five one-trial fresh processes. A grid point counted as crossing when at least three of five runs contained an attributed minor GC. A majority-directed binary search located a 2,000-closure bracket; the reported threshold is its midpoint. Holdout predictions were serialized before the corresponding result file existed, and the runner refuses to overwrite a result.

A configured maximum is not the current semi-space

--max-semi-space-size is a maximum, not a request to begin at that size. The Node CLI documentation says exactly that. v8.getHeapStatistics().total_available_size is also the wrong diagnostic: here it was about 4.3 GB because it describes the whole heap.

The experiment instead recorded all fields returned by v8.getHeapSpaceStatistics() for new_space and new_large_object_space before and after every trial. Node also warns that the availability and interpretation of these spaces can change with V8 versions, so no single field is silently renamed “the actual semi-space.”

The max-only diagnostic makes the growth visible. These are medians from three fresh capturedLet(250000) processes:

Configured maximum	`new_space.space_size` before	after	Minor GCs
2 MiB	3.75 MiB	4.00 MiB	20
4 MiB	7.00 MiB	8.00 MiB	10
8 MiB	6.75 MiB	9.50 MiB	5
16 MiB	5.75 MiB	23.25 MiB	3
32 MiB	6.75 MiB	23.00 MiB	3

A “32 MiB semi-space” did not begin with 32 MiB committed, and two processes with different caps could begin in very similar states.

For the threshold regression, both --min-semi-space-size=S and --max-semi-space-size=S were set. V8 still committed pages lazily, so telemetry was retained, but this removed the most obvious max-only ambiguity. The regression and holdouts apply to this exact protocol; they are not a formula for an arbitrary already-running Node process.

Thresholds scale linearly with the configured semi-space

The measured first-GC brackets were:

`S`	`capturedLet` threshold	`capturedVar` threshold
2 MiB	41,000	101,000
4 MiB	55,000	135,000
8 MiB	95,000	189,000
16 MiB	177,000	317,000
32 MiB	335,000	579,000

Each midpoint has a grid half-width of 1,000 closures. Ordinary least squares with an intercept gives:

capturedLet:
  N_threshold = 17,750 + 9,907.26 × S_MiB
  slope SE = 111.20 closures/MiB
  slope 95% CI = [9,553.36, 10,261.15]
  R² = 0.99962

capturedVar:
  N_threshold = 66,833 + 15,916.67 × S_MiB
  slope SE = 212.55 closures/MiB
  slope 95% CI = [15,240.25, 16,593.09]
  R² = 0.99947

With S in bytes, the reciprocal slope estimates effective nursery bytes per iteration:

capturedLet: 105.84 B/iteration  (95% CI 102.19–109.76)
capturedVar:  65.88 B/iteration  (95% CI  63.19–68.80)
difference:   39.96 B/iteration  (95% CI  35.26–44.66)

The confidence intervals use five fitted midpoints (df = 3) and treat them as observed responses; they do not add the ±1,000 grid uncertainty. The difference interval uses a delta-method calculation that treats the two slope estimates as independent. R² is strong evidence of linear scaling in this controlled protocol, not a universal law of V8 heaps.

An independent heap method predicts the same extra bytes

The threshold fit above did not use heap-snapshot sizes. A GC-only pilot selected broad search brackets, but only fresh threshold runs entered the fit.

Now compare its 39.96 B estimate with the heap graph. After forced full GC, 10,000 live closures produced:

Case	Closures	Unique direct contexts	Closure `self_size`	Context `self_size`	`heapUsed` increase/item
`capturedVar`	10,000	1	64 B	40 B	72.03 B
`capturedLet`	10,000	10,000	64 B	40 B	112.02 B
`factoryVar`	10,000	10,000	64 B	40 B	112.02 B
`copiedVar`	10,000	10,000	64 B	40 B	111.95 B
`noCaptureVar`	10,000	1	64 B	48 B	72.01 B
`noCaptureLet`	10,000	1	64 B	48 B	72.02 B

capturedLet therefore retains one additional 40-byte system / Context per item. The process-level delta independently agrees:

112.02152 B - 72.02624 B = 39.99528 B per callback

The threshold-only estimate was 39.96 B, just 0.04 B (0.1%) below the snapshot result, and 40 B lies inside its 95% confidence interval.

The estimands are not identical: one is effective traffic at a nursery threshold and the other is retained self_size after full GC. Their numerical agreement supports a specific explanation: the main differential allocation in this code shape is the one additional context.

The sampling allocation profile is directionally consistent but less exact. Its main-stack difference was 44.83 B/item, about 12% above 40 B. That is acceptable for a sampled profile; it should not be presented as an exact object-size measurement.

This claim remains version- and shape-specific:

In V8 14.1, for this escaping-capture pattern, every per-item value was represented by a distinct 40-byte context.

It is not a JavaScript language guarantee.

A predictive wall-time model

The threshold equation alone predicts only whether a step occurs. To predict wall time, the training runs fitted three coefficients after GC attribution:

M_let(N) = 0.0472 + 0.0000423015 × N ms     (R² = 0.854)
M_var(N) = 2.4996 + 0.0000208576 × N ms     (R² = 0.834)
G_let,2(N) = -2.6180 + 0.000116553 × N ms   (R² = 0.760)

M was fitted only from fresh runs with no observed GC. G_let,2 was fitted from runs with exactly two attributed minor collections. The snapshot-constrained threshold model was:

N̂_let(S) = 15,577 + S_bytes / (64 B + 40 B)
N̂_var(S) = 61,038 + S_bytes / 64 B

The intercepts were estimated from the five training thresholds; the 64 B closure and additional 40 B context came from the independent snapshot.

The first holdout falsified part of the model

The first frozen holdout used an unmeasured 12 MiB setting:

predicted let threshold: 136,567
regression cross-check:  136,637

The plan predicted no minor GC at 130,000, two at 145,000, and no GC for capturedVar(145000). Across 21 fresh processes per target:

Target	Predicted minor GCs	Observed	Predicted wall	Observed mean
`let`, 130k	0	0 in 21/21	5.55 ms	4.78 ms
`let`, 145k	2	1 in 21/21	20.46 ms	10.74 ms
`var`, 145k	0	0 in 21/21	5.52 ms	4.94 ms

The byte model correctly placed the boundary and predicted the direction of the timing jump. The event-count submodel failed: this V8 state produced one minor GC, not two, so the wall-time magnitude was overpredicted by almost 2×.

That failure is part of the result. It shows why semi-space growth and post-collection state cannot be hidden inside the phrase “GC time.”

A revised holdout predicted the step out of sample

After that failure, the two-event rule was restricted to the power-of-two protocol represented by all five training settings. A second plan was frozen for the never-measured (min=max=64 MiB, N=630k/680k) workload—an extrapolation beyond the 32 MiB training maximum:

snapshot-constrained threshold: 660,855
unconstrained regression:       651,815

Before running it, the plan predicted:

Target	Minor GCs	Wall time
`let`, 630k	0	26.70 ms
`let`, 680k	2	105.45 ms
`var`, 680k	0	16.68 ms

The 21-process holdout produced:

Target	Observed minor GCs	Observed mean wall
`let`, 630k	0 in 21/21	23.89 ms
`let`, 680k	2 in 21/21	106.36 ms
`var`, 680k	0 in 21/21	17.21 ms

For the above-threshold let target, the wall-time error was 0.9%. The predicted jump was 78.75 ms versus 82.48 ms observed, a 4.7% error. The predicted let/var ratio was 6.32× versus 6.18× observed, a 2.3% error.

This is the missing out-of-sample check: snapshot bytes plus a known semi-space setting and N predicted the GC side of the boundary and approximately how large the wall-time step would be before that workload was run.

It is still a local model. It predicts the first step for this fresh-process protocol, Node/V8 version, and code shape—not arbitrary GC histories.

GC attribution must be delayed

Node delivers GC PerformanceEntry objects asynchronously. Reading the observer immediately after a timing window can report zero even when collection occurred inside it.

The corrected workers:

save every trial’s start and end time;
allow pending entries to arrive on later setImmediate turns;
match each entry by its own startTime and duration;
record detail.kind, so only minor events define the threshold.

This procedure is implemented in src/gc-trial-worker.mjs. The event timestamps, durations, and young-space telemetry are retained per run rather than summarized away.

The residual is not the pure cost of a binding

After attributed GC time is removed, a mutator-time difference remains. Allocation profiles cannot decompose CPU time into context allocation, initialization, closure linking, generated code, tiering, and barrier paths.

One candidate can at least be bounded experimentally. The original captured value is an SMI, which does not require a heap-reference store. A matched control used either one prebuilt SMI or one prebuilt boxed object, outside the timed window, through the same factory path.

Heap snapshots verified identical topology in both arms: 10,000 closures, 10,000 unique direct contexts, and 40 B per context. The timing protocol used 30 independent processes per arm, nine GC-free trials per process, and the process median. Before seeing the data, practical equivalence was defined as a boxed/SMI ratio in [0.90, 1.10].

geometric mean SMI:    2.045 ms
geometric mean boxed:  2.028 ms
boxed / SMI:           0.992
90% CI:                [0.906, 1.085]
Welch TOST p-value:    0.038

The confidence interval lies inside the equivalence bounds, so the tagged-reference initialization path is equivalent to the SMI path within ±10% in this kernel. It does not explain the residual at a practically large scale here.

This does not measure “all write barriers.” The context is newly allocated, so V8 may skip or fast-path a generational remembered-set update. A promoted context storing a young object would be a different experiment. The conclusion is deliberately narrow: pointer-valued context initialization in this code shape is not a remaining >10% explanation.

“Overlapping distributions” is not equivalence

The original application-like medians—3.63, 3.60, and 3.23 ms—were accompanied by overlapping intervals. That supports only “no stable multi-fold difference was established.” It does not prove equality.

The replacement experiment used 60 fresh processes. Each process created and drained 100,000 correct callbacks; all six execution orders were repeated ten times. The estimand was the paired mean log wall-time ratio. Equivalence was fixed in advance at ±10%—the largest difference this microbenchmark would call practically interchangeable and far below the original multi-fold claim—with two primary comparisons and Bonferroni-adjusted alpha = 0.025 per comparison.

Comparison	Geometric mean ratio	95% CI	TOST p	Equivalent?
`factoryVar / capturedLet`	0.872	[0.710, 1.072]	0.618	No
`copiedVar / capturedLet`	0.870	[0.727, 1.040]	0.648	No

The formal test did not establish ±10% equivalence. The data remain compatible with a moderate advantage for the alternatives, and GC occurrence still varied between otherwise balanced fresh processes. The defensible claim is therefore weaker than “they are the same”:

This experiment found no stable multi-fold advantage among the semantically correct implementations, but it did not establish practical equivalence within ±10%.

That distinction is exactly what a TOST is for.

Why `var` is not an optimization

Replacing let with one shared var removes thousands of contexts by removing the requirement to preserve thousands of values:

capturedLet(10000): [0, 5000, 9999]
capturedVar(10000): [10000, 10000, 10000]

That is not a faster implementation of the same program. It is a different program.

A factory or local copy preserves the behavior, but it also preserves the main allocation cost:

for (var i = 0; i < N; i++) {
  const value = i;
  callbacks.push(() => value);
}

The optimization target is not the keyword. It is the retained topology: how many closures escape, how many independent environments they require, how much state they retain, and whether allocation occurs in one latency-sensitive batch.

What the experiment establishes

For this Node/V8 version and code shape:

the first-GC threshold scales linearly with configured semi-space under the controlled protocol (R² > 0.999 for both cases);
threshold arithmetic independently estimates 39.96 additional B/iteration, while the heap graph measures one additional 40 B context;
a frozen second holdout correctly predicts 0 → 2 minor GCs and 106.36 ms wall time from a 105.45 ms point prediction;
constant SMI and boxed captures are equivalent within a predeclared ±10% bound for this initialization path;
the application-like data do not establish ±10% equivalence among the three correct source forms.

It does not establish that let is generally slow, that retained bytes always equal allocation traffic, that --max-semi-space-size is the current young-generation size, or that every V8 state will produce the same number of scavenges.

The final lesson is more specific—and more useful—than the original benchmark headline:

Independent semantic identity requires independent state. In this V8 pattern, that state costs one 40-byte context per escaping closure; enough contexts move the program across a predictable GC boundary, where a moderate allocation difference can become a multi-fold wall-time step.