DEV Community: Bruno Baruffaldi

Caveman Saves Tokens by Doing Less, Not Just Saying Less

Bruno Baruffaldi — Sun, 02 Aug 2026 00:00:00 +0000

Caveman promises an appealing optimization for coding agents:

Same answers, 65% fewer output tokens. Brain still big. Mouth small.

The idea is simple. Models often produce unnecessary explanations, narrate their tool calls, and pad conclusions with filler. Remove that verbosity and the agent should become faster and cheaper without becoming less capable.

Coding agents can consume a surprising number of tokens while inspecting repositories, running commands, and correcting their own mistakes. A small skill that removes wasted language sounds like an easy efficiency win. But a coding agent operates through a trajectory: call a tool, interpret the result, and decide whether to continue. An instruction that changes how the agent communicates can also change how it acts.

To measure that effect, I ran 60 tasks across Claude Code and Codex CLI using SWE Bench Verified, Terminal Bench 2, and USACO tasks. Caveman reduced Claude Code token usage by 18% and its reported cost by 25%, but in Codex token usage fell by only 3.9%.

The most important result was not how many tokens disappeared, but where they disappeared from. With Caveman enabled, Claude Code performed 17% fewer steps and made almost 20% fewer tool calls, and in Codex hidden reasoning tokens fell by 14%. Caveman didn’t merely make the agents speak less. It changed how much work they performed.

And if the skill mostly works by making the agent do less, adjusting the reasoning effort might be a better choice. Claude Code exposes reasoning effort as a native setting, and in this experiment medium effort reached roughly the same Pass@3 as Caveman with high effort while costing about 20% less. A style instruction that arrives at a similar operating point indirectly is harder to justify when a direct knob is one flag away.

What Caveman Promises

Caveman is a skill for Claude Code, Codex, Gemini, and other coding agents. It instructs the model to avoid narration, drop hedging, and communicate in compact fragments.

Its README summarizes the intended behavior with a memorable distinction:

Brain still big. Mouth small.

The main benchmark compares ten standalone answers, including prompts such as explaining Git rebase, React rerenders, and Docker multistage builds. The reported average falls from about 1,200 output tokens to under 300, the headline reduction of 65%.

That result is plausible for those prompts. They ask for prose, and Caveman aggressively compresses prose. The problem is the extrapolation.

A single conversational answer and a repository level coding task have very different token distributions. In a coding agent, useful output is source code, patches, and exact error messages. Caveman leaves much of that unchanged.

The project itself acknowledges this distinction. Its documentation warns that Caveman only targets output tokens, adds input tokens of its own, and may become net negative when the original model is already concise. It also describes the honest evaluation control as Caveman versus a simple terse instruction, rather than Caveman versus an unconstrained baseline.

Those caveats are good. Caveman is free, open source, and doesn’t collect telemetry. The issue is that the headline claims are much broader than the evidence supporting them.

The Experiment

I evaluated five agent configurations:

Configuration	Agent	Treatment
`claude-baseline`	Claude Code with Sonnet 5	High effort
`claude-caveman`	Claude Code with Sonnet 5	Caveman injected at session start (high effort)
`claude-medium`	Claude Code with Sonnet 5	Medium reasoning effort
`codex-baseline`	Codex CLI with GPT 5.6 Terra	High effort
`codex-caveman`	Codex CLI with GPT 5.6 Terra	Caveman forcibly activated (high effort)

The benchmark contained 60 tasks: 20 from SWE Bench Verified, 30 from USACO, and 10 from Terminal Bench 2. Each task was executed three times under every configuration, for 900 trials in total.

The claude-medium configuration is particularly important. Comparing only baseline against Caveman mixes together two possible effects:

Caveman may compress communication.
Caveman may cause the model to spend less effort.

The medium effort arm provides a native way to reduce model effort without installing Caveman. It lets us ask whether the skill offers something beyond indirectly making the agent do less work.

Results

Claude Code showed substantial savings:

Metric	Baseline	Caveman	Change
Total tokens	1,595,199	1,302,271	18.4% lower
Output tokens	19,860	16,059	19.1% lower
Reported cost	$1.003	$0.758	24.5% lower
Steps	25.5	21.2	17.0% lower
Tool calls	24.6	19.7	19.7% lower
Agent prose characters	2,956	1,741	41.1% lower
Pass@1	81.1%	80.0%	1.1 points lower
Pass@3	91.7%	86.7%	5.0 points lower

Codex showed a much smaller reduction:

Metric	Baseline	Caveman	Change
Total tokens	418,852	402,402	3.9% lower
Output tokens	5,252	4,757	9.4% lower
Reasoning tokens	2,542	2,200	13.5% lower
Reported cost	$0.257	$0.249	3.2% lower
Steps	17.8	17.5	1.8% lower
Tool calls	11.8	11.5	2.7% lower
Pass@1	89.4%	85.0%	4.4 points lower
Pass@3	93.3%	90.0%	3.3 points lower

If we stop here, Caveman looks like a reasonable tradeoff for Claude and a poor one for Codex. Claude saves approximately one quarter of its modeled cost while losing little Pass@1, while Codex gives up considerably more performance for a 3.2% cost reduction.

But these aggregated numbers hide the most important part of the experiment.

The Wrong Denominator

Caveman promotes a reduction in output tokens. Coding agent cost, however, isn’t dominated by visible prose.

For Claude Code, the baseline token distribution looked like this:

Quantity	Share
Cache hit rate	97.8%
Output tokens as a share of all tokens	1.24%
Visible prose as a share of output tokens	3.7%
Visible prose as a share of all tokens	0.046%

Directly generated prose accounts for only about 0.05% of Claude’s reported session token volume.¹ A 41% reduction therefore removes only about 0.02% of the total tokens directly.

Shorter replies also make subsequent contexts slightly smaller, so the full mechanical saving from prose compression is larger than 0.02%. But even a deliberately generous fixed trajectory estimate puts that effect well below 1% of total token volume. Most output tokens are code, diffs, and commands that must remain exact. That’s nowhere near the observed 18% reduction.

The missing savings have to come from somewhere else. The trajectory data provides the answer: with Caveman, Claude took 17% fewer steps and made almost 20% fewer tool calls. Each removed turn avoided processing a large context again.

The expensive part of an agent is taking another turn.

Decomposing the Reduction

The total token count can be decomposed approximately into two factors:

total tokens = number of steps × tokens per step

Applying that decomposition gives the following:

Comparison	Total tokens	Steps	Tokens per step
Codex baseline to Caveman	3.9% lower	1.8% lower	2.2% lower
Claude baseline to Caveman	18.4% lower	17.0% lower	1.7% lower
Claude baseline to medium	37.1% lower	20.7% lower	20.6% lower

For Claude, approximately 92% of the token reduction is explained by taking fewer steps. Tokens per step barely changed.

Caveman certainly changed the writing style. Visible prose fell by 41%, and prose per step also decreased. But this compressed prose was too small to explain the session level savings. The savings appeared because the model ended its trajectory earlier.

That interpretation is also visible in the tool usage:

Metric	Claude baseline to Caveman
Steps	17.0% lower
Tool calls	19.7% lower
Shell calls	19.0% lower
Agent execution time	15.6% lower

A style instruction propagated into the action policy of the agent. This doesn’t automatically mean that the removed steps were useful. Agents can overinspect repositories, repeat tests unnecessarily, and continue reasoning after they already have a valid solution. Some trajectories should be shorter.

But it does mean that Caveman shrinks the brain along with the mouth.

The Control That Changes the Conclusion

The strongest argument against using Caveman isn’t the small drop in quality. It’s the claude-medium control.

Both configurations reached exactly the same Pass@3:

Claude Caveman: 86.7%
Claude medium: 86.7%

But Caveman used 30% more tokens and cost 23% more than medium.

Mean cost per trial versus Pass@3. Better configurations move upward and to the left. Claude medium reaches the same Pass@3 as Claude Caveman at ~20% lower cost.

The chart makes the relevant comparison visible. On this plane, a configuration is dominated when another configuration achieves at least the same quality at a lower cost. Claude medium sits directly to the left of Claude Caveman: same Pass@3, lower cost.

The difference remains after accounting for successful trials:

Metric	Medium	Caveman	Caveman overhead
Pass@1	77.2%	80.0%	2.8 points higher
Pass@3	86.7%	86.7%	no change
Tokens per resolved trial	1.30 million	1.63 million	25.3% more
Cost per resolved trial	$0.749	$0.874	16.6% more

The quality comparison between Caveman and medium is inconclusive at Pass@1: Caveman was 2.8 points higher, but the difference was not statistically significant. The efficiency comparison is much clearer. Both configurations reached the same 87% Pass@3, yet Caveman consumed 25% more tokens per resolved trial and cost 17% more.

This comparison changes the practical recommendation. Caveman can reduce resource use, but it does so by indirectly altering the agent’s trajectory. When Claude already provides a direct control for effort, using a global style instruction to reach a similar operating point is difficult to justify.

Why Codex Saved Less

Caveman reduced Claude token usage by 18%, but reduced Codex usage by only 3.9%. One possible explanation is that Codex had much less waste available to remove.

Before applying Caveman, the Codex and Terra combination already used substantially shorter and cheaper trajectories:

Baseline metric	Claude Code with Sonnet 5	Codex with Terra
Tokens per trial	1,595,199	418,852
Steps per trial	25.5	17.8
Tokens per step	62,600	23,500
Pass@1	81.1%	89.4%
Pass@3	91.7%	93.3%

Codex reported around 74% fewer tokens while obtaining slightly better resolution rates. This difference appeared at two levels: Codex took around 30% fewer steps, and each step processed around 62% fewer tokens.

That doesn’t prove that GPT 5.6 Terra is intrinsically more efficient than Sonnet 5. The comparison mixes the model with the scaffold, the tokenizer, and everything else that differs between two harnesses.

The defensible conclusion is narrower:

In this harness, Codex with Terra started from a much more token efficient trajectory than Claude Code with Sonnet 5.

This provides a plausible explanation for the different Caveman results: Claude offered a long trajectory that the skill could truncate, while Codex already operated closer to a floor.

Caveman Needs Waste to Remove

An exploratory analysis inside the Codex results supports this interpretation.

I divided the tasks into four groups according to their baseline token consumption. Caveman increased median consumption among the cheaper tasks and only produced savings among the more expensive ones:

Baseline consumption group	Median token change with Caveman
First quartile	27% higher
Second quartile	11% higher
Third quartile	13% lower
Fourth quartile	9.5% lower

This analysis shouldn’t be treated as causal. The groups mix datasets, contain relatively few tasks, and are defined using observed baseline consumption. Regression to the mean may explain part of the pattern.

Still, the result is consistent with a floor effect. When the original trajectory is already compact, Caveman has little redundant work to remove. Its additional prompt and behavioral changes may cost more than they save. When a trajectory becomes unusually expensive, Caveman has more room to terminate it early.

The 3.9% aggregate reduction in Codex was therefore driven by the expensive tail. For the median Codex task, Caveman didn’t save tokens.

This suggests a better mental model:

Caveman behaves like an indirect effort limiter whose effectiveness depends on how inefficient the baseline agent already is.

Quality

Caveman didn’t improve aggregate Pass@1 or Pass@3 in any of the main comparisons.

For Claude:

Pass@1: 81.1% → 80.0%
Pass@3: 91.7% → 86.7%

For Codex:

Pass@1: 89.4% → 85.0%
Pass@3: 93.3% → 90.0%

The direction is consistent, but the experiment isn’t large enough to establish the magnitude precisely.

Bootstrap confidence intervals computed at the task level all crossed zero:

Comparison	Pass@3 change	95% confidence interval
Claude baseline to Caveman	5.0 points lower	11.7 lower to 1.7 higher
Claude baseline to medium	5.0 points lower	11.7 lower to 1.7 higher
Claude medium to Caveman	no change	8.3 lower to 8.3 higher
Codex baseline to Caveman	3.3 points lower	10.0 lower to 3.3 higher

We found no evidence that Caveman improves task resolution. Every aggregate quality comparison moved in the negative direction, but the experiment was not large enough to estimate effects of a few percentage points precisely.

There is one statistically detectable behavioral effect. Codex reasoning tokens fell by 14%, with a 95% confidence interval from 4.6% to 20% lower. Caveman doesn’t leave reasoning untouched, whatever the README says.

Other Evaluations

A JetBrains evaluation tested Caveman on Claude Code using Harbor and SkillsBench.

It measured an 8.5% reduction in output tokens, far below the advertised 65%, with no detectable quality difference. Its first ten task evaluation had suggested savings near 30%; the number fell after expanding the benchmark. Its explanation was structural: coding agent output is dominated by code, diffs, tool invocations, and exact strings that Caveman can’t safely compress.

That result is compatible with this experiment, but the medium effort arm adds an important control. Even when Caveman reduces a trajectory without producing a detectable quality loss, it may still be an inefficient way to reach that operating point.

Other evaluations tested an even simpler question: does Caveman outperform asking the model to “be brief”?

Max Taylor evaluated 24 prompts with prompt specific quality rubrics and found Caveman in roughly the same token and quality range as a two word concision instruction.

Alex Rios described the problem as the Terse Control Fallacy: comparing a sophisticated compression technique against silence instead of comparing it against the trivial alternative.

The Brevity Preprint

The Caveman README references the preprint Brevity Constraints Reverse Performance Hierarchies in Language Models as evidence that shorter answers can sometimes improve accuracy.

The paper evaluated 31 models on 1,485 questions and identified 115 cases where smaller models systematically outperformed larger ones. The authors associate part of this behavior with overthinking: larger models sometimes elaborate beyond what’s useful and introduce additional opportunities for error. On this selected subset, constraining responses improved large model accuracy from 40% to 67%.

This is a meaningful result, but also a targeted one. The brevity intervention was evaluated on problems already exhibiting unusual scaling behavior, rather than across the benchmark as a whole.

There’s also an important difference in settings. The study evaluates standalone questions: no tools, no repository, no decision about whether to keep exploring. A coding agent must repeatedly decide what to do next.

The paper doesn’t validate Caveman as an agent optimization. At most, it shows that excessive elaboration can hurt on a selected group of standalone reasoning questions.

Per-Dataset Breakdown

The aggregate results also hide substantial variation across task types:

Dataset and agent	Token change	Pass@1 change	Pass@3 change
SWE Bench with Claude	28.8% lower	no change	5.0 points lower
USACO with Claude	14.9% lower	no change	3.3 points lower
Terminal Bench with Claude	4.1% higher	6.7 points lower	10.0 points lower
SWE Bench with Codex	5.2% lower	11.7 points lower	5.0 points lower
Terminal Bench with Codex	4.0% lower	3.3 points lower	10.0 points lower
USACO with Codex	1.5% higher	no change	no change

Each row compares that agent’s Caveman run against its own baseline, broken out by dataset.

SWE Bench with Claude was Caveman’s strongest case. It preserved Pass@1 while reducing token usage by almost 29%.

Terminal Bench with Claude was the opposite. Token usage increased while both success metrics fell, but that subset contained only 10 tasks, so a difference of 10 Pass@3 points represents a single task. One contrast in that subset produced a token ratio above 100% because of one unusual task.

USACO with Codex showed no quality degradation, but the baseline already reached 100% Pass@3. That subset had little room to distinguish the configurations.

The result depends on the agent, the model, and the dataset. Probably on the individual task too.

Limitations

This experiment has several limitations:

The runs used Harbor as the evaluation harness. The reported cost is modeled rather than billed because they used subscription authentication rather than API keys.
Timed out runs didn’t report token or cost metrics. The estimated savings should therefore be treated as approximate rather than exact.
Claude and Codex receive Caveman through different integration mechanisms. That reflects how the skill is distributed for those tools, but it means cross agent comparisons mix the skill with its delivery path.
The experiment needs more tasks to estimate small quality effects. Most tasks were solved in all three attempts or failed in all three, and only about ten of the sixty produced mixed results within a configuration, so three trials don’t provide three independent observations. With sixty tasks and few discordant pairs, differences around five percentage points remain uncertain.
The medium arm controls for cost and effort, but not for style. A stronger future design would include a plain instruction such as “communicate briefly, but do not change your exploration or verification behavior.”

Should You Use Caveman?

I’d use Caveman primarily as a style preference. If you find terse status updates and compact final answers easier to read, that alone may be a good reason to enable it.

Enabling it, though, does more than change the wording. Caveman reaches into the trajectory itself: how much the agent reasons, how many tools it calls, and when it decides the task is done. If you’re worried about quality loss, raising reasoning effort may compensate, but that spends back the tokens Caveman saved.

If the goal is reducing cost, I’d prefer explicit controls such as reasoning effort or context management. These mechanisms target resource usage directly and are easier to evaluate.

Caveman makes an agent more concise. Making it more efficient is a different claim, and this experiment doesn’t support it.

The Broader Lesson

The most interesting result isn’t specific to Caveman.

A system prompt is part of an agent’s policy.

Instructions about tone, confidence, or verbosity may look cosmetic, but they can influence more than the text an agent produces. Because the agent repeatedly conditions its next action on those instructions, a request for concision can also affect how long it explores, how many tools it uses, and when it decides that the task is finished.

That’s what happened here. Caveman did save tokens in Claude Code, but most of the reduction came from shorter trajectories. Tokens per step barely moved. Codex, which already started from a shorter and cheaper trajectory, had much less room for that effect.

This is also why explicit controls matter. When the goal is to trade reasoning effort for cost, a native effort setting targets that tradeoff directly. In this experiment, medium reasoning reached the same Pass@3 as Caveman while using fewer tokens and costing less.

The broader lesson is simple: instructions that look like presentation choices can become behavioral interventions once they’re placed inside an agent loop.

Future Work

There are several ways to extend this experiment.

The first is simply scale. Sixty tasks were enough to reveal clear behavioral differences, but not enough to estimate small changes in resolution rate precisely. A larger evaluation would reduce that uncertainty and make dataset specific effects easier to distinguish. It would also be interesting to include newer and harder benchmarks such as DeepSWE.

The second is model diversity. This experiment only covers Claude Code and Codex with one model family each. Repeating it with models from other families, such as DeepSeek, Kimi, or Muse Spark, would help determine whether Caveman’s effects depend on the underlying model. The large difference observed between Claude Code and Codex already suggests that its impact may depend strongly on both the model and the agent scaffold.

The most important control, however, would be a minimal terse control. Just something like be concise or be brief added to the system prompt. A stronger follow up experiment would therefore compare baseline, a minimal terse control, Caveman, and native effort controls across more tasks and model families. That would make it possible to separate style, effort, and agent behavior much more cleanly.

References

Caveman repository: https://github.com/JuliusBrussee/caveman

Caveman evaluation directory: https://github.com/JuliusBrussee/caveman/tree/main/evals

JetBrains evaluation: https://blog.jetbrains.com/ai/2026/07/speak-to-ai-agents-like-cavemen-tosave-tokens/

Max Taylor, I benchmarked caveman against two words: https://www.maxtaylor.me/articles/i-benchmarked-caveman-against-two-words

Alex Rios, There is a reason that caveman is a character from the past: https://alexriosme.substack.com/p/there-is-a-reason-that-caveman-is

Brevity Constraints Reverse Performance Hierarchies in Language Models: https://arxiv.org/abs/2604.00025

How Do AI Agents Spend Your Money? Analyzing and Predicting Token Consumption in Agentic Coding Tasks: https://arxiv.org/abs/2604.22750

The prose estimate converts visible agent text into an approximate token count. ↩

Training with scikit-learn, Deploying with the Right Runtime

Bruno Baruffaldi — Sat, 28 Feb 2026 00:00:00 +0000

Compiling scikit-learn Trees to Source Code: Deploying ML with the Right Runtime

Deploying a machine learning model is not the same problem as training it.

Training environments are built for flexibility. They optimize for experimentation, rapid iteration, rich APIs, and numerical breadth. Deployment environments optimize for something very different: latency, memory footprint, startup time, packaging simplicity, and operational cost.

In this post I use decision trees and random forests as a controlled simplification of a broader issue: how we deploy ML models in production. Trees are useful here precisely because their inference logic is easy to reason about. They let us isolate the systems problem without hiding behind heavy linear algebra or GPU acceleration. The point is not that trees are special. The point is that deployment deserves its own engineering decisions.

TreeCompiler is a small proof of concept built around that idea. It compiles scikit-learn DecisionTreeClassifier and RandomForestClassifier models into standalone Python or Go source code. No scikit-learn. No numpy. No scientific stack at inference time.

The goal is not to replace frameworks. The goal is to make explicit a systems principle: inference deserves its own runtime.

A Quick Reminder: How Trees and Forests Work

A decision tree recursively partitions the feature space using axis-aligned splits. At each internal node, a single feature and threshold are chosen to reduce impurity (for example using Gini or entropy). Inference is simply walking the tree from root to leaf by evaluating comparisons like x[j] <= threshold. A random forest is an ensemble of such trees trained on bootstrapped samples and random feature subsets. At inference time, each tree produces a prediction (or probability vector), and the forest aggregates them, typically by averaging probabilities (soft voting).

If you want a deeper refresher, the scikit-learn documentation provides a clear overview of the algorithmic details: https://scikit-learn.org/stable/modules/tree.html

The important part for this discussion is this: inference is just branching logic and basic arithmetic. There is no heavy linear algebra involved.

Training Runtime vs Inference Runtime

When we talk about inference cost, we usually think about raw compute: CPU cycles per forward pass, vectorized math, and numerical throughput. For tree-based models, however, the computation itself is small. A decision tree prediction is mostly a sequence of comparisons, and a random forest repeats that process across multiple trees before averaging the results. That means the largest cost often comes from the machinery around the model rather than from the model logic itself: importing large numerical libraries, deserializing pickled objects, allocating arrays, shipping container images with hundreds of megabytes of dependencies, and managing compatibility across Python versions and architectures.

Those costs make sense in a training environment, where flexibility, experimentation, and rich APIs matter. They make much less sense in an inference environment, where the model has already been trained and the runtime only needs to execute a fixed set of operations. TreeCompiler explores what happens when we separate those concerns. By turning the learned tree structure into explicit source code, inference no longer depends on a general-purpose ML framework to load and execute a serialized artifact. The deployed program contains only the branching logic required for prediction, which reduces startup work, dependency loading, object allocation, and packaging overhead. In compiled targets, it also gives the compiler a simpler program to optimize: deterministic control flow, constant thresholds, direct comparisons, and small functions instead of a dynamic model object interpreted through a training framework.

What the Compiler Actually Does

Scikit-learn stores a trained decision tree as arrays: left children, right children, feature indices, thresholds, and class counts per node. Prediction consists of walking those arrays until a leaf is reached.

TreeCompiler extracts that structure into an intermediate representation and generates explicit source code with nested if/else branches.

For a small tree, the generated Python looks like this:

def predict_proba(x):
    if x[2] <= 2.45:
        return [1.0, 0.0, 0.0]
    else:
        if x[3] <= 1.75:
            return [0.0, 0.91, 0.09]
        else:
            return [0.0, 0.02, 0.98]

The generated code has no imports and no runtime dependencies, only comparisons and simple arithmetic. For a random forest, TreeCompiler generates one function per tree and performs soft voting by averaging the predicted probabilities. Once inference is represented as explicit source code, the deployment shape changes: scikit-learn is no longer required at runtime, and the model is no longer a serialized artifact loaded by an external framework, but part of the executable itself. That shift directly affects latency, memory usage, packaging, and deployment complexity.

Benchmark Setup

To make the impact concrete, I benchmarked three deployment strategies on AWS Lambda, all using the same model:

RandomForestClassifier, 50 trees, depth 8, 4 classes, 20 features.

Each function was configured with 128MB of memory, which is the minimum allocation for AWS Lambda.

Deployment strategies:

scikit-learn inside a Docker image.
Python code (generated source, standard Python runtime).
Compiled Go (generated source compiled into a static binary).

Python and Go expose two different optimization surfaces. In Python, the generated tree code is still executed by an interpreter, so the runtime sees the model as bytecode to evaluate step by step. In Go, the generated source is compiled ahead of time into native machine code. That gives the compiler a chance to apply optimizations before the function ever runs: lowering comparisons and arithmetic into direct CPU instructions, choosing efficient register and stack layouts.

Once inference is reduced to pure branching logic, the runtime itself becomes the dominant factor.

Python vs Compiled Python (Same Language, Different Runtime Shape)

Metric	scikit-learn (Docker)	Python Compiled
Cold init	2.05s	450.8ms
Warm p50	22.4ms	1.8ms
Warm p95	30.5ms	15.7ms
Memory	201 MB	127 MB

Cold init measures the time required to initialize the execution environment before the first request. Warm p50 and warm p95 measure latency after the environment is already running: p50 captures the median request, while p95 captures tail latency near the slowest 5% of requests. Memory is the peak memory consumption observed during execution. Under those definitions, compiled Python changes the shape of the service even though the model is the same: cold initialization drops from 2.05 seconds to 450.8 milliseconds, roughly 4.5x faster, or about a 78% reduction in startup time. Warm p50 falls from 22.4ms to 1.8ms, over 12x faster, roughly a 92% reduction in median latency. Warm p95 drops from 30.5ms to 15.7ms, almost 2x faster at the tail. Memory decreases from 201MB to 127MB, a 37% reduction.

Those improvements do not come from changing the forest, the features, or the prediction rule. They come from changing what has to exist at inference time. The generated Python function still runs on the Python interpreter, but it no longer needs to import and initialize scikit-learn, load a pickled object graph, allocate NumPy structures, or carry a full scientific stack just to execute a sequence of branches. Even within the same language, replacing a framework-driven runtime with explicit prediction code produces structural gains in startup time, latency, and memory usage.

Compiled Python vs Compiled Go (Different Runtime Model)

Metric	Python Compiled	Go Compiled
Cold init	450.8ms	78.3ms
Warm p50	1.8ms	1.1ms
Warm p95	15.7ms	11.5ms
Memory	127 MB	21 MB

Moving from pure Python to compiled Go pushes the same generated model into a different execution model. Cold initialization drops from 450.8ms to 78.3ms, roughly 5.7x faster, or an 83% reduction. Warm p50 improves from 1.8ms to 1.1ms, about 1.6x faster, while warm p95 drops from 15.7ms to 11.5ms, improving tail latency by roughly 27%. Memory falls from 127MB to 21MB, an 83% reduction, or about 6x smaller. The warm latency gains are smaller than the scikit-learn to compiled Python jump because both versions already execute minimal prediction logic. The remaining difference comes from the runtime around that logic.

In the Python version, the generated code is explicit, but it still runs through the Python interpreter and Python object model. In the Go version, the same tree structure is compiled ahead of time into a static binary, so the executable can start with much less runtime machinery and the compiler can lower the generated comparisons, constants, and function calls into native code. For a workload made almost entirely of deterministic branches and simple arithmetic, that matters: there is little numerical work left to optimize, so startup cost, memory footprint, and runtime dispatch become the dominant factors.

Translating This Into Economic Impact

In serverless systems, latency and cost are tied to the same underlying factors: how long the function runs, how much memory is allocated, and how much work has to happen before the first request can be served. AWS Lambda charges per request and per GB-second, so reducing execution time matters, but the configured memory size matters just as much. This is why peak memory is not only an operational metric, but also a cost signal. A function that peaks around 21MB can safely fit inside the 128MB minimum allocation. A function that peaks near 201MB would normally need at least a 256MB allocation to leave a safe margin.

The cost impact becomes clearer if we turn the benchmark into a concrete traffic scenario. Assume 100 million invocations per month on x86 Lambda, excluding the free tier and excluding surrounding services such as API Gateway, CloudWatch Logs, storage, and networking. The request charge is the same for all implementations, so the interesting part is the duration cost. For the estimate below, the scikit-learn Docker version is modeled at 256MB because of its observed memory usage, while compiled Python and compiled Go are modeled at 128MB. Average billed duration is approximated as:

warm p50 latency + cold start rate × cold init time

Cold start rate	scikit-learn Docker	Compiled Python	Compiled Go	Go vs scikit-learn
0%	$29.33/month	$20.38/month	$20.23/month	31% lower
1%	$37.88/month	$21.31/month	$20.39/month	46% lower
10%	$114.75/month	$29.77/month	$21.86/month	81% lower

This table also shows why a latency chart alone does not fully explain the deployment impact. In the warm-dominated case, request charges become the floor, so the total bill cannot fall by 12x even though median latency does. The compute component does fall almost that much: in the 1% cold-start scenario, moving from scikit-learn Docker to compiled Go reduces duration cost from $17.88/month to $0.39/month, a roughly 98% reduction. Under burstier traffic, the difference is larger because cold initialization becomes part of the cost profile. Reducing cold starts from 2.05 seconds to 78.3 milliseconds changes both user-visible latency and billed initialization work.

The broader point is not that every tree model will save exactly this amount. The point is that deployment shape changes the cost equation. Removing the training stack from the inference path reduces package size, initialization work, runtime memory pressure, and execution overhead. For low-volume workloads, the dollar difference may be small because the request charge and free tier dominate. For high-volume or bursty workloads, the same structural changes can translate into meaningful reductions in billed compute and a lower need to overprovision memory.

Broader Lesson

This idea is not limited to trees.

In a previous post, I explored a similar separation between experimentation tooling and production inference when migrating PCA projections from scikit-learn to Faiss for scalable vector search. The theme was the same: use rich libraries for training and validation, but deploy inference using the runtime that best matches operational constraints. In that case, benchmarking showed roughly a 1.77× improvement in throughput simply by moving the projection step to a runtime designed for high-performance inference.

Trees simply make that principle easier to visualize because their inference logic is explicit and finite.

The broader lesson is straightforward.

Inference is a systems problem as much as it is a modeling problem. The largest optimizations are not just in the math. They are in choosing the right runtime for the job.

References

TreeCompiler repository: https://github.com/barufa/TreeCompiler

scikit-learn DecisionTreeClassifier documentation: https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html

scikit-learn Tree module overview: https://scikit-learn.org/stable/modules/tree.html

Related post on separating training and inference runtimes: https://stuckinalocalminima.com/blog/2025/sklearn-faiss

Amazon S3 Vectors: What It Is, Where It Fits, and the Gotchas Nobody Tells You

Bruno Baruffaldi — Fri, 05 Dec 2025 00:00:00 +0000

S3 Vectors adds native vector storage + ANN search to S3 via vector buckets and vector indexes. It’s cost‑oriented, elastic, and ideal for big volumes with sub‑second queries when throughput is moderate. It’s in preview and has sharp edges: hard Top‑K=30 , only float32 vectors. This post focuses on where it fits and the gotchas that actually matter in design.

Why S3 Vectors matters

Embeddings are everywhere (RAG, recommendations, search). Most teams start with a hosted vector DB and quickly hit two things:

Cost at scale : millions to billions of vectors add up.
Ops overhead : clusters, replicas, upgrades, and capacity planning.

S3 Vectors flips that default: store vectors in S3 with a purpose‑built index that you can query directly. You don’t manage nodes; you pay S3‑style pricing for storage + per‑request querying. If your workload tolerates low to mid QPS but needs large capacity and durability, this is a compelling baseline.

Key Concepts

Vector bucket : a special S3 bucket type for vectors.
Vector index : a logical container inside a vector bucket where you write/query vectors (all vectors in an index share dimension and distance metric ).
Vector : { key, values: float32[d], metadata }, where metadata can be filterable/non‑filterable.
Queries : approximate nearest neighbor ( ANN ) with optional metadata filters.

Key properties you’ll design around:

Dimension : 1…4096 (set at index creation).
Distance : Cosine or Euclidean (immutable per index).
Throughput : write RPS per index is low; design for batching and sharding.
Retrieve limit : Top-K is capped at 30 per similarity query — and there is no pagination mechanism for ANN queries.

Hard limits (and how to design around them)

Top‑K ≤ 30 (no pagination). If you need >30 another vector database could be more suitable.
Inserts : up to 500 vectors per PutVectors; 5 write RPS per index.
Data types : float32 only for vector values. If you pass other types, S3 Vectors converts to float32.
Immutable index schema : dimension, distance, and non‑filterable metadata keys can’t be changed.

Things AWS docs don’t spell out (yet)

CloudFormation / Terraform : no native resources to create vector buckets/indexes in preview — automate with CLI/SDK from CDK or CFN Custom Resources. See AWS re:Post confirmation (no CFN/Terraform support as of Sep 2025).
Dedup across indexes? Not documented. Pricing/examples suggest you pay per copy per index. If you want to “reuse” embeddings, prefer a single index with metadata/filters, or partition by namespaces/tenants.
One bucket with many indexes or many buckets? Best practice: one bucket with multiple indexes and access control via IAM/bucket policies. Use many buckets only for strong isolation (accounts/regions/KMS/lifecycle). See S3 Vectors best practices.

Cost intuition (rule‑of‑thumb)

S3 Vectors aims to lower total cost for large volumes with moderate QPS: you pay S3‑like storage + per‑request costs for index/query. No cluster provisioning , and batch ingest jobs are cheap.

When to choose S3 Vectors vs alternatives

Need	S3 Vectors	OpenSearch (Serverless/Managed)	SaaS Vector DB
Massive volume, low cost, low‑to‑mid QPS	Yes	Maybe (if QPS climbs)	Maybe
Ultra‑low latency, high QPS, complex filters	Can fall short	Yes	Yes
Simple ops (no clusters)	Yes	Less	Less
Bedrock tie‑in (embeddings/KB)	Yes	Indirect	Varies

S3 Vectors vs Upstash Vectors (serverless)

High‑level

Aspect	S3 Vectors	Upstash Vectors
Nature	Object store with native vector indexes (preview)	Managed serverless vector DB
Pricing model	Storage + per‑request query (see AWS announcement)	Per‑request pricing (e.g., $0.4/100K req in PAYG) + storage (e.g., $0.25/GB)
Query limits	Top‑K ≤ 30 , no pagination in `QueryVectors`	Client‑set `topK` with pagination via Resumable Query (cursor)
Filtering	Metadata filters; declare non‑filterable keys at index create	Metadata filtering ; namespaces for isolation
Data type	`float32` only for vector values	Upstash accepts numeric vector payloads (not advertised as multiple dtypes; equivalently treated as float arrays).
Multi‑tenancy	One bucket, many indexes (index per tenant) recommended	Multiple indexes + namespaces ; multiple DBs by project

Links

Implications

If you need scroll/infinite‑results UIs , Upstash’s resumable query is simpler. In S3 Vectors you’ll fan‑out across segments and re‑rank to bypass Top‑K=30. The fan-out could end in a clique with means some vectors would never be retrieved.
If your profile is huge corpuses + moderate QPS + lowest storage cost , S3 Vectors is attractive. For real‑time UX with richer ranking/pagination, Upstash is frictionless.
Multi‑tenant: both solve it, but S3 Vectors centralizes observability/logging per bucket; Upstash leans on namespaces and per‑index isolation.

Closing

S3 Vectors won’t replace every vector DB —but it changes the starting point. If your priority is cost and simplicity with large datasets and moderate QPS, start with S3 Vectors and scale out to OpenSearch or a dedicated vector DB only where the latency/query profile demands it.

Questions you want me to benchmark next? Throughput vs. shards, recall vs. filters, or evaluator design for reranking under Top‑K=30.

From scikit-learn to Faiss: Migrating PCA for Scalable Vector Search

Bruno Baruffaldi — Sat, 19 Jul 2025 00:00:00 +0000

Why using Faiss

Faiss is a high‑performance library for vector similarity search and related primitives (clustering, compression, linear transforms like PCA). t scales to millions–billions of vectors on CPU and GPU and it is a much faster implementation of PCA. In practice this reduces memory, latency, and Python overhead.

Why migrate PCA to Faiss?

If you’re already using scikit-learn for training, why switch to Faiss for deployment?

Training PCA in sklearn is convenient, but for the deployment implementation is slow.
Faiss offers faster, more efficient kernels for applying PCA at scale.
You can migrate a trained sklearn.PCA to a faiss.PCAMatrix without retraining.

Principal Component Analysis

PCA (Principal Component Analysis) is a linear dimensionality reduction technique. It projects data into a lower-dimensional space using the eigenvectors of the covariance matrix. You can check this video for a detail exaplanation.

Sklearn

We’ll focus on the essential operation of PCA: projecting vectors using transform().

Given :

X as the input data
skl_pca as the trained PCA object from sklearn

You can project X into the PCA-transformed space like this:

X_transformed = X @ skl_pca.components_.T

If whitening was applied during PCA fitting, you’ll also need to scale the output:

scale = xp.sqrt(skl_pca.explained_variance_)
min_scale = xp.finfo(scale.dtype).eps
scale[scale < min_scale] = min_scale
X_transformed /= scale

For reference, see the official implementation.

Faiss

In Faiss, after training a PCAMatrix, the transformation looks slightly different:

 X_transformed = X @ faiss_pca.A.T + faiss_pca.b

Here, A is the components matrix, and b is a bias vector.

Migrating from `sklearn` to `Faiss`

To migrate from a trained sklearn.PCA model to a faiss.PCAMatrix, you need to extract:

A: the transformed components matrix
b: the bias vector to match sklearn’s behavior

Depending on whether whitening is used:

Without whitening:

A = skl_pca.components_
b = -skl_pca.mean_ @ A.T

With whiten=True:

# escala cada fila de skl_pca.components_
A = skl_pca.components_ / sqrt(skl_pca.explained_variance_)[:, None]
b = -skl_pca.mean_ @ A.T

After these definitions we can get:

X @ A.T + b == sklearn.PCA.transform(X)

Code

Let’s create a small PCA model using the USPS digits dataset:

from sklearn.datasets import fetch_openml
from sklearn.model_selection import train_test_split
from sklearn.decomposition import PCA

X, y = fetch_openml(data_id=41082, as_frame=False, return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=42, test_size=3_000)

skl_pca = PCA(n_components=32, random_state=42)
skl_pca.fit(X_train)

Now, let’s migrate the trained model to Faiss:

import sklearn
import faiss
import numpy as np

def sklearn_pca_to_faiss(skl_pca) -> faiss.PCAMatrix:
    d_in = skl_pca.components_.shape[1]
    d_out = skl_pca.n_components_

    # Build A: rows are components; include whitening if requested
    if getattr(skl_pca, "whiten", False):
        scale = np.sqrt(skl_pca.explained_variance_)[:, None]
        A = (skl_pca.components_ / scale).astype(np.float32)
    else:
        A = skl_pca.components_.astype(np.float32)

    faiss_pca = faiss.PCAMatrix(d_in, d_out, 0.0, False) # eigen_power handled manually
    faiss.copy_array_to_vector(A.reshape(-1), faiss_pca.A)

    mean = skl_pca.mean_.astype(np.float32)
    faiss.copy_array_to_vector(mean.reshape(-1), faiss_pca.mean)

    # Choose bias so that X @ A^T + b == (X - mean) @ A^T
    b = -mean @ A.T # shape (d_out,)
    faiss.copy_array_to_vector(b.reshape(-1), faiss_pca.b)

    faiss_pca.is_trained = True
    return faiss_pca

faiss_pca = sklearn_pca_to_faiss(skl_pca)

Important: Use Faiss’s copy_array_to_vector utility to load arrays into Faiss structures. See this file for implementation details.

Validation

Always validate that the migration preserves results:

import numpy as np
import faiss
import sklearn
import time

X = np.random.randn(1_000_000, faiss_pca.d_in).astype(np.float32)
# Check over some random vectors
np.testing.assert_allclose(skl_pca.transform(X), faiss_pca.apply_py(X), atol=1e-5)
# Check over train vectors
np.testing.assert_allclose(skl_pca.transform(X_train), faiss_pca.apply_py(X_train), atol=1e-5)
# Check over test vectors
np.testing.assert_allclose(skl_pca.transform(X_test), faiss_pca.apply_py(X_test), atol=1e-5)
print("OK: sklearn == faiss")

# sklearn
t0 = time.perf_counter(); _ = skl_pca.transform(X); t1 = time.perf_counter()
# faiss
t2 = time.perf_counter(); _ = faiss_pca.apply_py(X); t3 = time.perf_counter()

print(f"sklearn.transform: {(t1-t0):.3f}s | {(X.shape[0]/(t1-t0)):.0f} vec/s")
print(f"faiss.apply_py : {(t3-t2):.3f}s | {(X.shape[0]/(t3-t2)):.0f} vec/s")
print(f"Speedup: {((t1-t0)/(t3-t2)):.1f}x")

See the complete code here.

Benchmarking on an Asus Zenbook 14 showed throughput rising from 803,290 vec/s to 1,418,897 vec/s — roughly a 1.77× improvement.

Conclusion

Migrating from scikit-learn to Faiss for PCA application is a straightforward optimization with real-world impact. You can keep sklearn for training and validation, then deploy the exact same projection using Faiss—boosting inference performance without retraining.

This method is simple, deterministic, and production-ready. And with just a few lines of code, you bridge the gap between experimentation and scalable deployment.

DVC + Many Files: A Strategy for Efficient Large Dataset Management

Bruno Baruffaldi — Sun, 30 Jun 2024 00:00:00 +0000

Implementing DVC at my workplace had been successful for most tasks, but dealing with a dataset containing 1 million images was tough. Uploading and downloading the dataset took several hours, so we had to use some clever tricks to streamline the workflow. In this post, I’ll share the valuable lessons we learned to help you avoid similar pitfalls.

Context

In an effort to improve data management during experiments, I decided to incorporate DVC. DVC (Data Version Control) is a tool that stores your data in remote storage and simplifies its versioning with Git. It hashes the data and saves the hash along with other metadata in a source-control file that is added to the repository. The actual data is then transferred to a corresponding folder in the remote storage. That way, DVC allows you to version your data by tracking a “pointer” with Git and restore the corresponding files with a single DVC command.

At first, everything worked smoothly. We could upload and download data without any issues. It was easy to track which version of our code was using which version of the data for running our experiments. However, things got complicated when we tried to work with datasets containing around 1 million images and beyond.

It turns out that DVC struggles with handling a large number of files (>200k)[1][2]. In our case, it took over 8 hours to upload the data with DVC and another 3 hours to download it. These times were clearly unmanageable for us. Once the download was complete, DVC needed an additional ~1.5 hours to check out the files. The conclusion was clear: we had a huge performance issue!

Problem

DVC has to check every file to make sure it’s the right one and avoid uploading the same file twice. This computation takes time and could be inefficiently implemented. On top of the overhead per request, there’s another problem that we ran into. It costs a lot of money to send 1 million requests to the storage account. So, we had to come up with a solution to make DVC work for us.

Solution

The solution we decided to implement was to zip the images in several archives. That way, we could reduce the number of files DVC had to check. The easiest way to do this was to use the zip command in Linux. We created a script that zipped the images in groups and uploaded them to the remote storage. This way, we reduced the number of files DVC had to check from 1 million to ~1000. This change significantly improved the upload and download times.

Here is the script we used to zip the images:

#!/bin/bash

# The directory structure should be as follows:
# imagenet/
# ├── n01440764
# │   ├── n01440764_10026.JPEG
# │   ├── n01440764_10027.JPEG
# │   ........
# ├── n01440765
# │   ├── n01440765_1000.JPEG
# │   ........
# .........

directory_path="imagenet"
max_processes=$(nproc --all)

# Function to handle ZIP operation using parallel
zip_directories() {
    find "$directory_path" -mindepth 1 -maxdepth 1 -type d | \
    parallel --progress -j "$max_processes" 'zip -qr {}.zip {} && rm -rf {}'
}

# Function to handle UNZIP operation using parallel
unzip_files() {
    find "$directory_path" -type f -name "*.zip" | \
    parallel --progress -j "$max_processes" 'unzip -q {} && rm {}'
}

# Main script logic
if [ $# -ne 1 ]; then
    echo "Usage: $0 [ZIP|UNZIP]"
    exit 1
fi

action=$1

case $action in
    ZIP)
        zip_directories
        ;;
    UNZIP)
        unzip_files
        ;;
    *)
        echo "Error: Unrecognized option '$action'. Usage: $0 [ZIP|UNZIP]"
        exit 1
        ;;
esac

exit 0

This solution has some points to consider:

I strongly recommend grouping the files in a way that makes sense for your project. In our case, we use some metadata (eg. labeling session) that naturally partition the dataset. This way, we could easily navigate through the dataset and find the images we needed without having to download/update the whole dataset.
Adding new data to the dataset should not modify the existing archives. This way, DVC will not store the same files twice. The new data should be added to a new archive. This is easy to achieve based on the metadata used to partition the dataset. By using the labeling sessions, you can add new zips without changing the existing ones.

Another solution was proposed at this article, which uses Parquet for partitioning the data instead of zipping. This clever solution is more efficient for some cases, but it requires more effort to implement and may not apply to CV datasets.

Summary

DVC is a great tool for managing data in machine learning projects. However, it struggles with large datasets containing a large number of files. To overcome this limitation, we zipped the images in groups and uploaded them to the remote storage. This change significantly improved the upload and download times as it reduced the number of files being tracked. I hope this post helps you avoid similar pitfalls when working with large datasets in DVC. If you have any questions or suggestions, feel free to leave a comment below. I’d love to hear from you!

DEV Community: Bruno Baruffaldi

Caveman Saves Tokens by Doing Less, Not Just Saying Less

What Caveman Promises

The Experiment

Results

The Wrong Denominator

Decomposing the Reduction

The Control That Changes the Conclusion

Why Codex Saved Less

Caveman Needs Waste to Remove

Quality

Other Evaluations

The Brevity Preprint

Per-Dataset Breakdown

Limitations

Should You Use Caveman?

The Broader Lesson

Future Work

References

Training with scikit-learn, Deploying with the Right Runtime

Compiling scikit-learn Trees to Source Code: Deploying ML with the Right Runtime

A Quick Reminder: How Trees and Forests Work

Training Runtime vs Inference Runtime

What the Compiler Actually Does

Benchmark Setup

Python vs Compiled Python (Same Language, Different Runtime Shape)

Compiled Python vs Compiled Go (Different Runtime Model)

Translating This Into Economic Impact

Broader Lesson

References

Amazon S3 Vectors: What It Is, Where It Fits, and the Gotchas Nobody Tells You

Why S3 Vectors matters

Key Concepts

Hard limits (and how to design around them)

Things AWS docs don’t spell out (yet)

Cost intuition (rule‑of‑thumb)

When to choose S3 Vectors vs alternatives

S3 Vectors vs Upstash Vectors (serverless)

Closing

From scikit-learn to Faiss: Migrating PCA for Scalable Vector Search

Why using Faiss

Why migrate PCA to Faiss?

Principal Component Analysis

Sklearn

Faiss

Migrating from sklearn to Faiss

Code

Validation

Conclusion

DVC + Many Files: A Strategy for Efficient Large Dataset Management

Context

Problem

Solution

Summary

Migrating from `sklearn` to `Faiss`