Token-Level Filtering Costs 0.15 to 0.34 ms per Token on CPU

#llm #performance #opensource #python

Most guardrail benchmarks compare detection quality. This one measures the tax: how much time does token-level filtering add to each decoding step?

The answer on CPU is 0.147 to 0.343 ms per token for 100 to 10,000 banned patterns. The per-token cost is nearly flat in pattern count. What scales is the one-time automaton build.

Method

synthetic deterministic tokenizer, vocabulary padded to 50,257;
random logits, sequence length 32;
batch 1 and 8;
200 timed calls after warm-up;
CPU only, PyTorch 2.12.1, Python 3.14.8;
no GPU number is claimed.

python3 benchmarks/benchmark_processor.py \
  --patterns 100,1000,10000 --batches 1,8 --iterations 200

Results

Patterns	Build	Batch 1 mean	Batch 1 p95	Batch 8 per item
100	0.4 ms	0.147 ms/token	0.165 ms	27 us
1,000	4.3 ms	0.170 ms/token	0.207 ms	32 us
10,000	40.8 ms	0.343 ms/token	0.429 ms	96 us

How to read it

The inference path is O(1) per token with respect to pattern count. The automaton, failure links and danger mask are built once, then each step applies a precomputed mask. That is why 100 and 1,000 patterns cost almost the same per token, and why 10,000 patterns cost roughly twice as much rather than a hundred times more.

Build time is the real scaling cost. Forty milliseconds for 10,000 patterns is a startup cost, not a per-request cost, as long as the processor is built once and reused. Rebuilding it per request would dominate everything else.

Batching amortizes Python and tensor overhead. At batch 8, the per-item cost falls to 27 to 96 microseconds.

Do not convert milliseconds into a percentage without measuring your model step. 0.343 ms is about 1 percent of a 30 ms decoding step and about 0.3 percent of a 100 ms step. The percentage belongs to your workload, not to the filter.

What is not measured

detection recall, false positives or resistance to paraphrased attacks;
GPU performance;
end-to-end latency including network, sampling and framework overhead;
the effect of a penalty on model quality.

An older changelog mentioned a vendor figure of roughly 0.001 ms per token on GPU. We removed it from the current documentation because it is not reproducible here. If you need a GPU number, run the script on the target card and publish the JSON next to the claim.

Reproduce

git clone https://github.com/Resk-Security/resk-logits
cd resk-logits
pip install -e ".[dev]"
python3 benchmarks/benchmark_processor.py \
  --patterns 100,1000,10000 --batches 1,8 --iterations 200

The raw JSON lands in benchmarks/results.json. The full methodology is in BENCHMARKS.md.

DEV Community