Half Your Prefill Compute Can Go to 1% of Requests

A capacity spreadsheet has a row for tokens per request. Mine says 1,536. That number is a lie, and it is a lie in a specific direction: it under-describes the requests that actually cost you the node.

Here is the same statement in a form you can check. Take a request-length distribution with a Pareto tail of index α = 1.2 and a 256-token floor. The median request is 456 tokens. The mean is 1,536. The 99th percentile is 11,900. The top 1% of requests, by token count, carries 46.4% of all prefill tokens.

Ten thousand requests out of a million spend half the compute. A plan built on the mean cannot see that, because the mean is computed from the same tail it is supposed to warn you about.

Two systems that already measured this

I did not invent the statistic. Hurricanes and earthquakes have been counted this way for decades, and the numbers are uglier than most serving teams expect. The concentration figures below are dataset work by Бакуго Кацуки.

System	Top 1% share	Implied Pareto α	Gini
Hurricane ACE, top 1% of storms	7.63% of total ACE	2.27	0.28
Hurricane ACE, top 13.5% of storms	50% of total ACE	1.53	0.49
Earthquake energy, top 1% of quakes	90% of total energy	1.02	0.96 (reported 0.98)

One labeling note, because a reader has already caught it: the Gini in the earthquake row is mine, derived from the two reported concentration points as an implied α = 1.02, not a count over a raw catalog. Бакуго's own figure comes from 81,772 events, M5.0+, 1976-2024, log energy. When a two-point inference and a catalog disagree by 0.02, the catalog is right and the inference is the cheaper thing to say.

Two things are worth staring at.

First, the earthquake row. The top 1% of earthquakes holds 90% of the energy. That is not a tail; that is the distribution wearing the tail as a disguise. A capacity plan for earthquakes would be a plan for the top 1% and a rounding error for everything else.

Second, the two hurricane rows disagree. The top-1% share implies α = 2.27; the top-13.5% share implies α = 1.53. If ACE were a clean Pareto, those would be the same number. They are not, which means the concentration of hurricanes is not a law with a parameter. It is a measurement you have to take at the scale you care about.

That matters for anyone tempted to import an α from someone else's workload. α is a thermometer, not a constant of nature.

The same statistic on serving load

I do not have a public trace of LLM request lengths, so I am not going to pretend these next numbers are measured. They are computed from the model, with the parameter stated, and the script at the bottom reproduces every one of them. Run it against your own histogram and the table becomes yours.

α	Top 1% share of tokens	p99 / mean	mean / median
1.10	65.8%	5.98	5.86
1.20	46.4%	7.74	3.37
1.50	21.5%	7.18	1.89
2.00	10.0%	5.00	1.41
2.27	7.6%	4.25	1.32

Note the middle column. p99/mean is not monotonic in α; it peaks around α = 1.2 and then falls, because the mean itself is inflating as fast as the percentile. If you are using p99/mean as your heaviness metric, you cannot tell α = 1.2 from α = 2.0. Top-1% share separates them by a factor of 4.6.

The mean does not converge

This is the part that should change how you sample.

For α below 2, the variance of a Pareto is infinite. The sample mean is still consistent in theory, but the number of samples needed to get close is not the number anyone collects. I simulated 400 independent samples at each size and asked how often the sample mean lands within 10% of the true mean:

α	n = 1,000	n = 10,000	n = 100,000
1.20	11.5%	20.2%	41.5%
1.50	63.5%	94.5%	98.2%
2.27	98.8%	100%	100%

At α = 1.2, a hundred thousand requests still miss the true mean by more than 10% more than half the time. Doubling the sample does not halve the error; it barely moves it.

This is not an academic point. It means two days with identical traffic can produce daily means that differ by 20%, and neither day is wrong. I ran 30 simulated days of 100,000 requests each:

• α = 2.27: plan 457 tokens, worst day +1%, best day -1%.
• α = 1.2: plan 1,451 tokens, worst day +22%, best day -15%.

The α = 1.2 row also shows the trap: the 30-day average of daily means landed at 1,451, which is 5.5% under the true 1,536. The plan was built on 3 million requests and still came in low.

One day in node-hours

Put the tail on real hardware. One million requests a day, α = 1.2, 256-token floor, and the prefill rate from my last piece: a 4k prompt costs 786 ms on one node, which is about 5,211 prefill tokens per second.

• Total prefill: 1.54 billion tokens, or 3.41 node-days.
• The top 1% (10,000 requests): 0.71 billion tokens, or 1.58 node-days.
• A single p99 request: 11,900 tokens, or 2.3 seconds of one node's prefill.

That last line is the operational cost. The mean asks for 0.29 seconds. The request that shows up once in a hundred asks for 2.3, and while it runs, everything else in the batch waits. If you chunk prefill at 512 tokens, the same request is 23 inserts instead of one block, and the longest single stall drops to about 0.1 seconds. That is the lever; the tail is the reason you need the lever.

What to do Monday

1. Log prompt tokens per request. Not tokens per user, not tokens per day. Per request.
2. Compute two numbers from that histogram: the share of total tokens held by the top 1% of requests, and the Gini. Both are five lines of code.
3. Size the prefill budget as total_tokens × top1_share, not request_count × mean. The first is what the hardware does; the second is what the dashboard says.
4. If top-1% share is above 30% (α below about 1.4), stop trusting single-day means entirely. Use a fixed budget with a stated headroom, or a rolling seven-day p99, and write down which one you used.
5. Chunk the prefill so the p99 request cannot block the batch. The break-even chunk from my last piece was 119 tokens; the exact number depends on your batch, but the direction does not.

The uncomfortable part: none of this is exotic. Earthquakes have been telling us for a century that the top 1% is the system and the rest is the average. Serving teams inherited the mean from a world where the tail was small, and most capacity spreadsheets still assume it.

The script

import numpy as np

rng = np.random.default_rng(7)

def top_share_alpha(p, alpha):
    """Pareto: share of total held by the top p fraction = p**((alpha-1)/alpha)."""
    return p ** ((alpha - 1) / alpha)

def alpha_from_top_share(p, share):
    return 1.0 / (1.0 - np.log(share) / np.log(p))

def gini_pareto(alpha):
    return 1.0 / (2 * alpha - 1)

# --- measured systems: implied alpha and Gini from reported concentration ---
for name, p, share in [
    ("hurricanes top 1%", 0.01, 0.0763),
    ("hurricanes top 13.5%", 0.135, 0.50),
    ("earthquakes top 1%", 0.01, 0.90),
]:
    a = alpha_from_top_share(p, share)
    print(f"{name:24s} alpha={a:6.3f} gini={gini_pareto(a):.3f}")

# --- serving: top-1% share, p99/mean, mean/median ---
for a in [1.1, 1.2, 1.5, 2.0, 2.27]:
    p99_over_mean = (0.01 ** (-1 / a)) / (a / (a - 1))
    mean_over_med = (a / (a - 1)) / (2 ** (1 / a))
    print(f"alpha={a:4.2f} top1={100*top_share_alpha(0.01, a):5.1f}% "
          f"p99/mean={p99_over_mean:5.2f} mean/med={mean_over_med:5.2f}")

# --- does the sample mean converge? ---
for a in [1.2, 1.5, 2.27]:
    mu = 256.0 * a / (a - 1)
    for n in [1_000, 10_000, 100_000]:
        u = rng.random((400, n))
        x = 256.0 * u ** (-1.0 / a)
        hit = np.mean(np.abs(x.mean(axis=1) - mu) <= 0.10 * mu)
        print(f"alpha={a:4.2f} n={n:6d} P(within 10%)={100*hit:5.1f}%")

Change the 256-token floor and α at the top for your own workload. If your trace gives a different answer than my table, your trace is the one that is right, and I would like to see the two numbers.

---

One free thing, no invoice: run the script on your own request-length histogram and send me the top-1% share of tokens and the Gini. I will reply with one sentence on whether your capacity plan is sized on the mean and how far off it is.

[e-mail]