A quota admits requests. A round-robin dispatches requests. A rate limit counts requests. What a tenant consumes is residency - how long its sequences sit in the running batch taking a slot of every decode iteration - and residency is the output length. The exchange rate between one request and one millisecond of GPU is therefore a number nobody chose.
const B = running.length;
const dt = iterBase + iterSlope*B + prefillCost*prefillTokensAdmittedThisStep;
const perSeq = (iterBase + iterSlope*B) / B; // what one decode token costs
for (const v of running){ v.gen++; gpu[v.tenant] += perSeq; }
perSeq is the whole accounting system: summed over every iteration it equals the busy time exactly, so the four tenant shares add to 1.000000. Four tenants share one replica under continuous batching, send identical prompts, and offer identical load in the resource itself - 750.0 ms of machine per second each - so all four stay backlogged. Same 120 seconds, same GPU, same bill; only the split moves: https://dev48.infy.uk/ai/days/day71-multi-tenant-fairness.html
| scheduler | classify | chat | summarise | agent | max/min |
|---|---|---|---|---|---|
| first-come-first-served | 25.9 | 26.8 | 24.7 | 22.6 | 1.18 |
| equal requests/min quota | 7.4 | 11.3 | 30.3 | 51.0 | 6.87 |
| VTC, shipped weights | 12.8 | 17.2 | 31.1 | 38.9 | 3.05 |
charge dt / B
|
25.0 | 25.0 | 25.0 | 25.0 | 1.00 |
Turning on the fairness feature made the split 5.8x less fair, with every tenant inside its quota and every dashboard green. The deciding quantity is the realised output length - 23.7 tokens for classify, 353.8 for the agent team. It is in no operator config, it is not max_tokens, and it is not knowable at admission: the model decides it one token at a time, after the request is in.
VTC (Sheng et al., OSDI 2024) reproduces and then inverts. It beats the quota at 3.05x and is flat under floods while FIFO decays 1.17 to 7.00, and at equal offered load it is 2.6x worse than no scheduler at all, crossing back only past a flood of about 3.3x. Its own counter comes out at a ratio of 1.014 while it hands out the machine at 3.05x. Not a bug, a different quantity.
The shipped decode weight is 2. This machine's own rate is (iterBase + iterSlope*B) / (B * prefillCost) = 15.42 at a batch of 24, and 27.92 at 8, 12.29 at 48. The right constant is a function of the load the scheduler is deciding, so the repair is to delete it and charge dt / B as it is spent: 1.00x at every flood and spread, and 93,932 decode tokens against FIFO's 92,864. Better on both axes - the only dominant point on the page.
What the measurement took back
I had the slot cap written up as the cheap fix before I ran it. Cap each tenant at maxBatch / n = 6 running sequences and decode time splits 24.7 / 25.2 / 25.2 / 24.9, a ratio of 1.023 - provably fair in the thing it controls. It scores 1.92x overall, worse than doing nothing, because prefill is 31.3% of this machine and the cap says nothing about it. Six slots of 24-token answers turn over fifteen times faster than six of 354-token answers: classify burns 18.2% of the replica on prefill, the agent 1.3%.
172 verifier assertions, and an independently written reference simulator agreeing exactly over 1,728 configurations x 9 policies. One AI idea a day, computed rather than quoted: https://dev48.infy.uk/aifromzero.php
Top comments (0)