Ten reviews per LLM call cut the bill by 57%. A thousand per call only reaches 65%.

You have 100,000 customer reviews and a prompt that tags one of them as positive or negative and says what it's about. Sending them one by one is the obvious first version. Sending ten per request cuts the bill by 57%. Sending a thousand per request cuts it by 65%, and those last eight points are where the trouble starts.

You pay for the rules once per request

You pay per token, a chunk of text about three-quarters of an English word. Input tokens are what you send, output tokens are what the model writes back, and output usually costs several times more.

A labeling request has two parts: the instructions (the labels, the rules, a couple of examples) and the review itself. The API remembers nothing between calls, so the instructions travel with every request. In my test, a one-review request sends 313 tokens, and 295 of them are the same every time: the rules plus a few tokens of message formatting. Label 100,000 reviews one at a time and you pay for those rules 100,000 times.

Pack ten reviews into one request and the rules are paid once per ten. Each review now costs a tenth of the instructions, its own text, a short [7] label in front of it, and its line of the answer. The first part shrinks as you pack more. The rest doesn't shrink at all.

Where the curve goes flat

My test task is a made-up clothing store with two labels per review: sentiment (positive, negative, mixed) and topic (shipping, quality, sizing, price, support, other). I wrote 20 short reviews and their correct answers myself, and counted everything with tiktoken, OpenAI's open-source token counter, using o200k_base, the encoding of current GPT models. I didn't call a model: the replies are the right answers in the requested format. Prices are gpt-5.6's list prices, the ones the vendor publishes, $2 per million input tokens and $12 per million output, from the catalog of the LLM gateway I work on, read on 21 October 2026.

Reviews per call	Tokens per review, in / out	Tokens in one reply	100,000 reviews	Saved
1	313 / 13	13	$78.38	—
5	90 / 19	96	$41.00	48%
10	54 / 19	192	$33.82	57%
100	24 / 19	1,920	$27.83	64%
1,000	21 / 19	19,201	$27.25	65%

Even with free instructions the bill can't drop below $27.18, because the reviews and the answers stay. So 65% is the ceiling, and ten per call already covers 87% of the way there. Packing also adds a cost: each answer line needs an "id", so output per review grows from 13 tokens to 19, at six times the input price. At one per call output was 20% of the bill. At ten it's 68%, and packing can't touch that part.

A bigger request means a bigger failure

With one review per call, a bad reply loses one review. With a hundred per call, a reply that stops halfway can lose all hundred, and you still pay for the half that arrived. Replies break for boring reasons: the connection drops, the model skips an item, or it hits max_tokens, the cap you set on how long its answer may be.

The answer format decides how much you keep. I cut a ten-review reply at 70% of its length. Written as one JSON array, it doesn't parse at all, so 0 of 10 labels survive. Written as JSON Lines, one JSON object per line, 6 of 10 lines parse and only 4 reviews go back for a second try. Both formats cost the same: 194 and 192 tokens.

So ask for one object per line that carries the id, set max_tokens to the number of items times one answer line with room to spare, and check finish_reason, the field in the response that reads length when the reply hit the cap.

Find your own flat spot

Count your instructions, one item and one answer line on your real prompt, then plug them in:

# Token counts from your own prompt, prices in $ per 1M tokens
INSTR_1, INSTR_N = 295, 325  # instructions: one-item version, packed version
ITEM, TAG = 18, 3            # tokens per item, plus its "[7] " label
OUT_1, OUT_N = 13, 19        # answer per item: alone, and with its "id" key
PRICE_IN, PRICE_OUT = 2.00, 12.00
ITEMS = 100_000

def bill(n):
    if n == 1:
        tokens_in, tokens_out = INSTR_1 + ITEM, OUT_1
    else:
        tokens_in, tokens_out = INSTR_N / n + ITEM + TAG, OUT_N
    return ITEMS * (tokens_in * PRICE_IN + tokens_out * PRICE_OUT) / 1e6

for n in (1, 5, 10, 20, 100):
    print(f"{n:>4} per call: ${bill(n):6.2f}  ({1 - bill(n) / bill(1):.0%} saved)")

It prints $78.20 at one per call, $33.50 at ten and $27.65 at a hundred, within 35 cents of the table because the inputs are rounded. Stop where doubling the batch adds only a couple of points; here that's ten to twenty. This isn't a vendor's batch API, which gives a discount for waiting hours for results but still sends one request per review, rules included. The two stack.

Then read the reply line by line and resend only what's missing:

import json

def parse_packed(reply, sent_ids):
    labels = {}
    for line in reply.splitlines():
        try:
            item = json.loads(line)
            labels[item["id"]] = (item["sentiment"], item["topic"])
        except (ValueError, KeyError, TypeError):
            continue  # cut-off or malformed line: that item gets resent
    missing = [i for i in sent_ids if i not in labels]
    return labels, missing

reply = '''{"id": 1, "sentiment": "negative", "topic": "sizing"}
{"id": 2, "sentiment": "negative", "topic": "shipping"}
{"id": 3, "sentiment": "posi'''
labels, missing = parse_packed(reply, [1, 2, 3, 4])
print(f"parsed {len(labels)}, resend {missing}")

It prints parsed 2, resend [3, 4].

What this doesn't solve

I didn't measure accuracy. Models can skip, merge or mislabel items in a long list, and my replies are perfect by construction, so compare a sample of packed results with one-at-a-time ones before you switch. The counts come from OpenAI's tokenizer, and Claude and Gemini split text differently. gpt-5.6 can also spend hidden thinking tokens, billed as output, that none of these numbers include. And short reviews favour packing: the longer each item, the smaller the instructions' share and the smaller the saving.

The gateway I mentioned sells gpt-5.6 at $1.70 / $10.20, 15% under list. That takes ten per call from $33.82 to $28.75 and leaves the curve where it was. It also fills in a max_tokens of 4,096 when a request has none, which the official API doesn't, so at 19 tokens per review a call with more than about 213 reviews gets cut off there.

The takeaway

Instructions are billed per request, items per item. Pack enough to turn the instructions into a rounding error, about ten here, and stop: past that point you're buying risk, not savings.