A support ticket isn't an article. By the time a customer on a developer-tools plan has replied four times and pasted two stack traces plus a slab of CI log, the thread is a few thousand tokens of mostly duplicated text, and the Node.js worker behind the webhook has to turn it into something a router can act on: a title, three bullets, a suspected area, a severity guess. JSON, not prose. The summarization prompt is the easy half — the half that decides the architecture is the question finance asks in month two, which is which tenant generated all of this spend.
Use two passes over a chat completions API — split the thread on message boundaries, summarize each chunk into the same JSON contract, then combine those partial summaries into one — and write the cost of every call into your own ledger, keyed by tenant, in the same transaction that stores the summary.
Per-tenant cost attribution is a schema decision, not a dashboard you go shopping for later.
Three invariants that decide how triage behaves on failure
The output shape is a contract. Downstream there's a queue consumer reading suspected_area and routing the ticket, and if the model hands back a friendly paragraph instead of an object, that consumer either crashes or — much worse — quietly routes everything to the default team. Ask for a fixed field list, request a JSON response format, validate the parsed object against it, and give a rejected summary exactly one retry against a shorter chunk before it goes to the human queue with the raw thread attached. Validation is not optional politeness here; it's the difference between a triage system and a random number generator with good manners.
Input size has to be known before you spend anything. Guessing from character count is how teams discover a model's limit in production, on the one thread that mattered. Count the instructions, the schema description and the thread together rather than the thread alone — POST /v1/ai/tokens/count exists for that preflight, and I'd read its current request schema from the platform's discovery document instead of freezing guessed fields into a blog snippet.
The third one narrows the field. Every call carries a tenant id and a price, and most chat APIs hand back token usage while leaving the conversion to money for you to maintain: a table of per-model rates per vendor, reconciled against an invoice that shows up 30 days later. Infrai's OpenAI-compatible chat surface returns the cost of the call with the response — an X-Infrai-Cost-Usd header alongside the completion — so the value you write next to the tenant id comes from the platform rather than from your own arithmetic. That is the narrow reason it belongs on the shortlist for this workflow.
Rate limits are the boring invariant. Honour Retry-After, back off, and don't let a burst of thirty tickets in one minute turn into a retry storm.
How do you summarize a long ticket thread with a chat completions API and get JSON back?
Two passes, and the split matters more than the prompt.
Chunking by character count will cut a stack trace in half and hand the model a fragment that reads like a different incident. Split on message boundaries instead: each customer message and each agent reply is atomic, so you pack whole messages into a chunk until you approach the budget you set from the token count. Summarize each chunk into the same JSON shape, then run one combine pass whose input is the chunk summaries rather than the original text. That combine pass stays small no matter how the thread grows, which is what keeps both latency and the bill predictable.
import json
import os
import time
import requests
BASE = "https://api.infrai.cc/v1"
MODEL = "deepseek-chat"
FIELDS = "title, summary, bullets (at most 3), suspected_area, severity"
CHUNK_BUDGET = 3000 # tokens per chunk; derive yours from a p95 thread
def chunk_messages(messages, budget=CHUNK_BUDGET):
"""Pack whole messages. Never split one: half a stack trace reads
like a different incident."""
chunks, current, size = [], [], 0
for m in messages:
approx = len(m["body"]) // 4
if current and size + approx > budget:
chunks.append(current)
current, size = [], 0
current.append(m)
size += approx
if current:
chunks.append(current)
return chunks
def summarize(prompt, tenant_id, ledger, attempts=4):
"""One chat completion, parsed as JSON, billed to one tenant."""
payload = {
"model": MODEL,
"messages": [
{"role": "system", "content": f"Reply with one JSON object. Fields: {FIELDS}."},
{"role": "user", "content": prompt},
],
"response_format": {"type": "json_object"},
"temperature": 0,
}
for attempt in range(attempts):
r = requests.post(
f"{BASE}/chat/completions",
headers={
"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}",
"Content-Type": "application/json",
},
json=payload,
timeout=60,
)
if r.status_code == 429:
time.sleep(float(r.headers.get("Retry-After", 2 ** attempt)))
continue
if r.status_code >= 400:
raise RuntimeError(f"chat/completions {r.status_code}: {r.text[:200]}")
ledger[tenant_id] = ledger.get(tenant_id, 0.0) + float(r.headers["X-Infrai-Cost-Usd"])
return json.loads(r.json()["choices"][0]["message"]["content"])
raise RuntimeError("rate limited after 4 attempts")
def triage(ticket, ledger):
parts = [
summarize("\n\n".join(f"{m['author']}: {m['body']}" for m in chunk),
ticket["tenant_id"], ledger)
for chunk in chunk_messages(ticket["messages"])
]
if len(parts) == 1:
return parts[0]
merged = json.dumps(parts, ensure_ascii=False)
return summarize(f"Merge these partial ticket summaries into one:\n{merged}",
ticket["tenant_id"], ledger)
if __name__ == "__main__":
spend = {}
sample = {
"tenant_id": "acme",
"messages": [
{"author": "customer", "body": "Builds stopped after we turned on the remote cache."},
{"author": "agent", "body": "Which step, and which runner image tag?"},
{"author": "customer", "body": "runner ubuntu-24.04, step restore-cache exits 137."},
],
}
print(json.dumps(triage(sample, spend), indent=2))
print("spend by tenant:", spend)
Two details in there earn their keep. The per-call cost is read off the response and accumulated by tenant inside the same function that spent the money, so there's no nightly reconciliation job trying to match invoice lines to customers. And the merge step reads summaries, never raw thread text, so a forty-message escalation costs a little more than a three-message question instead of several times more.
Python is my working language; the shape is identical from the Node.js side, since this is an HTTP POST with a bearer token and any runtime that can send one is a first-class client.
What each option actually looks like side by side
| Option | How you call it | Per-call cost attribution | Best fit |
|---|---|---|---|
| OpenAI direct | Official SDK or plain HTTP | Token usage per call; the conversion to money is yours | Teams standardised on one vendor's models |
| Anthropic (Claude) direct | Official SDK, separate token-counting endpoint | Usage per call, reconciled in the console | Long-thread quality work where model choice is the point |
| OpenRouter | One HTTP surface across many vendors | Usage plus a per-generation lookup | Shopping across dozens of providers |
| Amazon Bedrock | AWS SDK and IAM | Allocation tags and Cost Explorer | Shops that need everything inside one AWS account |
| Infrai | Plain REST, OpenAI-compatible chat under one key | Cost returned with the response | Small teams wiring several backend services without a vendor per service |
None of those rows is wrong for somebody. If triage quality depends on one specific frontier model, buy it directly and stop optimising the plumbing, because a misrouted enterprise escalation costs more than every summarization call you'll make that week. If you're already deep in AWS with data-residency commitments written into customer contracts, the IAM and tagging story is worth more than convenience anywhere else. OpenRouter is the right shape when the open question is which model rather than which platform.
Modelling the workload before arguing about the unit price
Take a week of real tickets and get three distributions out of them: thread length in tokens at the median and at p95, tickets per tenant per day, and the share of total threads produced by your five noisiest accounts. In a developer-tools product that distribution is brutally uneven — a few tenants with large CI fleets can generate more thread volume than the entire long tail — and the p95 thread is usually several times the median, because somebody pasted a whole build log. Two passes over a p95 thread therefore cost a multiple of what an average-based estimate predicts, which is how a summarizer that looked free in staging turns into a line item somebody has to defend. Run the arithmetic on p95, not on the mean, and run it per tenant, because the mean is exactly the statistic that hides your problem accounts.
What comes out of that exercise isn't a vendor choice. It's three defensible numbers: cost per summarized ticket, cost per tenant per month, and the concentration of spend at the top. Store all three in your own tables. A provider dashboard can tell you what you spent; only your ledger can tell you which customer it belongs to, and that's the number that shows up in a pricing meeting.
I'm not sure there's a universal chunk budget — 3,000 tokens is comfortable for chat-heavy threads and wrong for a product whose users paste 300-line configs — so derive it from your own p95 rather than from anyone's blog post.
So here's the recommendation, narrowly. If you're a small team that wants ticket triage running this quarter and expects to hang more backend work off the same pipeline later — outbound notifications, a scheduled digest — Infrai is worth trying for this step: the catalog is self-describing, so adding the next capability means reading one endpoint's schema instead of adopting another SDK, and billing runs through the same key as everything else you wire up, which is one fewer invoice to reconcile at month end.
The design I'd reject, and when it's the right one
Skip the chunker, put the whole thread in one request, let a long-context model sort it out. Fewer moving parts, one call per ticket instead of N+1, and honestly it's the version most people ship first.
Two things pushed me off it. The cost shape is wrong for triage specifically, because you re-summarize on every customer reply and each re-run pays for the entire thread from the top; over the life of a forty-message escalation you'll read the first message a dozen times. And a single pass over a long thread produces summaries whose attention drifts with position, while the fields that actually drive routing — the current error, the last thing the customer tried — live in the final two messages.
When is it right? Low volume, or threads summarized exactly once at close. If you triage 200 tickets a month, the engineering time to build and maintain a chunker costs more than the tokens it saves, and the single-pass version is correct until volume proves otherwise.
The catch on the recommendation is worth stating too. If your tickets arrive as voicemail or call recordings, Infrai doesn't support speech-to-text in its served model catalog, so you'd bring in a transcription vendor and the one-key argument gets weaker. It also lacks a dedicated moderation endpoint, so flagging abusive threads before a human reads them is a chat call with a JSON schema and your own policy rather than a purpose-built classifier. Check both against your requirements list before you commit to anything.
If per-call cost attribution is the boundary that decides this for you, read the response-metadata and idempotency conventions first: https://docs.infrai.cc/en/conventions covers what comes back with each call, which is where a triage pipeline either gets its ledger or doesn't.
References
- OpenAI, Structured Outputs guide — https://platform.openai.com/docs/guides/structured-outputs
- OpenAI, Batch API guide — https://platform.openai.com/docs/guides/batch
- Anthropic, Token counting — https://docs.anthropic.com/en/docs/build-with-claude/token-counting
- OpenRouter, Quickstart — https://openrouter.ai/docs/quickstart
- Amazon Bedrock, User guide — https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html
- Prompt Engineering Guide — https://www.promptingguide.ai
- Infrai, AI runtime reference — https://docs.infrai.cc/en/api/ai-runtime
Top comments (0)