TL;DR: For low-cost property-management support triage, use an API for bounded summarization: normalize the ticket once, reduce repeated tokens, persist the structured result, and retain the raw conversation only for the period your audit policy requires. These token reduction strategies also work in a Node.js worker, even though the validation rules below are shown in Python. Measure tokens before using the API, reject malformed output, and do not confuse a shorter prompt with a cheaper or safer system. The expensive part is usually repeated context, not the final few sentences.
A ticket such as The boiler is still out after the contractor visit; please call me after 6pm needs a queue, urgency, property identifier, and a short reason. It does not need the entire email thread copied into every retry. That distinction matters more than shaving adjectives from a prompt.
Shorter is not automatically safer.
Consider a tenant who sends three replies in one thread. The newest message says the leak is contained, the quoted message says water is entering the bedroom, and an automated footer repeats the property address twice. A naive character limit can preserve the footer and the reassuring sentence while dropping the safety detail. A field-aware normalizer can keep the newest unquoted text, extract the address into a controlled field, retain safety terms from the quoted context, and mark the record as requiring review. That is a longer implementation than text[:4000], but it prevents a token reduction strategy from silently changing the operational meaning of the ticket. The API call should receive the normalized representation, while the original thread remains in the short-lived evidence store under the retention policy.
What should the storage bill actually retain?
I start with the retention question because token reduction is only half of the runtime cost. A useful record has three layers: the normalized input, the model-produced decision, and enough provenance to reproduce the decision. Keeping all three forever is easy to implement and difficult to defend. Keeping only the summary makes later disputes hard to investigate.
For a first pass, retain the raw ticket for 30 days, the normalized text and structured decision for 180 days, and an immutable hash of the input with the processing metadata for the longer audit period. Those are policy examples, not universal legal requirements; a property operator must set the periods with its legal and support teams. The engineering point is that different data has different value. I first thought the token count would dominate the retention decision; later, the more useful distinction was whether a record could explain a disputed routing result without exposing the whole conversation.
| Record | Needed for triage | Useful for investigation | Retention decision |
|---|---|---|---|
| Raw message and attachments | No, after normalization | High | Short, policy-defined window |
| Normalized ticket text | Yes | High | Longer than raw content |
| Structured decision | Yes | High | Operational and audit window |
| Input hash and runtime metadata | No | Medium | Long-lived integrity record |
The trade-off is uncomfortable: deleting raw text reduces exposure and storage, but it removes the easiest evidence when a resident contests an urgency label. I would rather make that loss explicit than hide it behind an apparently low token count. A retention job should record what it deleted and why, while the triage table keeps the decision, schema version, tokenizer name, and processing timestamp.
How can low-cost summarization use an API without losing token-critical detail?
Use a pipeline with a narrow contract. First strip quoted reply chains, signatures, tracking parameters, and duplicated notification text. Then preserve the fields that can change routing: stated problem, location, access constraints, safety language, requested time, and the ticket identifier. Redaction must happen before token counting, and token counting must happen before the request. The same strategies apply when a Node.js service calls an API; the language changes, while the boundary and the evidence do not.
A tokenizer is an estimator for a particular encoding, not a universal measure of meaning. The tiktoken project documents a byte-pair encoding library and provides encoding utilities; use the encoding appropriate to the runtime configuration, then compare the count with the service's actual usage telemetry. A count made with a different encoding can be directionally useful and operationally wrong. In an API integration, call the counter a local estimate and keep the measured usage returned by the service as the accounting record.
Here is the important boundary in Python: the function prepares data and enforces a local input limit, but the network adapter remains generic. It also keeps the summary cache key tied to content, policy, and schema rather than to a mutable ticket status. Version 1 of this boundary should be boring: a normalized payload, one bounded request, and one validated write.
import hashlib
import json
from dataclasses import dataclass
from typing import Any
import tiktoken
@dataclass(frozen=True)
class TicketInput:
ticket_id: str
text: str
policy_version: str
schema_version: str
def token_count(text: str, encoding_name: str) -> int:
encoding = tiktoken.get_encoding(encoding_name)
return len(encoding.encode(text))
def cache_key(ticket: TicketInput) -> str:
payload = json.dumps(
{
"text": ticket.text,
"policy_version": ticket.policy_version,
"schema_version": ticket.schema_version,
},
sort_keys=True,
separators=(",", ":"),
).encode("utf-8")
return hashlib.sha256(payload).hexdigest()
def prepare(ticket: TicketInput, encoding_name: str, max_tokens: int) -> dict[str, Any]:
count = token_count(ticket.text, encoding_name)
if count > max_tokens:
raise ValueError(f"ticket exceeds local input limit: {count} > {max_tokens}")
return {
"ticket_id": ticket.ticket_id,
"text": ticket.text,
"input_tokens_estimate": count,
"cache_key": cache_key(ticket),
"policy_version": ticket.policy_version,
"schema_version": ticket.schema_version,
}
The limit is a guardrail, not a license to truncate blindly. A hard cut in the middle of a sentence can remove not from do not enter, or separate a unit number from an access instruction. Prefer field-aware normalization and an explicit overflow state. Tickets that still exceed the limit can go to manual review or a separate chunking path, with that route visible in metrics.
What does a correct structured result look like?
Summarization is a poor success criterion for triage. The queue needs a valid object. A minimal contract might contain category, urgency, property_id, requested_callback_window, reason, and needs_human_review. Every field needs an allowed shape, and every missing value needs a defined representation. Empty strings and nulls are not interchangeable if downstream routing treats them differently.
Validate the decoded object before writing it to the queue. Check the schema, enum values, identifier format, and cross-field rules. For example, needs_human_review should be true when the category is unknown, when safety language is present, or when the property identifier is absent. That rule belongs in deterministic code; asking a model to remember it in prose makes a storage record look complete while the routing decision remains ambiguous.
from typing import Any
ALLOWED_URGENCY = {"routine", "priority", "emergency", "unknown"}
def validate_decision(value: Any) -> dict[str, Any]:
if not isinstance(value, dict):
raise ValueError("decision must be an object")
required = {
"category",
"urgency",
"property_id",
"requested_callback_window",
"reason",
"needs_human_review",
}
if set(value) != required:
raise ValueError("decision fields do not match the contract")
if value["urgency"] not in ALLOWED_URGENCY:
raise ValueError("unsupported urgency")
if not isinstance(value["needs_human_review"], bool):
raise ValueError("needs_human_review must be boolean")
if not isinstance(value["reason"], str) or not value["reason"].strip():
raise ValueError("reason must be non-empty")
return value
Do not retry a malformed response with the same unbounded prompt. Record the validation failure, apply a bounded retry policy, and route the ticket to a human when the contract still fails. A retry that creates a second stored decision without linking it to the first is an audit defect, even if the second answer looks better.
Which failures survive a shorter prompt?
Token reduction does not fix duplicate delivery, stale cache entries, or a queue worker that crashes after the model call and before the database commit. Give each processing attempt an idempotency key derived from the ticket revision, policy version, and schema version. Store an attempt state such as started, validated, persisted, or manual_review; do not infer state from whether a summary column happens to be non-null.
This is where a low-cost design can become expensive.
If the worker commits the ticket state before it commits the decision, a retry can send the same text again and create two plausible summaries. If it commits the decision first, a crash can leave a decision with no queue transition. The durable answer is a transaction or an outbox-style handoff that makes the state transition explicit; prompt shortening cannot repair that ordering problem.
The cache deserves the same skepticism as the prompt. A ticket can be edited after a resident adds a safety detail. If the key uses only ticket_id, the old classification can be served as new. If the key includes the normalized content hash and policy versions, an edit creates a new decision while identical retries reuse the existing one. That is more storage than a single mutable row, but it gives the operator a defensible history.
Observe four quantities separately: normalized input tokens, output tokens, validation failures, and repeat-processing rate. Aggregate them by ticket class and policy version. A falling average token count can conceal a rising overflow rate; a stable model response can conceal duplicate writes. I would page on the operational consequence, not on a vanity token number.
A practical decision rule
Keep the input bounded by removing repeated context and retaining decision-bearing fields. Count with the runtime's intended encoding. Require a strict structured object, validate it locally, and persist the exact versions that shaped it. Cache by content and policy, not by a mutable identifier alone. Retain raw text for a declared period, then delete it through an auditable job while keeping the minimum provenance needed to explain the routing result.
The deliberate loss is raw evidence after the retention window. The benefit is a smaller exposure surface and a storage policy that can be explained field by field. If the business cannot accept that loss, the answer is a longer approved retention period or a separate controlled archive, not a more aggressive prompt rewrite.
Top comments (0)