Short answer: store the review unit, its source digest, the prompt/schema version, the submission manifest, and the terminal result as separate records. A batch is useful only when a worker can restart without guessing which game-code changes were already summarized. The model API is one stage in that process; it is not the system of record.
For a gaming team reviewing changes to gameplay code, the practical output is a structured finding, not a paragraph that happens to look sensible. I would use a batched workflow when reviews may wait, then make publication conditional on three checks: the result belongs to the intended change, it matches the declared schema, and every input has a terminal state. That choice favors recoverability over the pleasant simplicity of one request per diff.
The important design question is therefore storage-first: what must remain true after a worker restart, a partial result, or a revised instruction? Once those invariants are explicit, the batch submission mechanism becomes an adapter instead of the architecture.
Which records must survive a restart?
Start with one durable review unit per meaningful change. A useful key can combine the commit identifier, file path, and hunk range, but the key should be stored as data rather than reconstructed from a log line. Store the exact source text or a protected reference to it, a digest of that source, and the instruction version used to interpret it.
Five records are enough for a small implementation:
| Record | Minimum purpose | Why it exists |
|---|---|---|
| Review unit | Stable ID, source digest, source location | Defines what is being reviewed |
| Contract | Schema version and allowed values | Defines what a valid finding means |
| Manifest item | Review ID, payload digest, batch ID | Connects local work to remote work |
| Submission | Batch ID, request state, timestamps | Makes retries and polling resumable |
| Finding | Parsed result, validation state, evidence | Separates accepted output from raw output |
This is deliberately more storage than a demo needs. It is less storage than an audit trail needs, and structured review sits uncomfortably between those two cases. A random request ID is insufficient: after a lost response, the worker cannot tell whether it should wait, reconcile, or submit a duplicate.
I also keep raw responses apart from accepted findings. Raw model output can be valuable when a validator rejects a record, but it should have stricter access and retention rules than the normalized result. Source code and player-facing data can both contain information that does not belong in ordinary application logs.
What does a valid game-code finding look like?
The contract should be narrower than the model's vocabulary. For this example, a review item accepts one of three risk values and requires an evidence array, even when the array contains an explicit statement that no supplied evidence was found. Missing information should remain visible; it should not become a guessed low-risk result.
Here is a request shape I would put in a fixture before connecting a network client. The schema_version value is an application choice, not a claim about any model service.
review_request = {
"review_id": "commit-abc123:combat/damage.py:18-31",
"source_digest": "sha256:example-digest",
"schema_version": 1,
"instruction_version": "review-rules-1",
"text": "changed gameplay code goes here",
"output_contract": {
"risk": ["low", "medium", "high"],
"summary_max_chars": 600,
"evidence_required": True,
},
}
The digest binds the request to a particular diff. The instruction and schema versions bind it to a particular interpretation. I start this example at contract version 1; changing the enum or the 600-character limit should create version 2 rather than silently changing version 1. If either changes, the idempotency identity changes too. Otherwise a replay can return a result produced under yesterday's contract and make it look current.
There is a subtle distinction here. Valid JSON is a serialization property; a valid finding is an application property. A result can parse successfully while naming a file that was not in the supplied change or using probably-medium as a risk value. The validator must reject both cases, and the stored state should say why.
How should a GPT summarization workflow combine batch processing?
Treat the manifest as the expected set, not as an incidental request log. Before submission, write one manifest item for each review unit and record its payload digest. After result retrieval, map records by review_id, reject duplicate terminal results, and compare the set of returned IDs with the set of manifest IDs.
The reconciliation rule is simple:
def reconcile(manifest_items, returned_results):
expected = {item["review_id"] for item in manifest_items}
seen = {}
errors = []
for result in returned_results:
review_id = result.get("review_id")
if review_id in seen:
errors.append((review_id, "duplicate result"))
else:
seen[review_id] = result
missing = expected - set(seen)
unexpected = set(seen) - expected
errors.extend((review_id, "missing result") for review_id in missing)
errors.extend((review_id, "unexpected result") for review_id in unexpected)
return seen, errors
An absent line is not an empty summary. It is an incomplete batch. That distinction matters to a review queue because an empty summary can be published and forgotten, while an incomplete state can be retried or shown to an operator.
A transport failure is also ambiguous. The worker may have lost the response after the remote system accepted the submission. The recovery path should look up the local submission and its idempotency identity before creating another batch. If that identity cannot be reconciled, quarantine the submission for operator review rather than quietly multiplying work.
Which failure modes deserve different states?
One retry branch hides too much. I use separate states for transport uncertainty, item validation failure, and completeness failure because each one has a different recovery action.
| State | Evidence | Next action |
|---|---|---|
| Submission uncertain | No confirmed remote state | Reconcile by stored identity |
| Item rejected | Response exists but violates the contract | Quarantine the item and retain the reason |
| Batch incomplete | One or more manifest IDs have no terminal result | Retry unresolved items after reconciliation |
| Accepted | ID, digest, and fields pass validation | Make the finding available to the review consumer |
The last row is intentionally strict. A completed remote job is not the same as a completed application workflow. Completion at the service boundary only says that the service reached its own terminal state; it does not prove that the consumer received one usable finding for every game-code change.
The worker can remain small if it delegates those decisions to storage and validation functions. A Node.js worker can implement these same boundaries around its model API client; the example is Python because the contract is easier to inspect without hiding it behind a framework.
def process_batch(items, client, store):
manifest = [store.create_manifest_item(item) for item in items]
batch_id = client.submit([item["payload"] for item in manifest])
store.record_submission(batch_id, manifest)
state = client.wait_for_terminal_state(batch_id)
store.record_remote_state(batch_id, state)
if state != "completed":
store.mark_submission_for_reconciliation(batch_id)
return
results = client.results(batch_id)
accepted, rejected = validate_and_reconcile(manifest, results)
store.record_rejected(rejected)
store.record_accepted(accepted)
The placeholder wait_for_terminal_state represents a bounded polling policy, not an unlimited loop. Its interval and timeout should be chosen from the service's documented limits and the review latency the team can tolerate. A worker restart must resume from the recorded submission state rather than submit the same manifest blindly.
How do competing API shapes change the storage contract?
The application should depend on capabilities: submit work, inspect state, retrieve results, and normalize errors. Those capabilities appear in different forms across real services. Some expose delayed batch jobs, some expect the client to manage concurrency around synchronous generation, and some expose relevance ordering for supplied documents rather than text generation. A reranking interface cannot be treated as a summarization interface merely because both accept text.
That difference changes the storage adapter, not the invariants. A delayed job needs a durable remote ID and reconciliation policy. A synchronous path needs an attempt record, timeout classification, and an idempotency decision. A service that returns ordered documents needs a different output contract altogether. I would compare these options on result identity, item-level errors, input and output limits, retention behavior, rate-limit semantics, and whether schema enforcement is observable. A single latency number cannot answer those questions.
This is also where the batch trade-off becomes concrete. Batching adds a manifest, delayed visibility, and reconciliation work. In return, it can group work whose consumer does not need an answer before the next editor action. For a tiny interactive review, synchronous generation may be easier to operate, but only if its retry and timeout states are just as explicit. Convenience is not a durability guarantee.
What should a safe rollout prove first?
Begin with replayable fixtures, not live traffic. Include a short gameplay diff, malformed JSON, an unsupported risk value, a wrong source digest, one missing output, one duplicate output, and a result with an unexpected review ID. The assertions should inspect stored states and rejection reasons, not merely check that a function raised an exception.
For the first rollout, write accepted findings beside the existing review output and compare a sample. Do not let the new path replace the canonical result until the manifest, digest, and completeness checks have been exercised on real-shaped data. A second step can publish only records that pass all three checks. Retain manifests long enough to explain a disagreement, with access controls suited to source code.
Operational measurements should include acceptance rate, item-level rejection rate, unresolved-work age, completeness lag, retry count, and validation failures grouped by rule. Batch completion by itself is a weak metric; it can be green while a required review item is missing.
The decision rule is compact: choose batching when delayed processing is acceptable and durable reconciliation is worth the extra records. Choose a synchronous path when interaction is immediate and the same structured contract can be enforced before publication. In either path, the model call remains replaceable. The review unit, its evidence, and its state transitions are what the gaming workflow must trust.
Sources
References:
Top comments (0)