The job summarised support tickets
into a weekly report.
One section per product.
It read well.
It read confidently.
It arrived every Monday.
Nobody noticed
that the last two products
had not appeared
for a month.
The tickets had grown.
The report had grown with them.
And the call to the model
had a maximum output length,
set once,
when the report was half the size.
When a model hits that limit
it does not wrap up.
It does not apologise.
It stops,
mid list,
sometimes mid word,
and hands back what it has.
The response does say so.
Every serious API
returns the reason a generation ended.
It finished on its own,
or it hit the limit,
or a stop sequence,
or a filter.
The code read the text
and never read the reason.
Prose hides it best.
A report of eight sections
that ends after six
looks like a report of six.
Structured output
hides it differently.
Truncated JSON fails to parse,
so somebody adds a repair step
that closes the brackets,
and now half an answer
validates.
So read the stop reason
on every call,
the way you read
an HTTP status.
Finished naturally is success.
Hit the limit is not an answer.
It is an error,
or a signal to continue,
but never a result
to pass downstream.
Set the limit
with real headroom,
and alarm when calls get close to it,
because slow growth
is exactly how you get there.
Do not repair broken output
into valid output.
If it does not parse,
it is incomplete,
and incomplete
should be loud.
Where the shape is known,
check the shape.
Eight products went in.
Count eight sections
coming out.
And log the reason
next to the cost,
so the week the summaries
started stopping early
shows up on a graph
instead of in a meeting.
A model that runs out of room
does not tell you in the text.
It tells you in the field
you were not reading.
– Serguey Asael Shinder
Top comments (0)