Somebody on the team
was tired of the marketing copy
saying seamless
four times a paragraph.
So they found a setting.
Frequency penalty,
turned up to one point five.
It worked.
The copy got more varied.
The setting lived
in the shared client config.
Two weeks later
the invoice extractor,
which used the same client,
started losing rows.
Fifty invoices in,
fifty JSON objects out,
and the first twenty
were perfect.
After that
the keys began to wander.
supplier became vendor.
total became amount_due,
then sum.
One record
wrote invoice number
with a space in it.
The parser
expected exact names,
skipped what it did not recognise,
and reported success.
Here is what the setting does.
At every step
the model scores
each possible next token.
A frequency penalty
lowers the score of a token
by a fixed amount
for every time
it has already appeared
in the answer.
Twice,
two steps down.
Thirty times,
thirty steps down.
Its sibling,
the presence penalty,
lowers it once,
as soon as it has been used at all.
In prose
that pushes away
the tired word.
In a list of fifty records
the most repeated tokens
are exactly the ones
that must not change.
The key names.
The quotes.
The braces.
By record twenty,
supplier
had been written twenty times,
and every one of them
counted against the next.
The model was not confused.
It was obeying.
Writing the same word again
had become the expensive choice,
so it picked the nearest cheap one.
Code fails the same way.
A variable used often
gets renamed halfway down a function.
A closing bracket
starts losing to something else.
So keep penalties
out of anything
with a fixed vocabulary.
JSON,
code,
SQL,
labels from a list.
Set them per task,
not in a client
that every task shares.
Where the API offers it,
use structured output
with a schema,
so the keys are enforced
while the answer is written,
not hoped for afterwards.
And make the parser loud.
An unknown key
is an error,
not a row to skip.
Repetition
is a fault in an essay.
In a format
it is the whole point.
– Serguey Asael Shinder
Top comments (0)