The supplier emails were a mess.
Free text,
forwarded chains,
the odd photo of a printed page.
So you gave them to a model
with a schema.
Order number.
Quantity.
Delivery date.
Strict structured output,
so the reply
always parsed.
It always parsed.
For six weeks
the warehouse booked slots
straight from the extracted dates.
Then a lorry turned up
on a day nobody expected,
and another failed to turn up
on a day somebody did.
The email behind the second one
said, in full,
we will confirm the date next week.
The model had returned
a delivery date anyway.
The fourteenth of March.
Plausible.
Well formatted.
Nowhere in the email.
Here is the mechanism.
Strict mode
does not ask the model nicely.
It constrains the decoding.
At every step,
only tokens that keep the output
valid against your schema
are allowed out.
In the strict mode
most teams use,
every field has to be listed
as required.
Optional is spelled differently.
A field that may be empty
has to accept null
as one of its types.
Yours did not.
So when the email
held no date,
the one thing the model
was not allowed to produce
was the truth.
Something had to go in the field,
and the most likely string
shaped like a date
went in.
The schema guaranteed the shape.
Nothing guaranteed the content.
So give every field
a legal way to say not stated.
Make the date
a string or null,
and tell the model
that null is the right answer
when the text does not say.
Ask for a second field
beside each value,
the exact words it came from.
Then check in code
that those words
actually appear in the email.
A date with no quote
goes to a person.
And count the nulls each week.
If they drop to zero,
the suppliers have not got tidier.
Your schema has started
filling the gaps again.
A form with no box for I don't know
does not get fewer unknowns.
It gets confident guesses
in the place where the answer should be.
– Serguey Asael Shinder
Top comments (0)