Amazon's internal ticket labeler paid Claude Sonnet 4.6 to write about 170 tokens of JSON per ticket. The only field anyone used was one label picked from a list of 20.
The replacement, a small open-source decision model called Strands Decider, writes one token and a confidence score. Amazon reports roughly 98% lower cost (about $22,200 down to about $500 per million tickets) and call time dropping from 4.2s to 0.9s.
The catch: that baseline is a mid-tier model at list price. With Haiku 5.5 pricing on the same token counts, the LLM side lands near $740 per million tickets, and a single self-hosted instance only wins above roughly 1,900 tickets an hour. Most of the saving came from using an expensive model for a job that never needed words.
Two things worth copying either way:
- Sort your JSON keys. Key order changes token order, and in my CPU test one ticket flipped labels from reordering alone. Sorting removed the drift on all 16 tickets.
- Gate on confidence. Auto-apply at 0.9 and up, fall back below. Every wrong answer in my 26-case smoke test scored under 0.8.
I ran the 2B model on a plain CPU box and redid the break-even math against Haiku and Google's new Decisions API in the full write-up:
Top comments (0)