DEV Community

Glenn Allogho
Glenn Allogho

Posted on Originally published at Medium

Your agent pays for an essay when it needs a tick box

Amazon's internal ticket labeler paid Claude Sonnet 4.6 to write about 170 tokens of JSON per ticket. The only field anyone used was one label picked from a list of 20.

The replacement, a small open-source decision model called Strands Decider, writes one token and a confidence score. Amazon reports roughly 98% lower cost (about $22,200 down to about $500 per million tickets) and call time dropping from 4.2s to 0.9s.

The catch: that baseline is a mid-tier model at list price. With Haiku 5.5 pricing on the same token counts, the LLM side lands near $740 per million tickets, and a single self-hosted instance only wins above roughly 1,900 tickets an hour. Most of the saving came from using an expensive model for a job that never needed words.

Two things worth copying either way:

  • Sort your JSON keys. Key order changes token order, and in my CPU test one ticket flipped labels from reordering alone. Sorting removed the drift on all 16 tickets.
  • Gate on confidence. Auto-apply at 0.9 and up, fall back below. Every wrong answer in my 26-case smoke test scored under 0.8.

I ran the 2B model on a plain CPU box and redid the break-even math against Haiku and Google's new Decisions API in the full write-up:

https://medium.com/data-science-collective/amazon-cut-an-ai-bill-by-98-the-real-lesson-is-the-1-token-3f45ba3e0e09?sk=b69edafdc23c30d40d4e9aafeb842589

Top comments (0)