On 20 December 2025, I was fortunate to be a part of the AWS Community Day Kochi. There were fantastic sessions going on throughout the event, but there was one presentation in particular that caught my whole attention and wouldn’t let go.
The speaker came on stage and delivered a big result right out of the gate – they reduced their AI operational costs by 14x using AWS Bedrock. But this wasn’t a highlight reel of quick success. Let’s face it, tech talks that only display the wonderful stuff don’t educate us much. Instead, there was a clear story of how the squad failed again and again on the route to the big victory.
This session was pure gold if you are a developer, or a firm seeking to scale AI features without burning through your entire runway. Here’s a look at the trip, the technological challenges, and how they finally cracked the puzzle.
The Business Problem: A 3-Body Challenge
But before we get to the solution, we need to understand the nightmare the team was dealing with. They named it the “3-Body Challenge.”
The trouble is...they were drowning in data. Specifically, they were being overwhelmed with unstructured communications about cargo bookings.
The emails were bilingual and consisted of a crazy mix of Japanese and English content.
They needed their system to be able to correctly extract 23 complicated entities from these emails, such as Air Waybill (AWB) numbers, Flight Numbers, Weights and Dimensions.
The Accuracy vs. Cost Dilemma
The aim was to carry out real-time automatic Named Entity Recognition (NER). The system needed to be low latency, and have a very high accuracy rate of over 95% (f1 score), to be useful in the production pipeline.
You might be wondering why not just throw a Large Language Model (LLM) at it? They did well. And the LLM readily met the precision required. But the operating cost at that high volume was a deal breaker.
Sound familiar? This is a trap many teams get into. They had designed a system that worked well, but they knew they could never build a business around a 14x cost problem.
Rethinking the Core Problem
The first important change in their thinking was in how they approached Named Entity Recognition. Instead than using typical BIO (Beginning, Inside, Outside) tagging, they defined NER as a Sequence-to-Sequence (Seq2Seq) task.
Generating Structured JSON
To grasp this, picture it like an e-commerce checkout system. You don’t want the system to just highlight random goods in a shopping cart, you want it to create a well-formed receipt.
The input (sequence 1) in their case was the raw, jumbled email text prompt asking the model to extract all 23 entities as a JSON array. The expected output (sequence 2) was the exact JSON text produced that matched those entities.
The Technical Goal: Knowledge Distillation
To achieve the high accuracy of a big LLM without the massive price, they turned to a concept known as Knowledge Distillation.
Think of your database as a huge library and the “Teacher Model” as the chief librarian who has read and comprehended every book. The teacher is large, complex and expensive to consult. The purpose of distillation is to compress the knowledge and transfer it to a “Student Model”. The student is smaller, considerably faster and much cheaper to run, offering you the best of both worlds, great precision and low cost.
Evaluating the Distillation Options
The speaker outlined the major routes he may take to achieve this knowledge transfer:
Option 1: Logit-Based (e.g., DistilBERT): This method uses a metric called KL Divergence to match the student's final output probabilities (logits) to the teacher's. It is easy, fast and effective. But it typically misses a lot of the sophisticated internal “reasoning” of the teacher model.
Option 2: Feature-Based (e.g., TinyBERT): That is, to align the internal hidden states and attention mappings of the two models. these transfer knowledge really deep. The negatives? It's quite brittle. It requires model architectures to be same and is quite sensitive to throwing
shape_mismatcherrors.Option 3: Token-Based: Here, the teacher model is compared with the final output sequence of the pupil token by token. It learns from the teacher's soft labels and is suitable for generative Seq2Seq jobs such as the JSON extraction they needed.
They decided to go with the Token-Based method as they were generating JSON arrays. And now it gets interesting, and by fascinating I mean extremely frustrating for their engineering team.
The Engineering Nightmares
Attempt 1: The Token Mismatch Wall
Their initial approach was a custom token distillation built with PyTorch. They sought to distill the Llama 3-8B model to the Llama 3-1B model with their specialized Seq2Seq task.
In fairness, the logic was reasonable, but the technological reality was a failure. For token-based distillation, you need absolute, perfect token alignment in order to effectively distill that output JSON. They ran into a big problem: the Llama 3 tokenizer and their Japanese/English bilingual text were misaligned.
They were trying to compare output sequences that just didn’t line up, and the training loss got wildly unstable. This caused a constant token_mismatch_error.
Attempt 2: The Brittle Architecture Wall
Try 2. No way they were giving up. They tried to run a Logit/Feature-Based distillation technique with a library named TextBrewer.
This, too, died a technical death, as the solution was too brittle and required completely matching architectures. The library was quite strict about requirements and their specific Llama models were incompatible.
The operation failed again, generating a shape_mismatch_error. The team found that they were spending 100% of their time fighting engineering difficulties and 0% of their time on true data research.
The Pivot: Isolating the Real Problem
The team stepped back and saw their problem was not a terrible theory. Their problem was bad engineering. They were getting beaten on the two hardest segments of the pipeline:
Tokenizer and Architecture Alignment.
Setting up stable, distributed training environments.
In short, they required a completely managed solution to do the heavy lifting on the engineering requirements things like framework selection, dataset preparation, and normalizations so they could focus entirely on addressing their business challenge.
Enter Amazon Bedrock Model Distillation
They migrated their whole pipeline to Amazon Bedrock. The new method looked like this:
Take the user prompts and feed them into the large Teacher model.
Generate high-quality synthetic data.
Use that data to train the smaller Student model and transfer the knowledge.
Deploy the customized distilled model for real-world inference.
The workflow was pleasantly simple. The prepared prompt dataset was just taken by a data scientist and turned into a JSONL file and uploaded to an Amazon S3 bucket. They then picked whatever instructor and student models they wanted from within the Amazon Bedrock service to start the distillation job.
What made this one work while all the others didn't? It wasn’t magic, the speaker pointed out. I checked the CloudWatch training logs and it just... worked.
It solved alignment: By using the compatible Nova Pro and Nova Lite model family, there was zero token mismatch.
It solved stability: The managed service handled all the complex orchestration behind the scenes.
Their training loss went from 0.05 to a very accurate 0.008 in just 4 epochs and 70 total steps.
The Results: Fast, Accurate, and Cheap
They ran the newly distilled Nova Lite model that took the jumbled input prompt and produced a perfectly formatted JSON output array of the desired entities.
The bottom line? The stats tell the story:
The Teacher (Nova): Achieved a 97% overall F1 score (96.3% English, 95.4% Japanese) but cost 14x more to run.
The Student (Nova Lite): Achieved a 95.085% overall F1 score (96.535% English, 93.635% Japanese) at the 1x baseline cost.
They hit their >95% accuracy goal while entirely eliminating the 14x operational cost overhead!.
A Deeper Look at the Errors
Always seeking improvement, the team took a further look at the little 1.9% accuracy disparity between teacher and student.
Language: The student did significantly worse on the Japanese text (93.6% vs 95.4% for the teacher). Their next immediate step to fill the gap is to extend and increase the modest 150-sample Japanese dataset.
Complexity: Approximately 20% of the student’s errors were in the “long-tail” or multi-part entities. For example elaborate layered instructions such as “Fragile; refrigerate below 4°C” In these tough edge instances, the teacher model’s richer baseline reasoning nevertheless came out on top.
Key Takeaways
Generative NER is a game-changer: By moving from typical BIO tagging to a Seq2Seq technique, you gain amazing flexibility when you need to handle complex multi-entity extractions.
Your task dictates your method: If you pick a Seq2Seq task you are forced in a Token-Based distillation technique. Just be ready for the brittle engineering and alignment needs that are part of it.
The real win is cost, not speed: The big 14x improvement was about operational savings. Both models had similar inference performance but the smaller Nova Lite reduced the hefty financial burden of provisioning large LLM throughput.
Offload the engineering: If your team spends 100% of its time resolving architecture incompatibilities, they’re not conducting data science. Get rid of those inflexible tokenizer bottlenecks altogether with managed services like AWS Bedrock.
Conclusion
Sitting in the audience in Kochi at the conclusion of the day, I was reminded that the road to a massive technical win is seldom a straight line. The team didn’t just build a 14x cheaper AI, they failed forward, knew when they were in an engineering trap, and pivoted to a managed solution that allowed them to focus on solving their underlying business challenge.
If you’re producing AI solutions, don’t be scared to change your architecture when operational costs start to threaten your business model. At times, the smartest technological option you can make is simply to let a managed service do the heavy lifting.
About the Author
As an AWS Community Builder, I enjoy sharing the things I've learned through my own experiences and events, and I like to help others on their path. If you found this helpful or have any questions, don't hesitate to get in touch! 🚀
🔗 Connect with me on LinkedIn
References
Event: AWS Community Day Kochi
Topic: 14x Cheaper AI: A Real-World LLM Distillation Case Study on Bedrock
Date: December 20, 2025
Top comments (0)