Storm Reply · Generative AI · AWS · 2026
Introduction
The digital publishing industry is undergoing a fundamental shift in how audiences consume news. Readers increasingly expect audio, not just articles read aloud, but engaging, podcast-style content that fits into their daily routines. For media platforms competing in this space, the ability to generate high-quality audio summaries at scale is becoming a decisive competitive advantage. The challenge is that AI models capable of doing this are proliferating rapidly, and choosing the wrong one, at scale, across hundreds of publishers, has a direct and compounding impact on listener engagement and editorial reputation.
The Problem
BotTalk is a Text-to-Speech platform built exclusively for digital publishers and media companies. With over 800 voices across 120+ languages, BotTalk converts written news into natural-sounding audio for publisher clients across Europe. As audience demand shifted toward richer formats, BotTalk expanded its offering to include podcast-style content summaries, conversational scripts that do not simply restate facts, but reframe news in a format a listener actually wants to hear.
This created an immediate operational problem. The market for foundation models is crowded and fast-moving. New models are released regularly, each with different strengths, cost profiles, and behavioural tendencies. BotTalk needed to know, with confidence, which model produces the best podcast-style summaries from news articles. Not on a generic benchmark. On their content, against their editorial standards, for their publishers.
The existing approach, manually testing models in a playground, reading outputs, forming opinions, could not answer that question reliably. It did not scale across content types or languages, produced results that varied by reviewer, and left no audit trail for decisions that directly affected publisher output quality. There was no repeatable process and no shared evidence base.
BotTalk's requirements were clear:
- Systematic comparison of multiple foundation models against the same article and scenario
- Objective quality scoring calibrated to podcast content, not generic text quality
- Accessible results for both technical and editorial stakeholders, without AWS expertise
- Cost control given that every model invocation carries a direct Bedrock API cost
- Repeatability so the same experiment can be run as new models enter the market
The Solution
Storm Reply designed and delivered a fully serverless AI evaluation pipeline on AWS, purpose-built to turn model selection from a subjective exercise into a structured, repeatable experiment.
The core decision was to build around AWS Step Functions as the orchestration layer, giving BotTalk a transparent, auditable workflow with built-in checkpointing. If any step fails, intermediate results are preserved in Amazon S3 and the pipeline can resume without repeating expensive model invocations. This was a direct response to BotTalk's need for cost control and reliability at scale.
For model access, Storm Reply chose Amazon Bedrock's Converse API over model-specific integrations. This decision was deliberate: a unified interface means BotTalk can add any new Bedrock model, Claude, Llama 3, Amazon Nova, DeepSeek R1, or whatever ships next, with a configuration change and no engineering work. The evaluation framework is future-proof by design.
The pipeline operates as follows:
- Prompt optimisation — A Prompt Agent powered by Amazon Bedrock refines the input instructions before any model is invoked, ensuring all models compete on equal footing under a well-structured prompt
- Parallel model invocation — Multiple foundation models are called simultaneously via AWS Lambda using Python's ThreadPoolExecutor, reducing total experiment time to that of the slowest model rather than the sum of all
- AI-powered scoring — A dedicated LLM-as-Judge instance evaluates every output across five dimensions: Accuracy, Fluency, Completeness, Neutrality, and Prompt Compliance. The rubric is deliberately strict — scores above 95 are almost never awarded — to force genuine differentiation rather than clustering all outputs around an unhelpful median
- Human-in-the-loop reporting — Results are consolidated into a self-contained HTML comparison report delivered via a presigned S3 URL within seconds. Editorial and technical stakeholders review outputs side by side, see colour-coded scores, and make the final model selection — bringing human editorial judgment into the decision at exactly the right moment, without requiring any AWS expertise
The entire stack — IAM roles, Lambda functions, Step Functions state machine, API Gateway, S3 buckets, and CloudWatch billing alarms — is codified in Terraform and deployable in a single command.
The Achievements
BotTalk moved from ad hoc playground testing to a structured, evidence-based model selection process within the project timeline of April to June 2026.
Before: Model selection relied on individual judgment, could not be repeated consistently, and produced no documented rationale. Editorial and technical teams had no shared framework for evaluating quality.
After: A single API call triggers a complete evaluation experiment. A clear comparison report is returned in seconds. Every decision is backed by scored, comparable data — with a human making the final call.
Concrete outcomes:
- Human-in-the-loop at the right stage — The pipeline automates everything up to the decision point. A clear comparison report is returned in seconds, giving editorial and technical stakeholders the scored evidence they need to make a confident, final model selection. Human judgment is preserved where it matters most — on the decision itself, not on running the experiment
- Repeatable experiments — The same evaluation can be run against any new model or content type with zero code changes, keeping BotTalk current as the Bedrock model catalogue evolves
- Editorial confidence — Publisher-facing teams can now point to scored evidence when explaining model choices, strengthening BotTalk's credibility with its clients
The project directly led to a Phase 2 engagement, in which Storm Reply extended the pipeline into audio quality assurance — automatically detecting mispronunciations in TTS output using Amazon Transcribe and Amazon Bedrock, reducing manual editorial review from 20+ flagged items per article to only the 2–3 genuine errors requiring human attention.
Reach Out to Us
Storm Reply is the AWS specialist within the Reply Group, supporting customers in designing and delivering scalable cloud and AI solutions — from early architecture through to production.
Interested in modernizing your cloud infrastructure and building enterprise-grade solutions? Storm Reply is driven by continuous learning and practical innovation. We specialize in designing and delivering scalable AWS architectures that support customers throughout their cloud journey, from early assessment to production-ready deployment.
With deep experience in AWS architecture, data engineering, and security best practices, we help enterprises migrate with confidence and move faster on their cloud transformation goals.
Let’s connect and explore how we can support your modernization initiatives.
🌐 Website: https://www.stormreply.cloud/
💼 LinkedIn: https://www.linkedin.com/company/storm-reply/posts/?feedView=all
Date: January 2026
Top comments (0)