Introduction
When I first started building an AI-powered application using Amazon Bedrock, everything looked perfect.
The architecture was simple, clean, and easy to understand.
Client
│
AWS Lambda
│
Amazon Bedrock
│
Response
During development, I tested the application with only a few requests. AI responses were generated successfully, users received their results within seconds, and there were no noticeable performance issues. At that stage, I believed the architecture was production-ready.
However, reality was very different once multiple users started using the application simultaneously.
What worked flawlessly during development quickly became unreliable in production.
This article shares the real architectural challenge I encountered, why it happened, and how introducing Amazon SQS completely transformed the reliability and scalability of the application.
The Project
The application allows users to submit company information and receive an AI-generated due diligence report powered by Amazon Bedrock.
Each request involves several steps:
- Receiving the user's request
- Validating the payload
- Fetching additional company information
- Building a detailed prompt
- Sending the prompt to Amazon Bedrock
- Generating a large AI response
- Processing the output
- Storing the generated report
- Returning the final result to the user
Although this workflow sounds straightforward, AI inference is significantly slower than a typical REST API request.
A single report could take anywhere from 20 to 90 seconds depending on:
- Prompt size
- Amount of company data
- Model response length
- Current Bedrock workload
For one user, this was acceptable.
For dozens of concurrent users, it became a serious architectural problem.
My Initial Architecture
The first version of the system looked like this:
Client
│
▼
AWS Lambda
│
▼
Amazon Bedrock
│
▼
Return AI Response
The Lambda function performed every task synchronously.
It waited for Amazon Bedrock to finish generating the report before sending a response back to the client.
At first glance, this architecture seems reasonable.
Unfortunately, production traffic exposed several hidden problems.
The Production Issues
After deployment, more users began submitting report generation requests at the same time.
Almost immediately, several issues appeared.
1. Long Waiting Times
Users had to keep their browser open while waiting for the AI model to finish.
Some reports required over a minute to complete.
Many users assumed the application had frozen and refreshed the page, unknowingly creating duplicate requests.
2. Lambda Execution Time Increased
Since Lambda remained active throughout the entire AI generation process, execution times became unnecessarily long.
Long-running Lambda functions increase both execution cost and the likelihood of timeout failures.
3. Traffic Spikes Overwhelmed the System
During busy periods, many requests reached Lambda simultaneously.
Every Lambda invocation immediately attempted to call Amazon Bedrock.
This created sudden spikes in downstream traffic.
Even though AWS Lambda scales automatically, sending every request directly to the AI model at the same time is not always the most efficient or resilient design.
4. Temporary Failures Meant Lost Requests
Sometimes an AI request failed because of a temporary service interruption, a network issue, or a transient downstream error.
Without a buffering mechanism, the entire user request failed and had to be submitted again manually.
5. No Visibility Into Processing
The client had no way to determine whether:
- the request was queued,
- the AI model was still processing,
- the report had completed successfully, or
- the request had failed.
From the user's perspective, everything simply appeared to be "loading."
Understanding the Root Cause
The core problem was architectural rather than service-related.
The application relied entirely on synchronous processing.
The client waited for every step of the workflow to complete before receiving any response.
This approach is suitable for lightweight APIs but not for long-running AI workloads where inference time is unpredictable.
What the application needed was a way to separate request acceptance from request processing.
The Solution: Introducing Amazon SQS
Instead of invoking Amazon Bedrock immediately, I redesigned the architecture around asynchronous messaging using Amazon SQS.
The updated architecture became:
Client
│
▼
API Gateway
│
▼
AWS Lambda
│
▼
Amazon SQS
│
▼
Lambda Worker
│
▼
Amazon Bedrock
│
▼
Store Generated Report
│
▼
Notify / Poll Client
This small architectural change dramatically improved the overall system.
How the New Workflow Operates
Step 1
The client submits a report generation request.
Instead of waiting for AI processing to complete, the API immediately validates the request.
Step 2
The Lambda function creates a message containing the request details and places it into Amazon SQS.
The client instantly receives a confirmation along with a tracking ID.
The request is now safely stored in the queue.
Step 3
Lambda workers automatically consume messages from the queue.
Each worker independently processes a single report request.
Step 4
The worker invokes Amazon Bedrock to generate the AI report.
Since this happens in the background, users no longer have to wait for the model to finish before receiving an acknowledgment.
Step 5
Once processing completes, the report is stored (for example, in Amazon S3 or a database), and the request status is updated.
The client can either poll for completion or receive a notification when the report is ready.
Why Amazon SQS Made Such a Difference
Automatic Retry
Distributed systems occasionally experience temporary failures.
Instead of losing a request immediately, Amazon SQS keeps the message available.
If a Lambda worker encounters a transient error, the message becomes visible again after the visibility timeout expires, allowing another processing attempt.
This significantly improves resilience without requiring custom retry logic for every failure scenario.
Built-in Scalability
One of the biggest advantages of SQS is that it naturally absorbs sudden traffic spikes.
Instead of sending hundreds of AI requests directly to Amazon Bedrock at once, incoming requests are buffered in the queue.
Lambda workers then process messages at a pace the system can handle.
As demand grows, AWS automatically scales the number of Lambda workers consuming messages from the queue.
The application remains responsive because the API is no longer blocked by long-running AI inference.
Dead Letter Queue (DLQ)
No system can guarantee that every request will succeed.
Some messages may fail repeatedly because of invalid input, corrupted data, or unexpected processing errors.
By configuring a Dead Letter Queue (DLQ), failed messages are automatically moved aside after a defined number of unsuccessful processing attempts.
This prevents problematic messages from blocking the queue while preserving them for investigation and replay.
Instead of silently losing data, operations teams have full visibility into failed requests.
Improved Reliability
The queue acts as a protective buffer between the public API and the AI processing layer.
Even if Amazon Bedrock experiences temporary latency or downstream services become slower than usual, new user requests continue to be accepted and safely stored.
This decoupled architecture makes the entire system more resilient to fluctuations in traffic and temporary service disruptions.
Results After the Migration
After moving to an event-driven architecture with Amazon SQS, the application behaved much more predictably under load.
Some of the key improvements included:
- Immediate API responses instead of long waits.
- Better handling of concurrent user traffic.
- Reduced risk of request failures during temporary service interruptions.
- Automatic retry behavior for transient processing errors.
- Better visibility into request status.
- Isolation of failed messages using a Dead Letter Queue.
- Improved scalability without redesigning the application.
Most importantly, users no longer experienced failed requests simply because AI processing took longer than expected.
Lessons Learned
One of the biggest lessons I learned while building production AI systems is that the challenge is rarely the AI model itself—it is the architecture around it.
Amazon Bedrock provides powerful foundation models, but long-running AI inference should not be tightly coupled to user-facing APIs.
By introducing Amazon SQS, I transformed the application from a synchronous workflow into a resilient, event-driven system capable of handling production traffic more effectively.
If you're building AI applications on AWS, don't wait until production traffic exposes architectural bottlenecks. Designing with asynchronous processing, retries, buffering, and fault isolation from the start will save you significant time and improve the experience for both your users and your operations team.
Serverless AI applications are most successful when every component has a single responsibility, and Amazon SQS is one of the simplest yet most effective services for achieving that goal.
Top comments (1)
I was particularly intrigued by the production issues you encountered, especially the long waiting times and increased Lambda execution times, which are common pitfalls when dealing with synchronous architectures. The introduction of Amazon SQS as a message queueing service seems like a great solution to decouple the Lambda function from the AI generation process, allowing for greater scalability and reliability. I've had similar experiences with long-running tasks in my own applications, and implementing a queue-based architecture has been a game-changer. Did you consider implementing any additional features, such as retries or dead-letter queues, to further enhance the resilience of your application?