DEV Community: saif ur rahman

Drowning in 10,000+ Pages? A Scalable AI Architecture for Turning Unstructured Documents into Actionable Knowledge

saif ur rahman — Thu, 03 Sep 2026 18:44:40 +0000

Introduction

Organizations across almost every industry eventually run into the same wall: information grows faster than any team can reasonably review it. A single business, research project, investigation, healthcare workflow, insurance operation, financial process, compliance program, or enterprise archive can accumulate thousands — sometimes tens of thousands — of pages of documents.

Those documents rarely arrive in one clean format. A typical collection might include PDFs, scanned PDFs, Word documents, spreadsheets, emails, reports, forms, tables, images, diagrams, handwritten notes, and historical records, mixing structured and unstructured data freely.

The hard part was never storage. The hard part is turning thousands of heterogeneous pages into something that can be searched, understood, connected, summarized, and verified. A simple document search system isn't enough for that job, and a large language model on its own isn't either. What actually works is a document intelligence and retrieval pipeline — an architecture that converts raw files into a structured knowledge layer, so that AI can reason over 10,000+ pages without ever needing the entire archive stuffed into a single prompt.

At a high level, the full pipeline looks like this:

Documents
   ↓
Document Processing
   ↓
OCR / Extraction
   ↓
Classification & Separation
   ↓
Chunking
   ↓
Embeddings
   ↓
Vector Search
   ↓
Evidence Retrieval
   ↓
AI Reasoning
   ↓
Summaries / Answers / Reports

This approach lets an application work with 10,000+ pages without repeatedly sending the entire document collection to a large language model.

1. The Real Problem Isn't Document Storage

Uploading 10,000 pages into cloud storage is trivial — S3 handles that without complaint. The difficulty starts the moment someone asks a real question: "Find everything related to the previous condition." "What changed between 2020 and 2024?" "Show all records that mention this subject." "Summarize the entire history."

At that point the system has to understand the content, not just hold onto it. It helps to think of the problem in layers:

                    10,000+ PAGES
                           │
        ┌──────────────────┼──────────────────┐
        ▼                  ▼                  ▼
     Storage            Processing          Understanding
        │                  │                  │
        ▼                  ▼                  ▼
       S3              OCR/Parsing          AI Models
                                               │
                                               ▼
                                         Retrieval
                                               │
                                               ▼
                                            Answer

The most important insight in the whole architecture is this: document storage, document understanding, retrieval, and reasoning are four different problems, and they shouldn't all be handled by one service.

2. Why 10,000+ Pages Break the Naive Approach

The obvious first idea is:

10,000 pages
     ↓
Large Language Model
     ↓
Summary

That architecture creates several problems:

Context limitations. Even models with very large context windows shouldn't be handed an entire archive for every operation.
Cost. Reprocessing the full corpus repeatedly gets expensive fast.
Latency. Massive prompts take longer to process.
Retrieval quality. When thousands of unrelated pages are dumped into one context, the model has to do the hard work of separating signal from noise itself.
Lack of provenance. A generic summary often can't tell you exactly which document or page backs up a given statement.
Repeated computation. Unchanged documents shouldn't have to be reanalyzed every time a new question comes in.

The better pattern flips the order of operations:

Process once
    ↓
Store structured knowledge
    ↓
Search efficiently
    ↓
Send only relevant evidence to the model

3. The Solution: A Knowledge Pipeline, Not a Prompt

Rather than treating uploaded files as one giant prompt, the goal is to convert them into a reusable knowledge system:

Raw Documents
      │
      ▼
Document Processing
      │
      ▼
Normalized Content
      │
      ├───────────────┐
      ▼               ▼
Structured Data      Chunks
      │               │
      ▼               ▼
 PostgreSQL       Embeddings
                      │
                      ▼
                 Vector Index
                      │
                      ▼
                 Retrieval
                      │
                      ▼
                 AI Reasoning

This produces a two-speed architecture. The first stage — ingestion — is expensive but happens mostly once. The second stage — search and question-answering — is fast and runs every time a user interacts with the system.

4. Stage One: Ingesting the Documents

Ingestion starts when a large collection is uploaded — a project folder full of PDFs, scans, spreadsheets, and images:

Case / Project / Dataset
│
├── document-001.pdf
├── document-002.pdf
├── report-003.docx
├── scan-004.pdf
├── image-005.png
├── spreadsheet-006.xlsx
└── ...

All original files go into durable object storage, organized by project:

s3://document-ai/

projects/
  PROJECT-001/

    original/
    processed/
    pages/
    ocr/
    chunks/
    summaries/
    reports/

The original data should always remain untouched. AI processing creates derived representations rather than replacing the source.

5. Determining What Each Document Actually Contains

A 10,000-page archive is almost never one uniform dataset:

10,000 pages
      │
      ├── Reports
      ├── Forms
      ├── Correspondence
      ├── Financial records
      ├── Technical records
      ├── Historical records
      ├── Images
      └── Scanned documents

The application therefore needs document classification. Once classified, each document carries metadata like:

{
  "document_id": "DOC-00021",
  "type": "technical_report",
  "date": "2025-04-12",
  "page_start": 143,
  "page_end": 161
}

Once documents are classified, downstream processing becomes much more intelligent.

6. Splitting Large Files into Logical Documents

A single large PDF often contains dozens of separate logical documents stitched together:

Pages 1–20
    → Report A

Pages 21–37
    → Report B

Pages 38–55
    → Form Set

Pages 56–92
    → Technical Documentation

Instead of asking "what is this 10,000-page file?" the system can ask "what is this 15-page report?" — a much easier question that makes classification, summarization, metadata extraction, and retrieval significantly more precise.

Managed document services such as Amazon Bedrock Data Automation can help with document processing and splitting workflows; AWS currently documents splitting for source documents up to 3,000 pages. For collections beyond that, the workload can be divided into multiple source files or processing units.

7. Handling Scanned Documents

A large share of enterprise archives aren't really text documents at all — they're images sitting inside PDFs:

PDF
 ↓
Scanned Page
 ↓
Image
 ↓
No searchable text

A normal PDF text parser may return "" even though a human can clearly read the page. This is why OCR is a required processing stage:

Scanned page
     ↓
OCR
     ↓
Machine-readable text

Amazon Textract is one AWS option for extracting text from documents, while open-source OCR engines such as PaddleOCR can provide another implementation path. The important architectural principle is: never discard the original page image after OCR. Store the original page alongside the OCR text, because OCR can contain errors.

8. OCR Is Not the Same as AI Understanding

These steps are often incorrectly treated as one operation. They are actually different:

Image
 ↓
OCR
 ↓
Text
 ↓
Embedding
 ↓
Vector search
 ↓
LLM reasoning

OCR answers: what text is visible?
Embedding answers: what is the semantic representation of this content?
Vector search answers: which stored pieces of content are semantically similar to this query?
LLM answers: what does the retrieved evidence mean in context?

This separation makes the architecture much easier to scale.

9. Documents Are Multimodal, Not Just Text

A single page may contain text, a table, a chart, an image, and a diagram all at once:

                    SOURCE PAGE
                         │
              ┌──────────┴──────────┐
              ▼                     ▼
            Text                  Visual
              │                     │
              ▼                     ▼
        Text Embedding       Multimodal Embedding

Amazon Nova Multimodal Embeddings is designed for multimodal retrieval and supports text, images, documents, video, and audio in a common embedding framework — especially valuable for collections containing important visual information.

10–11. Chunking: Turning Documents into Searchable Units

After extraction, large documents must be divided into smaller logical pieces. Instead of creating one embedding for a 20-page report:

20 pages
 ↓
1 vector

the report is split into logical chunks:

20 pages
 ↓
Section 1
Section 2
Section 3
Section 4
...
 ↓
Multiple vectors

There's no universal perfect chunk size — the better strategy is structure-aware chunking, respecting headings, paragraphs, sections, tables, page boundaries, and semantic meaning rather than cutting text blindly every N characters.

A practical starting point for many document-search systems is around 700–1,000 words per chunk with moderate overlap, but the real answer comes from benchmarking sizes like 500, 800, 1,200, and 1,500 words against real queries. The goal is maximum retrieval accuracy with minimum unnecessary context.

12–13. Embeddings: Turning Content into Searchable Meaning

A keyword search for "thermal issue" may fail to match "the machine experienced repeated overheating problems," even though the concepts are clearly related. A semantic embedding captures that relationship:

Text
 ↓
Embedding Model
 ↓
[0.14, -0.82, 0.31, ...]

The numbers themselves aren't meaningful to a human — their purpose is to place semantically related content near each other in vector space.

For text-heavy workloads, Amazon Titan Text Embeddings V2 is a practical option, supporting multiple output dimensions including 1024. For mixed text and visual content, Nova Multimodal Embeddings becomes attractive:

Text-heavy
    ↓
Titan Text Embeddings V2
    ↓
S3 Vectors

Multimodal
    ↓
Nova Multimodal Embeddings
    ↓
S3 Vectors

The final choice should be made through retrieval benchmarking rather than model specifications alone.

14–15. Vector Search and Metadata

Once millions of words become vectors, the application needs an efficient way to find similar content:

Document chunks
      ↓
Embeddings
      ↓
S3 Vectors
      ↓
Vector index

When a user searches "previous equipment failures," the system generates a query embedding and searches the index:

Question
   ↓
Embedding Model
   ↓
Query Vector
      ↓
S3 Vectors
      ↓
Top-K similar chunks

A vector alone isn't enough — every vector should carry metadata back to its source:

{
  "project_id": "PROJECT-001",
  "document_id": "DOC-00421",
  "document_type": "report",
  "year": 2024,
  "page_start": 143,
  "page_end": 145,
  "chunk_id": "CHUNK-00872",
  "source_s3_key": "projects/PROJECT-001/chunks/CHUNK-00872.json"
}

This hybrid approach — metadata filtering plus semantic similarity — is much stronger than pure vector search on its own.

16–17. Retrieve Evidence Before Generating

One of the key lessons from this architecture: do not send raw embedding vectors to the language model and expect it to summarize them. The correct pipeline is:

Source content
     ↓
Embedding
     ↓
Vector index
     ↓
Query
     ↓
Nearest vectors
     ↓
Original source text
     ↓
AI model
     ↓
Answer

For example, a user asking "Find records discussing previous system failures" triggers this flow:

User Question
       ↓
Query Embedding
       ↓
Vector Search
       ↓
Top 5–20 Results
       ↓
Retrieve Source Chunks
       ↓
Build Context
       ↓
Nova
       ↓
Answer

The final model receives something like:

Question:
Find records discussing previous system failures.

Evidence 1:
Report A, page 18
"Repeated overheating was observed..."

Evidence 2:
Maintenance Report, page 4
"Previous thermal failures..."

Evidence 3:
Inspection Report, page 12
"The system had experienced..."

The model can then reason over these specific pieces of evidence, and the answer stays traceable back to a document and page.

18–19. Hierarchical Summarization

A 10,000-page collection shouldn't have only one summary. Instead, create multiple levels:

10,000+ Pages
      ↓
Documents
      ↓
Document Summaries
      ↓
Category Summaries
      ↓
Year / Time Summaries
      ↓
Cross-Document Analysis
      ↓
Final Summary

Organized another way:

PROJECT
│
├── Technical
│   ├── 2022
│   ├── 2023
│   └── 2024
│
├── Financial
│   ├── 2022
│   ├── 2023
│   └── 2024
│
├── Operational
│   ├── 2022
│   └── 2024
│
└── Correspondence

Instead of feeding 500 documents into one huge prompt:

500 documents
      ↓
one huge prompt

the hierarchy compresses the corpus step by step:

500 documents
      ↓
500 document summaries
      ↓
20 category summaries
      ↓
5 yearly summaries
      ↓
1 final synthesis

This is effectively a compression hierarchy — the amount of information reaching the final reasoning model shrinks dramatically at each level while the important signal survives.

20–21. Batch Processing for Large Archives

A 10,000-page system shouldn't process every document through a synchronous API request. Instead:

Upload
  ↓
Create Processing Job
  ↓
Queue
  ↓
Workers
  ↓
Parallel Processing
  ↓
Store Results

Independent tasks run asynchronously:

Document 1 → Summary
Document 2 → Summary
Document 3 → Summary
...
Document 500 → Summary

Amazon Bedrock Batch Inference is designed for exactly this kind of asynchronous batch model processing using data stored in S3, and it's especially useful for work that doesn't depend on live user interaction.

22. Not Everything Should Be Batch

Batch processing is excellent for classification, summaries, extraction, and embedding generation. But a live user question needs a real-time path:

Understand query
      ↓
Search vectors
      ↓
Retrieve evidence
      ↓
Reason
      ↓
Respond

A good architecture therefore combines:

Batch processing
+
Real-time retrieval
+
Real-time reasoning

23. The Role of the Backend

A backend framework such as AdonisJS can function as the orchestration layer. Its role isn't to perform every AI operation itself:

AdonisJS
│
├── Authentication
├── Project Management
├── Upload APIs
├── Job Creation
├── Processing Status
├── AWS Integration
├── Search API
├── Report API
└── Business Logic

The heavy work runs through background workers.

24. Queue-Based Processing Architecture

A robust workflow chains jobs together:

Upload
  │
  ▼
Create Project
  │
  ▼
Document Processing Job
  │
  ▼
Classification Job
  │
  ▼
OCR Job
  │
  ▼
Chunking Job
  │
  ▼
Embedding Job
  │
  ▼
Vector Index Job
  │
  ▼
Summary Job
  │
  ▼
Analysis Job
  │
  ▼
Final Report Job

Each job maintains its own status, progress, attempts, error, start time, and completion time.

This means a failed operation doesn't require processing the entire archive again.

25. PostgreSQL, S3, and S3 Vectors Have Different Jobs

A common architecture mistake is trying to put everything into the vector database. Instead:

PostgreSQL stores projects, documents, document types, dates, structured fields, summaries, timeline, processing status, users, and jobs.

S3 stores original documents, page images, OCR files, chunks, and generated reports.

S3 Vectors stores embeddings, vector metadata, and the retrieval index.

This gives each storage layer a clear responsibility.

26. The Full Knowledge Lifecycle

                RAW INFORMATION
                       │
                       ▼
              Document Processing
                       │
                       ▼
               Normalized Content
                       │
             ┌─────────┴─────────┐
             ▼                   ▼
       Structured Data          Chunks
             │                   │
             ▼                   ▼
       PostgreSQL           Embeddings
                                 │
                                 ▼
                            S3 Vectors
                                 │
                                 ▼
                             Retrieval
                                 │
                                 ▼
                          Source Evidence
                                 │
                                 ▼
                             AI Reasoning
                                 │
                ┌────────────────┼───────────────┐
                ▼                ▼               ▼
             Answers         Summaries        Reports

27. Cost Optimization

Large-document AI systems should be designed around cost per useful answer, not simply cost per page. The expensive mistake:

10,000 pages
     ↓
Large model
     ↓
Every question

The scalable strategy:

Process once
     ↓
Store knowledge
     ↓
Retrieve only relevant information
     ↓
Use AI reasoning on small context

28. Use Different Models for Different Jobs

Classification
      ↓
Small / inexpensive model

High-volume extraction
      ↓
Small / inexpensive model

Document summary
      ↓
Cost-efficient model

Semantic search
      ↓
Embedding model

Complex synthesis
      ↓
More capable reasoning model

In the AWS ecosystem, Amazon Nova models can be divided according to task complexity rather than treating one model as the universal solution.

29. Managed Automation vs. a Custom Pipeline

Managed document intelligence:

S3
 ↓
BDA
 ↓
Structured document output
 ↓
Embeddings
 ↓
Vector search

Custom pipeline:

S3
 ↓
Parser
 ↓
OCR
 ↓
Classification
 ↓
Chunking
 ↓
Embeddings
 ↓
Vector search

Neither is universally superior — the choice depends on engineering budget, processing volume, document diversity, required accuracy, operational complexity, and AWS cost. A hybrid architecture is often the most practical choice.

30. The Hybrid Architecture

                         10,000+ PAGES
                                │
                                ▼
                               S3
                                │
                                ▼
                    Document Processing
                         BDA / OCR
                                │
                                ▼
                       Classification
                                │
                                ▼
                     Document Separation
                                │
                    ┌───────────┴───────────┐
                    ▼                       ▼
             Structured Data              Text
                    │                       │
                    ▼                       ▼
              PostgreSQL                 Chunking
                                            │
                               ┌────────────┴────────────┐
                               ▼                         ▼
                      Text Embeddings            Visual Embeddings
                               │                         │
                               └────────────┬────────────┘
                                            ▼
                                        S3 Vectors
                                            │
                                            ▼
                                      Semantic Search
                                            │
                                            ▼
                                      Evidence Retrieval
                                            │
                                            ▼
                                      Amazon Nova
                                            │
                          ┌─────────────────┼─────────────────┐
                          ▼                 ▼                 ▼
                       Answers          Analysis           Reports

31. Reliability Is More Important Than a Beautiful Demo

A production system needs to answer: where did this statement come from? Every important generated claim should be traceable to:

Project
 ↓
Document
 ↓
Page
 ↓
Chunk
 ↓
Original source

For example:

Finding:
The issue was documented before 2022.

Evidence:
Document: Maintenance Report
Page: 17
Date: 2021-08-04

This is much more trustworthy than a paragraph generated without provenance.

32. The Source-of-Truth Principle

AI should never become the permanent source of truth. The source of truth remains the original document:

Original
 ↓
OCR
 ↓
Extracted text
 ↓
Structured facts
 ↓
Embeddings
 ↓
Summaries
 ↓
AI analysis

When the AI says something important, the application should be able to travel backward through that chain.

33. Contradiction Detection, Timelines, and Missing Information

Once documents are structured and searchable, the same architecture supports higher-level analysis. For example:

Document A:
"System was operational."

Document B:
"System was offline."

Document C:
"Maintenance required due to failure."

The system can identify a potential contradiction — but this should be treated as an AI-generated analytical finding, not automatically as established truth. The application should show both source statements so the user can verify the issue.

Dates extracted across the archive can be assembled into a chronological layer:

2021-02-10 → First reported issue
2021-06-15 → Maintenance performed
2022-01-09 → New failure
2022-03-21 → Investigation
2023-04-17 → Final report

And gaps in the record can be flagged the same way — if the archive contains 2021, 2022, and 2024 reports but nothing for 2023, the AI can flag that 2023 records may be missing, again as a finding that requires verification rather than an absolute fact.

34. From Search Engine to Knowledge System

Early on, the application may look like:

Upload
+
Search

After building the complete pipeline, it becomes:

DOCUMENT INTELLIGENCE PLATFORM
│
├── Search
├── Question Answering
├── Summarization
├── Timeline
├── Classification
├── Structured Extraction
├── Contradiction Detection
├── Evidence Retrieval
├── Reports
└── Cross-Document Analysis

The vector database is only one component. The real product is the knowledge layer built around the document corpus.

35. A Practical MVP

Phase 1: S3 upload, PDF/document extraction, OCR, document classification, chunking, embeddings, S3 Vectors, semantic search.

Phase 2: Document summaries, category summaries, structured extraction, AI Q&A.

Phase 3: Timeline, contradiction detection, missing information, final reports.

This reduces implementation risk by sequencing the hardest infrastructure first and the more advanced analytical features later.

36. Recommended AWS Architecture

                         USER
                           │
                           ▼
                       Next.js
                           │
                           ▼
                       AdonisJS
                           │
            ┌──────────────┼──────────────┐
            ▼              ▼              ▼
          S3            PostgreSQL       SQS
            │                              │
            │                              ▼
            │                           Workers
            │                              │
            ▼                              ▼
     Document Processing          OCR / Classification
                                           │
                                           ▼
                                       Chunking
                                           │
                                           ▼
                                      Embeddings
                                           │
                                           ▼
                                      S3 Vectors
                                           │
                                           ▼
                                      Retrieval
                                           │
                                           ▼
                                         Nova
                                           │
                        ┌──────────────────┼─────────────────┐
                        ▼                  ▼                 ▼
                     Answers           Analysis           Reports

This architecture separates storage, processing, orchestration, retrieval, and reasoning.

37. Ten Rules for Building This Kind of System

Never treat 10,000 pages as one prompt. Break information into manageable, meaningful units.
Preserve the original documents. AI-generated derivatives should never replace source evidence.
OCR and embeddings solve different problems. OCR extracts text; embeddings create searchable semantic representations.
Vectors are for retrieval, not generation. Retrieve the source content before asking a reasoning model to answer.
Use metadata with vectors. Category, date, document ID, page and source information greatly improve retrieval.
Use batch processing for independent large-volume work. Don't make a single HTTP request responsible for processing an entire archive.
Use hierarchical summarization. Document → category → period → final synthesis.
Keep evidence provenance. Every important AI finding should be traceable to its source.
Use different models for different jobs. Cheap models for repetitive work; stronger models for difficult reasoning.
Benchmark the complete pipeline. Accuracy, retrieval quality, processing time, cost, and reliability matter more than any individual model specification.

38. The Final Architecture in One Picture

                ┌─────────────────────────────┐
                │       10,000+ PAGES         │
                │ PDFs / DOCX / Scans / Images│
                └──────────────┬──────────────┘
                               │
                               ▼
                         ┌───────────┐
                         │    S3     │
                         └─────┬─────┘
                               │
                               ▼
                    ┌─────────────────────┐
                    │ Document Processing │
                    │     BDA / OCR       │
                    └──────────┬──────────┘
                               │
                               ▼
                     ┌──────────────────┐
                     │ Classification   │
                     │ & Separation     │
                     └────────┬─────────┘
                              │
                ┌─────────────┴──────────────┐
                ▼                            ▼
       ┌─────────────────┐          ┌────────────────┐
       │ Structured Data │          │ Logical Chunks │
       └────────┬────────┘          └───────┬────────┘
                │                           │
                ▼                           ▼
          PostgreSQL                  Embedding Model
                                            │
                              ┌─────────────┴─────────────┐
                              ▼                           ▼
                       Text Embedding             Multimodal
                                                    Embedding
                              │                           │
                              └─────────────┬─────────────┘
                                            ▼
                                      ┌────────────┐
                                      │ S3 Vectors │
                                      └──────┬─────┘
                                             │
                                             ▼
                                      Query / Search
                                             │
                                             ▼
                                     Relevant Evidence
                                             │
                                             ▼
                                      Amazon Nova
                                             │
                     ┌───────────────────────┼──────────────────────┐
                     ▼                       ▼                      ▼
                  Q&A Answer             Analysis               Report

Conclusion

The 10,000+ page problem is not fundamentally a problem of storing large files. Storage is relatively easy.

The real challenge is transforming massive amounts of heterogeneous information into a system that can answer questions accurately, retrieve supporting evidence, understand relationships across documents, and generate useful summaries.

The solution is not:

10,000 pages → one giant AI prompt

It is:

10,000+ pages
      ↓
Document processing
      ↓
OCR / extraction
      ↓
Classification
      ↓
Logical separation
      ↓
Chunking
      ↓
Embeddings
      ↓
Vector indexing
      ↓
Semantic retrieval
      ↓
Relevant source evidence
      ↓
AI reasoning
      ↓
Answers + summaries + reports

This architecture turns an enormous document collection into a reusable knowledge layer. Once that knowledge layer exists, the same underlying infrastructure can support many applications across industries: enterprise archives, compliance, research, insurance, healthcare, financial analysis, engineering, investigations, operations, and other document-heavy workflows.

The key idea is simple: do the expensive document understanding once, store the resulting knowledge intelligently, retrieve only what matters, and use AI reasoning on the relevant evidence rather than the entire archive. That's what makes AI analysis of 10,000+ pages practical, scalable, and economically sustainable.

How Amazon SQS Saved My AI Application: Lessons Learned While Building a Production AI System on AWS

saif ur rahman — Wed, 15 Jul 2026 12:54:08 +0000

Introduction

When I first started building an AI-powered application using Amazon Bedrock, everything looked perfect.

The architecture was simple, clean, and easy to understand.

Client
   │
AWS Lambda
   │
Amazon Bedrock
   │
Response

During development, I tested the application with only a few requests. AI responses were generated successfully, users received their results within seconds, and there were no noticeable performance issues. At that stage, I believed the architecture was production-ready.

However, reality was very different once multiple users started using the application simultaneously.

What worked flawlessly during development quickly became unreliable in production.

This article shares the real architectural challenge I encountered, why it happened, and how introducing Amazon SQS completely transformed the reliability and scalability of the application.

The Project

The application allows users to submit company information and receive an AI-generated due diligence report powered by Amazon Bedrock.

Each request involves several steps:

Receiving the user's request
Validating the payload
Fetching additional company information
Building a detailed prompt
Sending the prompt to Amazon Bedrock
Generating a large AI response
Processing the output
Storing the generated report
Returning the final result to the user

Although this workflow sounds straightforward, AI inference is significantly slower than a typical REST API request.

A single report could take anywhere from 20 to 90 seconds depending on:

Prompt size
Amount of company data
Model response length
Current Bedrock workload

For one user, this was acceptable.

For dozens of concurrent users, it became a serious architectural problem.

My Initial Architecture

The first version of the system looked like this:

                Client
                   │
                   ▼
             AWS Lambda
                   │
                   ▼
          Amazon Bedrock
                   │
                   ▼
          Return AI Response

The Lambda function performed every task synchronously.

It waited for Amazon Bedrock to finish generating the report before sending a response back to the client.

At first glance, this architecture seems reasonable.

Unfortunately, production traffic exposed several hidden problems.

The Production Issues

After deployment, more users began submitting report generation requests at the same time.

Almost immediately, several issues appeared.

1. Long Waiting Times

Users had to keep their browser open while waiting for the AI model to finish.

Some reports required over a minute to complete.

Many users assumed the application had frozen and refreshed the page, unknowingly creating duplicate requests.

2. Lambda Execution Time Increased

Since Lambda remained active throughout the entire AI generation process, execution times became unnecessarily long.

Long-running Lambda functions increase both execution cost and the likelihood of timeout failures.

3. Traffic Spikes Overwhelmed the System

During busy periods, many requests reached Lambda simultaneously.

Every Lambda invocation immediately attempted to call Amazon Bedrock.

This created sudden spikes in downstream traffic.

Even though AWS Lambda scales automatically, sending every request directly to the AI model at the same time is not always the most efficient or resilient design.

4. Temporary Failures Meant Lost Requests

Sometimes an AI request failed because of a temporary service interruption, a network issue, or a transient downstream error.

Without a buffering mechanism, the entire user request failed and had to be submitted again manually.

5. No Visibility Into Processing

The client had no way to determine whether:

the request was queued,
the AI model was still processing,
the report had completed successfully, or
the request had failed.

From the user's perspective, everything simply appeared to be "loading."

Understanding the Root Cause

The core problem was architectural rather than service-related.

The application relied entirely on synchronous processing.

The client waited for every step of the workflow to complete before receiving any response.

This approach is suitable for lightweight APIs but not for long-running AI workloads where inference time is unpredictable.

What the application needed was a way to separate request acceptance from request processing.

The Solution: Introducing Amazon SQS

Instead of invoking Amazon Bedrock immediately, I redesigned the architecture around asynchronous messaging using Amazon SQS.

The updated architecture became:

                  Client
                     │
                     ▼
               API Gateway
                     │
                     ▼
                 AWS Lambda
                     │
                     ▼
                Amazon SQS
                     │
                     ▼
              Lambda Worker
                     │
                     ▼
             Amazon Bedrock
                     │
                     ▼
          Store Generated Report
                     │
                     ▼
             Notify / Poll Client

This small architectural change dramatically improved the overall system.

How the New Workflow Operates

Step 1

The client submits a report generation request.

Instead of waiting for AI processing to complete, the API immediately validates the request.

Step 2

The Lambda function creates a message containing the request details and places it into Amazon SQS.

The client instantly receives a confirmation along with a tracking ID.

The request is now safely stored in the queue.

Step 3

Lambda workers automatically consume messages from the queue.

Each worker independently processes a single report request.

Step 4

The worker invokes Amazon Bedrock to generate the AI report.

Since this happens in the background, users no longer have to wait for the model to finish before receiving an acknowledgment.

Step 5

Once processing completes, the report is stored (for example, in Amazon S3 or a database), and the request status is updated.

The client can either poll for completion or receive a notification when the report is ready.

Why Amazon SQS Made Such a Difference

Automatic Retry

Distributed systems occasionally experience temporary failures.

Instead of losing a request immediately, Amazon SQS keeps the message available.

If a Lambda worker encounters a transient error, the message becomes visible again after the visibility timeout expires, allowing another processing attempt.

This significantly improves resilience without requiring custom retry logic for every failure scenario.

Built-in Scalability

One of the biggest advantages of SQS is that it naturally absorbs sudden traffic spikes.

Instead of sending hundreds of AI requests directly to Amazon Bedrock at once, incoming requests are buffered in the queue.

Lambda workers then process messages at a pace the system can handle.

As demand grows, AWS automatically scales the number of Lambda workers consuming messages from the queue.

The application remains responsive because the API is no longer blocked by long-running AI inference.

Dead Letter Queue (DLQ)

No system can guarantee that every request will succeed.

Some messages may fail repeatedly because of invalid input, corrupted data, or unexpected processing errors.

By configuring a Dead Letter Queue (DLQ), failed messages are automatically moved aside after a defined number of unsuccessful processing attempts.

This prevents problematic messages from blocking the queue while preserving them for investigation and replay.

Instead of silently losing data, operations teams have full visibility into failed requests.

Improved Reliability

The queue acts as a protective buffer between the public API and the AI processing layer.

Even if Amazon Bedrock experiences temporary latency or downstream services become slower than usual, new user requests continue to be accepted and safely stored.

This decoupled architecture makes the entire system more resilient to fluctuations in traffic and temporary service disruptions.

Results After the Migration

After moving to an event-driven architecture with Amazon SQS, the application behaved much more predictably under load.

Some of the key improvements included:

Immediate API responses instead of long waits.
Better handling of concurrent user traffic.
Reduced risk of request failures during temporary service interruptions.
Automatic retry behavior for transient processing errors.
Better visibility into request status.
Isolation of failed messages using a Dead Letter Queue.
Improved scalability without redesigning the application.

Most importantly, users no longer experienced failed requests simply because AI processing took longer than expected.

Lessons Learned

One of the biggest lessons I learned while building production AI systems is that the challenge is rarely the AI model itself—it is the architecture around it.

Amazon Bedrock provides powerful foundation models, but long-running AI inference should not be tightly coupled to user-facing APIs.

By introducing Amazon SQS, I transformed the application from a synchronous workflow into a resilient, event-driven system capable of handling production traffic more effectively.

If you're building AI applications on AWS, don't wait until production traffic exposes architectural bottlenecks. Designing with asynchronous processing, retries, buffering, and fault isolation from the start will save you significant time and improve the experience for both your users and your operations team.

Serverless AI applications are most successful when every component has a single responsibility, and Amazon SQS is one of the simplest yet most effective services for achieving that goal.

Your Prompt Isn't the Problem: Why System Prompts Matter More Than User Prompts in Production AI Applications

saif ur rahman — Fri, 26 Jun 2026 03:32:00 +0000

Introduction

When developers first start building AI applications, they usually focus on one thing:

The prompt.

Questions like:

How can I make Claude respond better?
How do I reduce hallucinations?
Why is my AI giving inconsistent results?
Should I use Chain of Thought?

become common.

Most teams spend days optimizing user prompts.

Very few spend time designing system prompts.

And that's where the real problem begins.

Recently, while building an AI-powered Due Diligence and Compliance Reporting platform using Amazon Bedrock and Claude, we discovered that prompt quality wasn't our biggest issue.

The real issue was a lack of system-level instructions.

The Problem

Our application generated forensic risk reports.

The workflow was simple:

User Input
      ↓
Claude
      ↓
Generated Report

Users provided:

{
  "companyName": "Microsoft Corporation",
  "country": "United States"
}

along with intelligence gathered from:

Companies House
OFAC
OpenSanctions
News APIs
Regulatory Sources

The AI then generated a complete report.

Everything seemed fine.

Until we started testing at scale.

Symptoms We Observed

The exact same data often produced different outputs.

Sometimes Claude generated:

Low Risk

For the same company.

Minutes later:

Medium Risk

for nearly identical input.

Other times:

Sections appeared in different orders
Risk scores changed
HTML formatting broke
Compliance recommendations varied
Findings were summarized differently

The model wasn't hallucinating.

It was doing exactly what we asked.

The problem was that we hadn't told it enough.

The Original Prompt

Our first implementation looked like this:

Generate an integrity due diligence report for the company using the data below.

Then we appended the API results.

That was it.

No structure.

No scoring methodology.

No formatting rules.

No output constraints.

The model had too much freedom.

Why This Happens

LLMs are prediction engines.

If instructions are vague:

Generate a report

the model must decide:

Format
Structure
Tone
Risk methodology
Recommendation logic

on its own.

Different reasoning paths produce different outputs.

This creates inconsistency.

And inconsistency is dangerous in production systems.

The Real Solution

We stopped optimizing the user prompt.

Instead, we designed a comprehensive system prompt.

Architecture changed from:

User Prompt
      ↓
Claude

to:

System Prompt
      ↓
User Prompt
      ↓
Claude

The system prompt became the source of truth.

What We Added

Output Constraints

Instead of:

Generate a report

we specified:

Output MUST be valid HTML.
Do NOT use markdown.
Do NOT use emojis.
Do NOT use conversational language.

Now every response followed the same format.

Fixed Section Order

We enforced:

1. Executive Summary
2. Entity Overview
3. Registry Findings
4. Sanctions Analysis
5. PEP Analysis
6. Litigation Review
7. Adverse Media Review
8. Risk Assessment
9. Recommendation

The model could no longer rearrange sections.

Deterministic Risk Scoring

Before:

Assess risk.

After:

Sanctions = 30%
PEP = 20%
Corruption = 20%
Litigation = 15%
Media = 15%

Every report now followed the same methodology.

Anti-Hallucination Rules

One of the most important additions was:

Do not invent information.
Use only provided data.
If data is unavailable, explicitly state:
"No data available from provided sources."

This dramatically improved reliability.

Before vs After

Before

Medium Risk

Reason:
Potential concerns observed.

No explanation.

No evidence.

No consistency.

After

Risk Score: 25

Sanctions:
0/100

Evidence:
No OFAC matches found.

Source:
OFAC API

Now every score was traceable.

The Hidden Benefit

Most teams think prompts only improve output quality.

In reality, strong system prompts also improve:

Maintainability

When requirements change:

Add ownership analysis

you update one system prompt.

Not every user prompt.

Debugging

When issues occur:

Why did risk increase?

you can inspect scoring rules directly.

Compliance

Auditors want repeatable processes.

System prompts create consistency.

Ad hoc prompting does not.

A Production Pattern

Today our AI architecture looks like this:

System Prompt
      ↓
API Data
      ↓
User Instructions
      ↓
Claude
      ↓
Structured HTML Report

The system prompt defines behavior.

The user prompt provides context.

This separation dramatically improves reliability.

Lessons Learned

The biggest mistake we made was treating prompts like chat messages.

Production AI systems are not chatbots.

They are software systems.

Software systems require:

Rules
Constraints
Validation
Predictability
Repeatability

System prompts provide those guarantees.

Best Practices for Production AI

1. Keep User Prompts Small

User prompts should contain:

Data
Context
Specific Request

Nothing more.

2. Move Rules to System Prompts

Examples:

Output format
Scoring logic
Compliance requirements
Validation rules

3. Prevent Hallucinations Explicitly

Always include:

Do not invent information.

4. Define Failure Behavior

Specify:

If data unavailable:
State that clearly.

Never leave the model guessing.

5. Standardize Output

Use:

JSON
HTML
XML
Markdown

but choose one and enforce it.

Final Thoughts

Many AI teams spend weeks optimizing prompts.

Few invest time designing system prompts.

Yet system prompts are often the difference between:

Interesting Demo

and

Production Application

If your AI outputs are inconsistent, unpredictable, or difficult to maintain, don't start by rewriting your user prompts.

Start by asking:

Does my model actually know the rules it's supposed to follow?

Because most of the time, the prompt isn't the problem.

The missing system prompt is.

The Day DNS Broke Our Deployment: Solving a Serverless Framework S3 Resolution Failure on AWS

saif ur rahman — Wed, 24 Jun 2026 12:03:50 +0000

Introduction

As engineers, we often spend hours optimizing code, improving prompts, and scaling infrastructure.

But sometimes the biggest production issues come from something much simpler.

A DNS lookup.

Recently, while deploying a serverless AI application to AWS, I encountered an error that completely blocked deployment.

The application hadn't changed.

AWS was healthy.

Permissions were correct.

The deployment package was valid.

Yet every deployment failed.

The error looked like this:

Error:
getaddrinfo EAI_AGAIN serverless-framework-deployments-eu-north-1-xxxxxxxx.s3.eu-north-1.amazonaws.com

At first glance, it looked like an AWS outage.

It wasn't.

This is the story of how a simple DNS resolution issue brought an entire deployment pipeline to a halt—and how we fixed it.

The Project

The application was an AI-powered Due Diligence Platform built with:

AWS Lambda
Amazon Bedrock
Amazon DynamoDB
Amazon SQS
Serverless Framework

Deployment flow:

Developer
     ↓
Serverless Framework
     ↓
S3 Deployment Bucket
     ↓
CloudFormation
     ↓
Lambda Functions

Every deployment package is first uploaded to an S3 bucket created by the Serverless Framework.

Only after the upload succeeds does CloudFormation update the stack.

The Error

During deployment, the terminal suddenly returned:

serverless deploy

✖ Error:
getaddrinfo EAI_AGAIN serverless-framework-deployments-eu-north-1-xxxxxxxx.s3.eu-north-1.amazonaws.com

The deployment stopped immediately.

No Lambda updates.

No CloudFormation changes.

Nothing.

First Assumption: AWS Was Down

The first thing I checked was AWS Service Health.

Everything was operational.

S3 was healthy
CloudFormation was healthy
Lambda was healthy

No incidents were reported.

Second Assumption: IAM Permissions

The next suspect was permissions.

I verified:

aws sts get-caller-identity

Response:

{
  "Account": "123456789012",
  "Arn": "arn:aws:iam::123456789012:user/developer"
}

Credentials were valid.

Permissions were correct.

Still failing.

Third Assumption: Serverless Framework Bug

I upgraded Serverless Framework.

npm install -g serverless

Deployment still failed.

The Real Problem

The key clue was:

EAI_AGAIN

This is not an AWS error.

It is a DNS resolution error.

Specifically:

EAI_AGAIN
=
Temporary DNS lookup failure

The operating system could not resolve the S3 endpoint hostname.

The request never reached AWS.

How We Confirmed It

I manually tested DNS resolution:

nslookup google.com

Intermittent failures appeared.

Then:

nslookup s3.eu-north-1.amazonaws.com

The same issue occurred.

This confirmed that the problem existed locally.

Not in AWS.

Root Cause

The machine was using an unstable DNS resolver.

Under heavy network usage, DNS lookups occasionally timed out.

When Serverless Framework attempted to upload artifacts to S3:

Serverless
      ↓
DNS Lookup
      ↓
Failure
      ↓
Deployment Stops

No connection to AWS was ever established.

The Fix

We switched to reliable public DNS servers.

Linux:

sudo nano /etc/resolv.conf

Added:

nameserver 8.8.8.8
nameserver 1.1.1.1

Then restarted networking:

sudo systemctl restart NetworkManager

Validation

After updating DNS:

nslookup s3.eu-north-1.amazonaws.com

Returned instantly.

Deployment succeeded:

serverless deploy

Output:

✔ Service deployed successfully

Additional Improvements

To avoid future issues, we added several safeguards.

Retry Logic

serverless deploy || serverless deploy

Useful for CI/CD jobs when transient network issues occur.

Connectivity Check

Before deployment:

curl https://s3.eu-north-1.amazonaws.com

If connectivity fails:

Stop deployment

This prevents wasting build minutes on doomed deployments.

AWS Credential Validation

Added:

aws sts get-caller-identity

to deployment pipelines.

This immediately detects expired or invalid credentials.

Lessons Learned

The biggest lesson was simple:

Not every AWS deployment error is actually an AWS problem.

Sometimes:

DNS fails
Local networking fails
VPNs interfere
Corporate firewalls interfere

And the cloud gets blamed.

Production Debugging Framework

When deployment issues occur, I now follow this order:

Step 1 — Validate AWS Credentials

aws sts get-caller-identity

Step 2 — Validate Internet Connectivity

ping google.com

Step 3 — Validate DNS

nslookup s3.eu-north-1.amazonaws.com

Step 4 — Validate AWS Services

aws s3 ls

Step 5 — Run Deployment

serverless deploy

This process has saved hours of troubleshooting.

Final Thoughts

As engineers, we often expect complex problems to have complex causes.

This incident reminded me that some of the most disruptive failures originate from the most basic layers of infrastructure.

A single DNS lookup failure stopped an entire deployment pipeline.

The code was correct.

AWS was healthy.

The architecture was sound.

But none of that mattered until the network could resolve a hostname.

Sometimes the fastest fix isn't changing code.

It's understanding where the request actually fails.

Key Takeaway

Before blaming AWS:

Check credentials
Check connectivity
Check DNS
Check local networking
Then investigate cloud services

You'll save yourself hours of debugging.

Building an AI-Powered Risk Intelligence System Using Serverless Architecture

saif ur rahman — Fri, 17 Apr 2026 17:20:07 +0000

Introduction

Organizations today require faster, more reliable ways to assess risk across entities such as companies, vendors, and partners. Traditional due diligence processes rely heavily on manual effort, fragmented data sources, and static reporting, which limits scalability and slows decision-making.

An AI-powered risk intelligence system solves this by automating data collection, analysis, and reporting. When combined with a serverless architecture, it becomes highly scalable, cost-efficient, and resilient without the need to manage infrastructure.

This article explains not only the concept but also how to practically achieve this using AWS services, focusing on architecture, services, and flow in a clear and implementation-oriented manner.

Understanding the Goal

The system aims to:

Collect data from multiple external sources
Analyze risk signals using AI
Apply consistent scoring logic
Generate structured reports automatically
Scale without manual infrastructure management

End-to-End Flow (Simple Overview)

A request is submitted (e.g., company name)
The system queues the request for processing
Background workers fetch data from APIs
AI analyzes the data and generates a report
The report is stored and made available to users

How to Achieve This Using AWS Services

1. Request Handling Layer

At the entry point, you need a way to accept incoming requests.

You can use:

Amazon API Gateway → to expose an HTTP endpoint
AWS Lambda → to process incoming requests

What happens here:

The user sends a request (company name, country, etc.)
Lambda validates the request
A unique report ID is generated
The request is stored for tracking
A message is sent to a queue for processing

This ensures the system responds quickly without waiting for heavy processing.

2. Asynchronous Processing with Queue

Instead of processing everything immediately, the request is placed in a queue.

You can use:

Amazon SQS (Simple Queue Service)

Why this is important:

Prevents timeouts
Handles high traffic smoothly
Allows retry if something fails
Decouples request from processing

The queue acts as a buffer between incoming requests and background workers.

3. Worker Layer (Background Processing)

The actual processing happens in a worker.

You can use:

AWS Lambda (triggered by SQS)

What the worker does:

Reads message from queue
Calls multiple external APIs
Collects raw data
Handles failures safely
Prepares data for AI processing

This layer is the core of data aggregation.

4. External Data Integration

The worker integrates with multiple external sources such as:

Sanctions databases
Watchlists
Corporate registries
News and media APIs

Best practices:

Call APIs in parallel (faster execution)
Use safe wrappers (so one failure doesn’t break everything)
Log responses for traceability
Normalize data into a consistent structure

5. Data Normalization

Different APIs return different formats. Before sending data to AI, you must standardize it.

This step ensures:

Consistent structure
Easier AI understanding
Better accuracy in results

Typical normalized structure includes:

Input data
Sanctions data
PEP/watchlist data
Corporate registry data
News/media data

6. AI Processing Layer

This is where intelligence is applied.

You can use:

Amazon Bedrock (for accessing foundation models)

What happens here:

The normalized data is sent to the model
A structured prompt guides the model
The model analyzes risk indicators
Assigns scores per category
Generates a structured report (HTML or text)

Key advantage:

No need to train your own model
Access to advanced models through API
Fast integration with serverless systems

7. Report Generation

The AI generates a structured report, typically in:

HTML format (for web display)
Optional PDF format (for sharing)

Reports usually include:

Executive summary
Risk analysis sections
Scoring tables
Final recommendation

8. Storage Strategy

You need to store both metadata and reports.

Metadata Storage

Use:

Amazon DynamoDB

Store:

Report ID
Status (Pending, Processing, Completed)
Risk level
Timestamps

Report Storage

Use:

Amazon S3

Store:

HTML reports
PDF files

Why separate storage:

DynamoDB is optimized for quick lookups
S3 is optimized for large file storage

9. Status Tracking

Users should be able to check report progress.

You can implement:

API to fetch report status
Query DynamoDB using report ID

Possible states:

PENDING
PROCESSING
COMPLETED
FAILED

10. Error Handling and Reliability

In distributed systems, failures are expected.

Best practices:

Use retry mechanisms (built into SQS + Lambda)
Wrap API calls in safe handlers
Log errors properly
Avoid system-wide failure due to one API

11. Security Considerations

Use IAM roles to control access
Secure API endpoints
Encrypt data in transit and at rest
Avoid exposing sensitive data

Why Serverless Works Best Here

Serverless architecture provides:

Automatic Scaling

Handles thousands of requests without manual intervention

Cost Efficiency

You only pay when the system runs

No Infrastructure Management

No servers to maintain or monitor

High Availability

Built-in fault tolerance across services

Key Design Principles

Decoupling

Each component works independently (API, queue, worker)

Fault Tolerance

Failures are isolated and handled gracefully

Deterministic AI Output

Strict prompts ensure consistent and reliable reports

Performance Optimization

Parallel API calls reduce processing time

Challenges and Practical Solutions

Challenge: External APIs are unreliable

Solution: Use safe wrappers and fallback logic

Challenge: Large reports

Solution: Store in S3 instead of database

Challenge: Inconsistent data formats

Solution: Strong normalization layer

Challenge: AI unpredictability

Solution: Use structured prompts and constraints

Real-World Use Cases

KYC and AML screening
Vendor risk assessment
Investment due diligence
Compliance monitoring
Third-party verification

Future Enhancements

Real-time monitoring and alerts
Risk dashboards with analytics
Entity matching using embeddings
Continuous data refresh pipelines

Conclusion

Building an AI-powered risk intelligence system using serverless architecture is both practical and powerful. By combining AWS services with generative AI, it is possible to create a system that is scalable, reliable, and capable of producing high-quality, structured risk reports automatically.

The key lies in designing a clean flow:

Accept request
Queue it
Process asynchronously
Aggregate data
Apply AI
Store and deliver results

This approach transforms traditional due diligence into a modern, intelligent, and automated system capable of supporting real-world compliance and risk decision-making at scale.

Struggling with AI Hallucinations? Here’s How I Solved It in Production

saif ur rahman — Wed, 01 Apr 2026 09:06:46 +0000

When I started building real-world Generative AI applications, everything seemed promising at first. The model responses were fluent, confident, and surprisingly helpful.

But very quickly, a serious problem started to appear.

The AI was giving wrong answers with full confidence.

At times, it would:

Invent facts that didn’t exist
Provide outdated or irrelevant information
Generate responses that sounded correct but were completely inaccurate

This is what we call hallucination in Generative AI and it becomes a major issue when you move from experiments to production systems.

In this article, I’ll share what caused hallucinations in my system and how I fixed them using practical, production-ready approaches.

The Problem: Confident but Incorrect AI

The biggest issue with hallucinations is not just that the AI is wrong it’s that it sounds right.

For example, a user might ask:

“What is the refund policy for my subscription?”

Instead of saying “I don’t know,” the model might generate a completely fabricated policy.

This creates serious risks:

Loss of user trust
Incorrect business decisions
Poor customer experience

I realized quickly that relying only on a language model was not enough for real applications.

Why Hallucinations Happen

After analyzing the system, I found a few key reasons.

1. No Access to Real Data

The model was answering based on its training data, not my application’s actual data.

So it tried to “guess” answers.

2. Poor Prompt Design

My prompts were too open-ended.

I wasn’t guiding the model properly, which allowed it to generate uncontrolled responses.

3. Too Much Context or Irrelevant Data

Sometimes I was passing too much or low-quality context, which confused the model.

4. No Validation Layer

There was no system to verify whether the answer was correct before returning it to the user.

The Solution: What Actually Worked

Fixing hallucinations required a combination of techniques, not just one change.

1. Implementing Retrieval-Augmented Generation (RAG)

The biggest improvement came from moving to a RAG-based architecture.

Instead of letting the model generate answers freely, I forced it to use retrieved documents as context.

New flow:

User Query
   ↓
Retrieve Relevant Documents
   ↓
Send Context + Query to Model
   ↓
Generate Answer Based on Context

This ensured that responses were grounded in real data.

2. Strict Prompt Engineering

I changed my prompts to be more controlled and restrictive.

Example:

You are an AI assistant.

Answer ONLY using the provided context.
If the answer is not found, say:
"I cannot find the answer in the provided data."

This single change reduced hallucinations significantly.

3. Limiting Context to Relevant Data

Instead of sending large amounts of data, I:

Retrieved only top relevant documents
Filtered out noisy or irrelevant content

This improved both accuracy and performance.

4. Adding a Confidence and Fallback Mechanism

I introduced fallback logic:

If confidence is low → Ask user for clarification
If no relevant data → Return safe response
If uncertain → Escalate to human

This prevented the system from guessing.

5. Using Structured Outputs

Instead of free-form text, I started using structured responses:

{
  "answer": "...",
  "source": "...",
  "confidence": "high"
}

This made it easier to validate and debug responses.

6. Continuous Monitoring and Feedback

I added logging and monitoring to track:

Incorrect responses
User feedback
Edge cases

Over time, this helped improve the system significantly.

Real Impact After Fixing Hallucinations

After applying these changes, I saw clear improvements:

More accurate responses
Reduced false information
Better user trust
More stable production behavior

The system became reliable enough for real users not just demos.

Key Lessons I Learned

Looking back, here are the most important lessons:

Never trust raw LLM output in production
Always ground responses in real data
Prompt design matters more than expected
Less context is often better than more
Add fallback mechanisms early
Monitor everything

Final Thoughts

Hallucinations are one of the biggest challenges in building real-world AI systems.

But they are not impossible to solve.

With the right architecture especially using RAG, structured prompts, and validation layers you can turn an unreliable system into a production ready solution.

If you’re building AI applications today, don’t aim for perfect models.

Aim for controlled, reliable systems.

That’s what actually works in production.

Why Traditional Call Centers Are Dying (And What Replaces Them)

saif ur rahman — Tue, 31 Mar 2026 10:43:48 +0000

For decades, call centers have been the backbone of customer support. Long queues, scripted conversations, and “press 1 for support” menus became the standard experience across industries.

But today, that model is slowly breaking down.

Customers expect faster responses, more personalized interactions, and support that feels natural not mechanical. Businesses, on the other hand, are looking for ways to reduce operational costs while improving efficiency.

This shift is driving a major transformation: traditional call centers are fading, and a new generation of intelligent, cloud-powered systems is taking their place.

The Problems with Traditional Call Centers

Traditional call centers were designed for a different era when customer expectations were lower and technology was limited.

Today, their limitations are becoming more visible.

Rigid IVR Systems

Most systems rely on fixed IVR menus:

Press 1 for billing
Press 2 for support
Press 3 for sales

This approach often frustrates users, especially when their issue doesn’t fit neatly into predefined options.

Long Wait Times

Customers are frequently placed in queues, waiting minutes or even longer to reach an agent.

This leads to:

Poor customer satisfaction
Increased call abandonment rates

Lack of Personalization

Traditional systems treat every customer the same way.

They lack awareness of:

Customer history
Previous interactions
Account context

This forces customers to repeat information again and again.

High Operational Costs

Maintaining a large team of agents is expensive.

Costs include:

Staffing
Training
Infrastructure
Maintenance

Scaling such systems becomes difficult and inefficient.

What Customers Expect Today

Modern users expect a completely different experience.

They want:

Instant responses
Natural conversations (not menu-driven)
24/7 availability
Personalized support
Seamless transitions between channels

In short, they expect support systems to be as intelligent and responsive as the apps they use daily.

What is Replacing Traditional Call Centers?

Traditional systems are being replaced by cloud-based, AI-powered customer engagement platforms.

These systems combine:

Cloud contact centers
Generative AI
Automation workflows
Real-time data integration

Together, they create a smarter and more flexible support experience.

The Rise of Cloud Contact Centers

Cloud-based contact centers eliminate the need for on-premise infrastructure.

They offer:

Scalability on demand
Global availability
Easy integration with other systems

Instead of managing hardware, businesses can focus on improving customer experience.

Generative AI is Changing Everything

One of the biggest shifts is the introduction of Generative AI into customer support.

Unlike traditional systems, AI can:

Understand natural language
Generate dynamic responses
Handle complex queries
Maintain conversational context

For example, instead of navigating menus, a user can simply say:

“I was charged twice. Can you help me fix this?”

The system can understand the issue and respond intelligently.

From Chatbots to Intelligent AI Assistants

Early chatbots were rule-based and limited.

Modern AI assistants are far more advanced.

They can:

Understand intent
Perform actions
Ask follow-up questions
Handle multi-step workflows

For example:

Change subscription
Apply discount
Update payment

All within a single interaction.

Automation is Reducing Manual Work

Automation plays a key role in replacing traditional systems.

Tasks that previously required human agents can now be automated:

Ticket creation
Account updates
Status checks
Basic troubleshooting

This reduces workload on support teams and speeds up response times.

Smarter Call Routing and Decision Making

Modern systems use intelligent routing instead of static rules.

Calls can be routed based on:

Customer priority
Issue type
Agent expertise
Real-time availability

This ensures customers are connected to the right agent faster.

Omnichannel Support is the New Standard

Customers no longer rely only on phone calls.

They expect support across multiple channels:

Chat
Email
Mobile apps
Social platforms

Modern systems unify all these channels into a single experience.

Benefits of Modern AI-Powered Support Systems

The shift away from traditional call centers brings significant advantages.

Better Customer Experience

Faster responses
Natural conversations
Personalized interactions

Reduced Costs

Fewer manual processes
Lower infrastructure costs
Efficient resource utilization

Scalability

Handle thousands of requests simultaneously
No need for large physical infrastructure

Improved Efficiency

Faster resolution times
Reduced agent workload
Smarter decision-making

Challenges in the Transition

While the shift is powerful, it comes with challenges:

Designing intelligent workflows
Handling complex edge cases
Ensuring accuracy in AI responses
Managing data privacy and compliance

Organizations need to carefully design and monitor these systems to ensure reliability.

The Future of Customer Support

We are moving toward a future where:

AI handles most routine interactions
Human agents focus on complex cases
Systems understand user intent deeply
Conversations feel natural and seamless

Customer support is no longer just a service it is becoming a key part of the product experience.

Final Thoughts

Traditional call centers are not disappearing overnight, but their role is rapidly changing.

Rigid systems, long wait times, and manual processes are being replaced by intelligent, cloud-based, and AI-driven solutions.

For developers and businesses, this shift represents a major opportunity to build systems that are not only efficient but also genuinely helpful.

The future of customer support is not about handling more calls.

It’s about building smarter systems that solve problems before customers even feel the need to call.

How to Build a Smart Call Routing System in Amazon Connect

saif ur rahman — Tue, 31 Mar 2026 10:13:56 +0000

Customer experience is no longer just about answering calls it’s about routing customers to the right place, at the right time, with the right context.

Traditional call routing systems often rely on rigid IVR menus and predefined rules, which can frustrate users and increase wait times. Modern cloud-based systems allow us to design intelligent, flexible, and scalable routing strategies.

In this article, I’ll walk through how to design and build a smart call routing system using Amazon Connect, along with practical considerations and best practices.

What is Smart Call Routing?

Smart call routing is the process of directing incoming customer calls based on:

Customer intent
Call context
Business rules
Agent availability
Customer priority or profile

Instead of sending every caller through the same flow, smart routing ensures that each customer is handled efficiently and appropriately.

Why Traditional Routing Falls Short

Most traditional systems rely on:

Static IVR menus
Fixed routing rules
Limited personalization

This often results in:

Long wait times
Misrouted calls
Poor customer experience
Increased operational overhead

Smart routing introduces dynamic decision-making, which significantly improves efficiency and user satisfaction.

Key Components of a Smart Routing System

When building a smart routing system in Amazon Connect, there are a few core components to understand.

Contact Flows

Contact flows define how calls are handled inside Amazon Connect.

They allow you to:

Capture user input
Define routing logic
Integrate backend services
Control the overall call experience

Queues

Queues represent groups of agents.

Calls can be routed based on:

Department (billing, support, sales)
Skill set
Priority

Routing Profiles

Routing profiles determine how agents receive contacts.

They help manage:

Agent workload
Queue priority
Call distribution

AWS Lambda Integration

Lambda allows you to introduce dynamic logic into your routing system.

You can use it to:

Fetch customer data
Validate inputs
Apply intelligent routing rules
Integrate external systems

Designing a Smart Call Routing Flow

A well-designed routing system follows a structured approach.

Step 1: Capture Customer Intent

At the start of the call, identify the reason for the call.

This can be done using:

Keypad input (DTMF)
Voice input
AI-based intent recognition

Example:

Press 1 for Billing
Press 2 for Technical Support
Press 3 for Sales

Step 2: Identify Customer Context

Use backend systems to retrieve:

Customer profile
Previous interactions
Account status

This helps personalize the routing decision.

Step 3: Apply Routing Logic

Based on intent and context:

Premium customers → Priority queue
Technical issues → Specialized agents
General queries → Standard support

Step 4: Route to the Appropriate Queue

Send the call to the correct queue.

Amazon Connect automatically assigns the call to available agents based on routing rules.

Step 5: Handle Fallback Scenarios

Always include fallback mechanisms:

Offer callback if wait time is long
Redirect to voicemail if no agents are available
Route to a default queue in case of failure

Example Smart Routing Flow

Incoming Call
   ↓
Capture Intent
   ↓
Fetch Customer Data (Lambda)
   ↓
Apply Routing Logic
   ↓
Route to Queue
   ↓
Agent Interaction

Adding Intelligence with Dynamic Routing

To make your system more advanced, you can introduce dynamic routing.

This includes:

Prioritizing high-value customers
Routing based on real-time queue load
Using historical data
Integrating AI for intent detection

For example, a returning customer with an unresolved issue can be routed directly to a senior agent.

Example Lambda Logic

exports.handler = async (event) => {

  const customerId = event.customerId;

  const customer = await getCustomerData(customerId);

  let queue = "GeneralSupport";

  if (customer.isPremium) {
    queue = "PrioritySupport";
  } else if (customer.issueType === "billing") {
    queue = "BillingQueue";
  }

  return {
    queueName: queue
  };
};

This shows how routing decisions can be made dynamically using backend logic.

Best Practices for Smart Call Routing

To build an effective system:

Keep IVR menus simple and user-friendly
Avoid overly complex call flows
Use Lambda for dynamic decision-making
Monitor performance and adjust routing rules
Implement fallback options
Continuously improve based on analytics

Common Challenges

While building smart routing systems, you may face:

Complex call flow design
Handling edge cases
Balancing automation and human interaction
Maintaining low latency
Ensuring reliability

Proper planning and testing help overcome these challenges.

Benefits of Smart Call Routing

A well-designed routing system provides:

Faster resolution times
Improved customer satisfaction
Better agent utilization
Reduced operational costs
Scalable and flexible architecture

Final Thoughts

Smart call routing is a critical component of modern customer support systems.

With Amazon Connect, it becomes possible to design intelligent and scalable routing strategies without managing infrastructure.

By combining contact flows, backend logic, and real-time data, you can build a system that not only routes calls efficiently but also enhances the overall customer experience.

As customer expectations continue to grow, investing in intelligent routing systems is no longer optional it is essential.

Integrating Generative AI with Amazon Connect for Smarter Customer Support

saif ur rahman — Tue, 31 Mar 2026 06:03:43 +0000

Customer expectations have changed significantly in recent years. Users no longer want to wait in long queues or navigate complex IVR systems. They expect fast, intelligent, and personalized support experiences.

This is where combining cloud contact centers with Generative AI becomes a game-changer.

By integrating Generative AI with Amazon Connect, organizations can transform traditional support systems into intelligent, automated, and highly scalable customer engagement platforms.

In this article, we’ll explore how this integration works, why it matters, and how you can design a modern AI-powered customer support system.

What is Amazon Connect?

Amazon Connect is a cloud-based contact center service that allows businesses to set up customer support systems without managing infrastructure.

It provides:

Voice and chat support
Contact flows (IVR systems)
Call routing and queue management
Real-time analytics and reporting

Unlike traditional call centers, Amazon Connect is fully managed and scalable, making it ideal for modern applications.

Why Combine Generative AI with Amazon Connect?

Traditional contact centers rely on:

Static IVR menus
Predefined responses
Manual agent intervention

These approaches often result in:

Poor user experience
High operational costs
Slow response times

Generative AI solves these challenges by enabling:

Natural language conversations
Intelligent query understanding
Dynamic response generation
Context-aware interactions

This creates a more human-like and efficient support experience.

High-Level Architecture

Below is a simplified architecture of integrating Generative AI with Amazon Connect.

Figure 1: AI-Powered Contact Center Architecture

User Call / Chat  
↓  
Amazon Connect (Contact Flow)  
↓  
AWS Lambda  
↓  
Generative AI Model  
↓  
Response Generation  
↓  
Return Response to User

How the System Works

Let’s break down the flow step by step.

1. User Interaction

A user initiates interaction through:

Voice call
Chat interface

Amazon Connect captures the request using a contact flow.

2. Contact Flow Processing

Amazon Connect routes the request based on:

User input
Intent detection
Business logic

Instead of using static IVR, it can forward the request to a backend service.

3. AWS Lambda Integration

AWS Lambda acts as the backend logic layer.

It:

Receives user input
Processes the request
Calls the Generative AI model
Handles responses

4. Generative AI Processing

The AI model:

Understands user intent
Uses context (if available)
Generates a natural language response

This enables:

Dynamic conversations
Personalized answers
Reduced dependency on predefined scripts

5. Response Delivery

The generated response is sent back to Amazon Connect and delivered to the user through:

Voice (Text-to-Speech)
Chat message

Example Use Cases

1. Intelligent Customer Support

Users can ask questions like:

“Why was I charged twice?”

Instead of navigating menus, the AI:

Understands the query
Fetches relevant data
Generates a contextual response

2. Automated Ticket Handling

AI can:

Collect user information
Create support tickets
Provide status updates

3. FAQ Automation

Replace static FAQs with:

Dynamic AI responses
Context-aware answers

4. Call Summarization

After a call:

AI generates summaries
Helps agents review conversations
Improves productivity

Example Lambda Flow (Simplified)

exports.handler = async (event) => {

  const userInput = event.input;

  // Call Generative AI model
  const aiResponse = await generateResponse(userInput);

  return {
    message: aiResponse
  };
};

This simple flow shows how Lambda connects user input with AI output.

Benefits of This Architecture

Improved Customer Experience

Natural conversations
Faster responses
Personalized interactions

Reduced Operational Costs

Fewer human agents required
Automated workflows
Efficient handling of repetitive queries

Scalability

Handles thousands of requests
No infrastructure management

Flexibility

Easy to integrate with backend systems
Supports multiple communication channels

Best Practices

To build an effective AI-powered contact center:

Use clear and structured prompts
Add fallback mechanisms for failed responses
Maintain conversation context
Monitor AI outputs regularly
Ensure data privacy and security

Challenges to Consider

While powerful, this approach has challenges:

Handling complex edge cases
Avoiding incorrect AI responses
Managing latency
Ensuring compliance for sensitive data

Careful system design is required to address these issues.

The Future of AI in Contact Centers

The combination of cloud contact centers and Generative AI is shaping the future of customer support.

We are moving toward systems that can:

Understand user intent deeply
Automate multi-step workflows
Act as intelligent agents

In the future, these systems will evolve into fully autonomous AI-powered customer service platforms.

Final Thoughts

Integrating Generative AI with Amazon Connect enables organizations to build smarter, faster, and more efficient customer support systems.

Instead of relying on rigid workflows, businesses can create dynamic, intelligent, and scalable experiences that adapt to user needs in real time.

For developers and architects, this represents a powerful opportunity to build next-generation customer engagement platforms using AI and cloud technologies.

How Retrieval-Augmented Generation (RAG) Works on AWS

saif ur rahman — Fri, 06 Mar 2026 16:04:16 +0000

How Retrieval-Augmented Generation (RAG) Works on AWS

Generative AI models are powerful, but they have an important limitation: they only know what they were trained on. When you want an AI system to answer questions about your own documents, company knowledge bases, or internal data, relying solely on the model’s training data is not enough.

This is where Retrieval-Augmented Generation (RAG) becomes one of the most important architectural patterns in modern AI systems.

RAG allows generative AI models to access external knowledge sources in real time. Instead of guessing or relying only on training data, the model retrieves relevant information and then generates an answer based on that data.

In this article, we will explore what RAG is, why it matters, and how it can be implemented using AWS services to build scalable and production-ready AI systems.

What is Retrieval-Augmented Generation (RAG)?

Retrieval-Augmented Generation is an AI architecture that combines information retrieval with generative language models.

Instead of asking a language model to answer a question based only on its training data, a RAG system retrieves relevant documents from a knowledge source and provides them to the model as context. The model then generates a response based on those documents.

In simple terms:

RAG = Retrieve relevant information + Generate an intelligent answer

This approach enables AI systems to work with up-to-date, domain-specific, and private data.

Why RAG is Important for Real-World AI Applications

Without RAG, generative AI models often struggle with several challenges:

Outdated knowledge
Lack of domain-specific expertise
Hallucinations (incorrect answers)
Inability to access private or enterprise data

RAG addresses these issues by connecting the language model to external knowledge sources.

Some common real-world applications include:

Customer support assistants
Enterprise knowledge search systems
Legal and compliance assistants
Financial document analysis tools
Healthcare knowledge systems
Internal company knowledge bots

By retrieving relevant documents before generating a response, the AI system becomes more accurate, trustworthy, and explainable.

How RAG Works (Conceptual Flow)

A typical RAG system operates in two main phases.

1. Data Preparation Phase

In this stage, documents are processed and converted into a searchable format.

The typical steps include:

Collecting documents such as PDFs, HTML pages, text files, or databases
Splitting documents into smaller sections called chunks
Converting each chunk into vector embeddings
Storing embeddings in a vector database

These embeddings allow the system to perform semantic searches based on meaning rather than exact keyword matches.

2. Query and Generation Phase

When a user asks a question, the system performs the following steps:

The user query is converted into an embedding.
The system searches the vector database for similar embeddings.
The most relevant document chunks are retrieved.
The retrieved context is sent to a language model.
The model generates a response using the retrieved information.

This approach ensures the model answers questions using real documents instead of guesswork.

Core Components of a RAG System on AWS

When building RAG systems on AWS, several components work together to create a scalable pipeline.

Data Storage

Documents are typically stored in Amazon S3, which serves as the central repository for knowledge sources.

Embedding Generation

Embeddings are numerical representations of text used for semantic similarity search.

These embeddings can be generated using foundation models available through Amazon Bedrock.

Vector Storage

Vector databases store embeddings and allow similarity search operations.

Common options include:

Amazon OpenSearch Serverless (vector search capability)
Other vector databases integrated with AWS services

Retrieval Engine

The retrieval layer searches the vector database to find the most relevant document chunks for a given query.

Generative Model

Finally, a foundation model from Amazon Bedrock generates the response using the retrieved context.

RAG Architecture on AWS

A simplified serverless architecture for RAG might look like this:

User Query
   ↓
API Gateway
   ↓
AWS Lambda
   ↓
Embedding Generation
   ↓
Vector Search (OpenSearch)
   ↓
Retrieve Relevant Documents
   ↓
Foundation Model (Amazon Bedrock)
   ↓
Generated Answer

This architecture is scalable, serverless, and cost-efficient, making it suitable for production AI workloads.

Building RAG with Amazon Bedrock Knowledge Bases

AWS also provides Knowledge Bases for Amazon Bedrock, which simplifies the implementation of RAG.

Instead of building the entire pipeline manually, Knowledge Bases handle several tasks automatically:

Document ingestion
Chunking and embeddings
Vector indexing
Retrieval pipelines

Developers simply provide the documents, and the service manages the underlying infrastructure.

This significantly reduces operational complexity and allows developers to focus on building AI applications.

Techniques That Improve RAG Performance

The effectiveness of a RAG system depends heavily on how the retrieval pipeline is designed.

Several techniques can significantly improve performance.

Smart Document Chunking

Documents should be divided into meaningful sections rather than random segments.

Proper chunking improves:

Retrieval accuracy
Context understanding
Response relevance

For structured documents such as reports or articles, hierarchical chunking can preserve relationships between sections.

Hybrid Search

Hybrid search combines:

Semantic search (vector similarity)
Keyword search

This approach improves retrieval performance, especially for technical or domain-specific documents.

Reranking

Sometimes the initial retrieval step returns several loosely relevant results.

A reranker model evaluates those results and prioritizes the most relevant documents.

This allows the system to send fewer but higher-quality documents to the language model.

Context Window Optimization

Sending too many documents to the language model increases both cost and latency.

A well-designed RAG system retrieves only the most relevant chunks, ensuring efficient responses.

Benefits of Using RAG on AWS

Implementing RAG provides several benefits for enterprise AI systems.

Improved Accuracy

Responses are generated using real documents rather than relying solely on training data.

Reduced Hallucinations

The model is grounded in verified information.

Access to Private Data

Organizations can safely use internal knowledge bases.

Scalability

AWS services allow the system to scale automatically based on demand.

Cost Efficiency

Serverless architectures reduce infrastructure management overhead.

Common Use Cases of RAG

RAG is widely used across many industries.

Some examples include:

Customer Support Assistants

AI systems retrieve answers from support documentation.

Enterprise Knowledge Systems

Employees can search internal knowledge bases using natural language.

Legal Document Analysis

AI retrieves relevant clauses from contracts and policies.

Financial Research Tools

Analysts can query financial reports and market documents.

Healthcare Knowledge Systems

Medical professionals can access clinical documentation efficiently.

Challenges When Implementing RAG

Although RAG is powerful, designing an effective system requires careful planning.

Some common challenges include:

Data Quality

Poorly structured documents lead to poor retrieval results.

Chunking Strategy

Improper chunk sizes reduce the quality of context provided to the model.

Latency

Multiple retrieval steps can increase response time.

Security

Sensitive documents require proper access control.

AWS security features such as IAM and encryption help address these concerns.

Best Practices for Production RAG Systems

When building a production-ready RAG system, consider the following best practices:

Store documents in structured formats
Use semantic chunking strategies
Implement reranking for better retrieval accuracy
Monitor model outputs to detect hallucinations
Optimize the number of retrieved documents to reduce token costs
Apply strict access control for sensitive data

Following these practices ensures your RAG system remains reliable and efficient.

The Future of RAG and AI Applications

RAG is rapidly becoming the standard architecture for enterprise generative AI systems.

As foundation models continue to improve, the real competitive advantage will come from how effectively these models connect to real-world knowledge sources.

Combining RAG with technologies such as AI agents, automation workflows, and serverless cloud architectures will enable even more powerful and intelligent applications.

Final Thoughts

Retrieval-Augmented Generation bridges the gap between large language models and real-world knowledge.

By combining document retrieval with generative models, developers can build AI systems that are accurate, context-aware, and capable of answering complex questions based on real data.

AWS provides a powerful ecosystem of services that make building RAG systems scalable and production-ready. Whether you are developing an enterprise knowledge assistant, a customer support chatbot, or a document analysis platform, RAG is one of the most effective architectures available today.

Why Your LLM Pipeline Needs Circuit Breakers

saif ur rahman — Wed, 25 Feb 2026 09:00:51 +0000

Most LLM demos work perfectly.

Until they don’t.

You test your prompt in the playground. It responds beautifully. You wire it into production. A few users try it. Everything seems fine.

Then traffic increases.

Then Bedrock throttles.

Then retries start firing.

Then your queue depth spikes.

Then you accidentally DDoS your own model endpoint.

This is the moment most AI systems fail — not because of intelligence, but because of infrastructure.

If you're building a real production AI backend, you don’t just need prompts.

You need circuit breakers.

The Illusion of Reliability in LLM Systems

When we integrate an LLM into a system, it feels like calling any other API:

await callLLM(prompt)

But LLMs are not ordinary APIs.

They are:

Capacity-constrained
Rate-limited
Token-limited
Region-dependent
Occasionally throttled
Sometimes unavailable

And when they fail, they fail in bursts.

The Real Failure Modes

Let’s look at what actually happens in production.

1 Bedrock Throttling

You’ll see errors like:

ThrottlingException: Too many tokens per day

Or:

Rate exceeded

This is not a bug in your code.

This is capacity control.

But here’s where it becomes dangerous:

If your system retries immediately, you amplify the problem.

2 Retry Storms

Imagine 500 concurrent requests.

Each one gets throttled.

Each one retries instantly.

Now you have 1,000 requests.

They retry again.

Now you have 2,000.

You’ve created a retry storm.

Your queue explodes.
Your workers saturate.
Your AI endpoint collapses.

This is how fragile AI backends implode.

3 Naive Exponential Backoff Isn’t Enough

Most developers think this solves it:

retryWithExponentialBackoff()

That’s necessary.

But it’s not sufficient.

Because if the upstream dependency (Bedrock) is hard-throttled for minutes or hours, exponential backoff just spreads out the pain.

You still keep hitting a failing system.

What you actually need is a circuit breaker.

What Is a Circuit Breaker (In AI Context)?

A circuit breaker is a control mechanism that:

Detects repeated failures
Stops sending traffic to a failing dependency
Waits for recovery
Gradually restores traffic

It prevents cascading failures.

It protects your infrastructure from external instability.

In LLM systems, it’s mandatory.

Designing Circuit Breakers for LLM Pipelines

1. Failure Threshold Detection

Track consecutive failures:

ThrottlingException
Timeout
5xx responses
Token quota exceeded

If failure rate exceeds a threshold (e.g., 30% in 1 minute):

Trip the breaker.

Store this state in:

Memory (single worker)
Redis (multi-instance)
DynamoDB (serverless safe)

2. Open the Circuit

When open:

Do NOT call Bedrock.

Instead:

Return a graceful error
Queue for later processing
Route to fallback model

This prevents retry storms.

3. Half-Open State

After a cooldown (e.g., 60 seconds):

Allow limited traffic:

1 request
Then 5
Then 10

If successful → close the breaker.
If failed → reopen immediately.

Controlled recovery is critical.

Fallback Models: Your Safety Net

Circuit breakers should not just stop traffic.

They should degrade gracefully.

Primary model:

Claude Sonnet 4.5

Fallback model:

Claude 3 Sonnet

Emergency fallback:

Claude Haiku

If high-tier model fails:

Automatically switch to smaller model
Reduce max_tokens
Return simplified output

Users prefer partial functionality over total outage.

Auto-Disabling Failing Endpoints

In distributed AI systems, you might have:

Multiple regions
Multiple models
Multiple inference profiles

If one endpoint begins failing:

Disable it automatically.

Maintain a health registry:

us-east-1: unhealthy
eu-west-1: healthy

Route traffic only to healthy regions.

This is how resilient systems behave.

A Safe Architecture Pattern

API Gateway
    ↓
Request Lambda
    ↓
SQS
    ↓
Worker Lambda
    ↓
Circuit Breaker Layer
    ↓
LLM Call
    ↓
Fallback Router
    ↓
S3 + DynamoDB

Never let your worker blindly call the model.

Always call through a protective layer.

AI Systems Are Distributed Systems

LLM integration is not prompt engineering.

It’s distributed systems engineering.

If you wouldn’t connect your production system to a database without:

Connection pooling
Retry logic
Circuit breakers
Health checks

Then you shouldn’t connect it directly to an LLM either.

Final Thought

LLMs are probabilistic.

Infrastructure must be deterministic.

If you don’t design protective layers around your AI dependencies, your system will eventually fail under load.

Not because your model is bad.

But because your architecture is fragile.

And fragile systems don’t scale.

6 Mistakes Developers Make When Deploying Generative AI on AWS (And How to Fix Them)

saif ur rahman — Tue, 24 Feb 2026 10:27:27 +0000

Generative AI is everywhere right now.

We’re building AI report generators, document summarizers, compliance checkers, risk engines, chatbots — and most of them work perfectly in local development.

Until they hit production.

Then things start breaking.

Timeouts.

Retries gone wrong.

Users refreshing the page 10 times.

S3 buckets accidentally public.

No clear job status.

Lambda costs increasing silently.

I recently built a production-ready serverless Generative AI backend on AWS, and along the way I made (and fixed) almost every mistake in this list.

If you’re deploying GenAI workloads on AWS, especially with Lambda, this article will save you time, money, and headaches.

Let’s break it down.

Mistake #1: Blocking API Calls with LLM Requests

The Problem

The most common mistake I see:

// Inside API handler
const result = await callLLM();
return result;

Looks simple.

But here’s what happens in production:

API Gateway has a 29-second timeout
LLM calls can take 10–60 seconds
External APIs (news, sanctions, risk feeds) add latency
Users sit there waiting

Eventually:

Timeout.

And your user thinks your AI “doesn’t work”.

The Fix: Asynchronous Architecture with SQS

Instead of blocking the API, decouple it.

Better flow:

Client ↓ API Gateway ↓ Lambda (Request Handler) ↓ SQS ↓ Worker Lambda (long timeout) ↓ Bedrock / External APIs ↓ S3 + DynamoDB

The API only:

Validates input
Creates a report record
Sends message to SQS
Returns immediately

The worker handles heavy AI processing.

This removes timeouts completely and makes your system scalable.

Mistake #2: No Retry Logic for AI Failures

The Problem

LLMs fail.
External APIs fail.
Network calls fail.

If you call AI directly inside a request and it fails:

The user request fails
No retry
No recovery
No record of what happened

This is dangerous in compliance or risk systems.

The Fix: Let SQS Handle Retries

SQS + Lambda event source mapping automatically:

Retries failed messages
Respects visibility timeout
Supports Dead Letter Queues

Now if your worker fails:

The message returns to queue
Lambda retries
You can configure maxReceiveCount
You can attach a DLQ for failed jobs

You get retry logic without writing retry code.

That’s production engineering.

Mistake #3: No Status Tracking for AI Jobs

The Problem

User submits request.

Now what?

You have no idea if the job is:

Pending
Processing
Completed
Failed

Users refresh blindly.
You cannot build dashboards.
You cannot monitor performance.

The Fix: DynamoDB Lifecycle Tracking

Use DynamoDB as a job state tracker.

When request is created:

json { "status": "PENDING", "risk_level": null, "s3_url": null }

When worker starts:

status → PROCESSING

When completed:

status → COMPLETED risk_level → High s3_url → https://...

Now your frontend can:

Poll job status
Show progress
Display result when ready

This is how long-running AI jobs should be handled.

Mistake #4: Making S3 Buckets Public

The Problem

You generate AI reports and store them in S3.

Quick solution?

Make bucket public.

json { "Principal": "*", "Action": "s3:GetObject" }

Done.

Except now:

Anyone can download reports
Sensitive data is exposed
Compliance risk increases

And yes, I’ve seen this happen.

The Fix: Use Pre-Signed URLs

Keep your bucket private.

When job completes:

js const url = await getSignedUrl(s3Client, command, { expiresIn: 600 // 10 minutes });

Now:

URL works temporarily
Only authorized user gets access
Bucket remains private
You avoid major security risks

Public buckets and AI-generated reports should never mix.

Mistake #5: Weak Input Validation

The Problem

Most GenAI systems accept user input like:

json { "companyName": "...", "corporateNumber": "..." }

Without proper validation:

Invalid corporate numbers
Injection attempts
Broken workflows
Garbage-in → garbage-out AI responses

LLMs amplify bad input.

The Fix: Strong Validation Schema

Use a validation layer (e.g., Joi):

js corporateNumber: Joi.string() .pattern(/^[a-zA-Z0-9-]+$/) .required()

Validate:

Length
Format
Required fields
Country constraints

Never trust AI to fix bad input.

AI is powerful — not magical.

Mistake #6: Over-Permissive IAM Roles

The Problem

Many developers attach:

AdministratorAccess

To Lambda for convenience.

This is dangerous:

S3 access everywhere
DynamoDB access everywhere
Bedrock access unrestricted
Harder to audit

The Fix: Least Privilege IAM

Grant only what you need:

sqs:SendMessage
sqs:ReceiveMessage
dynamodb:UpdateItem
s3:PutObject
s3:GetObject
Specific resource ARNs

Your GenAI backend becomes:

More secure
Easier to audit
Production compliant

Security is part of AI engineering.

The Real Lesson

Generative AI is not just about prompting.

It’s about:

Architecture
Reliability
Security
Observability
Lifecycle management

If you treat LLMs like simple API calls, your system will fail at scale.

If you treat them like long-running distributed workloads, you’ll build something production-ready.

Final Thoughts

AI is the exciting part.

But infrastructure is what makes it usable.

The difference between a demo and a real product is not the model — it’s the backend design.

If you’re deploying Generative AI on AWS:

Use async patterns
Track state
Avoid blocking APIs
Secure your storage
Validate aggressively
Follow least privilege

That’s how you build AI systems that survive production.