DEV Community

Tijani Abagaro
Tijani Abagaro

Posted on

Architecting an Enterprise RAG Platform: Shifting from AI Hype to Production Trust on AWS

#aws #ai #architecture #devops

Moving Generative AI from a proof-of-concept sandbox into an enterprise-grade solution requires shifting our engineering focus from what AI can do to what enterprises can trust.

Modern organizations generate immense repositories of institutional knowledge—ranging from complex corporate refund policies to intricate regulatory guidelines. While data availability is rarely the issue, efficiently retrieving it remains a massive operational bottleneck. Workforce productivity drops significantly when employees are forced to manually navigate fragmented, disconnected data silos.

As companies rush to adopt Generative AI to bridge this gap, they encounter a critical barrier: trust and governance. Public, out-of-the-box LLMs operate without internal corporate context and are highly prone to hallucination. For an enterprise, an incorrect or completely fabricated answer introduces unacceptable operational, brand, and regulatory risks.

To solve this, I designed and open-sourced an end-to-end Enterprise Refund AI Assistant. Instead of relying blindly on a foundation model’s pre-trained data, this platform utilizes a robust, decoupled Retrieval-Augmented Generation (RAG) architecture. By separating data ingestion from live inference, the system ensures that every conversational output is strictly anchored, grounded, and fully traceable back to verified, authoritative enterprise documentation.

Architectural Topology: The Power of a 100% Serverless Footprint

Rather than provisioning monolithic servers or maintaining idle container clusters, the platform relies on a 100% serverless topology. This structural decision guarantees total operational elasticity: the entire environment automatically scales down to zero when idle, eliminating baseline infrastructure costs.

Instead of a standard service catalog, the system is segmented into functional layers chosen specifically for isolation, security, and low operational overhead.

  • Global Distribution Edge: Amazon CloudFront securely distributes a web interface hosted on Amazon S3, reducing latency while protecting backend resources.
  • Decoupled Compute Layer: AWS Lambda orchestrates backend processing without requiring server management.
  • Vector Search & Knowledge Core: Amazon OpenSearch Serverless provides scalable semantic indexing and vector search.
  • Managed Foundation Models: Amazon Bedrock delivers Titan Text Embeddings V2 for embeddings and Amazon Nova Lite for grounded response generation.

Decoupling the Core Pipelines

To maximize throughput while protecting inference latency, the platform completely separates asynchronous document ingestion from synchronous user inference.

Pipeline 1 — Event-Driven Document Ingestion 

S3 Document Upload
        │
        ▼
S3 Event Notification
        │
        ▼
Lambda Processor
        │
        ▼
Titan Text Embeddings V2
        │
        ▼
OpenSearch Serverless
Enter fullscreen mode Exit fullscreen mode
  1. Uploading a corporate policy PDF into Amazon S3 automatically triggers an ObjectCreated event.
  2. Lambda extracts the document and chunks it using a 1,000-character window with a 200-character overlap.
  3. Each chunk is converted into a vector embedding using Amazon Bedrock.
  4. Embeddings, metadata, and source text are indexed into OpenSearch Serverless.

Pipeline 2 — Live RAG Inference 

User Question
      │
      ▼
API Gateway
      │
      ▼
Lambda
      │
      ▼
OpenSearch Vector Search
      │
      ▼
Context-Augmented Prompt
      │
      ▼
Amazon Nova Lite
      │
      ▼
Grounded Response
Enter fullscreen mode Exit fullscreen mode
  1. API Gateway routes the request to the inference Lambda.
  2. The user query is embedded and matched against OpenSearch using semantic similarity.
  3. Relevant documentation is injected into a constrained prompt.
  4. Amazon Nova Lite generates a response using only the supplied context.
  5. Conversation history is stored in DynamoDB for multi-turn interactions.

Infrastructure as Code & GitOps

Every AWS resource is provisioned declaratively using Terraform.

Remote state management is implemented using Amazon S3 together with DynamoDB state locking.

terraform {
  backend "s3" {
    bucket         = "enterprise-refund-ai-tfstate"
    key            = "prod/platform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "enterprise-refund-ai-tflocks"
    encrypt        = true
  }
}
Enter fullscreen mode Exit fullscreen mode

GitHub Actions automates deployment using GitHub OpenID Connect (OIDC), eliminating long-lived AWS credentials and enabling secure, short-lived role assumption.

Benchmark Performance & Production Results

Note: These benchmark results were obtained in a controlled demonstration environment and are intended to illustrate the platform's performance characteristics.

Performance

  • Average end-to-end latency: 1.15 seconds
  • Vector search latency: ~120 ms
  • Nova Lite response generation: ~850 ms
  • Document ingestion: Under 4.2 seconds for a standard 20-page enterprise policy

Cost Efficiency

  • Idle compute cost: $0.00/month (serverless compute)
  • Estimated monthly orchestration cost: Less than $45 at approximately 10,000 queries/day
  • Estimated savings: ~75% compared to always-on infrastructure

Operational Governance

  • Static AWS credentials: None (OIDC only)
  • Hallucination behavior: During benchmark testing, out-of-scope questions returned "I cannot find that information in the approved documentation" rather than generating unsupported answers.

Senior Engineering Challenges & Resolutions

Challenge 1 — Chained Latency

Problem

API Gateway → Lambda → Bedrock → OpenSearch → Bedrock creates multiple network hops that can increase latency.

Solution

  • Modular Lambda functions
  • Python 3.12 runtime
  • Optimized memory allocation
  • boto3 connection reuse

Challenge 2 — OpenSearch Serverless Security

Problem

OpenSearch Serverless separates security, network, and data access policies, making Infrastructure as Code more complex.

Solution

Terraform dynamically provisions IAM execution roles and injects them directly into OpenSearch data access policies during deployment, eliminating manual configuration.

Explore the Blueprint

The complete production-ready implementation is available as open source.

📦 GitHub Repository

https://github.com/Tijani-Abagaro-GenAI-Cloud/enterprise-refund-ai-showcase

🎥 YouTube Walkthrough

https://youtu.be/OCUfAaRI8Ng

I welcome feedback from the AWS and AI community. If you've built enterprise Generative AI systems, I'd love to hear how you're approaching retrieval, grounding, security, and operational scalability.

If you find this project useful, I'd appreciate your feedback, suggestions, or a ⭐ on GitHub.

Top comments (0)