DEV Community

Nagachaitanya Inamdar
Nagachaitanya Inamdar

Posted on Edited on

The Ingestion Factory: Decoupling Document Uploads with SQS Claim-Check & 100% In-Memory Parsers (Part 2)

#architecture #aws #java #rag

Series: Designing an Enterprise RAG Platform from Scratch (Part 2 of 5)

1. The Trap: Synchronous Document Parsing

In Part 1, I shared why I set out to build DocLab—an enterprise-grade RAG platform on AWS using Java 21, Spring Boot 3, and pgvector.

Today, let’s talk about the first big hurdle every RAG engineer faces: Document Ingestion.

In most hobby tutorials, document ingestion looks like this:

User uploads file ──> Web Server parses file in-process ──> Generates vectors ──> Saves to DB ──> Returns 200 OK
Enter fullscreen mode Exit fullscreen mode

When you try this in production, you encounter two immediate disasters:

  1. The Web Thread Bottleneck: If an engineer uploads a 40MB PDF or a 100-page architectural specification, parsing that file can take 15 to 45 seconds. Synchronous worker threads in Tomcat or Netty get blocked waiting on CPU-intensive parsing. Under moderate load, your API gateway runs out of threads and returns 504 Gateway Timeout.
  2. The Cloud OCR Cost Explosion: Many teams blindly wire up cloud OCR services like AWS Textract or vision models for every incoming file. Textract charges \$1.50 per 1,000 pages for standard tables, and up to \$15.00 per 1,000 pages for complex queries. If your company processes 50,000 pages of internal documentation, you’ve blown your entire monthly cloud budget before a single user has asked a question.

I needed an ingestion architecture that was completely asynchronous, scale-resilient, and 100% cost-free on document parsing.

2. Decoupling Intake via the SQS Claim-Check Pattern

To prevent web thread starvation, I implemented the enterprise Claim-Check Pattern using Amazon S3 and Amazon SQS.

Ingestion Factory

[Client POST /upload] ──> [Spring Boot Gateway]
                                │
        ┌───────────────────────┴───────────────────────┐
        ▼                                               ▼
 [Amazon S3: Raw Bucket]                     [SQS: doclab-parsing-queue]
 (Stores raw unparsed bytes)                 (Lightweight metadata envelope)
        │                                               │
        └───────────────────────┬───────────────────────┘
                                ▼
                   [UniversalParserWorker]
Enter fullscreen mode Exit fullscreen mode

How the Ingestion Flow Works:

  1. Instant Acceptance: When a client sends a multipart/form-data request with a document (PDF, Word, Excel, JSON) along with tenant and role metadata, the Spring Boot controller writes the raw binary stream directly to an Amazon S3 raw bucket (raw/doc-{uuid}-v1.ext).
  2. The Claim-Check Envelope: The controller never touches or parses the file content. Instead, it emits a lightweight JSON message (a claim-check) to an Amazon SQS queue (doclab-parsing-queue): 
   {
     "document_id": "doc-eb677320",
     "version": 1,
     "document_title": "Production Service Catalog",
     "source_format": "JSON",
     "s3_bucket": "doclab-docs-19530061",
     "s3_raw_key": "raw/doc-eb677320-v1-service-catalog.json",
     "tenant_id": "engineering-dept",
     "allowed_roles": ["ENGINEERING"]
   }
Enter fullscreen mode Exit fullscreen mode
  1. Sub-50ms Response: The API returns HTTP 202 ACCEPTED with the documentId and status QUEUED in under 50 milliseconds. The web server thread is immediately released back to the connection pool.

3. Zero-Cost Universal Parsing (100% In-Memory)

Now that the file is safely in S3 and queued in SQS, background workers handle the heavy lifting. But how do we parse multiple enterprise formats without paying for cloud OCR APIs?

By building purely in-memory format-specific extractors using proven, industrial-strength open-source Java libraries:

  • PDF Documents: Parsed via Apache PDFBox 3.0.2. Extracts raw text streams, structural paragraphs, and page boundaries without spinning up heavy headless browsers.
  • Word Documents (.docx): Parsed via Apache POI 5.2.5 (XWPF). Traverses OpenXML document structures, preserving heading hierarchies, bolded section titles, and nested paragraph blocks.
  • Excel Spreadsheets (.xlsx): Parsed via Apache POI (XSSF). Converts multi-column tabular data into readable key-value contexts (e.g., Cluster Node: worker-node-01 | Instance Type: c5.xlarge | vCPU: 4 | Memory: 16GB).
  • Raw JSON Technical Schemas: Parsed via Jackson. Flattens complex configuration catalogs, API contracts, and service schemas into discrete property chunks.
  • Markdown & Architecture Decision Records (ADRs): Clean token-aware sliding window chunking with semantic heading preservation (# Title, ## Decision, ## Consequences).

Why 100% In-Memory?

Each extractor receives the raw byte[] from S3 directly into a memory stream, extracts text chunks with token metadata, and produces clean text blocks without writing a single temporary file to disk.

The Economic Impact:

  • Cost of processing 10,000 internal documents via cloud OCR: ~\$150.00
  • Cost of processing 10,000 internal documents via our in-memory engine: \$0.00

4. The Canonical Intermediate Representation (IR)

One of the biggest mistakes in RAG architectures is coupling file parsing directly to embedding generation. If you change your embedding model (e.g., moving from OpenAI text-embedding-3 to Amazon Titan Text v2), you shouldn't have to re-extract raw PDF bytes from scratch.

To solve this, our parser outputs a format-agnostic Canonical Intermediate Representation (IR) stored on S3 (ir/doc-{uuid}-v1.json):

{
  "document_id": "doc-eb677320",
  "version": 1,
  "title": "Production Service Catalog",
  "source_format": "JSON",
  "content_hash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
  "chunks": [
    {
      "chunk_index": 0,
      "section_title": "Property: checkout-service",
      "content": "{\n  \"tier\" : \"Tier-1\",\n  \"database\" : \"aurora-postgresql\",\n  \"scaling_policy\" : \"Target tracking CPU at 70%\"\n}"
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

Once the Canonical IR is written to S3, the parser emits a secondary claim-check to doclab-indexing-queue. The indexing worker simply reads the clean, normalized JSON IR and generates 1024-dimensional embeddings via Amazon Titan Text v2, completely isolated from raw file parsing.

5. Dual-Mode Execution: Automatic Backgrounding vs. Batch Draining

In real enterprise systems, you need flexibility:

  • When the Spring Boot server runs, documents must be processed automatically within 3 seconds without human intervention.
  • During high-volume batch loads or offline testing, operators need to drain queues manually via CLI without keeping web servers running.

I implemented a Dual-Mode Worker Architecture:

  1. Automated Mode: A Spring @Scheduled worker (BackgroundPipelineWorker) polls SQS every 3 seconds, automatically draining intake and indexing queues with thread-safe AtomicBoolean locks.
  2. Manual / CLI Mode: Standalone workers featuring auto-draining loops with maxNumberOfMessages(10) that drain entire backlogs until the queues are completely empty, then cleanly exit.

What’s Next in Part 3

Now that raw files are asynchronously parsed into a structured, canonical vector index on AWS, how do we retrieve them accurately?

In Part 3, we will dive into the Storage & Hybrid Retrieval Vault:

  • Why pure vector search is dangerous on technical error codes and schema keys.
  • Combining PostgreSQL 16 pgvector (Cosine HNSW) with native tsvector (BM25).
  • The exact mathematics of Reciprocal Rank Fusion (RRF).

Stay tuned for Part 3! If you have questions about handling multi-format parsing or SQS claim-check architectures, drop them in the comments below!

Top comments (0)