DEV Community: Ankur Jindal

AWS Lambda Managed Instances: What Min=0/Max=0 Actually Does in Production

Ankur Jindal — Thu, 17 Sep 2026 02:38:34 +0000

A hands-on walkthrough, including the exact failures and dead ends you'll hit setting this up.

Introduction: The Production Bottleneck

AWS Lambda Managed Instances (LMI) puts Lambda functions on EC2 capacity you control, while keeping Lambda's programming model. Most of the coverage since re:Invent 2025 has focused on two things: the removal of the memory ceiling, and the EC2-backed pricing model that lets you apply Savings Plans and Reserved Instances.

Almost none of it covers the scaling configuration — specifically, what happens when MinExecutionEnvironments and MaxExecutionEnvironments are both set to 0.

The assumption most engineers carry over from standard Lambda is that a minimum of zero means "no standing capacity, scale up from zero on the next request." That assumption is wrong for LMI, and it's wrong in a way that produces a hard invoke failure instead of a slow cold start. This post documents the actual mechanics, verified against a live AWS account, along with the setup friction that isn't in the AWS documentation.

The Mechanics of the Failure Chain

On Lambda Managed Instances, MinExecutionEnvironments and MaxExecutionEnvironments are not independent settings. AWS enforces a pairing rule: a minimum of 0 is only accepted when the maximum is also 0.

When both are set to 0, Lambda does not idle the function and wait for the next invocation. It deactivates the function version:

Every EC2 Managed Instance backing that version is terminated.
Instance charges continue until termination actually completes — the meter doesn't stop the moment you save the config.
The version's State flips to Deactivated.
Any invocation against a deactivated version returns an explicit error. There's no cold start, no queueing, no retry — it fails outright.
Reactivation is never automatic. You have to push a new scaling configuration with non-zero values, through the console, the PutFunctionScalingConfig API, or a scheduled action.

This is the opposite of how ReservedConcurrentExecutions=0 or standard Lambda concurrency throttling behaves, and it's the single most important operational difference between LMI and normal Lambda for anyone running non-continuous workloads.

Production Architecture Pattern & Walkthrough

System Architecture Overview

LMI has one prerequisite that standard Lambda doesn't: a Capacity Provider. This is a separate resource that defines:

The VPC, subnets, and security groups the EC2 instances launch into
Which instance architecture (x86_64 or arm64) and types are eligible
The scaling mode (Auto or Manual)
The IAM role Lambda uses to manage EC2 on your behalf (the "Operator Role")

A function is then attached to a Capacity Provider at creation time by setting its compute type. Only after that attachment exists — and only on a published version, not $LATEST — do MinExecutionEnvironments and MaxExecutionEnvironments become meaningful.

Setup Walkthrough: Step by Step

1. Create the Capacity Provider and attach it to the function.

In the Lambda console, this doesn't appear as "Compute type" the way older documentation describes it — in the current console it's the EC2 capacity provider toggle under Custom settings. Toggling it on opens a side panel to select the capacity provider ARN, memory size, and the execution-environment-memory-per-vCPU ratio:

The panel already resolves the capacity provider by ARN once created. Memory and the vCPU ratio (2:1, 4:1, or 8:1) are set per-function here, not per-capacity-provider.

2. Architecture must match between the function and the capacity provider.

The first real failure: creating the function returned this error on save —

"You cannot use a Lambda Managed Instances function with a capacity provider that does not support the architecture of the function."

The capacity provider's InstanceRequirements.Architectures is fixed at creation time (in this case, arm64). If the function's own architecture toggle doesn't match, function creation is rejected outright — there's no coercion or fallback. Confirmed via CLI:

{
    "CapacityProvider": {
        "CapacityProviderArn": "arn:aws:lambda:us-east-1:<ACCOUNT_ID>:capacity-provider:lambaprovisionedinstance",
        "State": "Active",
        "VpcConfig": {
            "SubnetIds": [
                "subnet-<redacted>",
                "subnet-<redacted>"
            ],
            "SecurityGroupIds": [
                "sg-<redacted>"
            ]
        },
        "PermissionsConfig": {
            "CapacityProviderOperatorRoleArn": "arn:aws:iam::<ACCOUNT_ID>:role/service-role/lambaprovisionedinstance-role-<redacted>"
        },
        "InstanceRequirements": {
            "Architectures": ["arm64"]
        },
        "CapacityProviderScalingConfig": {
            "ScalingMode": "Auto"
        }
    }
}

Fix: match the function's architecture toggle to whatever the capacity provider was created with. There's no cross-architecture capacity provider.

3. Publishing does not prompt for scaling configuration.

Creating and publishing the function does not ask for MinExecutionEnvironments or MaxExecutionEnvironments at any point. That configuration lives separately, under Configuration → Function scaling configuration, edited independently after the function exists:

With nothing configured, AWS's default behavior applies: 3 Managed Instances provisioned across availability zones before the version is marked Active, invisible in this panel until you explicitly edit it.

4. $LATEST is not invocable on LMI — only a published version is.

This is a second point of confusion that isn't obvious from the console. Querying $LATEST directly shows a state most engineers won't recognize:

{
    "Configuration": {
        "FunctionName": "lmi-function",
        "FunctionArn": "arn:aws:lambda:us-east-1:<ACCOUNT_ID>:function:lmi-function:$LATEST",
        "State": "ActiveNonInvocable",
        "Architectures": ["arm64"]
    }
}

LMI introduces $LATEST.PUBLISHED specifically for this: a version you can republish repeatedly without managing numbered versions. When created through the console, this is generated automatically. Querying it directly confirms the actual invocable state:

{
    "Configuration": {
        "FunctionName": "lmi-function",
        "FunctionArn": "arn:aws:lambda:us-east-1:<ACCOUNT_ID>:function:lmi-function:$LATEST.PUBLISHED",
        "Version": "$LATEST.PUBLISHED",
        "State": "Active",
        "Architectures": ["arm64"]
    }
}

Invoking the function via its unqualified ARN implicitly targets $LATEST.PUBLISHED, not $LATEST — the reverse of how unqualified invokes behave on standard Lambda.

5. Managed Instances are hidden from the EC2 console by default.

Once $LATEST.PUBLISHED showed Active, the expected 3 instances weren't visible anywhere in the EC2 console, and describe-instances with a capacity-provider tag filter returned nothing:

{
    "Reservations": []
}

The cause: Managed resource visibility, an account-wide EC2 setting (introduced April 2026) that hides EC2 resources provisioned by services like LMI, EKS Auto Mode, and ECS Managed Instances from the console and Describe* API calls by default for accounts that didn't have managed resources before the setting existed. The instances are running and billing regardless — visibility is purely a display filter.

Fix, via CLI:

aws ec2 modify-managed-resource-visibility --default-visibility visible

{
    "Visibility": {
        "DefaultVisibility": "visible"
    }
}

This is account-wide and Region-specific, and applies to all managed-resource offerings, not just LMI. After setting it, the instances appeared after a while in the EC2 console:

Two running, one terminated — consistent with AWS's documented default of 3 instances provisioned for availability across AZs, with normal instance replacement happening in the background.

6. Verifying the 0/0 deactivation.

Editing the Function scaling configuration to MinExecutionEnvironments=0 and MaxExecutionEnvironments=0 and saving triggers an immediate, visible state transition in the console:

"Deactivating version $LATEST.PUBLISHED."

Within a few minutes, all three EC2 instances move to Terminated, and invoking the function returns an explicit error referencing the deactivated state — not a timeout, not a cold start, not a retry. This is the core behavior the rest of this post is about: 0/0 is a deactivation switch, not a scale-to-zero setting.

Implementation Strategies & Deep-Dive Considerations

Pricing changes the idle-cost math. LMI drops Lambda's per-invocation duration billing entirely. Instead, pricing has three components: the standard $0.20 per million requests, standard EC2 instance charges (Savings Plans and Reserved Instances apply), and a flat 15% management fee on top of the on-demand EC2 price — applied even when Savings Plans or RIs are discounting the underlying instance cost. That fee doesn't pause during idle traffic. It only stops when instances are actually terminated, which only happens through deactivation or explicit scale-down.

For a workload with, say, 5 hours of daily invocation activity, this means: unless MinExecutionEnvironments is actively brought to 0/0 during the other 19 hours, you are paying full EC2 rate plus the management fee for the entire idle window — not "only for the 5 hours," the way standard Lambda duration billing would imply.

There's no scale-out SLA. AWS does not publish a guaranteed scale-out latency for LMI. The explicit documented warning is that traffic more than doubling within a 5-minute window can produce throttling while capacity catches up. Combined with the 0/0 deactivation behavior, this means bursty or unpredictable workloads should generally avoid 0/0 entirely — the risk isn't a slow response, it's outright invoke failures during both the reactivation window and any sufficiently sharp traffic spike even at steady-state minimums.

EventBridge Scheduler is the correct pattern for predictable idle windows, not manual toggling. A nightly batch job or a fixed 9-to-5 traffic pattern is exactly the case 0/0 is designed for — scheduled deactivation before the idle window, scheduled reactivation before traffic resumes, via PutFunctionScalingConfig calls triggered on a schedule rather than through the console.

Managed resource visibility is a one-time account setting, not a per-function toggle. Worth setting to visible early in any LMI adoption, since the default-hidden behavior for new accounts otherwise makes basic troubleshooting (like confirming instance count) unexpectedly difficult.

Architectural Tradeoffs & Operational Takeaways

Workload shape	Recommended scaling config	Why
Predictable idle windows (nightly batch, fixed business hours)	`0/0` + EventBridge Scheduler	Stops EC2 + management-fee billing during confirmed idle time; reactivation is scheduled, not reactive
Bursty or unpredictable traffic	`MinExecutionEnvironments > 0`	Avoids hard invoke failures during the deactivated window and reduces exposure to the "traffic doubles in 5 minutes" throttling case
Continuous, steady-state traffic	Standing non-zero minimum, sized to baseline	You're paying for compute regardless; this is the workload LMI's EC2 pricing model is built for

The mechanical rule worth internalizing: MinExecutionEnvironments=0 is not a cost-saving default the way it is on standard Lambda concurrency settings. It only saves cost when paired with Max=0, and pairing it with Max=0 converts idle time into a hard failure window unless you're actively scheduling reactivation around it.

Why Your Enterprise RAG Pipeline Is Failing Before the First Query Runs

Ankur Jindal — Fri, 11 Sep 2026 07:24:37 +0000

Introduction: The Ingestion Layer Is the Real Bottleneck

Most teams building Retrieval-Augmented Generation (RAG) systems invest their engineering effort in two places: the LLM and the retrieval layer. They tune prompts, experiment with embedding models, compare vector databases, and benchmark retrieval precision. When the system underperforms in production, they go back to those same two layers and tune again.

The failure, in the majority of enterprise RAG deployments, is not there.

It is at the ingestion layer — specifically at the document parsing stage — and it happens before a single query is ever issued.

This is not a problem that appears during proof-of-concept work. In a POC, teams typically use small, clean documents: a few well-structured PDFs, maybe a Markdown file or two. Basic parsers work fine on those inputs. The results look reasonable. The system gets approved for production.

In production, the document corpus looks different. It contains 200-page regulatory policy documents where tables start on page 22 and continue on page 23. It contains scanned audit reports where the actual content lives in raster images. It contains financial disclosures where running headers and footers break into the body text every page. It contains multi-column layouts where two entirely separate sections sit side by side on the same page.

A basic parser cannot handle any of these cases correctly. It does not fail loudly. It silently discards structural context, flattens layout into a text stream, and passes that degraded output downstream. Every component that follows — the chunker, the embedding model, the vector index — operates on corrupted input. The hallucinations and retrieval failures that appear at query time are traced back to the LLM or the retrieval logic. The actual root cause sits two stages upstream, in the parser, and is never touched.

This article walks through the precise mechanics of why this happens, and documents a production-grade ingestion pipeline built with Docling, LangChain, and a structured metadata extraction layer designed to address these failures at their actual source.

The Mechanics of the Document Parsing Failure Chain

Understanding why standard parsers fail requires understanding what a PDF actually is at the data level.

A PDF file is not a document. It is a page description language. Its internal representation does not encode semantic structure — headings, paragraphs, tables, or sections. It encodes rendering instructions: draw this glyph at this coordinate using this font at this size. There is no native concept of "this text belongs to this table cell" or "this heading is the parent of this section." That structure is a human visual inference from the rendered page layout.

When a library like PyPDF or PDFMiner processes a PDF, it extracts the character sequences and their approximate positions on the page. Without a layout model, it has no mechanism to determine reading order, column boundaries, or the relationship between spatially separated elements. Its default behaviour is to read the page sequentially — typically left to right, top to bottom — treating the full page width as a single text stream.

The failure modes this produces in enterprise documents are specific and compounding:

Multi-Column Layout Corruption

A two-column regulatory document has column A on the left and column B on the right. A sequential reader scans across the page at each y-coordinate, producing output that reads: left-column sentence fragment, right-column sentence fragment, left-column continuation, right-column continuation. The output is a mixed stream of two entirely unrelated content sections. Both sections are now unparseable as coherent text, and neither can be chunked into a semantically meaningful unit.

Cross-Page Table Severing

Enterprise documents routinely contain tables that span page boundaries. The table header row is on page 11; the data rows continue on page 12. A page-scoped parser processes each page independently. The table on page 11 is extracted as a fragment with no data rows. The continuation on page 12 is extracted as rows with no header context. Neither fragment contains enough information to be retrievable as a meaningful unit. When these fragments are chunked and embedded, they produce vectors that cannot answer any query correctly about that table's content.

Header and Footer Pollution

Page numbers, document titles, section labels, and legal footers appear on every page of an enterprise document. A parser with no layout model injects these strings into the body text at the point where they appear spatially. A chunk that should contain a clean paragraph of policy text instead contains: policy sentence fragment, "Page 47 of 312 — CONFIDENTIAL", remainder of policy sentence. The embedding of that chunk is polluted by the header token sequence, shifting the vector away from its true semantic position.

Silent Image Discard

Charts, diagrams, process flow diagrams, and scanned pages exist in enterprise documents as embedded raster images. A text-extraction parser has no path to retrieve content from these images. It silently skips them. In regulated banking environments — financial disclosures, audit frameworks, risk matrices — a significant fraction of the information that actually matters lives in these images. It never enters the RAG index.

The Embedding Corruption Effect

The cumulative effect of these failures arrives at the embedding model. The embedding model has no knowledge of what the original document looked like. It encodes the text sequence it receives. A corrupted chunk — mixed columns, severed table rows, injected headers — produces a vector in a region of the embedding space that does not accurately represent the semantic content of the source material. That vector is then stored in the index and retrieved in response to queries. The retrieved context is wrong. The LLM generates a response based on wrong context. This is the hallucination. Its origin is the parser.

The failure chain is deterministic:

Bad parsing
  → Corrupted text sequences
    → Structurally invalid chunks
      → Semantically inaccurate vectors
        → Irrelevant retrieval
          → LLM hallucination

No amount of prompt engineering, model selection, or retrieval tuning corrects a failure that originates at stage one.

Production Architecture Pattern and Code Walkthrough

System Architecture Overview

The production ingestion pipeline is composed of four distinct functional layers:

Discovery Layer — recursive filesystem traversal with format filtering
Parsing Layer — layout-aware document conversion using Docling's DocumentConverter with a configured PdfPipelineOptions pipeline
Chunking Layer — structure-aware tokenization and context enrichment using Docling's HybridChunker
Indexing Layer — LangChain Document construction with structured metadata, ready for embedding and vector store ingestion

The pipeline is designed to be format-agnostic at the discovery layer and format-specific at the parsing layer. PDF files pass through the full PdfPipelineOptions pipeline. DOCX and Markdown files pass through Docling's native handlers. This avoids the common mistake of running PDF-specific configuration against non-PDF inputs.

Core Implementation: `chunks.py`

The parsing configuration is the most critical engineering decision in the entire pipeline. Every downstream component inherits the quality of this stage.

from pathlib import Path
from docling.document_converter import DocumentConverter, PdfFormatOption
from docling.datamodel.base_models import InputFormat
from docling.datamodel.pipeline_options import (
    PdfPipelineOptions,
    TableFormerMode,
    HeadingHierarchyOptions
)
from docling.chunking import HybridChunker


def load_docs(dir_path):
    """Recursively discover supported documents. Returns Path objects."""
    ROOT = Path(dir_path)
    EXTS = {".pdf", ".docx", ".md"}
    return [
        p for p in ROOT.rglob("*")
        if p.is_file() and p.suffix.lower() in EXTS and not p.name.startswith(".")
    ]


def parse_file(file_path):
    """
    Parse a single document using layout-aware Docling pipeline.
    Returns a (chunker, document) tuple for downstream chunking.
    """
    file_path = Path(file_path)

    # Pipeline configuration: heading hierarchy and parsed pages enabled;
    # table structure and OCR disabled for native-text PDFs.
    opts = PdfPipelineOptions(do_table_structure=False)
    opts.heading_hierarchy_options = HeadingHierarchyOptions(enabled=True)
    opts.generate_parsed_pages = True
    opts.do_ocr = False

    converter = DocumentConverter(
        format_options={InputFormat.PDF: PdfFormatOption(pipeline_options=opts)}
    )

    doc = converter.convert(file_path).document
    chunker = HybridChunker()
    return chunker, doc


def meta_data(metadata):
    """
    Extract structured metadata from a chunk's ChunkMeta object.
    Page number is derived from provenance data across all doc_items in the chunk.
    """
    page = (
        sorted({p.page_no for it in metadata.doc_items for p in it.prov})[0]
        if metadata.doc_items else 0
    )
    return {
        "page":     page,
        "filetype": metadata.origin.mimetype,
        "filename": metadata.origin.filename,
        "heading":  " > ".join(metadata.headings) if metadata.headings else ""
    }

Core Implementation: `index.py`

import os
from dotenv import load_dotenv
from llm import embed_documents
from chunks import load_docs, meta_data, parse_file
from langchain_core.documents import Document

load_dotenv()


def process_documents(dir_path):
    """
    End-to-end ingestion: discover → parse → chunk → contextualize → index.
    """
    docs = load_docs(dir_path)

    for doc_path in docs:
        chunker, parsed_doc = parse_file(doc_path)

        for i, chunk in enumerate(chunker.chunk(parsed_doc)):
            metadata = meta_data(chunk.meta)

            # contextualize() prepends the heading breadcrumb path to chunk text.
            # This is the text that goes to the embedding model — not chunk.text.
            text = chunker.contextualize(chunk)

            document = Document(
                page_content=text,
                metadata={**metadata, "chunk_index": i}
            )

            # embed_documents(document) — downstream call to vector store


if __name__ == "__main__":
    process_documents(os.environ.get("docs_path"))

Technical Walkthrough: Step by Step

Step 1 — Document Discovery (load_docs)

The discovery function uses Path.rglob("*") to traverse the directory tree recursively. It filters on a defined extension set {".pdf", ".docx", ".md"} and excludes hidden files by checking p.name.startswith("."). This prevents macOS resource fork files (.DS_Store, ._filename) from entering the pipeline and causing parse errors. The function returns Path objects rather than string paths, which allows the downstream parse_file function to leverage pathlib's existence check and type-safe path composition.

Step 2 — Layout-Aware Parsing (parse_file)

This is where the pipeline diverges from a naive implementation. Four specific pipeline options are set:

do_table_structure=False — This disables the TableFormer model, which is Docling's ML-based table cell recognition model. For native-text PDFs (PDFs with an embedded text layer), the structural relationships between cells can be inferred from the text positions without running the full vision model. Disabling it significantly reduces parse latency per document. For scanned documents or PDFs where tables are rendered as images, this flag must be set to True and the corresponding do_ocr=True flag must be enabled.

HeadingHierarchyOptions(enabled=True) — By default, Docling's layout model classifies page regions as SECTION_HEADER without assigning a level. Every heading in the exported document is treated as level 1, producing a flat structure with no parent-child relationship between sections. Enabling HeadingHierarchyOptions activates a post-processing stage that assigns heading levels using three signals in precedence order: PDF bookmarks from the document outline, leading numbering patterns in the heading text (e.g., 1., 1.1, 2.3.4), and visual font styling. This recovered hierarchy is what makes the HybridChunker's structural chunking meaningful.

generate_parsed_pages=True — This option keeps the raw parsed PDF cell data in memory after the layout analysis stage. It is specifically required for the font-style heading inference signal. Without it, the HeadingHierarchyOptions stage can still use bookmarks and numbering patterns, but font weight, slant, and size comparisons are skipped silently. In enterprise documents that have no bookmarks and no numbered headings — dense policy documents, legal agreements, free-form reports — font styling is often the only reliable heading signal available. This flag must be enabled to use it.

do_ocr=False — OCR is disabled for native-text PDFs. Running Tesseract or another OCR engine on a PDF that already has an embedded text layer introduces character recognition errors and significantly increases processing time. OCR should only be enabled for scanned documents or for pages that contain image-only regions.

Step 3 — Hybrid Chunking (HybridChunker)

HybridChunker operates in four sequential stages on the parsed DoclingDocument:

Hierarchical pass — walks the document's element tree and produces initial chunks at structural boundaries: paragraphs, list items, tables, and picture captions. Each chunk carries the heading path of the section it belongs to.
Merge pass — combines consecutive elements that share the same heading path and fit within the token limit. This prevents the common failure of one-sentence chunks that contain insufficient context for accurate retrieval.
Split pass — splits single elements that exceed the token limit. Tables use a LineBasedTokenChunker that repeats the header row for each split segment. Text elements use the semchunk algorithm, which splits at the most semantically meaningful boundary available rather than at a fixed character or token count.
Contextualization — the chunker.contextualize(chunk) call prepends the full heading breadcrumb path to the chunk text before it is passed to the embedding model. A chunk from a section 3 > 3.2 > Access Control Requirements produces an embedding that encodes both the section context and the content — significantly improving retrieval precision for queries that reference specific sections without quoting them verbatim.

Step 4 — Metadata Extraction (meta_data)

The meta_data function derives the page number from the provenance data attached to each doc_item in the chunk's metadata. A chunk that spans multiple pages will have prov entries from each page; the function takes the minimum page number as the canonical page reference. This is more accurate than line-based page attribution because the HybridChunker can merge elements from adjacent pages into a single chunk.

The heading field concatenates the heading path as a " > "-delimited breadcrumb string. This is stored as vector store metadata and enables filtered retrieval: a query scoped to a specific section of a document can filter on the heading field rather than relying solely on vector similarity.

Step 5 — LangChain Document Construction

Each processed chunk is wrapped in a LangChain Document object with page_content set to the contextualized text (not chunk.text) and metadata containing the structured fields plus the sequential chunk_index. The chunk_index enables re-ranking by document position as a secondary signal when multiple chunks from the same document are retrieved with similar cosine scores.

Implementation Strategies and Deep-Dive Considerations

Native PDF vs. Scanned Document Routing

The current implementation sets do_ocr=False globally. In a production corpus that contains a mix of native-text PDFs and scanned documents, this requires a pre-processing routing stage before the converter is invoked.

A practical approach is to probe the document's text layer density before routing:

import fitz  # PyMuPDF

def is_native_text_pdf(file_path: Path, sample_pages: int = 3) -> bool:
    """
    Sample the first N pages to determine if the PDF has a native text layer.
    Returns True for native-text PDFs, False for scanned/image-only PDFs.
    """
    doc = fitz.open(str(file_path))
    pages_to_check = min(sample_pages, len(doc))
    total_chars = sum(
        len(doc[i].get_text("text").strip())
        for i in range(pages_to_check)
    )
    doc.close()
    return total_chars > 100  # threshold: fewer than 100 chars implies image-only pages

Documents that fail the native-text check should be routed to a configuration with do_ocr=True and, depending on table density, do_table_structure=True with TableFormerMode.ACCURATE.

Heading Hierarchy Signal Selection

The HeadingHierarchyOptions signals apply in precedence order. For enterprise document corpora, the appropriate configuration depends on the document type:

Document Type	Recommended Signal Configuration
Regulatory PDF with bookmarked ToC	`use_bookmarks=True` (default), others as fallback
Technical specifications with numbered sections	`use_numbering=True`, `use_bookmarks=False` if ToC is absent or unreliable
Legal agreements, free-form policy documents	`use_style=True`, `use_font_style=True`, requires `generate_parsed_pages=True`
Mixed corpus	All signals enabled (default when `enabled=True`)

For a banking regulatory corpus — MAS TRM guidelines, internal policy documents, audit frameworks — the mixed configuration is appropriate because the document types are heterogeneous.

The `contextualize()` vs `chunk.text` Decision

This is a consistently misunderstood detail in Docling-based RAG implementations. chunk.text contains only the raw text of the chunk's content elements. chunker.contextualize(chunk) prepends the heading breadcrumb path to that text.

For a chunk with headings ["3. Security Controls", "3.2 Access Management"] and body text "All privileged access must be logged...", the outputs are:

chunk.text:
"All privileged access must be logged..."

chunker.contextualize(chunk):
"3. Security Controls
3.2 Access Management
All privileged access must be logged..."

The embedding model encodes both versions. The contextualized version places the vector in a region of the embedding space that reflects the section semantics, not just the sentence semantics. A query for "what are the access management logging requirements" retrieves the contextualized chunk with significantly higher cosine similarity than the raw text chunk, because the heading tokens anchor the embedding to the correct domain.

Always pass chunker.contextualize(chunk) to the embedding model. Store chunk.text separately if you need the raw text for display purposes.

Metadata as a Retrieval Filter Layer

The metadata schema produced by meta_data() — page, filetype, filename, heading, chunk_index — serves a dual purpose. It is stored alongside each vector in the vector store and enables metadata-filtered retrieval.

In regulated environments, this has a direct compliance application. A query scoped to a specific document version (filename filter) or a specific section (heading filter) can be constrained at the retrieval layer rather than relying on the LLM to perform that scoping from the retrieved context. This reduces the surface area for cross-document content bleed, where a retrieved chunk from document version N-1 is used to answer a query that should be answered from version N.

Architectural Tradeoffs and Operational Takeaways

Compute Overhead: `generate_parsed_pages=True`

Keeping the parsed PDF cells in memory (generate_parsed_pages=True) increases per-document memory consumption. For large documents — 200+ pages — this can add 300–500 MB of in-process memory overhead per concurrent parse operation. In a batch ingestion workload running on a fixed instance, the concurrency ceiling must be set to account for this.

The tradeoff is worth it for enterprise corpora where free-form documents without bookmarks are common. For corpora where all documents have reliable PDF outlines or numbered sections, generate_parsed_pages can be set to False to reduce memory pressure, with use_style=False in the HeadingHierarchyOptions.

`do_table_structure=False` Tradeoff

Disabling the TableFormer model eliminates the ML inference cost for table structure recognition, which is the most expensive single operation in the Docling pipeline for table-dense documents. For native-text PDFs, this is a reasonable tradeoff because cell relationships can be preserved through text-position inference.

The failure mode: if native-text PDFs contain tables that use borderless layouts or unconventional spacing, the text-position inference may still produce incorrect cell associations. A validation step that samples the parsed table output before indexing is strongly recommended for corpora with complex table structures.

Batch vs. Streaming Ingestion

The current implementation processes documents sequentially in a single process. For initial corpus ingestion at enterprise scale — thousands of documents — this does not meet practical time constraints. A production deployment should parallelise at the document level using a worker pool:

from concurrent.futures import ProcessPoolExecutor
import os

def process_documents_parallel(dir_path: str, max_workers: int = 4):
    docs = load_docs(dir_path)
    with ProcessPoolExecutor(max_workers=max_workers) as executor:
        futures = {executor.submit(parse_and_index, doc_path): doc_path for doc_path in docs}
        for future in futures:
            try:
                future.result()
            except Exception as e:
                print(f"Failed: {futures[future]} — {e}")

Worker count should be tuned based on available CPU cores and the memory overhead of generate_parsed_pages=True. On an m6i.4xlarge (16 vCPU, 64 GiB), a practical ceiling is 4–6 workers for large-document corpora.

Why This Architecture Scales Where Others Don't

The combination of layout-aware parsing, structure-driven chunking, and heading-enriched contextualization produces an index where retrieved chunks carry their structural position in the source document as part of the vector encoding. This has a measurable effect on retrieval precision for the query types that dominate enterprise RAG use cases: policy lookups, section-specific compliance checks, and cross-document comparisons.

The cost of this precision is paid once, at ingestion time. The alternative — fast parsing, naive chunking, no contextualization — produces an index that is cheap to build and unreliable to query. In a regulated environment, an unreliable RAG system is not a minor inconvenience. It is an audit risk.

Fix the ingestion layer. Everything downstream inherits its quality.

Every LLM Request Has Two Halves. Only One Uses Your GPU Cores

Ankur Jindal — Mon, 31 Aug 2026 06:50:21 +0000

Paste a long document into ChatGPT and hit enter.

Nothing happens for a second or two. Then the answer starts appearing, word by word, at a steady pace until it finishes.

You have seen this hundreds of times. Most people never think about it.

But those are two completely different things happening inside one request, running on two different parts of the same GPU, limited by two different bottlenecks. Once you know what they are, a lot of confusing things about serving AI models stop being confusing.

Including why a faster GPU sometimes makes no difference at all.

This is part two of a series. Part one covered the split between VRAM and cores. This post covers what each of them does when a request arrives.

The example we will follow

I am going to trace one question the whole way through:

what is the capital of france

Simple, short, and it produces a short answer. Perfect for watching the machinery.

Before anything happens, the model server has already loaded the model's weights into VRAM. That happened at startup and they stay there. Nothing about your request loads a model.

Your words get chopped into tokens, which are just pieces of words turned into numbers, and placed in VRAM alongside the weights.

Now the work begins.

Phase one: prefill, or "the pause"

Before the model can write anything, it has to read everything you gave it.

Here is the important part, and it is the opposite of what most people assume: it reads your entire prompt at once. Not word by word. All six words go to the cores together, in a single pass.

That is possible because your whole prompt is already there. Nothing has to wait for anything else. So thousands of cores fire simultaneously and chew through all of it in one burst.

flowchart LR
    A["what is the capital of france<br/>all 6 tokens together"] --> C
    W["16 GB of model weights"] --> C
    C["GPU CORES<br/>every core busy"] --> O1["First word: 'The'"]
    C --> O2["Notes saved to VRAM<br/>the KV cache"]

Two things come out of that single pass:

The first word of your answer. Just one.
A set of working notes, saved into VRAM. These notes summarise everything the model worked out about your prompt. The technical name is the KV cache.

The moment that first word appears is the moment the pause ends.

How long is the pause? It depends entirely on how much you gave it to read. Six words is nothing, so you would not notice. Paste a hundred-page contract and that is tens of thousands of tokens pushed through in one burst, and now the pause is real.

This phase has a name: prefill. And the pause has a name too: TTFT, time to first token. It is one of the two numbers every serving benchmark reports.

What is the bottleneck here? The cores. There is a mountain of maths to do and they are all doing it. This phase is compute-heavy.

Phase two: decode, or "the typing"

Now the model writes the rest of the answer. One word per pass.

To produce each word, the cores need three things: the notes from before, the word that was just produced, and the model's weights. All of it read out of VRAM.

flowchart LR
    N["Notes in VRAM<br/>the KV cache"] --> C
    L["The word just produced"] --> C
    W["16 GB of weights<br/>read again, in full"] --> C
    C["GPU CORES<br/>barely used"] --> O["One new word"]
    O -.->|"repeat"| N

That loop runs once per word until the answer is done:

Lap 1 → "The"
Lap 2 → "capital"
Lap 3 → "is"
Lap 4 → "Paris"

This is why the answer types itself out instead of appearing all at once. You are watching each lap finish. Streaming is not a feature someone built. It is just what happens when each word is sent the moment it exists.

This phase is called decode, and the gap between words is TPOT, time per output token. That is the second benchmark number.

What is the bottleneck here? Not the cores. Look at what has to happen for a single word: the entire model, all 16 GB of it, has to travel out of VRAM and past the cores. The cores glance at it, do a small amount of maths, and produce one word.

Then it all happens again for the next word.

The cores are mostly idle. The memory pipe is flat out. Decode is limited by how fast the GPU can read its own memory.

Why word three does not need a fresh pause

Here is the question I got stuck on, and it is worth its own section.

To predict word 3, the model needs to know everything before it. Your prompt, plus word 1, plus word 2. So does it read all of that again?

If it did, every single word would have its own pause. The whole thing would be unusable.

It does not, because of the notes.

Every word that gets produced is added to the notes as it goes. So by the time lap 3 starts, the notes already contain your prompt, "The", and "capital". The model reads those notes and only has to process the one word that just came out.

Lap 3 reads:  notes (prompt + "The" + "capital")  +  the word "capital"
Lap 3 does:   process one word
Lap 3 gives:  "is"

Nothing gets forgotten and nothing gets reprocessed. The model always sees the full history. It just never redoes the work.

You pay for the pause once, not once per word.

Now the part that changes how you buy hardware

Here is where this gets practical.

Every word requires one full read of the model out of VRAM. So the maximum speed of a model on a given card is a division:

How fast can the card read its own memory?
─────────────────────────────────────────  =  words per second
        How much is there to read?

Let me do it with real numbers.

Take an AWS g6e.xlarge. The GPU inside is a single NVIDIA L40S with 48 GB of VRAM. NVIDIA's datasheet gives its memory bandwidth as 864 GB/s.

Now put a model on it. Qwen3.8-27B is a 28-billion-parameter model. In its published form each weight takes 2 bytes, so:

28 billion × 2 bytes = 56 GB

That does not fit in 48 GB. So we use the Quantization version FP8 , where each weight takes 1 byte:

28 billion × 1 byte = 28 GB

Now the division:

864 GB/s ÷ 28 GB ≈ 31 words per second

That is the ceiling. Two numbers off two spec sheets, and you know roughly how fast that model will generate on that card. No deployment required to calculate this.

And notice what shrinking the model did. At 56 GB the ceiling would have been about 15 words per second. Halving the weights doubled the speed, because there is half as much to read every single lap.

A caveat worth stating plainly: that number is a theoretical ceiling for one user, and real throughput lands meaningfully below it. The notes have to be read too, and no software achieves perfect memory utilisation. Treat it as a sanity check, not a promise.

The AWS gap nobody mentions

Here is something that caught me out, and I suspect it catches out a lot of people.

That bandwidth figure, the number that decides your generation speed, is not on AWS GPU Instance page.

Look at an EC2 GPU instance table and you get the card name, GPU memory, vCPUs, system memory, and storage. Some tables have a column called "EBS Bandwidth" and another called "Network Bandwidth", which look like the answer and are not. Those are disk and network throughput. Neither has anything to do with VRAM.

The three bandwidths worth separating:

Bandwidth	Between what	Typical scale
Memory bandwidth	VRAM and cores	hundreds of GB/s
PCIe	GPU card and server	tens of GB/s
Network / EBS	Server and the outside world	Gbps

Only the first one governs how fast words come out, and AWS does not publish it.

So the workflow is: read the card name off the AWS table, then go to NVIDIA's datasheet for that card and find the field labelled Memory Bandwidth.

Everything above describes one user.

You might reasonably be looking at "31 words per second" and thinking that sounds poor for a server. And it would be, if that was all it could do.

But look again at what happens during decode. The GPU reads the entire model out of VRAM to produce one word for one person, and the cores sit mostly idle while it happens.

So why not use that same read to produce a word for fifty people at once?

That is exactly what real serving software does, and it is why the economics of hosting a model work at all. It also introduces a new limit, a different kind of bottleneck, and the reason ordinary load balancing falls apart completely for AI workloads.

That will cover in upcoming posts.

A GPU Is Two Things, and Only One of Them Holds Your Model

Ankur Jindal — Thu, 27 Aug 2026 08:39:52 +0000

I have run AWS infrastructure for years. When I started looking at how large language models actually get served, I hit a wall on something basic.

Someone said "the model sits on the GPU." Someone else said "the GPU does the calculation." Both sounded right. Neither told me where the model actually lives.

The problem is that "GPU" is not one thing.

Two parts, one board

You already know how a normal machine is laid out. A CPU does the work. RAM holds the data. They sit apart, connected by a bus.

A GPU card is the same pair, shrunk onto one board:

Cores — the chip in the middle. Thousands of small processors that do maths.
VRAM — memory chips around it. Holds the data.

So when people say different things, they are talking about different halves:

What they say	What they mean
"The GPU has 80GB"	The VRAM
"The GPU is fast"	The cores
"The model is on the GPU"	The weights are in VRAM

Neither half is "the GPU" on its own.

The cores hold nothing

This is the part that took me longest to accept.

The cores have almost no storage. They are hands, not shelves. The model does not get loaded into them once and stay there. It has to be streamed across from VRAM, continuously, for every calculation.

An 8 billion parameter model at full precision is about 16GB of numbers. Those numbers live in VRAM. To produce a single word of an answer, the GPU streams the whole thing across to the cores, does the maths, and gets one word back. Then it does it again for the next word.

That one fact explains why answers type themselves out instead of appearing all at once.

Where this bites you on AWS

Here is the practical part.

Look at a GPU instance in the EC2 pricing console:

g6.4xlarge    $1.3232/hr    16 vCPU    64 GiB    1 x 600 GB NVMe

That 64 GiB looks like plenty of room for a model. It is not GPU memory. It is the server's ordinary system RAM, and inference barely touches it.

The GPU on that instance is a single NVIDIA L4 with 24GB of VRAM. That 24GB is the only number that decides whether your model loads.

The EC2 pricing page has no GPU column at all. You have to cross-reference the instance-types page to find the card and its memory.

Worth memorising for the common G-family instances:

Family	GPU	VRAM
g4dn	T4	16GB
g5	A10G	24GB
g6	L4	24GB
g6e	L40S	48GB

One sentence to take away

Weights live in VRAM. Cores do the maths. Neither one is "the GPU" by itself.

Everything else in model serving follows from that split.

DEV Community: Ankur Jindal

AWS Lambda Managed Instances: What Min=0/Max=0 Actually Does in Production

Introduction: The Production Bottleneck

The Mechanics of the Failure Chain

Production Architecture Pattern & Walkthrough

System Architecture Overview

Setup Walkthrough: Step by Step

Implementation Strategies & Deep-Dive Considerations

Architectural Tradeoffs & Operational Takeaways

Why Your Enterprise RAG Pipeline Is Failing Before the First Query Runs

Introduction: The Ingestion Layer Is the Real Bottleneck

The Mechanics of the Document Parsing Failure Chain

Multi-Column Layout Corruption

Cross-Page Table Severing

Header and Footer Pollution

Silent Image Discard

The Embedding Corruption Effect

Production Architecture Pattern and Code Walkthrough

System Architecture Overview

Core Implementation: chunks.py

Core Implementation: index.py

Technical Walkthrough: Step by Step

Implementation Strategies and Deep-Dive Considerations

Native PDF vs. Scanned Document Routing

Heading Hierarchy Signal Selection

The contextualize() vs chunk.text Decision

Metadata as a Retrieval Filter Layer

Architectural Tradeoffs and Operational Takeaways

Compute Overhead: generate_parsed_pages=True

do_table_structure=False Tradeoff

Batch vs. Streaming Ingestion

Why This Architecture Scales Where Others Don't

Every LLM Request Has Two Halves. Only One Uses Your GPU Cores

The example we will follow

Phase one: prefill, or "the pause"

Phase two: decode, or "the typing"

Why word three does not need a fresh pause

Now the part that changes how you buy hardware

The AWS gap nobody mentions

Everything above describes one user.

A GPU Is Two Things, and Only One of Them Holds Your Model

Two parts, one board

The cores hold nothing

Where this bites you on AWS

One sentence to take away

Core Implementation: `chunks.py`

Core Implementation: `index.py`

The `contextualize()` vs `chunk.text` Decision

Compute Overhead: `generate_parsed_pages=True`

`do_table_structure=False` Tradeoff