DEV Community: Ankur Jindal

Hybrid Search and Reranking: What Each One Actually Buys You

Ankur Jindal — Tue, 06 Oct 2026 12:29:22 +0000

Enterprise RAG Engineering — Part 2 of 3

Where Part 1 Left Off

Part 1 argued that most enterprise RAG systems fail at the ingestion layer, before a single query is ever issued — and that the fix is layout-aware parsing rather than a better model.

We built that stage. Docling with HeadingHierarchyOptions enabled to recover section structure, generate_parsed_pages on so the font-style signal survives, HybridChunker to split at structural boundaries rather than fixed token counts, and metadata carried through to every chunk: page number, filename, and the heading breadcrumb the chunk sits under.

The result is an index where structure survived. Tables did not get flattened into a single text stream. Headings stayed attached to the content they govern. Chunks mean something on their own.

That was the write path. This article is about the read path — what happens when the first query arrives.

Because a corpus can be indexed perfectly and still fail to return the right chunk.

The Query That Breaks Pure Semantic Search

The default retrieval pattern in almost every RAG tutorial is three lines long. Embed the question, run a nearest-neighbour search, return the top k. It works well enough in a proof of concept that most teams ship it.

Then a user searches for WH-1000XM5.

The system returns wireless headphones. All of them. Not that one.

This is not a tuning problem, and no amount of prompt engineering fixes it. It is a property of how embeddings work.

An embedding model compresses a span of text into a fixed-width vector that encodes what the text is about. A model number is not about anything. It is an opaque string with no distributional meaning, and whatever position it occupies in vector space is determined almost entirely by the words around it. Two chunks that both discuss wireless headphones sit close together regardless of which specific model each one names.

The same failure appears with any exact identifier: a part number, an error code, a control reference, a contract clause, an internal project name. Enterprise corpora are full of them, and users search by them constantly.

Lexical search has the exact opposite profile. It matches the token and ranks by term statistics, so it finds WH-1000XM5 on the first hop. And it fails completely when a user asks "how long does the battery last" against a page that only ever says "playback time" — because there is no shared token to match.

Neither retriever is better. They fail on different things. That is the entire argument for running both.

Why You Cannot Just Run Both and Add the Scores

Running two retrievers is easy. Merging their results is where hybrid search is actually decided, and the obvious approach fails in a way worth understanding precisely.

The two scores are not comparable

Cosine distance is bounded. For normalised vectors it falls in a known range, and a given value means roughly the same thing across queries and across corpora.

Lexical ranking functions are not bounded. Postgres ts_rank_cd and BM25 both produce unbounded positive numbers whose magnitude depends on term frequency, match proximity, document length, and the statistics of the corpus being searched. Index another ten thousand documents and the same document scores differently against the same query.

So one retriever returns 0.83 and the other returns 24.7, and there is no constant that reconciles them — because the relationship between the two scales is not fixed.

Normalising first makes it worse

The usual workaround is min-max normalisation: scale each list to the range 0 to 1, then blend with a weight.

This introduces a subtler failure. The normalisation is computed over the candidates that were returned, not over the corpus. If the lexical arm returns 50 documents that are all weak matches, the best of those 50 normalises to 1.0 — the same score a perfect match would receive.

The normalisation destroys exactly the signal you were trying to preserve, and it does so silently, in precisely the case where the retriever had nothing good to offer.

Rank fusion sidesteps the problem instead of solving it

Reciprocal Rank Fusion, introduced by Cormack, Clarke and Büttcher at SIGIR 2009, takes the position that the scores are unsalvageable and discards them entirely. Each document is scored on its position in each list:

RRF(d) = Σ  1 / (k + rank_i(d))
         i

rank_i(d) is the document's 1-indexed position in list i, and k is a smoothing constant. The original paper fixed k = 60 during a pilot investigation and found it near-optimal across TREC collections without being especially sensitive to the exact value. It is now the default in Elasticsearch and OpenSearch.

Because it consumes only rank positions, RRF can fuse any number of ranked lists from any retrievers, whatever they score and however they score it.

What the constant 60 does

Rank	Contribution	Share of rank-1
1	1 / 61 = 0.01639	100%
2	1 / 62 = 0.01613	98.4%
20	1 / 80 = 0.01250	76.3%
100	1 / 160 = 0.00625	38.1%

The curve is deliberately flat at the top. Moving from rank 1 to rank 2 costs 1.6% of the score. Moving from rank 1 to rank 20 costs about 24%.

The consequence is that agreement outranks confidence. A chunk ranked second by both retrievers scores roughly 0.0322. A chunk ranked first by one retriever and missing from the other scores 0.0164. The chunk both retrievers found wins by a factor of two, despite neither ranking it first.

With k = 0 the formula collapses to pure reciprocal rank, where rank 1 scores 1.0 and rank 2 scores 0.5, and a single retriever's top hit dominates everything. The 60 damps that until consensus between independent signals becomes the deciding factor.

How We Built It

Both retrieval arms, the fusion, and the payload fetch run in one database round trip.

Why one statement

The alternative is two queries, both result sets pulled into the application, and fusion in Python. That costs two round trips plus a merge loop, and it moves ranking logic away from the layer that holds the data.

The database can express RRF directly. ROW_NUMBER() produces the ranks. FULL OUTER JOIN produces the union. The arithmetic is two divisions and an addition per row. The rows are already there — making the database do the merge is strictly cheaper than shipping them out to merge elsewhere.

The query

# Builds an OR query from the question's words, entirely in the database.
# Same stemming and stopword handling as the stored index itself.
ANY_TSQUERY = r"""CAST(array_to_string(ARRAY(
    SELECT '''' || replace(replace(lexeme, E'\\', E'\\\\'), '''', '''''') || ''''
    FROM unnest(tsvector_to_array(to_tsvector('english', :question))) AS lexeme
), ' | ') AS tsquery)"""

HYBRID_SQL = f"""
WITH keyword AS (
    SELECT id,
           ROW_NUMBER() OVER (ORDER BY ts_rank_cd(fts, {ANY_TSQUERY}) DESC) AS rank
    FROM chunks
    WHERE fts @@ {ANY_TSQUERY}
    ORDER BY ts_rank_cd(fts, {ANY_TSQUERY}) DESC
    LIMIT :pool
),
semantic AS (
    SELECT id, ROW_NUMBER() OVER (ORDER BY embedding <=> CAST(:qv AS vector)) AS rank
    FROM chunks
    ORDER BY embedding <=> CAST(:qv AS vector)
    LIMIT :pool
)
SELECT c.id, c.content, c.page, c.heading,
       COALESCE(1.0/(60+k.rank), 0) + COALESCE(1.0/(60+s.rank), 0) AS score,
       k.rank AS keyword_rank,
       s.rank AS vector_rank
FROM keyword k
FULL OUTER JOIN semantic s USING (id)
JOIN chunks c USING (id)
ORDER BY score DESC
LIMIT :pool;
"""

The decisions inside it

Building the lexical query in the database, not in Python. The obvious approach is to split the question in Python and join the terms with an OR operator. That introduces a second tokenizer — and Python does not apply the same stemming or stopword list that produced the stored index. The result is a query whose terms do not match what was indexed.

ANY_TSQUERY avoids this by running the question through to_tsvector('english', ...), the same function with the same configuration that produced the stored vectors, then unnesting the lexemes, quoting each, and joining them. Query-side and index-side analysis are identical because they are the same call.

The replace calls escape backslashes and quotes so a lexeme containing either cannot break out of the quoted literal.

OR, not AND. Terms are joined with the OR operator, making this recall-oriented: a chunk matching any term is a candidate. In a fusion pipeline that is correct. The lexical arm's job is to surface anything plausible; precision is handled downstream.

FULL OUTER JOIN, not INNER JOIN. This is the structural decision that makes the fusion behave correctly.

An INNER JOIN keeps only chunks that both retrievers found — which discards exactly the cases hybrid search exists to catch. The SKU that only the lexical arm found. The paraphrase that only the vector arm found. A LEFT JOIN privileges one arm arbitrarily.

With FULL OUTER JOIN, a chunk present in one list and absent from the other survives with a NULL rank on the missing side, and COALESCE(..., 0) contributes zero for that side rather than propagating NULL through the arithmetic. A single-arm hit scores about 0.0164. A two-arm hit scores up to 0.0328. Both are retrievable; the one both retrievers agreed on ranks higher.

Deferring the payload fetch. The CTEs carry only ids and ranks. Both arms rank over the full table, and carrying content through them would mean materialising large text values for rows about to be discarded. Joining for the payload after the fusion keeps the intermediate result sets narrow.

Two different depths. pool controls how deep each arm searches and how many fused candidates survive. The request's k controls how many documents come back at the end. They are not the same parameter, and binding the request's k to the SQL's final LIMIT would cap the candidate pool at the number of results the user asked for — starving the reranker of exactly the candidates it exists to re-order.

What Hybrid Search Bought Us

The gains are capability-level, and they are specific.

Identifier queries work. A search for a model number, a part code or a clause reference now returns the chunk containing that string, because the lexical arm finds it on exact match. Before, those queries returned topically adjacent chunks and nothing else. This is the single largest practical difference for an enterprise corpus.

Paraphrase queries still work. The vector arm is unchanged, so questions phrased conceptually against documents using different vocabulary behave exactly as they did. Hybrid search is additive — nothing that worked before stopped working.

Single-arm hits are not lost. Because of the FULL OUTER JOIN, a chunk only one retriever found still reaches the candidate set. It ranks below chunks both arms agreed on, which is the correct priority, but it is retrievable. An INNER JOIN implementation would silently drop exactly the results that justify running two retrievers.

The candidate pool is wider without being noisier. Two independent retrievers surface a more diverse set of candidates than either alone, and RRF orders them by cross-retriever agreement rather than by whichever scoring function happened to produce a larger number.

No scale calibration to maintain. Because RRF consumes only rank positions, there is no normalisation constant that drifts as the corpus grows, and no weight to re-tune after an ingestion run. The fusion behaves the same on 1,000 chunks as on 100,000.

One round trip, not two. The entire fusion happens inside the database. There is no second query, no application-side merge loop, and no window where the two result sets could come from different states of the data.

A note on evidence: these are mechanical gains, demonstrable by construction. Quantified relevance improvement requires running a labelled evaluation set against both configurations — which is the right next step, and which the candidate-depth and threshold parameters should be tuned against rather than by intuition.

Why We Added a Reranker

Hybrid search produces a better candidate set. It does not produce a better answer, and the reason is worth stating plainly.

The fused score is not a relevance score

The maximum possible RRF score in a two-retriever system is 2/61 — about 0.0328 — achieved by a chunk ranked first by both arms. That ceiling is fixed by the formula, not by match quality.

A chunk that answers the question perfectly and a chunk that is merely well-positioned in both lists receive identical scores if they occupy identical ranks. The number tells you where a chunk sat in two ordered lists. It tells you nothing about whether the chunk answers anything.

This has a direct operational consequence: any threshold applied to a fused score is a threshold on rank position, which is not the same thing as a threshold on relevance and will behave unpredictably as the corpus changes.

What a reranker does differently

Both retrieval arms score the query and the document separately. The vector arm embeds each independently and compares the results. The lexical arm computes term statistics without ever considering the query and document jointly. That independence is what makes them fast and indexable — and it is also their ceiling.

A cross-encoder reranker processes the query and the document together in a single forward pass. It can model the interaction between them: whether this specific document actually answers this specific question, rather than whether they occupy nearby regions of a vector space or share tokens.

That is far more expensive per document, which is exactly why it runs last, over a pool of around 50 candidates rather than the whole corpus. The architecture is deliberate: cheap indexed retrieval narrows millions of chunks to tens, then expensive joint scoring orders those tens.

In this system that stage is Amazon Bedrock's amazon.rerank-v1:0, called through the bedrock-agent-runtime endpoint. The API accepts up to 1,000 inline sources per request, so a 50-candidate pool needs no batching.

What it bought us

A real quality signal. The reranker's relevanceScore is the only number in the entire pipeline that measures whether a chunk answers the question. Everything upstream measures position or proximity.

A threshold that means something. Because that score reflects relevance rather than rank, a minimum threshold becomes a meaningful gate. Chunks that survived retrieval on position alone but do not actually answer the question get dropped before they reach the prompt.

Variable result counts, honestly. A request asking for three results may return fewer — or none. In one evaluation run against a compliance corpus, a question with k=3 returned a single source scoring 0.2067, and the generated answer matched the expected ground truth. The system was not failing. It was reporting that one chunk in the corpus answered that question.

That behaviour is only possible because a relevance score exists to threshold against. Without it, three chunks come back regardless, and the two weak ones go into the prompt alongside the good one.

Cleaner context for generation. Fewer, better chunks means a shorter prompt, fewer tokens, and less opportunity for the model to anchor on a weakly relevant passage. The quality gate pays for itself at the generation stage as well as the retrieval stage.

A usable degraded path. Because RRF ordering already exists before the reranker runs, a rerank failure has a sensible fallback: return the fused results truncated to k. This matters in practice — rerank endpoints throttle, and during testing a real ThrottlingException was absorbed this way without failing the request.

try:
    reranked_results = await rerank_documents(k, question, documents)
except Exception:
    logger.exception("rerank failed, falling back to RRF order")
    return list(islice(documents, k))

Two things about this fallback are deliberate. It catches bare Exception, which is correct here because any rerank failure should degrade rather than fail, and enumerating every exception a client library might raise is a losing game. And it skips the relevance threshold, because fallback documents carry no score to filter on — the caller gets up to k RRF-ranked chunks that may include weak matches. That is the trade: a complete answer on a weaker ranking, rather than no answer. The absent score should be visible to the caller rather than silently defaulted.

What You Give Up Without Each One

Without hybrid search

Exact-identifier queries fail silently. This is the expensive one. A user searching for a model number, an error code or a clause reference gets plausible-looking results that do not contain what they asked for. There is no error and no empty result — just confidently wrong context handed to the model, which then produces a fluent answer about the wrong thing. The failure is invisible until someone checks the citation.

Rare and domain-specific terms degrade. Jargon that is thin in the embedding model's training data has no reliable position in vector space. It embeds near whatever it superficially resembles. Lexical matching does not care how common a term is — it either appears or it does not.

Recall depends entirely on one model's view of meaning. If the embedding model's notion of similarity does not fit your domain, there is no second signal to compensate. Every retrieval failure traces to the same single point.

Negation and precise phrasing blur. Embeddings capture topic well and precise logical structure poorly. Two passages stating opposite conclusions about the same subject can sit close together in vector space.

Without a reranker

You have no relevance signal at all. This is the fundamental loss. Retrieval scores — fused or raw — measure position and proximity. Nothing in the pipeline measures whether a chunk answers the question. Any threshold you set is a threshold on something other than what you care about.

Every query returns exactly k results, whether or not they are any good. A question your corpus cannot answer returns k chunks anyway, because the retrievers always return their top k by position. Those chunks go into the prompt, and the model does what models do with weakly relevant context — it finds something to say.

You cannot distinguish "nothing matched" from "something matched." The system loses the ability to say I don't have that, which in a compliance or support context is often the most valuable answer available.

The top of the list is nearly undifferentiated. The flat RRF curve that makes consensus robust also means ranks 1 through 5 are separated by a few percent. Served directly, that ordering is weak — the fusion was designed to select a candidate set, not to produce a final ranking.

Prompt context gets noisier. More marginal chunks reach the model, consuming tokens and competing for attention with the one that actually answers the question.

The combined picture

Hybrid search without reranking gives you a wider net and no way to judge what you caught. Reranking without hybrid search gives you good judgement applied to a candidate set that may never have contained the right answer.

The two solve different halves of the same problem. Retrieval decides what is available to answer with. Reranking decides what is worth answering with.

What's Next

Everything described here runs against a database on a laptop. Part 3 takes it to AWS: S3 as the document store, Amazon OpenSearch Service as the vector and lexical index, and an event-driven ingestion path that starts when a file lands in a bucket rather than when someone runs a script.

That move changes where this article's fusion logic lives. The FULL OUTER JOIN and the hand-written 1/(60+rank) arithmetic become a search pipeline declaration — OpenSearch implements RRF as a score-ranker-processor with a configurable rank_constant that defaults to the same 60. Part 3 covers what that buys, what it costs, and what the migration does to retrieval quality measured against the same evaluation set.

References

Cormack, Clarke and Büttcher, Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods, SIGIR 2009
Amazon Bedrock Rerank API reference
PostgreSQL text search controls

AWS Lambda Managed Instances: What Min=0/Max=0 Actually Does in Production

Ankur Jindal — Thu, 17 Sep 2026 02:38:34 +0000

A hands-on walkthrough, including the exact failures and dead ends you'll hit setting this up.

Introduction: The Production Bottleneck

AWS Lambda Managed Instances (LMI) puts Lambda functions on EC2 capacity you control, while keeping Lambda's programming model. Most of the coverage since re:Invent 2025 has focused on two things: the removal of the memory ceiling, and the EC2-backed pricing model that lets you apply Savings Plans and Reserved Instances.

Almost none of it covers the scaling configuration — specifically, what happens when MinExecutionEnvironments and MaxExecutionEnvironments are both set to 0.

The assumption most engineers carry over from standard Lambda is that a minimum of zero means "no standing capacity, scale up from zero on the next request." That assumption is wrong for LMI, and it's wrong in a way that produces a hard invoke failure instead of a slow cold start. This post documents the actual mechanics, verified against a live AWS account, along with the setup friction that isn't in the AWS documentation.

The Mechanics of the Failure Chain

On Lambda Managed Instances, MinExecutionEnvironments and MaxExecutionEnvironments are not independent settings. AWS enforces a pairing rule: a minimum of 0 is only accepted when the maximum is also 0.

When both are set to 0, Lambda does not idle the function and wait for the next invocation. It deactivates the function version:

Every EC2 Managed Instance backing that version is terminated.
Instance charges continue until termination actually completes — the meter doesn't stop the moment you save the config.
The version's State flips to Deactivated.
Any invocation against a deactivated version returns an explicit error. There's no cold start, no queueing, no retry — it fails outright.
Reactivation is never automatic. You have to push a new scaling configuration with non-zero values, through the console, the PutFunctionScalingConfig API, or a scheduled action.

This is the opposite of how ReservedConcurrentExecutions=0 or standard Lambda concurrency throttling behaves, and it's the single most important operational difference between LMI and normal Lambda for anyone running non-continuous workloads.

Production Architecture Pattern & Walkthrough

System Architecture Overview

LMI has one prerequisite that standard Lambda doesn't: a Capacity Provider. This is a separate resource that defines:

The VPC, subnets, and security groups the EC2 instances launch into
Which instance architecture (x86_64 or arm64) and types are eligible
The scaling mode (Auto or Manual)
The IAM role Lambda uses to manage EC2 on your behalf (the "Operator Role")

A function is then attached to a Capacity Provider at creation time by setting its compute type. Only after that attachment exists — and only on a published version, not $LATEST — do MinExecutionEnvironments and MaxExecutionEnvironments become meaningful.

Setup Walkthrough: Step by Step

1. Create the Capacity Provider and attach it to the function.

In the Lambda console, this doesn't appear as "Compute type" the way older documentation describes it — in the current console it's the EC2 capacity provider toggle under Custom settings. Toggling it on opens a side panel to select the capacity provider ARN, memory size, and the execution-environment-memory-per-vCPU ratio:

The panel already resolves the capacity provider by ARN once created. Memory and the vCPU ratio (2:1, 4:1, or 8:1) are set per-function here, not per-capacity-provider.

2. Architecture must match between the function and the capacity provider.

The first real failure: creating the function returned this error on save —

"You cannot use a Lambda Managed Instances function with a capacity provider that does not support the architecture of the function."

The capacity provider's InstanceRequirements.Architectures is fixed at creation time (in this case, arm64). If the function's own architecture toggle doesn't match, function creation is rejected outright — there's no coercion or fallback. Confirmed via CLI:

{
    "CapacityProvider": {
        "CapacityProviderArn": "arn:aws:lambda:us-east-1:<ACCOUNT_ID>:capacity-provider:lambaprovisionedinstance",
        "State": "Active",
        "VpcConfig": {
            "SubnetIds": [
                "subnet-<redacted>",
                "subnet-<redacted>"
            ],
            "SecurityGroupIds": [
                "sg-<redacted>"
            ]
        },
        "PermissionsConfig": {
            "CapacityProviderOperatorRoleArn": "arn:aws:iam::<ACCOUNT_ID>:role/service-role/lambaprovisionedinstance-role-<redacted>"
        },
        "InstanceRequirements": {
            "Architectures": ["arm64"]
        },
        "CapacityProviderScalingConfig": {
            "ScalingMode": "Auto"
        }
    }
}

Fix: match the function's architecture toggle to whatever the capacity provider was created with. There's no cross-architecture capacity provider.

3. Publishing does not prompt for scaling configuration.

Creating and publishing the function does not ask for MinExecutionEnvironments or MaxExecutionEnvironments at any point. That configuration lives separately, under Configuration → Function scaling configuration, edited independently after the function exists:

With nothing configured, AWS's default behavior applies: 3 Managed Instances provisioned across availability zones before the version is marked Active, invisible in this panel until you explicitly edit it.

4. $LATEST is not invocable on LMI — only a published version is.

This is a second point of confusion that isn't obvious from the console. Querying $LATEST directly shows a state most engineers won't recognize:

{
    "Configuration": {
        "FunctionName": "lmi-function",
        "FunctionArn": "arn:aws:lambda:us-east-1:<ACCOUNT_ID>:function:lmi-function:$LATEST",
        "State": "ActiveNonInvocable",
        "Architectures": ["arm64"]
    }
}

LMI introduces $LATEST.PUBLISHED specifically for this: a version you can republish repeatedly without managing numbered versions. When created through the console, this is generated automatically. Querying it directly confirms the actual invocable state:

{
    "Configuration": {
        "FunctionName": "lmi-function",
        "FunctionArn": "arn:aws:lambda:us-east-1:<ACCOUNT_ID>:function:lmi-function:$LATEST.PUBLISHED",
        "Version": "$LATEST.PUBLISHED",
        "State": "Active",
        "Architectures": ["arm64"]
    }
}

Invoking the function via its unqualified ARN implicitly targets $LATEST.PUBLISHED, not $LATEST — the reverse of how unqualified invokes behave on standard Lambda.

5. Managed Instances are hidden from the EC2 console by default.

Once $LATEST.PUBLISHED showed Active, the expected 3 instances weren't visible anywhere in the EC2 console, and describe-instances with a capacity-provider tag filter returned nothing:

{
    "Reservations": []
}

The cause: Managed resource visibility, an account-wide EC2 setting (introduced April 2026) that hides EC2 resources provisioned by services like LMI, EKS Auto Mode, and ECS Managed Instances from the console and Describe* API calls by default for accounts that didn't have managed resources before the setting existed. The instances are running and billing regardless — visibility is purely a display filter.

Fix, via CLI:

aws ec2 modify-managed-resource-visibility --default-visibility visible

{
    "Visibility": {
        "DefaultVisibility": "visible"
    }
}

This is account-wide and Region-specific, and applies to all managed-resource offerings, not just LMI. After setting it, the instances appeared after a while in the EC2 console:

Two running, one terminated — consistent with AWS's documented default of 3 instances provisioned for availability across AZs, with normal instance replacement happening in the background.

6. Verifying the 0/0 deactivation.

Editing the Function scaling configuration to MinExecutionEnvironments=0 and MaxExecutionEnvironments=0 and saving triggers an immediate, visible state transition in the console:

"Deactivating version $LATEST.PUBLISHED."

Within a few minutes, all three EC2 instances move to Terminated, and invoking the function returns an explicit error referencing the deactivated state — not a timeout, not a cold start, not a retry. This is the core behavior the rest of this post is about: 0/0 is a deactivation switch, not a scale-to-zero setting.

Implementation Strategies & Deep-Dive Considerations

Pricing changes the idle-cost math. LMI drops Lambda's per-invocation duration billing entirely. Instead, pricing has three components: the standard $0.20 per million requests, standard EC2 instance charges (Savings Plans and Reserved Instances apply), and a flat 15% management fee on top of the on-demand EC2 price — applied even when Savings Plans or RIs are discounting the underlying instance cost. That fee doesn't pause during idle traffic. It only stops when instances are actually terminated, which only happens through deactivation or explicit scale-down.

For a workload with, say, 5 hours of daily invocation activity, this means: unless MinExecutionEnvironments is actively brought to 0/0 during the other 19 hours, you are paying full EC2 rate plus the management fee for the entire idle window — not "only for the 5 hours," the way standard Lambda duration billing would imply.

There's no scale-out SLA. AWS does not publish a guaranteed scale-out latency for LMI. The explicit documented warning is that traffic more than doubling within a 5-minute window can produce throttling while capacity catches up. Combined with the 0/0 deactivation behavior, this means bursty or unpredictable workloads should generally avoid 0/0 entirely — the risk isn't a slow response, it's outright invoke failures during both the reactivation window and any sufficiently sharp traffic spike even at steady-state minimums.

EventBridge Scheduler is the correct pattern for predictable idle windows, not manual toggling. A nightly batch job or a fixed 9-to-5 traffic pattern is exactly the case 0/0 is designed for — scheduled deactivation before the idle window, scheduled reactivation before traffic resumes, via PutFunctionScalingConfig calls triggered on a schedule rather than through the console.

Managed resource visibility is a one-time account setting, not a per-function toggle. Worth setting to visible early in any LMI adoption, since the default-hidden behavior for new accounts otherwise makes basic troubleshooting (like confirming instance count) unexpectedly difficult.

Architectural Tradeoffs & Operational Takeaways

Workload shape	Recommended scaling config	Why
Predictable idle windows (nightly batch, fixed business hours)	`0/0` + EventBridge Scheduler	Stops EC2 + management-fee billing during confirmed idle time; reactivation is scheduled, not reactive
Bursty or unpredictable traffic	`MinExecutionEnvironments > 0`	Avoids hard invoke failures during the deactivated window and reduces exposure to the "traffic doubles in 5 minutes" throttling case
Continuous, steady-state traffic	Standing non-zero minimum, sized to baseline	You're paying for compute regardless; this is the workload LMI's EC2 pricing model is built for

The mechanical rule worth internalizing: MinExecutionEnvironments=0 is not a cost-saving default the way it is on standard Lambda concurrency settings. It only saves cost when paired with Max=0, and pairing it with Max=0 converts idle time into a hard failure window unless you're actively scheduling reactivation around it.

Why Your Enterprise RAG Pipeline Is Failing Before the First Query Runs

Ankur Jindal — Fri, 11 Sep 2026 07:24:37 +0000

Introduction: The Ingestion Layer Is the Real Bottleneck

Most teams building Retrieval-Augmented Generation (RAG) systems invest their engineering effort in two places: the LLM and the retrieval layer. They tune prompts, experiment with embedding models, compare vector databases, and benchmark retrieval precision. When the system underperforms in production, they go back to those same two layers and tune again.

The failure, in the majority of enterprise RAG deployments, is not there.

It is at the ingestion layer — specifically at the document parsing stage — and it happens before a single query is ever issued.

This is not a problem that appears during proof-of-concept work. In a POC, teams typically use small, clean documents: a few well-structured PDFs, maybe a Markdown file or two. Basic parsers work fine on those inputs. The results look reasonable. The system gets approved for production.

In production, the document corpus looks different. It contains 200-page regulatory policy documents where tables start on page 22 and continue on page 23. It contains scanned audit reports where the actual content lives in raster images. It contains financial disclosures where running headers and footers break into the body text every page. It contains multi-column layouts where two entirely separate sections sit side by side on the same page.

A basic parser cannot handle any of these cases correctly. It does not fail loudly. It silently discards structural context, flattens layout into a text stream, and passes that degraded output downstream. Every component that follows — the chunker, the embedding model, the vector index — operates on corrupted input. The hallucinations and retrieval failures that appear at query time are traced back to the LLM or the retrieval logic. The actual root cause sits two stages upstream, in the parser, and is never touched.

This article walks through the precise mechanics of why this happens, and documents a production-grade ingestion pipeline built with Docling, LangChain, and a structured metadata extraction layer designed to address these failures at their actual source.

The Mechanics of the Document Parsing Failure Chain

Understanding why standard parsers fail requires understanding what a PDF actually is at the data level.

A PDF file is not a document. It is a page description language. Its internal representation does not encode semantic structure — headings, paragraphs, tables, or sections. It encodes rendering instructions: draw this glyph at this coordinate using this font at this size. There is no native concept of "this text belongs to this table cell" or "this heading is the parent of this section." That structure is a human visual inference from the rendered page layout.

When a library like PyPDF or PDFMiner processes a PDF, it extracts the character sequences and their approximate positions on the page. Without a layout model, it has no mechanism to determine reading order, column boundaries, or the relationship between spatially separated elements. Its default behaviour is to read the page sequentially — typically left to right, top to bottom — treating the full page width as a single text stream.

The failure modes this produces in enterprise documents are specific and compounding:

Multi-Column Layout Corruption

A two-column regulatory document has column A on the left and column B on the right. A sequential reader scans across the page at each y-coordinate, producing output that reads: left-column sentence fragment, right-column sentence fragment, left-column continuation, right-column continuation. The output is a mixed stream of two entirely unrelated content sections. Both sections are now unparseable as coherent text, and neither can be chunked into a semantically meaningful unit.

Cross-Page Table Severing

Enterprise documents routinely contain tables that span page boundaries. The table header row is on page 11; the data rows continue on page 12. A page-scoped parser processes each page independently. The table on page 11 is extracted as a fragment with no data rows. The continuation on page 12 is extracted as rows with no header context. Neither fragment contains enough information to be retrievable as a meaningful unit. When these fragments are chunked and embedded, they produce vectors that cannot answer any query correctly about that table's content.

Header and Footer Pollution

Page numbers, document titles, section labels, and legal footers appear on every page of an enterprise document. A parser with no layout model injects these strings into the body text at the point where they appear spatially. A chunk that should contain a clean paragraph of policy text instead contains: policy sentence fragment, "Page 47 of 312 — CONFIDENTIAL", remainder of policy sentence. The embedding of that chunk is polluted by the header token sequence, shifting the vector away from its true semantic position.

Silent Image Discard

Charts, diagrams, process flow diagrams, and scanned pages exist in enterprise documents as embedded raster images. A text-extraction parser has no path to retrieve content from these images. It silently skips them. In regulated banking environments — financial disclosures, audit frameworks, risk matrices — a significant fraction of the information that actually matters lives in these images. It never enters the RAG index.

The Embedding Corruption Effect

The cumulative effect of these failures arrives at the embedding model. The embedding model has no knowledge of what the original document looked like. It encodes the text sequence it receives. A corrupted chunk — mixed columns, severed table rows, injected headers — produces a vector in a region of the embedding space that does not accurately represent the semantic content of the source material. That vector is then stored in the index and retrieved in response to queries. The retrieved context is wrong. The LLM generates a response based on wrong context. This is the hallucination. Its origin is the parser.

The failure chain is deterministic:

Bad parsing
  → Corrupted text sequences
    → Structurally invalid chunks
      → Semantically inaccurate vectors
        → Irrelevant retrieval
          → LLM hallucination

No amount of prompt engineering, model selection, or retrieval tuning corrects a failure that originates at stage one.

Production Architecture Pattern and Code Walkthrough

System Architecture Overview

The production ingestion pipeline is composed of four distinct functional layers:

Discovery Layer — recursive filesystem traversal with format filtering
Parsing Layer — layout-aware document conversion using Docling's DocumentConverter with a configured PdfPipelineOptions pipeline
Chunking Layer — structure-aware tokenization and context enrichment using Docling's HybridChunker
Indexing Layer — LangChain Document construction with structured metadata, ready for embedding and vector store ingestion

The pipeline is designed to be format-agnostic at the discovery layer and format-specific at the parsing layer. PDF files pass through the full PdfPipelineOptions pipeline. DOCX and Markdown files pass through Docling's native handlers. This avoids the common mistake of running PDF-specific configuration against non-PDF inputs.

Core Implementation: `chunks.py`

The parsing configuration is the most critical engineering decision in the entire pipeline. Every downstream component inherits the quality of this stage.

from pathlib import Path
from docling.document_converter import DocumentConverter, PdfFormatOption
from docling.datamodel.base_models import InputFormat
from docling.datamodel.pipeline_options import (
    PdfPipelineOptions,
    TableFormerMode,
    HeadingHierarchyOptions
)
from docling.chunking import HybridChunker


def load_docs(dir_path):
    """Recursively discover supported documents. Returns Path objects."""
    ROOT = Path(dir_path)
    EXTS = {".pdf", ".docx", ".md"}
    return [
        p for p in ROOT.rglob("*")
        if p.is_file() and p.suffix.lower() in EXTS and not p.name.startswith(".")
    ]


def parse_file(file_path):
    """
    Parse a single document using layout-aware Docling pipeline.
    Returns a (chunker, document) tuple for downstream chunking.
    """
    file_path = Path(file_path)

    # Pipeline configuration: heading hierarchy and parsed pages enabled;
    # table structure and OCR disabled for native-text PDFs.
    opts = PdfPipelineOptions(do_table_structure=False)
    opts.heading_hierarchy_options = HeadingHierarchyOptions(enabled=True)
    opts.generate_parsed_pages = True
    opts.do_ocr = False

    converter = DocumentConverter(
        format_options={InputFormat.PDF: PdfFormatOption(pipeline_options=opts)}
    )

    doc = converter.convert(file_path).document
    chunker = HybridChunker()
    return chunker, doc


def meta_data(metadata):
    """
    Extract structured metadata from a chunk's ChunkMeta object.
    Page number is derived from provenance data across all doc_items in the chunk.
    """
    page = (
        sorted({p.page_no for it in metadata.doc_items for p in it.prov})[0]
        if metadata.doc_items else 0
    )
    return {
        "page":     page,
        "filetype": metadata.origin.mimetype,
        "filename": metadata.origin.filename,
        "heading":  " > ".join(metadata.headings) if metadata.headings else ""
    }

Core Implementation: `index.py`

import os
from dotenv import load_dotenv
from llm import embed_documents
from chunks import load_docs, meta_data, parse_file
from langchain_core.documents import Document

load_dotenv()


def process_documents(dir_path):
    """
    End-to-end ingestion: discover → parse → chunk → contextualize → index.
    """
    docs = load_docs(dir_path)

    for doc_path in docs:
        chunker, parsed_doc = parse_file(doc_path)

        for i, chunk in enumerate(chunker.chunk(parsed_doc)):
            metadata = meta_data(chunk.meta)

            # contextualize() prepends the heading breadcrumb path to chunk text.
            # This is the text that goes to the embedding model — not chunk.text.
            text = chunker.contextualize(chunk)

            document = Document(
                page_content=text,
                metadata={**metadata, "chunk_index": i}
            )

            # embed_documents(document) — downstream call to vector store


if __name__ == "__main__":
    process_documents(os.environ.get("docs_path"))

Technical Walkthrough: Step by Step

Step 1 — Document Discovery (load_docs)

The discovery function uses Path.rglob("*") to traverse the directory tree recursively. It filters on a defined extension set {".pdf", ".docx", ".md"} and excludes hidden files by checking p.name.startswith("."). This prevents macOS resource fork files (.DS_Store, ._filename) from entering the pipeline and causing parse errors. The function returns Path objects rather than string paths, which allows the downstream parse_file function to leverage pathlib's existence check and type-safe path composition.

Step 2 — Layout-Aware Parsing (parse_file)

This is where the pipeline diverges from a naive implementation. Four specific pipeline options are set:

do_table_structure=False — This disables the TableFormer model, which is Docling's ML-based table cell recognition model. For native-text PDFs (PDFs with an embedded text layer), the structural relationships between cells can be inferred from the text positions without running the full vision model. Disabling it significantly reduces parse latency per document. For scanned documents or PDFs where tables are rendered as images, this flag must be set to True and the corresponding do_ocr=True flag must be enabled.

HeadingHierarchyOptions(enabled=True) — By default, Docling's layout model classifies page regions as SECTION_HEADER without assigning a level. Every heading in the exported document is treated as level 1, producing a flat structure with no parent-child relationship between sections. Enabling HeadingHierarchyOptions activates a post-processing stage that assigns heading levels using three signals in precedence order: PDF bookmarks from the document outline, leading numbering patterns in the heading text (e.g., 1., 1.1, 2.3.4), and visual font styling. This recovered hierarchy is what makes the HybridChunker's structural chunking meaningful.

generate_parsed_pages=True — This option keeps the raw parsed PDF cell data in memory after the layout analysis stage. It is specifically required for the font-style heading inference signal. Without it, the HeadingHierarchyOptions stage can still use bookmarks and numbering patterns, but font weight, slant, and size comparisons are skipped silently. In enterprise documents that have no bookmarks and no numbered headings — dense policy documents, legal agreements, free-form reports — font styling is often the only reliable heading signal available. This flag must be enabled to use it.

do_ocr=False — OCR is disabled for native-text PDFs. Running Tesseract or another OCR engine on a PDF that already has an embedded text layer introduces character recognition errors and significantly increases processing time. OCR should only be enabled for scanned documents or for pages that contain image-only regions.

Step 3 — Hybrid Chunking (HybridChunker)

HybridChunker operates in four sequential stages on the parsed DoclingDocument:

Hierarchical pass — walks the document's element tree and produces initial chunks at structural boundaries: paragraphs, list items, tables, and picture captions. Each chunk carries the heading path of the section it belongs to.
Merge pass — combines consecutive elements that share the same heading path and fit within the token limit. This prevents the common failure of one-sentence chunks that contain insufficient context for accurate retrieval.
Split pass — splits single elements that exceed the token limit. Tables use a LineBasedTokenChunker that repeats the header row for each split segment. Text elements use the semchunk algorithm, which splits at the most semantically meaningful boundary available rather than at a fixed character or token count.
Contextualization — the chunker.contextualize(chunk) call prepends the full heading breadcrumb path to the chunk text before it is passed to the embedding model. A chunk from a section 3 > 3.2 > Access Control Requirements produces an embedding that encodes both the section context and the content — significantly improving retrieval precision for queries that reference specific sections without quoting them verbatim.

Step 4 — Metadata Extraction (meta_data)

The meta_data function derives the page number from the provenance data attached to each doc_item in the chunk's metadata. A chunk that spans multiple pages will have prov entries from each page; the function takes the minimum page number as the canonical page reference. This is more accurate than line-based page attribution because the HybridChunker can merge elements from adjacent pages into a single chunk.

The heading field concatenates the heading path as a " > "-delimited breadcrumb string. This is stored as vector store metadata and enables filtered retrieval: a query scoped to a specific section of a document can filter on the heading field rather than relying solely on vector similarity.

Step 5 — LangChain Document Construction

Each processed chunk is wrapped in a LangChain Document object with page_content set to the contextualized text (not chunk.text) and metadata containing the structured fields plus the sequential chunk_index. The chunk_index enables re-ranking by document position as a secondary signal when multiple chunks from the same document are retrieved with similar cosine scores.

Implementation Strategies and Deep-Dive Considerations

Native PDF vs. Scanned Document Routing

The current implementation sets do_ocr=False globally. In a production corpus that contains a mix of native-text PDFs and scanned documents, this requires a pre-processing routing stage before the converter is invoked.

A practical approach is to probe the document's text layer density before routing:

import fitz  # PyMuPDF

def is_native_text_pdf(file_path: Path, sample_pages: int = 3) -> bool:
    """
    Sample the first N pages to determine if the PDF has a native text layer.
    Returns True for native-text PDFs, False for scanned/image-only PDFs.
    """
    doc = fitz.open(str(file_path))
    pages_to_check = min(sample_pages, len(doc))
    total_chars = sum(
        len(doc[i].get_text("text").strip())
        for i in range(pages_to_check)
    )
    doc.close()
    return total_chars > 100  # threshold: fewer than 100 chars implies image-only pages

Documents that fail the native-text check should be routed to a configuration with do_ocr=True and, depending on table density, do_table_structure=True with TableFormerMode.ACCURATE.

Heading Hierarchy Signal Selection

The HeadingHierarchyOptions signals apply in precedence order. For enterprise document corpora, the appropriate configuration depends on the document type:

Document Type	Recommended Signal Configuration
Regulatory PDF with bookmarked ToC	`use_bookmarks=True` (default), others as fallback
Technical specifications with numbered sections	`use_numbering=True`, `use_bookmarks=False` if ToC is absent or unreliable
Legal agreements, free-form policy documents	`use_style=True`, `use_font_style=True`, requires `generate_parsed_pages=True`
Mixed corpus	All signals enabled (default when `enabled=True`)

For a banking regulatory corpus — MAS TRM guidelines, internal policy documents, audit frameworks — the mixed configuration is appropriate because the document types are heterogeneous.

The `contextualize()` vs `chunk.text` Decision

This is a consistently misunderstood detail in Docling-based RAG implementations. chunk.text contains only the raw text of the chunk's content elements. chunker.contextualize(chunk) prepends the heading breadcrumb path to that text.

For a chunk with headings ["3. Security Controls", "3.2 Access Management"] and body text "All privileged access must be logged...", the outputs are:

chunk.text:
"All privileged access must be logged..."

chunker.contextualize(chunk):
"3. Security Controls
3.2 Access Management
All privileged access must be logged..."

The embedding model encodes both versions. The contextualized version places the vector in a region of the embedding space that reflects the section semantics, not just the sentence semantics. A query for "what are the access management logging requirements" retrieves the contextualized chunk with significantly higher cosine similarity than the raw text chunk, because the heading tokens anchor the embedding to the correct domain.

Always pass chunker.contextualize(chunk) to the embedding model. Store chunk.text separately if you need the raw text for display purposes.

Metadata as a Retrieval Filter Layer

The metadata schema produced by meta_data() — page, filetype, filename, heading, chunk_index — serves a dual purpose. It is stored alongside each vector in the vector store and enables metadata-filtered retrieval.

In regulated environments, this has a direct compliance application. A query scoped to a specific document version (filename filter) or a specific section (heading filter) can be constrained at the retrieval layer rather than relying on the LLM to perform that scoping from the retrieved context. This reduces the surface area for cross-document content bleed, where a retrieved chunk from document version N-1 is used to answer a query that should be answered from version N.

Architectural Tradeoffs and Operational Takeaways

Compute Overhead: `generate_parsed_pages=True`

Keeping the parsed PDF cells in memory (generate_parsed_pages=True) increases per-document memory consumption. For large documents — 200+ pages — this can add 300–500 MB of in-process memory overhead per concurrent parse operation. In a batch ingestion workload running on a fixed instance, the concurrency ceiling must be set to account for this.

The tradeoff is worth it for enterprise corpora where free-form documents without bookmarks are common. For corpora where all documents have reliable PDF outlines or numbered sections, generate_parsed_pages can be set to False to reduce memory pressure, with use_style=False in the HeadingHierarchyOptions.

`do_table_structure=False` Tradeoff

Disabling the TableFormer model eliminates the ML inference cost for table structure recognition, which is the most expensive single operation in the Docling pipeline for table-dense documents. For native-text PDFs, this is a reasonable tradeoff because cell relationships can be preserved through text-position inference.

The failure mode: if native-text PDFs contain tables that use borderless layouts or unconventional spacing, the text-position inference may still produce incorrect cell associations. A validation step that samples the parsed table output before indexing is strongly recommended for corpora with complex table structures.

Batch vs. Streaming Ingestion

The current implementation processes documents sequentially in a single process. For initial corpus ingestion at enterprise scale — thousands of documents — this does not meet practical time constraints. A production deployment should parallelise at the document level using a worker pool:

from concurrent.futures import ProcessPoolExecutor
import os

def process_documents_parallel(dir_path: str, max_workers: int = 4):
    docs = load_docs(dir_path)
    with ProcessPoolExecutor(max_workers=max_workers) as executor:
        futures = {executor.submit(parse_and_index, doc_path): doc_path for doc_path in docs}
        for future in futures:
            try:
                future.result()
            except Exception as e:
                print(f"Failed: {futures[future]} — {e}")

Worker count should be tuned based on available CPU cores and the memory overhead of generate_parsed_pages=True. On an m6i.4xlarge (16 vCPU, 64 GiB), a practical ceiling is 4–6 workers for large-document corpora.

Why This Architecture Scales Where Others Don't

The combination of layout-aware parsing, structure-driven chunking, and heading-enriched contextualization produces an index where retrieved chunks carry their structural position in the source document as part of the vector encoding. This has a measurable effect on retrieval precision for the query types that dominate enterprise RAG use cases: policy lookups, section-specific compliance checks, and cross-document comparisons.

The cost of this precision is paid once, at ingestion time. The alternative — fast parsing, naive chunking, no contextualization — produces an index that is cheap to build and unreliable to query. In a regulated environment, an unreliable RAG system is not a minor inconvenience. It is an audit risk.

Fix the ingestion layer. Everything downstream inherits its quality.

Every LLM Request Has Two Halves. Only One Uses Your GPU Cores

Ankur Jindal — Mon, 31 Aug 2026 06:50:21 +0000

Paste a long document into ChatGPT and hit enter.

Nothing happens for a second or two. Then the answer starts appearing, word by word, at a steady pace until it finishes.

You have seen this hundreds of times. Most people never think about it.

But those are two completely different things happening inside one request, running on two different parts of the same GPU, limited by two different bottlenecks. Once you know what they are, a lot of confusing things about serving AI models stop being confusing.

Including why a faster GPU sometimes makes no difference at all.

This is part two of a series. Part one covered the split between VRAM and cores. This post covers what each of them does when a request arrives.

The example we will follow

I am going to trace one question the whole way through:

what is the capital of france

Simple, short, and it produces a short answer. Perfect for watching the machinery.

Before anything happens, the model server has already loaded the model's weights into VRAM. That happened at startup and they stay there. Nothing about your request loads a model.

Your words get chopped into tokens, which are just pieces of words turned into numbers, and placed in VRAM alongside the weights.

Now the work begins.

Phase one: prefill, or "the pause"

Before the model can write anything, it has to read everything you gave it.

Here is the important part, and it is the opposite of what most people assume: it reads your entire prompt at once. Not word by word. All six words go to the cores together, in a single pass.

That is possible because your whole prompt is already there. Nothing has to wait for anything else. So thousands of cores fire simultaneously and chew through all of it in one burst.

flowchart LR
    A["what is the capital of france<br/>all 6 tokens together"] --> C
    W["16 GB of model weights"] --> C
    C["GPU CORES<br/>every core busy"] --> O1["First word: 'The'"]
    C --> O2["Notes saved to VRAM<br/>the KV cache"]

Two things come out of that single pass:

The first word of your answer. Just one.
A set of working notes, saved into VRAM. These notes summarise everything the model worked out about your prompt. The technical name is the KV cache.

The moment that first word appears is the moment the pause ends.

How long is the pause? It depends entirely on how much you gave it to read. Six words is nothing, so you would not notice. Paste a hundred-page contract and that is tens of thousands of tokens pushed through in one burst, and now the pause is real.

This phase has a name: prefill. And the pause has a name too: TTFT, time to first token. It is one of the two numbers every serving benchmark reports.

What is the bottleneck here? The cores. There is a mountain of maths to do and they are all doing it. This phase is compute-heavy.

Phase two: decode, or "the typing"

Now the model writes the rest of the answer. One word per pass.

To produce each word, the cores need three things: the notes from before, the word that was just produced, and the model's weights. All of it read out of VRAM.

flowchart LR
    N["Notes in VRAM<br/>the KV cache"] --> C
    L["The word just produced"] --> C
    W["16 GB of weights<br/>read again, in full"] --> C
    C["GPU CORES<br/>barely used"] --> O["One new word"]
    O -.->|"repeat"| N

That loop runs once per word until the answer is done:

Lap 1 → "The"
Lap 2 → "capital"
Lap 3 → "is"
Lap 4 → "Paris"

This is why the answer types itself out instead of appearing all at once. You are watching each lap finish. Streaming is not a feature someone built. It is just what happens when each word is sent the moment it exists.

This phase is called decode, and the gap between words is TPOT, time per output token. That is the second benchmark number.

What is the bottleneck here? Not the cores. Look at what has to happen for a single word: the entire model, all 16 GB of it, has to travel out of VRAM and past the cores. The cores glance at it, do a small amount of maths, and produce one word.

Then it all happens again for the next word.

The cores are mostly idle. The memory pipe is flat out. Decode is limited by how fast the GPU can read its own memory.

Why word three does not need a fresh pause

Here is the question I got stuck on, and it is worth its own section.

To predict word 3, the model needs to know everything before it. Your prompt, plus word 1, plus word 2. So does it read all of that again?

If it did, every single word would have its own pause. The whole thing would be unusable.

It does not, because of the notes.

Every word that gets produced is added to the notes as it goes. So by the time lap 3 starts, the notes already contain your prompt, "The", and "capital". The model reads those notes and only has to process the one word that just came out.

Lap 3 reads:  notes (prompt + "The" + "capital")  +  the word "capital"
Lap 3 does:   process one word
Lap 3 gives:  "is"

Nothing gets forgotten and nothing gets reprocessed. The model always sees the full history. It just never redoes the work.

You pay for the pause once, not once per word.

Now the part that changes how you buy hardware

Here is where this gets practical.

Every word requires one full read of the model out of VRAM. So the maximum speed of a model on a given card is a division:

How fast can the card read its own memory?
─────────────────────────────────────────  =  words per second
        How much is there to read?

Let me do it with real numbers.

Take an AWS g6e.xlarge. The GPU inside is a single NVIDIA L40S with 48 GB of VRAM. NVIDIA's datasheet gives its memory bandwidth as 864 GB/s.

Now put a model on it. Qwen3.8-27B is a 28-billion-parameter model. In its published form each weight takes 2 bytes, so:

28 billion × 2 bytes = 56 GB

That does not fit in 48 GB. So we use the Quantization version FP8 , where each weight takes 1 byte:

28 billion × 1 byte = 28 GB

Now the division:

864 GB/s ÷ 28 GB ≈ 31 words per second

That is the ceiling. Two numbers off two spec sheets, and you know roughly how fast that model will generate on that card. No deployment required to calculate this.

And notice what shrinking the model did. At 56 GB the ceiling would have been about 15 words per second. Halving the weights doubled the speed, because there is half as much to read every single lap.

A caveat worth stating plainly: that number is a theoretical ceiling for one user, and real throughput lands meaningfully below it. The notes have to be read too, and no software achieves perfect memory utilisation. Treat it as a sanity check, not a promise.

The AWS gap nobody mentions

Here is something that caught me out, and I suspect it catches out a lot of people.

That bandwidth figure, the number that decides your generation speed, is not on AWS GPU Instance page.

Look at an EC2 GPU instance table and you get the card name, GPU memory, vCPUs, system memory, and storage. Some tables have a column called "EBS Bandwidth" and another called "Network Bandwidth", which look like the answer and are not. Those are disk and network throughput. Neither has anything to do with VRAM.

The three bandwidths worth separating:

Bandwidth	Between what	Typical scale
Memory bandwidth	VRAM and cores	hundreds of GB/s
PCIe	GPU card and server	tens of GB/s
Network / EBS	Server and the outside world	Gbps

Only the first one governs how fast words come out, and AWS does not publish it.

So the workflow is: read the card name off the AWS table, then go to NVIDIA's datasheet for that card and find the field labelled Memory Bandwidth.

Everything above describes one user.

You might reasonably be looking at "31 words per second" and thinking that sounds poor for a server. And it would be, if that was all it could do.

But look again at what happens during decode. The GPU reads the entire model out of VRAM to produce one word for one person, and the cores sit mostly idle while it happens.

So why not use that same read to produce a word for fifty people at once?

That is exactly what real serving software does, and it is why the economics of hosting a model work at all. It also introduces a new limit, a different kind of bottleneck, and the reason ordinary load balancing falls apart completely for AI workloads.

That will cover in upcoming posts.

A GPU Is Two Things, and Only One of Them Holds Your Model

Ankur Jindal — Thu, 27 Aug 2026 08:39:52 +0000

I have run AWS infrastructure for years. When I started looking at how large language models actually get served, I hit a wall on something basic.

Someone said "the model sits on the GPU." Someone else said "the GPU does the calculation." Both sounded right. Neither told me where the model actually lives.

The problem is that "GPU" is not one thing.

Two parts, one board

You already know how a normal machine is laid out. A CPU does the work. RAM holds the data. They sit apart, connected by a bus.

A GPU card is the same pair, shrunk onto one board:

Cores — the chip in the middle. Thousands of small processors that do maths.
VRAM — memory chips around it. Holds the data.

So when people say different things, they are talking about different halves:

What they say	What they mean
"The GPU has 80GB"	The VRAM
"The GPU is fast"	The cores
"The model is on the GPU"	The weights are in VRAM

Neither half is "the GPU" on its own.

The cores hold nothing

This is the part that took me longest to accept.

The cores have almost no storage. They are hands, not shelves. The model does not get loaded into them once and stay there. It has to be streamed across from VRAM, continuously, for every calculation.

An 8 billion parameter model at full precision is about 16GB of numbers. Those numbers live in VRAM. To produce a single word of an answer, the GPU streams the whole thing across to the cores, does the maths, and gets one word back. Then it does it again for the next word.

That one fact explains why answers type themselves out instead of appearing all at once.

Where this bites you on AWS

Here is the practical part.

Look at a GPU instance in the EC2 pricing console:

g6.4xlarge    $1.3232/hr    16 vCPU    64 GiB    1 x 600 GB NVMe

That 64 GiB looks like plenty of room for a model. It is not GPU memory. It is the server's ordinary system RAM, and inference barely touches it.

The GPU on that instance is a single NVIDIA L4 with 24GB of VRAM. That 24GB is the only number that decides whether your model loads.

The EC2 pricing page has no GPU column at all. You have to cross-reference the instance-types page to find the card and its memory.

Worth memorising for the common G-family instances:

Family	GPU	VRAM
g4dn	T4	16GB
g5	A10G	24GB
g6	L4	24GB
g6e	L40S	48GB

One sentence to take away

Weights live in VRAM. Cores do the maths. Neither one is "the GPU" by itself.

Everything else in model serving follows from that split.

DEV Community: Ankur Jindal

Hybrid Search and Reranking: What Each One Actually Buys You

Where Part 1 Left Off

The Query That Breaks Pure Semantic Search

Why You Cannot Just Run Both and Add the Scores

The two scores are not comparable

Normalising first makes it worse

Rank fusion sidesteps the problem instead of solving it

What the constant 60 does

How We Built It

Why one statement

The query

The decisions inside it

What Hybrid Search Bought Us

Why We Added a Reranker

The fused score is not a relevance score

What a reranker does differently

What it bought us

What You Give Up Without Each One

Without hybrid search

Without a reranker

The combined picture

What's Next

References

AWS Lambda Managed Instances: What Min=0/Max=0 Actually Does in Production

Introduction: The Production Bottleneck

The Mechanics of the Failure Chain

Production Architecture Pattern & Walkthrough

System Architecture Overview

Setup Walkthrough: Step by Step

Implementation Strategies & Deep-Dive Considerations

Architectural Tradeoffs & Operational Takeaways

Why Your Enterprise RAG Pipeline Is Failing Before the First Query Runs

Introduction: The Ingestion Layer Is the Real Bottleneck

The Mechanics of the Document Parsing Failure Chain

Multi-Column Layout Corruption

Cross-Page Table Severing

Header and Footer Pollution

Silent Image Discard

The Embedding Corruption Effect

Production Architecture Pattern and Code Walkthrough

System Architecture Overview

Core Implementation: chunks.py

Core Implementation: index.py

Technical Walkthrough: Step by Step

Implementation Strategies and Deep-Dive Considerations

Native PDF vs. Scanned Document Routing

Heading Hierarchy Signal Selection

The contextualize() vs chunk.text Decision

Metadata as a Retrieval Filter Layer

Architectural Tradeoffs and Operational Takeaways

Compute Overhead: generate_parsed_pages=True

do_table_structure=False Tradeoff

Batch vs. Streaming Ingestion

Why This Architecture Scales Where Others Don't

Every LLM Request Has Two Halves. Only One Uses Your GPU Cores

The example we will follow

Phase one: prefill, or "the pause"

Phase two: decode, or "the typing"

Why word three does not need a fresh pause

Now the part that changes how you buy hardware

The AWS gap nobody mentions

Everything above describes one user.

A GPU Is Two Things, and Only One of Them Holds Your Model

Two parts, one board

The cores hold nothing

Where this bites you on AWS

One sentence to take away

Core Implementation: `chunks.py`

Core Implementation: `index.py`

The `contextualize()` vs `chunk.text` Decision

Compute Overhead: `generate_parsed_pages=True`

`do_table_structure=False` Tradeoff