Every Option on Layer 0 of the Agentic RAG Stack Is Paid. Here Is the Whole Free Route, Measured

The reference "Complete Agentic RAG Tech Stack" is nine layers and about fifty-three tools. I went through it tool by tool to find out how much of it you can actually build without a credit card, and the answer is: most of it. Roughly thirty-one entries are genuinely open source and another six are open weights, so about thirty-seven of fifty-three are buildable free. Agent frameworks, vector databases, memory — those layers are almost entirely open.

Then there is layer 0.

Inference and deployment is the one layer where every single option listed is a proprietary, billed endpoint: Bedrock AgentCore, Microsoft Foundry, Gemini Enterprise, Together, Groq, Modal. There is no open row. Which means the free route at layer 0 is not "pick the open-source one" — it is self-host open weights on hardware you already have, and take on the arithmetic that the managed endpoints were doing for you.

That arithmetic is the whole of this post. I built it as Arc Rector, a stack-in-a-box where all nine layers are swappable adapters behind one interface and every default runs with zero vendor bills and no API keys.

👉 Live, everything computed in your browser: https://dev48.infy.uk/arcrector/level0-inference.html

The layer is two methods

Before any of the sizing maths, it is worth seeing how small the contract is, because that is what makes "self-host or rent" a config line rather than a rewrite.

class Inference(ABC):
    """L0 -- inference & deployment. Turns a prompt into text."""
    name: str = "inference"

    @abstractmethod
    def complete(self, prompt: str, system: str = "", **kwargs) -> str: ...

    def available(self) -> bool:
        """Cheap liveness probe. Adapters override with a real health check."""
        return True

Five implementations sit behind that — Ollama, vLLM, llama.cpp, NVIDIA NIM and an echo control — and everything above layer 0 calls exactly those two methods. Streaming is deliberately not in the interface: adding it would mean changing the signature across all five adapters at once, and I would rather have the constraint visible than have a half-supported feature.

Weights are bytes, and bytes are the first wall

The only sizing formula that never lies to you:

weights_bytes = params * bits_per_weight / 8

Eight billion parameters at fp16 is 8.03e9 × 16 / 8 = 16.1 GB, before a single token of context exists. Seventy billion at fp16 is 141 GB — two H100s, and again, before any context. At 4.5 bits the 70B is 39.7 GB and fits on one 48 GB card with room to spare.

This is why quantisation is a precondition at layer 0, not an optimisation. It is also the only reliable way to predict a failure that otherwise arrives as a process kill with no message.

Two traps ride along with it. The parameter count on a model card is the whole model including embeddings and the output head, and quantisers routinely leave those at higher precision — so a q4_K_M file is never exactly 4 × params / 8. And the weights number in isolation is a lie of omission, for a reason we get to in two sections.

4-bit quantisation is not 4 bits per weight

Rounding a whole tensor with one shared scale is useless: a single outlier sets the range and every ordinary weight collapses onto the same step. So real formats are group-wise — a block of 32 to 256 weights gets its own scale, and in asymmetric schemes its own minimum. Those are exactly GGUF's d and dmin fields.

def quantise_group(vals, bits=4):
    mn, mx = min(vals), max(vals)
    levels = 2 ** bits - 1              # 15 at 4 bits
    scale  = (mx - mn) / levels or 0.0
    codes  = [clamp(round((v - mn) / scale), 0, levels) for v in vals]
    return scale, mn, codes             # GGUF calls these d and dmin

def dequantise(codes, scale, mn):
    return [c * scale + mn for c in codes]

Now do the bookkeeping most write-ups skip. Every group carries an fp16 scale and an fp16 min of its own, so:

effective bits = bits + (16 + 16) / group_size

group size	effective bits per weight
32	5.000
64	4.500
128	4.250
256	4.125

At group 32, "4-bit" is exactly 5.0 bits per weight. q8_0 is exactly 8.5 for the same reason. Drag the group size up and you trade accuracy for that overhead in visible numbers — the page runs a real group-wise quantiser over 4096 seeded weights with real scales, real round-and-clamp and real reconstruction error, so you can watch the trade rather than read about it.

Real q4_K_M goes further: it quantises the scales themselves to 6 bits in super-blocks, keeps selected tensors at higher precision, and fuses dequantisation into the matmul — which is why its measured bits-per-weight is a fraction rather than a round number. And AWQ and GPTQ pick scales to minimise error on activations rather than on the weights in isolation, which buys back most of the quality at 4 bits. There is no calibration data in my browser panel; the format is real, the calibration is not.

The KV cache is the number that actually decides

Attention needs the key and value vectors of every previous token to produce the next one. Recomputing them per step would be quadratic, so every serving engine caches them — and that cache, not the weights, is what decides whether a long context is affordable.

def kv_cache_bytes(layers, kv_heads, head_dim, seq_len, batch, elem_bytes):
    return 2 * layers * kv_heads * head_dim * seq_len * batch * elem_bytes

The 2 is K and V. For Llama 3.1 8B at fp16 — 32 layers, 8 KV heads, head dim 128 — that is 2 × 32 × 8 × 128 × 2 = 128 KiB per token. So:

• 4096 tokens, one sequence → kv_cache_bytes(32, 8, 128, 4096, 1, 2) = 536,870,912 bytes = 512 MiB exactly
• the same context across 32 concurrent sequences → 16 GiB

The quantised weights were 4.5 GB. The cache is the bigger number.

Three consequences follow, and all three are things people discover the expensive way:

1. num_ctx is a reservation, not a measurement. Engines reserve for the declared maximum, so ARC_L0_INFERENCE__NUM_CTX=8192 costs 1 GiB per sequence on the 8B whether your prompts are 8,000 tokens or 200.
2. Batch size multiplies the cache linearly. Concurrency is bought in memory.
3. Quantising the KV cache to fp8 halves it, and is often a better trade than quantising the weights further. The cache is where the memory went.

Use kv_heads, never n_heads — it is a factor of four

The single most common back-of-envelope error at this layer. In classic multi-head attention every query head owns its own key and value heads. Grouped-query attention keeps the query heads and shares each K/V head across a group of them:

# config.json, llama3.1:8b
"num_hidden_layers":     32,
"num_attention_heads":   32,   # query heads
"num_key_value_heads":    8,   # <- the one the formula wants

Llama 3.1 8B is 32 query heads → 8 KV heads, so the cache is exactly 4× smaller than MHA would make it. The 70B is 64 query heads → still 8 KV heads: 8× smaller, and that reduction is the only reason it can be served at long context at all.

Read num_key_value_heads off the model config before you size anything. It is not the number on the marketing page, and getting it wrong makes you conclude a model needs four times the memory it does.

Prefill and decode are two different machines

Prefill processes the whole prompt in parallel — one big matmul per layer over every prompt token at once — so it is compute-bound and scales with FLOPs. Decode emits one token at a time, and each step streams the entire set of weights out of memory to produce that single token, so it is memory-bandwidth-bound and barely notices a faster GPU.

# decode step, batch 1 : stream 4.5 GB of weights -> 1 token
# decode step, batch 8 : stream 4.5 GB of weights -> 8 tokens
#                        same bytes moved, 8x the output
#
# prefill, 1000 tokens : ~1000x the arithmetic of one decode step
#                        adding a second prompt genuinely doubles the work

That split explains nearly everything about serving:

• TTFT and tokens/sec are different bugs. A slow first token means a long prompt or a queued prefill. Slow subsequent tokens mean you are bandwidth-limited, and the fix is batching or a smaller model.
• Quantisation speeds up decode far more than prefill — fewer bytes to stream per token.
• Batching is nearly free during decode, because the weights are being streamed anyway.
• CPU-only is brutal for exactly this reason: system RAM bandwidth is an order of magnitude below a GPU's, and decode is the phase that lives on it.

Continuous batching is a queue manager, not a bigger batch

Static batching collects N requests, runs them together, and waits for all N before starting the next batch. Because real output lengths differ by an order of magnitude, most slots spend most of the batch finished and idle. Continuous batching (Orca's idea, vLLM's headline feature) schedules at the granularity of one decode step: a finished sequence is evicted immediately and a waiting one admitted into its slot, mid-batch.

# one engine step, decode-priority with chunked prefill
for r in active:                       # decode: 1 token per live sequence
    if r.prefilled >= r.prompt and budget >= 1:
        budget -= 1; r.generated += 1
        if r.generated >= r.output: r.done = step

for r in active:                       # prefill: whatever budget is left
    if r.prefilled < r.prompt and budget > 0:
        take = min(r.prompt - r.prefilled, budget)
        r.prefilled += take; budget -= take

The page runs that as a genuine discrete-event simulator — real admission, real eviction, a shared per-step token budget, and per-request completion steps that are counted rather than assigned. Watch the two effects fight: continuous batching drains the queue in fewer steps because slots never idle, and lengthens every step because more sequences are live. Push the slot count up and at least one individual request finishes later than it would have under static batching, while total throughput rises.

That trade is the entire point, and it is why the honest question is never "which engine is faster":

TTFT        time to first token     -> prefill + queueing
TPOT        time per output token   -> bandwidth + batch size
throughput  tokens/sec across all concurrent requests
p95         the tail, which is what users complain about

# a change that improves throughput 3x and p95 TTFT 2x is a
# REGRESSION for a chat product and a WIN for an eval harness.

Decide which one you are before you benchmark.

And this is why Arc Rector's default is Ollama and not vLLM. vLLM is the better engine and the production answer — but its advantage only exists when there is concurrency to exploit, and a single-user localhost stack has none. Choosing vLLM there would be picking the winner of a race nobody is running, and paying for it with a CUDA requirement.

available() — a probe that can actually return False

The base implementation returns True, and a probe that always says yes is worse than no probe: it converts a clean startup failure into a confusing mid-run one. So each adapter overrides it with the cheapest check that can genuinely fail.

def available(self) -> bool:
    """True only when the server is up AND this model is actually pulled."""
    try:
        r = self._http.get(f"{self.base_url}/api/tags", timeout=5)
        if r.status_code != 200:
            return False
        return any(m.get("name") == self.model
                   for m in r.json().get("models", []))
    except Exception:
        return False

Checking that the server answers is not enough. A running Ollama daemon is perfectly happy while the model you named was never pulled, and that failure surfaces mid-run as a 404 that looks like a broken pipeline. So the probe lists the installed models and requires yours to be among them.

NIM is the interesting exception: its probe checks only that the API key is present, because a live health check would spend free-tier credits on nothing. That is a deliberate trade of probe strength for not burning your quota, and it is the kind of decision that only shows up when you actually run the thing.

arc-rector doctor calls every level's probe and prints the exact remedy — for layer 0 that is the ollama pull you are missing. The web UI's /api/health runs the same probes on purpose, so the page can never report healthier than the CLI.

The control adapter, which is the best debugging tool in the repo

echo has no model in it at all. It parses the numbered context block out of the prompt and returns the first sentence of each entry with its [n] marker attached.

class EchoInference(Inference):
    name = "echo"
    def complete(self, prompt, system="", **kwargs) -> str:
        entries = self._parse_entries(self._extract_context(prompt))
        if not entries:
            return "I don't know based on the provided documents."
        parts = []
        for marker, body in entries[: self.max_sentences]:
            sentence = self._first_sentence(body)
            if sentence:
                parts.append(f"{sentence} [{marker}]")
        return " ".join(parts) or "I don't know based on the provided documents."

That is enough to exercise retrieval, citation formatting, guardrails, memory and tracing with byte-identical output on every run, which is what the offline test suite needs. It is also a diagnostic worth stealing outright: if your answer looks the same under echo as under a real model, your retrieval is doing the work and the model is adding nothing. Same idea as plotting the random control in a pruning experiment — the thing that turns an impression into a measurement.

Two gotchas that only appear if you run them

The three hosted-or-server adapters — vLLM, llama.cpp and NIM — all speak OpenAI-shaped /v1/chat/completions, so their complete() bodies are near-identical and the difference between them is a base_url, a header and an error message. Ollama's native API is the odd one out (/api/chat, message.content at the top level, sampling knobs nested under options) — and it stays the default anyway, because it is the one that also fetches the weights, manages the model store and picks the GPU offload split.

Living in that shared body:

• vLLM serves exactly the model it was launched with. A mismatched model id is a 404, and GET /v1/models is how you find the real name.
• llama-server ignores the field entirely. It serves the one GGUF you gave it, whatever you put in the request. Which is worse — you can point it at the wrong quantisation for a week and nothing will tell you.

⚖️ Open weights are not open source

This is the part that has to be said plainly, because "free" above meant free of bills, not free of terms.

The Ollama runtime is MIT. The checkpoint you pull through it is not. Those are two different licences and only one of them is OSI-approved.

Llama 3.x ships under the Llama Community Licence: an acceptable-use policy you must pass on to your own users, naming and attribution requirements, and a clause that revokes the grant for any product with more than 700 million monthly active users unless Meta grants it separately. Gemma carries Google's own terms with a use-restriction policy and a downstream-notice obligation. Qwen and DeepSeek publish their own terms, and they differ by model within the same family — some tiers are Apache-2.0 or MIT and some are not. Read the card for the exact checkpoint you are pulling, not the family it belongs to.

And in almost every case the training data and the training code are not released at all, so you cannot reproduce the model, audit what went into it, or fix it at the source. That is the substantive difference from open source. It is not a licensing technicality.

The genuinely open end exists and is worth knowing: Mistral 7B (Apache-2.0), the Apache-2.0 Qwen 2.5 tiers, Nomic Embed v1.5 (Apache-2.0, and the embedding default in this stack), and OLMo, which publishes data and training code as well as weights.

Say open weights when that is what you mean. Before anything commercial ships, answer three questions: does the licence restrict my use case or my user count; must I pass the acceptable-use policy to my own users; is the naming and attribution requirement satisfied.

What actually ran

The whole stack is deployed on an Oracle Always-Free VM — 2 OCPU, 12 GB, CPU-only ARM. A real query there returned a cited answer in about 100 seconds, against 21 real 768-dimensional vectors in Qdrant. No GPU, no API key, no vendor bill.

The documented default is llama3.1:8b; the model that actually ran is llama3.2:3b, because an 8B on that box answers at roughly 2–5 tokens per second, which is minutes for a paragraph. Changing your mind about that costs one environment variable, which is the entire argument for making layer 0 configuration rather than architecture:

ARC_L0_INFERENCE=llamacpp            arc-rector ask "..."   # swap the level
ARC_L0_INFERENCE__MODEL=llama3.2:3b  arc-rector ask "..."   # swap one setting

And the honest matrix, which is the repo's credibility and never gets inflated: Ollama and echo are ✅ — run and observed to work. vLLM, llama.cpp and NIM are 📄 — real adapters, written and shipped, never executed here. vLLM needs a CUDA GPU; NIM needs a key I did not want the default to require. Saying so is cheaper than being found out.

Verification

216 pytest cases, and they run with no Docker, no Ollama, no network and no weights — because offline mode swaps every level to its dependency-free adapter, echo included. The level page above ships four live panels that genuinely compute in your browser: a real group-wise quantiser, a memory calculator using the published layer and head geometry of Llama 3.2 3B / 3.1 8B / 3.1 70B, a real discrete-event batching scheduler, and an exact longest-common-prefix cache calculator. No API, no model download, no server.

What that page is not: there is no GPU and no model on it. The quantiser is the format, not the fused kernel. The memory model covers weights and KV cache only — CUDA context, activation buffers and fragmentation are a flat allowance you can see in the table, not a measurement. The scheduler's cost model is token count, not FLOPs, so prefill and decode are charged almost identically there and are nothing alike on hardware.

And the framing for the whole project, which I would rather state than have inferred: this is a complete, correct, zero-cost starting point — not a production-ready system. PRODUCTION.md in the repo lists exactly what is missing, including auth, multi-tenancy, rate limits, retention and backups.

If you take one thing from this: at layer 0 the free route is real, but it is not free of arithmetic. Size the KV cache, not the weights — and check num_key_value_heads before you believe your own number.

Live page: https://dev48.infy.uk/arcrector/level0-inference.html
Repo: https://github.com/dev48v/arc-rector