Llama Is Not Open Source. The Clause That Decides It Has a Number In It: 700 Million Users

Every other layer of an agentic RAG stack is an interface with adapters behind it. Swap Qdrant for Chroma, swap Langfuse for nothing at all, and the rest of the system does not notice.

The model is the one layer with no adapter of its own. You change a string in a config file and the whole thing is a different system:

l0_inference:
  model: llama3.1:8b     # <- this line is the layer

This is Level 2 of nine in Project Arc Rector — an agentic RAG stack built entirely from free, self-hostable parts, with every level swappable. The page for this layer runs a real byte-pair encoder, a real n-gram language model with real temperature/top-k/top-p sampling, and real budget arithmetic, all in your browser: https://dev48.infy.uk/arcrector/level2-models.html

Repo: https://github.com/dev48v/arc-rector

Start with the licence, because it is the expensive mistake

Open weights and open source are not the same thing, and the gap has a legal department in it.

Open source in the OSI sense means anyone may use the thing for any purpose, in any field, with no additional restriction. MIT and Apache-2.0 qualify. Open weights means only that the trained parameters are downloadable. The licence attached to them may restrict precisely what open source may not.

family	licence	the catch
Llama 3.1 / 3.2	Llama Community Licence — not OSI open source	An acceptable-use policy you inherit, naming and attribution requirements on derivatives, and a separate licence required from Meta if you had over 700 million monthly active users at the release date
Gemma 2 / 3	Gemma Terms of Use — bespoke	A prohibited-use policy that travels with the weights and with anything you fine-tune from them, plus a duty to pass the restrictions downstream
Qwen 2.5 / 3	Apache-2.0 on many sizes	The licence is per model, not per publisher — some sizes in the same family ship under a separate Qwen licence
Mistral 7B / Mixtral	Apache-2.0	None worth the paragraph. This is why you can stop reading and start shipping
DeepSeek	MIT on code for several releases	A separate model licence rides alongside

A model can be free of charge, free of an API key, running entirely on hardware you own — and still be commercially restricted. Running it locally removes the vendor bill. It does not remove the licence.

The Qwen row is the one that actually bites, because it defeats the habit everybody has: you check a licence once, at the family level, and then pull whatever size fits. Check the specific repo you are pulling.

The model does not read your prompt. It reads tokens.

A byte-pair encoder learns its vocabulary by counting which adjacent pairs of bytes co-occur most often, then merging the winner, over and over. The page trains one live so you can watch the merge table form.

Two properties survive at any scale, and both cost you money:

Digits are pre-tokenised one at a time. 1234567.89 costs ten tokens no matter how many merges you train. That is a large part of why models are bad at arithmetic — the number never becomes a single thing they can hold.

Anything outside the merge table falls back to raw bytes. A Bengali word at three bytes per character costs three tokens per character. Same tax on a real API, and it is invisible until the bill arrives.

So a table of part numbers can cost four times what an English paragraph of the same visual size costs. If your retrieval corpus is structured data, your token budget is not the one you estimated from word count.

The context window is a budget, and something is always evicted

num_ctx  = 4096          the whole desk top
  system prompt
+ memories
+ numbered chunks        <- top_k of these were retrieved
+ the question
+ room for the answer    <- 512, and you must reserve it

Raise top_k and more chunks are retrieved. That does not mean more chunks are read. The packer fills what is left after the fixed costs, and the rest are dropped — retrieved, ranked, and never seen by the model.

This is the silent one. With Ollama, num_ctx is the whole window, and an over-long prompt is truncated to fit rather than rejected. Nothing warns you. The model answers from half a document and reports no problem, because from its point of view there was none.

The page shows the eviction directly: kept chunks renumbered 1..n over what actually fitted, evicted ones listed beside them.

Sampling is the same arithmetic at every scale

The page's generator is an n-gram, not a transformer — no attention, no learned embeddings, and it cannot generalise one character beyond its counts. That is stated on the page rather than glossed over.

But everything after the distribution arrives is identical to what a 70B model does: temperature rescaling of the log-probabilities, top-k truncation, top-p nucleus truncation, renormalising what survives, drawing from the cumulative distribution, and perplexity as the exponential of the mean negative log-probability. Same lines of arithmetic. A transformer's only contribution would be a far better distribution to start from — which is exactly why sampling parameters transfer between models and prompts often do not.

Two smoothing mechanisms are visible and both matter. Stupid backoff drops the oldest character when the full context was never seen. Add-k gives every unseen character a sliver of mass — and turning add-k up dissolves coherence, because that leaked mass is the tail temperature then amplifies.

What fits is one multiplication

parameters x bytes-per-weight = weights in memory

That is the whole calculation, and it decides which model you are allowed to run. Decoding one token reads every weight once, so the ceiling on tokens per second is memory bandwidth divided by model size in bytes — an upper bound from arithmetic, never a benchmark.

The measured reality on this project's CPU-only development box: an 8B model runs at roughly two to five tokens a second, and a single generation was timed at 45–60 seconds; the answer recorded in the README took 139.8 s.

That is not a footnote, it is a design constraint that propagated through the whole stack. It is why the Mem0 adapter carries a wall-clock watchdog. It is why every Ragas metric hit its 180-second default timeout and returned n/a. And it is why the documented default is Llama 3.1 8B while the real default on the box that produced every number in the repo is llama3.2:3b at roughly 2 GB.

One environment variable moves between them:

ARC_L0_INFERENCE__MODEL=llama3.2:3b

The web UI's stack sidebar reads the live resolved config rather than the file on disk, so the L2 row changes the moment you set it. A stack diagram that shows what is configured rather than what is running is decoration.

The three things I would take away

1. Check the licence per model, not per family, and know that self-hosting removes the bill and not the terms.
2. Your token budget is not your word count — digits and non-Latin scripts are charged differently, and a high top_k retrieves chunks the packer then silently drops.
3. Bandwidth divided by model size is your speed ceiling, and on CPU it is low enough to change what the rest of your stack must do about timeouts.

Level 3 is agent frameworks, where the question stops being "which model" and starts being "who decides what happens next".

The whole stack, nine levels, all free to self-host: https://dev48.infy.uk/arcrector.php