From API to GPU, Week 2: What Actually Happens Behind the API

Phase 1 of 8: Comfortable running local models. Week 2 of 32.

In Week 1 I went through the machine and proved the GPU works. This week I run an
actual language model on it and, for the first time, look at what sits behind the
"AI API" I have called for years. The short version: the API is not magic. It is
an HTTP layer over a local process that loads a file of numbers into memory and
does the matrix math from Week 1. By the end of this post that sentence will be
clear, backed by real commands and real output you can reproduce.

I am using Ollama, a runtime that makes running a local model about as easy as
running a container. Other names you will see in this space are llama.cpp for
lightweight local inference, vLLM for high-throughput serving, and NVIDIA
TensorRT-LLM for optimized NVIDIA inference. Hugging Face Transformers is also
common, but it is a broader Python framework for running and training models,
not a ready-made local model service. These tools overlap, but they are not exact
replacements. I chose Ollama because it gives me a CLI and local HTTP API with
very little setup.¹

Everything below runs on the DGX Spark and is reached over SSH as spark, the
same setup as Week 1.

Concept 1: a model is not a runtime

The first idea to keep straight, because it holds for the rest of the series, is
the split between the model and the runtime.

• The model is trained data: weights plus the metadata needed to use them. On its own it does nothing. It is data on disk.
• The runtime is the program that loads those weights into memory and runs the math to turn your prompt into text. Ollama is the runtime here.

If you know Docker, there is a useful analogy, but it is not exact. A model
package is like an image made of versioned, content-addressed layers. Ollama is
like the engine that pulls those layers and starts the workload. Unlike a
container image, the main model layer is trained numeric data, not an app and
its operating-system files.

Ollama calls each stored package file a blob. Here, a blob is just a file
kept under a name derived from its content digest. The inspection below follows
the package index to the model blob and checks that it is the expected file.

Rather than run a wrapper script, I inspected the model with a few direct
commands, one at a time. Each command answers a single question, so you can
paste it, read the output, then move to the next.

First, list what Ollama has downloaded locally:

ssh spark 'ollama list'

NAME                       ID              SIZE      MODIFIED
nomic-embed-text:latest    0a109f422b47    274 MB    2 days ago
llama3.2:3b                a80c4f17acd5    2.0 GB    3 days ago
phi4:latest                ac896e5b8b34    9.1 GB    7 days ago
qwen3.6:35b-a3b-bf16       94061ddd23a7    71 GB     8 days ago

Next, ask Ollama for the focused facts about Phi-4. The Ollama server listens on
localhost:11434 on the Spark, so run this in a shell on the Spark
(ssh spark):

curl -s http://localhost:11434/api/show -d '{"model": "phi4"}' | jq '{
  format: .details.format,
    architecture: .details.family,
    parameters: .details.parameter_size,
    context_length: (.model_info | to_entries
      | map(select(.key | endswith(".context_length"))) | first.value),
    embedding_length: (.model_info | to_entries
      | map(select(.key | endswith(".embedding_length"))) | first.value),
    quantization: .details.quantization_level,
    capabilities,
    runtime_parameters: (.parameters | split("\n")
      | map(select(length > 0) | gsub(" +"; " ")))
  }'

{
  "format": "gguf",
  "architecture": "phi3",
  "parameters": "14.7B",
  "context_length": 16384,
  "embedding_length": 5120,
  "quantization": "Q4_K_M",
  "capabilities": [
    "completion"
  ],
  "runtime_parameters": [
    "stop \"<|im_start|>\"",
    "stop \"<|im_end|>\"",
    "stop \"<|im_sep|>\""
  ]
}

That call gives readable model facts, including "format": "gguf". For a
normal check, this is how I know Ollama identifies the model as GGUF. I do not
need to locate the raw file and inspect its bytes just to answer that question.

To see the exact files that make up the package, read Ollama's manifest. It is a
small JSON file on the Spark, so jq can read it directly. I select just the
package version and the file list:

jq '{schemaVersion, layers: [.layers[] | {mediaType, digest, size}]}' \
  /usr/share/ollama/.ollama/models/manifests/registry.ollama.ai/library/phi4/latest

{
  "schemaVersion": 2,
  "layers": [
    {
      "mediaType": "application/vnd.ollama.image.model",
      "digest": "sha256:fd7b6731c33c57f61767612f56517460ec2d1e2e5a3f0163e0eb3d8d8cb5df20",
      "size": 9053114464
    },
    {
      "mediaType": "application/vnd.ollama.image.template",
      "digest": "sha256:32695b892af87ef8fca6e13a1a31c67c1441d7398be037e366e2fc763857c06a",
      "size": 275
    },
    {
      "mediaType": "application/vnd.ollama.image.license",
      "digest": "sha256:fa8235e5b48faca34e3ca98cf4f694ef08bd216d28b58071a1f85b1d50cb814d",
      "size": 1084
    },
    {
      "mediaType": "application/vnd.ollama.image.params",
      "digest": "sha256:45a1c652dddc9efdcefa977ab81cfbe26b6e52bc8e78f2f4c698538783e0ac80",
      "size": 82
    }
  ]
}

The registry path ends in library/phi4/latest, so this manifest is the index
for the model name phi4 and tag latest. It does not hold the 9.1 GB of
weights. It lists the files that make up the Ollama package, much like a lock
file maps package names to exact artifacts.

Each entry under layers is one file in the package (one blob). It has three
useful fields:

• mediaType says what the file's role is. This package has a model, a prompt template, a license, and default parameters.
• digest is the file's content identity. The sha256 prefix names the hash algorithm, and the characters after the colon are the hash of the file's bytes.
• size says exactly how many bytes that file should contain.

The model layer's digest is also how Ollama names the file on disk. It stores
each blob under a blobs directory using the digest as the filename, with the
colon changed to a hyphen, so sha256:fd7b...df20 becomes sha256-fd7b...df20.
That file is 9,053,114,464 bytes, the size shown above, and its SHA-256
matches the digest. The linked investigation independently checks the file
header with xxd: bytes 47 47 55 46 spell GGUF in ASCII. That deeper check
is useful when reusing the raw file in another runtime, but .details.format
is the simple Ollama API answer.²

So ollama show and the manifest answer different questions. The metadata call
gives readable model facts such as architecture and context length. The
manifest tells Ollama which exact files form the runnable package and where to
find them by content ID.

You might now wonder whether another runtime can use the same GGUF file. That is
useful, but it is not part of the main Ollama lesson. I answer it near the end in
Can llama.cpp reuse this Ollama model?.

I am using phi4, Microsoft's 14.7B-parameter model. At 16 bits per weight,
14.7 billion weights alone would need about 29.4 GB. The local package is only
9.1 GB, which leads to the first surprise.

A quick tour of what each line means, since these terms come up constantly:

Field	Value	Plain meaning
format	gguf	model file format reported by Ollama
architecture	phi3	the neural-network design (phi4 reuses the phi3 family)
parameters	14.7B	how many trained weights the model has
context length	16384	the most tokens (prompt + reply) it can consider at once
embedding length	5120	width of each token vector; covered later
quantization	Q4_K_M	the weights are stored at about 4 bits each, not 16

That last row answers the 9.1 GB surprise. This build is quantized: its
weights use a mixed low-bit Q4_K_M representation instead of 16-bit values.
Four bits per weight would be about 7.35 GB before metadata and quantization
overhead, so a 9.1 GB model layer is reasonable.
Quantization is a whole phase later in this series (weeks 14 to 16). For now the
only thing to take away is that Ollama runs lower-precision weights, and that is
why this 14.7B model takes much less space than its 16-bit source model.

The three stop entries (<|im_start|>, <|im_end|>, <|im_sep|>) are
special tokens: markers the model was trained to use as turn boundaries in a
chat. The runtime watches for them to know when the model has finished its reply.
They show up again in the next section, so keep them in mind.

Concept 2: the API is just HTTP over a local process

Ollama exposes a local HTTP API on port 11434,³ the same shape of
thing as any cloud AI API. The difference is that the "server" is a process on
my own machine.
I called the generate endpoint with curl and formatted the reply with jq.
Run it directly on the Spark:

curl -s http://localhost:11434/api/generate -d '{
  "model": "phi4",
  "prompt": "Explain Kubernetes scheduling in three sentences.",
  "stream": false
}' | jq

Here is the complete response, nothing removed. It is long, but seeing all of it
is the point, because two parts of it explain how the model actually works:

{
  "model": "phi4",
  "created_at": "2026-07-14T03:11:50.862545331Z",
  "response": "Kubernetes scheduling is the process of assigning workloads, such as pods, to appropriate nodes within a cluster based on resource availability and constraints. The scheduler evaluates factors like CPU, memory requirements, affinity/anti-affinity rules, taints/tolerations, and node selectors to determine the best fit for each pod. This ensures optimal utilization of resources, workload distribution, and adherence to specified policies, thereby maintaining efficient cluster operation and performance.",
  "done": true,
  "done_reason": "stop",
  "context": [
    100264, 882, 100266, 198, 849, 21435, 67474, 38952, 304, 2380,
    23719, 13, 100265, 198, 100264, 78191, 100266, 198, 42, 30927,
    38952, 374, 279, 1920, 315, 61853, 990, 33785, 11, 1778,
    439, 55687, 11, 311, 8475, 7954, 2949, 264, 10879, 3196,
    389, 5211, 18539, 323, 17413, 13, 578, 29909, 67349, 9547,
    1093, 14266, 11, 5044, 8670, 11, 51552, 14, 15719, 71260,
    13797, 5718, 11, 259, 1673, 82, 5640, 22847, 811, 11,
    323, 2494, 57137, 311, 8417, 279, 1888, 5052, 369, 1855,
    7661, 13, 1115, 26420, 23669, 50549, 315, 5070, 11, 54696,
    8141, 11, 323, 71628, 311, 5300, 10396, 11, 28592, 20958,
    11297, 10879, 5784, 323, 5178, 13
  ],
  "total_duration": 3974942694,
  "load_duration": 100037676,
  "prompt_eval_count": 18,
  "prompt_eval_duration": 46132000,
  "eval_count": 89,
  "eval_duration": 3817895000
}

Reading it top to bottom:

The response field is the answer, a real reply from a model running on my
hardware with no network call leaving the box.

The context field is the part worth staring at. You might be wondering what
those numbers are. They are tokens. A model does not read text; it reads
token IDs, which are integers. Before anything runs, a tokenizer splits the
text into tokens and maps each one to an integer. And this array is not just the
prompt, it is the whole conversation as tokens: the chat template, my question,
and the model's full answer. Decoding the first several so it is not a mystery:

• 100264 is the special marker <|im_start|> (start of a turn), and 882 is the word "user". So the conversation begins "start of turn, user".
• 849, 21435, 67474, 38952 are "Ex", "plain", " Kubernetes", " scheduling". Notice "Explain" is split into two tokens. Tokens are often sub-word pieces, not whole words.
• 304, 2380, 23719, 13 are " in", " three", " sentences", ".", finishing my prompt.
• 100265 is <|im_end|> (end of turn), then 100264 78191 100266 is the start of the assistant's turn ("start, assistant, separator").
• 42, 30927 onward ("K", "ubernetes", ...) is the model's answer, token by token, which is exactly the text in the response field above.

The special IDs 100264, 100265, 100266 are the chat markers
(<|im_start|>, <|im_end|>, <|im_sep|>), the turn boundaries from the
model's embedded tokenizer and chat template. Two questions probably popped into
your head reading that, the same ones that popped into mine: wait, how do I even
know 849 is "Ex" or that 100264 is <|im_start|>? And how does Ollama know
which tokenizer to use? Both have clear answers, but they would break the flow
here, so I answer them at the end. If you want them now, jump to
the token questions, answered.

This is why token counts matter everywhere. On a cloud API you pay per token. In
a model, tokens are what fill the context window, and the whole conversation is
carried forward as this growing list of integers. When people say "tokens per
second," this array is the unit being counted.

Concept 3: inference has two measured phases

The bottom of that same output has the timing, and it turns vague words like
"latency" into measured numbers. All the durations are in nanoseconds, so here
they are converted:

Field	Raw value	Converted value
load_duration	100,037,676 ns	0.10 s
prompt_eval_count	18	18 input tokens
prompt_eval_duration	46,132,000 ns	46 ms
eval_count	89	89 output tokens
eval_duration	3,817,895,000 ns	3.82 s
total_duration	3,974,942,694 ns	3.97 s

prompt_eval_duration is the time spent reading the tokenized input. That is
the prefill phase. eval_duration is the time spent generating output
tokens. That is the decode phase. load_duration is time Ollama spent
loading or preparing the model for this request.

One headline number can be calculated directly:

• Tokens per second (generation speed): 89 tokens divided by 3.82 s is about 23.3 tokens/sec.

I cannot honestly calculate time to first token (TTFT) from this
non-streaming response. Adding load and prefill gives 146 ms of server work
before decode, but that is not the same as observing when the first token reaches
the client. To measure TTFT, the client must request a streamed response and
timestamp the first non-empty token. I do that in the benchmark later in this
post.

The timing split also shows that generating text has two phases that behave
differently:

1. Prefill (prompt_eval): the model reads all 18 prompt tokens at once. This is fast (46 ms) because the tokens are processed together, which is the parallel, compute-heavy work the GPU from Week 1 is good at.
2. Decode (eval): the model generates output tokens one at a time, each one depending on the previous. This is the slow, sequential phase, and it is where memory bandwidth matters. Each token passes through the model layers and uses their weights again, while the key-value cache avoids recalculating all prior tokens from scratch.

That is the core of Week 2. The API is HTTP over a local process, the "text" is
really a stream of integer tokens, and a request is load, then prefill, then
decode, each one measurable. Now I can start changing the settings that shape the
output.

Concept 4: temperature, or how random the model is allowed to be

Back in Concept 2 I ran the same prompt twice and got slightly different wording
each time. That is not a bug. It is a setting called temperature, and it
controls how random the model is allowed to be when it picks each next token.

The setting is options.temperature. I sent the same coffee-shop prompt twice
at each temperature to watch it change. I changed the value in this direct call
and ran it twice:

curl -s http://localhost:11434/api/generate -d '{
  "model": "phi4",
  "prompt": "Give me a one-sentence tagline for a coffee shop.",
  "stream": false,
  "options": { "temperature": 0 }
}' | jq -r .response

Twice at temperature 0:

temperature=0
run 1: "Awaken Your Senses, One Cup at a Time."
run 2: "Awaken Your Senses, One Cup at a Time."

These two runs matched. The same call with "temperature": 1.2 gives variety:

temperature=1.2
run 1: "Where every sip is a moment of delight."
run 2: "Where Every Cup is a Perfect Brew—Awaken Your Senses."

Same input, two different answers.

So what is temperature actually doing? At each step, the model does not just pick
one next token. It produces a probability for every token in its vocabulary, like
"there is a 40% chance the next token is Awaken, 8% chance it is Where, and so
on". Temperature reshapes that list before one token is picked. Top-p and seed
also affect the choice, but each control has a different job:

Control	Changes	Lower/same	Higher/different
Temperature	Probability shape	Favors likely tokens	Allows more variety
Top-p	Candidate set	Fewer likely tokens	More possible tokens
Seed	Random sequence	Same starting sequence	Another sequence

Top-p is also called nucleus sampling. At 0.9, it keeps the most likely
tokens whose probabilities add up to 90%, then samples only from that set.
Temperature and top-p can be used together, but while learning I would change
one at a time so I know which setting changed the output.

You might also see a seed beside temperature. A model runtime uses a
pseudorandom number generator when it samples from the possible next tokens.
Pseudorandom means the values look random, but they are generated by a formula.
The seed is the starting number for that formula, like starting with the same
shuffle of a deck. The same seed gives the sampler the same sequence of random
choices when the model, prompt, options, runtime, and execution conditions stay
the same.

I tested that with temperature 1.2, where sampling has room to vary:

for RUN in 1 2 3 4; do
  printf "run=%s seed=42: " "$RUN"
  curl -s http://localhost:11434/api/generate -d '{
    "model": "phi4",
    "prompt": "Reply with one invented coffee shop name and nothing else.",
    "stream": false,
    "options": {"temperature": 1.2, "seed": 42, "num_predict": 16}
  }' | jq -r .response | tr '\n' ' '
  printf "\n"
done

run=1 seed=42: Espresso Enchantments
run=2 seed=42: Espresso Enchantments
run=3 seed=42: Espresso Enchantments
run=4 seed=42: Espresso Enchantments

These four warm-model runs matched. A fixed seed improves repeatability, but it
is not a universal guarantee. A different runtime version, model build,
hardware path, parallel execution order, or other nondeterministic GPU behavior
can still change the result. A seed controls the sampler's random sequence; it
does not freeze the whole software and hardware stack.

The practical takeaway is simple. Start near zero when repeatability matters,
and raise temperature when you want variety. Add a fixed seed when you want more
repeatable comparisons, and record the runtime and model version too. Everything
else this week uses the same options object in the API call.

Concept 5: chat is a list of role-tagged messages

So far I used /api/generate, which takes one plain prompt. Ollama also has
chat interfaces. I will use them in increasing depth:

1. The CLI proves I can have a one-shot conversation.
2. The HTTP API exposes the role-tagged message structure.
3. The Python client keeps that structure across multiple turns.

First, chat through the CLI

I keep using phi4, already pulled in Concept 1, so there is nothing new to
download. Ollama's CLI is terminal-aware, so I forced SSH to allocate a terminal
with -tt:

ssh -tt spark \
  'ollama run --nowordwrap phi4 \
  "Reply with exactly these three words: local model ready"'

Local model ready.

Connection to spark closed.

This proves the CLI can send a prompt and display a reply. It does not show the
message roles or how a client maintains a conversation. For that, I need the
HTTP API.

Next, inspect messages through the API

The /api/chat endpoint takes a messages array. Each item has a role that
says who wrote it and content that holds the text:

Role	Who writes it	What it does
system	Application	Sets behavior, style, or limits
user	Person using the app	Carries the user's prompt or follow-up
assistant	Model	Stores a previous model reply in the history

The system prompt is an instruction the application places before the user
conversation. It sets the expected role, style, or limits of the assistant. It
is configuration, not a security boundary. A user can still send conflicting
instructions, so production systems need checks outside the prompt too.

The user prompt is simply the content of a message with role: "user".
After the model replies, the client stores that answer as an assistant message.
The next request sends the earlier messages plus the new user message.

Why resend them? The Ollama API does not remember this client's conversation
between independent HTTP requests. I sent two consecutive requests to the same
running Ollama process. First, I included the earlier user question and
assistant reply:

curl -s http://localhost:11434/api/chat -d '{
  "model": "phi4",
  "stream": false,
  "messages": [
    {"role": "system", "content": "Reply with only a name or UNKNOWN."},
    {"role": "user", "content": "My cluster is called Atlas."},
    {"role": "assistant", "content": "Atlas"},
    {"role": "user", "content": "What name did I give it?"}
  ],
  "options": {"temperature": 0, "seed": 42}
}' | jq '{reply: .message, prompt_tokens: .prompt_eval_count}'

{
  "reply": {
    "role": "assistant",
    "content": "Atlas"
  },
  "prompt_tokens": 50
}

Then I sent only the follow-up, with no earlier turns:

curl -s http://localhost:11434/api/chat -d '{
  "model": "phi4",
  "stream": false,
  "messages": [
    {"role": "system", "content": "Reply with only a name or UNKNOWN."},
    {"role": "user", "content": "What name did I give it?"}
  ],
  "options": {"temperature": 0, "seed": 42}
}' | jq '{reply: .message, prompt_tokens: .prompt_eval_count}'

{
  "reply": {
    "role": "assistant",
    "content": "UNKNOWN"
  },
  "prompt_tokens": 31
}

Request	Reply	Prompt tokens
Earlier turns included	Atlas	50
Earlier turns omitted	UNKNOWN	31

What changed between the two calls? Only the messages array.

In the first request, the JSON body included the earlier line My cluster is
called Atlas, the model's earlier answer, and the new follow-up. Ollama could
therefore pass all of that text to the model, so the answer was Atlas.

In the second request, the conversation history contained only What name did I
give it?. There was no mention of Atlas in its messages, so the model answered
UNKNOWN. Ollama did not look up the earlier HTTP call and add it automatically.

That is what the client owns the conversation state means. The application,
such as this Python client, keeps the message list in memory or a database. For
each new turn, it appends the latest user message and sends the relevant history
again. Ollama processes the messages it receives in that request.

Resending history has a cost. The request with earlier turns contained 50 prompt
tokens; the request without them contained 31. The 19-token difference includes
the added message text, role labels, and chat-template separators. I measured
the total difference here; I did not split it token by token. All 19 additional
tokens used positions in the context window.

Finally, implement the conversation in Python

The required Python client now implements the message flow I just tested. It
uses /api/chat, keeps prior turns in the messages
list, and streams the reply as it arrives. With "stream": true, Ollama sends
one JSON object per line; each text fragment is in message.content. The loop
prints each fragment and joins them into the completed assistant message. Here
is the complete file:

#!/usr/bin/env python3
"""Small command-line chat client for Ollama's streaming /api/chat endpoint."""

import argparse
import json
import os
import urllib.error
import urllib.request


def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--model", default=os.getenv("MODEL", "phi4"))
    parser.add_argument("--url", default="http://localhost:11434")
    parser.add_argument("--system", default="You are a concise assistant.")
    parser.add_argument("--temperature", type=float, default=0.0)
    parser.add_argument("--top-p", type=float, default=0.9)
    parser.add_argument("--num-ctx", type=int, default=4096)
    parser.add_argument("--once", help="Send one prompt and exit")
    return parser.parse_args()


def request_chat(
    url: str,
    model: str,
    messages: list[dict[str, str]],
    options: dict[str, float | int],
) -> str:
    body = {
        "model": model,
        "messages": messages,
        "stream": True,
        "options": options,
    }
    request = urllib.request.Request(
        f"{url.rstrip('/')}/api/chat",
        data=json.dumps(body).encode(),
        headers={"Content-Type": "application/json"},
    )
    parts: list[str] = []
    try:
        with urllib.request.urlopen(request) as response:
            for line in response:
                event = json.loads(line)
                text = event.get("message", {}).get("content", "")
                if text:
                    print(text, end="", flush=True)
                    parts.append(text)
    except urllib.error.URLError as exc:
        raise SystemExit(f"Ollama request failed: {exc}") from exc
    print()
    return "".join(parts)


def main() -> None:
    args = parse_args()
    messages: list[dict[str, str]] = []
    if args.system:
        messages.append({"role": "system", "content": args.system})
    options = {
        "temperature": args.temperature,
        "top_p": args.top_p,
        "num_ctx": args.num_ctx,
    }

    if args.once:
        messages.append({"role": "user", "content": args.once})
        request_chat(args.url, args.model, messages, options)
        return

    print(f"Chatting with {args.model}. Type /exit to quit.")
    while True:
        try:
            prompt = input("you> ").strip()
        except (EOFError, KeyboardInterrupt):
            print()
            return
        if prompt.lower() in {"/exit", "/quit"}:
            return
        if not prompt:
            continue
        messages.append({"role": "user", "content": prompt})
        print("model> ", end="", flush=True)
        answer = request_chat(args.url, args.model, messages, options)
        messages.append({"role": "assistant", "content": answer})


if __name__ == "__main__":
    main()

I used --once for a repeatable blog example and passed a system prompt. If I
leave off --once, the same program opens an interactive loop and keeps each
user and assistant turn in the next request.

ssh spark '~/venvs/w1/bin/python - \
  --model phi4 \
  --system "Answer in exactly one short sentence." \
  --once "What does the Kubernetes scheduler do?"' \
  < public/week-02-first-local-model/chat_client.py

The Kubernetes scheduler assigns pods to nodes based on resource availability,
constraints, and policies.

This is the same local runtime and the same phi4 model through three
interfaces: Ollama's CLI, /api/generate with curl, and /api/chat from
Python. Only the client interface changed.

Concept 6: measure cold starts and verify context allocation

Imagine Phi-4 is behind a chat application. No one has used it for a while, so
Ollama has unloaded it from memory. A user sends a message and waits about six
seconds before seeing the first word. They send another message a minute later
and see the first word in a fraction of a second.

That difference is what this experiment measures:

Request state	What Ollama must do	What the user notices
Cold	Load the model, then process the prompt	Long wait for first text
Warm	Reuse the model already in memory	First text arrives quickly

A cold request starts with the model unloaded, so Ollama must load it before
inference. A warm request starts with the model already resident in memory
and ready for reuse.

Before running anything, I expected the first request to have a much higher
time to first token (TTFT) because it includes model loading. I expected the
next requests to start quickly. I did not expect the output generation rate to
change much, because all runs use the same model and output length.

I also used the experiment to answer a separate configuration question:

1. What does num_ctx change? It changes how much token capacity Ollama allocates. It does not make a short prompt longer or automatically faster.
2. How different is the first request from later requests? A streamed client can measure real time to first token for both a cold and warm model.

These are two checks in one script, not a claim that context size causes the
cold-start difference. The cold-versus-warm runs measure model loading. The
2,048-versus-4,096 setting verifies how I request context capacity.

The model metadata showed a maximum context length of 16,384 tokens. The
configured context size tells Ollama how much of that capacity to allocate
for a run. The prompt does not become 4,096 tokens just because I set
num_ctx to 4,096. It means the prompt, chat history, and generated reply may
use up to that budget.

A larger context also needs a larger key-value cache, often shortened to KV
cache. That cache holds intermediate values from earlier tokens so decode does
not repeat all prior work for every new token. More context gives the cache room
for more tokens, which uses more memory.

The benchmark changes num_ctx from 2,048 to 4,096. At each setting it runs
three requests in this exact order:

Run	Model state	Purpose
1	Cold	Measure the user-visible cost of loading Phi-4
2	Warm	Measure a request with Phi-4 already loaded
3	Warm	Repeat the warm measurement once more

It fixes temperature and seed, limits each reply to 96 tokens, and calls
ollama stop phi4 before run 1 at each context setting. Runs 2 and 3 do not stop
the model, so they reuse it in memory.

Most importantly, it requests a streamed response. The client starts a clock
before the HTTP request and stops the TTFT clock when the first non-empty text
chunk arrives. That is a real client-observed TTFT measurement.

Here is the full benchmark:

#!/usr/bin/env python3
"""Measure client TTFT and server timings from Ollama's streaming API."""

import argparse
import json
import subprocess
import time
import urllib.request
from pathlib import Path
from typing import Any


def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--model", default="phi4")
    parser.add_argument(
        "--prompt",
        default="Explain Kubernetes scheduling in three sentences.",
    )
    parser.add_argument("--contexts", default="2048,4096")
    parser.add_argument("--runs", type=int, default=3)
    parser.add_argument("--url", default="http://localhost:11434")
    parser.add_argument("--output", type=Path)
    return parser.parse_args()


def unload(model: str) -> None:
    subprocess.run(
        ["ollama", "stop", model],
        stdout=subprocess.DEVNULL,
        stderr=subprocess.DEVNULL,
        check=False,
    )
    time.sleep(1)


def run_once(
    url: str,
    model: str,
    prompt: str,
    num_ctx: int,
    cold: bool,
) -> dict[str, Any]:
    if cold:
        unload(model)
    body = {
        "model": model,
        "prompt": prompt,
        "stream": True,
        "options": {
            "temperature": 0,
            "seed": 42,
            "num_ctx": num_ctx,
            "num_predict": 96,
        },
    }
    request = urllib.request.Request(
        f"{url.rstrip('/')}/api/generate",
        data=json.dumps(body).encode(),
        headers={"Content-Type": "application/json"},
    )

    started = time.perf_counter()
    first_token_at: float | None = None
    final: dict[str, Any] = {}
    with urllib.request.urlopen(request) as response:
        for line in response:
            event = json.loads(line)
            if event.get("response") and first_token_at is None:
                first_token_at = time.perf_counter()
            if event.get("done"):
                final = event
    finished = time.perf_counter()

    if first_token_at is None or not final:
        raise RuntimeError("Ollama stream ended without timing data")
    eval_seconds = final["eval_duration"] / 1e9
    return {
        "context": num_ctx,
        "cold": cold,
        "ttft_ms": round((first_token_at - started) * 1000, 1),
        "client_total_ms": round((finished - started) * 1000, 1),
        "load_ms": round(final["load_duration"] / 1e6, 1),
        "prompt_eval_ms": round(final["prompt_eval_duration"] / 1e6, 1),
        "prompt_tokens": final["prompt_eval_count"],
        "output_tokens": final["eval_count"],
        "tokens_per_second": round(final["eval_count"] / eval_seconds, 1),
    }


def main() -> None:
    args = parse_args()
    contexts = [int(value) for value in args.contexts.split(",")]
    rows: list[dict[str, Any]] = []
    print("ctx  run  cold  ttft_ms  total_ms  load_ms  out_tok  tok/s")
    for context in contexts:
        for run_number in range(1, args.runs + 1):
            row = run_once(
                args.url,
                args.model,
                args.prompt,
                context,
                cold=run_number == 1,
            )
            rows.append(row)
            print(
                f"{context:>4}  {run_number:>3}  {str(row['cold']):>5}  "
                f"{row['ttft_ms']:>7.1f}  {row['client_total_ms']:>8.1f}  "
                f"{row['load_ms']:>7.1f}  {row['output_tokens']:>7}  "
                f"{row['tokens_per_second']:>5.1f}"
            )

    result = {
        "model": args.model,
        "prompt": args.prompt,
        "runs_per_context": args.runs,
        "measurements": rows,
    }
    if args.output:
        args.output.write_text(json.dumps(result, indent=2) + "\n")


if __name__ == "__main__":
    main()

I ran three requests per context size. The first is cold because the script
stops Phi-4 before it. The next two reuse the loaded model.

ssh spark '~/venvs/w1/bin/python - --contexts 2048,4096 --runs 3' \
  < public/week-02-first-local-model/benchmark.py

ctx  run  cold  ttft_ms  total_ms  load_ms  out_tok  tok/s
2048    1   True   6034.1   10129.6   5908.4       96   23.4
2048    2  False    196.1    4288.0    140.4       96   23.5
2048    3  False    151.9    4244.7    104.1       96   23.5
4096    1   True   6260.8   10369.2   6150.1       96   23.4
4096    2  False    174.3    4262.8    117.8       96   23.5
4096    3  False    173.5    4260.8    124.8       96   23.5

The cold requests took about 6.0 to 6.3 seconds before the first token. The warm
requests took 151.9 to 196.1 ms. This is the cold-start effect I could not
measure from the earlier non-streaming call.

Generation stayed at 23.4 to 23.5 tokens/sec. The short 18-token input did not
use the extra context capacity, so doubling the setting did not make this test
faster. It just allowed a larger KV cache. ollama ps confirmed the last
requested allocation:

ssh spark 'ollama ps'

NAME           ID              SIZE      PROCESSOR    CONTEXT    UNTIL
phi4:latest    ac896e5b8b34    9.7 GB    100% GPU     4096       4 minutes from now

What did this experiment prove?

• A cold Phi-4 request took about 6 seconds to produce its first token because Ollama had to load the model.
• Warm requests produced the first token in about 0.15 to 0.20 seconds.
• Ollama accepted the final 4,096-token setting and reported CONTEXT 4096.
• Doubling the requested context did not change throughput for this 18-token prompt. Both settings stayed near 23.5 tokens/sec.

It did not measure the memory difference between the two context settings,
and it did not test a long prompt. Those need a separate experiment. The lesson
here is simpler: keep models warm for latency, and treat context size as a
capacity setting rather than a speed setting.

The token questions, answered

Earlier, in the token section (Concept 2), I parked two questions to keep the
main thread moving: how do I even know that 849 is "Ex", and how does Ollama
know which tokenizer to use? Here are the answers. You do not need them to follow
the rest of the post, so read on only if you want to poke at the internals.

Wait, how do I even know what each token ID means?

I did not guess. A runnable model package needs a tokenizer, the exact
rulebook that maps text to token IDs and back. The ordinary Phi-4 token IDs in
the API response match OpenAI's cl100k_base encoding. I verified that match
with decode_tokens.py instead of inferring it from the size of the IDs.

#!/usr/bin/env python3
"""Week 2 - show how text becomes tokens for a cl100k model like phi4.

Encodes the prompt and decodes individual token IDs, so you can match the
`context` array from the generate API back to real words.

Run with a Python that has tiktoken installed, e.g. the Week 1 venv:
    ~/venvs/w1/bin/python -m pip install -q tiktoken
    ~/venvs/w1/bin/python decode_tokens.py
"""
import tiktoken

PROMPT = "Explain Kubernetes scheduling in three sentences."
# A few IDs taken from the generate API's context array.
SAMPLE_IDS = [849, 21435, 67474, 38952, 304, 2380, 23719, 13, 882, 78191]


def main() -> None:
    enc = tiktoken.get_encoding("cl100k_base")

    print("encode:", enc.encode(PROMPT))
    print("decode individual IDs:")
    for t in SAMPLE_IDS:
        print(f"  {t:>6}  {enc.decode([t])!r}")


if __name__ == "__main__":
    main()

Then I ran the file with the pinned tiktoken 0.13.0 package:

ssh spark '~/venvs/w1/bin/python -' \
  < public/week-02-first-local-model/decode_tokens.py

encode: [849, 21435, 67474, 38952, 304, 2380, 23719, 13]
decode individual IDs:
     849  'Ex'
   21435  'plain'
   67474  ' Kubernetes'
   38952  ' scheduling'
     304  ' in'
    2380  ' three'
   23719  ' sentences'
      13  '.'
     882  'user'
   78191  'assistant'

The first line reproduces the exact prompt tokens from the context array, which
confirms phi4 really does use this vocabulary. Decoding the individual IDs gives
back the words, including user and assistant. This proves the ordinary
prompt and role IDs match cl100k_base. It does not prove what the added special
IDs mean. I verify those from the model file in the next section.

Is cl100k a universal standard? No. It is one of a small set of named
vocabularies from OpenAI's tiktoken library, where each vocabulary (such as
cl100k_base for GPT-4 and the newer o200k_base for GPT-4o) is tied to specific
models.⁴ Other model families use entirely different tokenizers. Llama
models, for example, use a different scheme with different token IDs, so the same
word gets a different number there. Each model comes with its own vocabulary, and
you can always load that exact tokenizer to see how it splits text.

And how does Ollama know which tokenizer to use?

The runtime is what turns the prompt into tokens, so this is a fair question. It
does not keep a list of models and guess. The tokenizer is packed inside the
model file. Ollama stores models in the GGUF format,⁵ and a GGUF file
carries the tokenizer metadata next to the weights.

There are two files involved:

1. The small manifest file is /usr/share/ollama/.ollama/models/manifests/registry.ollama.ai/library/phi4/latest. It is a JSON index for the Phi-4 package. It does not contain the tokenizer. Its model-layer entry tells Ollama which blob file to open.
2. The model blob is in /usr/share/ollama/.ollama/models/blobs. Its filename is sha256-fd7b6731c33c57f61767612f56517460ec2d1e2e5a3f0163e0eb3d8d8cb5df20. This 9 GB file is the actual GGUF model. It contains the model weights, tokenizer vocabulary, special tokens, and chat template.

I resolved that path directly before using Python:

ssh spark '
  MODELS_DIR=/usr/share/ollama/.ollama/models
  MANIFEST_FILE=manifests/registry.ollama.ai/library/phi4/latest
  MANIFEST="$MODELS_DIR/$MANIFEST_FILE"
  DIGEST="$(jq -r '\''.layers[]
    | select(.mediaType == "application/vnd.ollama.image.model")
    | .digest'\'' "$MANIFEST")"
  BLOB_FILE="blobs/${DIGEST/:/-}"
  printf "models directory: %s\n" "$MODELS_DIR"
  printf "manifest file: %s\n" "$MANIFEST_FILE"
  printf "model digest: %s\n" "$DIGEST"
  printf "GGUF file: %s\n" "$BLOB_FILE"
  stat -c "GGUF bytes: %s" "$MODELS_DIR/$BLOB_FILE"
'

models directory: /usr/share/ollama/.ollama/models
manifest file: manifests/registry.ollama.ai/library/phi4/latest
model digest: sha256:fd7b6731c33c57f61767612f56517460ec2d1e2e5a3f0163e0eb3d8d8cb5df20
GGUF file: blobs/sha256-fd7b6731c33c57f61767612f56517460ec2d1e2e5a3f0163e0eb3d8d8cb5df20
GGUF bytes: 9053114464

The digest changes from sha256:... in the manifest to sha256-... in the
blob filename. Only the colon becomes a hyphen. The Python script automates
these same steps, opens that GGUF blob, and reads a few tokenizer fields so I do
not have to inspect a 9 GB binary file by hand:

#!/usr/bin/env python3
"""Read selected tokenizer metadata directly from an Ollama GGUF model layer."""

import argparse
import json
import os
from pathlib import Path

from gguf import GGUFReader


def parse_args() -> argparse.Namespace:
  parser = argparse.ArgumentParser(description=__doc__)
  parser.add_argument("--model", default=os.getenv("MODEL", "phi4"))
  parser.add_argument(
    "--models-dir",
    type=Path,
    default=Path("/usr/share/ollama/.ollama/models"),
  )
  return parser.parse_args()


def field(reader: GGUFReader, name: str):
  value = reader.get_field(name)
  if value is None:
    raise KeyError(f"Missing GGUF field: {name}")
  return value.contents()


def main() -> None:
  args = parse_args()
  name, separator, tag = args.model.partition(":")
  if not separator:
    tag = "latest"
  manifest = (
    args.models_dir
    / "manifests"
    / "registry.ollama.ai"
    / "library"
    / name
    / tag
  )
  data = json.loads(manifest.read_text())
  digest = next(
    layer["digest"]
    for layer in data["layers"]
    if layer["mediaType"].endswith("image.model")
  )
  blob = args.models_dir / "blobs" / digest.replace(":", "-")
  reader = GGUFReader(blob)
  tokens = field(reader, "tokenizer.ggml.tokens")
  template = field(reader, "tokenizer.chat_template")

  print("tokenizer model:", field(reader, "tokenizer.ggml.model"))
  print("tokenizer pre:", field(reader, "tokenizer.ggml.pre"))
  print("token count:", len(tokens))
  for token_id in (100257, 100264, 100265, 100266):
    print(f"token {token_id}: {tokens[token_id]!r}")
  for marker in ("<|im_start|>", "<|im_end|>", "<|im_sep|>"):
    print(f"template contains {marker}: {marker in template}")


if __name__ == "__main__":
  main()

I ran it with the pinned gguf 0.19.0 package:

ssh spark '~/venvs/w1/bin/python -' \
  < public/week-02-first-local-model/inspect_tokenizer.py

tokenizer model: gpt2
tokenizer pre: dbrx
token count: 100352
token 100257: '<|endoftext|>'
token 100264: '<|im_start|>'
token 100265: '<|im_end|>'
token 100266: '<|im_sep|>'
template contains <|im_start|>: True
template contains <|im_end|>: True
template contains <|im_sep|>: True

Everything the runtime needs is in the file:

• tokenizer.ggml.model = 'gpt2' is the tokenizer type. It is a byte-pair encoding (BPE), the same family GPT-2 and GPT-4 use.
• tokenizer.ggml.tokens is the full vocabulary, all 100,352 entries, the list that maps text pieces to IDs.
• The three IDs are the exact special-token strings claimed earlier.
• The embedded chat template contains those same markers. Phi-4's model card uses this ChatML-style prompt layout.⁶

So Ollama reads the vocabulary from the model file itself and uses it to turn my
prompt into token IDs. The ordinary IDs used in this prompt are compatible with
cl100k_base, while the GGUF adds Phi-4's special chat tokens. A Llama model's
GGUF carries a different vocabulary, and Ollama uses that instead. Same runtime,
different tokenizer per model, because the tokenizer travels with the model.

Results

Here is the Week 2 result in one place:

Item	Measured result
Runtime	Ollama 0.31.2
Main model	Phi-4, 14.7B parameters, Q4_K_M
Model layer	9,053,114,464 bytes
Context settings tested	2,048 and 4,096 tokens
Cold TTFT	6,034.1 to 6,260.8 ms
Warm TTFT	151.9 to 196.1 ms
Generation speed	23.4 to 23.5 tokens/sec
Python deliverable	Streaming command-line chat client
Benchmark deliverable	Cold/warm TTFT and throughput benchmark

I also used every interface in the plan: interactive CLI, curl, and Python.
The commands changed temperature and context size. The Python client used a
system prompt, and the benchmark compared repeatable runs with measured timing.

What surprised me

The first surprise was the model size. Phi-4 has 14.7 billion parameters, but
its quantized model layer is 9.1 GB instead of the roughly 29.4 GB needed for
16-bit weights alone.

The bigger surprise was the cold start. Generation speed stayed near 23.5
tokens/sec, but the first token took about 6 seconds when Ollama had to load the
model. Once warm, it arrived in about 0.15 to 0.20 seconds.

Changing context capacity from 2,048 to 4,096 did not improve this short
request. A larger limit gives the KV cache room for more tokens. It does not make
an 18-token prompt larger or automatically faster.

Mistakes and troubleshooting

I first called the output of ollama show a manifest. That was wrong. Model
metadata is a readable summary. The manifest is the JSON index that points to
the model, template, license, and parameter layers.

I also tried to estimate TTFT by adding load_duration and
prompt_eval_duration from a non-streaming response. That is useful server
timing, but it is not client-observed TTFT. The fixed benchmark streams the
response and timestamps the first non-empty text chunk.

One smaller issue came from Ollama's terminal-aware CLI. Its one-shot response
was not visible through plain SSH capture, so I used ssh -tt to allocate a
pseudo-terminal. The Python and curl clients did not need that workaround.

Production implications

A production service should keep commonly used models warm. Otherwise, a user
can wait seconds for model loading before generation even starts. Streaming also
matters because it lets the user see the first token instead of waiting for the
whole reply.

Context size is a capacity and memory decision. Setting it to the model maximum
for every request can reserve more KV-cache memory than the workload needs. The
right setting comes from measured prompt and conversation lengths.

The system prompt is not an access-control system. If I expose Ollama beyond
localhost, I still need a gateway with authentication, authorization, TLS, rate
limits, request-size limits, and logging. Prompt instructions do not replace
those controls.

What I will learn next

Week 3 moves from running a packaged model to opening a Hugging Face repository
and inspecting its files. I will compare base and instruct models, read the
model card and license, then inspect configuration, tokenizer, chat-template,
and Safetensors files.⁷

Run it yourself

The public Week 2 lab contains the chat client, benchmark, scripts, pinned
requirements, raw results, and model-run record.⁸

Can llama.cpp reuse this Ollama model?

This is optional reading. It is useful if you want to understand whether a model
file belongs to one runtime or can move between compatible runtimes.

For this Phi-4 download, yes. Ollama reports details.format = "gguf", and the
independent file-header check agrees. llama.cpp loads local GGUF files, so it
can use this same 9,053,114,464-byte model blob without downloading the weights
again.⁵ The content matters, not the missing .gguf extension on Ollama's
content-addressed filename.

A file header alone does not prove another runtime can load the model and
generate text, so I installed a pinned CUDA build of llama.cpp and pointed it
at the exact model blob named by Ollama's manifest digest. It loaded the file
and generated a reply:

Check	Result
Model blob	same file, Ollama manifest digest fd7b...df20
llama.cpp build	pinned commit 571d0d54
Device used	CUDA0 (NVIDIA GB10)
GPU layers offloaded	41/41
Generated reply	Shared model works.

That is a concrete yes. The full investigation records the failed first check,
CUDA toolkit discovery, pinned ARM64 build, scripts, commands, raw output, and
the apparent terminal hang.²

One limit remains. The model blob is only one layer of the Ollama package. The
manifest also lists a prompt template, license, and default parameters as
separate layers. llama.cpp does not read those Ollama layers just because it
can read the model blob. This Phi-4 GGUF carries its own tokenizer and chat
template, but I still set the seed, temperature, context, and GPU layers
explicitly. Same weights do not guarantee the same output from two runtimes.

This result is specific to this Phi-4 file and this pinned llama.cpp commit.
For another Ollama download, I would repeat the test because an older runtime
may not support a newer model architecture.

---

1. Official project docs for the alternatives named here:
   llama.cpp at https://github.com/ggml-org/llama.cpp, vLLM at
   https://docs.vllm.ai, NVIDIA TensorRT-LLM at
   https://nvidia.github.io/TensorRT-LLM/, and Hugging Face Transformers at
   https://huggingface.co/docs/transformers/index ↩

2. Full Ollama-to-llama.cpp investigation, including the pinned
   CUDA build and raw command output:
   https://github.com/dramasamy/from-api-to-gpu/blob/main/week-02-first-local-model/ollama-model-in-llamacpp.md ↩

3. Ollama API reference:
   https://docs.ollama.com/api/introduction ↩

4. tiktoken, OpenAI's tokenizer library, lists the standard vocabularies
   and which model each one belongs to. See the model-to-encoding table at
   https://github.com/openai/tiktoken ↩

5. GGUF is the model file format from the llama.cpp project. It stores the
   weights plus metadata such as the tokenizer and chat template. See
   https://github.com/ggml-org/ggml/blob/master/docs/gguf.md ↩

6. Microsoft Phi-4 model card, including its chat input format:
   https://huggingface.co/microsoft/phi-4#input-formats ↩

7. Week 3 roadmap:
   https://github.com/dramasamy/from-api-to-gpu/blob/main/roadmap/week-03.md ↩

8. Week 2 companion lab:
   https://github.com/dramasamy/from-api-to-gpu/tree/main/week-02-first-local-model ↩