From API to GPU, Week 1: Understanding NVIDIA DGX Spark Environment

I've used AI through APIs for years — POST a prompt, get tokens back, ship the
feature. I have never once deployed a model myself. No PyTorch, no GPU memory
math, no idea what actually happens between my HTTP request and the text that
comes back. This series is me closing that gap on purpose, one week at a time,
on an NVIDIA DGX Spark.

I'm a software engineer and technical program manager. I'm comfortable with
Linux, Python, Docker, Kubernetes, and APIs. I'm a complete beginner at machine
learning. So Week 1 is deliberately unglamorous: before running any model, I
want to know the machine — what CPU and GPU it has, how its memory works, and
what the NVIDIA software stack underneath is actually made of. Every claim below
is backed by a real command and its real output, so you can run the same thing
on your own box and compare.

About this series

From API to GPU is a 32-week journey from AI-API consumer to
local LLM systems architect — running, optimizing, and eventually
fine-tuning models on local hardware, documenting each week as a hands-on lab
plus a blog post. The full week-by-week plan lives in the roadmap¹, and
every week's runnable code lands in the companion GitHub repo². If you have
a similar machine, you can follow along and reproduce every result.

The plan runs in eight phases, with a parallel CUDA track starting around week 5:

Phase	Weeks	Focus
1	1–4	Comfortable running local models
2	5–8	Enough ML to understand inference
3	9–13	Transformers and terminology
4	14–16	Quantization and model formats
5	17–20	Inference engineering
6	21–24	Production model services
7	25–28	RAG and application integration
8	29–32	Fine-tuning

The goal of this first post is narrow on purpose: stand up and understand
the environment. By the end you'll be able to inventory a DGX Spark, read what
nvidia-smi tells you, explain how its unified memory differs from a normal
GPU, untangle the NVIDIA driver/CUDA-runtime/toolkit layers, and prove the GPU
is usable from PyTorch with a measured CPU-vs-GPU speedup. No model yet — that's
week 2. This is the foundation everything else builds on.

All the commands in this post are packaged as a runnable script in the companion
repo under week-01-environment/³ — clone it, set up an SSH alias spark
that reaches your DGX Spark, and run ./inventory.sh to reproduce everything
here. The repo holds the commands and scripts; this post holds the explanations,
so neither repeats the other.

Here's the whole inventory script, so you can see exactly what it runs without
cloning anything. It just wraps each command in an SSH call to the Spark and
prints a labelled section:

#!/usr/bin/env bash
# Week 1 — DGX Spark machine inventory. Prereq: an SSH alias `spark`.
set -euo pipefail
SPARK_HOST="${SPARK_HOST:-spark}"

run() { echo "=== {% katex inline %}1 ==="; ssh "{% endkatex %}SPARK_HOST" "$2" 2>&1 || true; echo; }

run "uname"      'uname -a'
run "os-release" 'cat /etc/os-release'
run "lscpu"      'lscpu'
run "memory"     'free -h'
run "storage"    'lsblk'
run "gpu"        'nvidia-smi'
run "cuda-nvcc"  'nvcc --version || /usr/local/cuda/bin/nvcc --version'
run "cuda-dirs"  'ls -d /usr/local/cuda*'
run "python"     'python3 --version'
run "docker"     'docker version'

The rest of this post walks through the interesting parts of that output one
section at a time.

Where I started: two machines, and a rule I broke immediately

The setup is two machines. A MacBook Pro is the control machine — for writing,
editing, and opening SSH sessions. An NVIDIA DGX Spark is the workhorse — where
every model and every GPU command actually runs. I reach it over SSH as spark.

A theme for this whole series is that I verify facts with a command instead of
assuming them — even the obvious ones. So rather than start by stating what my
machines are, I'll show them. First the control machine:

# On the control MacBook
uname -m && sysctl -n machdep.cpu.brand_string

x86_64
Intel(R) Core(TM) i5-1038NG7 CPU @ 2.00GHz

An Intel x86_64 Mac. This matters more than it looks, because it's a
different architecture from the Spark — so I want it on the record, not assumed.
Now the Spark:

ssh spark 'uname -a'

Linux spark-66b9 6.17.0-1026-nvidia ... aarch64 aarch64 aarch64 GNU/Linux

The Spark is aarch64 — ARM64, the same CPU family as phones and Apple
Silicon, but a different architecture from my Intel Mac.

Machine	Role	Architecture	Verified by
MacBook Pro	control / authoring	x86_64 (Intel i5)	uname -m
DGX Spark	model + GPU work	aarch64 (ARM64)	uname -a

This is not trivia. Because the two machines are different architectures, a
Python wheel or Docker image built for my Intel Mac will not necessarily run on
the ARM64 Spark. That is exactly why all the real work in this series happens
over ssh spark, on the box itself — and why "it works on my laptop" means
nothing here.

CPU versus GPU, and why models love the GPU

The Spark's CPU is a 20-core ARM chip:

ssh spark 'lscpu'

Architecture: aarch64
CPU(s):       20
Model name:   Cortex-X925   (10 performance cores)
Model name:   Cortex-A725   (10 efficiency cores)

Twenty cores sounds like a lot, but for running a model the CPU is mostly a
traffic director: it runs the OS, my Python, and the data loading. The heavy
lifting happens on the GPU. Here's the intuition that finally made "why a GPU"
click for me.

A neural-network layer boils down to one operation repeated endlessly: multiply
a big grid of numbers (the model's weights, $W$ ) by a list of numbers (the
input, $x$ ) and add them up — a matrix multiplication. One output value is:

$$y_i = \sum_j W_{ij}\, x_j$$

A model with billions of parameters does billions of these multiply-adds for a
single token. The magic property is that they're independent: computing
$y_1$ doesn't need $y_2$ . In terms I already know, that's embarrassingly
parallel — like the map phase of a MapReduce where no shard waits on another.

That's the whole reason a GPU wins:

Aspect	CPU (the 20 ARM cores)	GPU (the NVIDIA GB10)
Parallel workers	a few strong cores	thousands of small cores
Good at	branching logic, one-at-a-time	the same math on huge data at once
Analogy	a few expert chefs	a stadium of line cooks

A CPU is a handful of very smart workers doing tasks in sequence. A GPU is
thousands of simpler workers all doing the identical multiply-add on different
numbers simultaneously. Since a model is nothing but that identical operation
repeated, the GPU is the right tool.

One caveat I'm carrying forward: those cores are useless if you can't feed
them numbers fast enough, so memory bandwidth — not raw compute — usually
limits how fast a model runs. I'll test the CPU-vs-GPU speed difference directly
with a matmul benchmark once PyTorch is installed; my prediction is a 10x–50x
GPU speedup, with a slow first GPU run due to one-time warmup.

Reading nvidia-smi: my new top for the GPU

nvidia-smi is the command I'll run every day from now on. It's the GPU
equivalent of top or docker stats. Here's the real output, lightly trimmed:

ssh spark 'nvidia-smi'

NVIDIA-SMI 580.159.03   Driver Version: 580.159.03   CUDA Version: 13.0
GPU 0: NVIDIA GB10   Persistence-M: On
Temp  Perf  Pwr:Usage/Cap   Memory-Usage    GPU-Util  Compute M.
35C   P8    4W / N/A        Not Supported   0%        Default

Field by field, and why each one will matter later:

Field	Value	Meaning
Driver Version	580.159.03	kernel driver talking to the GPU
CUDA Version	13.0	max CUDA the driver supports
GPU name	NVIDIA GB10	the device (Grace-Blackwell)
Temp	35C	die temperature (heat → throttling)
Perf	P8	clock state, P0 = max … P8 = idle
Pwr:Usage/Cap	4W / N/A	current vs max power draw
Memory-Usage	Not Supported	would show VRAM used/total — see below
GPU-Util	0%	% of last sample the GPU was busy

The bottom of the output also has a process table listing every PID holding the
GPU — right now just Xorg, GNOME, and Firefox using it for the desktop. That's
my first stop whenever I hit "out of memory": find the offender, like lsof on
a stuck port.

For Week 1 I'm using nvidia-smi purely as an inventory tool — what GPU,
what driver, what max CUDA, who's using it. The deeper use (streaming monitors,
reading utilization and memory bandwidth to decide if a workload is
compute-bound or memory-bound) is a profiling skill I'm deliberately saving for
the CUDA track around Week 5, so I don't tangle the two learning tracks.

The surprise: unified memory, and a blank that isn't a bug

The one field that stopped me was Memory-Usage: Not Supported. On a normal PC
with a discrete GPU, that column is how you answer "did my model fit? how much
VRAM is left?" On the Spark it's blank — and that's not a bug, it's the whole
point of the machine.

A traditional GPU has its own separate memory (VRAM), physically distinct from
system RAM. The DGX Spark's GB10 is a Grace-Blackwell superchip that fuses the
ARM CPU and the GPU onto one package and gives them one shared memory pool:

ssh spark 'free -h'

               total   used   free   available
Mem:           121Gi   6.2Gi  67Gi   115Gi

Traditional discrete GPU	DGX Spark (GB10)
System RAM (e.g. 64 GB) + separate VRAM (e.g. 24 GB)	one shared 121 GiB pool
Copy data RAM → VRAM over PCIe	CPU and GPU read the same memory
"Will it fit?" limited by VRAM (24 GB)	limited by total RAM (121 GB)

This is unified memory. nvidia-smi reports "Not Supported" for GPU memory
because there is no separate VRAM to report — the GPU's memory is the system's
121 GiB. In infra terms, a discrete GPU pays a "copy tax" moving weights across
the PCIe bus, like shuffling data between two services with separate caches;
unified memory removes that hop, like two services sharing one in-memory cache.

The practical consequence for me: on the Spark, the ceiling on model size isn't
a stingy 24 GB of VRAM — it's 121 GB. But I have to track model memory
differently, via free -h or PyTorch's own counters, not the nvidia-smi
memory column. The trade-off (which I'll measure later) is that shared memory
usually has lower peak bandwidth than a top-end discrete card's dedicated VRAM,
so the Spark trades some raw speed for the ability to fit much larger models.

Driver vs CUDA runtime vs CUDA toolkit

The most confusing part of the NVIDIA stack for a newcomer is that "CUDA" isn't
one thing — it's three separate layers, installed and versioned independently.
Mapping each to infrastructure I already understand finally made it stick:

Layer	What it is	Infra analogy	Who needs it
NVIDIA driver	kernel module that talks to the GPU	a device driver	everyone using the GPU
CUDA runtime (libcudart)	shared libs an app calls to run GPU work	the .so libs you link	anyone running GPU programs
CUDA toolkit	the nvcc compiler, headers, profilers	gcc + headers + build tools	only people compiling CUDA

The insight that unblocked me: you can run GPU code without the toolkit.
PyTorch ships its own copy of the CUDA runtime inside its wheel. So to run
models I need the driver (system-level) plus a CUDA-enabled PyTorch (which
brings its own runtime). I do not need nvcc — that's only for compiling
custom CUDA kernels, a much-later CUDA-track activity.

With that lens, two clues from the inventory make sense. First, the header line
CUDA Version: 13.0 is the maximum CUDA the driver supports — a ceiling, not
what's installed. That's the number that matters this week: when I install
PyTorch, I must pick a CUDA build ≤ 13.0 so the driver can run it.

Second, the alarming-looking one:

ssh spark 'nvcc --version'

bash: nvcc: command not found

This looks broken but isn't. It only means the toolkit's compiler isn't on my
PATH. The driver and runtime clearly work — nvidia-smi talks to the GPU. And
the toolkit is physically installed:

ssh spark 'ls -d /usr/local/cuda*'

/usr/local/cuda  /usr/local/cuda-13  /usr/local/cuda-13.0

So nvcc exists; it's just at /usr/local/cuda/bin/nvcc, not on PATH. Rather
than assert that, I confirmed it by calling the full path:

ssh spark '/usr/local/cuda/bin/nvcc --version'

Cuda compilation tools, release 13.0, V13.0.88

The toolkit is version 13.0.88, matching the driver's CUDA 13.0 ceiling — a
healthy, consistent stack. Nothing is broken; command not found was a PATH
issue, not a missing install. If I ever want nvcc on PATH, it's one line:
export PATH=/usr/local/cuda/bin:$PATH. But to run models, I never need it.

Installing PyTorch, the right way: a virtual environment

PyTorch is the Python library I'll use to talk to the GPU. Before installing it,
one habit worth keeping: never install into the system Python. I use a
virtual environment (venv) — an isolated per-project Python with its own
packages, exactly like a per-service dependency sandbox so one project's
libraries can't break another's. On the Spark that's why the earlier
python3 -c "import torch" failed: the system Python genuinely has no torch, and
I want to keep it that way.

If you're following along, first check whether you already have PyTorch — many
setups ship with it:

ssh spark 'python3 -c "import torch, sys; print(torch.__version__)" \
  || echo "no torch in this Python"'

I didn't, so I made a clean venv and installed torch into it:

ssh spark 'python3 -m venv ~/venvs/w1'
ssh spark '~/venvs/w1/bin/python -m pip install --upgrade pip'
ssh spark '~/venvs/w1/bin/python -m pip install torch numpy'

The install is packaged as setup.sh in the companion repo, which just runs the
three steps above against requirements.txt:

#!/usr/bin/env bash
set -euo pipefail
VENV="{% katex inline %}{VENV:-{% endkatex %}HOME/venvs/w1}"
HERE="{% katex inline %}(cd "{% endkatex %}(dirname "$0")" && pwd)"
python3 -m venv "$VENV"
"$VENV/bin/python" -m pip install --upgrade pip
"{% katex inline %}VENV/bin/python" -m pip install -r "{% endkatex %}HERE/requirements.txt"

The interesting part is what pip pulled in. Remember the stack tops out at CUDA
13.0 — and without me specifying any special index, PyPI served an ARM64 +
CUDA 13 build automatically:

Successfully installed torch-2.13.0 nvidia-cuda-runtime-13.0.96
  nvidia-cudnn-cu13-9.20.0.48 nvidia-cublas-13.1.1.3 ... (aarch64 wheels)

This is the payoff of the "PyTorch ships its own CUDA runtime" point from
earlier: I did not install the CUDA toolkit or touch nvcc. Torch brought
its own CUDA 13 runtime libraries (libcudart, cuDNN, cuBLAS) as ordinary
Python wheels, matched to the ARM64 architecture and the driver's CUDA 13
ceiling. The driver was the only piece I needed pre-installed.

Proving the GPU actually works from Python

Now the moment this whole week builds to: does Python see the GPU? The
validation script (validate_gpu.py in the repo) is deliberately tiny:

import torch

print("PyTorch:", torch.__version__)
print("CUDA available:", torch.cuda.is_available())
print("CUDA version (torch):", torch.version.cuda)

if torch.cuda.is_available():
    print("Device:", torch.cuda.get_device_name(0))
    print("Capability:", torch.cuda.get_device_capability(0))
    x = torch.rand(3, 3, device="cuda")   # put a real tensor on the GPU
    print("Tensor on:", x.device)

Run it with the venv's Python:

ssh spark '~/venvs/w1/bin/python validate_gpu.py'

PyTorch: 2.13.0+cu130
CUDA available: True
CUDA version (torch): 13.0
Device: NVIDIA GB10
Capability: (12, 1)
Tensor on: cuda:0

Every line matters. CUDA available: True means PyTorch found a usable GPU
through the driver. 2.13.0+cu130 confirms it's a CUDA 13 build. Device: NVIDIA
GB10 is our chip. Capability: (12, 1) is the GPU's compute capability —
NVIDIA's versioning for GPU features; 12.x is the Blackwell generation. And
Tensor on: cuda:0 is the real proof: I created a tensor and it physically lives
in GPU memory, not on the CPU. This is the line that says "I can run models on
this machine now."

Testing the prediction: CPU vs GPU

Earlier I predicted the GPU would beat the CPU by 10x-50x on a large matrix
multiply. Time to measure instead of hand-wave. The benchmark
(benchmark.py) multiplies two 4096x4096 matrices 20 times on each device and
averages. Two details make the GPU timing honest: a warmup (the first GPU
call pays a one-time kernel-load cost, so I run a few throwaway iterations
first), and torch.cuda.synchronize() before stopping the clock (CUDA launches
kernels asynchronously, so without a sync I'd be timing queueing, not
computing):

def bench(a, b, sync=False):
    if sync: torch.cuda.synchronize()
    t0 = time.perf_counter()
    for _ in range(ITERS):
        c = a @ b
    if sync: torch.cuda.synchronize()
    return (time.perf_counter() - t0) / ITERS

cpu_s = bench(a, b)                       # on CPU
ag, bg = a.cuda(), b.cuda()
for _ in range(3): _ = ag @ bg            # warmup
gpu_s = bench(ag, bg, sync=True)          # on GPU

ssh spark '~/venvs/w1/bin/python benchmark.py'

Matrix: 4096x4096 float32, iters: 20
CPU  ms/matmul: 173.8   GFLOP/s: 790.8
GPU  ms/matmul: 7.53    GFLOP/s: 18263.5
Speedup (CPU/GPU): 23.1 x

A single 4096x4096 float32 matmul is about 137 billion floating-point
operations. The 20-core ARM CPU chews through it in ~174 ms (~0.79 TFLOP/s); the
GB10 does it in ~7.5 ms (~18.3 TFLOP/s) — a 23x speedup, squarely inside my
predicted 10x-50x range. That number is the reason this hardware exists: the
same math, done thousands-at-a-time instead of a-few-at-a-time. (These are the
raw values in results.json in the repo; yours will differ by machine.)

What surprised me

• A GPU with no VRAM number is a feature, not a fault. Unified memory reframes "will it fit?" from ~24 GB to 121 GB.
• nvcc: command not found is not an error state — the CUDA that matters for running models lives inside PyTorch, not in the toolkit. Installing torch pulled its own CUDA 13 runtime as plain wheels; I never touched the toolkit.
• Verifying every fact with a command — even the "obvious" ones — already caught assumptions I would otherwise have carried into later weeks.
• The GPU's 23x matmul win was real and measurable on day one, not a slide.

What I'll do next

Week 1 is done: I know the machine, I understand the NVIDIA stack, and I've proven
the GPU is usable from Python with a real speedup. Week 2 leaves inventory behind
and runs an actual language model with Ollama — a model runtime with a local
HTTP API — where I'll start measuring the things that matter for serving: tokens
per second and time to first token.

---

1. 32-week roadmap —
   https://github.com/dramasamy/from-api-to-gpu/tree/main/roadmap ↩

2. Companion code repository —
   https://github.com/dramasamy/from-api-to-gpu ↩

3. Week 1 lab —
   https://github.com/dramasamy/from-api-to-gpu/tree/main/week-01-environment ↩