The same 64 GB USB stick answered a question in 5 seconds on my Mac, once the model was loaded. On a Windows laptop with a Celeron, the same question, from the same stick, with the same model, took more than 6 minutes.
That was expected. What I didn't expect was the gap between two virtual machines on the same mini PC: one answered in 32 seconds, the other in 2 minutes and 40 seconds. Same processor, same 4 cores, same 8 GB of RAM. The most likely cause was a single dropdown in the VM settings.
A few weeks ago I ran AI offline for 7 days on a mini PC that stayed at home. This time I wanted the opposite: an offline AI on a USB stick, with the runner and the models on the stick, that I could plug into any computer and use without installing anything.
The results first
| Machine | Model | Ready | Answer speed | Answer time |
|---|---|---|---|---|
| Mac M3 | Llama 3B | 29 s | 39 tokens/s | 5 s |
| Mac M3 | Qwen3 8B | 117 s | 17.6 tokens/s | 12 s |
| Kali VM | Llama 3B | 39 s | 7.1 tokens/s | 32 s |
| Mint VM | Llama 3B | 41 s | 1.5 tokens/s | 2 min 40 s |
| Mint VM | Qwen3 8B | 85 s | 0.67 tokens/s | 5 min 46 s |
| Celeron laptop | Llama 3B | 69 s | 0.6 tokens/s | 6 min 22 s |
| Celeron laptop | Qwen3 8B | crashed | n/a | n/a |
Answer time is for the same question on every machine, once the model was loaded; ready time is how long the model took to load from the stick. Running with no internet at all (Wi-Fi off) was confirmed on the Mac with the 3B model, at the same speed. How I measured is further down.
So, does it run anywhere? The 3B model started on all four systems without installing anything, but it was only comfortable to use on two of them. The 8B model crashed on the Celeron. On the Mac it feels like a normal chat. On the Kali VM it's slow but usable. On the Mint VM and the Celeron it works the way a fax works.
This post covers:
- What goes on the stick, and why each choice (file system, runner, models) matters.
- How to build the same stick, step by step, with checksums.
- How I measured, on macOS, Windows, Linux Mint and Kali.
- Four things the numbers taught me, including that VM setting.
- What "no installation" really means, and when this is a bad idea.
What goes on the offline AI USB stick
The idea only works if three things travel together: a runner that starts on every operating system, model files every system can read, and a file system all of them can write to.
| Piece | My choice | Why |
|---|---|---|
| Stick | Kingston, 64 GB | What I had. About 7.8 GB ended up used |
| File system | exFAT | Reads and writes on macOS, Windows and Linux, and allows files over 4 GB |
| Runner | llamafile 0.10.6 (368 MB) | One executable for macOS, Windows and Linux |
| Small model | Llama 3.2 3B Instruct, Q4_K_M (2.0 GB) | Fast enough for weak CPUs |
| Medium model | Qwen3 8B, Q4_K_M (5.0 GB) | Better answers, to find each machine's limit |
Why exFAT. FAT32 can't store a file over 4 GB, so the Qwen model wouldn't fit. macOS can read NTFS but not write to it without extra software. exFAT is in Apple's list of Disk Utility formats, Windows supports it natively, and Linux has had a native exFAT driver in the kernel since 5.7. Any current Ubuntu, Mint or Kali has it.
Why llamafile. llamafile is a Mozilla project that combines llama.cpp with Cosmopolitan Libc, so a single file runs "on most operating systems and CPU architectures, with no installation". Its supported systems include macOS, Windows 10+, Linux and the BSDs.
Why the models are separate files. llamafile can bundle a model inside the executable, but its README warns that "only executables under 4GB can run on Windows". With the weights as separate .gguf files, the 368 MB runner stays under that limit, so the same model files can be used on every system. Whether a model fits a given machine is another question, as the Celeron showed.
Build the same stick, step by step
You need at least 8 GB free on the stick and about 7.5 GB on your computer for the downloads. Format the stick as exFAT first (Disk Utility on macOS, "Format" in Windows Explorer). I named mine Offline AI.
Step 1: Download the runner and the models
These commands are for macOS or Linux. They download to a folder on your disk first, so you can verify the files before they touch the stick:
mkdir -p ~/offline-ai/llamafile ~/offline-ai/models && cd ~/offline-ai
curl -fL -o llamafile/llamafile-0.10.6 \
https://github.com/mozilla-ai/llamafile/releases/download/0.10.6/llamafile-0.10.6
curl -fL -o models/Llama-3.2-3B-Instruct-Q4_K_M.gguf \
https://huggingface.co/bartowski/Llama-3.2-3B-Instruct-GGUF/resolve/main/Llama-3.2-3B-Instruct-Q4_K_M.gguf
curl -fL -o models/Qwen3-8B-Q4_K_M.gguf \
https://huggingface.co/unsloth/Qwen3-8B-GGUF/resolve/main/Qwen3-8B-Q4_K_M.gguf
Step 2: Check the files before you trust them
A stick that travels between computers carries an executable you will run on all of them. Before copying anything, compare each file with the SHA-256 published by its source (the GitHub release page and the Hugging Face file pages). These are the values I verified:
d579f61dcd3a306f518e6d90e599d77793ed5f09543023d09c96ad35fcfa63f0 llamafile/llamafile-0.10.6
6c1a2b41161032677be168d354123594c0e6e67d2b9227c84f296ad037c728ff models/Llama-3.2-3B-Instruct-Q4_K_M.gguf
120307ba529eb2439d6c430d94104dabd578497bc7bfe7e322b5d9933b449bd4 models/Qwen3-8B-Q4_K_M.gguf
Save them as SHA256SUMS and run shasum -a 256 -c SHA256SUMS (macOS) or sha256sum -c SHA256SUMS (Linux). Every line must say OK.
Step 3: Copy to the stick, add the Windows copy, check again
STICK="/Volumes/Offline AI" # on Linux: /media/$USER/Offline AI
cp -R llamafile models SHA256SUMS "$STICK/"
cp "$STICK/llamafile/llamafile-0.10.6" "$STICK/llamafile/llamafile.exe"
echo "d579f61dcd3a306f518e6d90e599d77793ed5f09543023d09c96ad35fcfa63f0 llamafile/llamafile.exe" >> "$STICK/SHA256SUMS"
cd "$STICK" && shasum -a 256 -c SHA256SUMS # on Linux: sha256sum -c SHA256SUMS
The echo line adds the Windows copy to the checksum list with the same hash as the verified original, so the check covers the file you will actually run on Windows. On Windows itself, Get-FileHash .\llamafile\llamafile.exe -Algorithm SHA256 must print the same value.
The .exe copy is the same file under another name, because Windows only runs files that end in .exe (the README says to rename it; a second copy keeps the stick usable everywhere). Checking again on the stick catches a bad copy.
Keep the checksums somewhere other than the stick. If someone swaps the executable, they can rewrite SHA256SUMS next to it too. A copy in a repository, a note, or this article is the reference that can't be swapped along with the file.
Step 4: Talk to it
On macOS or Linux, from the stick:
./llamafile/llamafile-0.10.6 -m models/Llama-3.2-3B-Instruct-Q4_K_M.gguf
On Windows, in PowerShell from the stick:
.\llamafile\llamafile.exe -m .\models\Llama-3.2-3B-Instruct-Q4_K_M.gguf
I ran these two commands as written on the Mac and on Windows. On Linux and macOS, if you get Exec format error or a WINE/run-detectors message, start it through the shell: sh ./llamafile/llamafile-0.10.6 -m models/Llama-3.2-3B-Instruct-Q4_K_M.gguf. The troubleshooting page mentions this for older zsh, older Fish and Python. My benchmark script hit it: starting llamafile from Python's subprocess failed with Exec format error until I ran it through sh.
Without --server, --chat or --cli, llamafile starts a chat in the terminal and a server at http://localhost:8080, with a web chat page inherited from llama.cpp. My server log also warned that CORS allowed all origins and no API key was set. If you only want the terminal chat, --chat runs it without the server.
How I measured
To compare machines, I wrote a small benchmark script (Python for macOS and Linux, PowerShell for Windows). It starts llamafile in server mode, times how long it takes to be ready, then asks the same three questions: summarize a paragraph with numbers, fix a retry function, and explain HTTP 429.
| Runner | llamafile 0.10.6 |
| Models | Llama 3.2 3B Instruct Q4_K_M, Qwen3 8B Q4_K_M |
| Settings | 4,096 context, temperature 0, seed 42, up to 200 answer tokens, Qwen3 thinking off |
| Runs | 3 per question (1 on Kali) |
| When | November 17, 2026 |
| Type | Controlled test, not real use |
| Machine | CPU | RAM | GPU used |
|---|---|---|---|
| Mac | Apple M3 | 16 GB | Yes, Metal |
| Windows laptop | Intel Celeron N4000, 2 cores | 8 GB | Not confirmed (see Lesson 3) |
| Linux Mint VM | 4 vCPUs of a Ryzen 3 3300U (mini PC, Proxmox) | 8 GB | No (no GPU passed through) |
| Kali VM | 4 vCPUs of the same Ryzen 3 3300U | 8 GB | No (no GPU passed through) |
Ready time counts from starting the program until the server answers its health check, with the stick freshly plugged in. The VMs got the stick through Proxmox USB passthrough. Answer time is for the HTTP 429 question (58 prompt tokens for Llama, 37 for Qwen, and 200 answer tokens), calculated from llamafile's own timings and taken as the median of the runs. It is close to what you feel when you ask something and wait. Speeds are medians too.
Only one run was offline: a repeat of the 3B model on the Mac with Wi-Fi off (the script records whether the machine could reach the internet), which ran at the same speed, about 40 tokens/s. The numbers in the results table were measured with the network on; the VMs stayed online because I manage them over the network. Nothing in the setup uses the network, but I only proved it with that one run.
Lesson 1: on my Mac, the stick only slowed the start
I ran the same models from the Mac's internal SSD for comparison:
| Model | Ready from SSD | Ready from stick | Speed from SSD | Speed from stick |
|---|---|---|---|---|
| Llama 3B | 2.1 s | 29 s | 34 tokens/s | 39 tokens/s |
| Qwen3 8B | 8.7 s | 117 s | 16.8 tokens/s | 17.6 tokens/s |
On my Mac, the USB stick added startup time. Once loaded, the models generated at similar speeds from USB and SSD. The model is read into memory at startup, and the answers are computed from memory.
Two caveats. The SSD runs were single runs and came out slightly slower, which I read as noise, not as the stick being faster. And they came right after I had downloaded and checked those files, so the system may have served part of them from its memory cache, which flatters the SSD. The direction is clear anyway. When the model was still in memory from a previous run on the stick, the Mac was ready in 1 second.
In this test, a faster stick would have shortened the start, not the answers.
Lesson 2: the VM that could see AVX2 was five times faster
The Mint and Kali VMs run on the same mini PC, with the same 4 vCPUs and 8 GB each. Kali answered in 32 seconds, Mint in 2 minutes and 40 seconds. Reading the prompt was 9 times faster on Kali (17.7 vs 1.9 tokens per second).
I looked at the Proxmox configuration of both:
LinuxMint cores: 4 cpu: x86-64-v2-AES memory: 8192
KaliLinux2025 cores: 4 cpu: host memory: 8192
x86-64-v2-AES is the default CPU type when you create a VM in the Proxmox web UI. It presents a generic processor so the VM can move between hosts with different CPUs. The x86-64-v2 level stops before AVX, AVX2 and FMA, which arrive in x86-64-v3. The Proxmox docs list AVX, AVX2, FMA and F16C among the flags x86-64-v3 adds over x86-64-v2-AES. The Ryzen 3 3300U has AVX2 and FMA, but the Mint VM never sees them: its CPU type hides AVX, AVX2 and FMA. With host, Kali sees the real processor.
llamafile's support page says it enables AVX2, FMA, F16C and similar instructions at runtime when the CPU has them. When the VM hides them, those faster paths are off.
My caveat: this is the likely cause, not a proven one. Mint and Kali also differ in distribution and kernel. A clean test would switch the Mint VM to host and run it again, and I didn't. Still, a 9x gap in prompt reading on identical hardware, with the faster VM being the one that can see AVX2, points in one direction.
You can check your own VM in one line:
grep -o -w -E 'avx2|fma' /proc/cpuinfo | sort | uniq -c
No output means the VM can't see them. To expose the host CPU flags in Proxmox, set the CPU type to host with qm set <vmid> --cpu host, or --cpu x86-64-v3 if you need live migration across hosts that all support v3. Then run qm reboot <vmid> (or stop and start the VM) so the change takes effect. The trade-off is real: with host, migrating the VM to a node with a different CPU can fail, which is why Proxmox doesn't make it the default.
Lesson 3: where the floor is
The Celeron N4000 is a 2-core, low-power laptop chip. It ran the 3B model at 0.6 tokens per second, which means a 200-token answer takes more than 6 minutes. The Qwen3 8B model started loading and then the process ended with Terminating on uncaught SIGSEGV. The log shows llamafile had switched to GPU mode, probably trying the Celeron's Intel UHD 600, and the Mint VM ran the same model in 8 GB, so I don't know whether memory or the GPU path caused it. Running it again with --gpu disable would tell, and I haven't.
Something I can't explain: llamafile's support page says AMD64 processors "must have AVX" or llamafile "will print an error and refuse to run". The N4000's feature list on Notebookcheck goes up to SSE4.2, with no AVX. It ran anyway, slowly. The Mint VM is a second case: its CPU type exposes no AVX at all, and llamafile ran there too. My guess is that the 0.10 rebuild changed this and the page wasn't updated, but that is a guess.
The Celeron and the Mint VM both ran the 3B model without AVX. On both, the benchmark answer took minutes (6 min 22 s and 2 min 40 s). My slowest machine that was still usable had AVX2, 4 cores and 8 GB.
Lesson 4: small models can be confidently wrong
Speed is only half of it. I kept every answer, and the 3B model made two mistakes worth showing.
Asked to fix a retry function with exponential backoff, Llama 3.2 3B wrote this and called it exponential:
wait_time = min(10, attempt * 2) # exponential backoff
wait_time *= random.uniform(0.8, 1.2) # add jitter
attempt * 2 grows linearly (0, 2, 4, 6 seconds), and the first retry waits zero seconds. Qwen3 8B got it right with initial_delay * (2 ** attempt). Asked what HTTP 429 means, Llama listed "Session expiration" as a cause. RFC 6585 defines 429 as rate limiting: the client sent too many requests in a given amount of time.
The bigger model wasn't immune either. When I chatted with Qwen3 8B directly and asked for the capital of Portugal, it first wrote a paragraph of reasoning to itself, then answered Lisbon correctly, then added a "fun fact": "Lisbon has been the capital since the 12th century, making it one of Europe's oldest continuously inhabited cities." Lisbon became the capital in the 13th century (1255 or 1256, depending on the source), and being the capital has nothing to do with being continuously inhabited.
That reasoning paragraph is Qwen3's thinking mode, which is on by default. The model card documents two ways to turn it off: enable_thinking=False in the chat template (what my benchmark sends through the API), or, according to the model card, adding /no_think to a message, which switches it off for that turn while thinking mode is on.
Offline, there is nobody to check the answer but you. Use these models for drafts, summaries of text you give them, and code you will read anyway. Don't use them as a source of facts.
What "no installation" really means
I installed nothing on any machine. That doesn't mean nothing was written, or that nothing was needed:
- On Unix systems, llamafile extracts a small loader named
apeto$TMPDIR. - On the Mac, the first run wrote the Metal GPU support files to
~/.llamafile(about 3 MB in my case). The docs say the GPU path on Apple Silicon needs the Xcode Command Line Tools, which my Mac already had.
If you need a machine to keep nothing, delete ~/.llamafile and $TMPDIR/.ape-1.10 afterwards.
When this is a bad idea
| Situation | Why |
|---|---|
| A company laptop | Running an unsigned executable from a USB stick may break your company's policy, and the docs note that endpoint protection such as CrowdStrike can kill the process |
| You have internet and need correct facts | A hosted model is far more capable than a 3B or 8B model on a laptop CPU |
| A CPU (or VM) without AVX | On my Celeron and Mint VM it started, but the benchmark answer took minutes |
| A Mac without the Xcode Command Line Tools | The Metal GPU path needs them, so expect CPU speed or a failed start (I didn't test this) |
| A shared stick you don't control | Verify the checksums against a copy you keep elsewhere before every use |
Where it does make sense: a trip without reliable internet, a machine you can't install anything on, working with text you can't send to an external service, or a demo that must work without Wi-Fi.
FAQ
Why not Ollama? Ollama is a separate build per operating system, and its normal setup is an installer or a service. You could carry its standalone binaries for each system, but llamafile is one file that runs on all of them.
Which model should I put on the stick? For most laptops without a strong GPU, a 3B model in Q4. Add an 8B model only if the machines you'll use have 16 GB of RAM and a recent CPU or an Apple Silicon GPU.
Can it use a GPU on Windows or Linux? According to the support page, yes for NVIDIA, AMD and Vulkan-capable GPUs, with Windows marked as best-effort in the 0.10 series. My VMs had no GPU passed through, and I didn't test GPU offload on Windows.
Takeaway
-
The 3B model ran on all four systems; the 8B model crashed on the Celeron. exFAT plus llamafile plus separate
.gguffiles, nothing installed (the Mac already had the Xcode Command Line Tools). - On my Mac, the stick only slowed the start. 29 s versus 2 s to load the 3B model, and a similar answer speed once loaded (39 vs 34 tokens/s).
- Check your VM's CPU type before blaming the model. Same mini PC, 32 seconds versus 2 minutes 40 seconds, and the slow VM couldn't see AVX or AVX2.
- Without AVX, it started but answers took minutes. The Celeron and the Mint VM both ran the 3B model without AVX: 6 min 22 s and 2 min 40 s per answer.
- Small models can be confidently wrong. Read the code, don't trust the facts.
What would you use an AI on a USB stick for: travel, a machine you can't install on, or something I haven't thought of?
I'm Diogo Soares, a Staff AI Engineer working on AI agents and legacy modernization. If this was useful, follow me here on DEV.to and on Instagram at @diogosoares.dev, where every experiment becomes a Reel of under a minute. Every article also gets a video walkthrough on my YouTube channel, Diogo Soares Dev, so subscribe there for the next ones. Let's also connect on LinkedIn, or check out diogosoares.dev.
Top comments (0)