DEV Community

Diogo Soares
Diogo Soares

Posted on Edited on

I Put an Offline AI on a USB Stick. Does It Run Anywhere?

The same 64 GB USB stick answered a question in 5 seconds on my Mac, once the model was loaded. On a Windows laptop with a Celeron, the same question, from the same stick, with the same model, took more than 6 minutes.

That was expected. What I didn't expect was the gap between two virtual machines on the same mini PC: one answered in 32 seconds, the other in 2 minutes and 40 seconds. Same processor, same 4 cores, same 8 GB of RAM. The most likely cause was a single dropdown in the VM settings.

A few weeks ago I ran AI offline for 7 days on a mini PC that stayed at home. This time I wanted the opposite: an offline AI on a USB stick, with the runner and the models on the stick, that I could plug into any computer and use without installing anything.

The results first

Machine Model Ready Answer speed Answer time
Mac M3 Llama 3B 29 s 39 tokens/s 5 s
Mac M3 Qwen3 8B 117 s 17.6 tokens/s 12 s
Kali VM Llama 3B 39 s 7.1 tokens/s 32 s
Mint VM Llama 3B 41 s 1.5 tokens/s 2 min 40 s
Mint VM Qwen3 8B 85 s 0.67 tokens/s 5 min 46 s
Celeron laptop Llama 3B 69 s 0.6 tokens/s 6 min 22 s
Celeron laptop Qwen3 8B crashed n/a n/a

Answer time is for the same question on every machine, once the model was loaded; ready time is how long the model took to load from the stick. Running with no internet at all (Wi-Fi off) was confirmed on the Mac with the 3B model, at the same speed. How I measured is further down.

So, does it run anywhere? The 3B model started on all four systems without installing anything, but it was only comfortable to use on two of them. The 8B model crashed on the Celeron. On the Mac it feels like a normal chat. On the Kali VM it's slow but usable. On the Mint VM and the Celeron it works the way a fax works.

This post covers:

  1. What goes on the stick, and why each choice (file system, runner, models) matters.
  2. How to build the same stick, step by step, with checksums.
  3. How I measured, on macOS, Windows, Linux Mint and Kali.
  4. Four things the numbers taught me, including that VM setting.
  5. What "no installation" really means, and when this is a bad idea.

What goes on the offline AI USB stick

The idea only works if three things travel together: a runner that starts on every operating system, model files every system can read, and a file system all of them can write to.

Piece My choice Why
Stick Kingston, 64 GB What I had. About 7.8 GB ended up used
File system exFAT Reads and writes on macOS, Windows and Linux, and allows files over 4 GB
Runner llamafile 0.10.6 (368 MB) One executable for macOS, Windows and Linux
Small model Llama 3.2 3B Instruct, Q4_K_M (2.0 GB) Fast enough for weak CPUs
Medium model Qwen3 8B, Q4_K_M (5.0 GB) Better answers, to find each machine's limit

Why exFAT. FAT32 can't store a file over 4 GB, so the Qwen model wouldn't fit. macOS can read NTFS but not write to it without extra software. exFAT is in Apple's list of Disk Utility formats, Windows supports it natively, and Linux has had a native exFAT driver in the kernel since 5.7. Any current Ubuntu, Mint or Kali has it.

Why llamafile. llamafile is a Mozilla project that combines llama.cpp with Cosmopolitan Libc, so a single file runs "on most operating systems and CPU architectures, with no installation". Its supported systems include macOS, Windows 10+, Linux and the BSDs.

Why the models are separate files. llamafile can bundle a model inside the executable, but its README warns that "only executables under 4GB can run on Windows". With the weights as separate .gguf files, the 368 MB runner stays under that limit, so the same model files can be used on every system. Whether a model fits a given machine is another question, as the Celeron showed.


Build the same stick, step by step

You need at least 8 GB free on the stick and about 7.5 GB on your computer for the downloads. Format the stick as exFAT first (Disk Utility on macOS, "Format" in Windows Explorer). I named mine Offline AI.

Step 1: Download the runner and the models

These commands are for macOS or Linux. They download to a folder on your disk first, so you can verify the files before they touch the stick:

mkdir -p ~/offline-ai/llamafile ~/offline-ai/models && cd ~/offline-ai
curl -fL -o llamafile/llamafile-0.10.6 \
  https://github.com/mozilla-ai/llamafile/releases/download/0.10.6/llamafile-0.10.6
curl -fL -o models/Llama-3.2-3B-Instruct-Q4_K_M.gguf \
  https://huggingface.co/bartowski/Llama-3.2-3B-Instruct-GGUF/resolve/main/Llama-3.2-3B-Instruct-Q4_K_M.gguf
curl -fL -o models/Qwen3-8B-Q4_K_M.gguf \
  https://huggingface.co/unsloth/Qwen3-8B-GGUF/resolve/main/Qwen3-8B-Q4_K_M.gguf
Enter fullscreen mode Exit fullscreen mode

Step 2: Check the files before you trust them

A stick that travels between computers carries an executable you will run on all of them. Before copying anything, compare each file with the SHA-256 published by its source (the GitHub release page and the Hugging Face file pages). These are the values I verified:

d579f61dcd3a306f518e6d90e599d77793ed5f09543023d09c96ad35fcfa63f0  llamafile/llamafile-0.10.6
6c1a2b41161032677be168d354123594c0e6e67d2b9227c84f296ad037c728ff  models/Llama-3.2-3B-Instruct-Q4_K_M.gguf
120307ba529eb2439d6c430d94104dabd578497bc7bfe7e322b5d9933b449bd4  models/Qwen3-8B-Q4_K_M.gguf
Enter fullscreen mode Exit fullscreen mode

Save them as SHA256SUMS and run shasum -a 256 -c SHA256SUMS (macOS) or sha256sum -c SHA256SUMS (Linux). Every line must say OK.

Step 3: Copy to the stick, add the Windows copy, check again

STICK="/Volumes/Offline AI"   # on Linux: /media/$USER/Offline AI
cp -R llamafile models SHA256SUMS "$STICK/"
cp "$STICK/llamafile/llamafile-0.10.6" "$STICK/llamafile/llamafile.exe"
echo "d579f61dcd3a306f518e6d90e599d77793ed5f09543023d09c96ad35fcfa63f0  llamafile/llamafile.exe" >> "$STICK/SHA256SUMS"
cd "$STICK" && shasum -a 256 -c SHA256SUMS   # on Linux: sha256sum -c SHA256SUMS
Enter fullscreen mode Exit fullscreen mode

The echo line adds the Windows copy to the checksum list with the same hash as the verified original, so the check covers the file you will actually run on Windows. On Windows itself, Get-FileHash .\llamafile\llamafile.exe -Algorithm SHA256 must print the same value.

The .exe copy is the same file under another name, because Windows only runs files that end in .exe (the README says to rename it; a second copy keeps the stick usable everywhere). Checking again on the stick catches a bad copy.

Keep the checksums somewhere other than the stick. If someone swaps the executable, they can rewrite SHA256SUMS next to it too. A copy in a repository, a note, or this article is the reference that can't be swapped along with the file.

Step 4: Talk to it

On macOS or Linux, from the stick:

./llamafile/llamafile-0.10.6 -m models/Llama-3.2-3B-Instruct-Q4_K_M.gguf
Enter fullscreen mode Exit fullscreen mode

On Windows, in PowerShell from the stick:

.\llamafile\llamafile.exe -m .\models\Llama-3.2-3B-Instruct-Q4_K_M.gguf
Enter fullscreen mode Exit fullscreen mode

I ran these two commands as written on the Mac and on Windows. On Linux and macOS, if you get Exec format error or a WINE/run-detectors message, start it through the shell: sh ./llamafile/llamafile-0.10.6 -m models/Llama-3.2-3B-Instruct-Q4_K_M.gguf. The troubleshooting page mentions this for older zsh, older Fish and Python. My benchmark script hit it: starting llamafile from Python's subprocess failed with Exec format error until I ran it through sh.

Without --server, --chat or --cli, llamafile starts a chat in the terminal and a server at http://localhost:8080, with a web chat page inherited from llama.cpp. My server log also warned that CORS allowed all origins and no API key was set. If you only want the terminal chat, --chat runs it without the server.


How I measured

To compare machines, I wrote a small benchmark script (Python for macOS and Linux, PowerShell for Windows). It starts llamafile in server mode, times how long it takes to be ready, then asks the same three questions: summarize a paragraph with numbers, fix a retry function, and explain HTTP 429.

Runner llamafile 0.10.6
Models Llama 3.2 3B Instruct Q4_K_M, Qwen3 8B Q4_K_M
Settings 4,096 context, temperature 0, seed 42, up to 200 answer tokens, Qwen3 thinking off
Runs 3 per question (1 on Kali)
When November 17, 2026
Type Controlled test, not real use
Machine CPU RAM GPU used
Mac Apple M3 16 GB Yes, Metal
Windows laptop Intel Celeron N4000, 2 cores 8 GB Not confirmed (see Lesson 3)
Linux Mint VM 4 vCPUs of a Ryzen 3 3300U (mini PC, Proxmox) 8 GB No (no GPU passed through)
Kali VM 4 vCPUs of the same Ryzen 3 3300U 8 GB No (no GPU passed through)

Ready time counts from starting the program until the server answers its health check, with the stick freshly plugged in. The VMs got the stick through Proxmox USB passthrough. Answer time is for the HTTP 429 question (58 prompt tokens for Llama, 37 for Qwen, and 200 answer tokens), calculated from llamafile's own timings and taken as the median of the runs. It is close to what you feel when you ask something and wait. Speeds are medians too.

Only one run was offline: a repeat of the 3B model on the Mac with Wi-Fi off (the script records whether the machine could reach the internet), which ran at the same speed, about 40 tokens/s. The numbers in the results table were measured with the network on; the VMs stayed online because I manage them over the network. Nothing in the setup uses the network, but I only proved it with that one run.


Lesson 1: on my Mac, the stick only slowed the start

I ran the same models from the Mac's internal SSD for comparison:

Model Ready from SSD Ready from stick Speed from SSD Speed from stick
Llama 3B 2.1 s 29 s 34 tokens/s 39 tokens/s
Qwen3 8B 8.7 s 117 s 16.8 tokens/s 17.6 tokens/s

On my Mac, the USB stick added startup time. Once loaded, the models generated at similar speeds from USB and SSD. The model is read into memory at startup, and the answers are computed from memory.

Two caveats. The SSD runs were single runs and came out slightly slower, which I read as noise, not as the stick being faster. And they came right after I had downloaded and checked those files, so the system may have served part of them from its memory cache, which flatters the SSD. The direction is clear anyway. When the model was still in memory from a previous run on the stick, the Mac was ready in 1 second.

In this test, a faster stick would have shortened the start, not the answers.


Lesson 2: the VM that could see AVX2 was five times faster

The Mint and Kali VMs run on the same mini PC, with the same 4 vCPUs and 8 GB each. Kali answered in 32 seconds, Mint in 2 minutes and 40 seconds. Reading the prompt was 9 times faster on Kali (17.7 vs 1.9 tokens per second).

I looked at the Proxmox configuration of both:

LinuxMint      cores: 4  cpu: x86-64-v2-AES  memory: 8192
KaliLinux2025  cores: 4  cpu: host           memory: 8192
Enter fullscreen mode Exit fullscreen mode

x86-64-v2-AES is the default CPU type when you create a VM in the Proxmox web UI. It presents a generic processor so the VM can move between hosts with different CPUs. The x86-64-v2 level stops before AVX, AVX2 and FMA, which arrive in x86-64-v3. The Proxmox docs list AVX, AVX2, FMA and F16C among the flags x86-64-v3 adds over x86-64-v2-AES. The Ryzen 3 3300U has AVX2 and FMA, but the Mint VM never sees them: its CPU type hides AVX, AVX2 and FMA. With host, Kali sees the real processor.

llamafile's support page says it enables AVX2, FMA, F16C and similar instructions at runtime when the CPU has them. When the VM hides them, those faster paths are off.

My caveat: this is the likely cause, not a proven one. Mint and Kali also differ in distribution and kernel. A clean test would switch the Mint VM to host and run it again, and I didn't. Still, a 9x gap in prompt reading on identical hardware, with the faster VM being the one that can see AVX2, points in one direction.

You can check your own VM in one line:

grep -o -w -E 'avx2|fma' /proc/cpuinfo | sort | uniq -c
Enter fullscreen mode Exit fullscreen mode

No output means the VM can't see them. To expose the host CPU flags in Proxmox, set the CPU type to host with qm set <vmid> --cpu host, or --cpu x86-64-v3 if you need live migration across hosts that all support v3. Then run qm reboot <vmid> (or stop and start the VM) so the change takes effect. The trade-off is real: with host, migrating the VM to a node with a different CPU can fail, which is why Proxmox doesn't make it the default.


Lesson 3: where the floor is

The Celeron N4000 is a 2-core, low-power laptop chip. It ran the 3B model at 0.6 tokens per second, which means a 200-token answer takes more than 6 minutes. The Qwen3 8B model started loading and then the process ended with Terminating on uncaught SIGSEGV. The log shows llamafile had switched to GPU mode, probably trying the Celeron's Intel UHD 600, and the Mint VM ran the same model in 8 GB, so I don't know whether memory or the GPU path caused it. Running it again with --gpu disable would tell, and I haven't.

Something I can't explain: llamafile's support page says AMD64 processors "must have AVX" or llamafile "will print an error and refuse to run". The N4000's feature list on Notebookcheck goes up to SSE4.2, with no AVX. It ran anyway, slowly. The Mint VM is a second case: its CPU type exposes no AVX at all, and llamafile ran there too. My guess is that the 0.10 rebuild changed this and the page wasn't updated, but that is a guess.

The Celeron and the Mint VM both ran the 3B model without AVX. On both, the benchmark answer took minutes (6 min 22 s and 2 min 40 s). My slowest machine that was still usable had AVX2, 4 cores and 8 GB.


Lesson 4: small models can be confidently wrong

Speed is only half of it. I kept every answer, and the 3B model made two mistakes worth showing.

Asked to fix a retry function with exponential backoff, Llama 3.2 3B wrote this and called it exponential:

wait_time = min(10, attempt * 2)  # exponential backoff
wait_time *= random.uniform(0.8, 1.2)  # add jitter
Enter fullscreen mode Exit fullscreen mode

attempt * 2 grows linearly (0, 2, 4, 6 seconds), and the first retry waits zero seconds. Qwen3 8B got it right with initial_delay * (2 ** attempt). Asked what HTTP 429 means, Llama listed "Session expiration" as a cause. RFC 6585 defines 429 as rate limiting: the client sent too many requests in a given amount of time.

The bigger model wasn't immune either. When I chatted with Qwen3 8B directly and asked for the capital of Portugal, it first wrote a paragraph of reasoning to itself, then answered Lisbon correctly, then added a "fun fact": "Lisbon has been the capital since the 12th century, making it one of Europe's oldest continuously inhabited cities." Lisbon became the capital in the 13th century (1255 or 1256, depending on the source), and being the capital has nothing to do with being continuously inhabited.

That reasoning paragraph is Qwen3's thinking mode, which is on by default. The model card documents two ways to turn it off: enable_thinking=False in the chat template (what my benchmark sends through the API), or, according to the model card, adding /no_think to a message, which switches it off for that turn while thinking mode is on.

Offline, there is nobody to check the answer but you. Use these models for drafts, summaries of text you give them, and code you will read anyway. Don't use them as a source of facts.


What "no installation" really means

I installed nothing on any machine. That doesn't mean nothing was written, or that nothing was needed:

  • On Unix systems, llamafile extracts a small loader named ape to $TMPDIR.
  • On the Mac, the first run wrote the Metal GPU support files to ~/.llamafile (about 3 MB in my case). The docs say the GPU path on Apple Silicon needs the Xcode Command Line Tools, which my Mac already had.

If you need a machine to keep nothing, delete ~/.llamafile and $TMPDIR/.ape-1.10 afterwards.


When this is a bad idea

Situation Why
A company laptop Running an unsigned executable from a USB stick may break your company's policy, and the docs note that endpoint protection such as CrowdStrike can kill the process
You have internet and need correct facts A hosted model is far more capable than a 3B or 8B model on a laptop CPU
A CPU (or VM) without AVX On my Celeron and Mint VM it started, but the benchmark answer took minutes
A Mac without the Xcode Command Line Tools The Metal GPU path needs them, so expect CPU speed or a failed start (I didn't test this)
A shared stick you don't control Verify the checksums against a copy you keep elsewhere before every use

Where it does make sense: a trip without reliable internet, a machine you can't install anything on, working with text you can't send to an external service, or a demo that must work without Wi-Fi.


FAQ

Why not Ollama? Ollama is a separate build per operating system, and its normal setup is an installer or a service. You could carry its standalone binaries for each system, but llamafile is one file that runs on all of them.

Which model should I put on the stick? For most laptops without a strong GPU, a 3B model in Q4. Add an 8B model only if the machines you'll use have 16 GB of RAM and a recent CPU or an Apple Silicon GPU.

Can it use a GPU on Windows or Linux? According to the support page, yes for NVIDIA, AMD and Vulkan-capable GPUs, with Windows marked as best-effort in the 0.10 series. My VMs had no GPU passed through, and I didn't test GPU offload on Windows.


Takeaway

  • The 3B model ran on all four systems; the 8B model crashed on the Celeron. exFAT plus llamafile plus separate .gguf files, nothing installed (the Mac already had the Xcode Command Line Tools).
  • On my Mac, the stick only slowed the start. 29 s versus 2 s to load the 3B model, and a similar answer speed once loaded (39 vs 34 tokens/s).
  • Check your VM's CPU type before blaming the model. Same mini PC, 32 seconds versus 2 minutes 40 seconds, and the slow VM couldn't see AVX or AVX2.
  • Without AVX, it started but answers took minutes. The Celeron and the Mint VM both ran the 3B model without AVX: 6 min 22 s and 2 min 40 s per answer.
  • Small models can be confidently wrong. Read the code, don't trust the facts.

What would you use an AI on a USB stick for: travel, a machine you can't install on, or something I haven't thought of?


I'm Diogo Soares, a Staff AI Engineer working on AI agents and legacy modernization. If this was useful, follow me here on DEV.to and on Instagram at @diogosoares.dev, where every experiment becomes a Reel of under a minute. Every article also gets a video walkthrough on my YouTube channel, Diogo Soares Dev, so subscribe there for the next ones. Let's also connect on LinkedIn, or check out diogosoares.dev.

Top comments (0)