DEV Community

Cover image for I Turned a $10 USB Drive Into a Portable, Offline AI Assistant — Here's How You Can Too
Aman Kumar Dewangan
Aman Kumar Dewangan

Posted on

I Turned a $10 USB Drive Into a Portable, Offline AI Assistant — Here's How You Can Too

#ai #llm #opensource #tutorial

No cloud, no subscription, no internet required — how to run a quantized LLM entirely from a USB drive using llamafile.

If you've ever wanted your own private AI assistant — one that runs entirely on your machine, never sends a single token to a third-party server, and works on a plane with no wifi — this guide walks through exactly how to build one. By the end, you'll have a self-contained AI environment that lives on a USB drive and boots on any Windows laptop in under a minute.

Why Build This

Most people assume running a capable large language model requires a GPU rig, a cloud subscription, or at minimum a beefy always-on machine. That's no longer true. Thanks to model quantization and single-binary inference engines, a 7-8B parameter model can now run comfortably on consumer laptop CPUs, packaged into a single portable executable.

The benefits of a pendrive-based setup specifically:

  • Data sovereignty — nothing you type ever leaves the local machine or network.
  • Zero marginal cost — no per-token API billing, ever.
  • True portability — plug into any Windows machine, get the same AI, no installation.
  • Offline capable — once the files are loaded, no internet connection is needed at all.
  • A genuine systems lesson — you'll understand how LLM inference, quantization, and local networking actually fit together, instead of treating it as a black box behind an API.

What You'll Need

Component Purpose
A USB drive (16GB+, 32GB recommended) Storage for the engine and model
llamafile Single-executable LLM inference engine
A GGUF-format quantized model The actual language model weights
A Windows/macOS/Linux laptop Host machine to run it on
(Optional) A local chat UI like llama-ui Nicer front-end than the raw CLI

On model choice: pick any instruction-tuned open-weight model in GGUF format from Hugging Face — Qwen2.5/3, Llama 3, Mistral, and Gemma all have well-supported quantized releases. A Q4_K_M quantization is the sweet spot between size and coherence for CPU inference; it roughly halves the model's footprint versus full precision with minimal quality loss.

Step 1: Set Up the Folder Structure on Your Drive

Plug in your pendrive and lay out a structure like this:

E:\PortableAI\
├── bin\
│   └── llamafile\
│       └── llamafile.exe
├── models\
│   └── your-model-Q4_K_M.gguf
└── run-portable-ai.bat
Enter fullscreen mode Exit fullscreen mode

Keeping the engine and model in fixed relative paths means the launch script never has to hardcode a drive letter — it works whether the pendrive mounts as E:, F:, or anything else.

Step 2: Download llamafile

Grab the latest llamafile.exe release from the official llamafile repo. It's a single binary — no installer, no dependencies. Drop it into bin\llamafile\.

llamafile bundles a llama.cpp-based inference engine and a lightweight web server into one executable, which is exactly what makes this portable: one file runs the model and serves a chat interface over HTTP.

Step 3: Download a Quantized Model

Head to Hugging Face and search for GGUF builds of your model of choice (e.g., "Qwen2.5-7B-Instruct-GGUF"). Download the Q4_K_M variant — it typically lands in the 4–5GB range for a 7-8B model, which fits comfortably on any USB 3.0 drive alongside the engine.

Place the .gguf file in models\.

Step 4: Write the Launch Script

This is the part that makes the whole thing feel like a real product instead of a CLI toy. Create run-portable-ai.bat with the following:

@echo off
setlocal enabledelayedexpansion
set "DIR=%~dp0"
set "MODEL=%DIR%models\your-model-Q4_K_M.gguf"
set "BIN=%DIR%bin\llamafile\llamafile.exe"
set "HOST=0.0.0.0"
set "PORT=8080"

rem Optional shared password for the API/UI. Leave empty for no auth.
set "APIKEY=your-secret-key-here"

if not exist "%BIN%" (
    echo llamafile.exe not found at: %BIN%
    pause
    exit /b 1
)
if not exist "%MODEL%" (
    echo Model not found at: %MODEL%
    pause
    exit /b 1
)

rem Detect this machine's LAN IP so other devices on the network can connect.
set "LANIP="
for /f "delims=" %%a in ('powershell -NoProfile -Command "(Get-NetIPConfiguration ^| Where-Object {$_.IPv4DefaultGateway -ne $null -and $_.NetAdapter.Status -eq 'Up'} ^| Select-Object -First 1 -ExpandProperty IPv4Address).IPAddress" 2^>nul') do set "LANIP=%%a"
if not defined LANIP set "LANIP=<this-laptop-ip>"

set "AUTH="
if defined APIKEY set "AUTH=--api-key %APIKEY%"

echo ==========================================================
echo  Portable AI server starting
echo ==========================================================
echo   On this laptop : http://127.0.0.1:%PORT%/
echo   Other devices  : http://%LANIP%:%PORT%/
echo.
echo  Press Ctrl+C to stop the server.
echo ==========================================================

"%BIN%" -m "%MODEL%" --host %HOST% --port %PORT% %AUTH%
Enter fullscreen mode Exit fullscreen mode

A few implementation notes worth calling out:

  • %~dp0 resolves to the script's own directory — this is the trick that makes the whole thing drive-letter-agnostic. It works whether Windows mounts your pendrive as E:\ or G:\.
  • HOST=0.0.0.0 binds the server to all network interfaces, not just localhost — this is what lets other devices on the same wifi reach it. If you only want it accessible from the host laptop itself, set this to 127.0.0.1 instead.
  • The PowerShell one-liner pulls the first "up" network adapter with a default gateway, a reliable way to grab the actual LAN-facing IP rather than a VPN or virtual adapter address.
  • APIKEY gates access with a shared secret. Leave it blank for personal single-device use; set it if you're exposing the server to a shared network.

Step 5: Run It

Double-click run-portable-ai.bat. On first launch:

  1. Windows Firewall will prompt to allow llamafile.exe — accept for Private networks if you want other devices to reach it.
  2. The model loads into memory (a few seconds to a minute, depending on size and disk speed).
  3. Once ready, open http://127.0.0.1:8080/ in a browser — you'll see a built-in chat UI, ready to use, fully offline.

To use it from your phone or another laptop on the same wifi, browse to the LAN address the script printed (e.g., http://192.168.0.101:8080/).

Step 6 (Optional): A Nicer Front-End

llamafile's built-in UI is functional but basic. For a more polished chat experience, point a local UI like llama-ui at the same endpoint — it talks to llamafile's OpenAI-compatible API (/v1/chat/completions) and adds conversation history, model switching, and a settings panel for your API key.

Security Notes Before You Share This With Anyone

  • Binding to 0.0.0.0 with no API key means anyone on the same network can use your AI — fine at home, not fine on public/office wifi. Set APIKEY outside a fully trusted network.
  • This setup is for personal, local, authorized use. Don't port-forward it to the public internet without authentication and a reverse proxy — an open llamafile endpoint is an open compute resource for anyone who finds it.
  • Quantized models can still produce inaccurate output. Treat it like any other LLM: verify anything factual before relying on it.

What This Actually Demonstrates

This isn't just a neat trick — it's a working example of the shift happening in AI right now: inference moving from centralized cloud APIs to the edge. Quantization made 8B-parameter models small enough to run on a laptop CPU; single-binary engines like llamafile made deployment trivial enough that "portable AI on a USB stick" is now a weekend project instead of a research paper.

For anyone thinking about data privacy, offline-capable tooling, or just wanting to understand LLM infrastructure hands-on instead of through an API wrapper — this is one of the most direct ways to get there.

Have questions about adapting this for macOS/Linux, running a larger model, or securing it for multi-user access? Drop a comment below — happy to dig in.
Sent

Top comments (0)