I Turned a $10 USB Drive Into a Portable, Offline AI Assistant — Here's How You Can Too

Aman Kumar Dewangan — Mon, 24 Aug 2026 08:34:16 +0000

No cloud, no subscription, no internet required — how to run a quantized LLM entirely from a USB drive using llamafile.

If you've ever wanted your own private AI assistant — one that runs entirely on your machine, never sends a single token to a third-party server, and works on a plane with no wifi — this guide walks through exactly how to build one. By the end, you'll have a self-contained AI environment that lives on a USB drive and boots on any Windows laptop in under a minute.

Why Build This

Most people assume running a capable large language model requires a GPU rig, a cloud subscription, or at minimum a beefy always-on machine. That's no longer true. Thanks to model quantization and single-binary inference engines, a 7-8B parameter model can now run comfortably on consumer laptop CPUs, packaged into a single portable executable.

The benefits of a pendrive-based setup specifically:

Data sovereignty — nothing you type ever leaves the local machine or network.
Zero marginal cost — no per-token API billing, ever.
True portability — plug into any Windows machine, get the same AI, no installation.
Offline capable — once the files are loaded, no internet connection is needed at all.
A genuine systems lesson — you'll understand how LLM inference, quantization, and local networking actually fit together, instead of treating it as a black box behind an API.

What You'll Need

Component	Purpose
A USB drive (16GB+, 32GB recommended)	Storage for the engine and model
llamafile	Single-executable LLM inference engine
A GGUF-format quantized model	The actual language model weights
A Windows/macOS/Linux laptop	Host machine to run it on
(Optional) A local chat UI like llama-ui	Nicer front-end than the raw CLI

On model choice: pick any instruction-tuned open-weight model in GGUF format from Hugging Face — Qwen2.5/3, Llama 3, Mistral, and Gemma all have well-supported quantized releases. A Q4_K_M quantization is the sweet spot between size and coherence for CPU inference; it roughly halves the model's footprint versus full precision with minimal quality loss.

Step 1: Set Up the Folder Structure on Your Drive

Plug in your pendrive and lay out a structure like this:

E:\PortableAI\
├── bin\
│   └── llamafile\
│       └── llamafile.exe
├── models\
│   └── your-model-Q4_K_M.gguf
└── run-portable-ai.bat

Keeping the engine and model in fixed relative paths means the launch script never has to hardcode a drive letter — it works whether the pendrive mounts as E:, F:, or anything else.

Step 2: Download llamafile

Grab the latest llamafile.exe release from the official llamafile repo. It's a single binary — no installer, no dependencies. Drop it into bin\llamafile\.

llamafile bundles a llama.cpp-based inference engine and a lightweight web server into one executable, which is exactly what makes this portable: one file runs the model and serves a chat interface over HTTP.

Step 3: Download a Quantized Model

Head to Hugging Face and search for GGUF builds of your model of choice (e.g., "Qwen2.5-7B-Instruct-GGUF"). Download the Q4_K_M variant — it typically lands in the 4–5GB range for a 7-8B model, which fits comfortably on any USB 3.0 drive alongside the engine.

Place the .gguf file in models\.

Step 4: Write the Launch Script

This is the part that makes the whole thing feel like a real product instead of a CLI toy. Create run-portable-ai.bat with the following:

@echo off
setlocal enabledelayedexpansion
set "DIR=%~dp0"
set "MODEL=%DIR%models\your-model-Q4_K_M.gguf"
set "BIN=%DIR%bin\llamafile\llamafile.exe"
set "HOST=0.0.0.0"
set "PORT=8080"

rem Optional shared password for the API/UI. Leave empty for no auth.
set "APIKEY=your-secret-key-here"

if not exist "%BIN%" (
    echo llamafile.exe not found at: %BIN%
    pause
    exit /b 1
)
if not exist "%MODEL%" (
    echo Model not found at: %MODEL%
    pause
    exit /b 1
)

rem Detect this machine's LAN IP so other devices on the network can connect.
set "LANIP="
for /f "delims=" %%a in ('powershell -NoProfile -Command "(Get-NetIPConfiguration ^| Where-Object {$_.IPv4DefaultGateway -ne $null -and $_.NetAdapter.Status -eq 'Up'} ^| Select-Object -First 1 -ExpandProperty IPv4Address).IPAddress" 2^>nul') do set "LANIP=%%a"
if not defined LANIP set "LANIP=<this-laptop-ip>"

set "AUTH="
if defined APIKEY set "AUTH=--api-key %APIKEY%"

echo ==========================================================
echo  Portable AI server starting
echo ==========================================================
echo   On this laptop : http://127.0.0.1:%PORT%/
echo   Other devices  : http://%LANIP%:%PORT%/
echo.
echo  Press Ctrl+C to stop the server.
echo ==========================================================

"%BIN%" -m "%MODEL%" --host %HOST% --port %PORT% %AUTH%

A few implementation notes worth calling out:

%~dp0 resolves to the script's own directory — this is the trick that makes the whole thing drive-letter-agnostic. It works whether Windows mounts your pendrive as E:\ or G:\.
HOST=0.0.0.0 binds the server to all network interfaces, not just localhost — this is what lets other devices on the same wifi reach it. If you only want it accessible from the host laptop itself, set this to 127.0.0.1 instead.
The PowerShell one-liner pulls the first "up" network adapter with a default gateway, a reliable way to grab the actual LAN-facing IP rather than a VPN or virtual adapter address.
APIKEY gates access with a shared secret. Leave it blank for personal single-device use; set it if you're exposing the server to a shared network.

Step 5: Run It

Double-click run-portable-ai.bat. On first launch:

Windows Firewall will prompt to allow llamafile.exe — accept for Private networks if you want other devices to reach it.
The model loads into memory (a few seconds to a minute, depending on size and disk speed).
Once ready, open http://127.0.0.1:8080/ in a browser — you'll see a built-in chat UI, ready to use, fully offline.

To use it from your phone or another laptop on the same wifi, browse to the LAN address the script printed (e.g., http://192.168.0.101:8080/).

Step 6 (Optional): A Nicer Front-End

llamafile's built-in UI is functional but basic. For a more polished chat experience, point a local UI like llama-ui at the same endpoint — it talks to llamafile's OpenAI-compatible API (/v1/chat/completions) and adds conversation history, model switching, and a settings panel for your API key.

Security Notes Before You Share This With Anyone

Binding to 0.0.0.0 with no API key means anyone on the same network can use your AI — fine at home, not fine on public/office wifi. Set APIKEY outside a fully trusted network.
This setup is for personal, local, authorized use. Don't port-forward it to the public internet without authentication and a reverse proxy — an open llamafile endpoint is an open compute resource for anyone who finds it.
Quantized models can still produce inaccurate output. Treat it like any other LLM: verify anything factual before relying on it.

What This Actually Demonstrates

This isn't just a neat trick — it's a working example of the shift happening in AI right now: inference moving from centralized cloud APIs to the edge. Quantization made 8B-parameter models small enough to run on a laptop CPU; single-binary engines like llamafile made deployment trivial enough that "portable AI on a USB stick" is now a weekend project instead of a research paper.

For anyone thinking about data privacy, offline-capable tooling, or just wanting to understand LLM infrastructure hands-on instead of through an API wrapper — this is one of the most direct ways to get there.

Have questions about adapting this for macOS/Linux, running a larger model, or securing it for multi-user access? Drop a comment below — happy to dig in.
Sent

Getting Over Hacktoberfest 2020

Aman Kumar Dewangan — Tue, 03 Nov 2020 23:28:15 +0000

My First Open Source Contribution: Hacktoberfest 2020

My name is Aman Kumar Dewangan, pursuing B.Tech in Electrical Engineering from the National Institute of Technology Raipur. I am a proficient IoT developer, Electronics Enthusiast, Frontend Developer, and Begineer in Cloud Engineering.

Background

I took CSE as my extra subject in Class 12^th, hence I have a pretty good knowledge of coding in C++. With the time I learnt the implementation of C++ in Arduino and choose my Area of Intrest as IoT. I further learnt Python during my initial days of college and spent a significant time understanding the concepts of Machine Learning. As time spent I decided to move to Frontend Development and still learning many things related to it. I am familiar with many programming languages like C, C++, Java, Python, JavaScript etc... (!Alert: HTML and CSS are not programming languages.).

Progress

As I have recently learnt JavaScript and I realised that it is pretty much different from other languages and there is no textual content available on the Internet to explain it easily, so I wrote articles on JavaScript and how learning Javascript can actually become fun.

Contributions

Adding articles on JavaScript
Pulse Oximeter for Calidad Healthcare

Reflections

It was really enjoying and informational, we get to learn from each other through interaction and learn what mistakes we possibly make while developing something. If possible, I surely wish to participate in it next year as well.

DEV Community: Aman Kumar Dewangan