DEV Community: Mohammed Ali Chherawalla

How to Run Voice-to-Text Locally on Your Desktop (Whisper, Offline Dictation)

Mohammed Ali Chherawalla — Thu, 25 Jun 2026 05:22:23 +0000

OpenAI trained Whisper on 680,000 hours of audio, and the small models that came out of it run on a laptop CPU in real time. That power sits idle while you pay a monthly subscription to send every voice note to someone else's server for transcription. Off Grid AI Desktop is a free, open-source app that runs Whisper directly on your Mac or PC.

GitHub →

Free, open-source, runs offline. No account, no API key, no audio leaving your machine.

The Off Grid AI Desktop chat, running a local model fully on-device.

Why local dictation matters

Dictation is the fastest way to get words out of your head. You speak at about 150 words a minute. You type at maybe 40. The gap is the whole point.

But most dictation tools ship your voice to a cloud service. That means your half-formed ideas, your client names, your passwords read aloud by accident, all travel over the network. With local Whisper, none of it does. The audio is captured, transcribed, and turned into text on the same machine you are sitting at.

You get the speed of voice without handing your voice to anyone.

What You Need

Whisper is light. Most of these models run fine on hardware you already own.

Minimum

macOS: Apple Silicon (M1) or recent Intel, 8 GB RAM, macOS 13+
Windows: any 64-bit CPU, 8 GB RAM, Windows 10+
About 2 GB free disk for the app and a small Whisper model

Recommended

macOS: M2 or newer, 16 GB unified memory
Windows: NVIDIA GPU (CUDA) or a recent GPU with Vulkan, 16 GB RAM
A USB or built-in mic with a clean signal

The transcription model itself is a few hundred megabytes. You are not downloading anything close to the size of a chat LLM.

What Off Grid AI Desktop Can Do

Off Grid AI Desktop bundles whisper.cpp, a C++ port of Whisper that runs without Python and without a server you sign into. Here is what that buys you.

Talk into the composer. Click the mic, speak, and your words land as text in the chat box. You edit before you send. Dictate a long prompt to the local model instead of typing it.

Capture thoughts hands-free. Pacing the room, washing dishes, walking with the laptop open. Speak the idea while it is fresh and clean it up later.

Keep medical, legal, and personal audio private. If you dictate notes that name real people or describe real situations, on-device transcription means that audio never becomes someone else's training data or log entry.

Pair it with the local LLM. Dictate a rough question, let the on-device model answer, read the answer, dictate a follow-up. The whole loop stays on your hardware.

How It Works

The flow is short. You press the mic. The app records audio from your input device. whisper.cpp loads the model into memory and decodes the audio into text. The text appears in the composer. That is the entire path, and every step happens locally.

Whisper models come in sizes, and the size is a trade between speed and accuracy:

Model size	Disk	Good for	What to expect
tiny / base	~75–150 MB	quick notes, fast machines	very fast, looser on accents and proper nouns
small	~500 MB	everyday dictation	a solid balance for most people
medium	~1.5 GB	accents, technical terms	slower, noticeably more accurate

Start with small. Move up only if it mishears names or jargon you use a lot.

How Hardware Acceleration Works

Whisper is small enough that even pure CPU transcription is usable. But acceleration helps, especially on the larger models.

On macOS, whisper.cpp uses Metal to run the model on the GPU, and Apple Silicon's unified memory means the CPU and GPU share the same pool with no copying back and forth. That is why a quiet M-series fanless laptop handles this so well.

On Windows, you get GPU acceleration through CUDA on NVIDIA cards or Vulkan on a broader range of GPUs. No compatible GPU? The CPU path still works, just a bit slower on the bigger models.

The models are quantized, stored in a compressed numeric format, so they fit in normal RAM and load quickly. That quantization is the reason this runs on consumer hardware at all.

Tips for Cleaner Transcripts

A few habits raise accuracy more than swapping models does.

Get close to the mic. Whisper is robust to noise, but a clear signal always wins. Speak in full sentences rather than fragments, because the model uses context from surrounding words to resolve what it heard.

Pick the right model for your machine. If transcription lags behind your speech, drop to a smaller model. If it keeps mangling specific names, go one size up. You will find your match in a few minutes.

For long dictation, pause naturally between thoughts. Those pauses give the decoder clean boundaries and reduce run-on errors.

Privacy: Stronger Than Cloud Dictation

A cloud dictation service receives your raw audio. It may store it, log it, attach it to your account, or use it to improve its product. You are trusting a privacy policy you did not write and cannot enforce.

Off Grid AI Desktop receives nothing, because there is no server. The app is AGPL-3.0 open source, so you can read exactly what it does with your audio, which is process it on your machine and discard it. No telemetry. No account. No upload. Pull the network cable and dictation still works.

Getting Started

Open the GitHub repo and grab the latest release for macOS or Windows, or clone and build from source.
Install and launch Off Grid AI Desktop.
Download a Whisper model from inside the app (start with small).
Open a chat, click the mic in the composer, and allow microphone access when your OS asks.
Speak. Watch the text appear. Edit and send.

That is it. No sign-up wall, no key to paste.

What's Coming

More transcription languages and model sizes selectable in the app
Push-to-talk and hotkey dictation outside the composer
Tighter integration with the meeting recorder for live captions
Cross-device sync so a phrase dictated on one machine reaches another

The built-in model browser. Download text, vision, image, and voice models from Hugging Face, sized to your machine.

FAQ

Q: Is it really free?

Yes. The app is free and open source under AGPL-3.0. No trial, no paywall on dictation.

Q: Does it work offline?

Completely. After the model is downloaded, you can turn off your network and dictation keeps working.

Q: Which Whisper model should I use?

Start with small for everyday use. Go to medium if you need better handling of accents or technical vocabulary, and tiny or base if your machine is older.

Q: How much RAM do I need?

8 GB is enough for the smaller models. 16 GB gives you headroom to run the larger model alongside the local chat LLM.

Q: Does it work on Windows as well as Mac?

Yes. macOS uses Metal, Windows uses CUDA or Vulkan, and both fall back to CPU when there is no compatible GPU.

Q: Is my voice ever uploaded?

No. There is no server to upload to. Audio is transcribed on your machine and not retained.

Run Whisper on your own hardware, keep your voice to yourself.

GitHub →

How to Transcribe Meetings Locally in 2026 (Whisper, On-Device)

Mohammed Ali Chherawalla — Thu, 25 Jun 2026 05:22:18 +0000

OpenAI's Whisper model runs accurate speech-to-text on a laptop GPU, and the C++ port runs it without Python or a cloud account. That capability sits unused while teams pay per-minute fees to upload their calls to a transcription service. Off Grid AI Desktop is a free, open-source app that records and transcribes your meetings directly on your Mac or PC.

GitHub →

Free, open-source, runs offline. No account, no API key, no telemetry.

Meetings record and transcribe on-device, with a local summary and transcript.

What This Gets You

You want a searchable record of what was said, not a subscription to a notetaker bot. Here is the difference local transcription makes.

The audio never leaves your machine, so a confidential call stays confidential. There is no per-minute meter, so a three-hour planning session costs the same as a five-minute standup, which is nothing. The transcript and summary save to your disk, so you own the record instead of renting access to it.

What You Need

Whisper comes in several sizes. The bigger ones transcribe better and want more memory.

Tier	macOS	Windows	RAM / VRAM	Free disk
Minimum	Apple Silicon M1	Any modern CPU or iGPU	8 GB	10 GB
Recommended	M2 / M3 / M4	NVIDIA RTX (6 GB+ VRAM)	16 GB+	20 GB

On the minimum tier, a smaller Whisper model transcribes a one-hour call in a few minutes after it ends. The recommended tier runs the larger, more accurate models and finishes faster. CPU transcription works everywhere; a GPU just speeds it up.

How the Meeting Recorder Works

The recorder captures three streams at once: the screen video, the system audio coming out of your speakers, and your microphone. That covers both sides of a Google Meet or Zoom call, the remote participants on system audio and you on the mic.

When you stop, the bundled whisper.cpp engine transcribes the captured audio locally. There is no upload step. The transcription is the same model the cloud services use, running on your own hardware.

After the transcript is ready, the local LLM reads it and generates a title, a summary, and a list of the people mentioned. You get a usable meeting note without rewatching the recording or paying a notetaker.

Why On-Device Transcription Is Viable Now

Whisper ships as quantized weights, which stores the model at lower precision so it fits in consumer RAM without losing meaningful accuracy. That is the change that moved good transcription off the cloud and onto a laptop.

On macOS, transcription runs through Metal against Apple Silicon's unified memory. On Windows, you get CUDA on NVIDIA cards or a CPU path when there is no GPU. Either way the math runs locally, so the longer a call runs, the more you save versus a per-minute service.

Getting Cleaner Transcripts

Audio quality drives transcription quality more than model size does. Use a headset or a decent mic so your own track is clean, and the result improves immediately.

Pick the model to match the job. A smaller Whisper model is fine for a quick internal sync where you just need the gist. Reach for a larger model when accuracy matters, like a client call you will quote from later. Recording system audio captures the remote side at the source, which beats a microphone picking it up off your speakers.

Privacy: Stronger Than a Cloud Notetaker

A cloud notetaker uploads your full meeting audio and video to a server you do not control. The recording, the transcript, and the summary all live there. Off Grid AI Desktop keeps every part of that on your machine, because there is no server in the path.

It is AGPL-3.0 licensed, so the code is auditable. There is no account and no telemetry. The recorder requires an explicit start and stop, and a visible recording indicator stays on the whole time, so nothing records silently in the background.

Getting Started

Download or clone from the GitHub repo.
Install and launch the app on your Mac or PC.
Open the Models browser and download a Whisper model plus a local LLM for summaries.
Start a recording before your call, confirm the indicator is on.
Stop when the call ends. Read the transcript, title, and summary.

git clone https://github.com/off-grid-ai/desktop
cd desktop
npm install
npm run dev

What's Coming

Cross-device sync so meeting notes follow you between machines.
Unified search across transcripts and the rest of your captured work.
More transcription models as new open-weight releases ship.

FAQ

Q: Is it really free?

Yes. The app is free and open-source under AGPL-3.0. There are no per-minute charges and no subscription.

Q: Does it work offline?

Yes. Recording and transcription both run with no network connection.

Q: Does it work with Zoom and Google Meet?

Yes. It records the screen, system audio, and mic, so it captures both sides of any call in your browser or a desktop client.

Q: How accurate is the transcription?

It runs Whisper, the same model behind many cloud transcription tools. Accuracy depends on the model size you pick and your audio quality.

Q: How much RAM do I need?

8 GB runs a smaller Whisper model. 16 GB or more is comfortable for the larger, more accurate ones.

Q: Is my meeting audio private?

Yes. The audio, transcript, and summary stay on your machine. Nothing uploads.

Transcribe your calls on hardware you already own. GitHub →

How to Run Text-to-Speech Locally on Your Desktop in 2026 (Kokoro, Offline Voice)

Mohammed Ali Chherawalla — Thu, 25 Jun 2026 05:21:42 +0000

Kokoro-82M is a text-to-speech model with 82 million parameters, small enough to fit in a few hundred megabytes, yet it produces voices that hold up against systems many times its size. That quality sits idle while you pay per character to a cloud TTS API that meters every word you synthesize. Off Grid AI Desktop is a free, open-source app that runs Kokoro directly on your Mac or PC.

GitHub →

Free, open-source, runs offline. No account, no API key, no text leaving your machine.

The Off Grid AI Desktop chat, running a local model fully on-device.

Why local text-to-speech matters

Cloud TTS bills by usage. A long article, a book chapter, a daily digest read aloud, those characters add up, and the meter keeps running. Worse, every line you send to be spoken is a line you hand to someone else's server.

Local TTS removes both problems. You synthesize as much as you want for free, and the text never leaves your machine. Read your own notes back to you. Turn a chat answer into audio. Build a voice mode that talks to you while you work, all without a usage bill or a privacy trade.

You own the voice and you own the text.

What You Need

Kokoro is tiny by model standards, so the bar is low.

Minimum

macOS: Apple Silicon (M1) or recent Intel, 8 GB RAM, macOS 13+
Windows: any 64-bit CPU, 8 GB RAM, Windows 10+
About 2 GB free disk for the app and the voice model

Recommended

macOS: M2 or newer, 16 GB unified memory
Windows: NVIDIA GPU (CUDA) or a GPU with Vulkan, 16 GB RAM
Decent speakers or headphones

At 82 million parameters, Kokoro runs comfortably on a CPU. A GPU makes it snappier, but it is not required.

What Off Grid AI Desktop Can Do

Off Grid AI Desktop bundles Kokoro-82M, an open-weight, multilingual TTS model, and wires it into the chat interface. Here is what you get.

Speak any message. Every reply from the local model has a Speak control. Click it and hear the answer read aloud in a natural voice. Useful when your eyes are busy or your screen is full.

Turn on voice mode. Flip auto-speak on and the model reads each new answer out loud as it arrives. Ask a question, then look away and listen. The chat becomes a conversation you can have hands-free.

Read long text without straining your eyes. Paste a document or generate a summary, then have it read back to you while you rest your eyes or walk around.

Stay private with sensitive text. If the words being spoken include client details, health notes, or anything personal, on-device synthesis means that text is never sent anywhere to be voiced.

How It Works

The path is short and stays on your machine. You click Speak, or auto-speak triggers on a new message. The app passes the text to Kokoro running locally. The model generates an audio waveform. Your speakers play it. No request leaves the device.

Kokoro is multilingual and ships with a set of voices. Picking one is a matter of taste:

Choice	What it affects	What to expect
Voice	timbre and accent	pick the one that is easiest for you to listen to
Auto-speak on	reads every reply	best for hands-free, listen-while-you-work sessions
Auto-speak off	per-message Speak only	best when you want audio only sometimes

There are no per-character costs to manage, so synthesize freely.

How Hardware Acceleration Works

Because Kokoro is small, the model loads fast and synthesizes quickly even on modest hardware.

On macOS, generation runs on Apple Silicon with Metal acceleration, and unified memory keeps the CPU and GPU working from the same pool. A fanless laptop handles this without breaking a sweat.

On Windows, you get acceleration through CUDA on NVIDIA cards or Vulkan on a wider range of GPUs, with a CPU fallback that still performs well given the model's size.

The model weights are quantized, stored compactly, so they fit in ordinary RAM and start fast. That is what makes a high-quality voice viable on a normal desktop.

Tips for Better Playback

A few small choices improve the listening experience.

Pick a voice you can listen to for a while, not just the first one. You will be hearing it a lot, and comfort matters more than novelty.

Keep auto-speak off when you are skimming and on when you commit to listening. Toggling it deliberately stops the app from reading things you only meant to glance at.

For long passages, let the audio play in the background while you do something else. Local synthesis has no meter, so there is no reason to ration it.

If a name or acronym is pronounced oddly, edit the text slightly before speaking it. Spelling something the way it sounds usually fixes the pronunciation.

Privacy: Stronger Than Cloud TTS

A cloud TTS API receives every string you want spoken. It can log that text, tie it to your account, and bill you for it. You are trusting a vendor with the exact words you are reading.

Off Grid AI Desktop sends nothing, because there is no server. The app is AGPL-3.0 open source, so you can verify that synthesis happens on your machine and the audio is played, not uploaded. No telemetry, no account, no usage meter. Disconnect from the internet and the voice still speaks.

Getting Started

Open the GitHub repo and download the latest release for macOS or Windows, or clone and build it yourself.
Install and launch Off Grid AI Desktop.
Download the Kokoro voice model from inside the app.
Open a chat, send a message, and click Speak on the reply.
Turn on auto-speak when you want every answer read aloud.

No sign-up, no key to paste, no character quota.

What's Coming

More Kokoro voices and languages selectable in the app
Adjustable speaking rate and pitch
Read-aloud for documents and captured notes, not just chat
Cross-device sync so a voice set on one machine carries to another

The built-in model browser. Download text, vision, image, and voice models from Hugging Face, sized to your machine.

FAQ

Q: Is it really free?

Yes. The app is open source under AGPL-3.0, and there is no per-character cost because synthesis runs locally.

Q: Does it work offline?

Yes. Once the voice model is downloaded, TTS runs with no network connection.

Q: Which model does it use?

Kokoro-82M, an open-weight multilingual TTS model that is small enough to run on a CPU yet sounds natural.

Q: How much RAM do I need?

8 GB is enough. 16 GB lets you run TTS alongside the local chat model comfortably.

Q: Does it work on Windows as well as Mac?

Yes. macOS uses Metal, Windows uses CUDA or Vulkan, and both fall back to CPU.

Q: Is the text I speak ever uploaded?

No. There is no server. Text is turned into audio on your machine and the audio is played locally.

Run a high-quality voice on your own hardware, keep your text to yourself.

GitHub →

How to Run a Private AI Meeting Notetaker (Zoom and Google Meet, On-Device)

Mohammed Ali Chherawalla — Thu, 25 Jun 2026 05:21:36 +0000

When you invite a cloud notetaker to a call, a bot joins, records everyone, and ships the audio to a vendor's servers. Half the room often does not know it is there. Off Grid AI Desktop is a free, open-source app that records, transcribes, and summarizes your meetings on your own Mac or PC, with the recording in plain sight and the data never leaving the machine.

GitHub →

Free, open-source, runs offline. No account, no API key, no telemetry.

Meetings record and transcribe on-device, with a local summary and transcript.

Why a Local Notetaker

A meeting note is supposed to save you time, not hand your conversations to a third party. Here is what changes when the notetaker lives on your desktop.

There is no bot joining the call, so the only consent at stake is yours and your participants', not a vendor's terms of service. The recording, the transcript, and the summary all stay on your disk. There is no per-minute fee, so it costs nothing to keep notes on every call you take.

What You Need

The notetaker runs Whisper for transcription and a local LLM for the summary. Both want a little memory.

Tier	macOS	Windows	RAM / VRAM	Free disk
Minimum	Apple Silicon M1	Any modern CPU or iGPU	8 GB	10 GB
Recommended	M2 / M3 / M4	NVIDIA RTX (6 GB+ VRAM)	16 GB+	20 GB

The minimum tier handles a smaller Whisper model and a compact LLM fine. The recommended tier runs larger models and finishes the transcript faster. Both macOS and Windows are supported.

Consent Comes First

The recorder does nothing until you press start. While it runs, a visible recording indicator stays on, so anyone watching your screen knows a capture is in progress, and you stop it with one action. There is no always-on listening and no bot lurking in the participant list.

That design is the point. A notetaker should make recording obvious, not hide it. You decide which calls get captured, and the indicator makes it honest for everyone in the room.

How It Works

The recorder captures the screen video, the system audio, and your microphone together, so both sides of a Zoom or Google Meet call are covered. When you stop, the bundled whisper.cpp engine transcribes the audio locally, with no upload.

Then the local LLM turns the transcript into a real note: a title, a summary, and the people who came up. You get the outcome of the call without rewatching it.

The summary does not just sit in a folder. It folds into your private memory and your Day, the persisted journal of what you did, so a meeting becomes part of a searchable record of your work instead of a one-off file you forget about.

Privacy: Stronger Than Otter or Fireflies

Cloud notetakers like Otter and Fireflies route your meeting audio and video through their servers. The recording, transcript, and summary live in their account, under their retention policy, with their access. Off Grid AI Desktop keeps all of it on your machine, because nothing in the pipeline touches a server.

It is AGPL-3.0 licensed, so the code is auditable rather than a black box. There is no account to breach and no telemetry phoning home. On an air-gapped machine, every part still works.

	Cloud notetaker	Off Grid AI Desktop
Where audio goes	Vendor's servers	Your disk
Bot in the call	Yes	No
Per-minute fee	Often	None
Code you can audit	No	Yes (AGPL-3.0)
Works offline	No	Yes

Getting Started

Download or clone from the GitHub repo.
Install and launch the app on your Mac or PC.
Open the Models browser and download a Whisper model plus a local LLM.
Press start before your call, confirm the recording indicator is on.
Stop when the call ends, then read the note in your Day.

git clone https://github.com/off-grid-ai/desktop
cd desktop
npm install
npm run dev

What's Coming

Cross-device sync so meeting notes follow you between machines.
Unified search across meeting summaries and the rest of your memory.
More transcription and summary models as new open-weight releases ship.

FAQ

Q: Is it really free?

Yes. The app is free and open-source under AGPL-3.0. No subscription and no per-minute charge.

Q: How is this different from Otter or Fireflies?

No bot joins your call, and nothing uploads. Recording, transcription, and the summary all happen on your machine.

Q: Does it work with Zoom and Google Meet?

Yes. It records the screen, system audio, and mic, so it captures both sides of any call.

Q: How does it handle consent?

Recording is explicit start-and-stop, and a visible indicator stays on the whole time. Nothing records silently.

Q: Does it work offline?

Yes. Recording, transcription, and summarizing all run with no network connection.

Q: Where do my meeting notes live?

On your disk, folded into your private memory and your Day. No vendor account holds them.

Keep your meeting notes private, on hardware you already own. GitHub →

How to Point Your IDE and Apps at a Local AI Model (Private, On-Device)

Mohammed Ali Chherawalla — Thu, 25 Jun 2026 05:21:01 +0000

Your editor, your terminal scripts, and half the AI tools you installed last month all speak the same protocol: the OpenAI HTTP API. They all assume that protocol points at a server you pay for. It does not have to. Off Grid AI Desktop is a free, open-source app that puts an OpenAI-compatible endpoint on your own Mac or PC, so every one of those tools can run against on-device models instead.

GitHub ->

Free, open-source (AGPL-3.0), runs offline. No account, no telemetry, no API key.

The local gateway: one OpenAI-compatible API at 127.0.0.1:7878 for every modality.

The address every tool can use

There is one endpoint to remember:

http://127.0.0.1:7878/v1

Anything that takes an OpenAI base URL takes this one. IDE extensions, CLI tools, a Python script, a browser extension, a shell alias. You give them this address and a placeholder key, and they get a private inference backend that works on a plane.

It is bound to loopback, so it answers only from your own machine. Nothing on your network or the internet can reach it. That is the point. Your code, your prompts, and your files go to a process you control, not to a vendor.

What You Need

Tier	macOS	Windows
Minimum	Apple Silicon (M1), 16 GB unified memory, macOS 13+, ~12 GB free disk	NVIDIA or recent CPU, 16 GB RAM, Windows 11, ~12 GB free disk
Recommended	M2/M3/M4, 24 GB+ unified memory	NVIDIA GPU (CUDA) or Vulkan GPU, 32 GB RAM

CPU fallback works on Windows when there is no GPU. It runs slower but it runs.

What you can wire up

The gateway is OpenAI-SDK compatible, so the list of things you can point at it is long. A few that developers reach for first:

IDE assistants and editor extensions that let you set a custom base URL. They send chat completions, the local model answers, your code never leaves the laptop.
A curl one-liner or a shell function for quick prompts from the terminal.
Scripts using openai-python or openai-node, where you change two arguments and the script now runs offline.
Any app that already speaks the OpenAI protocol or mirrors an Ollama-style models array, since the gateway exposes both shapes.

One endpoint covers more than text. You get vision, embeddings, speech-to-text, text-to-speech, and image generation behind the same OpenAI routes, so the tools you point at it are not limited to chat.

Point a script at it

Start with the smallest possible test. This confirms the endpoint answers.

curl http://127.0.0.1:7878/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local",
    "messages": [{"role": "user", "content": "Reply with just: it works"}]
  }'

Now the same in Python with the official SDK. The two lines that matter are base_url and api_key.

from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:7878/v1",
    api_key="local",  # any placeholder, the gateway ignores it
)

resp = client.chat.completions.create(
    model="local",
    messages=[{"role": "user", "content": "Summarize this commit message: fix off-by-one in pager"}],
)
print(resp.choices[0].message.content)

In Node, with the openai package:

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://127.0.0.1:7878/v1",
  apiKey: "local",
});

const resp = await client.chat.completions.create({
  model: "local",
  messages: [{ role: "user", content: "Rename this variable to be clearer: tmp2" }],
});
console.log(resp.choices[0].message.content);

Point your IDE at it

Most IDE assistants and editor AI extensions expose two settings: a base URL and an API key. Set them like this:

Base URL or API base: http://127.0.0.1:7878/v1
API key: any non-empty string, for example local
Model: local, or whatever id GET /v1/models reports

If the extension also asks for a provider type, choose OpenAI-compatible or custom. From there the extension's chat, inline completion, and edit features run against the model on your disk. To check which models are active and their kind (chat, vision, image, speech, transcription), call:

curl http://127.0.0.1:7878/v1/models

More than chat for your scripts

Because the same endpoint serves every modality, you can build small tools that would normally need three vendors.

Transcribe an audio file with whisper.cpp, sent as multipart:

curl http://127.0.0.1:7878/v1/audio/transcriptions \
  -F "file=@meeting.m4a" \
  -F "model=local"

Generate embeddings for a local search script, using all-MiniLM-L6-v2:

curl http://127.0.0.1:7878/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"model": "local", "input": "the cat sat on the mat"}'

There is also text-to-speech at /v1/audio/speech (Kokoro, WAV output, with voice ids from /v1/audio/voices) and image generation at /v1/images/generations. Same base URL, same placeholder key.

Handling slow calls in tooling

Some calls take a while. The first request to a modality downloads its model, and multi-step image generation runs for seconds to minutes. Rather than risk a client timeout in your script, opt into async with ?async=true, a body field "async": true, or the header Prefer: respond-async. You get a 202 with a poll_url, then poll GET /v1/requests/{id} until it finishes. For an IDE assistant doing short chat turns you will not need this, but a batch script will.

How it stays fast on a laptop

Models load on demand per modality and offload when the call ends, so a chat model and an image model never sit in RAM together. Your peak memory is set by the largest single job, not the sum of all of them.

The models themselves are quantized GGUF files at levels like q8_0 and Q4_K, which shrinks a model that wanted tens of gigabytes down to a handful. On macOS the GPU runs them on Metal over unified memory. On Windows it is CUDA for NVIDIA cards or Vulkan for others, with a CPU path as backup. That combination is why a consumer machine handles models that needed a rented server not long ago.

Privacy: stronger than a hosted backend

When your IDE talks to a hosted AI service, your source code goes to that service. It is logged, billed per token, and tied to an account.

When your IDE talks to 127.0.0.1:7878, the code goes to a process on your own machine and stops there. The gateway makes no outbound calls for inference. There is no telemetry and no account. The whole app is AGPL-3.0, so you can read what it does before you trust it with your repository. Disconnect from the network and every example above keeps working.

Getting Started

Clone or download from github.com/off-grid-ai/desktop.
Install and launch the app on macOS or Windows.
Confirm the gateway answers at http://127.0.0.1:7878/v1. Browse GET /docs for the Scalar playground or /openapi.json for the spec.
In each tool, set the base URL to that address and the key to any placeholder.
Run a request. The first call per modality downloads its model once, then everything runs offline.

What's Coming

Reaching this gateway from your other paired devices over the local mesh, so a second laptop or a phone can use the model on your desktop. This is roadmap, not shipped yet.
More bundled models across the modalities.
The on-device models are also exposed as MCP tools at POST /v1/.../mcp over Streamable HTTP, so MCP clients can call them. A separate article goes into that.

FAQ

Q: Will this work with my IDE extension?

If the extension lets you set a custom OpenAI base URL and key, yes. Set the URL to http://127.0.0.1:7878/v1 and the key to any placeholder.

Q: Is it really free?

Yes. AGPL-3.0, open source, no metered API. You run models on your own hardware, so there is no token bill.

Q: Does it work offline?

Yes. After each modality downloads its model once, every endpoint runs with no internet.

Q: Which clients are supported?

Anything that speaks the OpenAI HTTP API, including openai-python and openai-node. The gateway also mirrors an Ollama-style models array for tools that expect that.

Q: How much RAM do I need?

16 GB works on macOS and Windows. 24 GB or more helps with bigger models and image generation. Models load one at a time, so size for the heaviest single job.

Q: Is my code private?

The endpoint is bound to 127.0.0.1 and makes no outbound inference calls. No telemetry, no account, open source. Your repository stays on your disk.

Give your whole machine a private inference backend.

GitHub ->

One Local API for Chat, Vision, Images, Speech, and Embeddings (On-Device, 2026)

Mohammed Ali Chherawalla — Thu, 25 Jun 2026 05:20:55 +0000

A modern laptop has a GPU and unified memory that can run a chat model, a vision model, a diffusion model, and a whisper transcriber. Most of that silicon sits idle while you pay a monthly bill to call those same models on someone else's server, over the internet, with your data in transit. Off Grid AI Desktop is a free, open-source app that runs all of those models directly on your Mac or PC, and exposes them through one local HTTP API.

GitHub ->

Free, open-source (AGPL-3.0), runs offline.

The local gateway: one OpenAI-compatible API at 127.0.0.1:7878 for every modality.

The short version

Off Grid AI Desktop ships a local server at http://127.0.0.1:7878/v1. It speaks the OpenAI API shape. One base URL covers six modalities: chat, vision, embeddings, speech-to-text, text-to-speech, and image generation. It binds to loopback only, so it never faces the internet. There is no API key, no account, no cloud hop. You point any OpenAI SDK at the base URL and your existing code works.

That is the whole pitch. The rest of this post is one example call per modality so you can paste and run.

What you need

The app bundles the inference engines, so you only need the hardware to run the models.

Tier	Mac	Windows	RAM/VRAM	Disk
Minimum	Apple Silicon M1	NVIDIA GPU or modern CPU	16 GB	20 GB free
Recommended	M3 / M4	RTX with 8 GB+ VRAM	32 GB	40 GB+ free

Models are quantized GGUF, so they fit in consumer memory. On Mac you get Metal acceleration over unified memory. On Windows you get CUDA or Vulkan, with a CPU fallback.

Why one endpoint matters

You already know the OpenAI client. You set a base URL and a key, you call chat.completions.create, and you parse the response. Off Grid AI Desktop keeps that contract and changes only where the request goes. Your code stops sending bytes to a data center. Nothing leaves the machine.

That means you can take a script you wrote against the cloud and repoint it in one line:

from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:7878/v1",
    api_key="local",  # any placeholder works; no key is checked
)

The api_key is ignored. It exists so the SDK stops complaining. Now the same client object handles every modality below.

Chat

Standard chat completions. The local model runs the request and streams or returns the text.

curl http://127.0.0.1:7878/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {"role": "user", "content": "Summarize the CAP theorem in two sentences."}
    ]
  }'

You get back the usual choices[0].message.content. No streaming flag needed for a single response. Set "stream": true when you want tokens as they arrive.

Vision

The same chat endpoint takes images. Pass an image_url as a data URL, an http URL, or a local file path. The active vision model reads the pixels and answers.

curl http://127.0.0.1:7878/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{
      "role": "user",
      "content": [
        {"type": "text", "text": "What is in this screenshot?"},
        {"type": "image_url", "image_url": {"url": "file:///Users/you/Desktop/shot.png"}}
      ]
    }]
  }'

A file:// path stays on disk and never uploads anywhere. The model loads, reads the image, and offloads when done.

Embeddings

Vector embeddings for search and RAG, served by all-MiniLM-L6-v2.

curl http://127.0.0.1:7878/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"input": "local-first inference"}'

You get a data[0].embedding array of floats. Feed it to whatever vector store you run. Index your private documents without sending a single line of them off your machine.

Speech to text

Transcription runs whisper and handles multiple languages. Send the audio file as multipart form data.

curl http://127.0.0.1:7878/v1/audio/transcriptions \
  -F file=@meeting.m4a

The response carries the transcribed text. A recorded call, a voice memo, an interview: it gets turned into text on the device that holds the recording.

Text to speech

The reverse direction uses Kokoro and returns a WAV file. List the available voices first, then synthesize.

curl http://127.0.0.1:7878/v1/audio/voices

curl http://127.0.0.1:7878/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "input": "Your draft is ready for review.",
    "voice": "af_heart"
  }' --output speech.wav

You now have a WAV file on disk, generated locally. Pipe it into a notification, a reading-mode feature, or an accessibility flow.

Text to image and image to image

Image generation uses the local diffusion engine. Text to image is one POST.

curl http://127.0.0.1:7878/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{"prompt": "a terminal-green wireframe of a city at night"}'

For image to image, send the source as multipart to the edits endpoint, or reference it inline.

curl http://127.0.0.1:7878/v1/images/edits \
  -F image=@source.png \
  -F prompt="repaint in cinematic lighting"

The pixels render on your GPU. Nothing about the prompt or the output touches a remote service.

How the models share your memory

The app does not hold every model in RAM at once. Each modality loads on demand and offloads after the request. The chat model and the diffusion model never co-reside, so a 16 GB machine can serve all six modalities, one at a time, without thrashing.

That trades a small load cost on a cold call for a memory footprint that fits real hardware. If you call the same modality repeatedly, it stays warm.

For long jobs like image generation, opt into async with ?async=true or the header Prefer: respond-async. You get back 202 Accepted and a poll_url. Poll GET /v1/requests/{id} until it finishes.

curl "http://127.0.0.1:7878/v1/images/generations?async=true" \
  -H "Content-Type: application/json" \
  -d '{"prompt": "a quiet harbor at dawn"}'
# -> 202, { "poll_url": "/v1/requests/abc123" }

Privacy: stronger than a hosted API

A hosted multimodal API sees every prompt, every image, every audio file you send. It logs requests. It needs an account and a key tied to your identity. Off Grid AI Desktop binds to 127.0.0.1, so the server answers only your own machine. There is no telemetry. There is no account. The code is AGPL-3.0, so you can read exactly what it does. Your screenshots, recordings, and documents stay where they are.

Getting started

Download or clone from github.com/off-grid-ai/desktop.
Install and launch the app.
Pick and download a model for each modality you want from the built-in browser.
The gateway is live at http://127.0.0.1:7878/v1. Open /docs for interactive docs or /openapi.json for the full spec.
Point your OpenAI SDK at the base URL and call any modality above.

Run GET /v1/models any time to see which model is active per modality, tagged by kind: chat, vision, image, speech, transcription.

What's coming

Use the gateway from other paired devices over your local mesh, so a phone or tablet can call your desktop as the household's private inference backend. Local only, no cloud.
More bundled models across modalities.
Wider OpenAI surface coverage as the spec grows.

FAQ

Q: Is it really free?

Yes. The app is free and open-source under AGPL-3.0. There is no paid tier gating the gateway.

Q: Does it work offline?

Yes. Once the models are downloaded, every call runs on-device with no network.

Q: Do I need an API key?

No. The endpoint checks nothing. SDKs require a key field, so pass any placeholder string.

Q: Will my existing OpenAI code work?

If it uses the official openai-python or openai-node SDK, set base_url to http://127.0.0.1:7878/v1 and it works for chat, vision, embeddings, audio, and images.

Q: Can other apps on my network reach it?

No. The server binds to loopback only. It answers your machine and nothing else. Cross-device access over the local mesh is a separate, future feature.

Q: How much RAM do I need?

16 GB runs the modalities one at a time. 32 GB keeps more headroom and faster warm calls.

Stop renting access to models your own machine can run.

GitHub ->

A Local Code Sandbox: Run AI-Generated Code Safely On-Device (No Cloud)

Mohammed Ali Chherawalla — Thu, 25 Jun 2026 05:20:19 +0000

Every modern laptop ships with a browser engine that can run untrusted code in a locked box. That same sandbox sits idle while you paste AI-generated snippets into a cloud playground that logs everything you send it. Off Grid AI Desktop is a free, open-source app that runs a local code sandbox directly on your machine, with no network and no file access.

GitHub → Free, open-source, runs offline.

The Artifacts tab. HTML, React, and documents the model generated, rendered in a local sandbox.

The problem with running AI code

You ask a model to build a small UI. It hands back a block of React. Now what.

Option one: paste it into a browser console or a fresh HTML file. You read it line by line first, because you have no idea what it does. Option two: drop it into a cloud playground. Fast, but every keystroke leaves your machine and lands on someone else's server.

Both options leak. Cloud playgrounds see your code and often your prompt. Local files give the script your filesystem. Neither is what you want for code you did not write.

Off Grid AI Desktop gives you a third path. The model writes the code on-device. The code runs on-device. The sandbox cannot reach the internet and cannot read a single file.

What you need

This runs on macOS and Windows. The sandbox itself is light. The local model that writes the code is the part that wants memory.

Tier	Hardware	What you get
Minimum	16 GB RAM (Apple Silicon or a recent PC), 10 GB free disk	Small models write HTML and simple JS; preview renders instantly
Recommended	24 GB+ unified memory (Mac) or an 8 GB+ GPU (NVIDIA/Vulkan)	Larger models write React components and Mermaid diagrams without stalling

The preview pane is just a sandboxed browser frame. It costs almost nothing to render. Your hardware budget goes to the model.

What Off Grid AI Desktop can do here

Ask the local model for a component, a chart, or a layout. It writes the code. The app renders it live in a sandboxed iframe, and you flip between the source and the running result.

You get:

A live preview of model-written HTML, SVG, JS, and React. You see the thing run, not a description of it.
A Code/Preview toggle. Read the source, then watch it execute. One click between them.
Download. Save the output and use it wherever you want.
Mermaid diagrams. Ask for a flowchart in plain English, get a rendered diagram.

The point is speed with a hard safety boundary. You read code when you want to. You do not have to read it to know it cannot phone home or touch your disk.

How the sandbox works

The preview runs inside a sandboxed iframe. Two restrictions matter most.

No network access. The frame cannot make a fetch call, load a remote script, or open a socket. If the model writes code that tries to beacon out, the call goes nowhere.

No file access. The frame cannot read your home directory, your documents, or anything else on disk. It runs in its own walled box with nothing behind it.

So when the model writes React, where does React come from. Not a CDN. The runtimes ship inside the app.

Offline runtimes, no CDN

Most live-preview tools pull React, a JSX compiler, and a diagram library from a content delivery network at render time. That means a network call every time you preview, and a hard dependency on being online.

Off Grid AI Desktop bundles those runtimes in app resources:

React (UMD build) for components.
Babel-standalone to compile JSX in the browser frame.
Mermaid for diagrams.

They live on disk inside the app. No CDN, no fetch, no version that changes under you. The preview renders the same way on a plane as it does at your desk. This is also why the no-network rule does not break anything: the frame never needed the network in the first place.

model writes code  ->  Code/Preview toggle  ->  sandboxed iframe
                                                  no network
                                                  no file access
                                                  bundled React + Babel + Mermaid

A practical tip: read the code, then trust the box

The natural worry with AI code is that it does something you did not ask for. A normal sandbox asks you to audit every line before you run it. This one inverts the order.

Flip to Preview first. The frame has no network and no disk, so even hostile code has nothing to act on. Watch the behavior. If you like it, flip to Code and read the source, then download. You audit because you want clean code, not because you are scared of it.

For React work, keep components self-contained. The bundled UMD build covers React itself. Ask the model to write standard hooks and JSX rather than reaching for an obscure third-party package the bundle does not carry.

Privacy: stronger than a cloud playground

A cloud playground sees your prompt, your code, and your edits. Many keep them. Your draft of a half-working idea becomes a row in someone's database.

Off Grid AI Desktop sees none of that, because there is no server in the loop. The model runs on your machine. The sandbox runs on your machine. Off Grid AI Desktop is open source under AGPL-3.0, takes no account, sends no telemetry, and works with your network cable unplugged. The code you generate is yours and stays on your disk.

Getting started

Clone or download from GitHub: https://github.com/off-grid-ai/desktop
Install the app on macOS or Windows.
Launch it and pick a local model from the built-in models browser.
Open a chat and ask for something visual. Try "build a small React card component" or "draw a flowchart for a login flow in Mermaid."
When the code arrives, flip to Preview to watch it run, then to Code to read it. Download if you want it.

What's coming

More bundled runtimes so the sandbox covers a wider range of model output.
A shared local API so other on-device tools can reuse the same preview.
Cross-device sync so a sandbox you build on one machine shows up on another, still without a cloud middleman.

The Off Grid AI Desktop chat, running a local model fully on-device.

FAQ

Q: Is it really free?

Yes. Off Grid AI Desktop is free and open source under AGPL-3.0. No account, no card.

Q: Can the AI code reach the internet from the preview?

No. The sandboxed iframe has no network access. A fetch call or remote script load from inside the preview goes nowhere.

Q: Can the preview read my files?

No. The frame has no file access. It cannot read your home folder or anything else on disk.

Q: Does it need to be online to render React?

No. React, Babel-standalone, and Mermaid are bundled in app resources. There is no CDN call, so the preview works fully offline.

Q: What can the model write and preview?

HTML, SVG, plain JavaScript, React components, and Mermaid diagrams. Flip between Code and Preview, and download the result.

Q: Mac or Windows?

Both. The app runs on macOS (Apple Silicon) and Windows (CUDA or Vulkan GPU, with CPU fallback).

Run AI-generated code without handing it your network or your files. Off Grid AI Desktop on GitHub →

Skills: Give Your Local AI Reusable, Triggerable Automations (On-Device)

Mohammed Ali Chherawalla — Thu, 25 Jun 2026 05:20:14 +0000

You write the same prompt to your AI over and over. The proofread instructions. The standup format. The triage rules. Every time you retype them, or paste them from a notes file, and tweak the wording again.

Off Grid AI Desktop lets you save that prompt once as a skill, then run it with a slash command or let it fire on its own. All of it runs on your machine.

GitHub →

Free, open-source, runs offline. No account, no API keys, no telemetry.

The Off Grid AI Desktop chat, running a local model fully on-device.

What a skill is

A skill is a reusable instruction pack. If you have used Claude Code skills, the shape is familiar. You hand the model a folder of instructions, name it, and call it up whenever you need that behavior.

Each skill is a folder under a local .skills directory. Inside sits a SKILL.md file with two parts: frontmatter that gives the skill a name and a description, and a body that holds the actual instructions.

.skills/
  proofread/
    SKILL.md

A SKILL.md looks like this:

---
name: proofread
description: Fix grammar and tighten prose without changing meaning.
---

Read the text I give you. Correct spelling and grammar.
Tighten wordy sentences. Keep my voice and meaning intact.
Return the cleaned text, then a short list of what you changed.

If you do not need a folder, a bare proofread.md works too. Same idea, one file.

Running a skill on command

Type a slash command in chat to invoke a skill. /proofread pulls in that skill's instructions and applies them to your message.

This is the everyday use. You build a small library of skills for the things you ask for constantly, then call them by name instead of retyping the prompt.

A few skills worth writing first:

proofread: clean up grammar and tighten prose without changing your voice.
standup: turn a list of yesterday's work into a tight daily standup post.
triage: read a batch of messages or tasks and sort them by urgency with a one-line reason each.

Each one is a few lines of SKILL.md. Write them once and they are there forever.

Triggers: skills that run on their own

A skill does not have to wait for a slash command. You can give it a trigger and it runs by itself. When a triggered skill fires, it runs an action prompt you define, and it can reach for your MCP connector tools while it works.

There are three trigger types.

Schedule. The skill runs daily at a local time you set, like 09:00. Good for a morning summary or an end-of-day wrap.

Keyword. The skill fires when a keyword shows up in a newly captured observation. If "invoice" appears in what you captured, a billing skill can wake up and act on it.

Event. The skill fires on a new calendar event or a new approval. Useful for prepping context before a meeting or reacting the moment something lands in your queue.

A worked example: the daily standup

Say you want a standup summary in your hands every morning without thinking about it.

Write a standup skill. The body tells the model how you like your standup formatted: what you shipped, what is next, any blockers. Give it a schedule trigger at 09:00.

Each morning the skill runs its action prompt on its own. If you have connected a task tool through MCP, the skill can pull from it while it writes. The summary is ready when you sit down. You did not type a word.

How triggers reach your tools

A triggered skill can use the MCP connectors you have set up. Connect a tool like a task tracker or a notes tool, and a running skill can read from it as part of its work.

The local model does the reasoning. The connector fetches the data. The skill ties them together with the instructions you wrote. Your triage skill can read the day's items and sort them. Your standup skill can pull yesterday's closed tasks.

The pattern is simple to hold in your head. Connectors bring data in. The on-device model reasons over it. The skill says what to do.

Everything stays local

This is the part that separates it from cloud automation tools. The skill files live on your disk. The model that runs them is on your machine. The triggers fire on your machine.

Nothing leaves the device. Your .skills folder is yours to read, edit, version in git, and back up. There is no automation service watching your calendar from a server, and no prompt library sitting in someone else's account.

Tips for writing good skills

A few things that make skills work better.

Keep the description sharp. It tells the model what the skill is for, so a vague description gets vague behavior.

Be specific in the body. Spell out the format you want, the tone, the edge cases. The model follows clear instructions far better than hints.

Start with one skill you reach for daily. Get it right, then add the next. A handful of sharp skills beats a pile of half-written ones.

For triggered skills, scope the action prompt tightly. A skill that runs on its own should know exactly what to produce, since you will not be there to nudge it.

Privacy: stronger than cloud automation

Cloud automation platforms hold your triggers, your prompts, and the data they touch on their servers. Every run is logged somewhere you do not control.

Off Grid AI Desktop holds all of it on your machine. The skill files are local. The model is local. The triggers run local. The app is AGPL-3.0, so you can read exactly what it does. No account, no telemetry.

Getting started

Get the app from https://github.com/off-grid-ai/desktop, or clone and build it.
Install and launch it, then download a chat model from the built-in browser.
Create a .skills folder and drop in a proofread.md or a proofread/SKILL.md.
Type /proofread in chat to run it.
Add a trigger to a skill to have it run on a schedule, on a keyword, or on an event.

What's coming

A richer trigger set for more events around capture and approvals.
Sharing skill folders across your paired devices over the mesh.
A library of starter skills to fork and edit.

Actions: what to do, and what Off Grid proposes. Always your call.

FAQ

Q: Is it really free?

Yes. The app is free and open-source under AGPL-3.0. No account, no subscription, no API keys.

Q: How is a skill different from just pasting a prompt?

A skill is saved, named, and callable with a slash command. You write it once instead of retyping the prompt every time, and a skill can also run on its own with a trigger.

Q: Do skills work offline?

Yes. Skills are files on your disk and the model runs on your machine. The network is not needed.

Q: What can trigger a skill automatically?

A daily schedule at a local time, a keyword in a newly captured observation, or an event such as a new calendar entry or a new approval.

Q: Can a skill use my connected tools?

Yes. A triggered skill can use your MCP connector tools while it runs its action prompt.

Q: Is my data private?

Yes. Skill files, the model, and triggers all stay on your machine. There is no telemetry and the source is open.

Save your prompts as skills and let your local AI run them for you.

GitHub →

How to Build a Local AI Second Brain on Your Desktop in 2026 (100% Offline)

Mohammed Ali Chherawalla — Thu, 25 Jun 2026 05:19:39 +0000

A modern laptop GPU can run a capable language model and read text off a screenshot in the time it takes you to switch windows. That power sits idle while you pay a monthly subscription to a note-taking app that stores your life on someone else's server. Off Grid AI Desktop is a free, open-source app that turns your own machine into a second brain that builds itself.

GitHub →

Free, open-source, runs offline. No account, no API key, no data leaving your machine.

Day lays out your meetings, suggested actions, and to-dos in one place.

The problem with note-taking apps

A second brain is only as good as what you put into it. The catch is that putting things into it is work, and work is the thing you skip when you are busy. So the notes go stale, the system rots, and you are back to forgetting what you read last Tuesday.

The fix is to stop relying on manual capture. Off Grid AI Desktop watches your work, with your explicit permission, and writes the notes for you. You decide when it is on. It does the recording, reading, and summarizing. Your second brain fills itself.

What You Need

This is the heaviest of the local AI features, since it runs a chat model continuously. Give it room.

Minimum

macOS: Apple Silicon (M1), 16 GB RAM, macOS 13+
Windows: NVIDIA GPU (CUDA) or a GPU with Vulkan, 16 GB RAM, Windows 10+
About 10 GB free disk for the app and a small model

Recommended

macOS: M2 Pro or newer, 24 GB or more unified memory
Windows: NVIDIA GPU with 8 GB+ VRAM, 32 GB RAM
A fast SSD, since the memory database grows over time

The chat model is a quantized GGUF file. A larger model distills cleaner observations, so more memory pays off here.

How the capture-to-memory loop works

This is the part nothing else does the same way. The loop has five stages, and every one runs on your machine.

Capture. You turn on screen capture, per device, opt-in. A visible recording indicator stays on the whole time, so you always know it is running. Nothing is captured silently.
OCR. Each captured frame is read for text on-device. The pixels become words.
Distill. The local LLM reads that text and writes short observations, then pulls out the people, projects, companies, and concepts it mentions.
Store. Observations and entities go into a local database on your disk. No cloud, no sync to a vendor.
Reflect. You browse the result through several views, described below.

Capture is always something you switch on, with a light showing while it runs. It is never the default and never hidden.

What Off Grid AI Desktop Can Do

Once the loop is running, your second brain shows up in five places. Each one is a different angle on the same captured memory.

Day. A journal of your day, written for you in time blocks. Glance back and see what you actually worked on, not what you meant to.

Entities. A private CRM for everything, not just people. Projects, companies, and concepts each get a record, with merge, hide, and hierarchy controls and synthesis summaries the model writes from your captured context. The colleague you talked to last week is a record. So is the project you keep half-remembering.

Replay. A scrubbable movie of your day. Drag through time and see what was on screen, the way you would scrub a video.

Reflect. Trends across your Day and Week. Where your focus went, how much you context-switched, what pulled your attention. Patterns you cannot see from inside a busy day.

Actions. Action items the model detects in your communication, gathered for you to review. They are never sent on your behalf. You decide what to do with each one.

Everything in these views came from frames you chose to capture, read and summarized on your hardware.

How Hardware Acceleration Works

Running a chat model continuously is what makes the hardware matter.

On macOS, the model runs on Apple Silicon with Metal acceleration, and unified memory lets the CPU and GPU share one pool. That is why an M-series machine can keep a model resident and distill in the background without grinding.

On Windows, acceleration comes from CUDA on NVIDIA GPUs or Vulkan on a wider range of cards, with a CPU fallback for machines without a compatible GPU. More VRAM means a larger, sharper model.

The models are quantized GGUF files, compressed so they fit in consumer RAM or VRAM. Quantization, plus on-device OCR, is what makes a self-building memory practical on a desktop you already own.

Tips for a Cleaner Memory

A few habits keep your second brain useful instead of noisy.

Capture in sessions, not all day. Turn it on for deep work and off for breaks. You get richer observations from focused time and less clutter from idle browsing.

Tidy your entities now and then. Merge duplicates, hide the noise, set a hierarchy. A few minutes of cleanup makes the synthesis summaries far more accurate.

Pick a model that fits your RAM with headroom. If distillation lags, drop to a smaller quantization. The loop should feel like it is keeping up, not falling behind.

Review Actions on a schedule rather than reacting to each one. They are a queue you control, not an inbox firing at you.

Privacy: Stronger Than Cloud Note-Taking

A cloud note-taking app stores your work on its servers, indexes it, and ties it to your account. A second brain like this, built on captured screen content, would be a serious thing to hand to a vendor. So nothing here does.

Off Grid AI Desktop keeps every frame, observation, and entity on your disk. The app is AGPL-3.0 open source, so you can read exactly what it captures and where it stores it. No telemetry, no account, no upload. Capture only runs when you turn it on, with a visible indicator the whole time. Pull the network cable and your second brain keeps working.

Getting Started

Open the GitHub repo and download the latest release for macOS or Windows, or clone and build from source.
Install and launch Off Grid AI Desktop.
Download a chat model from inside the app.
Turn on screen capture, grant the OS permission, and confirm the recording indicator is showing.
Work for a while, then open Day, Entities, Replay, and Reflect to see what was built.

No sign-up, no key, no cloud account.

What's Coming

Cross-device sync so memory from your laptop and desktop join up
Unified search across Day, Entities, and captured observations
More capture sources beyond the screen
Richer Reflect trends over longer time spans

Reflect shows where your attention actually went across the day.

Replay is a scrubbable movie of your day, captured on-device.

Entities: a private CRM for the people, projects, and topics in your work.

FAQ

Q: Is it really free?

Yes. The app is free and open source under AGPL-3.0. The capture-to-memory loop is part of the open core.

Q: Does it work offline?

Yes. OCR, distillation, and storage all run on your machine, so it works with no network.

Q: Is the screen capture always on?

No. It is opt-in, per device, and only runs when you turn it on, with a visible recording indicator the whole time.

Q: How much RAM do I need?

16 GB is the floor because a chat model runs continuously. 24 GB or more lets you run a larger model for cleaner observations.

Q: Does it work on Windows as well as Mac?

Yes. macOS uses Metal and unified memory, Windows uses CUDA or Vulkan, and both fall back to CPU.

Q: Is any of my captured data uploaded?

No. There is no server. Frames, observations, and entities stay on your disk, and Actions are never sent on your behalf.

Build a second brain that builds itself, and keep every frame of it on your own machine.

GitHub →

How to Expose On-Device AI Models as MCP Tools (Local MCP Server, No Cloud)

Mohammed Ali Chherawalla — Thu, 25 Jun 2026 05:19:33 +0000

Your laptop can run a chat model, a vision model, a diffusion model, and a transcriber on its own GPU. Most setups still route every tool call through a cloud model that bills per token and reads everything you send. Off Grid AI Desktop is a free, open-source app that runs those models on your Mac or PC and serves them to any MCP client through a local MCP endpoint.

GitHub ->

Free, open-source (AGPL-3.0), runs offline.

The local gateway: one OpenAI-compatible API at 127.0.0.1:7878 for every modality.

What MCP is, in two paragraphs

The Model Context Protocol is a standard way for an AI client to call external capabilities. A client lists the tools a server offers, then invokes them with structured arguments and reads structured results. It is JSON-RPC underneath. The point is that any compliant client can use any compliant server without custom glue.

Most MCP servers wrap a SaaS API: a Notion server, a GitHub server, a database server. The tool runs somewhere remote and the data travels there. Off Grid AI Desktop does the opposite. It wraps your own on-device models as MCP tools. The capability and the data both stay on your machine.

The short version

Off Grid AI Desktop serves MCP at http://127.0.0.1:7878/mcp. It is Streamable HTTP, stateless JSON-RPC. The local chat, vision, image, speech, and embedding models show up as MCP tools. An MCP client connects, lists those tools, and calls them. Every call runs on-device. Nothing routes through a server anyone else owns.

What you need

The app bundles the inference engines, so the requirement is hardware to run the models.

Tier	Mac	Windows	RAM/VRAM	Disk
Minimum	Apple Silicon M1	NVIDIA GPU or modern CPU	16 GB	20 GB free
Recommended	M3 / M4	RTX with 8 GB+ VRAM	32 GB	40 GB+ free

Models are quantized GGUF that fit consumer memory. Mac uses Metal over unified memory. Windows uses CUDA or Vulkan, with CPU fallback.

How it works

The same app exposes an OpenAI-compatible HTTP API at http://127.0.0.1:7878/v1 and an MCP endpoint at http://127.0.0.1:7878/mcp. The HTTP API is for code that already speaks OpenAI. The MCP endpoint is for clients that speak MCP and want to treat your local models as tools inside a larger agent loop.

Both bind to loopback only. The server answers your own machine. There is no key, no account, no cloud hop.

Step 1: List the tools

MCP starts with a tools/list call. The server returns the local models as callable tools. Here is the raw JSON-RPC over the Streamable HTTP endpoint.

curl http://127.0.0.1:7878/mcp \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -d '{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "tools/list"
  }'

You get back a list of tools backed by your on-device models: chat, vision, image generation, speech, and embeddings. The endpoint is stateless, so each request stands alone. There is no session to keep open.

Step 2: Call a tool

A tools/call invokes one tool by name with arguments. This calls the local chat model as an MCP tool.

curl http://127.0.0.1:7878/mcp \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -d '{
    "jsonrpc": "2.0",
    "id": 2,
    "method": "tools/call",
    "params": {
      "name": "chat",
      "arguments": {
        "messages": [
          {"role": "user", "content": "Draft a one-line commit message for a README fix."}
        ]
      }
    }
  }'

The model runs on your GPU and returns the result through the MCP envelope. Swap the tool name and arguments to call vision, image generation, speech, or embeddings the same way.

Step 3: Point a client at it

Most MCP clients take a server URL in their config. Add the local endpoint as an HTTP MCP server.

{
  "mcpServers": {
    "offgrid-local": {
      "url": "http://127.0.0.1:7878/mcp"
    }
  }
}

Reload the client. The local models appear in its tool list. Now an agent running in that client can transcribe a file, describe an image, or generate one, and the work happens on your hardware.

Why route models through MCP at all

You can already hit the models over the HTTP API. MCP buys you something different. It lets a client that orchestrates many tools treat your private models as just more tools in the same loop. A planning agent can call a remote tool for one step and your local vision model for the next, through one protocol, with one tool list.

That keeps the sensitive steps local. The image you do not want to upload gets described by the on-device vision tool. The audio you do not want to send out gets transcribed by the on-device speech tool. The agent never has to leave the machine for those calls.

How the models share your memory

The app does not load every model at once. Each modality loads when its tool is called and offloads after. The chat model and the diffusion model never sit in RAM together. A 16 GB machine can back all the tools, one call at a time, without running out of memory.

Repeated calls to the same tool keep that model warm. A cold call pays a short load cost. For long-running tools like image generation, the underlying API supports async with a poll URL, so a slow render does not block.

Privacy: stronger than a hosted tool server

A hosted MCP server sees every argument you pass and every result it returns. It needs credentials and it logs traffic. The Off Grid AI Desktop MCP endpoint binds to 127.0.0.1, so it answers only your machine. There is no telemetry, no account, no key. The code is AGPL-3.0, so the behavior is auditable. Your prompts, images, and recordings stay on disk.

Getting started

Download or clone from github.com/off-grid-ai/desktop.
Install and launch the app.
Download a model for each modality you want to expose as a tool.
The MCP endpoint is live at http://127.0.0.1:7878/mcp. Run tools/list to confirm.
Add the URL to your MCP client config and reload.

The interactive HTTP docs at /docs and the spec at /openapi.json cover the companion /v1 API if you want both surfaces.

What's coming

Reach the MCP endpoint from other paired devices over your local mesh, so a phone or tablet can use your desktop's models as tools. Local only, no cloud.
More bundled models behind the tools.
Broader tool coverage as the on-device modalities grow.

FAQ

Q: Is it really free?

Yes. Free and open-source under AGPL-3.0. The MCP endpoint is part of the open core.

Q: Does it work offline?

Yes. Once models are downloaded, every tool call runs on-device with no network.

Q: Which MCP clients work?

Any client that supports an HTTP MCP server over Streamable HTTP. Point it at the URL and it lists the tools.

Q: Do I need an API key or account?

No. The endpoint checks neither. It is local-only by design.

Q: Can another machine reach my MCP endpoint?

Not today. It binds to loopback. Cross-device access over the local mesh is a separate, future feature.

Q: What models back the tools?

The same on-device models the app runs for chat, vision, image generation, speech, and embeddings. Run GET /v1/models to see which is active per modality.

Give your agent local tools that never phone home.

GitHub ->

How to Run a Local AI Image Studio on Your Desktop (SDXL, Z-Image, Offline)

Mohammed Ali Chherawalla — Thu, 25 Jun 2026 05:18:58 +0000

The GPU in a modern laptop can run the same image models that power paid services like Midjourney. That hardware sits idle while you pay a monthly fee to generate pictures on someone else's server. Off Grid AI Desktop is a free, open-source app that runs Stable Diffusion XL and Z-Image-Turbo directly on your Mac or PC.

GitHub →

Free, open-source, runs offline. No account, no API key, no telemetry.

On-device image generation in Off Grid AI Desktop. SDXL, Lightning, Turbo, and more, run from your chat.

What You Get Instead of a Subscription

You buy holes, not drill bits. Here is what a local image studio gets you over a cloud tool.

Your prompts never leave the machine. There is no per-image credit, no monthly cap, and no queue. You run as many generations as your hardware allows, at 2am, on a plane, with the WiFi off. The model weights live on your disk, so the service cannot be deprecated or rate-limited out from under you.

What You Need

The bigger the model, the more memory it wants. Two tiers cover most people.

Tier	macOS	Windows	RAM / VRAM	Free disk
Minimum	Apple Silicon M1	NVIDIA GTX 1660 or any modern iGPU	8 GB	15 GB
Recommended	M2 / M3 / M4	RTX 3060 or better (8 GB+ VRAM)	16 GB+	30 GB

On the minimum tier you run SD 1.5 and the few-step models comfortably. SDXL at full resolution wants the recommended tier. CPU-only generation works on Windows as a fallback, but it is slow, so expect minutes per image rather than seconds.

What Off Grid AI Desktop Can Do

The image studio is built on stable-diffusion.cpp, the same C++ engine that runs Stable Diffusion without Python or a server account.

You get text-to-image and image-to-image. Type a prompt and generate from scratch, or drop in a reference image and steer it with strength and a new prompt. A live per-step preview shows the picture forming as the model denoises, so you see early whether a generation is going anywhere before it finishes. Progress and an ETA sit next to it, and a cancel button stops a run you do not want.

Finished images open in a lightbox for full-size viewing. Everything you make lands in an artifacts gallery, so your history is browsable on disk instead of locked in a web account. Style presets like Sketch, Cinematic, and Anime prepend tuned prompt fragments, so you get a consistent look without memorizing keyword soup.

Which Models to Use

Different models trade speed for fidelity. Pick one to match your hardware and your patience.

Model	What it is	Best for
SD 1.5 / 2.1	The original compact models	Low-memory machines, fast iteration
SDXL	Higher resolution, stronger composition	Final quality on the recommended tier
SDXL-Lightning	SDXL distilled to a few steps	Near-SDXL quality, much faster
Z-Image-Turbo	2026 flagship, around 8 steps	Modern quality with a short step count

A good workflow is to draft with a few-step model, then re-run the prompt you like on full SDXL for the keeper. SDXL-Lightning and Z-Image-Turbo cut the step count dramatically, which is what makes local generation feel responsive instead of a chore.

How Hardware Acceleration Works

The models are large, so they get quantized. Quantization stores the weights at lower precision, which shrinks the file and the memory footprint enough to fit on consumer hardware. That is the change that made on-device image generation viable.

On macOS, generation runs through Metal against Apple Silicon's unified memory, so the GPU reads the same RAM as the CPU with no separate VRAM ceiling to fight. On Windows, you get CUDA on NVIDIA cards or Vulkan on others, with a CPU path when there is no usable GPU. The app picks the accelerated path it finds.

Keeping It Fast

Step count is the biggest lever on speed. A 30-step SDXL run takes far longer than an 8-step Turbo run, and for drafting you rarely need the extra steps. Start low and raise it only for the final image.

Resolution is the second lever. Doubling the side of an image roughly quadruples the work. Generate at a smaller size while you tune the prompt, then bump the resolution once the composition is right. If a generation is clearly wrong by step three, the cancel button saves you the rest.

Privacy: Stronger Than a Cloud Image Tool

A cloud image service sees every prompt you type and every reference image you upload. Some reserve the right to train on what you make. Off Grid AI Desktop sees none of it, because there is no server. The model runs locally, the output saves to your disk, and nothing is logged off-machine.

It is AGPL-3.0 licensed, so the code is auditable. There is no account, so there is no profile to leak. Run it on an air-gapped machine and every feature still works.

Getting Started

Download or clone from the GitHub repo.
Install and launch the app on your Mac or PC.
Open the Models browser and download an image model (start with SDXL-Lightning or Z-Image-Turbo).
Open the image studio, type a prompt, pick a style preset if you want one.
Generate. Watch the per-step preview, then open the result in the lightbox.

git clone https://github.com/off-grid-ai/desktop
cd desktop
npm install
npm run dev

What's Coming

More image models as new open-weight releases ship.
Cross-device sync so your artifacts gallery follows you between machines.
Unified search across generated images and the rest of your captured work.

Projects keep related chats, uploaded documents, and generations together.

FAQ

Q: Is it really free?

Yes. The app is free and open-source under AGPL-3.0. There are no image credits and no subscription.

Q: Does it work offline?

Yes. Once a model is downloaded, generation runs with no network connection.

Q: Which models can I run?

SD 1.5, SD 2.1, SDXL, SDXL-Lightning, and Z-Image-Turbo. You download them through the built-in Models browser.

Q: How much RAM do I need?

8 GB runs the compact and few-step models. 16 GB or more is comfortable for full SDXL.

Q: Does it run on Windows and Mac?

Both. macOS uses Metal on Apple Silicon. Windows uses CUDA, Vulkan, or a CPU fallback.

Q: Are my prompts private?

Yes. Prompts and images stay on your machine. There is no server to send them to.

Run a private image studio on hardware you already own. GitHub →

Connector Support in Off Grid AI Desktop: Private, Approval-Gated Integrations

Mohammed Ali Chherawalla — Thu, 25 Jun 2026 05:18:53 +0000

The Model Context Protocol lets an AI assistant talk to your tools through a standard interface, and the spec is open. Most products that adopt it still route your tool data through their cloud and let the model act on your behalf with no record. Off Grid AI Desktop runs MCP connectors locally: the connector fetches, the on-device model reasons, and every proposed action waits in an approval queue before anything happens.

GitHub → Free, open-source, runs offline.

Connectors in Off Grid AI Desktop. Authorized actions run only after you approve them.

What a connector does here

A connector is a bridge between the local model and an outside tool. Off Grid AI Desktop speaks MCP over two transports: stdio for local processes and HTTP for remote servers. That covers most MCP servers in the wild.

You add a connector, enable it, and test it. The app discovers the tools that connector exposes. From then on, the on-device model can call those tools inside a normal chat.

Verified connectors today include Notion, Linear, and Jira/Confluence. The framework is general MCP, so it is not a fixed list of three. If a server speaks MCP over stdio or HTTP, you can wire it in.

What you need

This runs on macOS and Windows.

Tier	Hardware	What you get
Minimum	16 GB RAM, 10 GB free disk	Connectors work; a small local model reads and summarizes fetched data
Recommended	24 GB+ unified memory (Mac) or 8 GB+ GPU (NVIDIA/Vulkan)	A larger local model handles multi-tool chats and longer context

Connectors themselves are cheap. The work sits with the local model that reads what they return.

The privacy model: fetch, reason, approve

Three steps, and the boundary between them is the whole point.

Fetch. The connector pulls data from the outside tool. A Notion page, a list of Linear issues, a Jira ticket. That data comes into the app.

Reason. The local model reads it. This is the step that usually happens on a vendor's server. Here it happens on your machine. Your Notion contents and your issue tracker do not pass through a model we host, because we host no model. There isn't one.

Approve. When the model proposes an action that changes something (creating an issue, editing a page, posting a comment), the action does not run. It lands in an approval queue. You read what it wants to do, then approve or reject. Every approval is written to an audit log.

Nothing executes without a logged approval. The model can suggest all day. It cannot act until you say so, and you can see afterward exactly what ran and when.

How it works end to end

outside tool (Notion / Linear / Jira)
        |
   connector (MCP: stdio or HTTP)  -- fetch
        |
   local model on your machine     -- reason
        |
   proposed action                 -- approval queue
        |
   you approve or reject           -- audit log

Read steps flow freely. The model can fetch and reason without bothering you. The gate is on writes. That is the line that keeps an autonomous loop from doing something you did not intend.

Authentication without a central OAuth client

Most integration products make you authorize through their cloud OAuth app. That app becomes a standing key to your account, held by a third party.

Off Grid AI Desktop uses local-friendly auth instead. Connectors authenticate with Dynamic Client Registration over OAuth, or with a token you hold. There is no central OAuth client that we own sitting between you and your tools. The credential lives on your machine and is used from your machine.

So the connection is yours in both directions. The data comes to you, and the key that opens it stays with you.

What you can do with it

Ask questions across your tools. "What Linear issues are assigned to me this sprint, and which ones touch the billing service." The model fetches and reasons locally.
Draft changes and review them. The model proposes a new ticket or a page edit. You approve it from the queue.
Keep a record. The audit log shows every action that ran, so you can answer "what did this thing actually do" with a fact, not a guess.

Each of these turns a connector from a data pipe into a working assistant that still answers to you.

A practical tip: scope connectors narrowly

Give each connector the smallest access that does the job. A read-mostly token on your issue tracker is enough for "summarize my open tickets." You do not need write access to ask questions.

When you do want the model to make changes, lean on the approval queue rather than pre-granting broad write scopes everywhere. The queue is the safety layer. Use it. Read each proposed action before you approve, especially the first few times you trust a new connector with writes.

Privacy: stronger than a cloud integration platform

A cloud integration platform holds an OAuth key to your account, routes your tool data through its servers, and runs the model there too. Three points where your data sits on someone else's hardware.

Off Grid AI Desktop collapses all three onto your machine. The credential is local. The fetched data stays local. The model that reads it is local. The app is open source under AGPL-3.0, takes no account, and sends no telemetry. Connectors reach out to the tools you name and nowhere else.

Getting started

Clone or download from GitHub: https://github.com/off-grid-ai/desktop
Install the app on macOS or Windows.
Open the connectors area and add a connector (stdio or HTTP), for example Notion, Linear, or Jira/Confluence.
Enable and test it. The app discovers the available tools.
Open a chat and ask a question that uses the connector. When the model proposes a write, approve it from the queue.

What's coming

More verified connectors on top of the general MCP support.
Per-connector permission profiles so you can pin read-only vs write access per tool.
Cross-device sync so a connector you set up on one machine is available on another, still without a cloud middleman.

Actions: what to do, and what Off Grid proposes. Always your call.

FAQ

Q: Is it really free?

Yes. Off Grid AI Desktop is free and open source under AGPL-3.0. No account, no card.

Q: What transports does it support?

MCP over stdio (local processes) and HTTP (remote servers). The framework is general MCP, not a fixed integration list.

Q: Which connectors are verified?

Notion, Linear, and Jira/Confluence today. Because the support is general MCP, other servers that speak stdio or HTTP work too.

Q: Does the model act on my behalf automatically?

No. Any action that changes something goes to an approval queue. Nothing executes without a logged approval, and the audit log records what ran.

Q: Where does my tool data go?

Into the app on your machine. The connector fetches it, the local model reasons over it on-device, and it does not pass through a server we host.

Q: How does auth work without a central OAuth app?

Connectors use Dynamic Client Registration over OAuth or a token you hold. There is no central OAuth client that we own. The credential stays on your machine.

Connect your tools, keep the keys and the data on your machine, and approve every action yourself. Off Grid AI Desktop on GitHub →