DEV Community

Jackson Ly
Jackson Ly

Posted on

Point any app at a local LLM on your Mac (OpenAI-compatible endpoints)

#ai #llm #mac #productivity

Most apps that grew an "AI" feature in the last two years talk to one of a handful of cloud APIs, and almost all of them speak the same dialect: the OpenAI Chat Completions format. That one detail is the reason you can pull the cloud out and run the whole thing locally on a Mac without the app ever noticing.

Here is the trick, why it works, and the gotchas that bite.

The one interface everything agrees on

OpenAI's /v1/chat/completions endpoint became the de facto standard. So when an app lets you "use your own key" or "set a custom base URL," it is almost always going to POST to {base_url}/chat/completions with a JSON body of messages and read back the same shape. It does not care what is on the other end, only that the response matches.

Local runners leaned into this. Both popular Mac ones expose exactly that endpoint:

  • Ollama serves an OpenAI-compatible API at http://localhost:11434/v1 (its native API lives on /api, but the /v1 path speaks the OpenAI dialect).
  • LM Studio has a built-in server you switch on from the Developer tab, serving on http://localhost:1234/v1.

So "make this app local" usually reduces to: point its base URL at one of those, put any non-empty string where it wants an API key, and pick a model you have pulled.

The 60-second version

Ollama:

brew install ollama        # or the .dmg from ollama.com
ollama serve &             # server on :11434
ollama pull llama3.1:8b    # pull a model once
Enter fullscreen mode Exit fullscreen mode

Confirm it speaks OpenAI:

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.1:8b",
    "messages": [{"role": "user", "content": "say hi in 3 words"}]
  }'
Enter fullscreen mode Exit fullscreen mode

If that returns a choices[0].message.content, any OpenAI-compatible client can use it. In the app, set:

  • Base URL: http://localhost:11434/v1
  • API key: ollama (or literally anything; it is ignored)
  • Model: llama3.1:8b

LM Studio is the same idea with a GUI: load a model, toggle the server on, and use base URL http://localhost:1234/v1.

Pointing real tools at it

The pattern shows up everywhere once you look for it. The official OpenAI SDKs are the clearest example: change one field.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
r = client.chat.completions.create(
    model="llama3.1:8b",
    messages=[{"role": "user", "content": "summarize this in one line: ..."}],
)
print(r.choices[0].message.content)
Enter fullscreen mode Exit fullscreen mode

Same code, cloud or local. Only the base_url changed. In JavaScript it is the baseURL option; in a lot of CLI and editor tools it is an OPENAI_BASE_URL environment variable. Some end-user apps expose it too: DEVONthink 4, for instance, lets its Chat point at Ollama or LM Studio directly, so search and summarize run over your own documents with nothing leaving the machine.

The honest gotchas

It is not always drop-in. The places it breaks, roughly in order of how often they catch people:

  1. Context window. A local model often defaults to a small context (Ollama defaults to 2048 tokens unless you raise num_ctx). If an app sends a big document and the model only reads the last 2k tokens, it looks like the model "ignored" your content. Raise the context in the model config, or send smaller chunks.
  2. Missing features. Some apps assume function calling, JSON mode, or vision. Not every local model or runner supports all of them, and the failure is often silent. Check the runner's compatibility notes for the specific feature you depend on.
  3. Streaming quirks. Most local servers stream fine, but a few clients expect particular SSE framing. If a response hangs, turn streaming off in the client as a test to isolate it.
  4. Model quality. A 7-8B model is not GPT-class. For classification, summarizing, tagging, and search over your own material it is often plenty; for long chains of reasoning it will disappoint. Match the job to the model rather than expecting parity.
  5. RAM. Everything is bounded by unified memory on a Mac. 16GB comfortably runs a capable 7-8B model; below that, stay in the 3B range and keep the context modest.

Why bother

Two reasons that actually matter. It is free after the download, so there is no per-token meter running while you iterate on a prompt fifty times. And nothing leaves the machine, which for anything sensitive (client data, personal notes, a private codebase) is the difference between "I can use AI on this" and "I am not allowed to."

The best part is that because it is one interface, you do not have to commit. Keep a cloud base URL for the genuinely hard reasoning and a local one for the bulk, private, high-volume work, and switch between them by changing a single string.

Written with AI assistance and edited by a human. Endpoint details reflect public docs as of July 2026 and move quickly, so check each project's current docs for exact paths and defaults.

Top comments (0)