Ollama Local vs Cloud: Where Should Your Coding Agent Run?

#ai #ollama #codingagents #tutorial

A coding agent can read a repository, inspect errors, and propose a patch. Before enabling that workflow, there is a less exciting question worth answering: where does the model process the code and prompt?

That question matters because “runs through Ollama” does not necessarily mean “runs on my machine.” Ollama supports both downloaded local models and models processed in its cloud.[1][2] Ollama says its cloud service processes prompts and responses but does not store or log that content and never trains on it.[5] It collects basic account information and limited usage metadata.[5] Ollama also documents a local-only mode that disables cloud features.[5] Those are different operating choices, even when the interface looks familiar.

The current interest in open-weight models makes this distinction timely. Ollama’s Qwen 3.5 library lists several model sizes and quantizations, including a 9B variant with a listed artifact-size range of 6.6 to 7.6 GB and a 27B variant listed at 17 to 20 GB.[4] Those are download-size ranges, not a promise that a machine with exactly that much memory will run them comfortably.[4] A model you can download may still be impractical on the computer you have.

This is a comparison of deployment paths, not a benchmark claiming one model is smarter. It draws on public Ollama documentation and model listings, not a hands-on performance test. The useful question is which trade-off fits the project’s code, available hardware, and security boundary.

The short version

Choose local inference when keeping source code and prompts off a model provider’s service is a hard requirement, you can provision enough hardware, and you can accept responsibility for model selection and updates.[2][5]
Choose cloud inference when you want to avoid downloading and operating a model, your approved data-handling rules permit sending prompts to a hosted service, and the available usage limits and model catalog fit your work.[1]
Choose a hybrid policy when tasks differ. Keep sensitive repositories local, and allow cloud inference only for approved projects or sanitized tasks.[1][5]

The model name does not settle the privacy question. The execution path does.[1][5]

What “local” and “cloud” mean in Ollama

Local mode: Ollama runs a model on a machine you control.[2][5] Its quickstart describes selecting a local model, and its OpenAI-compatible API example uses a loopback address for the local server.[2][3] The inference request is sent to that local server.[3] You still need to think about operating-system telemetry, agent extensions, logs, backups, and any tools the agent calls. Local inference limits what is sent to a remote model endpoint; it does not automatically make the whole agent workflow private.[5]

Cloud mode: Ollama processes the model request in its cloud.[1] The documentation says no model or app download is required for direct cloud requests, and shows authentication with an API key.[1] That is not the same as “nothing leaves the machine.” If a policy requires source code to remain on-premises, a cloud model is not an equivalent option just because the client library is familiar.[1][5]

A key distinction: privacy is not a model property alone. It depends on the destination of the request, the data included in context, the provider’s terms, account configuration, and the agent’s other integrations.[1][5] Review the current provider terms and organizational policy before sending code or credentials.

Compare the trade-offs that affect real work

Data boundary. A local model can keep prompts on your infrastructure when configured for local inference and when cloud features are disabled.[5] Confirm the network path rather than relying on a UI label. Ollama documents a setting for disabling cloud features.[1][5] With cloud inference, prompts are processed by the service.[1][5] The documentation says prompt content is not stored or logged and is not used for training, while limited account and usage metadata is collected.[5]

Hardware and latency. Local execution depends on available memory, compute, quantization, concurrent users, and model size. The Ollama library lists a range of Qwen 3.5 sizes, from small variants to larger ones; even one family can span very different hardware requirements.[4] The listed artifacts include a 9B model between 6.6 and 7.6 GB and a 27B model between 17 and 20 GB.[4] Cloud execution shifts model serving to the provider, but depends on network availability, service limits, and the provider’s current catalog.[1] Do not compare model-artifact size with a device’s total memory as if they were interchangeable numbers. Leave room for the operating system, context, and the coding agent itself.

Operational work. A local setup makes the operator responsible for choosing a suitable model, downloading it, allocating hardware, updating it, monitoring resource use, and checking that the local endpoint is not exposed beyond its intended network.[2][3] Cloud execution reduces that model-serving work, but introduces account, key, service-availability, usage-limit, and data-review responsibilities.[1] Ollama’s docs require an API key for direct cloud access and caution that secrets should stay out of browser code and source control.[1][3] Neither choice is maintenance-free; the work moves to a different layer.

Model choice and compatibility. Ollama documents OpenAI-compatible clients for both local and direct cloud access, but explicitly notes that its support covers a subset of the OpenAI API.[3] Cloud models must be selected from Ollama’s cloud catalog or API listing, while local requests use a locally available model name.[1][3] A shared client interface can ease switching, but it does not guarantee identical model behavior or support for every API feature.[3] Validate tool calls, structured outputs, streaming, and context handling with the exact model and client version you intend to use.[3]

Cost. Local inference has hardware, power, and operational costs even if there is no per-request bill. Cloud usage can be billed or limited according to the provider’s current plans and usage policy.[1] Because those details can change, do not hard-code a “local is free” or “cloud is cheaper” conclusion into an architecture decision. Estimate request volume, hardware utilization, and staff time, then check current plan details before committing.[1]

Try the same client shape with both paths

The OpenAI Python client can point to a local Ollama server.[3] The documentation’s local example uses an ollama placeholder API key that the local server ignores.[3]

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1/",
    api_key="ollama",  # required by the client, ignored locally
)

response = client.chat.completions.create(
    model="qwen3.5:9b",
    messages=[
        {"role": "user", "content": "Explain this test failure without changing files."}
    ],
)

print(response.choices[0].message.content)

The model tag above is listed in Ollama’s Qwen 3.5 library.[4] Pull the model before using it, and confirm that the selected quantization fits the hardware.[4] Treat the prompt as a simple connectivity example, not proof that this model is suitable for autonomous code changes.

For direct cloud requests, Ollama’s documentation uses its cloud API base URL and an OLLAMA_API_KEY environment variable.[1][3] It gives gemma4:31b as an example cloud model name.[1][3]

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://ollama.com/v1",
    api_key=os.environ["OLLAMA_API_KEY"],
)

response = client.chat.completions.create(
    model="gemma4:31b",  # example from Ollama's cloud documentation
    messages=[
        {"role": "user", "content": "Explain this test failure without changing files."}
    ],
)

print(response.choices[0].message.content)

This demonstrates a deployment distinction, not a controlled model comparison: the examples use different models.[3] For a fair quality evaluation, compare models on the same project tasks, with the same context, tool permissions, and scoring rules. Do not infer that cloud is more capable just because the example name differs, or that local is private if the agent sends source elsewhere through another integration.[3][5]

Connect a coding agent without widening its authority

Ollama’s quickstart documents launch paths for Claude Code, Codex CLI, and OpenCode, and says to choose a model when prompted.[2] That convenience does not decide what the agent may read or modify. Start with a disposable project or a low-risk task and keep the model-selection experiment separate from permission decisions.

A cautious evaluation can look like this:

Choose one representative task. Use a bug or test failure that can be evaluated without exposing secrets or customer data.
Use a clean worktree. Give the agent a branch or disposable copy, not a directory containing unrelated credentials or private notes.
Start read-only. Ask the agent to explain a failure or suggest a patch before granting write or shell permissions.
Record the exact model and route. Note whether the request went to the local server or Ollama’s cloud, plus the model tag and client version.
Check the result, not the confidence. Compare the proposed change with tests, expected behavior, and a human review.
Increase authority deliberately. If you later allow edits, keep approval for commands, network access, and irreversible actions.

The API compatibility layer is a convenience, not a security boundary.[3] A client configured with a local base URL can still coexist with an agent that uploads context to another service.[3][5] Check the full request path and every integration the agent uses.[3][5]

A decision checklist you can save

Before choosing local or cloud for a coding agent, answer these questions:

Data policy: Is code allowed to be sent to the selected hosted service? Does the policy distinguish source code, secrets, personal data, or regulated data?
Network proof: Can you confirm which endpoint receives prompts? Can you disable cloud features or restrict outbound traffic where required?
Hardware test: Does the chosen local model fit with the context length and concurrency you need, not just the downloaded artifact size?
Quality test: Have you measured the same coding tasks with the same acceptance criteria, rather than relying on model labels or community rankings?
Tool safety: Are file writes, shell commands, and network calls separately permissioned and reviewed?
Reliability plan: What happens when the network is down, the local machine is busy, a cloud model is retired, or usage limits are reached?
Cost model: Have you compared hardware and operating effort against current cloud usage terms for your expected workload?
Fallback rule: Which tasks may switch routes, and which must never leave the approved environment?

If those answers differ by project, do not force one route for every developer task. A policy can classify work: public sample projects may use cloud; private code may use local inference; credentials and production data may remain outside agent context entirely.[1][5] The policy should be visible and testable, not left to a developer noticing a model badge.

Verdict: choose the boundary first, then the model

Ollama Local is the stronger fit when a local-only data path is required and the operator is willing to operate the hardware and model.[2][5] Ollama Cloud is the more convenient fit when hosted processing is approved and you want to avoid downloading and serving the model yourself.[1] The technical interface overlaps, but the trust boundary and operational responsibilities do not.[1][2][3]

For a coding agent, that is the decision to make first. Then benchmark candidate models on representative tasks, with a fixed test set and carefully scoped tools. Ollama’s model listing shows that available model variants differ in artifact size, so an open-weight release does not by itself establish whether a model will run well on local hardware.[4] The inference route and agent integrations determine where context is sent.[1][3][5]

If you build with AI agents and developer tools, subscribe for more practical comparisons and implementation guides. Have you tried local or cloud inference with a coding agent? What did you choose, and what shaped that decision?