DEV Community

AI Engineering Foundry
AI Engineering Foundry

Posted on

Ollama with C# and .NET: Run a Local LLM in ASP.NET Core

#ai #webdev #programming #seo

Run a Local LLM with Ollama from C# and ASP.NET Core

Run a local LLM with Ollama and call it from C#: HttpClient, OllamaSharp and IChatClient, streaming, JSON output, embeddings, and troubleshooting.

run-ollama-with-dotnet`

Ollama with C#

local LLM in .NET, OllamaSharp, IChatClient, Microsoft.Extensions.AI, Ollama ASP.NET Core, Ollama embeddings C#, Ollama streaming C#, run LLM locally C#

Running an LLM locally is useful for development, experimentation, and privacy-sensitive workloads where sending data to a third-party API isn't an option.

In this guide we connect Ollama to an ASP.NET Core application, first with nothing but HttpClient, then with the approach we use in every .NET project on this site: OllamaSharp behind the Microsoft.Extensions.AI IChatClient interface. Along the way we cover streaming, typed JSON output, embeddings, choosing a model, and the two settings that silently broke our results until we found them.

In short

  • Install Ollama, ollama pull a model, and it serves an HTTP API on port 11434.
  • For a quick test, HttpClient + POST /api/generate is enough (Steps 2-3).
  • For real apps, register OllamaApiClient from OllamaSharp as an IChatClient. Your code stays the same if you later switch to OpenAI, Azure OpenAI or Claude.
  • Set a longer HTTP timeout, raise the context window, and use 127.0.0.1 instead of localhost on Windows.

Architecture

text
ASP.NET Core endpoint
↓
IChatClient / IEmbeddingGenerator (Microsoft.Extensions.AI)
OllamaApiClient (OllamaSharp)
↓ HTTP, 127.0.0.1:11434
Ollama server → model on GPU or CPU (qwen2.5-coder:7b, nomic-embed-text, …)

Your application code only depends on the interfaces at the top. Ollama, the model and the hardware underneath can change without touching your endpoints.

Step 1: Install and run Ollama

Install Ollama from ollama.com (Windows, macOS, Linux), then pull a model and start the server:

shell
ollama pull llama3.1
ollama serve

Ollama exposes a local HTTP API, by default on http://localhost:11434. A few things are easy to miss:

  • The server may already be running. On Windows and macOS the desktop app starts it, and the Linux installer sets up a service. If ollama serve fails because port 11434 is in use, that's why. Skip it.
  • Check it's up before writing C#: curl http://127.0.0.1:11434/api/tags or ollama list. If the model your code asks for isn't listed, every request fails.
  • Docker works too: docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama, then docker exec -it ollama ollama pull llama3.1. For NVIDIA GPUs add --gpus=all.

Choosing a model

Pick the model for the job, not the biggest one that fits:

Model Download Context What we use it for
qwen2.5-coder:7b 4.7 GB 32K Text-to-SQL and code generation
qwen2.5:7b 4.7 GB 32K Tool-calling experiments
llama3.1 (8B) 4.9 GB 128K General chat, Steps 1-3
nomic-embed-text 274 MB 2K Embeddings for RAG (Step 7)

The download size is roughly the minimum memory the weights need, and the context window needs more on top. Send one request, then run ollama ps. The PROCESSOR column shows 100% GPU, 100% CPU, or a split. A split means the model didn't fit in VRAM, and part of it runs on the CPU, which is much slower. 7B models are a good default for development, and also where we found real limits: in our tool-calling tests, a 7B model mostly answered directly instead of calling tools.

Step 2: Call it from ASP.NET Core

The smallest integration is a typed HttpClient that posts to /api/generate:

`csharp
public class OllamaClient(HttpClient http)
{
public async Task GenerateAsync(string prompt)
{
var response = await http.PostAsJsonAsync("/api/generate", new
{
model = "llama3.1",
prompt,
stream = false
});

    response.EnsureSuccessStatusCode();
    var result = await response.Content.ReadFromJsonAsync<OllamaResponse>();
    return result!.Response;
}
Enter fullscreen mode Exit fullscreen mode

}

public record OllamaResponse(string Response);
`

stream = false matters because /api/generate streams by default, and without it you'd get many JSON lines instead of one object. The reply also includes eval_count, prompt_eval_count and total_duration (in nanoseconds), which you can add to the record to log token counts and timings.

Step 3: Register the client

csharp
builder.Services.AddHttpClient<OllamaClient>(client =>
{
client.BaseAddress = new Uri("http://127.0.0.1:11434"); // avoid the IPv6 localhost penalty
client.Timeout = TimeSpan.FromSeconds(180); // the first request also loads the model
});

  • Timeout. HttpClient gives up after 100 seconds by default. A non-streaming request has to load the model (on the first call, or after Ollama unloads it, 5 minutes after the last request by default) and then generate the whole answer. That can take longer than 100 seconds.
  • 127.0.0.1, not localhost. When we traced the AI Database Agent, we found a 2-second IPv6 penalty on every new connection. Using the IPv4 address removed it.

Step 4: The better way: OllamaSharp + Microsoft.Extensions.AI

The hand-written client works, but you end up re-implementing chat history, streaming, tool calls and JSON schemas. In .NET there's a standard abstraction for this. Microsoft.Extensions.AI defines IChatClient (chat and streaming) and IEmbeddingGenerator (embeddings), plus middleware for logging, OpenTelemetry, caching and function invocation. OllamaSharp's OllamaApiClient implements both directly, so you don't need an adapter.

shell
dotnet add package OllamaSharp
dotnet add package Microsoft.Extensions.AI

Register it once in Program.cs:

`csharp
using Microsoft.Extensions.AI;
using OllamaSharp;

builder.Services.AddChatClient(sp =>
{
var http = new HttpClient
{
BaseAddress = new Uri("http://127.0.0.1:11434"),
Timeout = TimeSpan.FromSeconds(180),
};
return new OllamaApiClient(http, "qwen2.5-coder:7b");
})
.Use(inner => new ConfigureOptionsChatClient(inner,
o => o.AddOllamaOption(OllamaOption.NumCtx, 8192)))
.UseLogging();
`

  • OllamaApiClient(HttpClient, defaultModel) lets you control the timeout. The simpler new OllamaApiClient(new Uri(...), "model") constructor doesn't set one, so you get the 100-second default.
  • AddOllamaOption(OllamaOption.NumCtx, 8192) raises the context window on every request. This was the most important line in that project (see "Settings that matter").
  • .UseLogging() logs every request and response through your ILogger. In production you can add .UseOpenTelemetry() too.

Now inject IChatClient anywhere:

`csharp
app.MapPost("/chat", async (ChatRequest req, IChatClient chat, CancellationToken ct) =>
{
ChatResponse response = await chat.GetResponseAsync(req.Prompt, cancellationToken: ct);
return Results.Ok(new { answer = response.Text });
});

record ChatRequest(string Prompt);
`

For a system prompt or a multi-turn conversation, pass a list of ChatMessage objects instead of a string, and add the reply back with history.AddMessages(response). Ollama is stateless, so you send the whole history with every request.

Which .NET package should you use?

Option Status Use it when
OllamaSharp (OllamaApiClient) Active, implements IChatClient and IEmbeddingGenerator The default choice. Also exposes pulling and listing models, keep_alive, num_ctx.
Microsoft.Extensions.AI.Ollama Deprecated; NuGet says to use OllamaSharp Don't use it. Older tutorials still show it, with the old CompleteAsync method (now GetResponseAsync).
Semantic Kernel Ollama connector Still prerelease You already use Semantic Kernel. Otherwise IChatClient is simpler.
Raw HttpClient Always works Zero dependencies, or an endpoint no client wraps yet.
Ollama's OpenAI-compatible /v1 endpoint Supported You want to reuse OpenAI client code.

Checking the server and pulling a model from code

`csharp
var ollama = new OllamaApiClient(new Uri("http://127.0.0.1:11434"));

if (!await ollama.IsRunningAsync())
throw new InvalidOperationException("Ollama is not running on 127.0.0.1:11434.");

var models = await ollama.ListLocalModelsAsync();
if (!models.Any(m => m.Name.StartsWith("qwen2.5-coder:7b")))
{
await foreach (var status in ollama.PullModelAsync("qwen2.5-coder:7b"))
Console.WriteLine($"{status?.Percent}% {status?.Status}");
}
`

Step 5: Stream the response

With IChatClient, streaming is a different method on the same interface. GetStreamingResponseAsync returns an IAsyncEnumerable<ChatResponseUpdate>:

`csharp
app.MapPost("/chat/stream", async (ChatRequest req, IChatClient chat, HttpResponse response, CancellationToken ct) =>
{
response.ContentType = "text/plain; charset=utf-8";

await foreach (ChatResponseUpdate update in chat.GetStreamingResponseAsync(req.Prompt, cancellationToken: ct))
{
    await response.WriteAsync(update.Text, ct);
    await response.Body.FlushAsync(ct);
}
Enter fullscreen mode Exit fullscreen mode

});
`

Test it with curl -N. The text should arrive in pieces. If it arrives as one block, something between the server and the client is buffering. Passing ct matters: if the browser disconnects, the request is cancelled and the app stops reading the stream. For a real chat UI you'll want Server-Sent Events or SignalR.

Step 6: Get typed JSON back instead of text

When code consumes the answer, ask for a type. GetResponseAsync<T> generates a JSON schema from your C# type, and OllamaSharp passes it to Ollama's format parameter, which constrains the output:

`csharp
public enum TicketCategory { Bug, Billing, FeatureRequest, Other }
public record TicketTriage(TicketCategory Category, int Priority, string Summary);

var result = await chat.GetResponseAsync(
$"Triage this support ticket. Priority 1 (urgent) to 4 (low).\n\n{ticketText}",
cancellationToken: ct);

if (result.TryGetResult(out TicketTriage? triage))
{
// triage.Category, triage.Priority, triage.Summary
}
`

Valid JSON doesn't make the answer correct. A 7B model can return a well-formed object with the wrong category, so validate the values like any other untrusted input.

Step 7: Generate embeddings with the same library

The same OllamaApiClient class implements IEmbeddingGenerator. Pull an embedding model (ollama pull nomic-embed-text) and register it next to the chat client:

`csharp
IEmbeddingGenerator> embedder =
new OllamaApiClient(new Uri("http://127.0.0.1:11434"), "nomic-embed-text");

builder.Services.AddEmbeddingGenerator(embedder);

ReadOnlyMemory vector = await embedder.GenerateVectorAsync("How do I reset my password?");
Console.WriteLine(vector.Length); // dimension of the embedding model
`

Use one embedding model for both indexing and querying. If you change models, re-embed everything.

Settings that matter in practice

  • Context window (num_ctx). Ollama's context window defaults to a few thousand tokens, regardless of the model's maximum. When a prompt is longer, Ollama truncates it without an error. In our Text-to-SQL work, the schema prompt didn't fit in the 2,048-token default of the Ollama version we were running, so the model never saw part of the schema. Raising NumCtx to 8,192 fixed it. Set it per request (Step 4) or for the whole server (OLLAMA_CONTEXT_LENGTH). Newer versions raised the default, so check yours. A larger value uses more memory.
  • Model unloading (keep_alive). Ollama keeps a model in memory for 5 minutes after the last request, then unloads it, and the next request pays the load time again. For a demo or internal tool, set OLLAMA_KEEP_ALIVE (for example 30m, or -1 to keep it loaded).
  • Concurrency. By default each model handles one request at a time (OLLAMA_NUM_PARALLEL=1). If two users call your endpoint at once, the second waits for the first one's whole answer. Keep this in mind when you set timeouts and before you load-test.
  • Network exposure. Ollama listens on 127.0.0.1 only. To reach it from another machine or a container, set OLLAMA_HOST (for example 0.0.0.0:11434). Its API has no authentication, so don't expose it beyond a network you trust.

Troubleshooting

Symptom Likely cause Fix
Connection refused Ollama isn't running, or wrong host or port curl http://127.0.0.1:11434/api/tags. Check OLLAMA_HOST.
404 "model not found" The model name hasn't been pulled (llama3.1 ≠ llama3.1:70b) ollama list, then ollama pull <name>
TaskCanceledException after ~100 s Default HttpClient timeout while the model loads or generates Set Timeout, stream, or keep the model loaded
Every new connection ~2 s slow on Windows localhost resolves to IPv6 first Use http://127.0.0.1:11434
Answers ignore part of a long prompt Silent context truncation Raise num_ctx
Very slow tokens, fans at full speed Model runs (partly) on the CPU ollama ps. Use a smaller model or num_ctx.
ollama serve fails, port in use The server is already running You don't need ollama serve
Model returns JSON as text instead of calling a tool Small model + tool calling See the note below

A note on tool calling

IChatClient supports tool calling through UseFunctionInvocation() and AIFunctionFactory, and Ollama supports it for models trained for it. Whether a 7B model calls your tools reliably is a different question. In Part 5 of the AI Database Agent series, qwen2.5-coder:7b wrote a JSON object naming the tool into its text instead of making a structured call, so the experiments switched to qwen2.5:7b. With full context in the prompt, that model mostly answered directly instead of calling the tools.

What's next

With IChatClient in place, these are the natural next steps:

  • Generate SQL from plain English, safely: How to Build Text-to-SQL with ASP.NET Core
  • Answer questions from your own documents: Build RAG with ASP.NET Core and Qdrant
  • Decide where the AI endpoint should live: ASP.NET Core vs FastAPI for AI APIs
  • Before you let a model act on data: AI Agent Security: What Developers Need to Know

FAQ

Do I need OllamaSharp, or can I just use HttpClient?
HttpClient is enough for a single non-streaming call. OllamaSharp is worth adding once you need streaming, chat history, embeddings, structured output or tool calling. It also implements IChatClient, so you can swap Ollama for a hosted model later without rewriting your endpoints.

Is Microsoft.Extensions.AI.Ollama still supported?
No. The package is deprecated on NuGet, which recommends OllamaSharp instead. Use OllamaApiClient from OllamaSharp as your IChatClient.

Which Ollama model should I start with in C#?
Start with a 7B-8B model that fits entirely in your GPU memory (ollama ps should show 100% GPU). We use qwen2.5-coder:7b for SQL and code, llama3.1 for general chat, and nomic-embed-text for embeddings.

Why does my first request time out but later ones are fast?
The first request loads the model into memory, and Ollama unloads it after 5 idle minutes by default. Raise the HttpClient timeout, use streaming, or set OLLAMA_KEEP_ALIVE.

Can I use the same code with OpenAI, Azure OpenAI or Claude later?
Yes, as long as your code depends on IChatClient and IEmbeddingGenerator rather than on OllamaApiClient. Only the registration in Program.cs changes. Ollama-specific options like NumCtx are simply ignored by other providers.

Want to learn more: Run Ollama with .NET

Top comments (0)