Local AI with Foundry Local and PowerShell: chat, memory, tools; building a harness.

Since June 2026, the frontier labs have not stopped shipping: Fable, Sonnet 5, GPT-5.6 (Sol, Terra and Luna), Grok 4.5 trained on Cursor metrics, without talking about Chinese models DeepSeek R4, Kimi K3 and GLM 5.2. The race for the next frontier model is wide open.

A quieter signal runs alongside it: the move to open-weight models. Microsoft is testing DeepSeek, Uber Eats and Airbnb are running Qwen, Cursor uses Kimi. These models run on hardware you control, and they do real work for a fraction of the cost of a frontier API call.

Three reasons push in the same direction: API cost, privacy and data residency (GDPR and the Cloud Act discussion in the EU), and the fact that the gap between open-weight and frontier models has narrowed sharply in the last few months.

This is where Foundry Local fits. It ships SDKs (Python, C#, Rust, JS, C++) and a CLI exposing a REST API, in under 100 MB. It handles model acquisition and hardware acceleration for you: pick a model from the curated catalogue — Mistral, Phi, Qwen and others — and Foundry Local selects the right backend for your machine (CPU, GPU, NPU).

What Foundry Local doesn’t give you is the harness: the small program between the model and the real world that holds the state, executes the tools, and decides what comes back. This post is about writing it in PowerShell.

To install Foundry Local CLI, use the latest release on the GitHub page of the project.

_Everything below is in preview. This post uses the Foundry Local CLI 0.1.1 and has been tested on CLI 0.1.2. _

Scripts used in this post can be find on my GitHub repos

The CLI runs a local service. Everything else in this post talks about it, so start here.

>Foundry server status

You should have something like:

StateReady
PID60900
Started2026-07-13 18:58:02Z
Uptime1d 2h 13m 39s
Web URLs http://127.0.0.1:56172

If not, you can start the server.

>foundry server start

You can get a list of models available for your platform via this command.
>foundry model list

To get more information about a particular model, use this command:
>foundry model info mistral-7b-v0.2

Models are identified by an alias, and each alias can resolve to several variants. A variant is a build of the model for one acceleration backend, TensorRT or CUDA (NVIDIA), QNN (Qualcomm), or generic, and one device type (CPU, GPU, NPU).

Foundry Local picks a variant for you, but the pick is only as good as your hardware; a machine with no suitable GPU falls back to NPU.

The alias usually encodes the parameter count; for mistral-7b-v0.2, it is 7 billion. One exception is the Phi models, which use size tiers (mini, small…). More parameters mean a bigger memory footprint and generally more capability, though architecture and training quality matter at least as much.

The “model info” command also reports the context window, in tokens. This matters more than it looks: as you approach the limit, older turns get dropped, and the model starts answering confidently from a history it no longer has.

One last property: whether the model supports tool calling. The model never executes anything itself; it emits a structured request that your code executes and feeds back. In practice, below roughly 3B parameters, tool call reliability drops off fast.

To be able to use a model, it must be available in the local cache and loaded.

You can download a model:
>foundry model download qwen3.5-4b
And then load the model.
>foundry model load qwen3.5-4b
The latest command will download the model if it is not in the cache and load the model.
You can run the model by typing.
>foundry run qwen3.5-4b

You get a chat with the model; you can type /help to see available commands.

But this is just a chat. It is limited; there is no context, no memory, no tools, no harness.
To add more control, we can start building a solution in PowerShell using the included REST API.

The port is assigned at startup and changes when the service restarts. Don't hardcode it; read it back from the foundry server status command, or the SDK, before each session.
>foundry server status
If the service is not started, you need to use this command:
>foundry server start
To use the foundry CLI in PowerShell, we need to save the URI given by the command in a variable:

$restApiUri = "http://127.0.0.1:56172"

For the first script, we will do a simple chat, a simple question to the model.

$body = @{
        model    = "qwen3.5-4b"
        messages = @( @{ role = "user";   content = "Give the name of the capital of Belgium"   } )
    }

 $params = @{
        Method  = "POST"
        Uri     = "$($restApiUri)/v1/chat/completions"
        Body    = ($body | ConvertTo-Json -Depth 10)
        ContentType = "application/json"
    }

$result = Invoke-RestMethod @params

Foundry Local's REST API is OpenAI-compatible, which is the whole point: the body you build here is the same body you'd send to api.openai.com. Swapping a local model for a hosted one becomes a URL change.

The result contain an object with different property, among them the model, choices which is an array of HashMap containing the conversation with the model and total_tokens, a map listing used tokens, prompt_tokens, completion_tokens and total_tokens:
usage : @{prompt_tokens=11; completion_tokens=7; total_tokens=18}

It also contains the property Choices, an array that contains the interaction with the model. Normally the Model’s response is in the first (index 0). It contains the message, a map with a role (assistant), a content, the model response and the tool_call.

To get the message:

$result.choices[0].message.content

The script so far does nothing the CLI chat didn't. The first real gain is the system prompt: instructions that shape behaviour before the user ever speaks. It's the cheapest control surface you have.

To do so we need to modify the $message variable and add a hash table with the system role:

$messages = @(
    @{ role = "system"; content = "You are a PowerShell coding assistant. Only give code if requested." }
    @{ role = "user"; content = "Give me a PowerShell script to list all files in a given directory." }
)

This works, but the model has no memory. Every call is a cold start; nothing carries over. Memory here isn't a feature of the model; it's you resending the transcript on each interaction.

Which means the transcript needs to preserve insertion order and grow cheaply. A PowerShell array += reallocates on every append, so use a generic list:

[System.Collections.Generic.List[hashtable]] $Messages = @(
    @{ role = "system"; content = "You are a science popularizer specializing in physics." }
    @{ role = "user"; content = "Explain quantum computing in plain English in less than 200 words" }
)

The first interaction is similar to the two first examples. But for the next interaction with the model, the result is added to the message.

$assistantResponse = $result.choices[0].message.content
$messages.Add(@{ role = 'assistant'; content = $assistantResponse })

Then you can create a new message

$messages.Add(@{ role = 'user'; content = "Now explain it to a 5 year old" })

And start the conversation with the history.

Everything above sets up the last piece. Tool calling only works because of the transcript, the model's request, your result, and its final answer are three turns in the same history.

A model doesn’t execute tool, instead the model send back a request for the tool to be executed externally and return the result.

The function is a stub, it returns a canned answer so we can focus on the protocol, not on Azure stuff.

function Get-VnetPeeringStatus {
    param (
        [string]$VnetName,
        [string]$ResourceGroup
    )

    # Simulate a response from Azure
    return @{
        VnetName      = $VnetName
        ResourceGroup = $ResourceGroup
        PeeringStatus = 'Connected'
    }
}

To use a tool calling this function, the model needs to know its definition, which is a JSON array. You can define one or several tool.

$tools = @(
    @{
        type     = 'function'
        function = @{
            name        = 'Get-VnetPeeringStatus'
            description = 'Returns the peering status of an Azure VNet'
            parameters  = @{
                type       = 'object'
                properties = @{
                    VnetName      = @{
                        type        = 'string'
                        description = 'Name of the virtual network'
                    }
                    ResourceGroup = @{
                        type        = 'string'
                        description = 'Resource group containing the VNet'
                    }
                }
                required = @('VnetName', 'ResourceGroup')
            }
        }
    }
)

Then the message is needed:

[System.Collections.Generic.List[hashtable]] $Messages = @(
    @{ role = "system"; content = "You are an Azure infrastructure assistant. You have access to tools that query live Azure state. When a user asks about the status of a resource, call the appropriate tool instead of guessing. Only answer from tool results, never fabricate resource state." }
    @{ role = "user"; content = "Is the hub-vnet in rg-network-prod peered correctly?" }
)

Note how much heavier the system prompt is here. It tells the model to call the tool rather than guess, and to answer only from tool results. With small models this isn't optional, a 4B model asked about live infrastructure will happily invent a plausible peering status. The prompt is your first line of defence against this kind of hallucination.

Then we need to create the body:

$body = @{
    model       = $modelID
    messages    = $messages
    tools       = $tools
    tool_choice = @{ type = 'function'; function = @{ name = 'Get-VnetPeeringStatus' } }
} | ConvertTo-Json -Depth 10   

“tool_choice” pins which tool the model may call. Here I force Get-VnetPeeringStatus, useful for a demo, since it removes one source of nondeterminism. In real use you'd set auto and let the model decide, which is also where small models start to disappoint.

After send the request to the model, we need to scan the response to see if a “tool_calls” is returned.

$choice = $result.choices[0]

if ($choice.finish_reason -eq 'tool_calls') {
    # Append the assistant turn (with its tool_calls) to history
    $messages.Add(@{ role = 'assistant'; content = $choice.message.content })


    foreach ($call in $choice.message.tool_calls) {
        $args = $call.function.arguments | ConvertFrom-Json

        # Dispatch — never Invoke-Expression on model output
        $result = switch ($call.function.name) {
            'Get-VnetPeeringStatus' {
                Get-VnetPeeringStatus -VnetName $args.VnetName -ResourceGroup $args.ResourceGroup
            }
            default { @{ error = "Unknown tool: $($call.function.name)" } }
        }

        $messages.Add(@{ role = 'tool'; tool_call_id = $call.id ; content = ($result | ConvertTo-Json -Depth 5 -Compress) })

    }
    # Re-send with updated $messages to get the final answer
    $body = @{
        model       = $modelID
        messages    = $messages
    } | ConvertTo-Json -Depth 10        

     $params = @{
        Method  = "POST"
        Uri     = "$($restApiUri)/v1/chat/completions"
        Body    = $body
        ContentType = "application/json"
    }

    $result = Invoke-RestMethod @params
    $result.choices[0].message.content
}

The assistant content is added to the $message variable to keep the history.

Then for each call in the model message, the tool (the powershell function) is executed, and we add a new hashmap with the role tool, the tool_call_id and the result of the function.

Finaly, we resend the updated message to the model and this time it will reply to the question sent in the first interaction.

These are the basic interactions: chat, system prompt, memory, tools. Nothing here is PowerShell-specific, the same steps work the same way in Python or C#.

But look back at what got built. The model didn't remember anything; we resent the transcript. The model didn't call a function; it emitted a name and we decided whether to honour it. The model didn't check its own output; the switch did. Everything that made this useful happened outside the model, in a hundred lines of PowerShell sitting between it and the rest of the world.

That's the foundation of a harness. It's the small program that holds the state, owns the allowlist, executes the calls, and decides what goes back in. When you use a hosted product, the harness ships with it and you never see it. Foundry Local gives you a model and a port. The harness is yours to write, which is the cost, and the entire point: nothing reaches your infrastructure unless your code let it through.

One practical note before you build on this. Everything above talks to the REST API, which means you own the plumbing, discovering the port, resolving aliases to variants, catching a service that isn't up. I will post something about what you can do with the SDK.

Experiment with it; try it on some workflows but remember Foundry Local CLI is in preview, and you may encounter bugs and malfunctions.

I'm wrapping all of this into a PowerShell module; alias resolution, streaming, the tool dispatch loop. More on that in a follow-up.

Scripts used in this post can be find on my GitHub repos