DEV Community

Cover image for Living in the Terminal with your LLM
Shahriar Haque Abir
Shahriar Haque Abir

Posted on

Living in the Terminal with your LLM

My PowerShell 7 prompt is where I already live. Putting the LLM there means zero context switching and zero cloud.

TL;DR

I put an LLM directly into my PowerShell 7 profile so I can talk to it without leaving the terminal. The result is a terminal-native AI ops companion: system recon, curated news, command authoring and review, all invoked like any other command. The model runs locally through Ollama on loopback, so nothing leaves the machine. Every action the model suggests waits for a human Y/N. This post shows the integration, the use cases, a side-by-side comparison of hardware and model tiers, the trade-offs, and the guardrails that make it safe.

The Problem

The terminal is where I already work. Switching to a browser, a chat window, or a separate app to ask an AI a question breaks flow and loses context - the model doesn't know what's in my clipboard, my current directory, or my last error.

Three things made the obvious approaches fail:

  1. Cloud chat is disqualified for sensitive work. Pasting system output, file paths, or secrets into a browser is a non-starter on locked-down or air-gapped machines.
  2. Copy-paste round trips lose fidelity. By the time I paste Get-Process output into a chat and get an answer back, the moment has passed and the context is stale.
  3. An agent with shell access and no gate is a liability. Anything that can run Remove-Item or Invoke-WebRequest needs a human in the loop, not a system prompt that says "be careful".

What I Built

A PowerShell 7 profile that wires an LLM into the terminal as a set of first-class commands: ask, fix, brief, mem, news. The model (Ollama, loopback only) receives a curated context packet - time, health, cwd, $Error[0], git status, recent history - and answers in place. Every destructive suggestion is displayed, never executed. Mutating cmdlets in the full module require -Confirm/ShouldProcess and are logged.

Stack: PowerShell 7.2+ (7.6 LTS recommended), Ollama @ http://127.0.0.1:11434, nomic-embed-text for local vector memory, Pester 5+ for tests.

Getting Started: Wiring an LLM into Your Profile

The integration point is your PowerShell profile ($PROFILE). The goal: a function that takes a prompt, assembles context, calls the model, and prints the answer - callable like any cmdlet.

# ~/.config/powershell/Microsoft.PowerShell_profile.ps1 (or $PROFILE)
function Invoke-OpsAI {
    param(
        [Parameter(Mandatory, Position = 0, ValueFromRemainingArguments)]
        [string[]]$Prompt,
        [switch]$Stream
    )
    $text = $Prompt -join ' '
    $ctx  = Get-AmbientContext   # time, health, cwd, $Error[0], git, history
    $body = @{
        model  = 'qwen2.5:7b'
        prompt = "$ctx`n`nUser: $text"
        stream = $Stream.IsPresent
    } | ConvertTo-Json -Depth 6
    Invoke-RestMethod -Uri 'http://127.0.0.1:11434/api/generate' `
                      -Method POST -Body $body -ContentType 'application/json'
}
Set-Alias -Name ask -Value Invoke-OpsAI
Enter fullscreen mode Exit fullscreen mode

Three rules make this usable rather than fragile:

  1. Loopback only. The endpoint is 127.0.0.1. No DNS, no TLS, no egress. If the model isn't local, the command fails closed.
  2. Context is assembled by code, not by the model. The ambient packet is structured PowerShell output, not a dump of raw terminal scrollback.
  3. Output is structured. The function returns a [PSCustomObject] you can pipe, format, or convert to JSON - not a wall of text.

Architecture

Local context grounds an AI request; proposed remediations require confirmation and are audited.

Ports and adapters is doing real work here: the AI adapter is one adapter among five, so "use the model" is never a special path. It receives the same envelope as everything else. Swap Ollama for another backend and nothing else changes.

Engineering Philosophy

The design rests on a handful of principles, each visible in the sections above:

  1. The model is the least trusted component. Code assembles context; humans gate actions. The model never runs a command, never reads a secret, and never touches the system directly - it only reads a sanitized snapshot and proposes.
  2. Put the AI where you already work. The terminal is home. Zero context switching, zero copy-paste round trips, zero stale context.
  3. Loopback only. No egress, no cloud model for sensitive work. If the model isn't local, the command fails closed.
  4. Ports and adapters. The AI adapter is one adapter among five. Swap the backend and nothing else changes; the envelope is identical.
  5. Structured over raw. [PSCustomObject] output pipes, formats, and converts. Raw terminal dumps waste context and bury signal.
  6. Human-readable by default. Tables, color, truncation, and personal style. The tool adapts to taste, not the other way around.
  7. Validate, escape, mask, gate. Every input, every output, every action. No Invoke-Expression on model output, ever.
  8. Personalize the harness. Aliases, themes, verbosity, and guardrails live in config. The setup reflects how you work, not a generic template.

Daily Drivers

1. Curated news from favorite topics

A daily news command fetches RSS/JSON feeds, filters by topic keywords, and pipes the result through the model for a digest. Parameterization matters here - topic, source, and count are validated before any network call.

function Get-CuratedNews {
    param(
        [ValidateSet('ai','powershell','security','devtools')]
        [string]$Topic = 'ai',
        [ValidateRange(1,20)]
        [int]$Count = 5,
        [string]$FeedUrl = 'https://example.com/feed.xml'
    )
    # Validate URL scheme before fetching
    if ($FeedUrl -notmatch '^https://') { throw 'Feed URL must be HTTPS' }
    $items = Invoke-RestMethod -Uri $FeedUrl -TimeoutSec 10
    $filtered = $items | Where-Object { $_.title -match $Topic } | Select-Object -First $Count
    $prompt = "Summarize these headlines in 5 bullets, neutral tone:`n" + ($filtered | ConvertTo-Json)
    Invoke-OpsAI $prompt
}
Enter fullscreen mode Exit fullscreen mode
Set-Alias -Name news -Value Get-CuratedNews

function Add-MemoryNote {
    param([Parameter(Mandatory, ValueFromRemainingArguments)][string[]]$Note)
    $text = $Note -join ' '
    $embedding = Invoke-RestMethod -Uri 'http://127.0.0.1:11434/api/embeddings' `
        -Method POST -Body (@{ model = 'nomic-embed-text'; prompt = $text } | ConvertTo-Json) `
        -ContentType 'application/json'
    # Persist note + embedding to Memory/ (cosine search via Search-MemoryNote)
    [pscustomobject]@{ note = $text; vector = $embedding.embedding; saved = (Get-Date) }
}
Set-Alias -Name mem -Value Add-MemoryNote
Enter fullscreen mode Exit fullscreen mode

Security notes: validate the URL scheme, cap the count, validate the topic against an allow-list, set a timeout, and never interpolate user input directly into a shell string. Prefer Invoke-RestMethod with a parameter object over curl string building.

2. System recon and analysis

brief gathers system state - processes, services, disk, network, firewall, Defender - and pipes it through the model for a structured analysis with recommendations.

function Invoke-SystemBrief {
    $snapshot = [ordered]@{
        OS       = (Get-CimInstance Win32_OperatingSystem).Caption
        CPU      = (Get-CimInstance Win32_Processor).Name
        MemGB    = [math]::Round((Get-CimInstance Win32_ComputerSystem).TotalPhysicalMemory/1GB,1)
        DiskFree = (Get-PSDrive C).Free
        TopProc  = Get-Process | Sort-Object CPU -Descending | Select-Object -First 5 Name,CPU
        Services = Get-Service | Where-Object Status -eq 'Running' | Select-Object -First 10 Name
        Listeners= Get-NetTCPConnection -State Listen | Select-Object LocalPort,OwningProcess
    }
    $prompt = "Analyze this system snapshot. Flag anomalies, security risks, and 3 improvements:`n" + ($snapshot | ConvertTo-Json -Depth 4)
    Invoke-OpsAI $prompt
}
Set-Alias -Name brief -Value Invoke-SystemBrief
Enter fullscreen mode Exit fullscreen mode

The model never touches the system directly - it only reads a sanitized snapshot. Secrets in environment variables or config files are masked (Protect-OpsSensitiveText) before the packet is built.

3. Command authoring and review

Two modes: send a command to the model for review, or ask in plain language for a command.

# Review a command before running it
function Test-CommandSafety {
    param([Parameter(Mandatory)][string]$Command)
    $prompt = "Review this PowerShell command for correctness and risk. Reply with: SAFE/REVIEW/UNSAFE, reason, and a corrected version if needed:`n$Command"
    Invoke-OpsAI $prompt
}
Set-Alias -Name fix -Value Test-CommandSafety

# Ask for a command in plain language
function New-CommandFromIntent {
    param([Parameter(Mandatory, ValueFromRemainingArguments)][string[]]$Intent)
    $ctx = "PowerShell 7.6, modules: Az, Pester, PowershellOps. Aliases: ask, fix, brief, mem, news."
    $prompt = "$ctx`n`nWrite a PowerShell command or script for: $($Intent -join ' '). Output only code."
    Invoke-OpsAI $prompt
}
Set-Alias -Name gen -Value New-CommandFromIntent  # generate a command from plain language
Enter fullscreen mode Exit fullscreen mode

The harness supplies the model with the exact PowerShell version, installed modules, aliases, and guardrails - so suggestions match my environment instead of generic StackOverflow snippets.

Side-by-Side: Hardware and LLM Tiers

Not every machine can run every model. Pick the tier that fits the hardware, then accept the trade-offs.

Tier Hardware Model Latency Quality Use
Nano 8 GB RAM, no GPU phi3:mini (3.8B) <1s Basic Quick lookups, command syntax
Small 16 GB RAM, CPU qwen2.5:7b (7B) 2-4s Good Daily driver: news, brief, fix
Medium 32 GB RAM + GPU (8GB VRAM) qwen2.5:14b (14B) 1-2s Strong Deep system analysis
Large 48 GB+ VRAM (or 64 GB unified) llama3.1:70b (70B, Q4) 5-15s Excellent Complex reasoning, code review
Cloud Any + egress GPT-4 / Claude 1-3s Best Disqualified for sensitive/air-gapped work

Trade-offs:

  • Smaller models are fast and fit on laptops but hallucinate on complex multi-step reasoning and long context.
  • Larger models reason better but need VRAM; CPU-only inference at 70B is unusable (minutes per response).
  • Quantization (Q4_K_M) halves VRAM at ~2-3% quality loss - worth it on constrained hardware.
  • Context window scales with model: 4K (nano) → 32K (medium) → 128K (large). The ambient packet must fit; trim aggressively on small tiers.

Recommendation: start at Small (qwen2.5:7b on 16 GB). It handles 90% of daily tasks. Upgrade only when you hit a quality wall on deep analysis.

Ollama vs llama.cpp

Both run local models on your own hardware. The choice is about ergonomics vs control.

Dimension Ollama llama.cpp
Setup One-line install, ollama pull, ollama run. Zero config. Build from source or download binaries; manage GGUF files and flags manually.
API Clean HTTP REST (/api/generate, /api/chat, /api/embeddings). Stable contract. OpenAI-compatible server (llama-server) plus raw llama-cli. More flags, more power.
Model mgmt Built-in registry (ollama pull qwen2.5:7b). Auto-downloads, checksums, caching. You fetch GGUFs yourself (Hugging Face). You own versioning.
Quantization Picks sensible defaults; supports Q4/Q5/Q8 via Modelfiles. Full control: every quant type, custom splits, GPU layer offload (-ngl), thread tuning.
Performance Good defaults, slightly conservative. Often faster on the same hardware with tuned -ngl/-t/-b.
Modelfiles / system prompts First-class. Bake a system prompt + params into a named model (ollama create mymodel -f Modelfile). System prompt via flag; less ergonomic for named personas.
Embeddings /api/embeddings out of the box. llama-server --embedding, or separate tool.
Ecosystem Huge; most tools (LangChain, Open WebUI, Continue) target it. Lower-level; more DIY wiring.
Best for Daily driver, profile integration, stable contract. Benchmarking, squeezing max throughput, exotic quants, air-gapped custom builds.

Recommendation: start with Ollama for the profile integration - the REST contract is what Invoke-OpsAI calls, and Modelfiles let you bake the system prompt and guardrails into a named model. Move to llama.cpp only when you need to tune throughput on a specific GPU or run a quant Ollama doesn't package.

Mitigating Hallucination: Specialization Over Size

The brute-force answer to hallucination is a bigger model. The better answer is a smaller, specialized one used in a narrower way. The LLM in this setup is not a general-purpose assistant - it is a PowerShell-terminal specialist. That constraint is what makes small models reliable.

Principles:

  1. Narrow the task, don't widen the model. ask "explain Get-Process" is a documentation lookup. brief is a structured analysis over a fixed snapshot. fix is a command review. Each call has a bounded input and a bounded output. Small models excel at bounded tasks; they fail on open-ended reasoning.
  2. Ground every call in structured data. The ambient packet is deterministic PowerShell output, not free-form memory. The model summarizes and flags; it never invents the underlying facts.
  3. Specialize the prompt, not just the model. The system prompt injects the PowerShell version, installed modules, aliases, and guardrails. A 7B model with the right context outperforms a 70B model guessing at your environment.
  4. Finetune or train when the task is stable. If a task repeats (e.g., "review this command for risk"), a small finetuned model on that exact distribution beats a general model 10x its size. Training data is cheaper than GPU hours for a fixed task.
  5. Constrain the output. Ask for a specific shape - "Reply with: SAFE/REVIEW/UNSAFE, reason, corrected version." Constrained decoding and format instructions reduce hallucination more than extra parameters.
  6. Show sources. When the model cites a cmdlet or flag, require it to echo the input it based the answer on. If it can't, treat it as unverified.
  7. Human in the loop for the last mile. The model proposes; the human disposes. Hallucination that reaches a human is a nuisance; hallucination that executes is an incident.

The balance: bigger models buy you generality. Specialization - narrow tasks, grounded context, finetuned weights, constrained output, human gates - buys you reliability at a fraction of the cost. For terminal ops, reliability per watt beats raw capability. A 7B model that never hallucinates on Get-Process is more useful than a 70B model that sometimes invents flags.

Risks and Mitigation

Risk Mitigation
Command injection via model output Never Invoke-Expression on model output. Display suggestions; user copies or explicitly confirms.
Secret leakage in context Protect-OpsSensitiveText masks tokens, keys, passwords before the packet is built.
Prompt injection via fetched content Treat fetched web content as data, not instructions. Wrap in delimiters; instruct model to summarize only.
Unbounded network calls Validate URL scheme (HTTPS only), set timeouts, cap payload size.
Destructive actions Suggestions are displayed, never executed. Mutating cmdlets require -Confirm/ShouldProcess; logged to local audit file.
Model hallucination Ground answers in structured data, not free-form memory. Show sources in output.
Egress / data exfiltration Loopback endpoint only; no cloud model for sensitive work.
Stale context Rebuild the ambient packet per invocation; don't cache across calls.

Human-Readable Formatting and Personal Style

Raw PowerShell output is hostile to both humans and models. The profile applies a consistent formatting layer:

  • Tables for lists (Format-Table -AutoSize), cards for single objects, JSON for piping.
  • Color coding: errors red, warnings yellow, success green - via $PSStyle (PS 7.2+).
  • Truncation: cap output at N lines before sending to the model; summarize the rest.
  • Personal style: aliases (ask, fix, gen, brief, mem, news), prompt theme, and output verbosity are all configurable in ops-settings.json. The harness adapts to taste - verbose for learning, terse for ops.
# ops-settings.json
{
  "verbosity": "terse",
  "colorTheme": "dark",
  "maxContextLines": 40,
  "defaultModel": "qwen2.5:7b",
  "confirmDestructive": true
}
Enter fullscreen mode Exit fullscreen mode

The Harness: Context, Guardrails, and Control

The model is the least trusted component. The harness around it does the work:

  1. Context assembly - Get-AmbientContext builds a structured packet: timestamp, OS health, cwd, last error, git status, recent command history. This is deterministic code, not model inference.
  2. Command/alias/docs injection - the system prompt includes the PowerShell version, installed modules, available aliases, and relevant help topics so suggestions are environment-accurate.
  3. Guardrails -
    • No Invoke-Expression on model output.
    • All mutating cmdlets require -Confirm.
    • Sensitive text masked before context assembly.
    • URL scheme validation and timeouts on all network calls.
    • Model output is displayed, never executed automatically.
  4. Audit trail - every AI interaction and every confirmed action is logged to Reports/ with timestamp, prompt, and result.

Quick Start

  1. Install Ollama; pull a model: ollama pull qwen2.5:7b.
  2. Add Invoke-OpsAI and aliases to $PROFILE.
  3. Create ops-settings.json with your preferences.
  4. Add Get-AmbientContext and Protect-OpsSensitiveText helpers.
  5. Test: ask "explain Get-Process" and brief.

Verification

  • Loopback check: Test-NetConnection 127.0.0.1 -Port 11434 returns open.
  • No egress: disconnect network; ask and brief still work.
  • Secret masking: set $env:SECRET_TOKEN = 'abc123'; run brief; confirm token absent from logged packet.
  • Gate check: fix "Remove-Item C:\temp\*" returns a suggestion but does not execute.
  • Pester: Invoke-Pester → all tests green.

Takeaways

  • Put the LLM where you already work - the terminal. Zero context switching.
  • The model is the least trusted component. Code assembles context; humans gate actions.
  • Loopback only. No egress for sensitive work.
  • Route by hardware: nano for laptops, small for daily drivers, medium/large for deep analysis.
  • Validate, escape, mask, and gate - every input, every output, every action.
  • Personalize: aliases, themes, verbosity. The harness adapts to you.

Top comments (0)