Running LLM Agents at the Edge: A Practical Guide with NeoMind + Ollama

Everyone's building AI agents right now. Most of them live in the cloud — you send a request to OpenAI or Anthropic, get a response back, and hope the latency and cost stay reasonable. But what if your agent needs to control physical devices? Monitor factory sensors? Respond to security camera events in under 100ms?

Cloud-based agents add 200-800ms of round-trip latency per inference call. For a conversational chatbot, that's fine. For an autonomous agent monitoring a production line, it's the difference between catching a defect and shipping it.

This guide walks through building production-grade LLM agents that run entirely on edge hardware using NeoMind + Ollama. No cloud API keys. No data leaving your network. Sub-100ms inference on a $200 device.

What You'll Build

By the end of this guide, you'll have:

• A local LLM agent that monitors IoT device telemetry in real-time
• Natural language interaction with your agent via chat
• Autonomous decision-making based on sensor data patterns
• Multi-tier memory so the agent learns your environment over time
• Zero cloud dependency — everything runs on your LAN

Prerequisites

• Hardware: Any x86_64 or ARM64 machine with ≥8GB RAM (Raspberry Pi 5, Intel NUC, old laptop all work)
• OS: Linux (Ubuntu 22.04+), macOS, or Windows
• GPU: Optional but recommended. Ollama runs on CPU for smaller models (7B), but 13B+ benefits from GPU acceleration
• NeoMind: The edge AI platform (handles device connectivity, automation, UI)
• Ollama: The local LLM runtime (handles model serving and inference)

Step 1: Install NeoMind + Ollama

Install Ollama

curl -fsSL https://ollama.com/install.sh | sh

Pull a model suited to your hardware:

Hardware	Recommended Model	VRAM/RAM Needed	Inference Speed
Raspberry Pi 5 (8GB)	llama3.2:3b	~4 GB	~8 tokens/sec
Intel NUC (16GB)	llama3.1:8b	~6 GB	~15 tokens/sec
GPU (RTX 3060+)	llama3.1:70b (Q4)	~12 GB VRAM	~40 tokens/sec
Apple Silicon (M1+)	llama3.1:8b	~8 GB unified	~25 tokens/sec

ollama pull llama3.1:8b

Install NeoMind

curl -fsSL https://raw.githubusercontent.com/camthink-ai/NeoMind/main/scripts/install.sh | sh

Configure NeoMind to Use Ollama

Edit NeoMind's config (typically ~/.config/neomind/config.toml):

[ai]
backend = "ollama"
model = "llama3.1:8b"
api_url = "http://localhost:11434"
max_tokens = 2048
temperature = 0.7

Start NeoMind:

neomind start

Open the web UI at http://localhost:9375 and you'll see the AI chat panel ready to go.

Step 2: Connect Your First Device

Before the agent can monitor anything, it needs devices to talk to. NeoMind supports MQTT, BLE, and Webhook protocols out of the box.

Simulate a Device (for Testing)

If you don't have physical IoT devices yet, use NeoMind's device simulator:

neomind device simulate --type temperature-sensor --interval 5s

This creates a virtual temperature sensor that publishes readings to NeoMind's embedded MQTT broker every 5 seconds.

Connect a Real Device

For real devices, configure MQTT in NeoMind:

[mqtt]
enabled = true
port = 1883
# Devices connect to neomind-host:1883

Or use the auto-discovery feature — plug in a USB IoT gateway and NeoMind will detect and register it.

Step 3: Build Your First Autonomous Agent

Here's where it gets interesting. Instead of just chatting with the AI, we'll create an agent that autonomously monitors devices and takes action.

Define the Agent's Mission

In NeoMind's web UI, navigate to AI → Agents → New Agent:

name: "Temperature Guardian"
schedule: "every 5 minutes"
mission: |
  You are monitoring temperature sensors across a building.
  Your responsibilities:
  1. Check all temperature readings from the last 5 minutes
  2. Flag any reading above 28°C or below 16°C as anomalous
  3. If a sensor shows 3+ consecutive anomalous readings, alert the operator
  4. Log a summary of findings to the knowledge base

tools:
  - query_device_metrics
  - send_notification
  - update_knowledge_base

What Happens at Runtime

Every 5 minutes, NeoMind's agent runtime:

1. Wakes the agent and injects the current context (device states, recent history, knowledge base entries)
2. Runs inference via Ollama — the LLM analyzes the data and decides what actions to take
3. Executes tool calls — the agent queries specific device metrics, sends notifications if thresholds are breached, and updates its memory
4. Goes back to sleep until the next cycle

┌──────────────────────────────────────────────────┐
│            Agent Runtime Loop                     │
│                                                   │
│  ┌─────────┐    ┌──────────┐    ┌─────────────┐  │
│  │ Wake +  │───►│ LLM      │───►│ Execute     │  │
│  │ Inject  │    │ Inference│    │ Tool Calls  │  │
│  │ Context │    │ (Ollama) │    │ (typed)     │  │
│  └─────────┘    └──────────┘    └──────┬──────┘  │
│                                        │         │
│  ┌─────────────────────────────────────▼──────┐  │
│  │  Memory Update + Sleep until next cycle    │  │
│  └────────────────────────────────────────────┘  │
└──────────────────────────────────────────────────┘

Type-Safe Tool Dispatch

A critical design decision: NeoMind's agent tools are not string-based function calls. When the LLM decides to "query device temperature," it produces a structured command:

// The LLM output is deserialized into a typed enum
enum AgentToolCall {
    QueryDeviceMetrics { device_id: String, metric: String, since: Duration },
    SendNotification { channel: String, message: String, severity: Severity },
    UpdateKnowledgeBase { topic: String, content: String },
    ControlDevice { device_id: String, action: DeviceAction },
}

// Dispatch is a type-safe match — no eval(), no shell exec
match tool_call {
    AgentToolCall::QueryDeviceMetrics { device_id, metric, since } => {
        let readings = device_manager.query_range(&device_id, &metric, since).await?;
        Ok(ToolResult::MetricReadings(readings))
    }
    // ...
}

This eliminates injection attacks, hallucinated command strings, and the fragile JSON-to-shell pipelines that plague other agent frameworks.

Step 4: Multi-Tier Memory — Making the Agent Learn

A stateless agent is just a fancy cron job. NeoMind's memory system gives your agent persistent context across sessions:

Memory Tiers

Tier	Purpose	Example	Retention
Profile	User preferences and environment	"Operator prefers Celsius, works 8AM-6PM"	Permanent
Knowledge	Accumulated domain facts	"Server room #3 runs 2°C hotter than average"	Long-term
Task	Current mission state	"Monitoring anomaly on sensor-07 since 14:30"	Per-task
Session	Active conversation context	"User just asked about yesterday's temperature trends"	Per-session

Automatic Memory Extraction

The agent doesn't need explicit "save this" commands. NeoMind's memory engine automatically extracts and compresses important information from agent interactions:

Agent observes: "Sensor-07 has shown elevated readings (29.1°C, 29.4°C, 28.8°C) 
                 for the last 3 cycles. This is above the 28°C threshold."

→ Auto-extracted to Knowledge Base:
  "Sensor-07 recurring thermal anomaly, first detected 2026-08-19T14:30Z,
   readings consistently 28-30°C, may indicate HVAC issue in Zone 3"

Memory Compression

Long-running agents accumulate massive context. NeoMind compresses memory using:

• Temporal deduplication — merge repeated observations into time ranges
• Importance scoring — prioritize security events over routine readings
• Hierarchical summarization — compress daily summaries into weekly overviews

This keeps the context window manageable even after months of operation.

Step 5: Natural Language Interaction

While the autonomous agent runs in the background, you can also chat with it conversationally:

You: "What's the situation with sensor-07?"

Agent: "Sensor-07 in Zone 3 has been running hot since yesterday at 14:30. Readings have been consistently 28-30°C, which is 4-6°C above the zone average of 24°C. I've logged 12 anomalous readings. Based on the pattern, this looks like an HVAC degradation rather than a transient spike. Want me to create a maintenance ticket?"

You: "Yes, and also set up hourly monitoring for the next 48 hours"

Agent: "Done. I've created a maintenance ticket for Zone 3 HVAC inspection and scheduled myself for hourly checks on sensor-07 through August 21st. I'll alert you immediately if readings exceed 32°C."

The agent's response is grounded in its actual memory and device data — not hallucinated. NeoMind enforces this by injecting only verified data into the LLM context.

Step 6: Benchmarking — Real Numbers

Here's what we measured on common edge hardware running llama3.1:8b via Ollama:

Metric	Raspberry Pi 5	Intel NUC i5	Apple M1	RTX 3060
First token latency	1,200ms	400ms	350ms	150ms
Tokens/sec (gen)	8	18	28	45
Full agent cycle (5 devices)	6s	2.5s	1.8s	0.9s
RAM usage (idle)	4.2 GB	3.8 GB	3.5 GB	3.2 GB
RAM usage (inference)	6.1 GB	5.4 GB	5.0 GB	4.8 GB

Agent cycle = wake + context injection + inference + tool execution + memory update.

For comparison, the same agent using OpenAI's gpt-4o-mini API (cloud) adds ~800ms of network latency per inference call, plus $0.15 per 1M input tokens.

Advanced: Skill-Based Agent Customization

NeoMind's Skill System lets you fine-tune agent behavior without retraining:

# skills/factory-safety.yaml
name: "Factory Floor Safety Monitor"
trigger: "scheduled:every 2 minutes"
context: |
  You are a safety monitor for an industrial facility.

  Critical rules (NEVER override):
  - If CO2 > 1000ppm in any zone, immediately alert + activate ventilation
  - If temperature > 45°C near equipment, trigger emergency shutdown
  - If unauthorized motion detected after 22:00, alert security

  Normal operations:
  - Log all readings to knowledge base
  - Flag anomalies (>2σ from 7-day rolling average)
  - Generate daily safety summary at 18:00

tools:
  - query_device_metrics
  - send_notification
  - control_device
  - update_knowledge_base

Skills are YAML + Markdown files that the agent runtime injects into the LLM context. They provide guardrails and domain knowledge without requiring model fine-tuning.

Production Tips

1. Start with a smaller model — llama3.2:3b is fast enough for monitoring tasks and leaves headroom for the OS and NeoMind services
2. Use GPU offloading selectively — if you have a GPU, use it for the agent inference while keeping NeoMind's core services on CPU
3. Set memory limits — configure max_knowledge_entries and max_session_history to prevent memory bloat on long-running deployments
4. Monitor agent cycles — NeoMind's dashboard shows agent execution time, token usage, and tool call success rates
5. Layer models — use a fast small model for routine monitoring and a larger model (or cloud API) for complex analysis tasks

The Edge AI Stack, Simplified

┌─────────────────────────────────────────────┐
│              Your Edge Device                │
│                                              │
│  ┌──────────┐  ┌──────────┐  ┌───────────┐  │
│  │ NeoMind  │  │  Ollama  │  │  IoT      │  │
│  │ Platform │──│  LLM     │──│  Devices  │  │
│  │          │  │  Runtime │  │  (MQTT)   │  │
│  └──────────┘  └──────────┘  └───────────┘  │
│                                              │
│  Everything local. Zero cloud dependency.    │
└─────────────────────────────────────────────┘

That's the entire stack. Two components. One machine. No Kubernetes, no service mesh, no cloud account.

Get Started

• NeoMind: github.com/camthink-ai/NeoMind (Apache 2.0)
• Ollama: ollama.com
• Quick Start Guide: wiki.camthink.ai
• Community: Discord

Run your AI agents where your data lives — at the edge. No cloud required.