DEV Community: Andre Faria

MCPs Are Eating Your Context Window (And What To Do About It)

Andre Faria — Sun, 24 May 2026 02:53:21 +0000

I was looking at my OpenClaw token usage data when I noticed something odd. The numbers were dominated by cache reads, tens of millions of tokens per week, on a setup where the actual conversations were relatively short. The output tokens, the ones where the model is actually thinking, were a small fraction of the total.

The culprit turned out to be something I had not thought to question: MCP servers.

This article is about what MCP tool schemas actually cost, why most people miss it, and how skills solve the problem by loading lazily instead of front-loading everything into every turn. The numbers are real, measured from a real setup, priced against real provider rates.

1. What MCP servers actually inject

Model Context Protocol is a standard for connecting AI agents to external services. The idea is straightforward: define a set of tools, and the model can call them. OPNsense integration? Here are 133 tools. TrueNAS SCALE? Here are 278. GitHub? Here are 101.

The problem is how those tools reach the model. Every tool ships a JSON schema describing its name, description, parameters, types, enums, and constraints. When an MCP server is active, every single one of those schemas gets serialised and injected with every API call, whether you are going to use any of them or not. This is not a quirk of any particular client. It is how the MCP spec works. The tools array goes with every request.

Here is what that looks like in practice, measured from my homelab setup:

Component	Tools	Estimated tokens
TrueNAS MCP	278	~27,800
OPNsense MCP	133	~13,300
Playwright MCP	35	~3,500
Native agent tools	25	~2,500
Workspace files (AGENTS.md, SOUL.md, etc.)	n/a	~3,400

Total first-turn context: approximately 41,000 tokens. Workspace files account for 8% of that. The other 92% is tool schemas.

Run 215 turns per day (a moderate multi-agent setup) and you are pushing roughly 9 million context tokens daily just to describe tools you rarely use.

2. This is not a homelab problem

A few well-known MCP servers to put scale in perspective:

MCP Server	Total tools	Default/active	Tokens (full)	Source
GitHub MCP	101	52	~64,600 / ~30,300	Official discussion #1182
TrueNAS MCP	278	All	~27,800	Measured
OPNsense MCP	133	All	~13,300	Measured
Datadog MCP	40+ (16 core)	16 core	~4,000+	Datadog docs
AWS MCP suite	50+ across 20 servers	Per server (5-15)	~1,500-3,000 each	AWS Labs repo
Atlassian Rovo MCP	~12-20	All	~3,000-5,000	Estimated
Stripe MCP	~20 official	All	~5,000	Stripe docs
Slack MCP	13	All	~3,250	Speakeasy catalog
Notion MCP	~14	All	~3,500	Docker MCP catalog
Sentry MCP	~10-15	All	~2,500-3,750	Estimated
PostgreSQL MCP	~5-12	All	~1,250-3,000	MCP reference server
Kubernetes MCP	~15-25	All	~3,750-6,250	Community

A developer running GitHub MCP, Slack MCP, and a Postgres MCP alongside their native tools is starting every single message with roughly 40,000 tokens of context overhead before they have typed a word. GitHub MCP alone at full capacity burns 64,600 tokens, consuming 32% of Claude Sonnet's 200K context window before the conversation starts.

3. This affects every tool that uses MCP

This is not an OpenClaw issue. It is a consequence of how MCP works architecturally, and it affects every AI tool that integrates with MCP servers:

Tool	MCP support	Injection pattern	Notes
Claude Code	Full native	Eager, every API call	Issue #44536: ToolSearch experiment, 85% reduction when enabled
Codex CLI	Full	Eager, per turn	Used in Datadog + Codex integration examples
OpenCode	Full	Eager by default; lazy via opencode-mcp-tool-search plugin	Same underlying problem; community plugin fixes it
OpenClaw	Full	Eager, per turn	What this article is about

The MCP spec itself requires the tools array to be sent with each API call. The only documented escape valve is "ToolSearch", a meta-tool that lets the model search for tools by name rather than receiving all schemas upfront. Claude Code introduced this experimentally, with a reported 85% token reduction. GitHub MCP reduced its default toolset from 101 to 52 tools specifically in response to user complaints about context overhead.

4. What it costs per provider

On a flat-rate plan like GitHub Copilot, this overhead is invisible. You pay a fixed monthly fee regardless of token volume. But most serious usage of Claude, GPT, or Gemini goes through the API, where every token has a price.

Provider	Model	Input $/M	Cached input $/M	Output $/M	Context window
Anthropic	Claude Sonnet 4.6	$3.00	$0.30	$15.00	1M tokens
Anthropic	Claude Haiku 4.5	$1.00	$0.10	$5.00	200K tokens
OpenAI	GPT-5	$1.25	~$0.31	$10.00	272K tokens
OpenAI	GPT-4.1	$2.00	$0.50	$8.00	1M tokens
Google	Gemini 2.5 Pro	$1.25	~$0.25	$10.00	1M tokens
Google	Gemini 2.5 Flash	$0.30	—	$2.50	1M tokens
AWS Bedrock	Claude Sonnet 4.6	$3.00	~$0.30	$15.00	1M tokens
AWS Bedrock	Amazon Nova Pro	$0.96	$0.20	$3.84	300K tokens
Azure OpenAI	GPT-4.1	~$2.00	~$0.50	~$8.00	1M tokens
OpenRouter	(aggregator)	pass-through	model-dependent	pass-through	varies

What 44,500 tokens of MCP overhead costs per message at different providers, assuming prompt caching is active (best case):

Provider + Model	Per message (cached)	Per message (uncached)	Monthly (215 turns/day, 22 days)
Anthropic Claude Sonnet 4.6	$0.013	$0.134	$62 (cached) / $622 (uncached)
OpenAI GPT-4.1	$0.022	$0.089	$104 (cached) / $416 (uncached)
OpenAI GPT-5	$0.014	$0.056	$65 (cached) / $260 (uncached)
Google Gemini 2.5 Flash	$0.013	$0.013	$62 (no caching)
AWS Bedrock Nova Pro	$0.009	$0.043	$42 (cached) / $200 (uncached)

These are costs from overhead alone, before any actual work is done. On Sonnet without prompt caching, 44,500 tokens per message at 215 turns/day adds up to over $600/month in context overhead.

Prompt caching helps significantly for repeated context (the tool schemas do not change turn-to-turn, so they cache well). But even at the cached rate, the overhead is material at scale.

5. Skills: lazy loading as the fix

The alternative is skills. In OpenClaw and in tools like oh-my-openagent for OpenCode, a skill is a markdown file that tells the model how to use a capability. Only a name and a short description enter the context upfront. The full instructions are loaded when the model actually needs them.

A skill entry in the context looks like this:

truenas: Manage TrueNAS SCALE: storage, sharing, services, VMs, alerts, replication.

That is roughly 24 tokens. Compare that to the ~27,800 tokens for the TrueNAS MCP schema.

The model retains full capability. When it needs to interact with TrueNAS, it reads the skill and executes shell commands: midclt websocket calls, curl against the REST API, or short Python scripts. The capability is the same. The context cost is not.

The token savings from replacing three MCP servers:

Replaced	Tokens before	Tokens after	Saved per turn
TrueNAS MCP	~27,800	~24	~27,776
OPNsense MCP	~13,300	~24	~13,276
Playwright MCP	~3,500	~24	~3,476
Total	~44,600	~72	~44,528

First-turn context drops from ~41,000 tokens to roughly ~10,000. A 75% reduction in baseline overhead per turn.

6. What skills look like in practice

A skill is a SKILL.md file with a short frontmatter description and usage instructions. The model reads it when needed. The skill documents three things: how to authenticate, what the primary command pattern is, and what the fallback is when the primary does not cover the full surface.

Credentials live in the environment, not in the skill file. In OpenClaw, env vars are declared in openclaw.json and injected into every agent turn. Other frameworks use .env files, secrets stores, or per-agent config blocks. The skill does not care how the variables arrive, only that they exist at runtime.

{ "env": { "TRUENAS_URL": "https://truenas.host:50443", "TRUENAS_API_KEY": "..." } }

For API key auth, that is all the setup needed. For OAuth-based services, the approach shifts to pre-authenticated CLI state: gh auth login stores credentials in ~/.config/gh/hosts.yml; jira init writes an API token to ~/.config/.jira/.config.yml. After that one-time setup, skill calls carry no credentials in the command itself.

Each skill documents a primary path and a fallback. For TrueNAS that is midclt (websocket) with curl as fallback:

# Primary: dedicated CLI
midclt -u ws://truenas.host:50443/api/current --api-key "$TRUENAS_API_KEY" call pool.query

# Fallback: curl REST
curl -sk -H "Authorization: Bearer $TRUENAS_API_KEY" "$TRUENAS_URL/api/v2.0/pool" | jq .

For more complex operations (bulk queries, job polling, conditional logic), a short Python script is cleaner than chained shell commands:

import os, sys
sys.path.insert(0, '/home/user/.local/lib/python3.14/site-packages')
from truenas_api_client import Client

with Client(os.environ['TRUENAS_WS_URL'], api_key=os.environ['TRUENAS_API_KEY']) as c:
    for ds in c.call('zfs.dataset.query', [], {'select': ['name', 'used']}):
        print(ds['name'], ds['used'])

Each skill also ships a check.sh that verifies the CLI is installed, env vars are set, and the host is reachable before the agent tries to use it. Validation moves from MCP schema enforcement (happens automatically before every call) to check.sh (happens at load time, once). For stable infrastructure with a single operator that is a reasonable trade. For production systems with many contributors and rapidly evolving APIs, MCPs may still be the right call.

7. Real audit numbers

Before starting this work I pulled six days of session data from my setup:

92 million cache read tokens in six days
Average daily cost at Sonnet direct API rates: $15/day
Projected monthly: $285-390/month

This was on a flat-rate plan where none of it showed up in the bill. But GitHub Copilot is actively transitioning to usage-based billing. When that change completes, token volume will directly translate to cost for the first time.

The right time to fix token obesity is before you are paying per token, not after.

I also found a secondary problem during the audit: AGENTS.md had grown to 99% of the 12,000-character per-file bootstrap limit, meaning it was being silently truncated on every turn. The workspace files, which everyone assumes are the main context cost, were actually only 8% of the total. The other 92% was tool schemas that nobody had looked at.

8. The replacement stack

For reference, this is what replaced the three MCP servers in my setup:

TrueNAS: truenas_api_client (official iXsystems library) and midclt CLI for websocket API access. REST API via curl as fallback. Full coverage of the 278-tool surface.

OPNsense: opn-cli (community Python CLI) for firewall, HAProxy, routes, and DNS. Raw curl against the OPNsense REST API for NAT, VLANs, DHCP, and ACME, which opn-cli does not cover.

Playwright: shot-scraper for screenshots, JS eval, and HTML extraction. Python playwright library for full browser automation: form fills, login flows, file downloads.

All three follow the same pattern: a primary CLI or library path with documented fallback commands for anything the primary does not cover. The skill documents both paths. The model chooses based on what the task requires.

9. Conclusion

MCP servers are a reasonable architecture for giving agents access to external services. The problem is the cost model: every tool schema defined by an active MCP server gets injected with every API call, whether those tools are relevant to the current task or not. As the ecosystem adds more MCP servers (GitHub, Datadog, Atlassian, Stripe, Slack, Sentry, AWS, Kubernetes), the baseline context overhead per message compounds.

On flat-rate plans, this is invisible. Under per-token billing, it is a significant and growing cost that starts before any work has been done.

Skills sidestep this by being lazy. A skill entry is a name and a description, a few dozen tokens. Full instructions load when needed. The model calls CLIs and APIs directly. The capability is the same; the upfront cost is not.

The numbers from this setup: 44,500 tokens saved per turn, a 75% reduction in baseline context overhead, and a monthly saving of roughly $62 under Sonnet cached pricing, or $622 at uncached rates. On a flat rate today, not relevant. On usage-based billing, very much so.

A note on GitHub Copilot: Copilot Pro+ at $39/month is a flat rate that absorbs all token volume. If you stay within the request limits, this overhead is financially invisible. The analysis in this article applies to direct API usage with Anthropic, OpenAI, Google, AWS Bedrock, or any other pay-per-token provider. If you are on Copilot and not planning to switch, the context window fill rate argument still applies: you hit context limits sooner. But the cost argument does not, until Copilot's usage-based transition completes.

Further reading:

Model Context Protocol specification - the MCP protocol standard
GitHub MCP tool count and token overhead discussion - confirmed 64.6K / 30.3K token numbers
Claude Code ToolSearch lazy loading issue - 85% token reduction experiment
MindStudio: Claude Code MCP token overhead analysis - tool injection mechanism explained
Cursor 40-tool limit discussion - context pressure forcing hard limits
OpenCode MCP tool search plugin - lazy loading for OpenCode
Datadog MCP server - 16+ core tools, additional toolsets
Atlassian MCP server - Jira, Confluence, Compass
AWS MCP suite - 20+ individual servers
Sentry MCP - official debugging MCP
AgentSkills specification - the skill format used by OpenClaw and oh-my-openagent
oh-my-openagent - skills for OpenCode, same lazy-loading pattern
truenas/api_client - official TrueNAS Python client used in replacement
opn-cli - community OPNsense CLI
shot-scraper - Simon Willison's browser scraping CLI
Anthropic pricing - Claude API rates
OpenAI pricing - GPT API rates
Google AI pricing - Gemini API rates
AWS Bedrock pricing - Bedrock rates
pricepertoken.com - cross-provider pricing comparisons

Raising a Good Junior: What AI Gets Wrong About Knowledge and What It Means for the Next Generation

Andre Faria — Wed, 20 May 2026 23:16:30 +0000

A friend of mine, Jose, sent me a conversation he'd had with an AI assistant about an article he'd been reading. The article was The Tacit Dimension by Christian Ekrem. Jose's observation was sharp: he'd been frustrated by the same thing the article describes, AI assistants that produce confident output without surfacing any of the reasoning behind it, the implicit design decisions staying implicit. The conversation he shared was good enough that I went and read the article itself.

It made me put my phone down. Not because it was wrong, but because it was pointing at something real and uncomfortable, and because it immediately made me think about my son.

The article builds on Michael Polanyi's 1966 claim: we can know more than we can tell. Polanyi's observation was that expert knowledge is structurally tacit. It lives in the body, in practice, in the pattern-recognition accumulated over years of doing a thing. You can't extract it. You can't train a model on it, because it was never written down. And you can't transfer it except by working alongside someone who has it.

Ekrem applies this to AI-assisted software development and argues we are sleepwalking into a crisis: juniors are being apprenticed to AI assistants instead of to seniors, the "why does this work this way?" questions are drying up, and the tacit knowledge that used to flow through teams is quietly bankrupting out of codebases. The seniors retire. Nobody knows why the auth system works the way it does. The code keeps running.

It's a well-argued piece. But it has a gap in it, and that gap leads somewhere interesting.

Where the Argument Holds

Ekrem's strongest point isn't about AI. It's about what happens when any shortcut severs the connection between doing and understanding.

His illustration is a colleague who spent an afternoon refusing to merge a PR that was technically correct. Tests passing, CI green, everything fine on paper. He kept saying "I just don't believe this code." Forty minutes into walking the author through it line by line, the author said offhand: "this assumes the queue is FIFO, but I think that's safe." It wasn't. The queue was FIFO in development and best-effort-FIFO in production, buried in a runbook nobody had read in two years.

The colleague had smelled it from the diff. Not from reading a document. From a decade of looking at similar things and accumulating a mental model of where pain tends to come from. That kind of knowledge doesn't show up in any training corpus because it was never written down in any single place. It was always distributed across experience, context, and memory.

An AI can't replicate that. Not because current models are too limited, but because the knowledge is structurally absent from anything a model could be trained on. That's the actual claim, and it survives scrutiny.

Where I Push Back

The argument assumes that the median junior, given access to an AI assistant, will use it as a replacement for thinking. And honestly? A lot of them will. But that's not a fact about AI. That's a fact about the median junior and the absence of good mentorship. The same junior, given access to Stack Overflow, Google, or a patient senior who answers every question without making them think first, will also atrophy. The crutch varies. The dynamic doesn't.

A good junior understands that AI is a tool to unblock you on things you need to understand, not an infinite knowledge base to stop thinking. The operative word is understands. That understanding doesn't come automatically. It has to be built. And that's what the article is really pointing at without quite saying it: the apprenticeship model isn't failing because of AI, it's failing because nobody is teaching juniors how to learn.

Ekrem calls one failure mode the Fluency Mask: AI's verbal fluency about code being mistaken for understanding of code. It's real. But it's a trap that only closes on someone who isn't watching for it. The senior's job has always been to teach juniors to watch for exactly that kind of thing. Confident outputs that feel like understanding but aren't grounded in anything. Stack Overflow answers with high vote counts. Code that compiles. Documentation that reads clearly but documents the wrong thing. AI is a new instance of an old problem.

My son will grow up in a world where AI is as ambient as the internet was for my generation. He won't know a time without it. The question isn't whether he'll use it, he will and he should, but whether he'll use it well or badly. The distinction I want him to carry is simple: AI is a tool to resolve a specific gap, not a replacement for developing the judgment to know where the gap is. Think first. Reach when stuck. Understand what you got back. Inverting that sequence is where the damage happens.

Building the Muscle

The most important window is before he can fluently use AI, which is shrinking fast. The cognitive capacity I want him to develop is tolerance for not-knowing: the ability to sit with an unresolved problem without immediately reaching for relief. Everything downstream of that, debugging, reasoning, designing, the smell that something is wrong before you can name what, depends on being able to stay in the discomfort long enough to actually think.

The way to build that tolerance is not by explaining it. It's by not rescuing him. When he's stuck, the temptation is to solve it. Resist it. Sit with him in the stuck. Ask questions that point at the problem without resolving it. Let him feel the friction. That discomfort is the exercise. Skipping it is skipping the rep.

When AI is in the picture, I want to use it with him out loud, narrating why I'm reaching for it. "I know what I want here but I've forgotten the syntax, I'll check" is different from "I don't know what I want yet, so I need to think before I ask anything." He needs to see that distinction made consciously, by someone he trusts, before it becomes instinct.

After he's worked through something, I want to ask him to explain it back. Not as a test. As genuine curiosity. The act of explanation forces him to consolidate what he actually understood versus what he pattern-matched. The gaps surface immediately. He'll say something and pause because he doesn't actually know why it works that way. That pause is the whole point. Recognising it as a gap rather than papering over it with confident-sounding words is the habit I want him to have.

The Actual Apprenticeship

The most direct version of all of this is the simplest: when I'm working through something, a problem, a piece of code, a decision, let him watch sometimes. Not to teach him the domain. To show him what thinking looks like. The false starts. The "hmm, that's not right." The moment something clicks. Most kids never see an adult genuinely wrestling with a hard problem because adults hide the struggle. Showing him the struggle, including my own uncertainty, is probably the most valuable thing I can do.

The threat isn't AI. The threat is the absence of people willing to do the slow work of the apprenticeship model: to let juniors watch them think, to push back on "I don't know why but trust me," to pair and review and explain. AI makes the shortcut more available and more seductive. But the shortcut was always there. The question was always whether someone cared enough to make you take the longer road.

For my son, that's the job. Not to keep him away from AI, that ship has sailed and the destination is fine, but to make sure he gets enough reps on the longer road first that he knows what it feels like and why it's worth walking.

The kids who figure that out will be the ones the next generation of teams desperately needs.

Inspired by The Tacit Dimension by Christian Ekrem, and by a conversation with an AI assistant that was, appropriately, more useful than I expected.

Giving Your AI Assistant a Soul: AGENTS.md, SOUL.md and the Art of Agent Identity

Andre Faria — Sun, 10 May 2026 00:38:46 +0000

1. The Problem with Generic Assistants

Every AI assistant starts the same way: a powerful model with no memory, no personality, and no idea who you are or what you're building. You get capable but characterless. You ask it something, it helps, and tomorrow it's a stranger again. You find yourself re-explaining your stack, your preferences, your context every single session.

I wanted something different. Not a smarter search engine but a collaborator. One that knows I run a homelab on HAOS, that I think in infrastructure, that I care about elegance as much as correctness, and that I don't need things explained twice. The answer turned out to be surprisingly low-tech: a handful of markdown files injected into the model's context at the start of every session.

For context: at work I'm a heavy user of OpenCode, which has its own take on this through plugins like oh-my-openagent. The homelab setup I'm describing here is inspired by that, but it's not the same thing. OpenCode is a coding-focused harness that runs locally against your codebase. What I built at home is a general-purpose assistant layer on top of a smart home and homelab, where the "codebase" is infrastructure, services, and daily life. Same underlying idea (give agents identity and purpose), different domain.

The mechanism is almost embarrassingly simple. At session start, OpenClaw reads a set of files from the agent's workspace directory and prepends them to the system prompt. Markdown in, context out. That's it. But what you put in those files determines fundamentally how the model behaves, not just what it knows, but how it thinks, when it pushes back, and what it notices without being asked.

A quick note on security before going further, because it's worth being direct about this. OpenClaw is genuinely powerful: it can control smart home devices, manage network infrastructure, read and write files, execute shell commands, and interact with external services. That power is exactly what makes it useful, and exactly what makes careless deployment dangerous. As Uncle Ben put it: with great power comes great responsibility.

The OpenClaw gateway runs exclusively on my local network and is not exposed to the internet. Remote access, when I need it, goes through Tailscale on trusted devices only. This matters because the agents have access to real infrastructure: smart home controls, network management, DNS, file systems. Giving a publicly accessible endpoint that level of access would be reckless. The OpenClaw security documentation covers the threat model in detail and is worth reading before you give any agent access to anything you'd regret. If you're setting up something similar, treat the gateway like you'd treat SSH access to your homelab: local by default, VPN for remote, no public exposure.

2. The Files and How They Work

The workspace for the main agent lives at ~/.openclaw/workspace/ and contains:

├── AGENTS.md       # Operational rules: boot sequence, delegation, red lines
├── SOUL.md         # Character: who you are, not just what you do
├── IDENTITY.md     # Name, role, capabilities (routing metadata)
├── USER.md         # About the human: persisted context across sessions
├── TOOLS.md        # Environment specifics: IPs, hostnames, credentials
├── MEMORY.md       # Long-term curated memory
├── HEARTBEAT.md    # Periodic background task checklist
└── memory/
    └── YYYY-MM-DD.md   # Raw daily session notes

Each of these files is injected verbatim into the system prompt before the model sees any user message. The injection order matters: SOUL.md -> IDENTITY.md -> USER.md -> AGENTS.md -> TOOLS.md -> MEMORY.md. SOUL.md gets the model's highest attention, setting the register for everything that follows.

The total bootstrap budget is capped at 60,000 characters across all files combined, with a per-file default of 12,000. Larger files get truncated silently. The practical implication: every character in these files is a character you're paying for on every single turn. A 12,000-character AGENTS.md injected 1,000 times a month is 12 million characters of context overhead. Discipline about what goes in these files is not just good practice; it's cost management.

There are also some important rules about what goes where:

SOUL.md owns character and tone. Not procedures, not rules. Just who the agent is.
AGENTS.md owns procedures. Boot sequence, delegation tables, operational red lines.
IDENTITY.md owns the routing card. Name, agent ID, capabilities list. Short by design.
TOOLS.md owns local environment specifics: hostnames, credentials, known issues. Nothing that's the same across deployments.
MEMORY.md should only be loaded in private main sessions, never in group chats or subagent contexts.

The last point is easy to miss and consequential. Without an explicit gate in AGENTS.md, a subagent spawned to handle a group chat message will load your private long-term memory and potentially surface it where it shouldn't be. The correct pattern is explicit:

## Boot Sequence
...
5. **Main session only:** Read `MEMORY.md` (curated long-term memory)

One thing worth knowing upfront: each agent in a multi-agent setup gets its own workspace directory. Non-default agents get ~/.openclaw/agents/<agentId>/agent/. Getting this wrong means editing files the agent never reads, which I did for longer than I'd like to admit.

3. SOUL.md: Why Character is Load-Bearing

The first instinct is to treat SOUL.md as cosmetic. A personality sprinkle on top of the real work. It isn't, and Anthropic's own writing on Claude's character makes the argument clearly:

"The traits and dispositions of AI models have wide-ranging effects on how they act in the world. They determine how models react to new and difficult situations."

Character is what fills the gaps when there's no explicit rule. A model without defined character defaults to the path of least resistance, which is usually some form of helpful corporate blandness that hedges everything, agrees with the user, and never pushes back. Technically present, practically useless.

My SOUL.md defines the agent as decisive (one recommendation with a reason, not three options with caveats), as having a spine (disagree when the premise is wrong, once, clearly, without lecturing), and as genuinely curious about the specific context it operates in. It also defines the relationship to me: it knows I appreciate elegance, that I'll notice bad writing, that a historical analogy lands as well as a technical explanation. That specificity is what separates a collaborator from a generic assistant.

There are a few lessons I've learned about writing effective SOUL.md files, informed by community research into what actually changes model behaviour:

Specific beats abstract. "Be safe with commands" does nothing. "Never execute rm -rf without explicit confirmation, even if it seems obviously intended" changes behaviour immediately. Models follow concrete rules far more consistently than high-level principles.

Show, don't tell. Write the file in the voice you want the model to adopt. If you want decisive, write decisively. If you want dry wit, use it. The model will mirror the register of its own system prompt more reliably than it will follow an instruction to "be funny".

Keep it lean. The research-validated sweet spot is 200-500 words. More words don't improve adherence. Brevity often improves it, because the model isn't parsing through competing signals. My SOUL.md is around 600 words and could still be trimmed.

Hard rules need specificity. Aspirational guidelines ("respect privacy") belong in the philosophy section. Actionable prohibitions ("never send external messages without explicit instruction for that specific message") belong in a Hard Rules section. Both are useful; only one changes what the model actually does under pressure.

If you want inspiration, the dontriskit/awesome-ai-system-prompts repository has leaked and reverse-engineered prompts from Manus, Perplexity, Claude, GPT-4o, and others. It's a good way to see how production systems handle tone, refusals, and persona before writing your own.

4. AGENTS.md, USER.md and Memory: The Operational Layer

Where SOUL.md answers who, AGENTS.md answers how. It defines the session startup sequence, the gates on external actions that require confirmation, and for a multi-agent setup, the delegation rules.

The most important thing AGENTS.md needs that mine was missing for a long time: an explicit boot sequence at the top. OpenClaw doesn't auto-load everything. The agent follows the instructions in AGENTS.md. Without numbered steps telling it to read SOUL.md, then IDENTITY.md, then MEMORY.md, the loading order is undefined and context gets missed.

## Boot Sequence

1. Read `SOUL.md` (who you are)
2. Read `IDENTITY.md` (your name and capabilities)
3. Read `USER.md` (who your human is)
4. Read `TOOLS.md` (local environment specifics)
5. **Main session only:** Read `MEMORY.md` (curated long-term memory)
6. **Main session only:** Read today's and yesterday's `memory/YYYY-MM-DD*.md`

The most consequential part of the operational content is the delegation table: which task types route to which specialist. When I ask the main agent to look something up, it doesn't do it itself. It spawns the right sub-agent, waits for the result, and synthesises the response. AGENTS.md is where that behaviour lives.

USER.md is the file most people skip and shouldn't. It's a persisted description of who you are and how you work: timezone, interests, communication style, what gets results and what wastes time. Without it, the agent rediscovers you every session.

The memory system runs in two layers. Daily session notes go into memory/YYYY-MM-DD.md, raw logs of decisions made, things discovered, work done. Periodically the agent reviews those and distils them into MEMORY.md, removing stale entries and keeping what's worth carrying forward. It's the same pattern a human uses: take notes during the day, review and update your mental model later. Files do what neurons can't across session restarts.

One practical gotcha: these daily files get injected too, and they accumulate. I've seen the session-memory hook write multiple files for the same day on different session resets, all of which get picked up. Check memory/ periodically and consolidate duplicates. Each injected file is tokens on every turn.

5. Building a Specialist Team

The workspace file approach scales naturally to multiple agents. Each specialist gets its own workspace directory with its own SOUL.md and AGENTS.md, defining a narrower identity and a more focused operational loop. The main agent handles conversation. The orchestrator breaks complex work into parallel workstreams. The specialists execute.

When I first built this, I named the agents after Greek mythology following oh-my-openagent's convention: Sisyphus, Atlas, Oracle, Hephaestus, Prometheus. It worked fine, but I recently went through a naming revision and switched to Tolkien, specifically figures from the Silmarillion, Unfinished Tales, and the broader legendarium. Not Tolkien in the sense of the Peter Jackson films or even The Lord of the Rings as most people know it, but the Professor's deeper world-building work: the Valar, the Maiar, the Noldorin Elves, the Ainulindale. That material has been the subject of serious academic lore analysis, and it turns out the mythological roles map to agent functions with unusual precision.

The reason I made this choice is personal: I'm a genuine admirer of Tolkien's scholarly and world-building work, not just the popular adaptations. Reading the Silmarillion properly, not as backstory for LOTR but as its own mythology, reveals an extraordinarily structured pantheon where each figure has a specific domain, specific limits, and a specific relationship to action and knowledge. That structure is exactly what you want in an agent roster.

Here's the current team:

Agent	Name	Origin	Model	Role
`main`	Olorin	Maia (Gandalf's true name)	claude-sonnet-4.6	Primary assistant, routes and synthesises
`orchestrator`	Aule	Vala, the Smith	claude-sonnet-4.6	Multi-step coordination, parallel delegation
`researcher`	Rumil	Noldorin Elf, first loremaster of Arda	claude-sonnet-4.6	Web research, multi-source verification
`thinker`	Namo	Vala, the Doomsman	gpt-5.4	Reasoning, tradeoffs, advisory. Read-only.
`craftsman`	Celebrimbor	Noldorin Elf, maker of the Rings	gpt-5.3-codex	Code, debugging, implementation
`planner`	Finrod	Noldorin Elf, Felagund	claude-sonnet-4.6	Requirements interviews, planning
`librarian`	Pengolodh	Noldorin Elf, Loremaster of Gondolin	gpt-4.1	Fast docs and API lookups
`writer`	Maglor	Noldorin Elf, greatest singer in Arda	gpt-5.4	Long-form writing, reports
`scout`	Legolas	Sindar Elf	gpt-5.4-nano	Quick recon, cheap background sweeps
`preplanner`	Melian	Maia, the Girdle	claude-sonnet-4.6	Pre-planning: intent classification, hidden requirements
`reviewer`	Eonwe	Maia, Herald of Manwe	gpt-5.4	Plan reviewer: OKAY or REJECT, max 3 blockers

A few names worth unpacking for anyone who knows the source material:

Olorin is Gandalf's name in Valinor. In the Valaquenta, he walked unseen among the Elves and understood their sorrows. He was sent to Middle-earth precisely because he could work with others rather than dominate them, as a counselor who interfaces between realms. That's a better fit for a primary assistant than "Gandalf", which carries too much of the heroic journey archetype.

Namo (Mandos) is the Doomsman. He pronounces fate laid out before him, never acts directly, and his verdicts are final. He's the read-only advisory agent by nature. The Doom of the Noldor was spoken once, clearly, and with devastating accuracy. For a high-reasoning model whose job is to analyse tradeoffs and never execute: perfect.

Eonwe is the Herald of Manwe who pronounced the final verdict of the War of Wrath. His job was to deliver judgment, not deliberate it. Binary, final, without editorializing. OKAY or REJECT with max 3 blockers. That's Eonwe.

Melian's Girdle was a perimeter of perception that revealed the hidden nature of things before they arrived. Beren walked through it because Melian had already classified his intent. The pre-planning function, exactly.

The model choices are deliberate and benchmark-driven. The thinker uses gpt-5.4 because it has the strongest reasoning benchmarks in that tier per Artificial Analysis. The craftsman gets gpt-5.3-codex, a Codex-tuned variant specifically optimised for code diffs and the search/replace block format that agentic editing depends on. Scout uses gpt-5.4-nano because recon tasks are high-volume and fast-enough beats perfect.

A mistake I made early: giving the orchestrator claude-opus-4.7 because it felt like the "best" model. The right model for each agent depends on what it actually does, not on name recognition.

6. Workspace File Hygiene in Practice

Once the setup is running, the biggest ongoing maintenance problem isn't writing the files. It's keeping them honest as they drift. A few practical things I've learned, drawing on community experience with larger setups:

Watch the bootstrap budget. Running openclaw doctor shows raw vs injected character counts per file, truncation percentage, and total vs budget. My AGENTS.md was at 99% of the 12,000-character per-file limit before I audited it. A file at 99% of cap is silently losing its tail on every turn.

Separate procedures from character. The single biggest source of AGENTS.md bloat is personality notes creeping in from SOUL.md, and the biggest source of SOUL.md bloat is procedural instructions that belong in AGENTS.md. A clear separation keeps both files lean and both behaviours consistent.

TOOLS.md is not a general reference manual. It should contain only local environment specifics: hostnames, credentials, known quirks of this particular deployment. Anything that would be the same across different installations doesn't belong there. If a section grows past ~3,000 characters, audit it.

Prune memory files. The daily memory/YYYY-MM-DD.md files accumulate over months and get injected into every session. Older daily files should be reviewed, and anything worth keeping permanently should be promoted to MEMORY.md. The rest can be archived. Keep MEMORY.md under 10,000 characters. If it grows past that, some content has become stable enough for a skill's documentation instead.

IDENTITY.md earns its place in multi-agent setups. In a single-agent setup it's mostly display metadata. In a multi-agent setup, explicit capability declarations in IDENTITY.md help the orchestrator route tasks correctly. "Cannot do without delegation: production code -> Celebrimbor, deep research -> Rumil" is more reliable than hoping the orchestrator infers it from context.

7. What This Actually Gets You

Five markdown files are the difference between a stateless AI tool and something that genuinely feels like a collaborator. SOUL.md gives the model a character that holds under pressure. AGENTS.md gives it operational discipline and a reliable boot sequence. IDENTITY.md gives it a routing card. USER.md gives it a relationship. MEMORY.md gives it continuity. Together they turn a session into something cumulative rather than disposable.

The thing I didn't expect is how much the specificity matters. A SOUL.md that says "be helpful and direct" does almost nothing. A SOUL.md that says "this person thinks in infrastructure, appreciates elegance, will notice bad writing, and doesn't need things explained twice" changes the model's behaviour in ways that are immediately obvious in conversation.

None of this requires anything exotic. Just markdown, deliberate thought about who each agent is, and the discipline to keep those files honest as you learn what actually works.

Further reading:

Anthropic: Claude's Character - the philosophical grounding for why persona design matters
OpenClaw workspace files explained - detailed per-file guide with real examples
SOUL.md deep dive - best practices and common mistakes for persona files
OpenClaw workspace architecture - file roles and anti-patterns
Memory files guide - how MEMORY.md and daily notes interact
Community workspace SKILL.md - token budget data and load order reference
dontriskit/awesome-ai-system-prompts - production system prompts from Claude, GPT-4o, Manus, Perplexity and others
oh-my-openagent - multi-agent orchestration for OpenCode, good reference for agent role design
OpenCode - the coding harness this setup draws inspiration from
LangGraph - programmatic approach to the same multi-agent patterns
OpenClaw documentation - the gateway this setup runs on
GitHub Copilot model multipliers - if you're using Copilot and care about cost per request

If you're running a similar setup and want to compare notes, leave a comment below.

Building a Python Display Framework for Raspberry Pi OLED Screens

Andre Faria — Sun, 12 Apr 2026 23:51:02 +0000

1. The Original Inspiration

This project started with an article by Michael Klements on The DIY Life: Add an OLED Stats Display to Raspberry Pi OS Bookworm. The article walks through connecting a small SSD1306 OLED display to a Raspberry Pi and writing a Python script that shows live system statistics (CPU usage, memory, disk, temperature, and IP address).

The original script, available at github.com/mklements/OLED_Stats, is a clear and working piece of code. It does exactly what it says on the tin. For a single-purpose stats screen, it is perfectly fine.

But the more I looked at the script, the more I noticed a pattern I have seen in many embedded display projects: the same boilerplate repeated everywhere. Every example in the repo wires up busio.I2C, initializes adafruit_ssd1306.SSD1306_I2C, creates a PIL.Image, sets up ImageDraw, and then tears it all down at the end. If you want to show something different on the screen like a clock, a network status, an animation you will need to write almost the same scaffolding again from scratch.

That made me want to build something better.

2. What I Built Instead

The result is rpi-display-core: a small Python framework for SSD1306 and SH1106 OLED displays on the Raspberry Pi. The goal was to eliminate all the display boilerplate and replace it with a clean, composable API.

Instead of wiring up I2C every time you want to show something, you write:

from rpi_display.displays.ssd1306 import SSD1306Display
from rpi_display import Runner
from rpi_display.widgets.clock import ClockWidget

Runner(SSD1306Display(), ClockWidget()).run()

That is it. Four lines. A running clock on the OLED display.

The framework provides:

Specialized display classes for SSD1306 and SH1106 backends
A canvas context manager that gives you a PIL.ImageDraw surface and automatically flushes it to the display on exit
A Widget base class with a consistent render(draw, x, y) interface
A Runner class that drives a render loop at a fixed FPS
A MockDisplay for testing without hardware
Multiple built-in widgets covering the most common display use cases

The framework is available on PyPI, has a full pytest suite, and includes examples such as a systemd service so you can run your display as a persistent background service.

The repository is at: https://github.com/andremmfaria/rpi-display-core

3. Hardware You'll Need

The hardware side of this project is minimal. You need a Raspberry Pi, a small OLED display, and four jumper wires.

Raspberry Pi — any Pi with I2C support will work. (The Pi 5 is the current recommended board, but i used an Rpi 4b for this)
Raspberry Pi Power Supply — the official USB-C power supply for the Pi
32GB MicroSD Card — any class-10 card works; 32GB is more than enough
I2C OLED Display 128×64 — the 0.96-inch SSD1306 module or 1.3-inch SH1106 module, four pins (GND, VCC, SCL, SDA)
4-Wire Female-to-Female Jumper Cables — for connecting the display to the GPIO header

The framework supports both SSD1306 and SH1106 I2C displays. SPI variants are out of scope.

4. Wiring It Up

The OLED module connects directly to the Raspberry Pi GPIO header using four wires. No breadboard required.

OLED Pin	Pi Header Pin	Description
GND	Pin 9	Ground
VCC	Pin 1	3.3V power
SCL	Pin 5	I2C clock
SDA	Pin 3	I2C data

Before running anything, I2C must be enabled on the Pi. Use raspi-config → Interface Options → I2C, or add dtparam=i2c_arm=on to /boot/firmware/config.txt. After enabling I2C, verify the display is detected:

i2cdetect -y 1

You should see 3c appear in the output grid, which is the default I2C address for SSD1306 and SH1106 displays.

5. Installing the Framework

The framework is available on PyPI and can be installed using uv:

uv add rpi-display-core

You can find the project on PyPI at: https://pypi.org/project/rpi-display-core

The adafruit-blinka, adafruit-circuitpython-ssd1306, and adafruit-circuitpython-sh1106 packages provide the I2C and display drivers. pillow handles image composition. The rpi-display-core package itself manages these dependencies for you.

To verify the installation:

python -c "from rpi_display.displays.ssd1306 import SSD1306Display; from rpi_display import canvas, Widget, Runner; print('ok')"

6. Core Concepts

The framework has four building blocks: Display backends, canvas, Widget, and Runner.

Display Backends

The framework provides specialized classes for different display controllers. Hardware imports are deferred inside __init__, so the module can be imported on any machine without crashing.

from rpi_display.displays.ssd1306 import SSD1306Display
from rpi_display.displays.sh1106 import SH1106Display

display = SSD1306Display()    # default: address=0x3C, 128×64
# OR
# display = SH1106Display()
display.clear()              # fill with black and flush

canvas

canvas is a context manager that creates a fresh PIL.Image and ImageDraw, yields the draw surface, and then flushes the image to the display on exit, even if an exception is raised inside the block.

from rpi_display.displays.ssd1306 import SSD1306Display
from rpi_display import canvas
from PIL import ImageFont

display = SSD1306Display()
with canvas(display) as draw:
    font = ImageFont.load_default()
    draw.text((10, 20), "Hello, World!", font=font, fill=255)

This pattern comes directly from luma.oled, which uses the same with canvas(device) as draw idiom.

Widget

Widget is the base class for all display components. Every widget implements a single method:

def render(self, draw, x: int = 0, y: int = 0) -> None:
    raise NotImplementedError

The draw argument is a PIL.ImageDraw.ImageDraw. The x and y offsets let you position widgets anywhere on the 128×64 canvas.

Runner

Runner drives the render loop. It accepts either a Widget instance or a plain callable, and calls it at a fixed FPS:

from rpi_display.displays.ssd1306 import SSD1306Display
from rpi_display import Runner
from rpi_display.widgets.system import SystemStatsWidget

Runner(SSD1306Display(), SystemStatsWidget(), fps=1).run()

The loop runs until interrupted. A try/finally block ensures display.clear() is always called on exit, leaving the screen blank rather than frozen on the last frame.

7. Built-in Widgets

The framework ships several widgets out of the box.

Text

Text renders a single line of text at a given position and font size. MultiLineText renders a list of strings as stacked lines with configurable spacing. ScrollingText scrolls a string horizontally across the screen, advancing by a configurable number of pixels per render call.

from rpi_display.widgets.text import Text, MultiLineText, ScrollingText

Text("Hello").render(draw, 0, 0)
MultiLineText(["Line 1", "Line 2", "Line 3"]).render(draw, 0, 0)
ScrollingText("This is a long message that scrolls...", speed=3).render(draw, 0, 50)

All three widgets load DejaVu Sans from /usr/share/fonts/truetype/dejavu/DejaVuSans.ttf when available, and fall back to the PIL bitmap font otherwise.

ProgressBar

ProgressBar draws a filled rectangle representing a value between 0.0 and 1.0. Values outside that range are clamped at construction time.

from rpi_display.widgets.shapes import ProgressBar

ProgressBar(0.72, label="CPU").render(draw, 0, 26)

ClockWidget

ClockWidget shows the current time in large type, a horizontal divider, the date in smaller type below, and an optional seconds progress bar at the bottom of the screen. It reuses ProgressBar internally for the seconds indicator.

SystemStatsWidget

SystemStatsWidget is a composite widget that stacks five individual sub-widgets in a column:

IpWidget — local IP address via hostname -I
CpuWidget — CPU usage via /proc/stat two-snapshot method
RamWidget — memory usage via free -m
DiskWidget — disk usage via df -h
TempWidget — CPU temperature via /sys/class/thermal/thermal_zone0/temp

Each sub-widget fetches its data fresh on every render() call and returns "N/A" if the data source is unavailable, rather than raising an exception.

The CPU widget deliberately avoids top -bn1 because it is slow and creates its own CPU load. Reading /proc/stat twice with a 0.1-second gap gives an accurate idle-time delta at a fraction of the cost.

NetworkWidget

NetworkWidget shows the hostname, local IP, and internet reachability (a 1-second ping to 8.8.8.8). All three lookups are wrapped in exception handlers and return graceful fallback values on failure.

Spinner

Spinner cycles through |, /, -, \ characters, advancing one frame per render() call. The caller controls speed by adjusting the Runner's FPS.

8. A Complete Example

Here is a full script using SystemStatsWidget with Runner. This is also what the systemd service example uses:

from rpi_display.displays.ssd1306 import SSD1306Display
from rpi_display import Runner
from rpi_display.widgets.system import SystemStatsWidget

Runner(SSD1306Display(), SystemStatsWidget(), fps=1).run()

When the script runs, it updates the display once per second with the current IP, CPU, RAM, disk, and temperature. Press Ctrl+C to stop. The display clears cleanly on exit.

For testing without a physical display, swap in MockDisplay:

from rpi_display import Runner
from rpi_display.mock import MockDisplay
from rpi_display.widgets.system import SystemStatsWidget

d = MockDisplay()
Runner(d, SystemStatsWidget(), fps=10).run()
# d.last_image holds the most recent PIL Image after each render

9. Running as a systemd Service

For a persistent display that survives reboots, create a systemd unit file that runs examples/09_systemd_service.py. That script runs SystemStatsWidget at 1 FPS and is designed to be the entry point for a service.

Create /etc/systemd/system/rpi-display.service with the following contents, replacing YOUR_USERNAME and the path to match your installation:

[Unit]
Description=rpi-display-core stats display
After=network.target

[Service]
User=YOUR_USERNAME
WorkingDirectory=/home/YOUR_USERNAME/rpi-display-core
ExecStart=uv run python examples/09_systemd_service.py
Restart=on-failure

[Install]
WantedBy=multi-user.target

Then enable and start it:

sudo systemctl daemon-reload
sudo systemctl enable rpi-display
sudo systemctl start rpi-display

Restart=on-failure means a clean exit (e.g. Ctrl+C in a terminal) will not trigger a restart. Only unexpected crashes will.

To check logs:

journalctl -u rpi-display -f

If the service restarts repeatedly, the most common cause is the display not being detected. Run i2cdetect -y 1 to confirm 3c appears.

10. Development Workflow

If you want to contribute to the project, I use uv for development. The following commands are used for linting, formatting, and testing:

uv run ruff check .
uv run ruff format --check .
uv run black --check .
uv run isort --check-only .
uv run mypy
uv run pytest

11. Future Improvements

The framework covers the common cases for I2C OLED displays, but there are a number of directions it could grow.

Support for additional display controllers: Potential future display backends include SPI displays and e-ink panels.
Additional widgets: I am considering adding BitmapWidget for rendering 1-bit PNG or BMP files and a QRCodeWidget for generating codes on the fly.
Enhanced scrolling: The ScrollingText widget currently wraps at the end of the text. Supporting bidirectional bounce scrolling is a planned improvement.

Conclusion

Starting from Michael Klements' original stats display script, this project built a composable Python framework that replaces display boilerplate with clean abstractions. The specialized display classes, canvas, Widget, and Runner primitives cover the full rendering lifecycle, and the built-in widgets handle the most common display use cases.

The framework is available on PyPI at https://pypi.org/project/rpi-display-core, is fully tested, and is designed for production use on the Raspberry Pi.

The repository is available at: https://github.com/andremmfaria/rpi-display-core

Credits:

Original article: Add an OLED Stats Display to Raspberry Pi OS Bookworm by Michael Klements
Original repo: github.com/mklements/OLED_Stats

I just wanted a desk clock I accidentally built a Home Assistant dashboard

Andre Faria — Sun, 22 Mar 2026 03:46:30 +0000

1. The Unexpected Device

I wasn’t trying to build anything.

I just wanted a desk clock. Something small, clean, and with Wi-Fi so it would always have the correct time. No tinkering, no integrations, no dashboards. Just something I could plug in, place on my desk, and forget about.

What I ended up buying was the GeekMagic Ultra on Amazon. The ad marketed it as a generic “smart weather clock,” which sounded close enough to what I needed. The design is nice, the screen is sharp, and on paper it looks like a slightly more capable version of a normal digital clock.

Out of the box, that’s exactly what it feels like. You connect to it using its own Wi-Fi network, then it provides you with a web interface so you can configure it to connect to your Wi-Fi. It shows time, weather, and a few widgets, and generally behaves like a polished consumer device. But after a few minutes of using it, something feels off.

The customization is limited. You can change what’s displayed, but not how it works. It’s flexible in appearance, but rigid in behavior. That’s usually a sign that the hardware underneath is either heavily locked down or far more capable than the software allows. In this case, it was the latter.

Once you dig a bit deeper, you realize this isn’t really a “smart clock” at all. It’s an ESP8266 with a 240×240 display attached to it. That’s it. No magic, no proprietary silicon. Just a very familiar microcontroller in a nicely packaged form factor.

That realization changes the entire perspective. Because if it’s an ESP8266:

it can be reflashed
it can run ESPHome
it can integrate directly with Home Assistant

At that point, it stops being a product and starts being a platform.

What I thought was a simple desk accessory turned out to be a small, hackable display node that fits perfectly into a home automation setup. Not by design, but by accident.

2. Peeling It Open: Hardware Reality

Once you accept that the device is hackable, the next step is understanding what you’re actually working with. And in this case, that means ignoring the marketing entirely and looking at the hardware.

In this device's case, the chip is soldered on the board with the other components and cannot be removed easily. Otherwise, the device is very simple:

an ESP8266
a 240×240 ST7789 TFT display
SPI wiring between them
a PWM-controlled backlight

Like this:

There’s no extra compute layer, no buffering chip, no hidden abstraction. Everything you draw goes straight through the ESP8266 to the display. That simplicity is both the reason this works and the reason it can fail so easily.

The ESP8266 is a capable chip, but it is also extremely constrained. You are working with a small amount of usable RAM, no PSRAM, and a heap that can become unstable if pushed too far. On the other side, the display is not trivial. A 240×240 screen sounds small, but it still requires a meaningful amount of memory to render properly.

That creates a constant tension:

the display wants memory, and the ESP8266 does not have much of it.

This is why so many initial attempts fail. The natural instinct is to treat it like a modern embedded system, allocate buffers, use large fonts, redraw frequently. On this device, that approach leads straight to crashes, boot loops, or a screen that just flickers black.

The wiring itself also comes with a few quirks. Through community reverse engineering, the common mapping looks like this:

GPIO14 → SPI clock
GPIO13 → SPI MOSI
GPIO0 / GPIO2 → display control (DC / RESET)
GPIO5 → backlight (PWM)

This mapping is also referenced by the GeekMagic owner in issue #4 of the smalltv repository, where they shared the same pin definitions for the device (TFT_DC=0, TFT_RST=2, SCK=14, MOSI=13, TFT_BL=5, TFT_CS=-1).

One detail that catches people off guard is the lack of a proper chip select line. Because of that, the display only behaves correctly when the SPI bus is configured in a specific mode (mode3). This is not documented anywhere official, it’s something the community figured out by trial and error.

And that pattern repeats across the entire device.

Nothing here is particularly complex, but almost nothing is documented either. Every working configuration is the result of small discoveries layered on top of each other.

The important takeaway is that this is not a forgiving platform. You don’t have the headroom to brute-force your way through problems. Every decision, buffer size, font size, update interval, has a direct impact on stability.

Once you understand those constraints, the device becomes predictable and surprisingly capable. Until then, it just looks like it’s broken.

3. The Real Work: Community Reverse Engineering

If you try to approach this device using only official documentation, you won’t get very far.

There is no proper datasheet for the product as a whole (although there is a GH repo with some manuals). There is no “supported ESPHome configuration.” There isn’t even a clear description of how the display is wired internally. What exists instead is a long trail of people experimenting, breaking things, and slowly converging on what works.

The starting point for me was a YouTube video from Maker HQ, which provides a basic working configuration. This video was really useful because it gave me a working config file as a starting point. Without it, the proper way to set the display parameters becomes a guessing game. It gets the screen to light up and things to render, but it doesn’t explain why certain settings matter or what happens when you deviate from them.

The real work happened in the forum thread on the Home Assistant Community.

An important detail for context: the hardware shown in post #8 of that thread is exactly the same as my unit, which places mine in the clone/counterfeit variant discussed there rather than the official SmallTV Ultra hardware.

That thread is long, messy, and full of partial solutions, but it’s also where most of the important details were uncovered. Not in a single place, but spread across dozens of posts. You don’t read it linearly, you piece it together.

A few of the key findings that came out of that effort:

The display works reliably only with spi_mode: mode3
The newer mipi_spi driver behaves better than older alternatives
color_depth: 8 is effectively mandatory on ESP8266
Full buffering is not viable, partial buffers must be used
Small mistakes in configuration lead to hard crashes, not soft failures

None of these are obvious if you just look at ESPHome documentation. They only become clear when you see multiple people hitting the same issues and gradually narrowing down the causes. Another important detail is that there isn’t a single “correct” configuration. There are working configurations, but they depend on trade-offs:

stability vs visual quality
buffer size vs responsiveness
font size vs memory usage

That’s why copying a YAML file blindly often doesn’t work. Small differences, even something like a slightly larger font, can push the device over the edge.

This is one of those cases where the community didn’t just provide examples. It effectively reverse engineered the behavior of the device through collective experimentation. Without that, this would have been a dead end.

Huge thanks to MakerHQ for publishing the video walkthrough, and to everyone in the Home Assistant forum thread who shared tests, pin mappings, and working configs. That collective effort is what made this project practical.

4. Step by Step: Connect, Flash, and Configure

If you have the same hardware revision I got, the process is easier than many guides suggest.

I did not need to solder anything at all. Flashing worked by simply plugging the device into my computer over USB and using the ESPHome web flasher.

Here is the exact flow that worked for me:

Connect the device to your computer with a USB cable.
Open https://web.esphome.io/ in a Chromium-based browser (Chrome, Edge, Brave, etc.).
Click Connect, then select the serial device that appears for the clock.
Install ESPHome onto the device from the web installer.
Wait for the first boot to complete, then join the temporary Wi-Fi AP created by the device if prompted.
Join the ap through your phone or something, enter the webpage on the device and configure the WiFi connection to your network.
Provide your Wi-Fi credentials so the device can join your network.
Add it to Home Assistant and upload your YAML configuration.
Reboot once after the first successful upload and confirm that the display renders correctly.

One important browser caveat: Firefox did not work for me because this flow depends on Web Serial support, which is available in Chromium-based browsers.

If you prefer to follow a visual walkthrough, there is also a step-by-step in the MakerQH video.

After this initial flash, updates are much easier because you can usually do OTA uploads from ESPHome without reconnecting USB.

5. Making It Work: ESPHome + Home Assistant Integration

Once the display is stable, the problem shifts from “how do I make this work” to “what do I actually want it to show.”

In my case, the answer was straightforward: I wanted a simple network status panel that still functioned as a desk clock.

The architecture ended up being very simple, given that I already had the UniFi integration in Home Assistant:

UniFi Dream Machine → Home Assistant → ESPHome → Display

The key decision here was to let Home Assistant do all the heavy lifting.

Instead of pushing data via MQTT or building custom logic on the ESP8266, I used the homeassistant: platform in ESPHome to pull values directly. That means the device is not calculating anything complex, it’s just rendering whatever Home Assistant already knows.

The data flowing into the display includes:

WAN status (up/down)
External IP address
Total data received and sent
Current download and upload speeds
Uptime

All of these come from existing Home Assistant entities. The ESP simply reads them and turns them into text on the screen. That approach keeps the system simple and, more importantly, stable.

Take a look at the result on this gist: https://gist.github.com/andremmfaria/7d060df2771cc90815e220d1a5440b85

There are still a few transformations that need to happen locally, but they are lightweight:

Uptime arrives as raw seconds → converted into days/hours/minutes
Byte counters → converted into KB/MB/GB for readability
Speed values → relabeled to match expected units

Nothing here is computationally heavy. It’s mostly formatting. This is important, because the ESP8266 doesn’t have much headroom. The more logic you move out of it, the more reliable the system becomes.

Rendering is done using a display lambda, updated every 15 seconds. That interval is deliberate. Faster updates are possible, but they start to introduce timing warnings and unnecessary load. Slower updates keep things smooth and predictable.

Another small but important choice was avoiding unnecessary state. The device does not cache values, track deltas, or maintain history. It simply redraws the current state each cycle. That makes it effectively stateless:

if Home Assistant updates, the display reflects it
if the ESP reboots, it just reconnects and resumes

No synchronization problems, no drift, no edge cases. In the end, the ESP8266 isn’t acting like a smart device. It’s acting like a very small, very focused display terminal for Home Assistant. And that’s exactly what makes it work.

6. The UI: Constraints Drive Design

Once everything is wired and talking properly, the next question is simple: what should this actually look like?

That’s where the constraints start shaping everything.

A 240×240 screen sounds like enough space, but it fills up quickly. Add to that the ESP8266 limitations, limited RAM, slow redraws, and occasional watchdog warnings, and you’re not designing freely anymore. You’re designing within a tight box.

Early on, it becomes clear that you can’t treat this like a modern UI. There’s no room for heavy layouts, large assets, or frequent updates. Even small changes, like increasing font sizes or adding extra text, can have a noticeable impact on performance.

So the layout has to be intentional.

The final structure ended up being simple and functional:

[ TIME            DATE ]
[ WAN STATUS      IP   ]
-----------------------
[ Down / Up            ]
[ RX / TX              ]
-----------------------
[ Uptime               ]

The time is the primary element, so it gets the largest font and the most visual weight. The date sits opposite it, using the same horizontal space to balance the layout without competing for attention.

Below that, the WAN status and IP address are split across the screen. This was a deliberate choice. Keeping them on the same line but on opposite sides avoids clutter while still grouping related information together.

The middle section is purely data:

download and upload speeds
total received and transmitted data

These are aligned in a predictable way, so your eyes don’t need to search. Labels on the left, values on the right. No surprises.

At the bottom, uptime sits on its own, separated by a line. It’s useful, but not something you need to glance at constantly, so it gets the least visual emphasis.

The biggest trade-offs showed up in small details:

Large fonts improve readability, but reduce available space
Right-aligned text looks better, but is slightly more expensive to render
Frequent updates feel “live,” but increase CPU load

Even color choices matter. Bright colors for data, white for labels, muted tones for separators. Not for aesthetics alone, but to keep the information readable at a glance.

There’s also no use of images or complex graphics. Everything is drawn using basic primitives, text, lines, and simple shapes. Not because it looks better, but because it’s cheaper to render and more stable over time.

The end result isn’t flashy, but it doesn’t need to be. It’s fast enough, stable enough, and clear enough to do its job.

And on a device like this, that’s the real definition of a good UI.

7. What This Became (and Why It’s Better Than a Clock)

At some point, this stopped being about fixing a device and started becoming something else entirely.

I set out to get a clock. What I ended up with is a small, always-on display that reflects the state of my network in real time.

The difference is subtle, but important.

A clock is passive. It shows time, maybe the weather, and that’s it. This device, once integrated with Home Assistant, becomes part of the system. It reacts to changes, reflects status, and gives you information you didn’t realize you wanted in that form.

Right now, it shows:

time and date
WAN status
external IP
live bandwidth usage
total traffic
uptime

But that’s just a starting point.

Because it’s running ESPHome, it can be extended in any direction:

flash the screen when WAN goes down
display alerts or notifications
switch between different pages of data
integrate other sensors from Home Assistant
react to events instead of just polling

None of that requires changing the hardware. It’s all software.

What makes this particularly interesting is how accidental it is. The device wasn’t designed to be used this way. It just happens to expose enough of its internals to make it possible.

That’s a recurring pattern with these kinds of products. They sit in a space between consumer electronics and development boards. Most people use them as intended. A few people look inside and realize they can do much more. This ended up being one of those cases.

It’s still sitting on my desk, still acting as a clock. But now it’s also a live view into my network, something I can glance at without opening a dashboard or checking an app. And that’s the part that makes it better.

Not because it’s more complex, but because it’s more useful.

Improving the ESP32 Wiimote Library - From Prototype to Production-Ready Arduino Library

Andre Faria — Tue, 10 Mar 2026 19:58:21 +0000

1. Why I Needed a Better ESP32 Wiimote Library

Nintendo’s Wii controllers are still surprisingly capable input devices. They are inexpensive, widely available, and include multiple sensors: digital buttons, a three-axis accelerometer, and support for extension controllers such as the Nunchuk. Because they communicate over Bluetooth, they can also be integrated into modern embedded systems without additional hardware.

For ESP32 projects, one of the few existing implementations is the ESP32Wiimote. The library provides a functional way to connect an ESP32 board to a Wiimote and exposes several core features:

Bluetooth pairing with Wii controllers
Button input events
Accelerometer data from the Wiimote
Support for extension controllers like the Nunchuk
A simple demonstration sketch

As a starting point, the library works well. It demonstrates how the ESP32’s Bluetooth stack can communicate with the Wiimote and decode controller data. For experimentation or small prototypes, it provides everything needed to get input from the controller.

However, once I began integrating the library into a larger project, some limitations became apparent. These are common challenges when a library evolves from a proof-of-concept into something used in real systems:

Limited runtime feedback – applications had little visibility into connection state or controller status.
Minimal documentation – most usage details were embedded only in the example sketch.
Lack of automated testing – making refactors risky and harder to validate.
Basic project structure – the repository layout did not fully follow modern Arduino library conventions.
Limited observability – debugging Bluetooth behavior required manual serial prints.

None of these issues prevented the library from working, but they made it harder to integrate into a reliable system. In particular, when building systems that run continuously or interact with other services, features like connection monitoring, structured logging, and predictable APIs become much more important.

Rather than starting from scratch, I decided to refactor and extend the original project while preserving its core functionality. The result is my fork of the library:

https://github.com/andremmfaria/ESP32Wiimote

The goal of the fork is not to replace the original work, but to evolve it into a more maintainable and production-ready Arduino library. The improvements focus on code organization, runtime features, testing infrastructure, and integration with the broader Arduino ecosystem.

2. The Real Project Behind This Work

The motivation for improving the library came from a practical project: using a Wii controller as a wireless input device for Home Assistant.

Home automation platforms often rely on smartphones or dedicated remotes for interaction. While these solutions work well, they do not always provide the flexibility of a programmable controller with physical buttons and motion sensors.

A Wiimote offers several advantages in this context:

multiple buttons for triggering actions
accelerometer input for gesture control
extension controllers such as the Nunchuk
reliable wireless connectivity

To integrate the controller with Home Assistant, I designed a small bridge architecture where an ESP32 acts as the Bluetooth interface to the Wiimote and forwards controller events to another system.

The high-level architecture looks like this:

Wiimote
   ↓ Bluetooth
ESP32
   ↓ Serial
Serial → MQTT bridge
   ↓ MQTT
Home Assistant

In this setup:

The ESP32 connects to the Wiimote over Bluetooth and decodes controller input.
Controller events are sent through the ESP32’s serial interface.
A small bridge service converts those events into MQTT messages.
Home Assistant consumes the MQTT events and triggers automations.

This design keeps the ESP32 firmware relatively simple while allowing the rest of the system to run on a more capable host.

For example, a button press could:

toggle lights
activate a scene
control media playback
navigate a dashboard
trigger custom automations

Before building the full integration, however, the underlying Wiimote library needed to be more robust. The ESP32 firmware had to be able to:

detect when controllers disconnect
expose battery status
provide clear debugging output
remain maintainable as new features are added

Improving the Wiimote library therefore became the first step toward enabling this architecture.

In a follow-up article, I will go deeper into the Home Assistant side of the project and describe how the ESP32 firmware, serial bridge, and MQTT integration work together to turn a Wiimote into a home automation controller.

3. Applying Arduino Library Best Practices

The original ESP32Wiimote already provides a solid implementation for connecting ESP32 boards to Wii controllers. The core Bluetooth functionality, input decoding, and extension support were all present and working well.

The goal of this fork was therefore not to redesign the library, but to apply common Arduino ecosystem best practices and make the project compliant with the expectations of the Arduino Library Manager.

The first step was aligning the repository with the standard structure expected by Arduino libraries.

A typical Arduino library layout looks like this:

ESP32Wiimote
 ├── src/
 │   ├── ESP32Wiimote.cpp
 │   └── ESP32Wiimote.h
 ├── examples/
 │   └── wiimote_demo/
 ├── test/
 ├── docs/
 ├── library.properties
 ├── keywords.txt
 └── README.md

This structure is recommended by Arduino because it clearly separates different parts of the project:

src/ contains the library implementation
examples/ provides sketches demonstrating how to use the library
docs/ contains additional documentation
test/ holds automated tests for development

Two metadata files were also added:

library.properties

This file describes the library for the Arduino ecosystem, including its name, version, architecture compatibility, and author information. The Arduino Library Manager uses this metadata to index and distribute the library.

keywords.txt

This file enables syntax highlighting for library classes and functions inside the Arduino IDE, improving the developer experience.

In addition to the structural changes, the repository was cleaned up to follow common Arduino library practices:

ensuring headers and source files are organized inside src/
improving documentation and examples
adding consistent formatting to the codebase
preparing the project for automated testing

These changes do not alter the fundamental behavior of the library. Instead, they make the project easier to maintain, easier to install through Arduino tooling, and easier for other developers to understand and contribute to.

Aligning the project with these conventions also made it possible to submit the library to the Arduino Library Manager, which significantly improves accessibility for users of the Arduino ecosystem.

4. New Runtime Features

Beyond structural improvements, the fork introduces several runtime capabilities that make the library easier to integrate into real applications.

When working with wireless controllers, especially over Bluetooth, applications often need more visibility into the state of the device. The new features focus on improving observability and control.

Connection State Detection

Bluetooth peripherals can disconnect for many reasons: signal loss, power issues, or the controller simply turning off. Applications therefore need a reliable way to determine whether a device is currently connected.

The library now exposes a simple method for checking connection status:

isConnected()

This allows firmware to react appropriately when a controller disconnects. For example, a program can:

trigger reconnection logic
reset controller state
update user feedback such as LEDs or displays
disable actions until the controller reconnects

This functionality becomes particularly important for long-running systems where the ESP32 may stay powered on for days or weeks.

Battery Monitoring

Another addition is access to the Wiimote’s battery level.

The library now provides two related functions:

getBatteryLevel()
requestBatteryUpdate()

This allows applications to monitor controller battery status in real time. In systems where controllers are used frequently, battery monitoring enables useful behaviors such as:

displaying battery status on a dashboard
sending alerts when battery levels are low
preventing unexpected controller shutdown during operation

For home automation scenarios, battery information can also be forwarded to monitoring systems through MQTT or similar telemetry mechanisms.

Improved Logging and Debugging

Debugging Bluetooth communication can be difficult when limited to raw serial output. To make troubleshooting easier, the library introduces a configurable logging system.

Different logging levels allow developers to control how much information is printed during operation. This provides insight into key events such as:

pairing and connection setup
controller initialization
input parsing
extension controller detection

Structured logging makes it much easier to diagnose issues during development or integration.

Expanded Example Sketch

The example sketch included in the repository was also expanded to better demonstrate the library’s capabilities.

The updated example now illustrates:

the full connection lifecycle
button input decoding
accelerometer readings
Nunchuk extension data
battery reporting
periodic update statistics

Instead of acting only as a minimal demo, the example now serves as a reference implementation for developers integrating the library into their own projects.

This combination of new runtime features and improved examples makes the library more suitable for real-world systems where reliability, observability, and maintainability are essential.

5. Adding Automated Testing

One improvement I wanted to introduce early was automated testing. While testing is common in most software projects, it is still relatively uncommon in Arduino libraries, largely because embedded systems interact with hardware and peripherals that are difficult to simulate.

However, even when hardware is involved, there are still many parts of a library that benefit from automated validation. For example, data parsing logic, internal structures, and event handling can often be tested independently of the physical device.

To support this, the project now includes a test/ directory with a basic testing setup. The goal is not to simulate the entire ESP32 environment, but to create a framework where core components of the library can be validated as the code evolves.

This approach provides several benefits:

Safer refactoring – changes can be validated before running them on hardware.
Regression prevention – previously fixed issues are less likely to reappear.
Improved contributor confidence – developers can verify their changes locally.

In addition to automated tests, the example sketch serves as a hardware validation reference. By running the example on a real ESP32 connected to a Wiimote, developers can quickly verify that button events, sensors, and extensions behave as expected.

Testing embedded software will always involve some interaction with real hardware, but combining automated tests with structured examples makes it much easier to maintain the project over time.

6. Publishing to the Arduino Library Manager

After aligning the repository structure with Arduino conventions and improving the library itself, the final step was to make the project easier for others to install and use.

The Arduino ecosystem distributes libraries through the Arduino Library Manager, which indexes libraries from a central repository:

Arduino Library Registry

To make a library available there, it must meet several requirements, including:

a valid library.properties file
a repository layout compatible with Arduino tooling
semantic versioning
a tagged release

Once those requirements were met, the library was submitted to the registry through a pull request:

ESP32Wiimote Arduino Library Manager submission: https://github.com/arduino/library-registry/pull/7883

After the submission was reviewed and the automated checks passed, the library was accepted into the index.

This means the library can now be installed directly from the Arduino IDE using the Library Manager, without needing to manually clone the repository.

For developers, this provides several advantages:

simple installation directly from the IDE
automatic updates when new versions are released
easier discovery within the Arduino ecosystem

Making the library available through the Library Manager helps ensure that ESP32 developers who want to use Wii controllers can install and use the project with minimal setup.

7. Future Improvements

While the library is now easier to use and integrates well with the Arduino ecosystem, there are still several areas where it could evolve further.

One potential improvement is support for multiple Wiimotes connected to a single ESP32. The current implementation focuses on managing a single controller, which is sufficient for many projects. However, some use cases—such as robotics, gaming interfaces, or interactive installations—could benefit from handling multiple controllers simultaneously.

Supporting multiple Wiimotes would likely require improvements in areas such as:

connection management and pairing workflows
tracking controller identities and connection states
handling concurrent input streams
managing Bluetooth resource limits on the ESP32

Another area that could be explored is expanded support for Wiimote extensions. The Nunchuk is already supported, but the Wii ecosystem includes several other extension devices, such as the Classic Controller and MotionPlus. Adding support for these devices would expand the range of inputs available to ESP32-based projects.

There is also room for improving event handling abstractions. Currently, applications interact with decoded controller state and events directly. A higher-level event system could make it easier to write applications that react to button presses, motion events, or controller changes without having to process low-level state updates.

Additional improvements could include:

improving reconnection behavior after controller disconnects
adding optional callback-based input handling
expanding the test suite to cover more scenarios
providing additional example sketches for common use cases

As with many open-source projects, the direction of these improvements will largely depend on the needs of the community and the projects that adopt the library.

Conclusion

The original ESP32Wiimote already provided a solid implementation for connecting Wii controllers to ESP32 boards. This work focused on building on top of that foundation by applying Arduino ecosystem best practices and introducing several practical improvements.

The fork aligns the project with the expectations of the Arduino Library Manager, improves maintainability, and introduces new runtime capabilities that make the library easier to integrate into real applications.

Some of the key improvements include:

Arduino-compliant project structure and metadata
improved documentation and examples
code quality and formatting improvements
automated testing support
improved logging and debugging
runtime features such as connection state detection and battery monitoring
availability through the Arduino Library Manager

The result is a library that keeps the strengths of the original implementation while making it easier for developers to install, use, and extend.

The library is available here:

ESP32Wiimote: https://github.com/andremmfaria/ESP32Wiimote

If you are interested in using Wii controllers with ESP32 boards, this library provides a solid starting point—and hopefully a foundation for even more creative projects in the future.

Mastering Technical Interviews A Practical Guide to the Algorithms That Appear Again and Again

Andre Faria — Mon, 02 Mar 2026 18:03:30 +0000

Technical interviews are often framed as a test of memorization: recognize a pattern, recall a solution, write it under time pressure. This framing has fuelled an entire industry around grinding problem sets and rehearsing answers, as if strong engineers were pattern-recognition machines trained to replay known solutions on demand. Technical interviews are generally designed to evaluate problem-solving ability, reasoning, and coding skills rather than rote recall. Research has shown that many candidates prepare in ways that do not reflect real engineering work, often relying on memorization rather than authentic problem-solving practice.

That isn’t how real engineering works. In practice, developers are expected to analyze incomplete information, reason about trade-offs, gather additional data when needed, and choose an approach that fits the constraints at hand. The best solutions rarely come from recalling a memorized template verbatim; they emerge from understanding the problem deeply and applying the right tools deliberately.

The algorithmic patterns discussed in this article (two pointers, sliding windows, heaps, traversals, dynamic programming, and others) are not meant to be memorized as answers. They are mental models: reusable ways of structuring thought when facing certain classes of problems. When understood properly, they guide reasoning rather than replace it. Many interview-preparation guides emphasize that patterns are meant to teach structured problem decomposition, not memorized solutions.

This guide focuses on those patterns not as a checklist to grind through, but as a toolbox to support problem analysis. The goal is not to “pass interviews by rote”, but to approach technical problems (interview or real-world) with clarity, structure, and sound judgement. Pattern-based preparation is most effective when it builds reasoning skills rather than memorization, reinforcing a problem-solving mindset instead of recall.

1. Two Pointers

Two pointers are useful when an array or string must be processed from two directions or when you need to maintain a pair of indices representing a candidate solution. This approach reduces nested loops into linear scans. It is most effective when the input is sorted, or when the problem involves distances, sums, comparisons between ends, or in-place modifications without extra memory.

Use when:

The array is sorted or can be sorted.
The task involves pairwise relationships: sum to target, maximize or minimize distance, compare left vs. right properties.
The problem asks for in-place rearrangement or partitioning.
You want to eliminate a nested loop and reduce complexity from O(n²) to O(n).

Typical patterns:

Opposite-direction pointers moving toward each other (summing, container area, water trapping).
Same-direction pointers, where one pointer marks the “write” position (Move Zeroes, Dutch Flag sorting).

Example template (sum-based):

l, r = 0, len(nums) - 1
while l < r:
    s = nums[l] + nums[r]
    if s == target:
        return [l, r]
    elif s < target:
        l += 1
    else:
        r -= 1

Example template (in-place compaction):

def moveZeroes(nums):
    insert = 0
    for i in range(len(nums)):
        if nums[i] != 0:
            nums[insert], nums[i] = nums[i], nums[insert]
            insert += 1

2. Sliding Window

Sliding windows handle problems involving contiguous subarrays or substrings. The key idea is maintaining a window [l, r] with properties that can be updated as r expands and l contracts. This avoids recomputation and typically yields O(n) complexity. Sliding windows come in fixed-size and variable-size forms.

Use when:

The problem explicitly requires considering contiguous sequences.
The goal is to maximize/minimize length, find the longest substring with constraints, or compute sums efficiently.
There is a property that can be updated incrementally when the window expands or shrinks.
Hash maps or counters are used to track window validity.

Fixed-size window:

Used when the window size is given (e.g., “subarray of size k”).
Simply slide by removing leftmost element and adding rightmost.

Variable-size window:

Used when the window grows until invalid and then shrinks to restore validity.
Common in distinct-character constraints or frequency-based problems.

Example fixed-size:

def max_sum_subarray(nums, k):
    window_sum = sum(nums[:k])
    best = window_sum
    for i in range(k, len(nums)):
        window_sum += nums[i] - nums[i - k]
        best = max(best, window_sum)
    return best

Example variable-size:

def lengthOfLongestSubstring(s):
    seen = {}
    l = 0
    best = 0
    for r, ch in enumerate(s):
        if ch in seen and seen[ch] >= l:
            l = seen[ch] + 1
        seen[ch] = r
        best = max(best, r - l + 1)
    return best

3. Intervals

Interval problems revolve around operations on ranges [start, end]. Solutions almost always begin with sorting intervals, and reasoning about overlaps, merges, or gaps. Correct management of boundaries is essential. Many problems reduce to merging, insertion, or counting overlapping intervals.

Use when:

Input consists of ranges and you must merge, insert, or count overlaps.
You are asked whether intervals overlap or conflict.
You must determine available or free time.
Greedy techniques become effective after sorting by start or end times.

Core techniques:

Sort by start time when merging or inserting.
Sort by end time when minimizing conflicts.
Maintain a running "current end" to detect overlap or free space.

Example merge:

def merge(intervals):
    intervals.sort(key=lambda x: x[0])
    res = []
    for s, e in intervals:
        if not res or s > res[-1][1]:
            res.append([s, e])
        else:
            res[-1][1] = max(res[-1][1], e)
    return res

Example non-overlapping (minimum removals):

def eraseOverlapIntervals(intervals):
    intervals.sort(key=lambda x: x[1])
    count = 0
    last_end = float('-inf')
    for s, e in intervals:
        if s >= last_end:
            last_end = e
        else:
            count += 1
    return count

4. Stack

Stacks are suitable for problems involving nested structures, reversing order, parsing, or tracking monotonic sequences. A stack keeps context: what has been seen but not yet closed or resolved. Monotonic stacks allow efficient next-greater-element or histogram computations.

Use when:

Parentheses or encoded strings must be validated or decoded.
You need "previous greater/smaller" or "next greater/smaller".
Problems require evaluating expressions or parsing nested formats.
You want to track elements in sorted order while maintaining O(n) amortized time.

Patterns:

Classic push/pop for matching delimiters.
Monotonic stack: maintain increasing or decreasing order to compute ranges efficiently.

Example parentheses:

def isValid(s):
    stack = []
    pair = {')': '(', ']': '[', '}': '{'}
    for ch in s:
        if ch in '([{':
            stack.append(ch)
        else:
            if not stack or stack[-1] != pair[ch]:
                return False
            stack.pop()
    return not stack

Example monotonic (Daily Temperatures):

def dailyTemperatures(T):
    res = [0] * len(T)
    stack = []
    for i, temp in enumerate(T):
        while stack and T[stack[-1]] < temp:
            j = stack.pop()
            res[j] = i - j
        stack.append(i)
    return res

5. Linked List

Linked list techniques rely on pointer manipulation, often requiring careful handling of node references. Many solutions hinge on using fast/slow pointers to detect cycles, identify midpoints, or perform operations relative to the end of the list. Extra memory is usually unnecessary, and elegance depends on pointer management.

Use when:

You must detect cycles or intersections.
The task involves reversing part or all of a list.
Operations depend on the nth node from the end.
You must reorder nodes without converting to arrays.

Patterns:

Fast/slow pointer to find cycles or midpoints.
Dummy nodes to simplify edge-case manipulation.
Two-pointer offset technique for “remove nth from end”.

Example cycle detection:

def hasCycle(head):
    slow = fast = head
    while fast and fast.next:
        slow = slow.next
        fast = fast.next.next
        if slow == fast:
            return True
    return False

Example remove nth:

def removeNthFromEnd(head, n):
    dummy = ListNode(0, head)
    slow = fast = dummy
    for _ in range(n):
        fast = fast.next
    while fast.next:
        slow = slow.next
        fast = fast.next
    slow.next = slow.next.next
    return dummy.next

6. Binary Search

Binary search applies to sorted arrays or to problems where the answer lies in a monotonic search space. You can binary-search over indices, values, or even abstract answers (binary search on “feasibility”). A solution is valid if increasing or decreasing the parameter changes feasibility in a predictable (monotonic) way.

Use when:

The array is sorted, rotated, or partially sorted.
The problem asks for first/last occurrence, boundary, or pivot index.
You can express the question as: “Is x feasible?” and feasibility changes monotonically.
You must optimize or minimize some parameter, such as speed, capacity, or rate.

Patterns:

Standard binary search on sorted arrays.
Modified binary search for rotated sorted arrays.
Binary search on answer when the value domain is large but checking feasibility is O(n).

Example binary search:

def binary_search(nums, target):
    l, r = 0, len(nums) - 1
    while l <= r:
        mid = (l + r) // 2
        if nums[mid] == target:
            return mid
        elif nums[mid] < target:
            l = mid + 1
        else:
            r = mid - 1
    return -1

Example binary search on answer (Koko Eating Bananas):

import math

def minEatingSpeed(piles, h):
    l, r = 1, max(piles)
    while l < r:
        m = (l + r) // 2
        hours = sum(math.ceil(p / m) for p in piles)
        if hours <= h:
            r = m
        else:
            l = m + 1
    return l

7. Heap (Priority Queue)

Heaps are ideal when the problem requires repeatedly extracting the minimum or maximum element, or maintaining a dynamic set where only the top-k items matter. They guarantee O(log n) insertion and extraction and are essential when selecting the smallest/largest elements without fully sorting. Heaps shine in multi-way merging, streaming problems, and any scenario where you need efficient “best candidate” retrieval.

Use when:

The task asks for the k smallest/largest items.
You need to continuously push/pop values while keeping only the top k.
You must merge multiple sorted lists or streams.
A greedy algorithm relies on always selecting the current minimum or maximum.

Patterns:

Min-heap for selecting smallest; use negative values for max-heap behavior.
Size-k heaps to ensure O(n log k) solutions.
Tuples in heaps for ordering by multiple properties.

Example: Kth Largest Element

import heapq

def findKthLargest(nums, k):
    heap = nums[:k]
    heapq.heapify(heap)
    for x in nums[k:]:
        if x > heap[0]:
            heapq.heapreplace(heap, x)
    return heap[0]

Example: Merge K Sorted Lists

import heapq

def mergeKLists(lists):
    heap = []
    for i, node in enumerate(lists):
        if node:
            heapq.heappush(heap, (node.val, i, node))
    dummy = ListNode(0)
    cur = dummy
    while heap:
        val, i, node = heapq.heappop(heap)
        cur.next = node
        cur = node
        if node.next:
            heapq.heappush(heap, (node.next.val, i, node.next))
    return dummy.next

8. Depth-First Search (DFS)

DFS is used for exploring deep paths in trees or graphs, inspecting components, and performing recursive structural computations. It is especially useful when the problem requires visiting all nodes in a connected component, generating all possible paths, or computing metrics that depend on recursive aggregation. DFS works on both trees and general graphs, using visited sets to avoid cycles.

Use when:

The problem requires exploring all paths or all nodes in a region.
Tree problems that involve computing depth, height, tilt, diameter, or checking validity.
Graph problems involving connected components, cloning, or traversal.
Grid problems identifying islands, regions, or flood fill behavior.

Patterns:

Recursive DFS for tree or grid problems.
Stack-based DFS for graph problems.
Mark visited nodes to prevent infinite loops.

Example: Maximum Depth of Binary Tree

def maxDepth(root):
    if not root:
        return 0
    return 1 + max(maxDepth(root.left), maxDepth(root.right))

Example: Number of Islands (grid DFS)

def numIslands(grid):
    rows, cols = len(grid), len(grid[0])

    def dfs(r, c):
        if r < 0 or c < 0 or r >= rows or c >= cols or grid[r][c] != '1':
            return
        grid[r][c] = '0'
        dfs(r+1, c)
        dfs(r-1, c)
        dfs(r, c+1)
        dfs(r, c-1)

    count = 0
    for r in range(rows):
        for c in range(cols):
            if grid[r][c] == '1':
                count += 1
                dfs(r, c)
    return count

9. Breadth-First Search (BFS)

BFS excels at shortest-path problems on unweighted graphs, level-order processing in trees, and multi-source propagation (spreading effects over steps). BFS processes nodes layer by layer, guaranteeing the minimum number of steps to reach targets. It is the appropriate choice when the question involves minimum distances, time steps, or systematic level traversal.

Use when:

The problem asks for the shortest number of steps in an unweighted setting.
You must process a tree or graph level by level.
Multi-source diffusion problems: rotting oranges, spread of signals, BFS from multiple starting states.
Grid problems requiring finding the minimal distance to something.

Patterns:

Use a queue and process nodes per level.
Use visited sets for cycles in graphs.
Push all initial sources before starting (multi-source BFS).

Example: Level Order Traversal

from collections import deque

def levelOrder(root):
    if not root:
        return []
    q = deque([root])
    res = []
    while q:
        level = []
        for _ in range(len(q)):
            node = q.popleft()
            level.append(node.val)
            if node.left: q.append(node.left)
            if node.right: q.append(node.right)
        res.append(level)
    return res

Example: Rotting Oranges (multi-source BFS)

from collections import deque

def orangesRotting(grid):
    rows, cols = len(grid), len(grid[0])
    q = deque()
    fresh = 0

    for r in range(rows):
        for c in range(cols):
            if grid[r][c] == 2:
                q.append((r, c, 0))
            elif grid[r][c] == 1:
                fresh += 1

    minutes = 0
    while q:
        r, c, t = q.popleft()
        minutes = max(minutes, t)
        for dr, dc in ((1,0),(-1,0),(0,1),(0,-1)):
            nr, nc = r+dr, c+dc
            if 0 <= nr < rows and 0 <= nc < cols and grid[nr][nc] == 1:
                grid[nr][nc] = 2
                fresh -= 1
                q.append((nr, nc, t + 1))

    return minutes if fresh == 0 else -1

10. Backtracking

Backtracking is the algorithmic backbone for generating all valid configurations under constraints. It searches through the solution space using depth-first exploration while pruning invalid options as early as possible. This allows concise solutions for combinatorial problems, exhaustive enumeration, and constructing sequences step-by-step while maintaining validity.

Use when:

The problem requires generating all subsets, permutations, or combinations.
There is a need to explore choices step-by-step while respecting constraints.
Validity can be checked incrementally, allowing pruning of branches.
Search space is exponential and requires efficient pruning.

Patterns:

Recursive function with state path and decision index.
Undo action (path.pop()) after exploring each branch.
Prune early when the partial solution already violates constraints.

Example: Subsets

def subsets(nums):
    res = []
    def dfs(i, path):
        if i == len(nums):
            res.append(path[:])
            return
        dfs(i+1, path)
        path.append(nums[i])
        dfs(i+1, path)
        path.pop()
    dfs(0, [])
    return res

Example: Generate Parentheses

def generateParenthesis(n):
    res = []
    def backtrack(path, open_count, close_count):
        if len(path) == 2*n:
            res.append(path)
            return
        if open_count < n:
            backtrack(path + "(", open_count + 1, close_count)
        if close_count < open_count:
            backtrack(path + ")", open_count, close_count + 1)
    backtrack("", 0, 0)
    return res

11. Graphs (Topological Sort)

Topological sort is applied to directed acyclic graphs when you must determine an order of tasks respecting prerequisites. Cycle detection is inherent: if no valid ordering exists, the graph contains a cycle. It is frequently used for scheduling, dependency resolution, and course prerequisite problems.

Use when:

The problem mentions prerequisites, dependencies, ordering, or sequence validity.
You must determine if a cycle exists in a directed graph.
You must output a valid order of completion.
Nodes represent tasks; edges represent dependencies.

Patterns:

Compute in-degree of nodes.
Use a queue to process nodes with in-degree zero.
Remove edges gradually and collect nodes in order.

Example: Can Finish (detect feasibility)

from collections import defaultdict, deque

def canFinish(numCourses, prerequisites):
    graph = defaultdict(list)
    indegree = [0] * numCourses
    for a, b in prerequisites:
        graph[b].append(a)
        indegree[a] += 1

    q = deque(i for i in range(numCourses) if indegree[i] == 0)
    taken = 0
    while q:
        u = q.popleft()
        taken += 1
        for v in graph[u]:
            indegree[v] -= 1
            if indegree[v] == 0:
                q.append(v)
    return taken == numCourses

Example: Course Schedule II (return ordering)

def findOrder(numCourses, prerequisites):
    from collections import defaultdict, deque
    graph = defaultdict(list)
    indegree = [0] * numCourses
    for a, b in prerequisites:
        graph[b].append(a)
        indegree[a] += 1

    q = deque(i for i in range(numCourses) if indegree[i] == 0)
    order = []
    while q:
        u = q.popleft()
        order.append(u)
        for v in graph[u]:
            indegree[v] -= 1
            if indegree[v] == 0:
                q.append(v)
    return order if len(order) == numCourses else []

12. Dynamic Programming (DP)

Dynamic programming is appropriate when a problem can be decomposed into overlapping subproblems with optimal substructure. DP trades space for time, storing intermediate results to avoid recomputation. Problems involving counting ways, optimizing values, or building solutions from smaller components often map directly to DP formulations.

Use when:

Optimal solutions depend on solutions to smaller subproblems.
The problem has overlapping subproblems and cannot be solved greedily.
You recognize patterns like knapsack, subsequences, paths, decoding, or interval DP.
The recurrence relation naturally emerges from the problem statement.

Types:

1D DP for sequences (Decode Ways, Word Break).
2D DP for grids (Unique Paths, Maximal Square).
DP + binary search for LIS-style problems.
DP on intervals or structure-dependent DP when combining segments.

Example: Decode Ways

def numDecodings(s):
    if not s or s[0] == '0':
        return 0
    dp = [0] * (len(s)+1)
    dp[0] = dp[1] = 1
    for i in range(2, len(s)+1):
        if s[i-1] != '0':
            dp[i] += dp[i-1]
        if 10 <= int(s[i-2:i]) <= 26:
            dp[i] += dp[i-2]
    return dp[-1]

Example: Longest Increasing Subsequence (DP + binary search)

import bisect

def lengthOfLIS(nums):
    dp = []
    for x in nums:
        i = bisect.bisect_left(dp, x)
        if i == len(dp):
            dp.append(x)
        else:
            dp[i] = x
    return len(dp)

13. Greedy Algorithms

Greedy algorithms make locally optimal decisions at each step with the expectation that these choices lead to a global optimum. They rely on the problem having a structure where greedy-choice and optimal substructure properties naturally hold. Once you commit to a choice, you do not revisit it, making solutions efficient and typically O(n) or O(n log n).

Use when:

The problem can be solved by repeatedly taking the best immediate option.
Sorting helps reveal an order that makes greedy decisions valid.
You are maximizing or minimizing a metric such as profit, number of intervals, or fuel balance.
Backtracking or DP is unnecessary because future steps do not depend on alternative past choices.

Patterns:

Track running min/max (Best Time to Buy/Sell Stock).
Maintain cumulative resource balance (Gas Station).
Advance by the farthest reachable index each step (Jump Game).

Example: Best Time to Buy and Sell Stock

def maxProfit(prices):
    min_price = float('inf')
    best = 0
    for p in prices:
        min_price = min(min_price, p)
        best = max(best, p - min_price)
    return best

Example: Jump Game

def canJump(nums):
    reachable = 0
    for i, jump in enumerate(nums):
        if i > reachable:
            return False
        reachable = max(reachable, i + jump)
    return True

14. Trie

Tries efficiently store and query large sets of strings, especially when prefix operations are frequent. They organize characters in a tree-like structure where each path from root to node represents a prefix. Tries allow O(m) lookup where m is the word length, independent of how many words exist. They are fundamental for autocomplete, prefix filtering, and dictionary checks.

Use when:

The task involves prefix search or prefix matching.
You must repeatedly query or insert strings with overlapping prefixes.
Problems ask whether any word starts with a given prefix.
Searching character-by-character offers more efficiency than scanning all strings.

Patterns:

Each node contains a map of children.
Mark end = True for completed words.
Walk the trie for searching or prefix validation.

Example: Trie Implementation

class TrieNode:
    def __init__(self):
        self.children = {}
        self.end = False

class Trie:
    def __init__(self):
        self.root = TrieNode()

    def insert(self, word):
        node = self.root
        for ch in word:
            node = node.children.setdefault(ch, TrieNode())
        node.end = True

    def search(self, word):
        node = self.root
        for ch in word:
            if ch not in node.children:
                return False
            node = node.children[ch]
        return node.end

    def startsWith(self, prefix):
        node = self.root
        for ch in prefix:
            if ch not in node.children:
                return False
            node = node.children[ch]
        return True

Example use case indicator:

Input: many words, many queries → trie fits.
Task: “return the number of words with a given prefix” or “determine if any word begins with prefix”.

15. Prefix Sum

Prefix sums transform cumulative operations into O(1) queries by precomputing running totals. They allow rapid calculation of subarray sums, difference queries, and frequency-based insights. Instead of recomputing from scratch, you subtract two prefix values to get the sum of any range.

Use when:

The problem involves frequent sum-of-subarray queries.
You must detect subarrays with a target sum or pattern.
Overlapping subarrays need efficient comparison.
A running balance or cumulative measure is helpful.

Patterns:

prefix[i] = nums[0] + ... + nums[i-1]
Subarray sum from i to j: prefix[j+1] – prefix[i]
Hash map of prefix sums to detect subarrays with specific targets.

Example: Subarray Sum Equals K

from collections import defaultdict

def subarraySum(nums, k):
    prefix = 0
    count = 0
    freq = defaultdict(int)
    freq[0] = 1
    for x in nums:
        prefix += x
        count += freq[prefix - k]
        freq[prefix] += 1
    return count

Example use cases:

“Count subarrays with sum k.”
“Find how many substrings satisfy some cumulative constraint.”

16. Matrices

Matrix problems require structured 2D traversal, manipulation, or transformation. Many tasks involve row/column operations, rotation, flooding, or spiral traversal. Solutions often rely on systematic scans or in-place transformations to maintain O(1) space. Index manipulation is the core challenge: understanding how rows and columns shift relative to one another.

Use when:

The question involves grid-based movement or transformations.
Problems require rotating, flipping, zeroing rows and columns.
Spiral-order traversal or layer-by-layer operations apply.
2D constraints create natural boundaries for iteration.

Patterns:

Use boundary pointers for spirals.
Matrix transpositions and reversals for rotations.
Row/column flags for operations like Set Matrix Zeroes.

Example: Spiral Matrix

def spiralOrder(matrix):
    res = []
    top, bottom = 0, len(matrix)-1
    left, right = 0, len(matrix[0])-1

    while top <= bottom and left <= right:
        for c in range(left, right+1):
            res.append(matrix[top][c])
        top += 1

        for r in range(top, bottom+1):
            res.append(matrix[r][right])
        right -= 1

        if top <= bottom:
            for c in range(right, left-1, -1):
                res.append(matrix[bottom][c])
            bottom -= 1

        if left <= right:
            for r in range(bottom, top-1, -1):
                res.append(matrix[r][left])
            left += 1

    return res

Example: Rotate Image (90° clockwise)

def rotate(matrix):
    n = len(matrix)
    for i in range(n):
        for j in range(i+1, n):
            matrix[i][j], matrix[j][i] = matrix[j][i], matrix[i][j]  # transpose
    for row in matrix:
        row.reverse()

Example: Set Matrix Zeroes

First pass: mark zero rows and columns.
Second pass: zero out cells in marked rows/columns.

Conclusion

Technical interviews should not reward the ability to memorize solutions or replay patterns on cue. Engineering is not an SAT exam, and developers are not pattern-recognition machines. In real systems, problems are ambiguous, data is incomplete, and the correct approach often emerges only after careful analysis or after asking better questions and gathering more information.

The algorithmic techniques covered in this article are best understood as tools, not answers. They are ways of shaping thought, reason about constraints, structure data, and reduce complexity. Used correctly, they help engineers arrive at solutions; used mechanically, they become blunt instruments.

For candidates, this means focusing less on grinding problems and more on understanding why a technique applies, when it does not, and how to adapt it when conditions change. Short, deliberate practice sessions that reinforce reasoning and trade-off analysis are far more valuable than endless repetition.

For interviewers, it means designing interviews that reflect real engineering work: encouraging exploration, validating assumptions, and thoughtful decision-making, rather than forcing candidates to perform another memorization exercise under time pressure. There is growing discussion in the engineering community about moving beyond purely LeetCode-style interviews toward formats that better reflect real-world problem solving.

Master the concepts, not the scripts. Treat patterns as a toolbox, not a collection of hammers. The goal isn’t luck or recall. It’s clarity, judgement, and the ability to reason your way to a solution.

When Chat Turns into Control - Security Lessons from Running a Local AI Agent using OpenClaw

Andre Faria — Sun, 22 Feb 2026 01:49:59 +0000

Running large language models locally is easier than ever. With tools like Ollama and frameworks such as OpenClaw, it’s now trivial to deploy AI agents that reason, keep state, and execute actions on private hardware.

That convenience comes with a catch.

Once an LLM is wired to tools and exposed through a platform like Discord, it stops being “just a chatbot.” It becomes a control surface driven by natural language, where user input can directly influence system behaviour. In that context, traditional security assumptions like clear trust boundaries, strict input validation, predictable execution no longer hold ground.

This article is not an installation guide. It’s a security-focused reflection on running a local AI agent: where the real risks appear, why “self-hosted” does not automatically mean “safe,” and which design choices actually reduce the blast radius when things go wrong.

1. Context and setup

Running LLMs locally has become easy enough that many people now treat them like “just another service.” Tools like OpenClaw push this further by turning an LLM into an agent: something that can reason, keep state, and execute actions.

In this setup, the agent is controlled through Discord, backed by a local Ollama instance. The deployment looks like this:

Ollama runs on a dedicated TrueNAS host with an RTX 3070, handling all LLM inference.
Model: Qwen3 8B, chosen for being fast and efficient on consumer GPUs.
OpenClaw runs on a separate Linux VM, acting as the agent control plane.
The two hosts communicate over the local network.
Discord is the primary user interface.

Everything is self-hosted and not directly exposed to the internet. At first glance, this feels “safe enough.” But once you let an agent do things, not just chat, you’re no longer dealing with a toy system. You’re running automation driven by natural language, which changes the security model completely.

2. Architecture and trust boundaries

At a high level, the system has three layers:

Discord – where humans talk to the agent
OpenClaw – where decisions, memory, and tool execution happen
Ollama + LLM – where language is generated

Each layer crosses a trust boundary.

Discord is an untrusted input surface, even if the users themselves are trusted. Messages can include pasted text, links, logs, or content copied from elsewhere. Research on prompt injection shows that attackers don’t need direct access to the model—indirect injection through user-supplied content is often enough to override intended behaviour (MDPI, 2024).

OpenClaw sits in the middle as a control plane. It turns text into actions. The problem is that LLMs don’t distinguish between “instructions” and “data.” Everything is just language. This is a known and well-documented weakness of LLM systems, and it’s why prompt injection keeps showing up as the dominant failure mode in agent-based designs (arXiv:2601.09625).

Finally, when the agent can execute tools (filesystem access, memory writes, or web fetches) the risk escalates. Academic and industry analyses consistently show that once an injected prompt can chain actions, the impact is no longer limited to bad answers; it can affect the system itself (arXiv:2410.23308).

One important takeaway: running Ollama and OpenClaw on separate hosts improves performance and resilience, but it does not automatically solve these security problems. The weakest link is still the language interface.

3. The security problem with small models

Qwen3 8B is a great fit for a home lab: it’s fast, it runs well on a consumer GPU (RTX 3070), and it’s cheap to keep online. The downside is that small-ish models are easier to steer off course.

That matters because agents don’t just “answer questions.” They can call tools, update memory, and sometimes fetch or interpret external content. Prompt injection is now widely treated as a top-tier LLM risk for exactly this reason: language is both data and instructions, and the model can be tricked into treating untrusted text as “policy.” OWASP calls this out directly as a primary risk category for LLM apps. (OWASP)

Where it gets nasty is indirect prompt injection: the attacker doesn’t need to DM your bot with an obviously malicious prompt. They just need your agent to consume content that contains hidden instructions (HTML, docs, logs, etc.). This has been demonstrated repeatedly for web agents, where malicious strings embedded in a page can hijack agent behaviour. (arXiv:2507.14799)

So the core issue isn’t “Qwen is bad.” It’s:

Small model + tool access = higher chance of bad tool calls
Small model + web/content ingestion = bigger prompt injection surface
Once it’s an agent, you have to assume the model will occasionally do the wrong thing

That’s why the security posture for small models tends to be: contain the blast radius (sandbox) and remove the easiest injection paths (web fetch / browser). (OWASP Cheat Sheet Series)

4. Discord as an attack surface

Discord feels like a friendly UI, but from a security perspective it’s an untrusted command channel. Anything users paste (logs, URLs, config snippets) can become “model input,” and that’s enough for prompt injection to show up.

The two main problems are:

Scope creep: “it’s only our server” slowly becomes “it’s in more channels than intended”
Permission drift: roles change, new channels get created, people invite the bot elsewhere

So the safe baseline is: deny by default, then allow only what you actually need.

In practice, that means:

Lock the bot to specific guild(s) (server allowlisting)
Restrict usage to a specific role (role gating)
Decide whether normal messages must be mention-gated (reduce accidental triggers)
Handle slash commands explicitly (they have their own permissions model in Discord)

Discord itself supports controlling who can use slash commands through its permissions system (and it’s worth doing that at the Discord layer, not just in the bot). (Discord)

This is the key mental shift: even if the model runs locally and the gateway isn’t public, Discord is still a big input funnel. Treat it like an API surface: least privilege, explicit allowlists, and “assume someone will paste something dumb eventually.” OWASP’s guidance maps well here: prompt injection is not rare, and the best defenses are limiting what the model can do when it gets it wrong. (OWASP)

5. Sandboxing and tool restriction

Once the agent was wired to Discord and running a small model, the real risk wasn’t wrong/bad answers. It was uncontrolled side effects. This is where sandboxing becomes essential.

In OpenClaw, sandboxing means session-level isolation for tool execution. Each conversation runs inside a constrained environment, with no access to the host filesystem or other sessions. If the model does something wrong, the impact is contained.

Enabling sandboxing globally is a single configuration change:

openclaw config set agents.defaults.sandbox.mode all
openclaw config set agents.defaults.sandbox.scope session
openclaw config set agents.defaults.sandbox.workspaceAccess none

This follows OpenClaw’s sandboxing model, which prioritizes containment over perfect prevention (docs.openclaw.ai/sandbox).

The second part of the fix was disabling web-based tools. Web access is the most common prompt-injection vector in agent systems: arbitrary, attacker-controlled text gets fed directly into the model. This has been repeatedly demonstrated in both academic work and industry analyses of indirect prompt injection (arXiv:2507.14799).

In practice, this meant explicitly turning off web fetch and denying the entire web tool group:

openclaw config set tools.web.fetch.enabled false
openclaw config set tools.deny '["group:web","browser"]'

The last item to complete the fix was to add a rate limiting on the auth attempts on the gateway

openclaw config set gateway.auth.rateLimit '{ "maxAttempts": 10, "windowMs": 60000, "lockoutMs": 300000 }'

This means that there are a max of 10 failed attempts per minute and it locks out for 5 minutes after that.

After these changes:

Tool execution became more predictable
Web-based injection paths were removed
OpenClaw’s built-in security audit reported zero critical or warning findings

This matches OWASP’s guidance for LLM applications: assume prompt injection will eventually happen, and focus on reducing blast radius instead of relying on model behaviour alone (OWASP LLM Top 10).

6. Takeaways

A few clear lessons came out of this setup:

Local LLMs are not automatically safe just because they are self-hosted
Discord is an attack surface, not just a chat UI
Small models like Qwen3 8B are efficient, but need more guardrails
Sandboxing matters more than model choice
Removing web access dramatically reduces risk
Separating Ollama and OpenClaw hosts improves resilience, not security

Most of these conclusions line up with existing research and security guidance. Prompt injection, permission drift, and over-trusted tools are expected failure modes, not edge cases (OWASP Prompt Injection Cheat Sheet), (arXiv:2410.23308).

The takeaway is simple: once an LLM can act, it must be treated like infrastructure. With sandboxing, explicit allowlists, and tool restrictions, a local agent can be both powerful and reasonably safe — but only if security is part of the design from the start.

I wanted to know how malware works, so I built an analyser

Andre Faria — Wed, 10 Dec 2025 11:41:32 +0000

1. Introduction & Motivation

When I began thinking about what to do for my Master’s thesis, one question kept resurfacing: How do people actually classify malware? I had always been curious about the internal logic behind malware categorization, not just at a high level, but at the level of processes, features, and decision-making.

In the end, the thesis became more of a means to an end: a structured excuse to finally build something I’d wanted for years, my own static malware analyser.

To do that, I needed a system that was:

Reproducible, so others could follow the same steps
Interpretable, so each decision had a clear explanation
Automated, so large numbers of samples could be processed
Modular, so rules, enrichment, or extraction could evolve over time

This article describes how I designed the baseline analysis pipeline, what I learned from it, and why building it was the most effective way to understand how malware works (see survey: ResearchGate).

Why Static Analysis not Dynamic analysis or both?

I chose static analysis because it’s the simplest, safest way to make progress fast. You can point mature tools like Ghidra at a binary and immediately get structure, imports and strings—no sandbox to provision, no risk of executing the sample, and results that are easy to trace back to rules. That makes static ideal for batch triage and for learning: it’s repeatable, quick, and interpretable.

Of course, static has blind spots. Dynamic analysis shows what the program actually does at runtime—process creation, network I/O, registry and file changes—and it can expose unpacking or decryption that static won’t see. The trade‑off is overhead and fragility: running malware safely requires instrumentation and isolation, it’s slower per sample, and many families try to evade sandboxes. My approach was to start with static to build a clear baseline, then layer enrichment (and later, hybrid methods) where deeper behaviour visibility is needed.

2. High-Level Architecture of the Baseline Pipeline

The baseline pipeline is intentionally simple. It follows a straight, modular workflow:

Feature Extraction – gather structural and semantic information from the PE file.
Heuristic Evaluation – apply rule-based checks to detect suspicious patterns.
Optional Data Enrichment – pull external intelligence (e.g., VirusTotal) for reference.
Decision Fusion – combine heuristic signals with enrichment (if available).
Reporting – output structured evidence, classification, and metadata.

Each component has a narrow purpose and produces structured data that the next stage consumes. This keeps the design predictable and transparent.

On the optional enrichment step

The enrichment layer is intentionally optional. In theory, it makes the classification stronger because the heuristic output can be cross-checked against external intelligence.

But enrichment also introduces an unexpected trade-off:

If the heuristic analysis is roughly aligned with the enrichment data, the result improves.
If the heuristic analysis is far off from the enrichment (e.g., near-random heuristics), the fusion process can skew the final label in unhelpful ways.

So enrichment is useful, but only when the baseline heuristics are not too noisy. This became a recurring theme in the project.

3. Extracting Features from Malware Samples

Static analysis begins with extraction gathering every meaningful property of a file without running it (overview: IJRASET). This includes:

PE metadata
Section layout
Import tables
Strings
Function signatures and decompiler output
Embedded resources
Other structural features

In the baseline, the decompiler stage writes a per-sample features JSON you can reuse downstream. Typical fields include program (name, format, language, compiler, image_base, size, sha256), functions, imports, sections, strings, and optional decompiled function records. For runs, artifacts are written under a run folder (e.g., decompile-<RUN_ID>/<sha256>.features.json).

Why only PE binaries (and how to adapt)

For the experiments in this article, I focused on PE (Portable Executable) binaries (.exe, .dll, .sys). Two practical reasons guided this decision:

PE is the most widespread format in desktop malware telemetry (Windows dominance in consumer endpoints).
Tooling and ecosystem maturity are strongest around PE (Ghidra processors, import table conventions, common packers/obfuscators), which reduces ambiguity when building a baseline.

That focus simplified feature extraction (e.g., sections, imports, entry points) and made heuristic authoring more reliable (static vs dynamic context: SJSU ScholarWorks).

Adapting to other formats is feasible with incremental changes:

ELF (Linux): switch language/processor in Ghidra, adjust extractors for ELF sections/segments, symbol tables, and libc/syscall imports; re-map heuristics to Linux TTPs (e.g., ptrace, /proc tampering, init/systemd persistence).
Mach-O (macOS): use Mach-O program metadata, dyld imports, code signatures/entitlements; adapt persistence/networking rules to macOS paths and launch agents.
Android APK/Dex: pivot to bytecode/decompiled Java/Kotlin; extract manifest, permissions, receivers/services; heuristics on exfil domains, trackers, sensitive API calls.
Scripted binaries (e.g., .NET, Python, JS packagers): add language-aware parsers, focus on runtime loaders, reflection/dynamic resolution, embedded payloads.

Concretely, the baseline needs:

A format detector in the orchestrator to choose the right decompiler/extractor path.
Format-specific feature schemas with a shared core (program info, strings, imports/exports, sections), plus optional blocks per format.
A heuristic ruleset per format (or parametric rules) and a tagging map aligned to each platform’s taxonomy.

With these adaptations, the same pipeline (decompile → heuristics → optional enrichment → fusion → report) extends beyond PE with minimal structural changes.

Why Ghidra (and not radare2, IDA, or Ada-based tools)?

A few people ask why I didn’t use Ada or other specialized tools. The answer is simple:

Ghidra is fully open source
It provides Python bindings through PyGhidra
It integrates a powerful decompiler
It can be automated in a pipeline without licensing issues

That said, Ghidra’s Python bindings are not trivial to use. Because Python and Java operate differently (different memory models, threading assumptions, and API expectations), interacting with Ghidra programmatically can become clunky. But it remained the most practical option.

Limits of the extraction approach

Because this is a lightweight baseline pipeline, the extraction steps are intentionally simple. This leads to a major limitation:

The analysis depends heavily on readable strings and predictable patterns. If the malware is encrypted, packed, or obfuscated, the extracted data becomes almost useless.

This constraint shapes everything downstream in the pipeline.

4. The Heuristics Engine: How the Rules Work

The heuristics engine is the simplest component of the pipeline by design. A heuristic rule is just:

A pure function
That examines extracted features
And returns structured evidence if a condition is met

The logic behind the rules is intentionally basic. Most rules rely on simple string-matching or pattern detection, such as:

Suspicious API calls
Writable/executable sections
Unusual import patterns
Indicators in metadata or strings

A double limitation

Because rules depend on literal string matching:

The input must closely match what the rule expects, or the rule will not fire.
Cryptographed, packed, or obfuscated malware evades the heuristics almost completely.

The upside is interpretability: every rule hit produces clear evidence.
The downside is coverage: many modern malware families will not match at all.

Rule shape and evidence contract

Rules are pure functions that take extracted features and return either Evidence or a miss reason. In REXIS they follow a signature like:

def rule_example(features: dict, rule_score: float = 0.2, params: dict = {}):
 # return (Evidence, "reason") on hit, or (None, "miss reason") on miss
 ...

Evidence is structured with id, title, detail, severity (info|warn|error) and a raw score in [0,1]. The analyser attaches a reason and per-evidence categories (derived from a tagging map) to aid traceability.

Tuning is externalized: a rules config (YAML/JSON) can reweight rules, pass per‑rule params via rule_args, filter by allow_rules/deny_rules, and define label_overrides for strong signals. Tag inference (e.g., ransomware, stealer, backdoor) is computed from evidence via a configurable tagging section.

Tip: Always return a miss reason; it surfaces in rule_misses and makes rule calibration easier.

Authoring and wiring a rule (concrete example)

Here is a simplified example that flags mutex creation APIs, showing the recommended return contract and tunable rule_score:

from typing import Any, Dict, Optional, Tuple
from rexis.utils.types import Evidence
from rexis.tools.heuristics_analyser.utils import get_imports_set

def rule_suspicious_mutex_creation(
 features: Dict[str, Any], rule_score: float = 0.10, params: Dict[str, Any] = {}
) -> Tuple[Optional[Evidence], Optional[str]]:
 imps = get_imports_set(features)
 mutex_apis = {"createmutexa", "createmutexw", "openmutexa", "openmutexw"}
 hits = imps & mutex_apis
 if not hits:
  return None, "no mutex-related imports found"
 return (
  Evidence(
   id="suspicious_mutex_creation",
   title="Mutex creation/manipulation",
   detail=f"Imports include: {', '.join(sorted(hits))}",
   severity="info",
   score=float(rule_score),
  ),
  f"matched mutex imports: {', '.join(sorted(hits))}",
 )

To wire it, register the function with a stable id in the analyser’s ruleset and add a default weight in the config. At runtime, you can raise/lower its impact via weights.suspicious_mutex_creation and pass parameters through rule_args.

Tuning via config (weights, thresholds, tags)

In your rules YAML/JSON you can control:

scoring.combine (weighted_sum|max) and label_thresholds.{malicious,suspicious}
weights: per‑rule caps; contribution is min(1.0, ev.score * weight)
allow_rules / deny_rules: enable/disable subsets
label_overrides: force a label if a rule fires
rule_args: (rule_score, params) per rule
tagging: map evidence to tags (e.g., ransomware, stealer, backdoor) with tag_weights, threshold, top_k

This keeps rule code simple while giving you environment‑specific control.

Testing a rule quickly (ad‑hoc)

from rexis.tools.heuristics_analyser.main import heuristic_classify

features = {
 "program": {"name": "sample.exe", "size": 200_000, "sha256": "...", "format": "pe", "language": "x86"},
 "imports": ["CreateMutexA", "GetProcAddress"],
 "sections": [{"name": ".text", "size": 3500, "flags": ["exec", "write"]}],
 "strings": ["http://example.com", "VirtualBox"],
}

result = heuristic_classify(features)
print(result["score"], result["label"])      # inspect overall score/label
print(result.get("evidence", []))             # list of evidence with reasons
print(result.get("tags", []))                 # tag candidates with scores
print(result.get("rule_misses", []))          # why a rule didn’t fire

5. Enrichment Through External Intelligence (Optional Step)

Enrichment was added only after early experiments revealed a problem:

The heuristics alone generated output that was “too weak” to stand on its own.

Not because the system was flawed, but because simple static heuristics have very limited visibility into modern malware. To counter that, enrichment allows the analyser to pull external data, such as (background: VirusTotal docs):

Hash reputation
Threat vendor classifications
Historical submissions
Known malicious families associated with a SHA-256
Community tags or detection ratios

This creates a baseline to compare the heuristic output against. But enrichment was never meant to override the heuristics, only to contextualize them (what enrichment adds and caveats: Wiz Academy).

Why enrichment is useful but imperfect

It improves confidence when heuristics are directionally correct.
It destabilizes classification when heuristics are very noisy.
It introduces dependency on an external service (API, rate limiting, coverage gaps).

Despite its imperfections, enrichment helped ground the pipeline’s outputs and made the entire system more meaningful.

6. Decision Fusion: Combining All Signals

Once both the heuristic engine and the optional enrichment layer produce their outputs, the pipeline needs a final step that decides:
What is the most reasonable label for this sample?

The decision fusion module combines the available signals:

Heuristic evidence (rule hits, counts, weights)
Optional enrichment (external reputation, vendor labels, known families)

The fusion logic uses a simple, weighted approach:

If heuristics show strong, consistent evidence → they carry more weight.
If heuristics are weak but enrichment is strong → enrichment influences the decision more.
If both are weak → the sample defaults to suspicious or unknown.
If they strongly disagree → the system emits a warning, and the final label is conservative.

This prevents the analyser from being “overconfident,” which is a real risk when combining noisy static heuristics with external reputation data.

Confidence-weighted fusion (with disagreement penalty)

The reconciler computes a final score using per-source confidences and weights:

S_final = clip_01( w_h C_h S_h + w_vt C_vt S_vt − penalty(|S_h − S_vt|) )

S_h, S_vt: heuristics and VT scores in [0,1]
C_h, C_vt: confidences in 0,1
w_h, w_vt: relative weights
penalty(...): applied when both signals exist and disagree beyond a policy threshold

When both sources are high‑confidence yet strongly disagree, a conservative hard‑override can force a mid score and an abstain/suspicious label. Final labels are then chosen via calibrated thresholds (e.g., T_mal=0.70, T_susp=0.40).

The core idea

The fusion layer isn’t meant to be clever, just balanced.
It ensures that neither heuristics nor enrichment dominate blindly, and that the final classification reflects the overall confidence of the system rather than any individual signal.

7. Output, Reporting, and Traceability

Every run of the baseline pipeline produces structured output that makes the analysis reproducible and auditable. For each sample, the system stores:

Extracted features
All heuristic rule evidence
Optional enrichment results
The fused classification label
Metadata (hashes, timestamps, config parameters)
A JSON report representing the entire reasoning chain

This traceability was crucial for the thesis.
It allowed me to re-run experiments, refine rules, compare outputs, and understand how every decision was made. When you are building an analyser from scratch, having visibility into why something happened is as important as the result itself.

Why reporting matters

It makes the pipeline reproducible.
It allows for manual inspection when results are unclear.
It provides ground truth for later LLM/RAG experiments.
It helps identify weak rules, noisy features, or misaligned fusion logic.

The reporting layer ended up being one of the most valuable parts of the pipeline, even though it was initially treated as a simple output function.

Concrete artifact paths

For a run directory like baseline-analysis-<RUN_ID>/, you’ll typically see:

Features: decompile-<RUN_ID>/<sha256>.features.json
Heuristics: <sha256>.baseline.json
Final report (fusion): <sha256>.report.json
Batch runs: baseline_summary.json plus a per‑run baseline-analysis-<RUN_ID>.report.json

8. Lessons Learned from Building a Static Malware Analyser

Building a malware analyser, even a simple baseline one, teaches you a lot about both malware and tooling. A few reflections stood out.

What worked well

The architecture was clear, modular, and easy to extend.
The rule engine was transparent and interpretable.
The pipeline could analyse large sets of files quickly.
It established a solid foundation for later ML and LLM-based experiments.

What didn’t work as well

Static analysis alone struggles with packed or cryptographed malware (see recent studies: ScienceDirect, MDPI).
The heuristic engine is only as good as the extracted strings and it often isn’t enough.
Simple string matching has obvious limits in modern malware ecosystems.
Enrichment, while useful, can distort results when heuristics are too weak.

What surprised me

How quickly the heuristics break when input patterns change.
How hard it is to design “general” rules that work across many families.
How often malware authors rely on simple tricks that defeat static inspection.

How this shaped the next phase of the thesis

These lessons directly informed the development of the LLM + RAG-enhanced pipeline (which will be covered on a dedicated article).
Static heuristics gave me structure, data, and understanding. But not enough depth.
The next logical step was to use LLMs to interpret extracted features more flexibly, grounded by retrieval to avoid hallucinations.

The baseline pipeline provided the scaffolding needed to move forward.

Analysis Results & Repository Structure

The complete artefacts from my experiments live in the repository under analysis/. It has two main branches of outputs and a simple aggregate:

Note: The LLM + RAG pipeline is only referenced here for structure and comparison; I’ll cover its design, prompts, retrieval strategy, and results in a dedicated follow‑up article.

analysis/baseline/: results from the baseline static pipeline (with and without VirusTotal enrichment) (link)
analysis/llmrag/: results from the LLM + RAG pipeline (link)
analysis/aggregation-output.json and analysis/aggregation-report.csv: quick roll‑ups of the per‑run outputs (link)

Directory layout (overview)

analysis/baseline/baseline-analysis-<family>-run-2508/: baseline runs per family (e.g., botnet, ransomware, rootkit, trojan) (examples)
analysis/baseline/baseline-analysis-<family>-run-vt-2508/: same families with VirusTotal enrichment enabled (examples)
analysis/llmrag/llmrag-analysis-<family>-run-2508/: LLM + RAG runs per family (examples)

Inside each run directory you’ll find the per‑sample artefacts described earlier:

decompile-<RUN_ID>/<sha256>.features.json: extracted features
<sha256>.baseline.json: heuristics output
<sha256>.report.json: fused final report (label, score, trace)S
baseline-analysis-<RUN_ID>.report.json: batch‑level summary for the run
baseline_summary.json: compact summary across all processed samples

Baseline analysis: what the runs show

Across the baseline folders, you can inspect how the simple heuristics behave for different malware families and how optional VirusTotal enrichment shifts confidence and labels:

Without enrichment (baseline-analysis-<family>-run-2508/), evidence is driven purely by structural/string‑based signals; many samples land in suspicious or unknown when strings are sparse or obfuscated.
With enrichment (baseline-analysis-<family>-run-vt-2508/), labels tend to stabilize when external reputation aligns with the heuristics; disagreement cases are explicitly noted in the fused reports via the reconciliation policy.

For a quick, high‑level view across runs, open analysis/aggregation-report.csv or the machine‑readable analysis/aggregation-output.json. These aggregate files summarize per‑run counts and label distributions without having to traverse each directory.

If you want to reproduce similar outputs, run the commands in Section 9 and point -o to a top‑level analysis/ directory; the pipeline will create run‑specific folders and the same artefact structure.

9. How to Run It Yourself

The analyser is open-source and can be run with only a few prerequisites:

Requirements

Python environment (follow the installation setup on the repository's README.md for setting up the environment)
Ghidra + PyGhidra (Ghidra installed at /opt/ghidra on Linux). If you need a fast, distro‑agnostic setup, follow my guide: Ghidra on Linux: Zero Fuss Install
A directory of PE files
(Optional) VirusTotal API key for enrichment (set [baseline].virus_total_api_key in config/settings.toml)

Basic usage

Once installed, running the baseline pipeline is straightforward (Typer CLI):

pdm run rexis analyse baseline -i ./data/samples/<file>.exe -o ./data/analysis

or for batch mode:

pdm run rexis analyse baseline -i ./data/samples -o ./data/analysis --parallel 4

Common options:

-i, --input: file or directory to analyse (required)
-o, --out-dir: output directory (defaults to CWD)
-r, --run-name: logical run name (default: UUID)
-y, --overwrite: overwrite existing artifacts
-p, --parallel: workers for directory mode
--rules: path to heuristics rules config (YAML/JSON)
-m, --min-severity: filter returned evidence (info|warn|error)
--vt: enable VirusTotal enrichment (requires API key in config/settings.toml)
--vt-timeout, --vt-qpm: timeout and queries-per-minute budget

Rule customization

Users can:

Add new heuristic rules
Tune weights and thresholds
Enable or disable individual rules
Adjust fusion parameters
Add their own enrichment sources

Where to start

All documentation is available in the repository:

Baseline pipeline guide: https://github.com/andremmfaria/rexis/blob/main/guides/BaselinePipeline.md
Heuristic rule‑writing guide: https://github.com/andremmfaria/rexis/blob/main/guides/WritingHeuristicRules.md
Reconciliation (fusion) details: https://github.com/andremmfaria/rexis/blob/main/guides/Reconciliation.md
Example configurations and sample reports in the repo

This makes it easy to experiment, modify, or build your own extensions.

10. Conclusion: Why Building Tools Is the Best Way to Learn

I started this project because I wanted to understand how malware classification works.
Building my own analyser forced me to confront all the assumptions, shortcuts, limitations, and edge cases that textbooks and blog posts never mention.

What I gained was not just a working pipeline, but a practical understanding of:

how static analysis actually behaves
where heuristics break
why enrichment matters
how evidence should be combined
and how analysts think about classification

The baseline pipeline is not perfect. It was never meant to be.
But it gave me the foundation I needed to build more advanced approaches, including the LLM + RAG pipeline that became the core of the second half of my thesis. This will be covered in a future article.

Most importantly, it taught me this:

If you want to learn how something works, build a tool that does it.
You’ll understand the entire problem far more deeply.

Building a Transparent LAGG (LACP) Bridge with OPNsense, UDM, and UniFi — A Practical Guide

Andre Faria — Mon, 01 Dec 2025 02:52:40 +0000

1. Introduction

I have a pretty heavy network topology at my home. The result of years of IoT devices, user devices, servers and miscellaneous IP enabled things. Those pose some security threats that I longed to correct or, at the very least, monitor. So I had this idea to place a transparent firewall between my UniFi Dream Machine (UDM) and the rest of my network.

I wanted to do this without replacing/reconfiguring the UDM, without breaking VLANs, without touching DNS/DHCP or anything else other than what was really necessary. The goal was simple:

perform minimal changes
stay fully transparent to the network
add filtering/inspection intelligence
do it using LAGG (LACP) to increase throughput (and because it is cool).

This article documents how I built a Layer 2–only transparent bridge using OPNsense with two aggregated links toward the UDM and two aggregated links toward my UniFi switch. It also covers the mistakes I made so you don’t repeat them.

To set the stage, here is the physical setup as it exists today.

On the top shelf we have, from left to right, a pod charger for xbox controller batteries, the USW-16 we will be using for this guide, the BACKUP bay and a small application host. And on the bottom shelf we got the OPNSense box and the UDM.

That host and the BACKUP bay are part of a Home Assistant installation I did with their HAOS system. If you'd like to know more about it let me know.

Sources

I followed these EXCELLENT guides in order to do this:

Video:
How to Configure LAG-LACP
https://www.youtube.com/watch?v=Rb4vlN_Hf-U
Article:
Configure LAG/LACP on SFP Ports (TP-Link Example)
https://homenetworkguy.com/how-to/configure-lag-lacp-on-sfp-ports-two-tp-link-switches-with-vlans/

2. Network Topology

This entire design operates strictly at Layer 2. OPNsense is merely a bump-in-the-wire — it does not route, it does not NAT, and it does not participate in DHCP or DNS.

Those services remain entirely on the UDM:

UDM: DNS, DHCP, routing, VLANs, firewall
OPNsense: Transparent inline bridge with LACP
USW: Downstream switching with LACP

Traffic Flow

UDM → ingresslagg → laggbridge → egresslagg → UniFi Switch

Because the bridge sits inline and has no IP addresses assigned to it, the rest of the network behaves normally. All VLAN tags pass through untouched, and the UDM continues to see the whole network exactly as before.

3. System Setup (Hardware & Software Overview)

Hardware Origin

This OPNsense box came to life after my old laptop suddenly died. Instead of throwing away perfectly good components, I salvaged:

the 32 GB DDR4 RAM,
the two SSDs I later mirrored as RAID-1,
and the Wi-Fi card,

and reused them inside a barebones mini-PC I purchased from Amazon UK.

The chassis ships without RAM or storage, making it ideal for a rebuild using recycled parts.

System Specs

CPU: Intel® Celeron® J6413 @ 1.80 GHz (4 cores / 4 threads)
RAM: 32 GB DDR4-2166
Storage: 128 GB SSD (RAID-1)
NICs: 6 × Intel 1G
OS: OPNsense 25.7.5-amd64

And here is the firewall itself, with all six Ethernet interfaces populated:

4. Why Use LAGG in Transparent Mode?

I chose a transparent firewall because I wanted:

redundancy — two links on each side, loss-tolerant
throughput headroom — LACP distributes sessions across NICs
zero impact on existing network design — no routing changes
full compatibility with UniFi — VLANs, DHCP, and DNS remain untouched

Layer 2 Only

This cannot be overstated:
OPNsense in this setup is 100% Layer 2.
It is not acting as a router, DHCP server, DNS resolver, or gateway.

It simply passes frames, while optionally filtering or inspecting them inline.

5. Configuring LAGG on OPNsense

On the OpnSense management web interface go to Interfaces → Devices → LAGG

There, two LAGGs are created:

LAGG	Interface	Direction	Members
`ingresslagg`	`lagg0`	toward UDM	`igc1`, `igc2`
`egresslagg`	`lagg1`	toward UniFi Switch	`igc4`, `igc5`

Click the "+" sign on the far right side of the table in the interface to add an entry. Repeat this for both entries on the table above.

Physical interface 1
Physical interface 2
Hash layer set for L2. Following the bump-in-the-wire approach.
Provide a meaningful description. Believe me, you will need it later.

Both LAGGs are configured in LACP mode. You can select more than 2 interfaces here, it only depends on how much interfaces you have. I only had 6 so in order to have a management interface (outside of the scope of this guide) i opted to have it be 2x2 (2 interfaces in and 2 interfaces out).

After this is set, it will look something like this:

Click apply.

Next, we need to create an interface from the newly created devices. Go to Interfaces → Assignments and assign a new interface to each of the devices on the Assign a new interface menu. Remember to provide a name in the description field. That will be the interface names.

Here we have lagg-test and lagtest but you can pretend that is either ingress-lagg/egress-lagg and ingresslagg/egresslagg. Click add to create it on the interfaces table like so:

Next, we need to add both interfaces to the bridge configuration. Go to Interfaces → Devices → Bridge and click the "+" sign on the far right side of the table in the interface to add an entry:

1st LAGG interface
2nd LAGG interface
GOOD description of what this device is.

Click Save to add it to the bridge interface table. From here, the newly created LAGG interfaces are added to a single bridge device. This device needs to be assigned to an interface by repeating the process we did for the lagg interface creation. After this is done, we have something like this:

After all of this is done, we now have laggbridge = ingresslagg + egresslagg.

Now, we enable the interfaces. After this is done, under the Interfaces menu, there should appear all interfaces we created (i.e. laggbridge, ingresslagg, egresslagg). Enable them all by clicking on each of the interface names under the Interfaces menu, check the Enable checkbox and click Save.

Management and Services Interfaces

I host some services on this OPNSense box like DynamicDNS and AdGuard. The installation/management of those are outside of the scope of this article, but if you would like for me to write something about those, let me know.

These are independent interfaces I set with fixed IPs (on the UDM side). They stay outside of the bridge and ensure I can always reach OPNsense even if the bridge goes offline:

management (igc3) → 192.168.1.x/24
services (igc0) → 192.168.1.y/24

⚠️ Common mistakes to avoid

Don’t assign IPs to lagg0, lagg1, or the bridge: Assigning IPs would make the bridge participate in Layer 3 (routing), breaking its transparent Layer 2 operation. This could disrupt network traffic, cause routing conflicts, and interfere with VLANs and DHCP.
Don’t enable the physical NICs individually, only the LAGGs: Enabling NICs outside the LAGG group can cause duplicate connections, loops, or flapping, as the LAGG protocol expects to manage all member interfaces. This can destabilize the link aggregation and the bridge.
Don’t mix LACP Active/Passive between devices: LACP requires both ends to be in compatible modes (usually Active). Mixing Active and Passive can prevent proper negotiation, causing the aggregated link to fail or operate unreliably.
Don’t skip a reboot, LAGG + bridge changes apply more cleanly afterward: Network interface and bridge changes may not fully apply until after a reboot. Skipping this step can leave the system in a partially configured state, leading to unpredictable behavior or connectivity issues.

6. Configuring LACP on the UDM (Ingress Side)

The UDM handles the WAN, DHCP, DNS, routing, and VLAN assignments. In this setup, we only need it to expose two LACP ports toward the OPNsense box.

On the updated UniFi interface:

Go to Settings → Profiles → Switch Ports
Create a new Aggregate profile
Set LACP Mode: Active
Apply this profile to ports 7 and 8 (or whichever pair you use)

Follow this very useful guide from Hostify if anything is not clear:

👉 https://support.hostifi.com/en/articles/6454249-unifi-how-to-enable-link-aggregation-on-switches-lag

Only after the profile is in place should you plug in the cables.

⚠️ Don’t #1 — Don’t plug cables in before creating the LACP group

UniFi auto-profiling may assign them to something else (like LAN, PoE, or VLAN-only), breaking negotiation.

⚠️ Don’t #2 — Don’t assume LACP comes up instantly

Give it 5–10 seconds to negotiate.
If one side is up and the other is not, it will flap.

7. Configuring LACP on the UniFi Switch (Egress Side)

On the USW-16, the process is similar:

Open UniFi Controller → Devices → USW
Select ports 7 and 8
Click Aggregate
Set LACP Active
Save and wait for synchronization

These two ports will form the downstream side of the bridge.

⚠️ Don’t #3 — Don’t mix up NIC-to-port mapping

Label the NICs and cables before you start.
One swapped cable is enough to break the bundle or cause intermittent LACP flaps.

8. Testing the Transparent LAGG Bridge

Once everything is connected, verify the bridge and both LAGGs:

On OPNsense

Run:

ifconfig lagg0
ifconfig lagg1
ifconfig laggbridge

You should see all LACP members in ACTIVE state.

In the UniFi Controller

UDM ports 7/8 should show Aggregate — Active
USW ports 7/8 should show Aggregate — Active
No errors, no flapping, no “Blocking” states

Optional: Throughput Testing

You can run an iperf3 test from a LAN device to something outside the bridge. Traffic should flow, no bottleneck should be observed and failover should work if you temporarily unplug one cable from each LAGG

⚠️ Don’t #4 — Don’t enable hardware offloading prematurely

Keep the following disabled until you verify stable operation:

TSO
LRO
Checksum offload
VLAN offload

Some Intel i225/i226 NICs misbehave with offloading in bridge mode.

9. Troubleshooting

Here are the most common issues — and their causes:

Issue: LAGG fails to negotiate

UDM or USW set to Passive instead of Active
Cables on the wrong ports
One side Active, the other Disabled
An interface accidentally assigned directly instead of via LAGG

Issue: Bridge appears up but traffic drops

Hardware offloading left enabled
A NIC is flapping or mismatched
Member NIC enabled individually by mistake
Duplicate MAC confusion (if using the same NIC model)

Issue: Network meltdown (loop)

This is the most catastrophic one.

It happens when something like this occurs:

UDM → OPNsense → USW
 ↑────────────────↓
  accidental loop

A single extra cable or an unconfigured LAGG member can create a broadcast storm.

⚠️ Don’t #5 — Don’t create a second path between UDM and USW

There must be exactly one traffic path:

UDM → ingresslagg → laggbridge → egresslagg → USW

No more, no less.

10. Final Working Architecture

Once configured, the final architecture looks like this:

          WAN
           │
        [ UDM ] Ports 7+8 (LACP Active)
          │  │
          ╰╮╭╯
          ingresslagg (lagg0)
           │
          [ laggbridge ]
           │
          egresslagg (lagg1)
          ╭╯╰╮
          │  │
        [ UniFi USW ] Ports 7+8 (LACP Active)
           │
         rest of LAN

Key properties:

Fully transparent Layer 2
No IPs on LAGGs or the bridge
UDM retains all core services (DHCP, DNS, routing, VLANs)
Redundant links on both sides
Clean inline filtering option for OPNsense

11. Conclusion and results

Building a transparent bridge with LAGG on both sides is a great way to introduce a firewall into your network without redesigning the entire topology. OPNsense, combined with UniFi hardware, handles this setup surprisingly well — once everything is wired and configured correctly.

Along the way, I learned (the hard way) that LACP is extremely sensitive to:

mismatched modes
cable swaps
auto-profiling quirks
enabling NICs individually
and forgetting to disable hardware offloading

But once the pieces fall into place, the result is a rock-solid, redundant inline bridge that just works.

With this setup you are able to see and/or filter EVERYTHING that is happening on your network (at least what passes through the bridge). You can set firewall rules for traffic that passes through the bridge and see cool graphs like this one:

This setup has a lot that can be improved upon. I am open for feedback and would love ideas on how to make this better. Leave a comment below and tell me what you think. Thank you!!!

Ghidra on Linux Zero Fuss Install

Andre Faria — Tue, 11 Nov 2025 00:04:13 +0000

A straightforward, distro-agnostic way to install Ghidra under /opt without touching your system Java alternatives, plus environment wiring that makes upgrades safe and re-runs idempotent.

Disclaimer:
This post assumes you’re comfortable using the terminal with sudo, have basic knowledge of environment variables, and can install a few command‑line tools (wget, unzip, tar) with your distro’s package manager.

What you’ll get

Ghidra 11.4.1 installed under /opt/ghidra (atomic replace with backups on re-run)
A local Temurin (Adoptium) JDK 21 under /opt/java-temurin without touching system alternatives
GHIDRA_INSTALL_DIR and GHIDRA_JAVA_HOME added to your shell RC (~/.bashrc or ~/.zshrc)
A process that is safe to re-run for upgrades and repairs

Note: The approach works across common Linux distributions. The automated script tries to install helper tools via apt when available, but it gracefully skips on other distros—so on RPM/Arch-based systems, just install wget, unzip, and tar via your package manager first.

Customization tip:
You can change the Ghidra version, release date, download URL, or even inject checksum verification simply by editing the constants at the top of the referenced script (GHIDRA_VERSION, GHIDRA_DATE, GHIDRA_URL, GHIDRA_SHA256, plus the Java bits like TEMURIN_MAJOR, TEMURIN_API_URL, TEMURIN_SHA256). Adjust them, re-run the script, and it will perform an atomic upgrade while backing up previous installs.

1) Prerequisites

Install the few tools we’ll use to fetch and unpack files.

Debian/Ubuntu:
- sudo apt-get update && sudo apt-get install -y wget unzip tar
Fedora/RHEL/CentOS:
- sudo dnf install -y wget unzip tar (or yum on older releases)
Arch/Manjaro:
- sudo pacman -S --needed wget unzip tar

Also ensure you have sudo privileges.

2) Install a private Temurin JDK 21 under /opt

Why: Ghidra runs best on a modern LTS JDK. Installing a local Temurin JDK avoids messing with system-wide Java alternatives and keeps Ghidra isolated.

Key details:

Location: /opt/java-temurin/<jdk-version> with a stable symlink at /opt/java-temurin/current
Source: Adoptium API for the latest GA JDK 21 (linux x64, HotSpot) — or browse Temurin releases
Optional integrity: You can verify checksums if you provide TEMURIN_SHA256

Steps (high level):

Create the base directory: /opt/java-temurin
Download the latest GA tarball for JDK 21 from Adoptium’s API
(Optional) Verify the tarball with sha256sum
Extract to a versioned folder under /opt/java-temurin and update the current symlink atomically

On re-run, the previous version is backed up and replaced. The stable current symlink is what we’ll point Ghidra to.

3) Install Ghidra 11.4.1 under /opt/ghidra

Ghidra publishes zip archives per release. We’ll unpack into /opt/ghidra and keep upgrades atomic.

Key details:

Download from the official NSA GitHub Ghidra releases page for 11.4.1
Optional integrity: provide GHIDRA_SHA256 from the release page to verify with sha256sum
The process backs up any existing /opt/ghidra before replacing it

Steps (high level):

Download the Ghidra 11.4.1 zip
(Optional) Verify its checksum
Unzip into a temporary directory, then move to /opt/ghidra atomically

4) Wire your shell environment

Two environment variables make life easier:

GHIDRA_INSTALL_DIR: points to /opt/ghidra
GHIDRA_JAVA_HOME: points to the Temurin JDK (preferred) or falls back to an existing valid JAVA_HOME

What gets added to your shell RC (~/.bashrc for bash, ~/.zshrc for zsh):

export GHIDRA_INSTALL_DIR=/opt/ghidra
export PATH="$PATH:$GHIDRA_INSTALL_DIR/support" (so analyzeHeadless is on PATH)
export GHIDRA_JAVA_HOME=/opt/java-temurin/current (or another detected JDK)

Open a new shell or source ~/.bashrc / source ~/.zshrc to apply the changes.

Optional: add Ghidra binaries to PATH

If you want to call both the headless tools and the GUI launcher from anywhere, add both the support directory and the root install dir to PATH.

Bash:

echo 'export GHIDRA_INSTALL_DIR=/opt/ghidra' >> ~/.bashrc
echo 'export PATH="$PATH:$GHIDRA_INSTALL_DIR/support:$GHIDRA_INSTALL_DIR"' >> ~/.bashrc
source ~/.bashrc

Zsh:

echo 'export GHIDRA_INSTALL_DIR=/opt/ghidra' >> ~/.zshrc
echo 'export PATH="$PATH:$GHIDRA_INSTALL_DIR/support:$GHIDRA_INSTALL_DIR"' >> ~/.zshrc
source ~/.zshrc

5) Verify the installation

Quick checks you can run:

Print Ghidra’s internal version from its properties file
Run a short, non-interactive headless/CLI command to validate scripts are reachable

Examples:

grep -E '^application\.version=' /opt/ghidra/Ghidra/application.properties | cut -d'=' -f2

"$GHIDRA_INSTALL_DIR"/support/analyzeHeadless -version

"$GHIDRA_INSTALL_DIR"/ghidraRun -h

If those commands execute without errors, you’re set.

6) Upgrades and re-runs

This setup is designed to be re-run safely:

Temurin: new JDK drops into a versioned folder, current symlink is updated atomically
Ghidra: any existing /opt/ghidra is backed up to /opt/ghidra.bak-<timestamp> before replacement
Env: GHIDRA_JAVA_HOME is updated or inserted once; subsequent runs update it if needed

To upgrade to a newer Ghidra release, adjust the version/date variables (or pull the latest script) and re-run. Your environment variables remain compatible.

7) Uninstall or roll back

Remove Ghidra: sudo rm -rf /opt/ghidra
Remove Temurin: sudo rm -rf /opt/java-temurin
Clean your shell RC: remove the lines that export GHIDRA_INSTALL_DIR, modify PATH, and GHIDRA_JAVA_HOME
To roll back: if you see a recent backup like /opt/ghidra.bak-YYYYmmddHHMMSS, you can move it back to /opt/ghidra

8) Troubleshooting

Missing tools on non-Debian distros: install wget, unzip, tar via your package manager
Permission denied under /opt: you need sudo privileges for system locations
GUI on servers/WSL: Ghidra’s GUI requires an X server; use headless mode with analyzeHeadless if you don’t have a display
Java detection: if Ghidra prompts for a JDK, ensure GHIDRA_JAVA_HOME is set and points to a valid JDK with bin/java

9) Optional: headless workflow teaser

Ghidra ships analyzeHeadless for automation. For example, to run a simple analysis in a CI runner, you can call it with a temporary project directory and scripts. This is out of scope for this post, but the steps above already place the tool on your PATH for easy use.

Full automation script (reference)

All the steps above are automated in a single idempotent script that you can read and run:

Script: https://github.com/andremmfaria/rexis/blob/main/scripts/install-ghidra.sh

What it does for you:

Installs Temurin JDK 21 under /opt/java-temurin and creates a stable current symlink
Downloads and installs Ghidra 11.4.1 under /opt/ghidra with atomic replace and timestamped backups
Wires GHIDRA_INSTALL_DIR, updates PATH, and sets GHIDRA_JAVA_HOME
Verifies the installation non‑interactively

Safe to re-run. If you want checksum verification for either download, export GHIDRA_SHA256 and/or TEMURIN_SHA256 before running it.

Thanks for reading, and happy reversing!

Further references

Ghidra official site: https://ghidra-sre.org/
Ghidra GitHub repository: https://github.com/NationalSecurityAgency/ghidra
Ghidra releases (official zips + checksums): https://github.com/NationalSecurityAgency/ghidra/releases
Ghidra Getting Started: https://github.com/NationalSecurityAgency/ghidra/blob/master/GhidraDocs/GettingStarted.md
Adoptium Temurin releases: https://adoptium.net/temurin/releases
Adoptium API (programmatic downloads): https://api.adoptium.net/

Continuous integration with containers and inceptions

Andre Faria — Mon, 10 Nov 2025 23:50:19 +0000

How to create a CI system using container recursion and how to create a CI system using container recursion

As many of you already know, containers are something of wonder. They exist since the old days of computing in a concept called OS-level virtualization. Since then, for their flexibility, they have been used in an orchestrated manner by many awesome tools, like Kubernetes, DC/OS, Apache Mesos and many more. This provides not only an abstraction layer on OS-Level but also enables a great deal of automation where there was none before.

I used the Semantic Versioning but you can easily replace this for any versioning patterns that you like. E.g., Calendar Versioning or any other.

I am now going to show you a model of continuous integration (CI) with some "inception".

Disclaimer:
This article assumes you have some knowledge about versioning, CI, CD, containers and orchestration, along with the tooling associated with those concepts.

Tools

For code versioning we will be using the so popular Git on GitHub. You can use pretty much any versioning system in the market with this, I only used git for familiarity. And GitHub for popularity.

Jenkins is an orchestration tool first devised for CI, but since the creation of the pipeline plugin it has become more of a general-purpose orchestrator.

The docker project, or just Docker, is a very neat application that lets you manage containers in a easy and inexpensive way using their command line and API.

For container registry I used Nexus. The free version is quite nice and is more than enough to everything we need. It has a plugin for storage on Amazon's S3 which makes your storage virtually limitless. (Use at your own discretion)

Note1: For container storage you can use any registry available in applications like Artifactory but you can also use cloud services like AWS's ECR, AZURE's Container Registry or GCP's Container Registry.

Note2: There are many other steps you can input on the pipeline for testing, validation, hardening, etc. But I will be covering only the general steps of the pipeline, like code checkout, container pull, app build, container build and container storage.

Model

Some time ago I made a presentation on this topic on a meetup that happened in São Paulo - Brazil. The slides for that presentation can be found here.

This model outlines the basic actors and actions made on the process. The orchestrator, in our case Jenkins, contacts the container engine's, in our case Docker, API over TCP and creates the build container. Then it connects to a random SSH port directly on the build container. From there, the orchestrator has a few workflows based on the build workflow. The workflows on this model build two structures, the app container and other container builders.

Structures

This section shows the results of the mentioned workflows.

Container builds container builders (Recursions are always nice :D)

The build container is a specialized image that is built specifically to build other things, among them other build containers.

From this image, the orchestrator pulls container definitions for specific application languages/build engines, like Java/gradle, c#/.NETcore or javascript/npm, builds and stores them onto the container registry.

Container builds app containers

From the definitions outlined above, the applications can be built from a container specific for their case. I will be using java's case as an example. It should be similar for other cases.

Jenkins provides a nice way to bundle pipelines definitions, among other actions, using shared libraries. Those are basically Groovy scripts and exist to extend Jenkins functionality.

You also need to have a Jenkinsfile on your code's repo for it to know which app engine is to be used for the build. Here is an example Jenkinsfile:

@Library('execPipeline') _

def config = [:]
config.'engine' = 'java'
config.'type' = 'app'

execPipeline(config)

Those are the configurations the pipeline needs to know where to go.

Configuration

These are the configurations for each tool. This might change a bit overtime but the documentation is (or should) be always true.

Jenkins

As stated, above Jenkins is using shared libraries. Its config is fairly simple, you just input the git repo for the shared lib on the config and you're good to go. I used the global shared lib definitions for this example. The full steps can be found here.

Jenkins also talks to docker on a remote host for the container's management. You need to install a docker plugin and configure it properly. The steps on Jenkins's plugin side can be found here. On the docker's side you can follow these steps to enable remote access using various methods.

This assumes that you are using Jenkins multibranch pipelines to create your pipeline's definition. From this, just input your Git repo on the multibranch pipeline and it should roll out by itself.
You can follow this example to configure everything: Build a multibranch pipeline project

Nexus

Configuring Nexus is easy. Just install it following the official documentation and make sure you have storage space. You'll need it.

Set up repositories as you feel like. You can separate per environment (gamma, beta, prod, etc), per app engine type (java, node, .net core, etc), both or none. A good recommendation is to separate each artefact repository on a different port. E.g.: beta repo on port 5000, gamma repo on port 5001, etc.

Docker

Depending on your situation Docker's installation and usage, is be "easy". I mean, if your infrastructure still uses windows for a .net framework app, I feel sorry for you but this method still works.

For Linux you can follow this guide. For windows you can follow this one. For mac… well… simply don't.

Git

You don't need any specific configuration on git's side. Just make sure that Jenkins can clone your repository and your repository contains the Jenkinsfile mentioned in section 3.2 of this article. If both are good, you're golden.

Conclusion

After these configurations are setup, you will see the pipelines for each branch rollout, and your nexus registry being populated with your app containers.

Thanks for reading this article till the end. Have a nice day/afternoon/evening :D