Building an Autonomous Agent on an M1 Mac, by Choice

#discuss #ollama #llm #agents

For about 3 months I've been running an autonomous agent — one that thinks up and writes its own social media posts and comments — unattended, 4 sessions daily, on a 16GB M1 Mac with small models in the 9B / E4B class. I'm about to publish what that operation taught me about hardening, as a series of 4 technical articles.

Before that, there's one thing I want to write down first: why small models.

I've been to the purchase page for a Mac Studio or a new MacBook Pro more than once or twice. Backing the agent with a large cloud model (Opus or the GPT family) has always been an option in the code. And yet I haven't bought, and I haven't switched. The 16GB M1 is not an economic constraint — it's a constraint I chose.

From the outside, building on small models looks like a cheap compromise. This article explains why it isn't, and states where I stand. It also serves as the hub for the 4-article series.

A model's intelligence hides the roughness of your design

Large models absorb sloppy prompts, ambiguous instructions, and missing guards with sheer intelligence. If all you want is to ship a product, that's a virtue. But if you want to become someone who can build things, it becomes a defect.

Because inside the thing that worked, you can no longer tell where your design ends and the model's intelligence begins. "It worked" and "I built it" are different things. Something you bludgeoned into working with model capability counts as a thing that ran — it doesn't become the ability to build.

Small models have no absorption capacity. So every design flaw comes to the surface.

In my operation, all of the following surfaced:

The context window being silently truncated
Outputs cut off midway
A runaway caused by one missing sampling parameter

In cloud or large-model environments, these rarely bother you. The environment has cushioning built in.

Context windows are in the 200K–1M token class, so truncation itself rarely happens. And when you do exceed the limit, you get an explicit error rather than a silent cut
Sampling defaults are managed on the API side; there is structurally no room to drop a parameter
Even when a slightly-off output slips through, a smart model running the ReAct pattern (where the LLM looks at each result and picks the next move) recovers on the next step and rounds the loop off plausibly

A local 16GB machine has none of this cushioning. You specify the context window yourself, overflow is silently truncated from the head, and the sampling settings are your own property. The same class of problems surfaced in a form that could not be hidden, and the only way to close them was design.

As long as you're running on cushioning, you can't tell whether your design is protecting you or the environment is.

Training under constraints outlasts the environment

There was a period when I did business automation in an environment where I couldn't freely bring in new tools. Security constraints, constraints on approved software, and organizational friction against the very act of "introducing something new." What I could use was software already installed on everyone's PC — Excel and Access — plus an approved RPA tool.

If you can't add tools, the only way forward is to know the combinations of the tools at hand, deeply. Within those limits, I pushed automation as far as the available tools would go. But what that experience really trained wasn't tool knowledge.

The substance of automation is decomposing work that had been running on personal, artisanal tacit knowledge. Up to which step can be written deterministically, and from where does it require holistic semantic judgment grounded in experience?

You examine that boundary as finely as you possibly can, and sort the work into deterministic parts and parts that need semantic judgment. Only the former may be handed to RPA; automation that misjudges the boundary always breaks.

That sorting maps directly onto how I design agents today.

Then	Now
Routine processing writable in Excel / Access / RPA	Scripts and code (deterministic)
Parts needing human semantic judgment	LLM (semantic judgment, probabilistic generation)

I don't throw deterministic work at the LLM¹. I use the LLM only where semantic judgment is required. And since an LLM is not a substitute for human judgment, output verification and final decisions stay on the code-and-human side.

Deciding which side a task belongs to — that judgment is the center of the design. It was trained in a constraint-riddled environment, and it has stayed with me as a principle after the constraints disappeared.

A design that runs on small models doesn't depend on the model

The small-model constraint has one more meaning beyond training: proof of a lower bound.

A design that runs unattended for 3 months on the 9B / E4B class is not plugging design holes with model intelligence. Semantic judgment remains the LLM's job, but every protective mechanism lives on the code side.

Inputs are estimated and guarded before sending
Output truncation is detected and discarded
Failures are structurally recorded by telemetry (the machinery that records the system's behavior)

All of these are model-independent designs. So the design thinking carries over intact if you migrate to a large model. What doesn't carry over are the calibration values and thresholds tied to a specific model (that migration problem is what series #3 covers).

The reverse doesn't hold. A design built on the assumption of a large model's intelligence won't run as-is in a smaller environment. Stacking upward is easy; stepping downward requires a rebuild.

Of course, with a large model the same thing would have run with far less effort. But with something that ran easily, you never find out where design ends and model intelligence begins. Whatever ran on the small model, I can call my own design. That is what I mean when I say this is not a compromise.

What the series covers technically

That was the "why small" part. The techniques that actually made unattended operation work on 16GB are covered in the 4 articles that follow (links will be added as they're published).

Local LLM failures happen quietly — diagnosing silent truncation of inputs, and designing a budget guard that protects before sending
The day I dropped a 1.8x faster backend — adopting MLX because it won the benchmarks, and retreating after 24 hours in production
The day you swap the model — blind A/B testing, and the migration problem of model-tied calibration values
You can't fix what you can't observe — how telemetry, replay, and tests quietly lie

The build log of this agent itself starts at Building an agent from scratch — Claude Code and security-first development (in Japanese). The episode where the fight with small models began in earnest is My agent's memory broke — a day wrestling a 9B model (in Japanese).

Summary

Choosing small models is not an economic compromise — it's a constraint chosen to expose design flaws
What constraints train is not tool knowledge but the ability to sort deterministic parts from parts that need semantic judgment. That sorting carries straight over to the code/LLM division of labor at the center of agent design
The design thinking that sustains unattended operation on small models is model-independent and carries over intact to larger models (only the calibration values need re-measuring)
Whether to stay or move depends on your goal. If you want something running as fast as possible, cloud is the better fit. If you want the ability to build, small models are worth staying with

Top comments (1)

Xiao Man • Jul 13

The line between "it worked" and "I built it" is the best framing I've seen for why constraints matter in agent design. Your point about large models absorbing sloppy design resonated hard — I've been running into the exact same issue in a different context: LLM quality gates where the judge model's intelligence hides the fact that the gate itself is broken.

The parallel to your deterministic-vs-semantic split is almost 1:1 with what zxpmail and others have been discussing in the agent quality space — "don't judge correctness, judge risk" is really the same boundary-drawing exercise you're describing. Code owns the flow, LLM owns the judgment call, and the router decides which side a task lands on.

Running 3 months unattended on 9B models is genuinely impressive. The silent truncation issue is the kind of failure mode that only surfaces when you remove the cushion. Looking forward to the series — the MLX benchmark-vs-production story alone sounds worth the read.