A story of building an AutoML subsystem in one agentic session.
Why the heck did it take 3 weeks?
Three takeaways, less about tools, more about designing agentic development flows.
Don't trust generic tips
It seems too early to take anyone's agentic development tips as they are.
'Plan first'-type tips can either burn tokens with no quality gains, or deliver exceptional results.
No technique works everywhere, and picking the right one is on you.
For me, choosing a workflow for an agent now comes down to estimating a few things:
- input specificity
- complexity and scale
- risks
- verification approach
- quality gates
- human interaction frequency
- future observability
All while keeping an eye on the scope of tasks the agent is capable of.
Time as a design tool
Coverage and mutation testing worked pretty well as quality gates, but I forgot one thing: scale.
9,000 tests spawned out of nowhere (yes, I was chilling).
With a few subagents working in parallel, by the end of the first week their work turned into 'waiting hours for a test suite'.
Which got me thinking:
When to write tests, and how granular?
Before or after the product code is written?
Before or after the product code is verified (business value delivered)?
I can't answer these questions, not after these 3 weeks and not intuitively either. But I see a few options when I look at the timeline:
- Test as you go - test each part of Product code, at the granularity you chose. Business value is verified last.
- Product code → verify business value → pin - get the business value working first, then write tests that pin it.
These timelines fit my case (a big-scale task where gates are heavy). They're not a recipe for yours.
My takeaway is that I now see agentic development as a process on a timeline, where I decide when the feedback loops, gates and my own check-ins happen.
Harness: back in the fold
Harness reliability is key for long-running tasks.
I don't wanna wake up and think 'is it stuck?'.
I thought I could build a perfect harness from scratch, but I failed.
Why exactly, I'm not sure. Two gut feelings:
- Models feel more natural in their native harnesses. In my experience, they got lost or interrupted way less there than in my custom setup. And that turns out to be critical for multi-week goals.
- I wanted too much. I even started my own PWA, aka 'Claude Code Remote', but then I did the math on tokens.. :) Keeping up with my fantasies and all the fancy stuff the community keeps inventing takes way more resources than it sounds.
Now I imagine a harness as two layers:
- core - tools, subagents, tasks, goals, etc.
- workflow - skills (e.g. ralph, consensus planning, repo-specific ones like code review), general instructions and preferences (decision logging, retrospectives against what you optimized for, model routing, etc.)
Looking back, what I tried to replace was the core. That's where I failed.
The workflow layer is different - it's mine, and it's the part worth building to fit my style.
So here's the harness that ended up working well for me.. plain Claude Code, with my workflows on top.
Turns out it's pretty boring :)
@ Human Generated Content
Top comments (1)
9,000 tests spawning in a week is the real story here. Coverage and mutation testing as gates sounds rigorous, but at that scale the tests become the bottleneck, not the agent.