DEV Community: Surat Mukker

Lesson 3 - Architecture: Learn to organize your thoughts

Surat Mukker — Thu, 30 Jul 2026 00:33:26 +0000

AI takes the path of least resistance.

That one characteristic explains most of what changed for me about architecting a system once an agent was in the loop.

It is genuinely faster than I am on frameworks, patterns, and the standard way to wire something up. Since it has read more on them than I have. But "least resistance" means it optimizes for the thing in front of it, e.g., getting an endpoint to work or a test to pass. It cannot optimize for the shape the system needs for your use case because it does not know all details. You still own 100% of that part.

Two ways least-resistance goes wrong

Left alone, the path of least resistance breaks in two opposite directions. It cuts a corner to make the immediate thing work: collapses a boundary, hardcodes a value, skips the seam that would have let two pieces move independently later. And when you try to correct it, it will over-engineer and reach for patterns, layers, and abstractions you did not ask for and don't need yet.

Both come from the same place: it is solving the prompt, not steering the architecture.

Here is the version I lived with. My app is layered the usual way: an API layer, a service layer under it, a data-access layer under that, with clear rules about what each one is allowed to do. Database transactions belong in the service layer. The agent kept ignoring that. Commits I had scoped to the service layer kept turning up in the data layer, or up in the API. The worst one was a transaction that opened in the service layer and got committed two layers down. If left at simple prompts, it will run and deliver you something that works, but you'll find that along the way it has quietly broken the boundary and created a brittle system.

Here is the flip side, from just the other day. I was working on a bug fix with the agent on its own branch off main. Mid-test, I hit a separate gap, related to the feature but not the bug, and asked the agent to fix that too. It sensibly put the gap on its own branch, but branched off main instead of off the bug-fix branch, and I missed that. Both were named fix/..., so at a glance nothing looked off. It made the change and, per my workflow, asked me to test before it committed. The gap was handled well, but the bug was back. When I flagged it, the agent agreed: of course, that's expected since this branch came off main.

So I asked it to reconcile the two. The plan it came back with made my head spin: commit the work on the stray branch, rebase that onto the bug-fix branch, fast-forward the target, delete the stray branch, then re-verify the whole thing. But nothing was committed yet. The entire fix was three commands: git stash, switch to the bug-fix branch, git stash pop, then commit on the right branch. When I asked why not just do that, it agreed at once: "you're right, that was over-engineered."

A cut corner does not stay cut

This is the part that surprised me. Even when there was a good reason at the time, a cut corner does not stay a local mistake. Future sessions keep finding it and repeating it. The agent reads the current state of the code, sees the shortcut, and treats it as the pattern to follow. Murphy's Law seems to be in full play here. You'll have 10 good examples it could have followed, but it will find the one that it should not.

That boundary problem was exactly this. It was not a one-time slip. The agent generated code that broke the layering over and over, every session, and every time I had to catch it and send it back. Early on I would just fix it by hand. That was the mistake. Fixing it myself did nothing to stop the next session from doing it again. So I stopped hand-fixing and started making the agent fix it and write the rule down, where the next session would read it and follow it. But it still is not 100% foolproof.

Two things turned it around:

Plan reviews and a convention check became a fixed part of my workflow. Until they were, a lot of my attention was going to catching these leaks instead of the actual work. Once they were in the loop the leaks mostly stopped, and I ran a dedicated cleanup milestone to fix the ones that had already slipped through.
Isolation. I keep a full separate system-test stack and a dev stack for manual testing, non-negotiable. For anything parallel, I use worktrees, so each line of work has its own isolated stack and nothing bleeds across. Exploration you leave sitting in the main context is exploration the agent keeps copying, so keep the main context clean.

The rhythm I settled into on new work: experiment, clean up, then move on. The cleanup is not optional. It's what stops today's throwaway work/context from unintentionally becoming part of your conventions.

You bring the vision; it fills in the frameworks

The division of labor that works for me:

You own the judgment calls and the direction. State the bigger vision up front, even if you don't have every detail yet. AI can help you work out the details, but only against a destination you have set.
It knows the frameworks; it does not know your context. It's more up to speed than we are on approaches and trade-offs in the abstract. It does not know your constraints, your users, your history, any of it, until you tell it.
When it hands you a choice you don't understand, make it explain. Don't rubber-stamp an approach because it sounds authoritative. Ask it to walk you through the options and the trade-offs, then you decide.

Example: When it was time to integrate LLMs into the product I was building, I decided to make it a plugin-based system that could be driven via configs and specs. I spent a good amount of time going back and forth with it on requirements and goals. After many iterations, I had a design for an aspirational system that was the shape I wanted. Then we went back and forth on pieces that we would build now and others that would come later as needed. And then I asked it to add a lever: because it all deploys to a single container anyway, it can run the code built in instead of loading plugins every time. The agent was fine building the plugin machinery, but it would not have warned me that the pure-plugin path is a very painful operational choice.

AI can build it. You still need to define precisely for it and own the decisions.

Destination and constraints first, then a phased plan

The biggest shift is where I start. Not "let me build this piece and figure the rest out later." That's how you get architected into a corner.

Start with where you want to go and what bounds you: resources, timing, process. Then work out a phased plan to get there, and write it to disk as it evolves. Don't wait for the plan to be final before it exists on paper. A plan being figured out in the open is worth more than a perfect one still in your head, and the document keeps the history of the decisions. When you need to change something later, it tells you why it was done that way, which is exactly what you likely have forgotten by then.

A phased plan that paid off: Coming back to the same example, the AI service started as a library, imported straight into the app. Then I needed a second app that wanted almost all the same prompts. The quick path was to import it as a library again. Instead, I set the goal for the service to stand on its own over HTTP so any app can share it, and phased the way there. Stand it up as a microservice. Build the new app against it over HTTP. Keep the original app on the library path behind a flag. Prove the HTTP path works with the new app first, then migrate the original app over. The flag meant nothing broke while the new shape was still being proven.

What I keep

Architecting with AI did not lower the bar on judgment. It raised it. And it's very empowering if used correctly. For me, the agent removed the friction that used to slow me down, but it cuts both ways: because it does not have context (either you didn't specify or it rolled out of the current window), it's just as likely to speed you toward the wrong structure as the right one. The direction has to come from you, stated early, written down, and enforced.

I am still refining how much of the planning to hand over and how much to keep in my own head. But organizing your own thinking pays off in spades: get that right, and you can get the agent to follow as well.

A note on tools: I keep this deliberately tool-agnostic. Where it matters, I mean "the agent" generically. The models and tools are moving fast, so anything specific here is a snapshot, not a verdict.

Lesson 2 - Security: Secure as you build

Surat Mukker — Tue, 14 Jul 2026 19:09:11 +0000

Filing security under later does not work. For most of my career, when a vulnerability got disclosed, attackers needed days, often weeks, to turn it into a weapon. Defenders had a head start, and it's not the case anymore.

A few days from calling the build ready, one of the packages I lean on got poisoned. A release of a core dependency, one that sat deep in my stack, went up on the package index carrying a deliberate compromise. Not a bug. A rigged version built to harvest secrets: the keys, tokens, and passwords a developer has lying around. It got caught and pulled fast, but that doesn't help when it's already sitting in your own dependency tree.

It hit hard because I already knew AI was making the challenge of securing what we build worse. I'd read about it and nodded along, then deferred the library hardening until the app was built and production-ready. The warning had been in front of me for months, filed under later.

Exposures

And it wasn't a one-off. That same week, a second dependency I relied on was hijacked in a similar way, a remote-access trojan slipped into its latest version. Two compromises, two packages I used, within seven days.
That is the pace now. Poisoned code goes live and lands in thousands of dependency trees before the advisory is even written. The same speed that lets us build lets them attack.

Security First

I threw out the ship-it-then-secure-it plan. I stopped what I was doing and dealt with security first.
In the AI era, security is not a phase after the build but a part of it.

Assessment

Before I changed a thing, I worked out whether it had actually reached me.

Was it ever pulled into a production build? No. I hadn't built for prod during the exposure window, so the rigged version never got there.
Locally? I'd been in active development with hot-reload on, so I never triggered the full rebuild that would have freshly installed it. Most likely clear, not certain.
What could it have reached? This package sat deep in my stack, close to my environment and my secrets, and the compromised code was built to hunt for exactly that. So I stopped guessing and treated every key and token as already compromised.

Mitigation

Once I knew roughly where I stood, I hardened past the incident, not just against it. Every dependency you add is someone else's attack surface borrowed into yours.

Rotated every token.
Properly segregated staging and production. They'd been sharing more than they should, a cache instance and some credentials I'd waved off as low-risk.
Pinned every dependency, frontend and backend, to exact versions by hash. Nothing floating, no silent updates.
Stood up a dependency-review framework so new bumps get looked at instead of auto-trusted.
Changed when I reach for a library at all. When the second one got hit, I didn't wait for a clean version. It was small enough that I pulled it out and wrote the few lines I actually needed myself.

Gap

The dependency scare wasn't the only place I'd gone loose. Around the same time, I ran a proper security review and found something closer to home.

Once I'd stopped reading every line, the agent had quietly dropped an authorization check. The scoping that should have kept one tenant's data walled off from another was just gone.
It hadn't caused a breach; the identifiers weren't guessable, but the gate was standing open.
That is now a standing rule: a security review runs as a gate inside the build loop, not a pass tacked on at the end. When AI is writing this much of the code, we pull every review left, security included, or the code rots as fast as we build it.

Manual Levers

Not all of it is automated, and that's deliberate. Service tokens rotate on their own now. Password rotation is streamlined, but I still choose to run it myself.

The Lane

The most important lane in software right now is AI-native security: prevention, threat assessment, and mitigation built for a world where attackers move at machine speed.
You don't bolt this on at the end. It decides whether everything else is even safe to ship.
That week took away the last illusion that there was time to prepare. Getting the edge back means defending as fast as they now attack.
It's a good time to be in this lane. If you're building here, or thinking seriously about it, I'd love to compare notes.

Lesson 0 - Learning to build with AI: Figuring Out Trust boundaries

Surat Mukker — Tue, 14 Jul 2026 03:08:09 +0000

After publishing the first lesson, I realized I should have started the series with why I started to build with AI, what I learned, and how it shaped my view of software development in the age of AI.

I've led engineering teams for about 25 years. Lately I've been back in the code myself. Over the last several months I built a product end to end, mostly on my own: an AI-native procurement tool, first plan through deployment, with AI in the loop the whole way.

Zero to production in about four months.
A few thousand tests behind it.
Multiple LLM providers, picked by the use case.
The prompt drives the choice of vendor and model based on what it needs.

I wanted to understand these tools by using them, not by reading about them. What follows is what I learned, the parts that held up and the parts that didn't.

The Plan

The plan was to build it mostly myself, use AI to move faster, and own every design decision. That last part never changed. AI helps me weigh options and think through how a choice plays out before I commit, but the call stays mine. After enough years leading teams, I don't like shipping a system I can't fully account for.

That ran straight into my own gap. I've built plenty in Java, some from scratch, some I picked up and extended. Python I'm comfortable in, but I'd never built a full app in it solo, start to finish. So I was leaning on AI hardest in exactly the place I was thinnest. That's what made the choice of tool matter more than I expected.

The Tools

I started on Gemini. Good at plenty, but it fell apart on the one thing that mattered right then: it could not get its own unit tests to pass. It made a mess of mocks and kept digging itself deeper into a hole instead of fixing it. It couldn't tell me the what or why of the failures.

I decided to try Claude, and asked it to help me understand the reasons for the failures and fix them. Within 30 minutes everything was passing.

One caveat, and it holds for the whole post: none of this is an endorsement. I'm not putting one tool above another. I happened to build on a couple of specific ones, and both have moved on a lot since late 2025, so that afternoon was a snapshot, not a verdict. What follows isn't about which agent to pick. It's about how to work with any of them, because the hard parts I hit are the same whatever is doing the typing.

In fact, this is exactly why I built multi-model validation into the architecture from the ground up, instead of trusting any one model's first answer. Different models catch different things. And even a single model gives you more than its first pass if you ask for it. Hand it a leading question, tell it to review an answer, including its own, and it turns honest and digs deeper instead of defending what it just said. The catch is on us. We have to learn to ask, and remember to.

Tests stayed my Achilles heel for a long time. Every time they failed I'd point the agent at them and it would spin and get lost. So I tried giving it the failures in batches, and it did much better. The real turning point, where the tests started to pay off, came after a complete refactor of the test system (see Lesson 1).

The bigger thing I took from all this: the value isn't autocomplete. Used well, it's a tool that levels you up in the areas you're thin on, and I had one of those open in front of me. I stopped treating it like a faster keyboard and started treating it like a fast, literal engineer I had to manage.

The Setbacks

The prototypes came fast, faster than I could keep up with, if keeping up means actually understanding what's running in my own system. Nobody warns you about that part. Getting AI to produce code is the easy bit. Staying on top of what it produces is the real work.

Here is the version that made it concrete. I wrote a plan, the agent executed it, and everything looked done. Days later, working on something else, I found it had quietly decided a few things weren't important and dropped them:

Audit date fields, cut because it judged them unnecessary.
A service boundary, collapsed because, in its words, it's all one container right now anyway.
The high-value tests, skipped while the easy ones got written, and it never said so out loud.

None of this was malicious. It had a reasonable-sounding rationale for every cut, and that is exactly what makes it dangerous. The corners it chooses to cut are the ones it can argue for, so scanning the diff and asking "does this look right" won't catch them. It always looks right.

The Fixes

Got the plan out of the chat and onto disk. Plan files, tracker tables. Intent and decisions live in files now, not in a conversation that vanishes the moment a test run derails it. That's not hypothetical: it once lost an entire plan mid-session because the plan only ever existed in the chat.
Turned my review instincts into gates. A loop the work has to pass through: plan, review the plan, implement, then conventions, security, and test gates before anything ships.
Managed AI as a Contributor. I didn't sit down and design that to mirror how I ran human teams. It just emerged. Then I noticed it was the same review culture I'd always run, rebuilt for a contributor that's faster and far more agreeable than any junior I've worked with.

One standing instruction mattered more than all the tooling: challenge me. Don't just do what I ask. If what I'm asking will age badly, say so before you build it. An agent won't push back unless you tell it to. A good engineer does it without being asked.

The Unlock

For a while I didn't even run the insights report. I figured it was a distraction I didn't want to deal with. One day I ran it anyway, and it changed how I worked. It was blunt: what was working, what was holding me back, concrete suggestions on both. Then the tool offered to help me act on the feedback. I said yes.

The best call I made was to spin those improvements out into their own separate project. Everyone had been talking about skills and commands and agents and posting their own, and honestly, it was a bit intimidating while I was still struggling just to keep the agent on the rails. That toolkit project is what opened the door to working with AI effectively and efficiently.

So my advice to you is just go ahead and start. Use AI in your real workflow and it will help you figure it out as you go. You don't need to train up or set aside time to learn it first. If you write code, open an AI-enabled IDE and start coding with AI. The rest falls into place on its own.

The Now

Today I barely type. I give requirements, clarify, iterate on a plan, and the agent does the build.

The honest part: it is not faster than working without the system I built, but it is a lot more efficient. The gates and the checks cost real time. What I get back is output that's actually to spec, and a system I still understand well enough to debug at 2 am. I traded speed for fidelity, on purpose. At a certain level of seniority, that's the trade you want.

The Future

I haven't pushed it to run on a cloud VM yet. It needs more harness around it, and when I get there, I want it multi-model and fully pipelined end to end. I have outlines of a framework, but I haven't given it the time it deserves yet. If anyone is interested in this or wants to collaborate, please reach out.

The Lesson

None of this replaced the engineering. It just moved. Less of my day is typing now. Most of it is deciding what good actually looks like and building the guardrails that hold when a fast, agreeable, slightly overconfident contributor is the one doing the typing.

I don't think the people who get the most out of AI are the ones who trust it the most. It's the ones who work out the few places it can't be left alone, and stay there. For me that took months of getting it wrong first, which is most of what this post was about.

The product was the reason I started. The method is what I'm keeping.

Lesson 1 - TDD with AI: getting tests that hold up when the agent writes them

Surat Mukker — Fri, 10 Jul 2026 19:43:42 +0000

I let the agent write the tests as it built the features. They passed. I ran them, tested by hand, everything green, let it commit. For a while that felt great.

Then I changed a model. A field moved, a type changed, and a pile of tests went red. Not the tests for the thing I changed, tests all over the suite. I fixed them. A week later I changed another model, and it happened again. This kept happening until I stopped and looked at why.

Here's what made it sting. I've been writing tests for a long time, back to my Java days, and the thing I believe about them is simple. A unit test doesn't pay you back the day you write it. It pays you back later, the day you change the code and it catches what you broke. You spend a little now to buy yourself safety in the future. That's the whole deal.

These tests ran that backwards. They cost almost nothing to write and a fortune to keep. Every change broke them, and not because they'd caught a real problem. The one thing tests are supposed to make safe, changing the code, was the exact thing they made painful. That's not a test suite, that's a tax.

What had actually happened

The agent had mocked almost everything. Every test built a stand-in for the objects it touched and set the fields inline, something like this, a hundred times over with small variations:

def test_total_includes_tax():
    order = Mock()
    order.subtotal = 100
    order.tax_rate = 0.1
    order.discount = 0
    assert compute_total(order) == 110

Two things made that fragile.

First, the shape of the object was spelled out in every single test. So the day an order grew a new field, or a field changed type, I wasn't editing one place. I was editing every test that had ever built an order by hand. That is the pile going red, every time, for the same boring reason.

Second, a lot of the tests weren't checking behavior at all. They were checking wiring: that this function called that function with these arguments. assert_called_once_with(...). Change how the pieces fit together, even when the result is identical, and those tests break.

I had seen the same thing a few weeks earlier in a smaller form. Ids that were supposed to be one type kept coming back as another in the tests, and every reconciliation broke a fresh pile. Same root cause. The tests knew about details they never should have.

Why the agent wrote them this way

The agent did exactly what I asked. I asked for tests that pass. I got tests that pass. I never asked for tests that survive change, so it never optimized for that.

Left alone, a coding agent does the least it can to get the tests passing. Mocking your own code is part of that: you skip building real data, and a lot of the time you just check that some function got called. It won't reach for a shared helper unless you tell it to, and it will happily agree that everything looks fine. That last part matters. These tools are agreeable by default. If you don't ask them to push back, they won't.

The fix

Two changes.

One. Build the object in one place, and build the real thing. A single factory with sane defaults, each test overriding only the field it cares about.

def make_order(**overrides):
    defaults = dict(subtotal=100, tax_rate=0.1, discount=0)
    return Order(**{**defaults, **overrides})   # the real model, not a mock

def test_total_includes_tax():
    order = make_order(tax_rate=0.2)
    assert compute_total(order) == 120

Now a new field is one line in the factory, not a change to three hundred tests. And I stopped handing tests fake versions of my own objects. The factory builds the real thing, the same model the app uses. Mocks are only for the true edges now: the AI provider, the payment processor, email. For the code that actually reads and writes rows, the tests run against a real test database instead of a wall of mocks. When a model changes, the real constructors change with it, and most of the tests never notice.

Two. Assert behavior, not calls. Check the number that came back, the row that landed in the database, the effect that actually happened. Not which function called which. This is the change that made the suite stop breaking on refactors, because the tests now cared only about outcomes, and outcomes didn't change when I moved the plumbing around.

I deleted a lot of tests along the way too. A test that checks a mock returns the value you just set on the mock is not testing anything. The agent had written plenty, because they pass, and passing was the target. Cutting them made the suite smaller and the coverage number more honest.

The part that stuck

The factories and the real database were mechanical. The fix that lasted was upstream, in how I set the work up in the first place. Two standing instructions went in and stayed.

Mock only the real edges of the system. Everywhere else, use real objects and check what actually happened, not which function called which.
Challenge me. If I ask for something that's going to age badly, tell me before you do it, don't just do it because I asked.

If I could keep only one, it would be the second. Getting the agent to push back helped more than any factory did. Once it started telling me when I was about to make a mess, I made fewer of them.

This one outgrew the tests. It's the same thing I learned running teams. Tell a dev team to build something a certain way and most of them build it. A few will stop and tell you it's wrong. An agent is more agreeable than any junior you've ever worked with, so it almost never will, unless you ask. Say it plainly: I want to be challenged. Then hold up your end. Listen, and either defend the call or drop it.

This wasn't really a mocking problem. The agent built exactly what I asked for, and none of what I actually needed. I asked for tests that pass. What I wanted was tests that hold up when the code moves. That gap is mine to close, not the tool's. It cost me weeks of re-fixing the same tests before I stopped and dealt with the cause, so I'm writing it down for whoever hits it next.