DEV Community: sainul ashiqu

System design as a macro skill for agents:Trade-off Knowledge

sainul ashiqu — Sat, 11 Apr 2026 18:02:46 +0000

On system design as a macro skill, and why the gap matters more as agents get better

There's a pattern I keep seeing. Someone hands a coding agent a project. The agent executes. The code works. Tests pass. And then, a few weeks later, things start quietly falling apart — not because of a bug, but because of a choice made early on. The wrong database for the access pattern. A queue where a simple job runner would've been fine. A microservice boundary that made sense in theory but turned the whole thing into a distributed monolith.

The code was fine. The architecture wasn't.

What a senior developer actually brings to a project isn't just coding ability. It's a kind of pre-loaded decision map. Before writing a single line, they already know which database fits this use case, which queue strategy makes sense at what scale, where the invisible ceilings are. Not because they looked it up — because they've already been through the moment where the thing broke and had to move off it.

That's the gap. The agent is an excellent executor. The system design judgment is in the person directing it.

When the person directing it is a senior developer, things go well. They think it through, break it into clear tasks, hand it to the agent, and the agent fires through it. The hard work — what to build and with what — happened before the agent touched anything.

But most people using agents right now aren't senior developers. And even developers who are solid coders sometimes haven't had enough breadth of production experience to know, say, when Cassandra becomes the wrong call, or why your MongoDB setup is going to start hurting at a specific scale.

There's a distinction worth making here, though. If you're building something for yourself — no other users, no scaling concerns, just automating something from your own workflow — this gap barely matters. The reason is that what you're bringing to the table is something the agent genuinely can't replicate: deep domain knowledge of your own process. You know every step, every edge case, every place it breaks. That clarity is actually what makes the agent effective. You're the expert on the what, they handle the how. Agents are surprisingly capable in this mode.

The problem starts the moment you need the thing to grow, or serve other people, or hold up under real load. That's system design territory. And that's where both the agent and the person hit the same wall.

The obvious response is: just look it up. Read the docs. Search the trade-offs.

The problem is that documentation is a reference, not a judgment. It tells you what something does. What it doesn't tell you is when not to use it, what it quietly fails at in production, what decision Discord made at 10M users that you'd be wise to borrow. That kind of knowledge lives in engineering post-mortems, conference talks, internal retrospectives, the kind of thing someone shares after getting burned.

There are tools trying to address part of this — Context7 and similar MCP doc-fetchers are genuinely useful for one thing: you've already picked a tool, and you need the API right now. "How do I configure a Kafka consumer?" — they go get the docs. That's good. But it's solving a micro-level problem. The question it can't answer is: should you be using Kafka at all? That question doesn't live in any documentation page.

Take Discord's message storage history as an example of the kind of judgment that matters.

They started with MongoDB — flexible schema, fast to build on, fine at the beginning. Then message volume grew and the read/write patterns stopped fitting a document store. They moved to Cassandra — better suited for write-heavy workloads, scales linearly. But Cassandra brought its own problems: compaction-related latency spikes, and JVM garbage collection turning into an operational headache. They eventually moved to ScyllaDB — same API as Cassandra, rewritten in C++, GC issues gone, costs down.

Three moves. Each one contains a real decision signal: when does MongoDB stop being the right call? What does Cassandra actually cost you, not in dollars but in operational complexity? When is the runtime itself the bottleneck?

A senior developer carries all of that — not as a memorized list, but as pattern recognition. That's what makes them a good system designer. It's a macro skill, not a micro one. It's not "how to use Kafka." It's "when Kafka is wrong for this, and what it'll cost you to find out the hard way."

This knowledge already exists. Most of it is scattered across engineering blogs, case studies, post-mortems, talks. The issue is that it's formatted for humans reading linearly — narratives, context, long explanations. That format makes sense for learning. It's terrible for retrieval at planning time.

An agent in planning mode doesn't need to learn system design. It just needs to know the trade-off at the moment of decision. Those are two completely different information needs, and almost all the existing material is optimized for the first one.

What the agent actually needs is something like: "Use Cassandra when you have write-heavy workloads that need horizontal scale. Don't use it when your team doesn't want to manage JVM tuning and compaction. If you're on this path, look at ScyllaDB." Terse. Judgment-first. Retrievable in two seconds.

The structure that fits this best is something like an Obsidian vault. Not a flat list of tools — a graph. Notes organized by domain, connected through backlinks, so the agent can start with a vague requirement, follow the links, and arrive at a set of relevant trade-offs without having to read forty pages to get there. Each entry covers: what it is, when to use it, when not to, what it pairs with, what it silently costs you, and real examples. Short. Dense. Token-efficient.

Some domains are self-contained — caching is mostly its own thing. Others bleed into each other — your database choice affects your queue strategy, which affects your deployment model. The backlinks capture that structure. The agent can navigate it the way a senior developer navigates their own mental model: start somewhere, follow the connections, arrive at the right place.

I put together a starting scaffold for exactly this — the folder taxonomy, the entry template, a few seeded entries to show the format. It's not complete; it's nowhere near complete. But the shape is there. github.com/saitrogen/system-design-skill-wiki

If you've moved off a tool in production because it hit a wall — that decision is exactly what should be in here. Not the full story, just the signal: what you were building, what broke, what you moved to, and what you wish you'd known earlier.

The goal isn't a new platform or a product. It's a structure that agents can actually use when they're planning, so the next person who hands a task to an agent doesn't end up with an architecture they'll regret in six months.

How I Fixed Next.js Deployments on Appwrite Sites with Turborepo (Monorepo)

sainul ashiqu — Sat, 21 Feb 2026 16:24:38 +0000

TL;DR: If your Appwrite Sites deployment builds successfully but gets stuck at "Finalizing" forever — the problem is that Appwrite's internal SSR bundler can't find your next.config.js and server.js at the monorepo root. Here's the exact fix.

The Problem

I'm building Mathsido, an ed-tech platform, using a Turborepo monorepo with Next.js 16 and deploying to Appwrite Sites. The folder structure looks like this:

mathsido/
├── apps/
│   ├── app/              ← Next.js app lives here
│   │   ├── next.config.js
│   │   ├── package.json
│   │   └── src/
│   └── landing/
├── packages/
│   ├── types/
│   ├── utils/
│   └── ui/
├── package.json          ← root workspace
├── pnpm-workspace.yaml
└── turbo.json

When I deployed to Appwrite Sites, the build would succeed perfectly — all pages compiled, static pages generated, everything green. But then the deployment would get stuck on "Finalizing" for hours, sometimes days, and the site would never become reachable.

Understanding Appwrite's Build Pipeline

Before we fix the problem, let's understand what happens behind the scenes when Appwrite Sites deploys a Next.js app. This is key to understanding why the build fails in a monorepo.

Appwrite's Internal Build Steps

Appwrite Sites uses open-runtimes under the hood. When you trigger a deployment, it runs through these stages:

1. Environment Preparation     → Sets up Node.js, installs dependencies
2. Build Command Execution     → Runs YOUR build command (e.g., `pnpm build:app`)
3. SSR Bundling                → Appwrite's INTERNAL step (this is where it breaks)
4. Build Packaging             → Creates the deployment archive
5. Edge Distribution           → Deploys to Appwrite's edge network
6. Screenshot Capturing        → Takes a preview screenshot

The critical step is Step 3: SSR Bundling. This is NOT your code — it's Appwrite's internal script that prepares your Next.js app for their serverless runner.

What Appwrite Expects (The Conventions)

Here's where it gets interesting. During the SSR Bundling step, Appwrite's runner makes several assumptions about your project structure:

What Appwrite Looks For	Expected Location	Why
`next.config.js` / `next.config.mjs` / `next.config.ts`	Root of the build directory (`/usr/local/build/`)	Appwrite runs `mv next.config.* .next/` to bundle the config
`.next/` folder	Root of the build directory	This is where Next.js build output lives
`.next/standalone/server.js`	Root of `.next/standalone/`	Appwrite boots your app using this entry point
`public/` folder	Alongside `server.js`	Static assets need to be accessible at runtime
`.next/static/`	Inside `.next/` next to `server.js`	Client-side JS/CSS bundles

These conventions work perfectly for a standard single-project Next.js app:

my-app/                        ← Appwrite builds from here
├── next.config.js             ✅ Found at root
├── public/                    ✅ Found at root
├── .next/
│   ├── static/                ✅ Found inside .next
│   └── standalone/
│       └── server.js          ✅ Found at standalone root
└── package.json

But in a Monorepo, Everything is Nested

When you use Turborepo with output: 'standalone' and outputFileTracingRoot (which you must set for monorepos), Next.js generates the standalone output relative to your monorepo root, not the app directory.

Here's what the build output actually looks like:

/usr/local/build/                  ← Appwrite's build root
├── next.config.js                 ❌ MISSING (it's in apps/app/)
├── .next/                         ❌ MISSING (it's in apps/app/.next/)
├── apps/
│   └── app/
│       ├── next.config.js         ← Appwrite can't find this
│       └── .next/
│           ├── static/
│           └── standalone/
│               ├── apps/
│               │   └── app/
│               │       └── server.js  ← Deeply nested!
│               └── node_modules/
├── package.json
└── turbo.json

This mismatch causes two cascading failures:

Failure #1: The `next.config.*` Error

During SSR Bundling, Appwrite runs something like:

mv /usr/local/build/next.config.* /usr/local/build/.next/

Since next.config.js lives inside apps/app/, not at the root, the mv command fails:

mv: can't rename '/usr/local/build/next.config.*': No such file or directory

In some cases, this error is non-fatal and the build continues, but it's the first red flag.

Failure #2: The "Finalizing" Hang (The Silent Killer)

Even if the build completes and packaging succeeds, Appwrite's runtime needs to find server.js to boot your app. It looks for it at:

.next/standalone/server.js

But in your Turborepo standalone output, server.js is deeply nested at:

.next/standalone/apps/app/server.js

Appwrite can't find the entry point, fails to start the server, can't take a screenshot, and the deployment gets stuck in "Finalizing" indefinitely. There's no error, no timeout — it just... hangs.

The Fix

The solution is to add a post-build step that restructures your output to match what Appwrite expects. We need to:

Copy the .next folder (with standalone output) to the monorepo root
Copy next.config.js to the root
Place public/ and .next/static/ where the runtime expects them
Create a root-level server.js wrapper that delegates to the real nested server

Step 1: Update `next.config.js`

First, make sure your Next.js config has these critical settings:

// apps/app/next.config.js
const path = require('path');

/** @type {import('next').NextConfig} */
const nextConfig = {
    reactStrictMode: true,

    // Required for Appwrite SSR deployment
    output: 'standalone',

    // Required for monorepo: tells Next.js to trace dependencies
    // starting from the monorepo root, not just this app's directory
    outputFileTracingRoot: path.join(__dirname, '../../'),

    // If you use node-appwrite SDK, mark it as external
    serverExternalPackages: ['node-appwrite'],

    // List your internal workspace packages
    transpilePackages: [
        '@mathsido/types',
        '@mathsido/utils',
        // ... your packages
    ],
};

module.exports = nextConfig;

Why output: 'standalone'? This tells Next.js to create a self-contained server that includes only the files needed for production. Appwrite's SSR runner specifically looks for this output format.

Why outputFileTracingRoot? In a monorepo, your app imports code from packages/. Without this setting, Next.js only traces files inside apps/app/, missing all your shared packages. Setting it to the monorepo root (../../ relative to apps/app/) ensures everything is included.

Step 2: Create the Appwrite Build Script

Add this script to your root package.json:

{
  "scripts": {
    "build:app": "turbo run build --filter=@mathsido/app",
    "build:app:appwrite": "pnpm build:app && rm -rf .next && mkdir -p .next/standalone && cp -r apps/app/.next/standalone/* .next/standalone/ && cp -r apps/app/public .next/standalone/apps/app/public && cp -r apps/app/.next/static .next/standalone/apps/app/.next/static && echo \"require('./apps/app/server.js')\" > .next/standalone/server.js"
  }
}

Let's break down what this script does:

# 1. Run the normal Turborepo build
pnpm build:app

# 2. Clean any previous .next at root (avoid conflicts)
rm -rf .next

# 3. Create the directory structure Appwrite expects
mkdir -p .next/standalone

# 4. Copy the entire standalone output to root .next/standalone/
cp -r apps/app/.next/standalone/* .next/standalone/

# 5. Copy public assets to where the server expects them
cp -r apps/app/public .next/standalone/apps/app/public

# 6. Copy static client bundles (JS/CSS) to the right place
cp -r apps/app/.next/static .next/standalone/apps/app/.next/static

# 7. Create a root server.js wrapper that delegates to the real one
echo "require('./apps/app/server.js')" > .next/standalone/server.js

The magic is in step 7: we create a tiny server.js at the root of standalone/ that simply requires the real server. When Appwrite runs node server.js, it finds our wrapper, which boots the actual Next.js server from its nested location.

Step 3: Configure Appwrite Build Settings

In the Appwrite Console, go to your Site → Settings → Build, and update:

Setting	Value
Framework	Next.js
Install command	`pnpm install`
Build command	`pnpm run build:app:appwrite`
Output directory	`.next/standalone`
Rendering mode	Server side rendering

Step 4: Deploy!

Push your changes and trigger a new deployment. You should now see the complete build pipeline succeed:

[open-runtimes] Environment preparation started.
[open-runtimes] Environment preparation finished.
[open-runtimes] Build command execution started.
...
✓ Compiled successfully
✓ Generating static pages (17/17)
...
[open-runtimes] Bundling for SSR started.
[open-runtimes] Bundling for SSR finished.      ← No more errors!
[open-runtimes] Build packaging started.
[open-runtimes] Build packaging finished.
[appwrite] Edge distribution started.
[appwrite] Edge distribution finished (6/6).    ← Success!
[appwrite] Deployment finished.

The Complete Picture

Here's a visual summary of the problem and the fix:

Before (Broken)

Appwrite looks for:              What Turborepo generates:
─────────────────                ────────────────────────
/usr/local/build/                /usr/local/build/
├── next.config.js    ❌ MISS    ├── apps/app/next.config.js
├── .next/                       ├── apps/app/.next/
│   └── standalone/              │   └── standalone/
│       └── server.js ❌ MISS    │       └── apps/app/server.js
└── public/           ❌ MISS    └── apps/app/public/

After (Fixed)

/usr/local/build/
├── next.config.js              ← Copied from apps/app/
├── .next/                      ← Copied from apps/app/.next/
│   └── standalone/
│       ├── server.js           ← Wrapper: require('./apps/app/server.js')
│       ├── apps/
│       │   └── app/
│       │       ├── server.js   ← Real Next.js server
│       │       ├── public/     ← Static assets
│       │       └── .next/
│       │           └── static/ ← Client bundles
│       └── node_modules/
└── package.json

Bonus: Fixing the Broken Lockfile Warning

You might also notice this warning in your build logs:

WARN  Ignoring broken lockfile: duplicated mapping key (1032:3)

This happens when pnpm-lock.yaml has duplicate entries. To fix it:

# Regenerate a clean lockfile
rm pnpm-lock.yaml
pnpm install

This is benign — pnpm will skip the broken lockfile and resolve fresh — but it adds ~10-15 seconds to every build. Fixing it once saves time on every deployment.

Key Takeaways

Appwrite Sites makes assumptions about where Next.js files live. These work fine for single-project repos but break with monorepos.
The next.config.* must exist at the build root. During SSR bundling, Appwrite runs mv next.config.* .next/ from the root directory.
server.js must be at .next/standalone/server.js. This is the entry point Appwrite's Node.js runner uses to boot your app.
The fix is a post-build restructuring script that copies outputs to where Appwrite expects them and creates a tiny server wrapper.
outputFileTracingRoot is mandatory for monorepos. Without it, Next.js won't include your shared packages in the standalone build.

Environment & Versions

For reference, here's the exact stack this was tested with:

Tool	Version
Next.js	16.1.6
Turborepo	2.8.10
pnpm	10.0.0 / 9.15.9 (Appwrite)
Node.js	22.x
Appwrite Sites	Cloud (Feb 2026)

Useful Links

If this helped you, share it with someone else stuck on the same issue. Happy deploying! 🚀

Tags: #nextjs #appwrite #turborepo #monorepo #deployment #devops #webdev #typescript