DEV Community: Asaduzzaman Pavel

NixOS vs Traditional Linux: Why I Made the Switch and What I Learned

Asaduzzaman Pavel — Thu, 09 Apr 2026 22:39:09 +0000

After years of distro-hopping and watching every Linux installation slowly fall apart, I finally found my home in NixOS. What started as curiosity about a "weird functional Linux distribution" has turned into genuine enthusiasm for what might be the future of operating systems.

The Breaking Point

Like many Linux users, I had grown tired of a system I couldn't trust. I didn't need a dramatic failure to convince me. I simply wanted a Linux distribution where I could describe my entire setup in code rather than accumulating changes through countless imperative commands. Plus, knowing I could roll back instantly if I messed up a configuration gave me the confidence to experiment freely.

The traditional Linux model of imperative package management, where you run commands that mutate your system state, had failed me one too many times. I needed reproducibility, reliability, and the confidence that my system would work the same way today as it did yesterday.

Discovering the Nix Way

NixOS approaches system configuration from a radically different angle. Instead of imperatively installing packages and modifying configuration files scattered across your filesystem, you declare your entire system configuration in a single file (or set of files), just like dotfiles or Infrastructure as Code (IaC) tools like Ansible. The Nix package manager then builds your system from this declarative specification. With the introduction of Flakes, this process has become more manageable, allowing me to pin exact versions of dependencies and manage my configurations, just like dotfiles or IaC with Ansible, more deterministically across different machines.

This means your system is reproducible. You can take your configuration.nix file or your Flake configuration, just like dotfiles or an Ansible playbook to any machine and rebuild an identical system. No more "works on my machine" problems. No more forgotten configuration tweaks that break when you need to set up a new development environment.

What I Actually Like About It

Atomic Upgrades and Rollbacks

Every system configuration in NixOS creates a new generation. If an update breaks something, you can roll back to any previous generation instantly. This isn't just theoretical, I've used this feature countless times when experimenting with new configurations or testing newer software.

Isolation and Reproducibility

Development environments in NixOS are truly isolated. I can have multiple versions of Node.js, Python, or any other runtime available simultaneously without conflicts. Each project can specify its exact dependencies, and Nix ensures they don't interfere with each other or with the base system.

Configuration as Code

My entire system configuration lives in Github private GitLab repo. I can see exactly what changed between any two points in time, collaborate on configurations with teammates, and maintain separate branches for different use cases (work laptop, home desktop, and server setup).

The Learning Curve

I won't lie. NixOS has a steep learning curve.

The documentation is genuinely terrible. I never even knew the official NixOS wiki (wiki.nixos.org) existed, and it turns out it's just a fork of the unofficial, backdated wiki (nixos.wiki), which is frequently the first result on Google and adds confusion with its mix of accurate and misleading information. You'll find yourself piecing together information from unofficial Discord channels (shoutout to the server owner NobbZ for their help), YouTube videos by creators like VimJoyner and EmergentMind for their guidance, the NixOS manual, Nixpkgs manual, GitHub issues, and random blog posts just to accomplish basic tasks.

The official documentation often shows you what to do but rarely explains why or provides context for beginners. Want to understand how overlays work? Good luck finding a comprehensive explanation that doesn't assume you're already familiar with advanced concepts. Need to configure a service? The options are documented, but figuring out how they interact or what a minimal working configuration looks like often requires diving into the source code. Even Flakes and Home Manager, while powerful for managing configurations just like dotfiles or IaC with Ansible, add their own complexity, with sparse official guides that can leave newcomers struggling.

But here's the thing: once you understand the core concepts and how to structure projects, it all makes sense. The initial investment in learning pays dividends in system reliability and maintainability. I spent more time in my first month with NixOS reading documentation than I had in years of using other distributions, but I've spent far less time dealing with broken systems since then.

Practical Advantages in Daily Use

Development Workflows

Using nix-shell, I can drop into any development environment instantly. Need to quickly test something with Node.js 14 and a specific set of npm packages? One command gives you an isolated environment with exactly those dependencies. Working on a legacy project that requires an older version of PostgreSQL? No problem, it won't interfere with the newer version you use for other projects. Flakes make this even easier by letting me define reproducible development environments with pinned dependencies in a single flake.nix file.

If you're a developer wondering whether Nix plays well with your stack, yes it does. I use it with SvelteKit, Node.js, Go, Python, PHP and it just works. With nix-direnv, your dev environment activates automatically when you enter a project directory. No manual shell switching, no version conflicts, no "it worked yesterday" moments.

Server Management

For servers, NixOS is a game-changer. Your entire server configuration is versioned and reproducible. Deploying the same configuration to multiple servers is trivial. Rolling back a problematic deployment is instant. The days of manually configuring servers and hoping you remember all the steps are over.

Package Management

The Nix package repository is massive and remarkably up-to-date. Binary caches mean you rarely need to compile from source, but when you do need a custom build, Nix makes it straightforward to override package definitions or create your own.

What I Miss (And Don't Miss)

I occasionally miss the simplicity of apt install or pacman -S for quick one-off installations. In NixOS, even temporary software installs require a bit more thought, whether you want something available in your shell, user profile, or system configuration.

That said, I can still run something temporarily with comma, a small tool that lets you run any package from nixpkgs without installing it. For example, instead of nix run nixpkgs#cowsay, you just type , cowsay. It leaves no trace and no clutter.

What I don't miss is the anxiety around system updates on traditional distributions. I don't miss hunting down scattered config files under /etc. I don't miss the slow accumulation of entropy that gradually breaks things. And I definitely don't miss the "hope and pray" approach to system maintenance.

The Community

The NixOS community is small but incredibly knowledgeable and helpful. The quality of discourse is high, and there's a strong culture of documenting solutions and sharing configurations. The ecosystem around Nix is growing rapidly, with tools like Home Manager for user-level configuration management.

Looking Forward

NixOS has fundamentally changed how I think about computing environments. The confidence that comes from having a completely reproducible, version-controlled system configuration is liberating. I can experiment fearlessly, knowing I can always roll back. I can maintain complex development environments without the fear of conflicts or bitrot.

Is NixOS for everyone? Probably not. If you just want a simple desktop that works out of the box and you're happy with the defaults, Linux Mint, Ubuntu or Pop!_OS might serve you better. But if you're a developer, system administrator, or anyone who values reproducibility and reliability over simplicity, NixOS might just change your life.

Resources

If you're curious or just getting started, here are some links I found useful:

Nix Pill - classic introduction to Nix
NixOS Manual - official system documentation
NixOS Wiki - official NixOS wiki
nix.dev – practical guide for using Nix in real projects
Zero to nix - beginner-friendly intro to the ecosystem
Vimjoyer Youtube - tutorials and walkthroughs
Emergent_Mind Youtube - deep dives into Nix concepts
NixOS Discord (unofficial) - active and helpful community
My Configuration - my personal NixOS setup

Notes on Testing: Why I Prefer Testcontainers Over Mocks

Asaduzzaman Pavel — Thu, 09 Apr 2026 22:38:52 +0000

I've wasted entire Fridays writing "perfect" mocks for my database layer. I'd spend hours defining exactly what GetByID should return, only to have the app crash in production because of a missing comma in my SQL or a misunderstood Postgres constraint. That's the problem: mocks don't test your code, they test your assumptions about your code. And usually, your assumptions are wrong.

...And that's why I've mostly moved to Testcontainers. I'd rather wait ten seconds for a real Docker container to spin up than spend ten minutes faking a database behavior that I'm only 80% sure about anyway.

Mocks are just lies we tell ourselves

When you mock a database repository, you're essentially saying: "When I call GetByID, return this static struct." It's fast, sure, but it completely ignores reality. It doesn't catch syntax errors, it doesn't catch unique constraint violations, and it definitely doesn't catch the weird way your specific version of Postgres handles JSONB columns.

I've seen too many projects where the tests were 100% green but the application was broken in its core because the mock didn't account for a simple foreign key constraint. With a real container, the database does the work for you.

The CI Headache (The Real Gripe)

Look, I love Testcontainers, but setting them up in a CI environment like GitHub Actions is an absolute pain in the neck. You end up down a rabbit hole of Docker-in-Docker (DinD) configurations, permission issues, and mounting /var/run/docker.sock into a runner that really doesn't want you to have that much power.

There's always that one morning where the CI pipeline just hangs indefinitely because the runner ran out of disk space while trying to pull the postgres:16-alpine image for the hundredth time. It's a trade-off. I'm trading "clean" CI for "reliable" code, but don't let anyone tell you it's a smooth setup. It's a battle.

How I actually use it in Go

I don't restart the container for every single test. That would be insane. I spin up one instance of Postgres at the start of the test suite, and then I use a fresh database or a schema migration for each test.

func TestRepository_CreateUser(t *testing.T) {
    ctx := context.Background()

    // Real Postgres 16 container
    pgContainer, err := postgres.RunContainer(ctx,
        testcontainers.WithImage("postgres:16-alpine"),
        postgres.WithWaitStrategy(
            wait.ForLog("database system is ready to accept connections").
                WithOccurrence(2).
                WithStartupTimeout(5*time.Second),
        ),
    )
    if err != nil {
        t.Fatal(err)
    }
    defer pgContainer.Terminate(ctx)

    // Now we test against a REAL database
    connStr, _ := pgContainer.ConnectionString(ctx, "sslmode=disable")
    db, _ := sql.Open("postgres", connStr)
    repo := repository.NewUserRepository(db)

    // This will ACTUALLY fail if my SQL is broken
    err = repo.Create(ctx, &domain.User{Email: "test@example.com"})
    assert.NoError(t, err)
}

Confidence Over Speed

Yes, it's slower. But I'd rather have a test suite that takes two minutes and actually tells me the truth than a suite that takes two seconds and lies to my face. When I'm using my hand-crafted SQL approach, I need to know that my queries are valid. Testcontainers is the only way I've found to get that confidence without actually deploying to a staging environment.

It's not perfect. The local Docker-on-Mac/Windows slowness is real, and the CI setup is a constant fight, but I'm never going back to heavy mocking for my data layer. It's just not worth the risk of a 3 AM production page.

Deploying SvelteKit to Cloudflare Workers for Free

Asaduzzaman Pavel — Thu, 09 Apr 2026 22:33:35 +0000

For years, deploying a full-stack web app meant either paying for a VPS or wrestling with complex container orchestration. Cloudflare Workers changes that. You can run SvelteKit with server-side rendering, API routes, and edge caching, all without spending a dime on the generous free tier.

This guide walks through the exact setup I use for this site. No theoretical fluff, just working configuration.

What you get for free

Cloudflare's free tier is surprisingly capable:

100,000 requests/day — enough for most personal projects and small sites
10ms CPU time per request — SvelteKit runs comfortably within this
1 GB of KV storage — for simple config or session data
Custom domains — connect your own domain at no extra cost

Compare this to Vercel's hobby tier (limited to 10s serverless function duration) or Netlify's build minute limits. Workers' V8 isolate model means faster cold starts and more predictable pricing if you ever need to scale.

Prerequisites

Node.js 18+ installed
A Cloudflare account (free tier works fine)
A SvelteKit project ready to deploy

If you don't have a SvelteKit project yet:

pnpx sv create my-app
cd my-app
pnpm install

Step 1: Install the adapter

Cloudflare Workers runs JavaScript in V8 isolates, not Node.js. SvelteKit needs an adapter to bridge this gap:

pnpm add -D @sveltejs/adapter-cloudflare

The adapter-cloudflare package handles both Workers and Pages deployment. For new Workers Static Assets (the modern approach), this is the adapter you want, not the older adapter-cloudflare-workers.

Step 2: Configure svelte.config.js

Replace the default adapter in your svelte.config.js:

import adapter from '@sveltejs/adapter-cloudflare';
import { vitePreprocess } from '@sveltejs/vite-plugin-svelte';

/** @type {import('@sveltejs/kit').Config} */
const config = {
    preprocess: vitePreprocess(),
    kit: {
        adapter: adapter({
            // See below for options
            config: undefined,
            platformProxy: {
                configPath: undefined,
                environment: undefined,
                persist: undefined
            },
            fallback: 'plaintext',
            routes: {
                include: ['/*'],
                exclude: ['<files>', '<build>', '<redirects>']
            }
        })
    }
};

export default config;

Key options explained:

config: Path to your Wrangler config file (we'll create this next)
platformProxy: Controls how local bindings are emulated during development
fallback: 'plaintext' gives you a simple 404 page; use 'spa' if you need client-side routing for unmatched paths
routes.exclude: Tells Cloudflare which requests can bypass the Worker and serve static assets directly. This saves you invocation costs.

Step 3: Create wrangler.jsonc

Create a wrangler.jsonc file in your project root:

{
    "name": "my-sveltekit-app",
    "main": ".svelte-kit/cloudflare/_worker.js",
    "compatibility_flags": ["nodejs_als", "nodejs_compat"],
    "compatibility_date": "2024-09-23",
    "assets": {
        "binding": "ASSETS",
        "directory": ".svelte-kit/cloudflare"
    },
    "routes": [
        {
            "pattern": "yourdomain.com",
            "custom_domain": true
        }
    ]
}

What each field does:

name: Your Worker's identifier in the Cloudflare dashboard
main: The entry point SvelteKit generates. Don't change this.
compatibility_flags: nodejs_als is required for SvelteKit's async context; nodejs_compat helps with NPM packages that use Node.js APIs
compatibility_date: Cloudflare's runtime version. Bump this periodically for new features.
assets: Tells Workers where your static files live
routes: Connects custom domains (you can add this later if you don't have a domain yet)

Step 4: Build and deploy

Build your app locally first to verify everything works:

pnpm build

This creates a .svelte-kit/cloudflare/ directory containing your optimized app.

Now install Wrangler CLI and deploy:

pnpm add -g wrangler
wrangler login
wrangler deploy

After the first deploy, Cloudflare gives you a *.workers.dev subdomain. Free hosting, no domain required.

Step 5: Connect a custom domain (optional)

If you have a domain managed by Cloudflare:

Go to Workers & Pages in your Cloudflare dashboard
Select your Worker
Click "Triggers" → "Add Custom Domain"
Enter your domain

If your DNS is elsewhere, add a CNAME record pointing to your *.workers.dev subdomain.

Common gotchas

Node.js compatibility

Not all pnpm packages work in Workers. Anything relying on fs, native bindings, or certain Node.js internals will fail. Check the Cloudflare Workers runtime API docs before adding heavy dependencies.

If you hit issues, try adding the nodejs_compat flag (shown in the config above). This enables polyfills for common Node.js modules.

Environment variables

Don't use process.env in Workers. Instead, use SvelteKit's built-in $env modules:

// +page.server.js
import { SECRET_API_KEY } from '$env/static/private';

export async function load() {
    // Use SECRET_API_KEY here
}

For Cloudflare-specific bindings (KV, Durable Objects), access them via the platform object:

// hooks.js or +server.js
export async function handle({ event, resolve }) {
    const { env } = event.platform;
    // env.MY_KV_NAMESPACE, env.MY_DURABLE_OBJECT, etc.
    return resolve(event);
}

Worker size limits

The final Worker bundle must stay under Cloudflare's size limits (currently around 1MB gzipped). If your build fails with a size error:

Check for large dependencies bundled on the server side
Move heavy libraries to client-only imports if possible
Use dynamic imports for code splitting

Testing locally

You can test the production build locally with Wrangler:

pnpm build
wrangler dev .svelte-kit/cloudflare/_worker.js

This runs the exact same code that deploys to production, including platform bindings if you've configured them in wrangler.jsonc.

GitHub Actions for CI/CD

For automatic deployments on push:

# .github/workflows/deploy.yml
name: Deploy
on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: pnpm install
      - run: pnpm build
      - run: pnpx wrangler deploy
        env:
          CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}

Create a Cloudflare API token with "Cloudflare Workers" edit permissions and add it as CLOUDFLARE_API_TOKEN in your repository secrets.

Why not Cloudflare Pages?

Both Workers and Pages can host SvelteKit, but there are differences:

Feature	Workers	Pages
Static assets	Via `assets` binding	Native
Serverless functions	Single `_worker.js`	`/functions` directory
Request limits	100k/day free	100k/day free
Build output	`.svelte-kit/cloudflare`	`.svelte-kit/cloudflare`

I prefer Workers because the adapter-cloudflare generates a single _worker.js file that handles routing, SSR, and static assets. It's simpler mentally. One entry point, one mental model.

Monitoring your usage

The Cloudflare dashboard shows your request volume and CPU time. Keep an eye on:

Requests: Free tier = 100k/day. At ~10k requests/day, you've got a 10-day buffer.
CPU time: SvelteKit usually runs under 5ms per request unless you're doing heavy computation.

If you're approaching limits, consider:

Prerendering more pages at build time
Caching API responses at the edge
Using KV for frequently-read, rarely-written data

Why I use this setup

Deploying SvelteKit to Cloudflare Workers gives you a production-grade hosting stack at zero cost. The edge runtime means faster response times for global visitors, and the free tier is genuinely usable, not just a teaser to get you hooked.

The setup is minimal: install an adapter, add a config file, run two commands. The rest is just SvelteKit doing what it does best.

SSH Config: From Spaghetti to Sanity

Asaduzzaman Pavel — Thu, 09 Apr 2026 22:33:19 +0000

For a long time, my "workflow" was just a series of Ctrl+R searches in my shell history, hoping to find that one specific IP address or long-forgotten identity file flag. It was "SSH Spaghetti." It worked, but it was friction-heavy.

I realized I needed a better way when our team moved behind a Bastion host (Jump Box). Suddenly, every time I wanted to check a log or run a quick query on an internal DB, I had to SSH into the Bastion, authenticate again, and then SSH into the internal server. It was exhausting. It broke my flow.

ProxyJump: making the infrastructure transparent

When I discovered ProxyJump, it felt like magic. I could define my Bastion and my internal servers once, and suddenly I was back to a single command: ssh internal-db. SSH handles the tunneling and identity forwarding behind the scenes.

Host bastion
  HostName 1.2.3.4
  User admin
  IdentityFile ~/.ssh/id_ed25519

Host internal-db
  HostName 10.0.1.50
  User postgres
  ProxyJump bastion

Instant connections with multiplexing

The second major shift was discovering multiplexing (via ControlMaster). I often have three terminal windows open for the same server: one for htop, one for logs, and one for a shell. I was used to the 2-3 second delay of the SSH handshake every time I opened a new tab.

Setting up connection reuse changed everything. The first connection took its usual time, but the second and third were instantaneous. It made the remote server feel like it was running locally.

The orphaned socket headache

...And then I realized that if the master connection hangs or the server reboots while I have multiple tabs open, the ControlPath socket gets orphaned. You'll try to SSH back in and get some cryptic error about the socket already being in use, or worse, it'll just hang indefinitely. I ended up having to write a tiny shell alias just to rm -rf ~/.ssh/sockets/* when things get weird. It's a trade-off I'm willing to make, but it's not the "fire and forget" solution people claim it is.

The IdentityFile trap

I once spent an hour debugging why I couldn't SSH into a new staging box. Turns out, because I had too many keys in my ssh-agent, the server was rejecting me for "Too many authentication failures" before it even got to the right key.

The fix is IdentitiesOnly yes. It tells SSH to only use the key specified for that host, rather than trying every key in your agent like a brute-force attacker. It's one of those things I think everyone should just have in their Host * defaults, but I'm not 100% sure if it breaks some edge cases with hardware tokens.

Keep-alives (a bit of a guess)

I use ServerAliveInterval 60 and ServerAliveCountMax 3, but honestly, those numbers are a total guess based on what seemed to work for my home office's flaky Wi-Fi. It stops those annoying "Broken Pipe" disconnects during my morning coffee breaks, but I've also seen it keep a "zombie" connection alive for 10 minutes after I've closed my laptop lid.

Host *
  ServerAliveInterval 60
  ServerAliveCountMax 3
  IdentitiesOnly yes
  ControlMaster auto
  ControlPath ~/.ssh/sockets/%r@%h:%p
  ControlPersist 10m

The real win here isn't saving a few seconds. It's removing the mental overhead of "getting there." Now, whether I'm jumping through a triple-bastion setup or just checking a dev box, the experience is identical. It's the "commute" of my workday, and I finally stopped dreading it.

Notes on Rolling Checksums: A Weekend with rsync's Adler32

Asaduzzaman Pavel — Thu, 09 Apr 2026 22:28:02 +0000

I've always been fascinated by how rsync can synchronize large files so quickly just by sending the differences. Instead of just reading about it again, I decided to actually implement the core engine behind it: the rolling checksum.

Specifically, I spent the weekend with the Adler32 rolling implementation, following the research from Andrew Tridgell's original thesis.

The Modulo Trap

The "rolling" part of Adler32 is mathematically brilliant because it lets you calculate a new hash in O(1) time by just subtracting the byte that left the window and adding the new byte that entered. The magic number is 65521 (the largest prime under 2^16), which keeps the sums a and b within 16 bits.

Here's the math translated into Go:

const mod = 65521

func (r *RollSum) Rotate(b byte) {
    if len(r.window) == 0 {
        return
    }

    leave, enter := r.circle(b)
    r.a = (((r.a + enter - leave) % mod) + mod) % mod
    r.b = (((r.b - r.windowLen*leave - 1 + r.a) % mod) + mod) % mod
}

...And here is where I lost an hour to a completely avoidable bug. In Go (and C, and many others), the % operator isn't actually a mathematical modulo, it's a remainder operator. If the dividend is negative after subtracting the leave byte, % can return a negative value.

To get a true mathematical modulo, I had to use the ((n % mod) + mod) % mod pattern. It's an ugly, verbose little detail that completely ruins the checksum if you forget it. I assumed the standard library hash/adler32 would have some magic helper for this, but it doesn't do rolling out of the box, so you're on your own.

Manual Window Management

To actually roll the hash, you need to know exactly which byte is falling out of the window. I implemented this using a simple slice:

func (r *RollSum) circle(b byte) (leave, enter uint32) {
    r.removed = r.window[0]
    r.window = append(r.window[1:], b)
    enter = uint32(b)
    leave = uint32(r.removed)
    return
}

This works, but it's honestly terrible for performance. Calling append and shifting the slice on every single byte means the garbage collector is going to have a field day. For a toy weekend project, a slice is fine for readability, but if I actually wanted to process gigabytes of data, I'd have to rewrite this entire thing using a pre-allocated circular buffer.

Proving I Didn't Break It

Because the math is so finicky, I had to write a battery of tests against Go's standard (non-rolling) hash/adler32 implementation just to prove my math wasn't hallucinating values.

package rollsum

import (
    "fmt"
    "hash/adler32"
    "testing"
    "github.com/stretchr/testify/assert"
    "github.com/stretchr/testify/require"
)

func classic(data []byte) uint32 {
    a := adler32.New()
    a.Write(data)
    return a.Sum32()
}

func TestRollInAndOut(t *testing.T) {
    a := []byte("Adler-32 is a checksum")
    roll := New(1 << 16)
    for _, x := range a {
        roll.In(x)
    }
    require.Equal(t, classic(a), roll.Sum32())

    // Testing the actual roll
    roll.Out()
    a = a[1:]
    assert.Equal(t, classic(a), roll.Sum32())
}

We spend so much of our time working with high-level abstractions like HTTP frameworks and ORMs that taking a weekend to deal with memory allocation and bitwise shifts feels almost therapeutic. I didn't build a production-ready rsync replacement. I just wanted to remember how the underlying mechanics actually work without a framework holding my hand.

How I Organized over 100 NixOS Modules Without Going Crazy

Asaduzzaman Pavel — Thu, 09 Apr 2026 22:27:46 +0000

My first NixOS flake was a 500-line monster. Everything in one file: hardware config, user packages, SSH settings, that one weird GTK theme I liked. It worked. I was proud of it. Then I bought a laptop and tried to reuse my "setup." Copy-paste, change a hostname, rebuild...

...And spent the next three hours untangling why my desktop's NVIDIA config was trying to load on a laptop with integrated graphics. That's when I realized "just split it into files" doesn't actually solve the problem.

The mystery meat problem

Most modular NixOS setups I see follow a pattern like this:

imports = [
  ./hardware
  ./programs
  ./services
  ./users
];

Looks clean, right? Except ./programs secretly imports ./programs/development.nix which sneaks in services.docker because "you probably want it." And six months later, you're hunting through five directories trying to figure out where that virtualisation.docker.enable = true is actually coming from.

I wanted something different. I wanted discoverable modules where each piece explicitly says "I provide this" and hosts explicitly say "I want this." No mystery meat. No accidental cross-contamination between machines.

The pattern: flake-parts + auto-discovery

I settled on a pattern using flake-parts that treats modules like a catalog rather than a tree. Here's the core of my flake.nix:

{
  outputs = inputs@{ flake-parts, ... }:
    let
      inherit (inputs.nixpkgs.lib.fileset) toList fileFilter;
      import-tree = path:
        toList (fileFilter
          (file: file.hasExt "nix" && !(inputs.nixpkgs.lib.hasPrefix "_" file.name))
          path);
    in
    flake-parts.lib.mkFlake { inherit inputs; } {
      imports = import-tree ./modules;
    };
}

The import-tree function recursively finds every .nix file in modules/ (ignoring files starting with _, those are helpers). This means adding a new module is literally just creating the file. No editing imports lists. No forgetting to wire things up.

How modules register themselves

Each module in my setup registers itself into a flake-wide namespace. Here's a simplified version of my Neovim module:

{ inputs, ... }:
{
  perSystem = { pkgs, ... }: {
    # Build my custom Neovim package here
    packages.neovim-nvf = inputs.nvf.lib.neovimConfiguration {
      pkgs = customPkgs;
      modules = [ ./neovim-config.nix ];
    }.neovim;
  };

  flake.modules.nixos.programs_neovim = { config, pkgs, ... }: {
    # NixOS system-level config
    environment.systemPackages = [
      config.packages.neovim-nvf
    ];

    # Persistence settings for impermanence
    custom.persist.home.directories = [ ".local/share/nvim" ];
  };
}

Notice the flake.modules.nixos.programs_neovim line. This module doesn't do anything by default. It just makes itself available as programs_neovim in my module catalog. It contains both the package definition (perSystem) and the system integration (flake.modules.nixos).

Hosts pick what they need

My host configurations read like a menu. Here's my desktop (xenomorph):

{ inputs, ... }@top:
{
  flake.modules.nixos.host_xenomorph = { config, ... }: {
    imports = with top.config.flake.modules.nixos; [
      # Core
      gui
      wm

      # Hardware
      hardware_nvidiagpu
      hardware_qmk

      # Programs I want on this machine
      programs_neovim
      programs_steam
      programs_vesktop
      programs_spicetify

      # Services
      services_docker
      services_flatpak
    ];

    # Host-specific settings
    custom.hardware.monitors = [
      { name = "DP-1"; width = 3440; height = 1440; }
    ];
  };
}

I can see at a glance what this machine has. I can diff it against my laptop config to see what's different. And if I want to test programs_steam on my laptop, I just add it to that host's imports. No touching the module itself.

The helper functions that make this work

The import-tree function I showed earlier handles the directory scanning:

import-tree = path:
  toList (fileFilter
    (file: file.hasExt "nix" && !(inputs.nixpkgs.lib.hasPrefix "_" file.name))
    path);

This uses fileFilter from inputs.nixpkgs.lib.fileset to find every .nix file while skipping private files (those starting with _). The toList converts the file set to a list of paths that can be imported into the flake.

Why this actually scales

I've got about 100 module files now. The pattern still works because:

Adding something is trivial. Create modules/gui/zed.nix with flake.modules.nixos.programs_zed = ... and it's immediately available to any host. No imports to edit.

Disabling is trivial. Remove programs_zed from the host's imports. The module file can stay. It just isn't used.

Testing is possible. Because modules are explicit, I can evaluate nixosConfigurations.laptop.config.programs.zed.enable and know exactly where that came from.

Cross-machine diffs are readable. I can literally diff two host files and see "this one has Steam, that one doesn't."

The honest trade-offs

This isn't free. The learning curve on flake-parts is real. I spent a weekend just understanding how perSystem interacts with flake.modules. And nix flake check is slower now because it has to evaluate the entire module tree even when you're just testing one host.

Debugging "where did this option come from?" also requires different tools. Instead of grep -r through directories, I'm using nixos-option and tracing through the flake-parts module system. It's different, not necessarily better.

Also, this pattern assumes you're all-in on Flakes. If you're still using nix-channel and configuration.nix, this approach won't work without major restructuring.

The "100+ modules" reality check

Let me be honest: I don't have 100 unique modules. I have about 40 actual feature modules, plus 60+ small files for shell aliases, individual GUI programs, and helper functions. But the pattern doesn't care. Whether it's 10 modules or 200, the structure scales the same way.

The real benefit isn't the number. It's that I can add a new program, configure it, set up persistence for its dotfiles, and expose it to hosts, all in one logical file. No jumping between home.nix, packages.nix, and persist.nix to add one tool.

Resources

If you want to dig deeper into my setup, here are the key files:

My flake.nix – The import-tree pattern
flake-parts.nix – System definitions and options
modules/shell/default.nix – Example of perSystem + module combo
modules/hosts/xenomorph/ – A complete host configuration

If you're just starting with NixOS, this might be overkill. But if you're staring at a 500-line flake.nix and dreading the next refactor, this pattern might save you from going crazy too.

Reproducible Dev Environments with Nix and direnv

Asaduzzaman Pavel — Thu, 09 Apr 2026 22:22:28 +0000

I still remember the morning I joined a new project and spent three hours fighting with Node.js versions. The README said "Requires Node 18," but my system had 20, and nvm was having one of its moods. By the time I got the environment working, I'd forgotten why I wanted to contribute in the first place. That's when I realized: your development environment shouldn't be a puzzle you solve before you can write code.

The Chaos of "Just Install It"

Traditional development setup follows a familiar pattern. Clone the repo, read the README, install the right language version, install dependencies, realize you need a specific database version, install that, configure environment variables, discover the project needs an older version of some CLI tool, fight with your package manager, and eventually — if you're lucky — run the application.

The problems multiply when you switch between projects. That Node 18 project conflicts with your Node 20 side project. The Python 3.9 dependency breaks your system tools. The database version that worked yesterday doesn't match what production uses today. And god help you if you need to onboard a new team member and watch them go through the same three-hour ritual. Honestly, I've been on both sides of that and it's painful every time.

What I needed was a way to declare my environment: "This project needs Node 18.17, PostgreSQL 15, Redis 7, and these specific CLI tools." And I needed it to basically just work when I entered the project directory.

Enter Nix and direnv

Nix solves the first problem. It's a package manager that can install any version of any package side-by-side without conflicts. direnv solves the second — it automatically activates environment variables and tools when you enter a directory.

Together, they create something pretty powerful: a development environment that activates itself when you cd into a project and disappears when you leave. No manual switching. No pollution of your global system. Just clean, isolated, reproducible environments.

How It Actually Works

When I enter a project directory, direnv detects a .envrc file and loads the environment defined there. That file tells Nix to build a shell with exactly the packages specified in the project's flake.nix. When I leave the directory, direnv unloads everything. My system remains pristine.

Here's a typical setup for a Go project I work on:

# flake.nix
{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    flake-utils.url = "github:numtide/flake-utils";
  };

  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = nixpkgs.legacyPackages.${system};
      in
      {
        devShells.default = pkgs.mkShell {
          buildInputs = with pkgs; [
            go_1_22
            golangci-lint
            gotools
            gopls

            # Project-specific tools
            postgresql_16
            redis
            jq

            # These versions are pinned by the flake.lock
            # Everyone on the team gets the exact same versions
          ];

          shellHook = ''
            echo "Go $(go version)"
            echo "PostgreSQL $(pg_config --version)"

            # Set up local project paths
            export PGDATA="$PWD/.postgres"
            export REDIS_DATA="$PWD/.redis"
          '';
        };
      });
}

And the .envrc file that activates it:

# .envrc
use flake

That's it. When I cd into this project, direnv loads the Nix flake, and I suddenly have Go 1.22, the exact linter version we use, PostgreSQL 16, Redis, and all the tools configured. When I cd out, they're gone. My system shell remains untouched.

But here's what confused me at first: the first time you enter the directory, direnv shows a big scary message about blocked execution and asks you to run direnv allow. I assumed this was an error message and something was broken. It's not — it's a security feature. direnv refuses to execute any environment code until you explicitly whitelist the directory. Once you run that command, you never need to do it again unless the .envrc file changes.

The Features That Matter

Version Pinning That Actually Works

The flake.lock file pins every dependency to an exact commit. When a teammate runs this project, they get bit-for-bit identical versions of every tool. No "it works on my machine" because everyone basically has the same machine, defined in code.

Want to update dependencies? nix flake update regenerates the lock file. Want to know exactly what changed? git diff flake.lock shows you every package version change. Want to roll back? git checkout the previous lock file. It's version control for your entire development environment.

True Isolation

Each project gets its own namespace of packages. I can have one project on Node 18.17 with a specific npm version, another on Node 20 with a different npm, and a third on some ancient Node 16 for legacy maintenance. They never interfere with each other or my system packages.

This extends to environment variables too. That shellHook in the flake sets project-specific paths and configuration. When I leave the directory, those variables disappear. No more accidentally running production commands with development credentials because you forgot to switch environment files. I've done that. It's not fun.

Instant Onboarding

New team member joins? They install Nix, clone the repo, cd into it, and run direnv allow. That's it. The entire environment builds automatically. They have the same Go version, the same linter, the same database, the same everything. Onboarding time drops from hours to minutes. Pretty much instantly, actually.

Real-World Project Examples

Different projects need different setups. Here's how I structure a few common scenarios:

Web Project with Database

{
  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = nixpkgs.legacyPackages.${system};
      in
      {
        devShells.default = pkgs.mkShell {
          buildInputs = with pkgs; [
            nodejs_20
            pnpm
            postgresql_16
            redis
          ];

          shellHook = ''
            # Start services automatically if not running
            if [ ! -d "$PGDATA" ]; then
              initdb --auth=trust --no-locale --encoding=UTF8
            fi
            pg_ctl start -l log/postgres.log 2>/dev/null || true
            redis-server --daemonize yes --dir "$PWD/.redis" 2>/dev/null || true

            echo "Node $(node --version)"
            echo "Database ready on localhost:5432"
          '';
        };
      });
}

Now entering the directory starts PostgreSQL and Redis automatically (if they're not already running) and stops them when I leave. The entire development stack is self-contained.

Multi-Language Project

Sometimes you need multiple ecosystems in one project:

{
  devShells.default = pkgs.mkShell {
    buildInputs = with pkgs; [
      # Backend
      go_1_22

      # Frontend
      nodejs_20
      pnpm

      # Infrastructure
      terraform
      kubectl
      awscli2

      # Common tools
      jq
      yq
      git
    ];
  };
}

Everyone on the team uses the exact same Terraform version. No more "upgrade your CLI" Slack messages when someone updates the infrastructure code.

The Honest Trade-offs

This setup isn't free. The first time someone runs a Nix environment, it can take a while to build — especially if binary caches miss. That "instant onboarding" becomes "wait 20 minutes while Nix compiles everything" on the first run. Subsequent runs are fast, thanks to caching, but that initial build can be painful.

There's also the learning curve. Nix has its own language, its own concepts, and documentation that often assumes you already understand those concepts. Teaching a junior developer Nix before they can write their first line of code is probably overkill. For simple projects, honestly, the overhead might not be worth it.

Docker Compose solves similar problems and might be a better fit if your team is already comfortable with it. The trade-off there is Docker's overhead and the need to run commands inside containers rather than your native shell. Nix gives you native performance with container-like isolation.

I also got burned once editing the shellHook in a flake, saving it, and wondering why my changes didn't take effect. I assumed direnv would reload automatically when the flake.nix changed. It doesn't — you need to run direnv reload or leave and re-enter the directory. Caching is great for speed, pretty confusing when you're debugging.

When This Shines

I basically reach for Nix + direnv when:

Multiple projects need different versions of the same tool
Team onboarding is painful and inconsistent
"Works on my machine" happens regularly
I want CI and local environments to match exactly
I need complex development stacks (multiple databases, message queues, etc.)

I don't reach for it when basically:

It's a quick one-off script
The team is new to development and already overwhelmed
The project only needs one common tool (like just Node.js)
Everyone is already happy with their setup

Getting Started

If you want to try this yourself:

Install Nix with Flakes enabled: Determinate Systems installer makes this easy
Install direnv: nix profile install nixpkgs#direnv (ironically) or use your system package manager
Hook direnv into your shell: direnv hook bash (or zsh, fish) in your shell config
Create a flake.nix in your project root (start with one of my examples above)
Create .envrc with just use flake
Run direnv allow in the project directory

That's it. Your environment now activates automatically.

If something doesn't load, run direnv status to see what's happening. It will tell you if the environment is blocked, if there were errors loading the flake, or if you're in a directory that's not whitelisted yet.

For CI/CD or one-off usage without direnv, you can also run nix develop directly in the project — it drops you into the same shell that direnv would have activated.

Resources

direnv documentation - Hook setup for your shell
nix.dev - Practical Nix guide, especially the flakes tutorial
Zero to Nix - Beginner-friendly intro
My dotfiles - Real-world examples of development shells

Neovim Keybindings and My Workflow

Asaduzzaman Pavel — Thu, 09 Apr 2026 22:22:12 +0000

I'm Asad, a backend software engineer always on the hunt for tools that can improve my productivity and make coding less tedious. Neovim has been one of those tools, and its keybindings have made a real difference in how I work.

Neovim's Modal Editing

Neovim is a fork of Vim and uses something called "modal editing." It feels strange at first, especially if you're used to regular text editors, but it becomes natural quickly. Neovim has different modes that define what your keystrokes do:

Normal Mode

This is your navigation mode, where you move through your code using single-key commands. j takes you down, k takes you up, and w jumps word to word, just to name a few.

Insert Mode

This is where you type, the standard text entry mode. Nothing fancy, just get your words down.

Visual Mode

Selecting text visually happens in this mode. Highlight, copy, cut, or do whatever you want with the selected text.

Command Mode

You can do searches, execute commands, and more in this mode. It's your way to access Neovim's deeper functionality.

Keybindings I Actually Use

Here's where the efficiency comes in. The keybindings in Neovim, especially in Normal Mode, become muscle memory once you learn them. Your fingers stay on the keyboard and your coding speed improves.

Single-key commands for moving around, deleting words (dw), copying (y), pasting (p), it's all designed for speed and precision.
Press /, type what you're looking for, and let Neovim find it for you. n takes you to the next match, N takes you back.
Managing tabs is simple. Open a new tab with :tabnew, switch between them with gt and gT, and close tabs with :q.
Neovim opens up a world of customization. :map allows you to create your own shortcuts, tailoring Neovim to fit your workflow.
Need to view multiple files at once? :split and :vsplit split the window horizontally and vertically.
Jump to specific lines with ease. Prefix a line number with : and hit Enter. For example, :42 takes you to line 42.
Want to swap out a specific word all over your current file? Use :%s/old/new/g to replace every instance of 'old' with 'new'.

Neovim is all about making it your own. You can create custom shortcuts and functions to fit your style.

How Neovim Changed My Workflow

Since I started using Neovim, my productivity has improved noticeably. I can move through my codebase without reaching for the mouse, keeping my hands on the keyboard. Neovim bends to my will. Custom plugins and keybindings make it a good match for my needs.

Useful Links

My dotfiles and neovim config
Neovim website
Quick reference guide
kickstart.nvim A great starting config for Neovim

I Assumed SvelteKit 5 Would Just Work. I Was Wrong About the Mental Model.

Asaduzzaman Pavel — Thu, 09 Apr 2026 22:16:54 +0000

I assumed SvelteKit 5 would be a drop-in upgrade. Update the package.json, run npm install, maybe fix a few type errors. That's how it worked with Svelte 3 to 4. This time, the code compiled fine on the first try — but my brain didn't.

The migration took three days longer than expected, not because of broken builds, but because I kept trying to think in Svelte 4 patterns while the framework wanted me to think in runes. Here's what actually caught me off guard.

The Migration Script Lied (Kind Of)

Running npx sv migrate svelte-5 worked surprisingly well. It converted my let declarations to $state(), my $: statements to $derived() and $effect(). The diff was thousands of lines, mostly mechanical changes. I committed everything and ran the dev server expecting green checkmarks.

But here's the part that tripped me up: the migration script can't migrate your understanding. It turned this:

let count = 0;
$: doubled = count * 2;

Into this:

let count = $state(0);
const doubled = $derived(count * 2);

What I didn't immediately grasp was that $derived runs differently than $:. In Svelte 4, that reactive statement ran once before render and only updated when its dependencies changed. In Svelte 5, $derived is lazy — it only runs when you read the value, and it's memoized. Which sounds better, until you're debugging and wondering why your console.log isn't firing when you expect it to.

I Kept Trying to Export Let

The second day was spent fighting with component props. The migration script converted export let propName to $props(), but I kept writing new components the old way out of habit. Muscle memory is real — I'd type export let automatically, then stare at the error for thirty seconds before remembering.

The new $props() syntax with destructuring felt verbose at first:

// Old way (Svelte 4)
export let title = 'Default';
export let required;

// New way (Svelte 5)
let { title = 'Default', required } = $props();

But I have to admit: renaming props is cleaner now. I used to hate writing export { className as class } or dealing with $$restProps. Now it's just let { class: className, ...rest } = $props() and everything behaves predictably. The migration guide wasn't lying when they said this reduces API surface area.

Slots to Snippets Broke My Brain

The biggest friction wasn't syntax — it was conceptual. Svelte 4 slots felt like HTML. You had a <slot />, you passed content between tags, it just worked. Svelte 5 snippets are more powerful but require a mental shift.

In Svelte 4, I would write:

<!-- Card.svelte -->
<div class="card">
    <slot name="header" />
    <slot />
</div>

In Svelte 5, that becomes:

<!-- Card.svelte -->
<script>
    let { header, children } = $props();
</script>

<div class="card">
    {@render header?.()}
    {@render children?.()}
</div>

The ?.() optional chaining matters — if you don't include it and the parent doesn't provide that snippet, you get a runtime error. I learned this the hard way with a custom modal component that crashed when I tried to render it without a header.

The `$app/stores` Deprecation Sneak Attack

SvelteKit 2.12 deprecated $app/stores in favor of $app/state. I did not see this coming because I was focused on the Svelte 5 migration, not SvelteKit changes. Suddenly my $page.params access felt wrong.

// Old way (deprecated)
import { page } from '$app/stores';
$page.params.slug;

// New way
import { page } from '$app/state';
page.params.slug; // No $ prefix needed!

The fine-grained reactivity is actually better — updating page.state no longer invalidates page.data — but finding all the $page references across a large codebase took longer than I expected. The npx sv migrate app-state command helped, but it only catches .svelte files, not the .svelte.ts utility functions I'd started writing.

The Dependency Version Trap

Before you even touch the migration script, check your package.json. SvelteKit 2 requires specific minimum versions across the entire ecosystem, and the migration script doesn't always handle peer dependencies correctly.

Here is what you actually need:

{
    "devDependencies": {
        "svelte": "^5.55.0",
        "@sveltejs/kit": "^2.56.0",
        "@sveltejs/vite-plugin-svelte": "^5.0.0",
        "vite": "^7.0.0",
        "typescript": "^5.0.0"
    }
}

The gotcha: @sveltejs/vite-plugin-svelte 5.x+ is now a peerDependency, not a direct dependency of SvelteKit. If you just bump SvelteKit and forget that one, you'll get cryptic errors about missing preprocessors or failed TypeScript compilation. The migration script will update it, but double-check — I've seen it miss this in projects with complex dependency trees.

I also had to bump the adapter — Cloudflare adapter 2.x doesn't work with SvelteKit 2.5x+. You need @sveltejs/adapter-cloudflare version 5.x minimum. Same story for the other adapters: Netlify needs 5.x, Node needs 5.x, Vercel needs 5.x. The adapter versioning now basically tracks SvelteKit minor versions, which is easier to remember but means you can't skip adapter updates anymore.

Node version matters too. SvelteKit 2 requires Node 18.13 or higher. If you're still on Node 16 in production — and I was, on an older VPS — the build will fail with errors that look like import path problems but are actually just "your Node is too old."

The migration script updates your package.json, but it doesn't always pick the right versions if you have conflicting constraints. I had to manually delete node_modules and lockfile twice before everything resolved correctly.

What Actually Annoyed Me

Here's my genuine complaint: the documentation presents Svelte 5 as "almost completely backwards-compatible," which is technically true at the compiler level but misleading at the human level. Yes, your Svelte 4 components work in a Svelte 5 app. But as soon as you start mixing patterns — using a Svelte 5 snippet inside a Svelte 4 slot-based component, for example — things get weird.

I spent two hours debugging why a snippet wasn't rendering before realizing the parent component was still using <slot /> instead of {@render children?.()}. The reverse doesn't work: you can't pass slotted content to a component that expects snippets.

Also, the migration script leaves createEventDispatcher calls untouched because it's "too risky." So now I have a codebase that's half callback props, half event dispatchers, and I have to remember which components use which pattern. It's not unmanageable, but it's not the seamless upgrade the blog posts promised either.

What I Think Now

After a week of working with it, I'm warming up to runes. Being able to pull reactive logic out of components into .svelte.ts files without stores is genuinely useful. I refactored a complex form validation system that was previously using four different stores into a single reusable function with $state and $derived, and it's easier to test now.

But I'd recommend a different migration strategy than the one I used. Don't run the migration script on your entire codebase at once. Start with leaf components — the simple presentational ones with no slots or events. Get comfortable with the new syntax there first. Then tackle the complex components with slots and callbacks. Save the global changes (like $app/stores to $app/state) for last.

I'm not 100% sure if Svelte 5 is actually faster for my use case. The bundle size dropped slightly — about 3KB gzipped — but that's in the noise for most apps. The real win, if there is one, will be in maintainability six months from now when I'm not debugging mysterious $: statement ordering issues.

How to Actually Read Nix Error Messages (Without Crying)

Asaduzzaman Pavel — Thu, 09 Apr 2026 22:16:38 +0000

I stared at the terminal for twenty minutes. The error message was ten lines long, referenced some file in the Nix store with a hash I'd never remember, and told me something about "infinite recursion" or "attribute missing" or... something. I don't even remember anymore. I just remember the feeling: Nix was speaking a language I didn't understand, and I was failing.

That was my first week with Nix. It took months before I could read an error message without my brain basically shutting down. But once you learn the pattern, once you understand what Nix is actually trying to tell you, the errors go from cryptic nonsense to useful hints.

Here's the thing nobody told me: Nix errors have a grammar. They follow a structure. Once you know how to parse them, they stop feeling like punishment and start feeling like maps. Pretty weird, but true.

The Anatomy of a Nix Error

Most Nix errors have four distinct layers. Let's look at a real one I hit last week:

error: infinite recursion encountered
       at /nix/var/nix/profiles/per-user/root/channels/nixpkgs/lib/attrsets.nix:123:5
          122|   recursiveUpdate = x: y:
          123|     mapAttrs (name: val:
             |     ^
          124|       if builtins.isAttrs val && builtins.isAttrs x.${name} or null
       … while evaluating a nested attribute set
       … while calling the 'map' builtin
       … in the expression at /home/user/flake.nix:42:8

Looks terrifying, right? But it's actually telling you four specific things. Here's the breakdown:

Error type (the first line): infinite recursion encountered. This is what went wrong, not necessarily where, but the class of problem.

Location (the second line): The file and line number where Nix crashed. Usually deep in nixpkgs. This is rarely where you need to fix anything.

Trace (the "… while" lines): The evaluation stack, showing the chain of function calls. Reading these bottom to top shows the path from your code to the crash site.

Your code (the last "… in" line): The last trace entry is almost always the closest to where you introduced the problem. This is where you start debugging.

The rule I learned: The error location at the top is where Nix crashed. Your bug is almost always in the last line of the trace, or just above it. Read the trace bottom-up. That's it. That's the whole trick.

The Five Error Classes You'll Actually See

Nix errors basically group into a handful of recurring types. Recognizing them on sight saves you most of the diagnostic work.

1. Infinite Recursion Encountered

This means Nix tried to evaluate an attribute that depends on itself. Common causes: using self or super incorrectly in an overlay, or a callPackage where an attribute refers back to the derivation being defined.

# Classic trap: referring to 'pkgs' before it's fully constructed
let
  pkgs = import <nixpkgs> { overlays = [ overlay ]; };
  overlay = final: prev: {
    myPkg = pkgs.stdenv.mkDerivation { ... }; # ← should be 'final.stdenv'
  };
in pkgs

Debug strategy: Add --show-trace to your nix build or nix eval call. The trace will point you to exactly which attribute triggered the cycle. Yeah, it's verbose, but it's worth it.

2. Value Is Not a Function / Not an Attribute Set

Nix's type system is lazy. Errors only surface during evaluation. This means you passed something of the wrong shape. Usually a missing argument, a typo in an attribute name, or forgetting to apply a function.

# Often caused by:
let
  f = { foo }: foo + 1;
in
f 42  # passing an int to a function expecting an attrset

When the error says "value is not an attribute set" and points into nixpkgs, it usually means you've passed a derivation where a set of options was expected, or vice versa. For instance, calling pkgs.python3.withPackages and forgetting the lambda. I did this one twice last month.

3. Attribute Not Found

The most common error for nixpkgs newcomers. You're trying to access an attribute that doesn't exist on a set. Check spelling, check whether the package is in the right channel, and make sure you haven't confused a package set (like pkgs.python3Packages) with a package (pkgs.python3). Seriously, this trips up everyone at first.

error: attribute 'pythonPackages' missing

# The attribute set 'pkgs' does have this path, just named differently:
# pkgs.python3Packages.requests   ← correct
# pkgs.pythonPackages.requests    ← does not exist

4. Builder Failed with Exit Code N

This is a build-time error, not an evaluation error. Nix successfully evaluated your derivation, but the build script failed. The error message itself is almost never useful. The real information is in the build log.

error: builder for '/nix/store/...-my-package-1.0.drv' failed with exit code 1
       For full logs, run:
         nix log /nix/store/...-my-package-1.0.drv

Critical: Always run nix log immediately. The exit code alone tells you basically nothing. The log tells you everything: missing headers, failed tests, network errors, wrong phase outputs. Seriously, always check the log first.

5. Hash Mismatch

A fetchurl, fetchgit, or similar fetcher got content that doesn't match the declared hash. This is either a stale hash in your derivation, or the upstream source changed in place (which happens more than you'd hope).

error: hash mismatch in fixed-output derivation '/nix/store/...':
         specified: sha256-AAAA...
         got:       sha256-BBBB...

Fix it by updating the hash. The "got" value is the correct one. Copy it into your derivation. For fetchgit, also update the rev if the source moved. Yeah, it's annoying when upstream changes things, but that's the price of reproducibility.

Reading the Trace Strategically

The "… while evaluating" lines form a stack trace in reverse evaluation order. Your instinct is to look at the top. Resist it. The top is the crash site, deep in nixpkgs or a library. The bottom is where your code called into the machinery.

"The trace is a telescope pointed backwards. The furthest point is where things broke. Your eye should start at the lens."

The mental model: Each "… while" is a stack frame. Nix was evaluating the bottom frame first: your flake.nix, your module, your overlay. It called into something, which called into something else, until finally it reached the frame that caused the failure.

My process:

Find the last "… in the expression" line. This is typically pointing at your code. Open that file and go to that line.
Read one frame up. The next "… while" line tells you what Nix was doing with your value: calling a function, iterating a list, constructing an attribute set.
Match the operation to the error type. If it says "while evaluating a nested attribute set" and the error is "infinite recursion", look for circular attribute references at your entry point.
Ignore frames that are entirely inside nixpkgs unless you're writing an overlay or patch. Those frames are symptoms, not causes.
When unsure, add builtins.trace probes. Drop builtins.trace "reached here: ${builtins.typeOf x}" x near your suspect expression to confirm what's being evaluated and in what shape.

Practical Debugging Commands

Knowing the theory is one thing. These are the commands you'll actually reach for when things break:

# Always add --show-trace for eval errors
nix build .#myPackage --show-trace

# Evaluate a single attribute without building
nix eval .#packages.x86_64-linux.myPackage.name

# Enter a nix repl and poke at the value directly
nix repl
:lf .                    # load current flake
packages.x86_64-linux.myPackage.buildInputs

# Read the full build log after a builder failure
nix log /nix/store/HASH-myPackage-1.0.drv

# Inspect what a derivation will build without building it
nix derivation show .#myPackage

Leveling Up: A Better REPL

The default nix repl is barebones. After months of typing :lf . and manually navigating attribute sets, I built a custom repl.nix that preloads all my hosts, configs, and helper functions. It looks like this:

# filename: repl.nix
{
  # Pass your host name when starting the repl
  host ? "xenomorph",
  ...
}:
let
  flake = builtins.getFlake (toString ./.);
  inherit (flake.inputs.nixpkgs) lib;
in
{
  inherit lib;
  inherit (flake) inputs;

  # Quick access to current host
  c = flake.nixosConfigurations.${host}.config;
  config = c;
  co = c.custom;
  pkgs = flake.nixosConfigurations.${host}.pkgs;

  # Helper functions for debugging
  keys = lib.attrNames;                    # List all attributes in a set
  deps = pkg: map (p: p.name or "unknown") (pkg.buildInputs ++ pkg.nativeBuildInputs or []);

  # Find where an option is defined (filters out nixos internals)
  where = path: let
    opt = lib.attrByPath (lib.splitString "." path) null
          flake.nixosConfigurations.${host}.options;
    decls = if opt != null && opt ? files then opt.files else [];
    isMine = f: !(lib.hasInfix "nixos/modules/" (toString f));
    in lib.filter isMine decls;

  # Reload without exiting repl
  reload = import ./repl.nix { inherit host; };
}
// flake.nixosConfigurations  # Merge in all host configs

Why this is better:

keys pkgs: Instead of typing :p pkgs and scrolling through 50,000 attributes, get a clean list
where "services.nginx.virtualHosts": Find exactly which file defines that option, filtered to ignore nixpkgs internals
deps pkgs.hello: See build inputs for any package without digging through nixpkgs source
c / config / co: Short aliases for the current host's config and custom options
reload: Re-import the repl.nix without exiting (great when you're editing the repl itself)

Using it with a helper script:

#!/usr/bin/env bash
if [[ -f repl.nix ]]; then
  nix repl --arg host '"xenomorph"' --file ./repl.nix "$@"
else
  # Fallback to regular repl if not in a project with repl.nix
  nix repl .
fi

Now when I hit an error, I can jump into the repl and inspect values immediately:

nix-repl> c.services.nginx.enable
true

nix-repl> where "services.nginx.virtualHosts"
[ /home/k1ng/nix/dotfiles.nix/modules/web/nginx.nix ]

nix-repl> keys co.persist.home
directories files

nix-repl> deps pkgs.neovim
[ "libuv" "msgpack" "tree-sitter" ... ]

The inspiration for this came from Brian McGee's post about Nix's slow feedback loop. The REPL becomes a real debugging tool, not just a calculator.

Pro tip: The nix repl is criminally underused. You can load any flake, inspect attribute values, call functions, and test expressions interactively, all without triggering a full build. I use it basically every day now.

Quick Reference

Error	What It Means	First Thing to Try
`infinite recursion`	Circular attribute dependency	Check overlays and `self`/`super` usage
`not a function`	Wrong type passed	Check arg shapes with `nix repl`
`attribute missing`	Typo or wrong channel	Run `nix search nixpkgs <name>`
`builder failed`	Build-time error	Always read the full log with `nix log`
`hash mismatch`	Update the hash	Copy the "got" value into your derivation
Reading traces	Read bottom-up	Your entry point is at the end, not the top

Nix errors reward patience and a systematic reading strategy. The wall-of-red sensation disappears as soon as you know where to look first. That's almost always the bottom of the trace, not the top.

Once that clicks, debugging Nix feels less like archaeology and more like following a well-marked trail. Still annoying sometimes, honestly, but at least you know where you're going.

Resources

If you're stuck on a Nix error, these resources actually help:

nix.dev Troubleshooting Guide - Official guide for common errors like database issues, broken binary caches, and macOS problems
nix.dev Nix Language Basics - Understanding the language fundamentals makes error messages way clearer
Nix Manual: Built-in Functions - Reference for debugging tools like builtins.trace
Nix Manual: nix repl - Documentation for interactive debugging
Nixpkgs Manual: Library Functions - When you need to understand what lib functions are doing
NixOS Discord - The unofficial Discord is more helpful than you'd expect. Bring your --show-trace output.
NixOS Discourse - Search here first. Most errors have been asked about before.
NixOS Wiki - Actually has useful debugging tips mixed in with the outdated stuff

How I Actually Structure My Go Services

Asaduzzaman Pavel — Thu, 09 Apr 2026 22:11:21 +0000

Every time I see a new Go project with a pkg/ directory, I already know it's going to be a junk drawer. People treat pkg/ like a "public" folder for things they might want to share later, but they never do. It just ends up being a collection of half-baked string helpers and logger wrappers that leak into every other part of the app.

I've shifted almost everything into internal/. If a piece of code isn't meant to be imported by another service, it has no business being public. It's a simple boundary that saves me from "accidental coupling" where some random utility function in Service A starts getting used by Service B just because it was easy to find.

The "Manual" Wiring

...And that's why I stopped using dependency injection frameworks like Uber's fx or Google's wire. I know, I know, "it's just boilerplate." But I'd rather write thirty lines of explicit struct initialization in main.go than spend twenty minutes debugging why a provider didn't register or why some magic reflection-based container is throwing a cryptic error at startup.

When I look at my main.go, I see the entire dependency graph in plain text. I can see exactly where the database pool is created and exactly which services are using it. There's no magic. If something is missing, the compiler tells me.

The Layout I Actually Use

It's not revolutionary, but it's flat enough that I don't get lost in a sea of nested folders.

.
├── cmd/service/main.go  # The Wiring
├── internal/
│   ├── api/             # Transport (HTTP/gRPC)
│   ├── domain/          # The Interfaces (Contracts)
│   ├── logic/           # The "Brain"
│   ├── store/           # The Persistence
│   └── config/          # Environment-first config
└── Makefile             # My actual UI

Hand-Crafted SQL (The Verbosity Tax)

I avoid ORMs. I also avoid most SQL generators. I've tried sqlc, and while it's better than most, I still find myself fighting it when I need a complex join or a specific Postgres quirk that the parser doesn't like.

I write raw SQL and map it to structs. For a long time, I complained about the "verbosity tax" of Go, all those rows.Scan(&item.ID, &item.Name...) calls that felt like 1990s coding. But honestly, if you're using pgx/v5, it's not even that bad anymore. Between pgx.CollectOneRow and pgx.RowToAddrOfStructByName, the generic-based helpers make the manual scanning part almost disappear for most queries.

// Using pgx/v5 generics
p, err := pgx.CollectOneRow(rows, pgx.RowToAddrOfStructByName[Account])
if err != nil {
    return nil, err
}

It's still manual in the sense that you're writing the query and choosing the mapping strategy, but it's not the carpal-tunnel-inducing slog I thought it was. It's the right balance of control without the mystery performance issues you get with a full-blown ORM.

The "Domain" as a Buffer

I treat internal/domain as a DMZ. It's nothing but interfaces and structs. No database tags, no JSON tags, no business logic. It defines what the service does without caring about the how.

The logic package implements these interfaces. It never imports api or store. This is the only way I've found to keep testing from becoming a nightmare. I don't need a real Postgres instance to test a pricing calculation; I just pass a mock that satisfies the domain.Store interface and I'm done.

The One-Minute Rule

If I can't find a specific database query or a business rule within 60 seconds of opening the project, the structure has failed. I rely on fzf for everything, but the layout is also built so that LSP "goto definition" actually lands me in the right implementation, not some generated proxy or a massive interface file. internal/store/postgres/account.go tells me exactly what I'm getting.

I assumed that more "sophisticated" layouts with transport/ and endpoints/ and service/ folders would help as the project grew, but it turned out to just be more cognitive load. Simple and flat usually wins.

Multi-Agent Orchestration in Go

Asaduzzaman Pavel — Thu, 09 Apr 2026 22:11:04 +0000

I spent a whole day debugging why my agent kept hallucinating tool calls. The model would output "Action: web_search" but forget the "Action Input" part, or mix up the format entirely. LangChainGo's ConversationalAgent expects a specific pattern: Thought, Action, Action Input. Local models basically struggle with it.

The fix wasn't tuning the prompt. It was abandoning the framework entirely.

This post shows the pipeline architecture I ended up with: three specialized agents (research, write, edit) that pass structured data between each other using direct LLM calls. No agent framework parsing. No format gymnastics. It just works. Honestly, I should have done this from the start.

Stop Overcomplicating This

Single agents are like hiring one person to be your lawyer, accountant, and chef. They can do all three, but badly. Splitting into specialists gives you clearer prompts, better error isolation, and the ability to parallelize. Your fact-checker agent works across content types without rewriting prompts.

I've tried supervisor patterns and handoff mechanisms. They add complexity you rarely need. The pipeline pattern (Researcher → Writer → Editor → Output) is the easiest to implement correctly and maps cleanly to Go's concurrency model. Supervisors make sense when you need runtime judgment about which specialist to invoke. Handoffs work for conversational bots that pivot mid-stream. For document generation and ETL workflows, use a pipeline.

The problem with LangChainGo's agent framework is basically that it requires models to output this exact format:

Thought: I need to search for this
Action: web_search
Action Input: AI impact on software engineering

Local models mangle this constantly. They skip the Thought section, hallucinate nonexistent actions, or merge fields into gibberish. Direct LLM calls eliminate this parsing overhead. You call GenerateContent(), manually invoke tools when needed, and pass results back. The code is simpler, debugging is easier, and local models behave predictably. No more regex parsing nightmares.

The Code

Three agents using direct LLM calls with streaming support, built on a shared BaseAgent foundation. Here's the actual project structure:

2-multi-agent-pipeline/
├── main.go
├── internal/
│   ├── agents/
│   │   ├── base.go              # Shared LLM calling logic
│   │   ├── simple_researcher.go # Research agent
│   │   ├── writer.go            # Writer agent
│   │   ├── editor.go            # Editor agent
│   │   ├── errors.go            # Structured error types
│   │   ├── options.go           # Configuration options
│   │   └── pipeline.go          # Pipeline runner abstraction
│   └── tools/
│       └── search.go            # Brave Search tool
├── go.mod
└── go.sum

Base Agent: The Foundation

All agents share common LLM interaction logic through a BaseAgent struct. This basically eliminates duplication and provides consistent streaming, retries, and logging:

// filename: base.go
package agents

import (
    "context"
    "fmt"
    "log"
    "strings"
    "time"

    "github.com/tmc/langchaingo/llms"
)

type StreamHandler func(chunk string)

// Agent is the common interface for all pipeline agents
type Agent interface {
    ExecuteWithStream(ctx context.Context, input string, handler StreamHandler) (string, error)
}

type BaseAgent struct {
    llm llms.Model
}

func NewBaseAgent(llm llms.Model) BaseAgent {
    return BaseAgent{llm: llm}
}

func (b *BaseAgent) callLLM(
    ctx context.Context,
    prompt, system string,
    temp float64,
    maxTokens int,
    handler StreamHandler,
    logPrefix string,
) (string, error) {
    content := []llms.MessageContent{
        llms.TextParts(llms.ChatMessageTypeSystem, system),
        llms.TextParts(llms.ChatMessageTypeHuman, prompt),
    }

    if handler != nil {
        result, err := b.streamLLM(ctx, content, temp, maxTokens, handler, logPrefix)
        if err == nil {
            return result, nil
        }
    }

    return b.simpleLLM(ctx, content, temp, maxTokens)
}

func (b *BaseAgent) streamLLM(
    ctx context.Context,
    content []llms.MessageContent,
    temp float64,
    maxTokens int,
    handler StreamHandler,
    logPrefix string,
) (string, error) {
    var response strings.Builder
    chunkCount := 0

    _, err := b.llm.GenerateContent(ctx, content,
        llms.WithTemperature(temp),
        llms.WithMaxTokens(maxTokens),
        llms.WithStreamingFunc(func(ctx context.Context, chunk []byte) error {
            if len(chunk) == 0 {
                if chunkCount < 5 && logPrefix != "" {
                    log.Printf("[%s] Empty chunk #%d received", logPrefix, chunkCount+1)
                }
                chunkCount++
                return nil
            }
            str := string(chunk)
            response.WriteString(str)
            if handler != nil {
                handler(str)
            }
            chunkCount++
            if chunkCount == 1 && logPrefix != "" {
                log.Printf("[%s] First chunk received (%d bytes): %.50s...", logPrefix, len(chunk), str)
            }
            return nil
        }),
    )

    if err != nil {
        return "", err
    }

    if logPrefix != "" {
        log.Printf("[%s] Total chunks: %d, Total bytes: %d", logPrefix, chunkCount, response.Len())
    }
    return response.String(), nil
}

func (b *BaseAgent) simpleLLM(
    ctx context.Context,
    content []llms.MessageContent,
    temp float64,
    maxTokens int,
) (string, error) {
    const maxRetries = 3
    var lastErr error

    for i := 0; i < maxRetries; i++ {
        resp, err := b.llm.GenerateContent(ctx, content,
            llms.WithTemperature(temp),
            llms.WithMaxTokens(maxTokens),
        )
        if err == nil {
            if len(resp.Choices) == 0 {
                return "", ErrNoResponse
            }
            return resp.Choices[0].Content, nil
        }

        lastErr = err
        if i < maxRetries-1 {
            log.Printf("[LLM] Retry %d/%d after error: %v", i+1, maxRetries, err)
            time.Sleep(time.Second * time.Duration(i+1))
        }
    }

    return "", fmt.Errorf("failed after %d retries: %w", maxRetries, lastErr)
}

Error Handling

Structured errors make debugging easier. Each agent failure basically includes context about which agent failed and in what phase:

// filename: errors.go
package agents

import (
    "errors"
    "fmt"
)

// Agent errors for better error handling
type Error struct {
    Agent string
    Phase string
    Cause error
}

func (e *Error) Error() string {
    return fmt.Sprintf("%s agent failed during %s: %v", e.Agent, e.Phase, e.Cause)
}

func (e *Error) Unwrap() error {
    return e.Cause
}

// Common errors
var (
    ErrNoResponse       = errors.New("LLM returned no response")
    ErrContextCancelled = errors.New("operation cancelled")
    ErrSearchFailed     = errors.New("search tool failed")
)

// Result holds agent output with metadata
type Result struct {
    Content   string
    CharCount int
    Phase     string
}

func NewResult(content, phase string) Result {
    return Result{
        Content:   content,
        CharCount: len(content),
        Phase:     phase,
    }
}

Configuration Options

Functional options pattern for agent configuration. Pretty standard Go approach:

// filename: options.go
package agents

import "github.com/tmc/langchaingo/llms"

// Option configures an Agent
type Option func(*config)

type config struct {
    temperature  float64
    maxTokens    int
    systemPrompt string
    logPrefix    string
}

func defaultConfig() config {
    return config{
        temperature: 0.5,
        maxTokens:   2000,
        logPrefix:   "Agent",
    }
}

func WithTemperature(t float64) Option {
    return func(c *config) {
        c.temperature = t
    }
}

func WithMaxTokens(n int) Option {
    return func(c *config) {
        c.maxTokens = n
    }
}

func WithSystemPrompt(p string) Option {
    return func(c *config) {
        c.systemPrompt = p
    }
}

func WithLogPrefix(p string) Option {
    return func(c *config) {
        c.logPrefix = p
    }
}

// LLMCaller handles LLM interactions with configuration
type LLMCaller struct {
    llm    llms.Model
    config config
}

func NewLLMCaller(llm llms.Model, opts ...Option) LLMCaller {
    cfg := defaultConfig()
    for _, opt := range opts {
        opt(&cfg)
    }
    return LLMCaller{llm: llm, config: cfg}
}

The Search Tool

The researcher needs search. I use Brave Search API because it returns clean results without the SEO noise of Google. Honestly, it's way better than dealing with Google's API nonsense:

// filename: search.go
package tools

import (
    "context"
    "encoding/json"
    "fmt"
    "io"
    "log"
    "net/http"
    "net/url"
    "os"
    "time"
)

type Search struct {
    client *http.Client
    apiKey string
}

func NewSearch() *Search {
    return &Search{
        client: &http.Client{Timeout: 10 * time.Second},
        apiKey: os.Getenv("BRAVE_API_KEY"),
    }
}

func (s *Search) Name() string { return "web_search" }

func (s *Search) Description() string {
    return "Search the web using Brave Search API. Returns titles, descriptions, and URLs."
}

type BraveResponse struct {
    Web struct {
        Results []struct {
            Title       string `json:"title"`
            URL         string `json:"url"`
            Description string `json:"description"`
        } `json:"results"`
    } `json:"web"`
}

func (s *Search) Call(ctx context.Context, input string) (string, error) {
    if s.apiKey == "" {
        return s.mockResults(input), nil
    }
    return s.braveSearch(ctx, input)
}

func (s *Search) braveSearch(ctx context.Context, input string) (string, error) {
    params := url.Values{"q": {input}, "count": {"10"}}
    apiURL := "https://api.search.brave.com/res/v1/web/search?" + params.Encode()

    req, err := http.NewRequestWithContext(ctx, "GET", apiURL, nil)
    if err != nil {
        return "", fmt.Errorf("create request: %w", err)
    }

    req.Header.Set("X-Subscription-Token", s.apiKey)
    req.Header.Set("Accept", "application/json")

    resp, err := s.client.Do(req)
    if err != nil {
        return "", fmt.Errorf("search request: %w", err)
    }
    defer resp.Body.Close()

    body, _ := io.ReadAll(resp.Body)

    if resp.StatusCode != http.StatusOK {
        return "", fmt.Errorf("API status %d: %s", resp.StatusCode, string(body))
    }

    log.Printf("[Brave] Response: %.300s...", string(body))

    var braveResp BraveResponse
    if err := json.Unmarshal(body, &braveResp); err != nil {
        return "", fmt.Errorf("parse results: %w", err)
    }

    return s.formatResults(input, braveResp.Web.Results), nil
}

func (s *Search) formatResults(
    query string,
    results []struct {
        Title       string `json:"title"`
        URL         string `json:"url"`
        Description string `json:"description"`
    },
) string {
    formatted := make([]map[string]string, len(results))
    for i, r := range results {
        formatted[i] = map[string]string{
            "title":   r.Title,
            "snippet": r.Description,
            "source":  r.URL,
        }
    }

    output := map[string]interface{}{
        "query":   query,
        "results": formatted,
    }

    data, _ := json.Marshal(output)
    return string(data)
}

func (s *Search) mockResults(input string) string {
    return fmt.Sprintf(`{"query": "%s", "results": [
        {"title": "Guide to %s", "snippet": "Key facts about %s", "source": "example.com"},
        {"title": "%s - Wikipedia", "snippet": "Encyclopedia article", "source": "wikipedia.org"},
        {"title": "Research on %s", "snippet": "Academic papers", "source": "research-db.example"}
    ]}`, input, input, input, input, input)
}

The Agents

Each specialized agent basically embeds BaseAgent and focuses on its specific task:

// filename: simple_researcher.go
package agents

import (
    "context"
    "fmt"
    "log"

    "github.com/tmc/langchaingo/llms"
    langchaintools "github.com/tmc/langchaingo/tools"
)

type SimpleResearcherAgent struct {
    BaseAgent
    tools []langchaintools.Tool
}

func NewSimpleResearcher(llm llms.Model, tools []langchaintools.Tool) *SimpleResearcherAgent {
    return &SimpleResearcherAgent{
        BaseAgent: NewBaseAgent(llm),
        tools:     tools,
    }
}

func (r *SimpleResearcherAgent) findSearchTool() langchaintools.Tool {
    for _, t := range r.tools {
        if t.Name() == "web_search" {
            return t
        }
    }
    return nil
}

func (r *SimpleResearcherAgent) ExecuteWithStream(
    ctx context.Context,
    topic string,
    handler StreamHandler,
) (string, error) {
    prompt := r.buildPrompt(ctx, topic)
    return r.callLLM(ctx, prompt, "You are a research specialist.", 0.5, 2000, handler, "Researcher")
}

func (r *SimpleResearcherAgent) buildPrompt(ctx context.Context, topic string) string {
    searchResults := r.getSearchResults(ctx, topic)

    if searchResults == "" {
        return fmt.Sprintf(`Research "%s". Provide comprehensive notes with facts, statistics, trends, and sources.`, topic)
    }

    return fmt.Sprintf(`Research "%s" based on these search results:

%s

Provide comprehensive notes with facts, statistics, trends, and sources.`, topic, searchResults)
}

func (r *SimpleResearcherAgent) getSearchResults(ctx context.Context, topic string) string {
    searchTool := r.findSearchTool()
    if searchTool == nil {
        return ""
    }

    results, err := searchTool.Call(ctx, topic)
    if err != nil {
        log.Printf("[Search] Failed: %v", err)
        return ""
    }

    log.Printf("[Search] Got %d chars", len(results))
    return results
}

Writer and Editor follow basically the same pattern with different prompts and temperatures:

// filename: writer.go
package agents

import (
    "context"
    "fmt"

    "github.com/tmc/langchaingo/llms"
)

type WriterAgent struct {
    BaseAgent
}

func NewWriter(llm llms.Model) *WriterAgent {
    return &WriterAgent{BaseAgent: NewBaseAgent(llm)}
}

func (w *WriterAgent) ExecuteWithStream(
    ctx context.Context,
    research string,
    handler StreamHandler,
) (string, error) {
    prompt := fmt.Sprintf(`Transform these research notes into an engaging article:

%s

Write with compelling introduction, clear section headings (Markdown ##), factual accuracy, strong conclusion, and professional tone.`, research)

    return w.callLLM(ctx, prompt, "You are a professional writer.", 0.7, 2000, handler, "Writer")
}

// filename: editor.go
package agents

import (
    "context"
    "fmt"

    "github.com/tmc/langchaingo/llms"
)

type EditorAgent struct {
    BaseAgent
}

func NewEditor(llm llms.Model) *EditorAgent {
    return &EditorAgent{BaseAgent: NewBaseAgent(llm)}
}

func (e *EditorAgent) ExecuteWithStream(
    ctx context.Context,
    draft string,
    handler StreamHandler,
) (string, error) {
    prompt := fmt.Sprintf(`Edit this article for grammar, clarity, structure, and tone:

%s

Output the polished version directly. No commentary.`, draft)

    return e.callLLM(ctx, prompt, "You are a meticulous editor.", 0.3, 2000, handler, "Editor")
}

Pipeline Runner

For more complex workflows, the PipelineRunner abstraction basically lets you compose agents declaratively:

// filename: pipeline.go
package agents

import (
    "context"
    "fmt"
)

// PipelineStep represents one stage in the pipeline
type PipelineStep struct {
    Name   string
    Agent  Agent
    Prompt func(string) string // Transform previous output into prompt
}

// PipelineRunner executes a sequence of agents
type PipelineRunner struct {
    steps []PipelineStep
}

func NewPipelineRunner(steps ...PipelineStep) *PipelineRunner {
    return &PipelineRunner{steps: steps}
}

// Run executes the pipeline sequentially
func (p *PipelineRunner) Run(
    ctx context.Context,
    initialInput string,
    onStep func(name, output string),
) (string, error) {
    input := initialInput

    for _, step := range p.steps {
        select {
        case <-ctx.Done():
            return "", &Error{Agent: step.Name, Phase: "execution", Cause: ErrContextCancelled}
        default:
        }

        prompt := input
        if step.Prompt != nil {
            prompt = step.Prompt(input)
        }

        output, err := step.Agent.ExecuteWithStream(ctx, prompt, nil)
        if err != nil {
            return "", &Error{Agent: step.Name, Phase: "execution", Cause: err}
        }

        if onStep != nil {
            onStep(step.Name, output)
        }

        input = output
    }

    return input, nil
}

// RunWithStream executes with streaming for the final step only
func (p *PipelineRunner) RunWithStream(
    ctx context.Context,
    initialInput string,
    handler StreamHandler,
) (string, error) {
    if len(p.steps) == 0 {
        return "", fmt.Errorf("no steps in pipeline")
    }

    input := initialInput

    // Run all but last step without streaming
    for _, step := range p.steps[:len(p.steps)-1] {
        output, err := step.Agent.ExecuteWithStream(ctx, input, nil)
        if err != nil {
            return "", &Error{Agent: step.Name, Phase: "execution", Cause: err}
        }
        input = output
    }

    // Final step with streaming
    lastStep := p.steps[len(p.steps)-1]
    return lastStep.Agent.ExecuteWithStream(ctx, input, handler)
}

Pipeline Runner

For more complex workflows, the PipelineRunner abstraction basically lets you compose agents declaratively:

// filename: writer.go
package agents

import (
    "context"
    "fmt"

    "github.com/tmc/langchaingo/llms"
)

// PipelineStep represents one stage in the pipeline
type PipelineStep struct {
    Name   string
    Agent  Agent
    Prompt func(string) string // Transform previous output into prompt
}

// PipelineRunner executes a sequence of agents
type PipelineRunner struct {
    steps []PipelineStep
}

func NewPipelineRunner(steps ...PipelineStep) *PipelineRunner {
    return &PipelineRunner{steps: steps}
}

// Run executes the pipeline sequentially
func (p *PipelineRunner) Run(
    ctx context.Context,
    initialInput string,
    onStep func(name, output string),
) (string, error) {
    input := initialInput

    for _, step := range p.steps {
        select {
        case <-ctx.Done():
            return "", &Error{Agent: step.Name, Phase: "execution", Cause: ErrContextCancelled}
        default:
        }

        prompt := input
        if step.Prompt != nil {
            prompt = step.Prompt(input)
        }

        output, err := step.Agent.ExecuteWithStream(ctx, prompt, nil)
        if err != nil {
            return "", &Error{Agent: step.Name, Phase: "execution", Cause: err}
        }

        if onStep != nil {
            onStep(step.Name, output)
        }

        input = output
    }

    return input, nil
}

// RunWithStream executes with streaming for the final step only
func (p *PipelineRunner) RunWithStream(
    ctx context.Context,
    initialInput string,
    handler StreamHandler,
) (string, error) {
    if len(p.steps) == 0 {
        return "", fmt.Errorf("no steps in pipeline")
    }

    input := initialInput

    // Run all but last step without streaming
    for _, step := range p.steps[:len(p.steps)-1] {
        output, err := step.Agent.ExecuteWithStream(ctx, input, nil)
        if err != nil {
            return "", &Error{Agent: step.Name, Phase: "execution", Cause: err}
        }
        input = output
    }

    // Final step with streaming
    lastStep := p.steps[len(p.steps)-1]
    return lastStep.Agent.ExecuteWithStream(ctx, input, handler)
}

Main Orchestrator

The pipeline wires everything together with detailed logging. Here's what that looks like:

// filename: main.go
package main

import (
    "context"
    "fmt"
    "log"
    "os"
    "time"

    "github.com/k1ng440/go-llm-demo/2-multi-agent-pipeline/internal/agents"
    "github.com/k1ng440/go-llm-demo/2-multi-agent-pipeline/internal/tools"
    "github.com/tmc/langchaingo/llms"
    "github.com/tmc/langchaingo/llms/ollama"
    langchaintools "github.com/tmc/langchaingo/tools"
)

type Pipeline struct {
    researcher *agents.SimpleResearcherAgent
    writer     *agents.WriterAgent
    editor     *agents.EditorAgent
}

func NewPipeline() (*Pipeline, error) {
    model := "minimax-m2.7:cloud" // Ollama Cloud; or "qwen3.5:9b" / "llama3.1:8b" for local

    llm, err := ollama.New(
        ollama.WithModel(model),
        ollama.WithPredictMirostat(0),
    )
    if err != nil {
        return nil, err
    }

    testCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()
    _, err = llm.Call(testCtx, "Hi", llms.WithTemperature(0.1))
    if err != nil {
        return nil, fmt.Errorf("model %s not available (run: ollama pull %s): %w", model, model, err)
    }

    searchTools := []langchaintools.Tool{tools.NewSearch()}

    return &Pipeline{
        researcher: agents.NewSimpleResearcher(llm, searchTools),
        writer:     agents.NewWriter(llm),
        editor:     agents.NewEditor(llm),
    }, nil
}

func (p *Pipeline) Run(ctx context.Context, topic string) (string, error) {
    log.Println("[1/3] Research...")
    log.Println("[Streaming] Starting...")
    research, err := p.researcher.ExecuteWithStream(ctx, topic, func(chunk string) {
        fmt.Print(chunk)
        os.Stdout.Sync()
    })
    fmt.Println()
    if err != nil {
        return "", fmt.Errorf("research: %w", err)
    }
    log.Printf("      -> %d chars", len(research))
    log.Printf("      -> Preview: %.100s...", research)

    log.Println("")
    log.Println("[2/3] Writing...")
    log.Printf("      -> Input: %d chars of research", len(research))
    log.Println("[Streaming] Starting...")
    draft, err := p.writer.ExecuteWithStream(ctx, research, func(chunk string) {
        fmt.Print(chunk)
        os.Stdout.Sync()
    })
    fmt.Println()
    if err != nil {
        return "", fmt.Errorf("writing: %w", err)
    }
    log.Printf("      -> %d chars", len(draft))
    log.Printf("      -> Preview: %.100s...", draft)

    log.Println("")
    log.Println("[3/3] Editing...")
    log.Printf("      -> Input: %d chars of draft", len(draft))
    log.Println("[Streaming] Starting...")
    final, err := p.editor.ExecuteWithStream(ctx, draft, func(chunk string) {
        fmt.Print(chunk)
        os.Stdout.Sync()
    })
    fmt.Println()
    if err != nil {
        return "", fmt.Errorf("editing: %w", err)
    }
    log.Printf("      -> %d chars", len(final))
    log.Printf("      -> Preview: %.100s...", final)

    return final, nil
}

func main() {
    log.Println("Multi-Agent Pipeline Demo")
    log.Println("=========================")

    pipeline, err := NewPipeline()
    if err != nil {
        log.Fatal(err)
    }

    topic := "The impact of AI on software engineering workflows"

    ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
    defer cancel()

    result, err := pipeline.Run(ctx, topic)
    if err != nil {
        log.Fatal(err)
    }

    fmt.Println("")
    fmt.Println("=========================")
    fmt.Println("FINAL OUTPUT")
    fmt.Println("=========================")
    fmt.Println(result)
    fmt.Printf("\nTotal: %d characters\n", len(result))
}

To actually run this:

# Get a Brave Search API key from https://brave.com/search/api/
export BRAVE_API_KEY="your-api-key-here"

# Pull a local model (skip if using Ollama Cloud)
ollama pull qwen3.5:9b

# Run the pipeline
go run main.go

For Ollama Cloud models like minimax-m2.7:cloud, you need a Pro or Max subscription. Local models like qwen3.5:9b or llama3.1:8b run on your own hardware.

Without BRAVE_API_KEY, the search tool basically returns mock data. Which is fine for testing.

Handling Failures

The naive pipeline fails on any error. Production basically needs fallback logic.

Retry the research step a few times before giving up. Here's how:

// filename: main.go
func (p *Pipeline) RunWithFallback(ctx context.Context, topic string) (string, error) {
    var research string
    var err error

    for i := 0; i < 3; i++ {
        research, err = p.researcher.ExecuteWithStream(ctx, topic, nil)
        if err == nil {
            break
        }
        log.Printf("[Pipeline] Research attempt %d failed: %v", i+1, err)
        time.Sleep(time.Second * 2)
    }

    if err != nil {
        log.Println("[Pipeline] Using fallback research")
        research = fmt.Sprintf("Basic information about: %s", topic)
    }

    // Continue with writer and editor...
}

Sometimes you basically want intermediate results even if later stages fail:

// filename: main.go
type PipelineResult struct {
    Research  string
    Draft     string
    Final     string
    Errors    []error
    Completed bool
}

func (p *Pipeline) RunPartial(ctx context.Context, topic string) PipelineResult {
    result := PipelineResult{}

    research, err := p.researcher.ExecuteWithStream(ctx, topic, nil)
    if err != nil {
        result.Errors = append(result.Errors, fmt.Errorf("research: %w", err))
        return result
    }
    result.Research = research

    draft, err := p.writer.ExecuteWithStream(ctx, research, nil)
    if err != nil {
        result.Errors = append(result.Errors, fmt.Errorf("writer: %w", err))
        result.Draft = "[Failed to generate draft]"
        result.Final = research // Return research as fallback
        return result
    }
    result.Draft = draft

    final, err := p.editor.ExecuteWithStream(ctx, draft, nil)
    if err != nil {
        result.Errors = append(result.Errors, fmt.Errorf("editor: %w", err))
        result.Final = draft // Return draft if editor fails
        return result
    }
    result.Final = final
    result.Completed = true

    return result
}

Parallel Execution

When agents can work independently, use goroutines. You'll need to import sync:

// filename: main.go
import "sync"

func (p *Pipeline) ResearchParallel(ctx context.Context, topics []string) ([]string, error) {
    results := make([]string, len(topics))
    errList := make([]error, len(topics))

    var wg sync.WaitGroup
    for i, topic := range topics {
        wg.Add(1)
        go func(idx int, t string) {
            defer wg.Done()
            results[idx], errList[idx] = p.researcher.ExecuteWithStream(ctx, t, nil)
        }(i, topic)
    }
    wg.Wait()

    var combined []string
    for i, err := range errList {
        if err != nil {
            log.Printf("[Parallel] Topic %d failed: %v", i, err)
            continue
        }
        combined = append(combined, results[i])
    }

    if len(combined) == 0 {
        return nil, fmt.Errorf("all parallel research tasks failed")
    }

    return combined, nil
}

Don't Let Context Bleed

The biggest trap in multi-agent systems is earlier agents' reasoning confusing later ones. When agents share memory, you basically get contamination.

Use explicit handoffs with structured data. Here's what I mean:

// filename: writer.go
func (w *WriterAgent) ExecuteFromResearch(
    ctx context.Context,
    research *ResearchOutput,
    handler StreamHandler,
) (string, error) {
    prompt := fmt.Sprintf(`
Write an article about "%s" using these research findings:

Key facts: %v
Sources: %v
Statistics: %v
Notes: %s`,
        research.Topic,
        research.KeyFacts,
        research.Sources,
        research.Statistics,
        research.RawNotes,
    )

    return w.callLLM(ctx, prompt, "You are a professional writer.", 0.7, 2000, handler, "Writer")
}

Debugging

When things break, you basically need to know which agent failed and why. The BaseAgent already logs extensively, but you can add a tracer wrapper:

// filename: tracer.go
package observability

import (
    "context"
    "log"
    "time"
)

type AgentTracer struct {
    AgentName string
}

func (t *AgentTracer) Trace(
    ctx context.Context,
    input string,
    fn func() (string, error),
) (string, error) {
    start := time.Now()
    log.Printf("[Trace] %s started | input: %d chars", t.AgentName, len(input))

    output, err := fn()

    duration := time.Since(start)
    if err != nil {
        log.Printf("[Trace] %s FAILED after %v | error: %v", t.AgentName, duration, err)
    } else {
        log.Printf("[Trace] %s completed in %v | output: %d chars", t.AgentName, duration, len(output))
    }

    return output, err
}

Picking Models (And Knowing When to Stop)

Not every agent needs the same model. Different models excel at different tasks.

Research is mostly factual lookup. Fast models (Minimax, Qwen3.5, Llama) do this well. Writing needs creativity and flow. Premium models (Claude Opus 4, GPT-4o) shine here. Editing is about following rules. Mid-tier models (Claude Sonnet, GPT-4o-mini) are basically sufficient.

The quality gap is task-dependent. A fast model extracting facts from Brave Search performs nearly as well as Opus 4. But ask it to write engaging prose and the gap is pretty obvious.

Start with fast models everywhere. Upgrade individual agents to premium only when you see specific quality gaps. Usually this means keeping research on fast models, upgrading writing to premium if the output needs to be engaging, and using mid-tier for editing, or skipping the editor entirely if the writer is good enough.

More agents is not always better. Agent overhead is real: each adds latency and debugging complexity grows quadratically with handoffs. I follow one rule: one agent per distinct kind of work. Research, writing, and editing are different. "Research part A" and "Research part B" are basically the same task. Use parallel execution, not separate agents.

Things That Will Bite You

The working code is on GitHub if you want to see the full implementation.

Before you try to productionize this, watch out for these issues:

Context Window Limits
Each agent passes its full output to the next. A 4K research output becomes a 6K draft, then an editor prompt that includes both. Cumulative growth adds up fast. Most local models have 32K-128K context windows, so you won't hit the hard limit immediately, but you pay for every token in latency and cost. Track your per-stage token counts or your pipeline will balloon unnecessarily.

Go Concurrency Traps
The parallel research example above shares the same llms.Model across goroutines. Most LLM providers rate-limit by API key, not by connection. If you spawn 10 parallel researchers against Ollama Cloud, you'll hit quota errors. Add a semaphore or pool your model instances.

Streaming and Timeouts
The ExecuteWithStream pattern looks clean but basically complicates error handling. If the LLM connection drops mid-stream, you get a partial response that looks like success. Check your response length against expected ranges, or validate the output has a proper ending marker before declaring victory.

Start with a 2-agent pipeline. Add the editor only when you see specific failure modes it would fix. Measure latency at each step. Fix the slowest agent first. That's basically it.