DEV Community: M Hossein

I switched on production evals for my LLM app — and they scored nothing

M Hossein — Tue, 23 Jun 2026 12:46:23 +0000

What data privacy taught me about online evals, and why I stopped treating LLM prompts like magic and started treating them like hostile user input.

The Context & The Constraint

I am building a meeting assistant that fact-checks claims in real time. Because it processes meeting audio in the EU, it is bound by strict data residency rules: personal data and transcripts cannot arbitrarily leave our European infrastructure.

This introduces a fundamental distributed systems headache when we introduce Online Evaluations.

Online evals run after you ship, on live traffic. There is no answer key. Instead, you score live outputs for qualitative properties: Is this answer actually grounded in its sources? Is the model hallucinating? To do this, an evaluator needs to look at the inputs and outputs.

I wired up tracing, wrote an LLM judge, enabled LangSmith’s online evaluator, and waited for the scores to roll in. Nothing came back. Not a failure, not an error—just an endless stream of empty dashboards. The reason is a trap that is trivial to fall into, and it requires rethinking how we handle telemetry at the edge.

The Naive Approach (The MVP)

The standard, happy-path implementation for online evals assumes a globally unified system where data flows freely. You pipe your inputs, outputs, and intermediate states to an observability platform, and a cloud-hosted LLM judge scores them asynchronously.

Because of EU data rules, I had aggressively masked my traces. I configured my LangSmith client to strip all inputs and outputs before they left my server. LangSmith kept the metadata (latency, tokens) and nothing else.

Great for privacy. Fatal for evaluations. The evaluator opened the trace, found an empty object, and scored nothing. It failed silently because it was doing exactly what I configured it to do. But fixing this by simply "turning off masking" wasn't a legal option, and running the judge synchronously in the application code is an operational death sentence.

The Architectural Evolution (Iterative Refinement)

Building a robust evaluation pipeline at the edge is not about just wiring APIs together; it is about respecting the physical limits of your compute environment and the failure modes of generative models.

Subsystem 1: Tracing Without Leaking

The Pitfall: Treating data masking as a boolean operation—send everything or send nothing. Sending nothing blinds your telemetry; sending everything violates data residency.
The Fix: The masking hook isn't a toggle; it’s a transformation function. We can store a derived, safe projection of the data.
The Hidden Pitfall: If you simply hash the text, you lose all semantic value for your downstream judge. If you extract entities, you risk accidentally leaking PII inside those entities.
The Definitive Fix: We emit structurally safe telemetry. No transcripts, no raw claims. We emit hashes for correlation, array lengths for shape validation, and enums for state.

new Client({
  hideInputs: () => ({}), 
  // Store a cryptographically safe, structural projection
  hideOutputs: (outputs) => (outputs?.cards ? projectForEval(outputs) : {}), 
});

function projectForEval(state) {
  return {
    verdicts: state.cards.map((c) => ({
      verdict: c.verdict,                 // Strict enum (SUPPORTED, CONTRADICTED)
      claimHash: sha256(c.claim),         // Correlation without exposure
      sourceDomains: c.sources.map(host), // FQDNs only, no paths
      evidenceLen: c.evidence.length,     // Shape validation
    })),
  };
}

This allows our online evaluators to run cheap, deterministic checks (e.g., Did every card cite a valid domain?) without exposing a single sensitive byte.

Subsystem 2: The LLM Judge & Edge Constraints

The Pitfall: To check if evidence is actually faithful to a transcript, an LLM judge must see the text. To keep the text in the EU, I initially ran the judge synchronously inside my Cloudflare Worker: await judge(card, sources);. This instantly triggered CPU and wall-clock timeouts. Cloudflare Workers are built for fast I/O, not blocking for 4 seconds while an LLM grades homework.
The Fix: Decouple the evaluation from the critical path using Cloudflare's ctx.waitUntil(), allowing the worker to return the user's response immediately while the judge runs in the background.
The Hidden Pitfall: The Poison Pill. LLMs are non-deterministic. If your background judge hallucinates malformed JSON or markdown backticks, const { score } = JSON.parse(llmOutput) will throw a runtime exception. Because this is happening in waitUntil(), the error is swallowed, and your telemetry pipeline silently drops the trace.
The Definitive Fix: Shift to an asynchronous queue with strict parsing and Dead Letter Queues (DLQ). The edge worker drops the task onto a Cloudflare Queue. A separate background consumer processes the LLM judgment with defensive validation (e.g., Zod). If the LLM returns garbage, it fails the parse and is routed to a DLQ, preserving the pipeline's integrity.

// Edge Worker: Fire and forget. Never block the user.
export default {
  async fetch(req, env, ctx) {
    const response = await handleMeeting(req);

    // Offload evaluation to a background queue
    await env.EVAL_QUEUE.send({
      runId: currentRunId,
      card: response.card,
      sources: response.sources
    });

    return response;
  }
}

// Queue Consumer: Defensive parsing.
export default {
  async queue(batch, env) {
    for (const msg of batch.messages) {
      try {
         const rawLLMOutput = await runLocalEUJudge(msg.body.card, msg.body.sources);

         // DEFENSIVE: Never trust LLM output structure
         const parsed = EvalSchema.safeParse(rawLLMOutput);
         if (!parsed.success) {
             await env.DLQ.send({ error: parsed.error, raw: rawLLMOutput });
             continue; 
         }

         // DEFENSIVE: Do not block main edge traffic on third-party API limits
         await env.LANGSMITH.createFeedback(msg.body.runId, { 
             key: "faithfulness", 
             score: parsed.data.score 
         });
      } catch (err) {
         console.error("Eval pipeline failed", err);
      }
    }
  }
}

Subsystem 3: The Prompt Versioning Illusion

The Pitfall: The rubric that dictates whether a claim is "supported" or "contradicted" lives in the prompt. Prompts are usually treated as disposable strings. LangSmith's Prompt Hub offers a neat solution: a UI to edit prompts and a :production label to pull them dynamically.
The Fix: Fetch the :production prompt at runtime so the application always uses the latest logic.
The Hidden Pitfall: Fetching a prompt mid-request on a stateless edge node is a blocking network call on the hot path. It adds 100ms+ of latency to every interaction and creates a single point of failure. If the Hub goes down, your app goes down.
The Definitive Fix: Treat prompts strictly as immutable source code. You cannot rely on a third-party UI state to dictate edge execution reality. CI/CD pulls the specific, version-pinned prompt at build time, writes it to a constant, and bundles it.

"One-click rollback" via a UI label is a dangerous illusion in distributed systems. If the prompt is baked into the build, changing the label does nothing until the CDN finishes propagating the new deployment. Architecture must respect the physical reality of the deployment pipeline.

The Takeaway

Whiteboard architectures assume networks are perfectly reliable, external APIs never degrade, and LLMs always return beautifully formatted JSON. Production environments laugh at these assumptions.

Online evaluations are not free. They cost compute, they cost latency, and they introduce entirely new failure domains into your infrastructure. Building an LLM app requires epistemic humility—accepting that the model will fail, the judge will hallucinate, and the network will stall.

By pushing heavy evaluations to asynchronous queues, defensively parsing every output like it's hostile user input, and binding prompts to immutable build artifacts, we turn a fragile "happy path" demo into a hardened, mechanically sound system. It’s boring plumbing, but boring plumbing is the only thing that survives production.

Stop Shipping Bloat: 7 Steps to 15x Faster, 90% Smaller Docker Builds

M Hossein — Thu, 11 Jun 2026 06:01:00 +0000

We’ve all seen it: a simple service wrapped in a Docker image that tips the scales at 1.2GB and drags down CI/CD pipelines with a 4-minute build time.

It’s a massive waste of storage, bandwidth, and engineering time. It doesn't have to be this way. With a few intentional tweaks to your Dockerfile, you can drop that image down to under 80MB and cut build times to less than 20 seconds.

Here is a practical checklist to optimize your container builds, moving from a bloated, slow configuration to a lean, production-ready image.

1. Leverage Multi-Stage Builds

Your compiler, development dependencies, and local build caches have no business being in your production environment. Multi-stage builds allow you to install tools and compile your application in an initial heavy stage, then copy only the compiled artifacts into a completely fresh, minimal final runtime stage.

2. Pick a Smaller Base & Pin Your Versions

Using a generic tag like node:20 pulls in a massive Debian image (~1.1GB) packed with build utilities you rarely need in production. Moving to node:20-slim (~240MB) or node:20-alpine (~135MB) dramatically shrinks your baseline.

Additionally, never use floating tags like latest or 20-alpine. If the underlying image updates overnight, your builds can break with zero code changes. Always pin explicit versions for both the language runtime and the OS base.

3. Order Layers by Invalidation Frequency

Docker caches layers sequentially. If a layer changes, every subsequent layer is invalidated and forced to rebuild from scratch.

Since your source code changes on every single commit, but your package dependencies don't, you should copy your dependency manifests and install packages before introducing your actual application code. This is the single biggest win for CI/CD performance.

The Antipattern vs. The Optimized Pattern

Let's look at a typical, inefficient Dockerfile versus an optimized version incorporating these rules.

The Bloated Approach (What to Avoid)

This single-stage approach copies everything at once, runs as root, uses a massive floating base image, and busts the cache on every minor code tweak.

# 1.1GB Floating Base
FROM node:20

# Everything is copied early - any code change busts the cache for dependency installs
WORKDIR /app
COPY . .

# Massive node_modules and build tools stay in the final image
RUN npm install
RUN npm run build

# Runs as root by default
EXPOSE 3000
CMD ["node", "dist/main.js"]

The Optimized Approach (Production-Ready)

Here is the exact same application rewritten using multi-stage builds, an Alpine base, pinned versions, correct layer ordering, non-root execution, and chained commands.

# ==========================================
# STAGE 1: Build & Compilation
# ==========================================
FROM node:20.11.1-alpine3.19 AS builder

WORKDIR /app

# Copy dependency files first to utilize cached layers
COPY package*.json ./

# Install all dependencies (including devDeps for build)
RUN npm ci

# Copy the rest of the application source code
COPY . .

# Build the production artifact
RUN npm run build

# ==========================================
# STAGE 2: Lightweight Production Runtime
# ==========================================
FROM node:20.11.1-alpine3.19 AS runner

WORKDIR /app

# Set production environment flags
ENV NODE_ENV=production

# Copy only the necessary runtime configuration and compiled artifacts
COPY package*.json ./

# Chained RUN: Install production-only dependencies and clean npm cache in one layer
RUN npm ci --only=production && npm cache clean --force

# Copy compiled JavaScript from the builder stage
COPY --from=builder /app/dist ./dist

# Run as a non-privileged system user instead of root
USER node

EXPOSE 3000

CMD ["node", "dist/main.js"]

4. Don't Skip the `.dockerignore`

If you don't explicitly ignore files, your local build context copies everything in your directory directly into the Docker daemon. This means your massive local node_modules, heavy .git history, temporary logs, and local environment variables are sent to the build context, needlessly breaking your cache.

Spend 10 seconds creating a .dockerignore file in your root directory:

node_modules
.git
.github
.env
*.log
npm-debug.log*
dist
build
.coverage

5. Chain Your `RUN` Commands

Every single RUN, COPY, and ADD instruction in a Dockerfile creates a new read-only layer. If you install an OS package on one line and delete its cache on the next line, that deleted data is still stored in the underlying layer history.

To actually shrink your image size, you must install, utilize, and clean up within a single chained RUN instruction using &&:

# DO NOT DO THIS: The apt cache lives forever in layer history
RUN apt-get update
RUN apt-get install -y curl
RUN rm -rf /var/lib/apt/lists/*

# DO THIS: Cleanup happens in the exact same layer mutation
RUN apt-get update && \
    apt-get install -y curl && \
    rm -rf /var/lib/apt/lists/*

Summary Checklist

Strategy	Action	Primary Benefit
Multi-Stage Builds	Separate compiling from running via `AS builder`.	Drops image size by omitting build-only tooling.
Slim Pinned Bases	Use tags like `node:20.11.1-alpine3.19`.	Drops baseline footprint; prevents floating breakages.
Smart Layer Ordering	Copy manifests and run installs before copying source.	Keeps expensive installation steps cached in CI/CD.
Add `.dockerignore`	Strip out local tooling, hidden files, and `node_modules`.	Speeds up build context initialization; protects cache.
Chain Commands	Combine commands and cleanups (`&&`) in single `RUN` lines.	Minimizes unnecessary filesystem layer overhead.
Drop Privileges	Switch context via `USER node` or an equivalent non-root UID.	Mitigates severe container-escape security vulnerabilities.

A Note on ML and Heavy Pipelines: While the examples above target Node.js, these exact structural patterns apply identically to Python or Go. Poor layer ordering and careless copying of dataset files or massive build-time dependencies (like heavy C++ bindings or CUDA build-essential wrappers) are often the silent killers of CI/CD performance in machine learning and data pipelines. Fix your layer sequences, isolate your dependencies, and keep your containers lean.

Multi-VPS Distributed n8n Cluster on Ubuntu 24.04 with an IPIP Tunnel

M Hossein — Wed, 10 Jun 2026 06:07:00 +0000

When running automation tools like n8n for personal or production workflows, you quickly run into resource walls if you stick to the default configuration. A standalone SQLite setup can experience performance drops under load, and heavy processing inside JavaScript/Python nodes can easily stall the primary web service or exhaust single-server resources.

In this guide, we will step through how to architecture a decentralized, enterprise-grade n8n cluster split across two separate Ubuntu 24.04 VPS instances using the open-source community edition.

We will completely isolate the execution engine from the control plane using a lightweight IPIP (IP-in-IP) tunnel, proxying incoming traffic via Nginx, and managing tasks asynchronously through a PostgreSQL 16 and Redis 7 backend backbone.

The Target Architecture

Instead of over-engineering the infrastructure with complex orchestrators, we split our environment into two lightweight planes over a private network connection:

VPS 1: The Control Plane (Main Node) – Handles the web UI, external webhook endpoints via Nginx SSL, the PostgreSQL database, and the Redis message broker.
VPS 2: The Execution Plane (Worker Node) – A stateless compute node dedicated purely to listening to the Redis queue, crunching complex data, and reporting back via the private tunnel network.

Step 1: Base Server Prep & Official Docker Installation

We begin by prepping both Ubuntu 24.04 LTS machines with baseline utilities, Python dependencies, and the official Docker engine repository.

Run these commands on both servers:

# Update base system packages
sudo apt update && sudo apt upgrade -y

# Install essential dependencies
sudo apt install -y curl wget unzip git vim build-essential net-tools

# Set up Python runtimes
sudo apt install -y python3 python3-pip python3-dev

Installing Docker using the Modern Apt Sources Method

Ubuntu 24.04 drops legacy apt-key procedures. We install the official engine using the modern .sources system:

# Add Docker's official GPG key
sudo apt update
sudo apt install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Register the Docker repository configuration using DEB822 format
sudo tee /etc/apt/sources.list.d/docker.sources <<EOF
Types: deb
URIs: https://download.docker.com/linux/ubuntu
Suites: \$(. /etc/os-release && echo "\${UBUNTU_CODENAME:-\$VERSION_CODENAME}")
Components: stable
Architectures: \$(dpkg --print-architecture)
Signed-By: /etc/apt/keyrings/docker.asc
EOF

# Install the engine components and the compose plugin
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Step 2: Provisioning a Persistent Private IPIP Tunnel

To safely expose PostgreSQL and Redis across our servers without leaving open holes to the public internet, we build a private point-to-point network using an kernel-level IPIP tunnel.

We want VPS 1 (Main) to live on private IP 10.0.0.1 and VPS 2 (Worker) on 10.0.0.2.

Create the following script named setup-tunnel.sh on both nodes:

#!/bin/bash
# Usage: sudo ./setup-tunnel.sh <MAIN_PUBLIC_IP> <WORKER_PUBLIC_IP> --install

if [ "$EUID" -ne 0 ]; then
  echo "❌ Please run as root (sudo)."
  exit 1
fi

MAIN_IP=$1
WORKER_IP=$2
INSTALL_PERSISTENCE=false

if [ "$3" == "--install" ]; then
  INSTALL_PERSISTENCE=true
fi

configure_tunnel() {
  echo "⚙️ Initializing IPIP Tunnel Configuration..."
  modprobe ipip

  if ip link show tun0 >/dev/null 2>&1; then
    ip link set tun0 down
    ip tunnel del tun0
  fi

  if ip addr | grep -q "$MAIN_IP"; then
    echo "🖥️  Configuring: MAIN VPS"
    ip tunnel add tun0 mode ipip remote "$WORKER_IP" local "$MAIN_IP" ttl 255
    ip addr add 10.0.0.1/30 dev tun0
    ip link set tun0 up
    echo "✅ Tunnel 'tun0' is UP. Local private IP: 10.0.0.1"
  elif ip addr | grep -q "$WORKER_IP"; then
    echo "👷 Configuring: WORKER VPS"
    ip tunnel add tun0 mode ipip remote "$MAIN_IP" local "$WORKER_IP" ttl 255
    ip addr add 10.0.0.2/30 dev tun0
    ip link set tun0 up
    echo "✅ Tunnel 'tun0' is UP. Local private IP: 10.0.0.2"
  else
    echo "❌ Error: Public IP does not match local interfaces."
    exit 1
  fi
}

configure_tunnel

if [ "$INSTALL_PERSISTENCE" = true ]; then
  SCRIPT_PATH="/usr/local/bin/ipip-tunnel-setup.sh"
  cp "$0" "$SCRIPT_PATH"
  chmod +x "$SCRIPT_PATH"

  cat <<EOF > /etc/systemd/system/ipip-tunnel.service
[Unit]
Description=Maintain Persistent IPIP Tunnel between n8n Nodes
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=$SCRIPT_PATH $MAIN_IP $WORKER_IP

[Install]
WantedBy=multi-user.target
EOF

  systemctl daemon-reload
  systemctl enable ipip-tunnel.service
  echo "🌟 Systemd persistence initialized."
fi

Execute this script on both servers using your public IPs to establish the link permanently:

chmod +x setup-tunnel.sh
sudo ./setup-tunnel.sh <MAIN_PUBLIC_IP> <WORKER_PUBLIC_IP> --install

Verify the interface link by runnning a quick check from your main node: ping -c 3 10.0.0.2.

Step 3: Deploying the Control Plane (VPS 1)

With our private tunnel channel open, we configure the orchestration layer on the Main server. Create a directory at /home/n8n/ to hold your configurations.

1. The Environment Setup (`/home/n8n/.env`)

N8N_HOST=n8n.your-domain.com
N8N_PORT=5678

GENERIC_TIMEZONE=Europe/Vienna
N8N_ENCRYPTION_KEY=generate_a_random_long_string_here

POSTGRES_USER=n8n_admin
POSTGRES_PASSWORD=choose_a_secure_db_password_here
POSTGRES_DB=n8n_storage

REDIS_PASSWORD=choose_a_secure_redis_password_here

N8N_DIAGNOSTICS_ENABLED=false
N8N_PERSONALIZATION_ENABLED=false

2. The Infrastructure Deployment (`/home/n8n/docker-compose.yml`)

Note that modern Docker Compose specs omit the obsolete version string attribute. We explicitly map database and queue services strictly onto the secure tunnel interface IP (10.0.0.1), ensuring zero external exposures.

services:
  postgres:
    image: postgres:16-alpine
    container_name: n8n_postgres
    restart: unless-stopped
    environment:
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_DB: ${POSTGRES_DB}
    volumes:
      - postgres_data:/var/lib/postgresql/data
    ports:
      - "10.0.0.1:5432:5432"
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER} -d ${POSTGRES_DB}"]
      interval: 5s
      timeout: 5s
      retries: 5
    deploy:
      resources:
        limits:
          memory: 1G

  redis:
    image: redis:7-alpine
    container_name: n8n_redis
    restart: unless-stopped
    command: redis-server --requirepass ${REDIS_PASSWORD}
    ports:
      - "10.0.0.1:6379:6379"
    volumes:
      - redis_data:/data
    healthcheck:
      test: ["CMD", "redis-cli", "-a", "${REDIS_PASSWORD}", "ping"]
      interval: 5s
      timeout: 5s
      retries: 5

  n8n_main:
    image: docker.n8n.io/n8nio/n8n:latest
    container_name: n8n_core
    restart: unless-stopped
    ports:
      - "127.0.0.1:${N8N_PORT}:5678"
    environment:
      - EXECUTIONS_MODE=queue
      - QUEUE_BULL_REDIS_HOST=redis
      - QUEUE_BULL_REDIS_PORT=6379
      - QUEUE_BULL_REDIS_PASSWORD=${REDIS_PASSWORD}
      - DB_TYPE=postgresdb
      - DB_POSTGRESDB_HOST=postgres
      - DB_POSTGRESDB_PORT=5432
      - DB_POSTGRESDB_DATABASE=${POSTGRES_DB}
      - DB_POSTGRESDB_USER=${POSTGRES_USER}
      - DB_POSTGRESDB_PASSWORD=${POSTGRES_PASSWORD}
      - N8N_ENCRYPTION_KEY=${N8N_ENCRYPTION_KEY}
      - GENERIC_TIMEZONE=${GENERIC_TIMEZONE}
      - TZ=${GENERIC_TIMEZONE}
      - N8N_HOST=${N8N_HOST}
      - N8N_PROTOCOL=https
      - WEBHOOK_URL=https://${N8N_HOST}/
      - EXECUTIONS_DATA_PRUNE=true
      - EXECUTIONS_DATA_MAX_AGE=168
      - OFFLOAD_MANUAL_EXECUTIONS_TO_WORKERS=true
      - N8N_DIAGNOSTICS_ENABLED=false
      - N8N_PERSONALIZATION_ENABLED=false
    volumes:
      - n8n_data:/home/node/.n8n
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy

volumes:
  postgres_data:
  redis_data:
  n8n_data:

Launch the core stack on VPS 1:

docker compose up -d

Step 4: Configure Nginx Reverse Proxy with WebSockets

n8n uses persistent WebSocket connections to update progress inside your browser canvas UI. If Nginx proxy buffering isn't optimized, your UI will become unreadably laggy or constantly drop connections.

Create an Nginx server block at /etc/nginx/sites-available/n8n:

server {
    listen 80;
    listen [::]:80;
    server_name n8n.your-domain.com;

    client_max_body_size 50M;

    location / {
        proxy_pass http://127.0.0.1:5678;

        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # WebSocket support configurations
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

        # Disable buffering to avoid streaming timeouts
        proxy_buffering off;
        proxy_read_timeout 24h;
        proxy_send_timeout 24h;
    }
}

Enable the configuration block and use Certbot to register a Let's Encrypt SSL certificate automatically:

sudo ln -s /etc/nginx/sites-available/n8n /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx

# Run Certbot to handle automatic HTTP -> HTTPS redirects
sudo certbot --nginx -d n8n.your-domain.com

Step 5: Deploying the Execution Worker (VPS 2)

The execution worker node is completely stateless, meaning it needs no persistent storage databases. It only requires a minimal Compose setup pointing back across the tunnel to VPS 1.

1. Create `/home/n8n/.env` on VPS 2

Copy your environment details from VPS 1 to ensure that credential cryptographic keys and database definitions match exactly:

GENERIC_TIMEZONE=Europe/Vienna
N8N_ENCRYPTION_KEY=6uwhZTfJM80Hysuow5ge27r8xLtkFNWq
POSTGRES_USER=n8n_admin
POSTGRES_PASSWORD=WfVKtYp37bMDT6daPaTDSUYucBhfUw32
POSTGRES_DB=n8n_storage
REDIS_PASSWORD=8sHkQV3lpEc0yD4emUSn0oka9PrpdJ1h

2. Create `/home/n8n/docker-compose.yml` on VPS 2

We append command: worker to shift the container's operational blueprint into an isolated worker state:

services:
  n8n_worker:
    image: docker.n8n.io/n8nio/n8n:latest
    container_name: n8n_worker_remote
    command: worker
    restart: unless-stopped
    environment:
      - EXECUTIONS_MODE=queue
      - QUEUE_BULL_REDIS_HOST=10.0.0.1
      - QUEUE_BULL_REDIS_PORT=6379
      - QUEUE_BULL_REDIS_PASSWORD=${REDIS_PASSWORD}
      - DB_TYPE=postgresdb
      - DB_POSTGRESDB_HOST=10.0.0.1
      - DB_POSTGRESDB_PORT=5432
      - DB_POSTGRESDB_DATABASE=${POSTGRES_DB}
      - DB_POSTGRESDB_USER=${POSTGRES_USER}
      - DB_POSTGRESDB_PASSWORD=${POSTGRES_PASSWORD}
      - N8N_ENCRYPTION_KEY=${N8N_ENCRYPTION_KEY}
      - GENERIC_TIMEZONE=${GENERIC_TIMEZONE}
      - TZ=${GENERIC_TIMEZONE}
    deploy:
      resources:
        limits:
          memory: 2G

Launch the worker node:

docker compose up -d

Step 6: Server Hardening & Security Policies

⚠️ The Golden Rule of SSH Configuration: Always test your new configurations in an entirely separate terminal window before closing your current active shell connection so you don't accidentally lock yourself out.

1. Restrict Network Interfaces via UFW

Configure standard server firewalls to lock out random external pings.

On VPS 1 (Main Node):

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw allow 22/tcp  # Change if using a custom SSH port
sudo ufw allow in on tun0
sudo ufw enable

On VPS 2 (Worker Node):

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow 22/tcp
sudo ufw allow in on tun0
sudo ufw enable

2. Prevent Docker from Bypassing Your Firewall

Docker manipulates system iptables directly. To guarantee it respects your interface limits, explicitly verify your Docker routing daemon defaults:

sudo nano /etc/docker/daemon.json

Ensure "iptables": true is explicitly present, then apply a restart using sudo systemctl restart docker.

3. Polish the SSH Configuration

Enforce cryptographic authentication over standard passwords:

sudo nano /etc/ssh/sshd_config

Update these properties to ensure safety:

PermitEmptyPasswords no
PermitRootLogin prohibit-password
PasswordAuthentication no
MaxAuthTries 3

Test using sudo sshd -t and trigger a refresh via sudo systemctl restart ssh.

Verification & Monitoring

Because the multi-node worker management interface is part of n8n's enterprise plan, you can easily verify that your cluster is healthy from the command line using your Redis backend.

Execute a client list inspection inside your Redis service on VPS 1:

docker exec -it n8n_redis redis-cli -a <YOUR_REDIS_PASSWORD> client list

Look for the worker's network mapping signature in the connection stream output:

id=51 addr=10.0.0.2:35510 ... name=bull:am9icw== ... cmd=brpoplpush

The presence of the brpoplpush command originating from your tunnel gateway IP 10.0.0.2 means your stateless execution worker is successfully listening to your queue broker. Your secure, distributed n8n cluster is now fully configured and running smoothly.

Stop Guessing, Start Profiling: A Dev's Guide to Go Mechanics

M Hossein — Tue, 09 Jun 2026 11:40:00 +0000

The Problem: The Mysterious 2-Second Freeze

Imagine your Go microservice is a chef in a busy kitchen. It processes orders (JSON payloads) super fast. Life is good, until every 15 minutes, the chef completely freezes for 2 seconds. Orders pile up.

Why did the chef freeze? The Trash.

When you parse JSON using the standard encoding/json library, you create a lot of garbage (heap allocations). Go has a Garbage Collector (GC) that acts like a janitor.

⚠️ The Myth: Many engineers think the GC yells "Stop-The-World," making the chef freeze while the janitor sweeps. Modern Go has a concurrent GC. Real STW pauses are sub-millisecond.

✅ The Reality: Mark Assist.
If the chef makes trash faster than the janitor can sweep it, the Go runtime triggers Mark Assist. The runtime literally hands the chef a broom and forces them to sweep instead of cooking. The chef isn't frozen; the chef is doing janitor work.

📊 Diagram 1: The Mark Assist Traffic Jam

Normal:  Chef Cooks ➔ Makes Trash ➔ Janitor sweeps concurrently ➔ Fast! (20ms)

Spike:   Chef Cooks ➔ Makes TOO MUCH Trash ➔ Janitor falls behind
                |
         💥 RUNTIME: "Chef, drop the spatula, grab a broom!" (Mark Assist)
         💥 Chef spends 2 seconds sweeping instead of cooking 💥

The Right Way: Mechanical Sympathy

We need to stop making trash so the chef never has to sweep. Here are the 3 big mechanical fixes you need to know.

Fix 1: Stop Making Trash (Size-Classed Pools)

The standard encoding/json library creates trash because it uses "reflection."

The Fix: Use fast parsers (like easyjson) and sync.Pool. A sync.Pool is like a shared toolbox. You take a wrench out, use it, wipe it clean, and put it back. No trash!

⚠️ Trap 1: Megamorphic Bloat
If you put a 4KB wrench in the toolbox, but someone stretches it to 60KB and puts it back, your toolbox permanently bloats.

⚠️ Trap 2: The Black Hole & Allocation Roulette
To fix bloat, you might create separate toolboxes for Small and Large jobs. But what happens to a 4KB wrench that stretches to 5KB?

If you put it in the 64K pool, you play Allocation Roulette: The next chef asks for a 60K wrench and gets your 5K one. It snaps, triggering a heap allocation.
If you use exact equality checks (if cap == 4096), you create a Black Hole: A third-party library slices your buffer down to 3KB. It's still perfectly capable of fulfilling a 2KB request, but your code throws it in the trash. Your pool empties, forcing heap allocations.

✅ The Solution: Utility Ranges (The Half-Broken Tool)
Toolboxes need a "good enough" range. If a 4KB wrench gets filed down to 3KB, it’s still pretty useful. Put it back. But if it gets filed down to 500 bytes, it's useless for a 4KB job. Let the Garbage Collector throw it away.

var pool4K  = sync.Pool{New: func() any { b := make([]byte, 0, 4096); return &b }}
var pool64K = sync.Pool{New: func() any { b := make([]byte, 0, 65536); return &b }}

func getBuffer(size int) *[]byte {
    if size <= 4096  { return pool4K.Get().(*[]byte) }
    if size <= 65536 { return pool64K.Get().(*[]byte) }
    return nil // Poison pill: Too big, don't pool.
}

func putBuffer(buf *[]byte) {
    *buf = (*buf)[:0] // Wipe the wrench clean!
    c := cap(*buf)

    // The 4K Toolbox: Keep it if it's between 2KB and 4KB
    if c >= 2048 && c <= 4096 {
        pool4K.Put(buf) 
        return
    }
    // The 64K Toolbox: Keep it if it's between 32KB and 64KB
    if c >= 32768 && c <= 65536 {
        pool64K.Put(buf)
        return
    }

    // If it falls in the dead-zones (like 5KB), it doesn't fit either box. 
    // Let the Garbage Collector eat it.
}

(Note: We pool pointers `[]byte because passing a slice []byte to an interface{}` creates a heap wrapper—defeating the whole purpose!)*

Fix 2: Protect the Database (The Cancellation Storm)

When your app gets hit by Mark Assist, database queries slow down. You add timeouts (e.g., context.WithTimeout(ctx, 500ms)).

⚠️ The Myth: A context timeout "violently drops" the database connection.
✅ The Reality: When a context expires, database/sql drivers (like Postgres) must send an Out-Of-Band (OOB) cancellation request. To do this, the driver opens a brand new TCP connection just to say "cancel my previous request!"

📊 Diagram 2: The Self-Inflicted DDoS

1. APP GETS MARK ASSIST (Slows down)
   App ➔ 500ms Timeout hits! Query must be cancelled.

2. THE CANCEL STORM:
   App ➔ Opens NEW TCP Connection ➔ "Hey DB, Cancel Query 1"
   App ➔ Opens NEW TCP Connection ➔ "Hey DB, Cancel Query 2"
   ... (5,000 new TLS handshakes attack the Database CPU) 💥

✅ The Solution: Put a centralized proxy (like PgBouncer) in the middle. The proxy absorbs the violent cancellation storms from your app, but keeps a calm, steady set of warm connections open to the database. (Don't use a sidecar per pod—200 sidecars still attack the DB!)

Fix 3: Don't Break the Chain (Failing Closed)

If a user sends a 50MB "Poison Pill" JSON, the standard answer is: "Send it to a Dead Letter Queue (DLQ) and keep processing."

⚠️ The Trap: This destroys ordered message queues (like Kafka CDC). If you skip "Step 1: Create Account" because it was a Poison Pill, and process "Step 2: Update Account", your database is corrupted.

✅ The Solution: Entity Quarantine
Halt processing for just that specific user ID, put that ID in a quarantine list (like Redis), and skip future messages for that user until a human fixes it.

⚠️ The Distributed State Fallacy (The Radio Check): What happens when the quarantine list (Redis) goes down?
You might think: "Just make the app sleep or return an error until Redis comes back!"
This is a fatal mistake. Kafka uses a "Radio Check" (a heartbeat). If your app goes completely silent because it's sleeping or frozen, the Kafka broker assumes your app died. It violently kicks your app out of the group and gives the broken messages to another pod... which also freezes and dies. You will crash your entire cluster in a Rebalance Storm.

✅ The Definitive Fix: Put them on Hold
When dealing with strict order, you cannot guess. If Redis is down, you must Pause the queue, but keep the radio heartbeat alive.

Is entity in Quarantine?
 ➔ Yes: Skip message.
 ➔ No: Process message.
 ➔ Redis Down: Tell Kafka to "Pause()" this specific partition. 
               Keep polling the radio to say "I'm alive, just on hold." 
               Alert humans.

Fix 4: See the Ghost (Continuous Profiling)

You can't fix what you can't see.

CI Guardrails: Add go test -benchmem to your pipeline. If a PR adds memory allocations on the critical path, it fails.
Continuous Profiling: Use tools like Parca or Pyroscope to constantly take pprof snapshots in production, and link them to your traces (OpenTelemetry).

💡 The Pro Move: When an alert fires, you click the exact trace that was slow, and it shows you the exact line of code that forced your app into Mark Assist. No guessing required.

Summary

Mark Assist, not STW: The GC forces your app to help clean. Stop making trash.
Utility Ranges: A 3KB buffer is still a useful half-broken tool for a 2KB job. Use ranges, not exact equality.
Timeouts cause Storms: Context cancellations open new TCP connections. Use a centralized proxy to absorb the DDoS.
Keep the Radio Check: If your quarantine check goes down, pause the partition but keep polling to prevent a cluster crash.

When you understand how the machine actually works—Mark Assist, utility ranges, OOB cancellations, and heartbeats—you stop guessing and start engineering systems that survive production.

The Blast Radius: Mentoring Senior Engineers into Systems Thinkers

M Hossein — Mon, 08 Jun 2026 05:42:00 +0000

The transition from Senior Engineer to Staff Engineer isn't just about writing more complex code. It is about expanding your field of vision. A Senior Engineer owns a service; a Staff Engineer owns the spaces between the services.

But how do you teach that? How do you take an incredibly talented coder who operates in a silo and turn them into a "Systems Thinker"?

Let’s look at a classic engineering management scenario, the standard naive reaction, and the Staff-level approach to building a resilient engineering culture.

The Scenario: The Siloed Senior Engineer

Let's say you have a Senior Engineer on one of your product squads named David. David is fantastic at writing Go. He is the technical owner of the Receipt PDF Generation service. His code is spotless, his test coverage is 90%+, and he ships features on time.

Last week, David pushed an optimization. He realized the PDF generator was making sequential database calls, so he rewrote the logic to aggressively parallelize the data fetching using Goroutines. He made the service 30% faster. He was thrilled.

The Consequence: He didn't realize that by aggressively parallelizing the fetches, he effectively DDOS'd a shared internal database with concurrent reads. He exhausted the connection pool, which temporarily degraded the performance of a critical, Tier-1 compliance API that shares the same database.

He simply didn't think about his downstream blast radius.

Phase 1: The Standard Management Reaction

When incidents like this happen, the standard reaction in immature engineering organizations is to treat it as a behavioral problem:

The Scolding: A manager pulls David aside, tells him to "be more careful," and demands he test better.
The Vague Mandate: David is told to "act more like a consultant" and "talk to other teams" to get a better image of what's going on.

The Pitfall: The Culture of Fear & Friction
Scolding an engineer for an optimization creates a culture of fear. The next time David sees a way to improve the system, he will keep his head down to avoid getting yelled at. Furthermore, telling a squad engineer to vaguely "act as a consultant" creates massive friction with their Product Manager, who expects them to be burning down Jira tickets, not wandering around Slack asking other teams what they are doing.

Worst of all, this reaction completely ignores the architectural failure.

The Staff-Level Insight: Human error is never the root cause; it is merely the symptom of a fragile system. If a single engineer optimizing a PDF generator can bring down a Tier-1 API, your architecture is broken.

Phase 2: The Force Multiplier Framework

As a Staff Engineer, your job is to use this incident as a Force Multiplier. You need to fix the architecture and mentor the engineer.

But first, you must reframe the failure for David before any public process begins. In your 1-on-1, you don't scold him. You validate his intent (making things faster) but challenge his context. You tell him: "The optimization was great engineering; the lack of defensive consumption was a systems failure. Let's fix the system together."

From there, here is the exact, actionable 4-step strategy to turn David from a Service Owner into a Systems Thinker over the next six months.

1. The Blameless Post-Mortem (Reframing the Incident)

You do not initiate a post-mortem to ask, "Why did David break the database?" You initiate a blameless post-mortem with David and the affected teams to ask a purely mechanical question:

"Why did our architecture allow a single service to DDOS a shared database without rate limits or circuit breakers tripping?" This immediately shifts the focus from human guilt to system resilience.

2. Vulnerability as Leadership (The Tribe Meeting)

We want David to present his work at the monthly engineering all-hands, but not just to show off his cool parallelization code. That sends the message: "Look at my optimization, ignore the outage."

Instead, you ask David to present the entire incident. He walks through his optimization, explains the unexpected downstream database locks, and shares the system-level fixes. When a highly respected Senior Engineer stands up in front of the tribe and says, "I brought down the API because I didn't think about the DB locks, and here is what I learned," it creates massive psychological safety. It teaches the entire organization that it is okay to take risks, fail, and learn.

3. Structured Cross-Pollination (The RFC Process)

You cannot just tell an engineer to "go see what other teams are doing." You must bake it into the engineering lifecycle—and you must protect their time to do it.

Instead of a vague consultant role, bring David into the formal RFC (Request for Comments) / Design Doc process. For the next three months, make David a mandatory reviewer on architecture design docs for the squads immediately upstream and downstream from him.

This forces him to read about other systems before they are built. He learns to spot API contract changes, database coupling, and capacity limits outside of his Go routines.

Crucially, as the Staff Engineer, you must act as an umbrella. When David’s PM complains that he is reviewing RFCs instead of burning down the sprint backlog, you step in. You translate the systems work into product terms: "David isn't doing admin work; he's preventing the next P1 outage that will wipe out our sprint velocity for a week." This shields David from organizational friction and proves that systems thinking is valued over ticket-churning.

4. Operational Empathy ("You Build It, You Run It")

Writing code in a silo is easy. Operating it in production is hard.

If your developers are isolated from the operational consequences of their code, they will never become Systems Thinkers. You must advocate for putting Senior Developers on the PagerDuty incident rotation alongside your SREs.

Nothing builds system-level empathy faster than waking up at 3:00 AM because another team's runaway retry-loop just exhausted the connections on your database. Shared operational pain is the ultimate catalyst for distributed systems thinking.

The Payoff: Six Months Later

If you execute this strategy, you don't just fix an architectural flaw; you fundamentally alter how an engineer views the world.

Six months from now, a product manager will ask David to add a new feature to the PDF service that requires fetching user data from a newly acquired third-party API.

The old David would have just written the integration, wrapped it in a Goroutine, and shipped it.

The new David—the Systems Thinker—will pause. He'll check the API's rate limits. He'll add a semaphore to bound his concurrency, and a circuit breaker to fail fast. He'll ping the upstream squad to ensure his new polling pattern won't overflow their message queue. He will look beyond his service and own the space between the services.

The Takeaway

Mentoring isn't just scheduling 1-on-1s and giving code review feedback. It is about intentionally designing the environment around the engineer.

By leveraging blameless post-mortems, encouraging vulnerable leadership, injecting engineers into cross-team RFCs (while shielding them from PM friction), and exposing them to production reality through PagerDuty, you stop fighting human nature. Instead, you build an engineering culture where "Systems Thinking" is simply the path of least resistance.

Architecting Strict Sequential Ordering in a Concurrent World

M Hossein — Sun, 07 Jun 2026 06:42:00 +0000

Imagine you are building a cloud-native backend for a high-frequency trading platform or a core banking ledger. To ensure mathematical immutability and prevent silent data tampering, compliance mandates that every transaction for a specific financial account must be cryptographically chained.

This means the signature of Transaction #50 must explicitly include the cryptographic hash of Transaction #49. You cannot sign them out of order, and the backend is strictly responsible for generating and validating this chain.

This introduces a massive distributed systems headache: How do you enforce strict, sequential ordering while maintaining the concurrency required to scale a modern cloud architecture? Let's walk through the evolution of this system, deconstruct exactly how the standard event-driven approach fails in production, and examine the Staff-level architecture required to fix it.

Phase 1: The MVP & The Database Bottleneck

In the early days, traffic is low. An account might see one transaction every few minutes. The "Happy Path" is simple:

The API receives a deposit request for Account A.
The API queries Postgres for the last_signature_hash.
The API computes the new hash in memory: SHA(last_hash + new_transaction_data).
The API writes the new transaction and updates the state.

The Pitfall: The Thundering Herd
To prevent two concurrent requests from reading the same previous hash, you wrap the database operation in a pessimistic lock: SELECT ... FOR UPDATE. This forces the database to serialize requests at the row level.
When a massive partner bank initiates a bulk sync, dumping 5,000 transactions for a single corporate account onto the API in two seconds, 4,999 concurrent threads immediately hit the FOR UPDATE lock and block. The database connection pool is instantly exhausted, latency spikes platform-wide, and the MVP dies.

The Insight: The database must be your last line of defense, not your primary queueing mechanism. Contention must be solved upstream in memory.

Phase 2: The Event-Driven Reality (Step-by-Step Fixes)

To protect the database, we introduce Kafka and Golang. We place a ledger-events Kafka topic between the API and the database. By using account_id as the Kafka message key, Kafka routes all traffic for a specific account to a single partition, ensuring it is processed by exactly one Go worker pod.

This looks great on a whiteboard. But under the microscope of production reality, it is riddled with architectural gaps. Here is how we systematically uncover and solve them.

Step 1: The Ingestion Illusion & Operational Memory

The Pitfall: The Overdraft Race Condition
Keying messages by account_id does not guarantee chronological order; Kafka only guarantees the order in which the broker receives the messages. If a client issues a $50 deposit and a $120 withdrawal in the same millisecond, network latency might cause the withdrawal to hit Kafka first. If the backend accepts Kafka's order as absolute, the account overdrafts and the withdrawal is incorrectly rejected.

The Fix: Client-Dictated Sequence & In-Memory Buffering
Chronological order in finance is about business logic, not just hashing. The client must provide a sequence ID (e.g., seq_1, seq_2). The Go worker enforces this sequence. If Kafka delivers seq_3 before seq_2, the Goroutine cannot process it. It must buffer seq_3 in memory and wait for seq_2 to arrive.

The Hidden Pitfall: The Auto-Commit Catastrophe
By buffering out-of-order messages in memory, we introduce a severe operational risk. If Kafka is set to auto-commit offsets, it will tell the broker "I have successfully processed up to seq_3" simply because it read it off the partition. If the pod crashes while seq_3 is sitting in the memory buffer waiting for seq_2, seq_3 is permanently lost.

The Final Fix: You must disable auto-commit. The Go worker must implement meticulous manual offset management, committing Kafka offsets only after the expected sequence is successfully flushed to the database ledger. Furthermore, to prevent memory leaks from permanently stalled sequences (e.g., a client bug where seq_2 is never sent), the in-memory buffer must have a strict TTL. If the gap isn't filled within 60 seconds, the chain halts and triggers an SRE alert.

Step 2: The Database Constraint & The Infinite Ledger

The Pitfall: The "Missing Tail" and the UNIQUE Loophole
To process a transaction, the worker needs the last hash. Querying the ledger directly (SELECT ... ORDER BY created_at DESC LIMIT 1) creates an $O(N)$ index scan bottleneck on high-volume accounts. If we fix this by keeping an account_state table as an $O(1)$ cache, we face a new problem: If a pod crashes and a Kafka rebalance occurs, two workers might read the cached state concurrently and attempt to build off the same hash.

The Fix: The Dual-Table Pattern & Fork Prevention
We use two tables updated in a single atomic transaction. account_state acts as our high-speed cache, and ledger acts as our immutable history. We add a safety net to the ledger:

ALTER TABLE ledger ADD CONSTRAINT unique_chain_link UNIQUE (account_id, previous_hash);

Nuance: This UNIQUE constraint prevents forks (two transactions claiming the same parent), but it doesn't guarantee continuity (ensuring the new hash actually links to the true tail). Continuity is guaranteed by Optimistic Concurrency Control (OCC) in Go. The worker reads the $O(1)$ cache, computes the hash, and attempts an INSERT into the ledger. If a rogue worker also read the stale state, the UNIQUE constraint causes the second INSERT to fail instantly—before touching the account_state cache. The Go worker catches this error, fetches the fresh cache state, computes a new hash, and retries cleanly.

The Hidden Pitfall: The Index Bloat
An immutable ledger table will grow infinitely. The UNIQUE (account_id, previous_hash) constraint requires a B-Tree index. On a high-frequency ledger with billions of rows, this index swells beyond available RAM, causing INSERT performance to degrade exponentially due to disk I/O.

The Final Fix: Implement Postgres Table Partitioning. Partition the ledger by date (e.g., ledger_2026_06) or account hash range. This keeps the active index sizes small and strictly in memory, preserving sub-millisecond insert times.

Step 3: Concurrency Anti-Patterns

The Pitfall: The Goroutine Memory Leak
To isolate processing per account, the Go worker uses dynamic Goroutines fed by a sync.Map. To clean up idle Goroutines, engineers often use a select loop with a time.After(15 * time.Minute) timeout. time.After allocates a new timer channel on the heap every single iteration of the loop, causing massive Garbage Collection (GC) pressure. Furthermore, sync.Map degrades under high-frequency writes.

The Fix: RWMutex & The Timer Reset Pattern
Use a standard Go map protected by a sync.RWMutex. Inside the Goroutine, instantiate exactly one timer and defer its stop, resetting it on every loop.

import "sync"

// Dispatcher safely manages the dynamic worker channels for each account.
type Dispatcher struct {
    mu       sync.RWMutex
    channels map[string]chan Transaction
}

func NewDispatcher() *Dispatcher {
    return &Dispatcher{
        channels: make(map[string]chan Transaction),
    }
}

func processAccountChain(accountID string, ch <-chan Transaction, dispatcher *Dispatcher) {
    // Idiomatic Go: create ONE timer, clean up on exit
    idleTimer := time.NewTimer(15 * time.Minute)
    defer idleTimer.Stop() 

    for {
        idleTimer.Reset(15 * time.Minute) // Reuse the same object

        select {
        case tx, ok := <-ch:
            if !ok { return }
            // Process transaction sequentially...

        case <-idleTimer.C:
            // Lock map, delete channel, safely exit
            dispatcher.mu.Lock()
            delete(dispatcher.channels, accountID)
            dispatcher.mu.Unlock()
            return 
        }
    }
}

Step 4: The Edge Case Reality

The Pitfall: The Poison Pill under Strict Sequence
Transaction seq_2 in a batch of 500 contains a corrupted, unparseable JSON payload. Because we have firmly established that the client dictates the sequence, the backend cannot simply drop seq_2 and chain seq_3 to seq_1. Doing so legally invalidates the client's intended sequence and corrupts the financial state.

The Fix: The State-Backed DLQ & Chain Halt
You cannot fix a broken link in a strict sequence, nor can you skip it. The Go worker must:

Route the bad payload to a Dead Letter Queue (DLQ).
Update the account_state table in Postgres: UPDATE account_state SET status = 'BLOCKED'.
Drop all subsequent messages (like seq_3 and seq_4) into a holding topic or reject them directly at the API layer.

The chain halts immediately. An SRE or automated reconciliation process must investigate, notify the client of the specific sequence failure, and force the client to re-issue the transactions from the exact point of failure.

The Takeaway

When business logic requires strict, sequential cryptographic chaining, high-concurrency event-driven architectures naturally fight against the requirement.

You cannot solve this by throwing more threads at the problem or locking down your database rows. You must sequence deterministically via client IDs, buffer appropriately at the ingestion layer, manage Kafka offsets manually, isolate processing via memory dispatchers, and leverage your partitioned database as a high-speed cache backed by an immutable, constraint-driven ledger. Architecture at this level is about anticipating the mechanical realities of the system, not just the happy path.

Polyglot Monorepo Magic: TypeScript, Python, and Go in One Repo

M Hossein — Sat, 06 Jun 2026 07:22:00 +0000

What is a Polyglot Monorepo?

A polyglot monorepo is a single Git repository containing services and packages written in multiple programming languages. The alternative — a polyrepo — means one repo per service, per team, per language. When your frontend team, your Go API team, and your Python ML team all operate in separate repos, sharing contracts between them becomes a coordination problem.

This repo is beacon-monorepo. It's a real-time analytics platform. It contains:

beacon-monorepo/
├── frontend/
│   └── web/           ← Next.js dashboard (TypeScript)
├── services/
│   ├── api/           ← REST API (Go)
│   ├── ingest/        ← Data ingestion service (Go)
│   ├── ml/            ← ML inference API (Python / FastAPI)
│   └── worker/        ← Background jobs (Python / Celery)
├── packages/
│   ├── ui/            ← Shared React components (TypeScript)
│   └── sdk/           ← TypeScript client SDK
├── pkg/
│   ├── shared/        ← Go shared utilities
│   └── store/         ← Go data access layer
├── libs/
│   └── shared/        ← Python shared models + Pydantic schemas
├── proto/             ← Protobuf definitions (source of truth)
├── gen/
│   ├── go/            ← Generated Go stubs
│   ├── python/        ← Generated Python stubs
│   └── ts/            ← Generated TypeScript stubs
├── Taskfile.yml       ← Root task orchestration (cross-language)
├── pnpm-workspace.yaml
├── package.json       ← Root (biome, lefthook, turbo)
├── pyproject.toml     ← uv workspace root
├── uv.lock
├── biome.json
├── .golangci.yml
└── go.work            ← (gitignored — local dev only)

The core idea: services are owned by different teams writing different languages, but they share contracts defined in proto/, config defined at the repo root, and a single task runner (Taskfile.yml) that orchestrates everything with one set of commands.

Tool 1: Three Workspace Managers — The Parallel Foundation

There is no single package manager that handles TypeScript, Python, and Go. Instead, three workspace managers coexist in the same repo — each governing its own language, each completely ignorant of the others.

Language	Manager	Config file	Dep linking mechanism
TypeScript	pnpm workspaces	`pnpm-workspace.yaml`	`workspace:*` symlinks
Python	uv workspaces	root `pyproject.toml`	`{ workspace = true }` editable installs
Go	go workspaces	`go.work`	local module overlay

They coexist peacefully because they key on different file extensions. pnpm reads package.json. uv reads pyproject.toml. Go reads go.mod. A directory like services/ml/ that contains a pyproject.toml is invisible to pnpm. A directory with only a package.json is invisible to uv.

TypeScript: `pnpm-workspace.yaml`

packages:
  - 'frontend/*'
  - 'packages/*'
  - 'gen/ts'          # generated TypeScript stubs — must be a workspace member

gen/ts/ must be a workspace member, not a file: dependency. A file: reference copies the package into node_modules at install time — after buf generate regenerates the stubs, the copies are stale until you re-run pnpm install. A workspace:* reference is a symlink — always live.

packages/sdk/package.json depends on the shared UI and the generated TypeScript stubs:

{
  "name": "@beacon/sdk",
  "dependencies": {
    "@beacon/ui": "workspace:*",
    "@beacon/proto-ts": "workspace:*"   // symlinked — reflects buf generate immediately
  }
}

Python: root `pyproject.toml`

[project]
name = "beacon-root"
version = "0.1.0"
requires-python = ">=3.12"

[tool.uv.workspace]
# List Python members explicitly — globs must not match dirs without pyproject.toml
members = [
    "services/ml",
    "services/worker",
    "libs/shared",
    "gen/python",      # generated proto stubs — must be listed so uv can resolve them
]

[tool.uv.sources]
shared = { workspace = true }   # the workspace:* equivalent for Python

services/ml/pyproject.toml declares its internal dep:

[project]
name = "ml"
dependencies = [
    "shared",           # internal — resolves via workspace
    "fastapi>=0.115",
    "torch>=2.3",
]

[tool.uv.sources]
shared = { workspace = true }

Why explicit members, not globs? uv errors if a glob matches a directory without a pyproject.toml. frontend/web/ and packages/ui/ have package.json but no pyproject.toml — a glob like packages/* would break uv. Be explicit.

Go: `go.work`

go 1.23

use (
    ./services/api
    ./services/ingest
    ./pkg/shared
    ./pkg/store
    ./gen/go          ← generated proto stubs (own go.mod)
)

Critical: add go.work and go.work.sum to .gitignore. CI must validate each Go module against its own go.mod independently. The workspace is a local dev convenience — not a CI mechanism.

`go.mod` plumbing — how CI works without `go.work`

This is the part most polyglot monorepo guides skip. When CI runs GOWORK=off, each Go module must declare its in-repo dependencies in its own go.mod using replace directives — otherwise the module can't find gen/go or pkg/shared.

services/api/go.mod:

module github.com/your-org/beacon/services/api

go 1.23

require (
    github.com/your-org/beacon/gen/go    v0.0.0
    github.com/your-org/beacon/pkg/shared v0.0.0
    // external deps...
)

// replace directives make GOWORK=off CI work:
// each module explicitly maps its in-repo deps to disk paths
replace (
    github.com/your-org/beacon/gen/go     => ../../gen/go
    github.com/your-org/beacon/pkg/shared => ../../pkg/shared
)

go work sync propagates the require versions from each go.mod. The replace directives work with or without go.work — so GOWORK=off CI and go.work local dev both resolve to the same local disk paths.

Run go mod tidy in each module after adding the replace directives. The v0.0.0 version is a placeholder that go mod tidy fills in as a pseudo-version.

# .gitignore
go.work
go.work.sum
.task/
.venv/
node_modules/
.turbo/
.next/
__pycache__/
*.pyc
services/*/bin/

What `uv sync` and `pnpm install` and `go work sync` each do

pnpm install          # links TypeScript packages via symlinks in node_modules
uv sync --group dev   # installs all Python workspace members as editable installs
go work init          # creates go.work on a fresh clone (only needed once — it's gitignored)
go work use ./services/api ./services/ingest ./pkg/shared ./pkg/store ./gen/go
go work sync          # syncs go.mod files across Go modules (run after adding new deps)

Fresh clone problem: go.work is gitignored, so a fresh clone has no workspace file. The go work init + go work use steps only run once per machine. The Go Taskfile handles this automatically.

The key insight: these three commands are completely independent. Running pnpm install does not affect Python deps. Running uv sync does not touch node_modules. None of them touch each other. The three workspace managers are parallel foundations — they don't compose, they coexist.

Tool 2: Task (Taskfile.yml) — The Cross-Language Orchestrator

Three workspace managers handle dependencies. Task handles tasks (build, test, lint, generate, dev). It's the single entry point for everything in the repo, regardless of language.

Task is a language-agnostic binary. Install it once:

brew install go-task
# or: sh -c "$(curl --location https://taskfile.dev/install.sh)" -- -d -b ~/.local/bin

The root `Taskfile.yml`

version: '3'

includes:
  ts:
    taskfile: ./Taskfile.ts.yml
    optional: true
  py:
    taskfile: ./Taskfile.py.yml
    optional: true
  go:
    taskfile: ./Taskfile.go.yml
    optional: true
  proto:
    taskfile: ./Taskfile.proto.yml
    optional: true

tasks:
  doctor:
    desc: Verify all required tools are installed
    cmds:
      - command -v node  || (echo "node not found — https://nodejs.org"  && exit 1)
      - command -v pnpm  || (echo "pnpm not found — npm install -g pnpm" && exit 1)
      - command -v uv    || (echo "uv not found — https://docs.astral.sh/uv" && exit 1)
      - command -v go    || (echo "go not found — https://go.dev/dl" && exit 1)
      - command -v buf   || (echo "buf not found — https://buf.build/docs/cli" && exit 1)
      - command -v jq    || (echo "jq not found — brew install jq" && exit 1)
      - echo "All tools present"

  setup:
    desc: Install all dependencies (run task doctor first)
    deps: [doctor]
    cmds:
      - pnpm install
      - uv sync --group dev   # single .venv at repo root; --group dev includes pytest/mypy/ruff
      - task: go:init         # creates go.work if missing (idempotent)
      - go work sync
      - task: proto:generate

  build:all:
    desc: Build everything in dependency order
    cmds:
      - task: proto:generate       # proto first — all languages depend on it
      - task: ts:build
      - task: go:build
      - task: py:build

  test:all:
    desc: Test all languages in parallel
    deps:
      - ts:test
      - go:test
      - py:test

  lint:all:
    desc: Lint all languages in parallel
    deps:
      - ts:lint
      - go:lint
      - py:lint

  dev:
    desc: Start all dev servers
    deps:
      - ts:dev
      - go:dev
      - py:dev

Language-specific Taskfiles

Taskfile.ts.yml — TypeScript:

version: '3'

tasks:
  build:
    sources:
      - 'frontend/**/*.ts'
      - 'frontend/**/*.tsx'
      - 'packages/**/*.ts'
      - '**/package.json'     # dep changes should bust the cache
      - 'pnpm-lock.yaml'
    generates: ['frontend/web/.next/**']
    cmds:
      - pnpm turbo run build

  lint:
    sources: ['frontend/**/*.ts', 'packages/**/*.ts', '**/package.json']
    generates: ['.task/ts-lint.stamp']
    cmds:
      - pnpm biome check .
      - touch .task/ts-lint.stamp

  test:
    sources:
      - 'frontend/**/*.ts'
      - 'packages/**/*.ts'
      - 'frontend/**/*.test.ts'
      - 'pnpm-lock.yaml'
    cmds:
      - pnpm turbo run test

  dev:
    cmds:
      - pnpm turbo run dev --filter=@beacon/web

Taskfile.go.yml — Go:

version: '3'

# Prerequisite: jq must be installed (brew install jq / apt install jq)

tasks:
  init:
    desc: Create go.work for local development (gitignored; run once per clone)
    status:
      - test -f go.work    # skip if go.work already exists
    cmds:
      - go work init
      - go work use ./services/api ./services/ingest ./pkg/shared ./pkg/store ./gen/go

  build:
    deps: [init]           # ensures go.work exists before building
    sources:
      - 'services/api/**/*.go'
      - 'services/ingest/**/*.go'
      - 'pkg/**/*.go'
      - '**/go.mod'        # dep changes should bust the cache
      - '**/go.sum'
    generates: ['services/api/bin/api', 'services/ingest/bin/ingest']
    cmds:
      - cd services/api && go build -o bin/api ./cmd/api
      - cd services/ingest && go build -o bin/ingest ./cmd/ingest

  lint:
    deps: [init]
    sources: ['services/**/*.go', 'pkg/**/*.go']
    generates: ['.task/go-lint.stamp']
    cmds:
      - go work edit -json | jq -r '.Use[].DiskPath' |
          xargs -P4 -I{} sh -c 'cd {} && golangci-lint run ./...'
      - touch .task/go-lint.stamp

  test:
    deps: [init]
    sources: ['services/**/*.go', 'pkg/**/*.go']
    cmds:
      - go work edit -json | jq -r '.Use[].DiskPath' |
          xargs -P4 -I{} sh -c 'cd {} && go test ./... -race'

  dev:
    deps: [init]
    cmds:
      - cd services/api && go run ./cmd/api

Taskfile.py.yml — Python:

version: '3'

tasks:
  build:
    sources:
      - 'services/ml/**/*.py'
      - 'services/worker/**/*.py'
      - 'libs/**/*.py'
      - '**/pyproject.toml'  # dep changes should bust the cache
      - 'uv.lock'
    generates: ['.task/py-build.stamp']
    cmds:
      - uv sync --group dev
      - touch .task/py-build.stamp

  lint:
    sources: ['services/ml/**/*.py', 'services/worker/**/*.py', 'libs/**/*.py', 'uv.lock']
    generates: ['.task/py-lint.stamp']
    cmds:
      - uv run ruff check services/ml services/worker libs
      - uv run ruff format --check services/ml services/worker libs
      - uv run mypy services/ml/ services/worker/ libs/   # from repo root — not per-service
      - touch .task/py-lint.stamp

  test:
    sources: ['services/ml/**/*.py', 'services/worker/**/*.py', 'libs/**/*.py']
    cmds:
      - uv run pytest services/ml services/worker libs -x

  dev:
    cmds:
      - uv run fastapi dev services/ml/src/ml/main.py

Key concepts:

deps: runs tasks in parallel — test:all runs all three test suites simultaneously
cmds: runs sequentially — build:all runs proto first, then the rest
sources/generates is Task's caching: if source files haven't changed since the last run (hash stored in .task/), the task is skipped entirely
Add .task/ to .gitignore
task build:all requires all three toolchains installed (node + pnpm, uv, go, buf, jq). Frontend-only or backend-only developers who run it without the full stack will get a "task not found" error. Document this in your repo README — or split task build:ts, task build:go, task build:py as the normal developer interface.

CI: affected-only runs

Task has no built-in --affected. In CI, detect changed language areas with git diff:

# ci/affected.sh
changed=$(git diff --name-only origin/main...HEAD)

echo "$changed" | grep -qE '^(frontend|packages)/' && task ts:test
echo "$changed" | grep -qE '^(services/(api|ingest)|pkg)/' && task go:test
echo "$changed" | grep -qE '^(services/(ml|worker)|libs)/' && task py:test
echo "$changed" | grep -qE '^proto/' && {
  task proto:generate
  task ts:test && task go:test && task py:test  # proto change affects all
}

A PR touching only services/ml/ won't re-run Go tests.

Tool 3: buf + Protobuf — The Contract Layer

This is the tool that makes a polyglot monorepo genuinely worth the complexity. A single .proto file becomes the source of truth for how your TypeScript frontend, your Go API, and your Python ML service all communicate. Change the proto once — regenerate — all three languages are in sync.

`buf.yaml` — workspace config at repo root

# buf.yaml
version: v2

modules:
  - path: proto/analytics
    name: buf.build/your-org/analytics

  - path: proto/events
    name: buf.build/your-org/events

deps:
  - buf.build/googleapis/googleapis

lint:
  use: [STANDARD]
  except: [PACKAGE_DIRECTORY_MATCH]

breaking:
  use: [FILE]

`buf.gen.yaml` — one command, three languages

# buf.gen.yaml
version: v2

inputs:
  - directory: proto/analytics
  - directory: proto/events

plugins:
  # Go: message stubs
  - remote: buf.build/protocolbuffers/go
    out: gen/go
    opt: paths=source_relative

  # Go: gRPC service stubs
  - remote: buf.build/grpc/go
    out: gen/go
    opt: paths=source_relative

  # Python: message stubs
  - remote: buf.build/protocolbuffers/python
    out: gen/python

  # Python: gRPC service stubs
  - remote: buf.build/grpc/python
    out: gen/python

  # Python: mypy type stubs (community plugin)
  - remote: buf.build/community/nipunn1313-mypy
    out: gen/python

  # TypeScript: Protobuf-ES (modern, works with Connect-RPC)
  - remote: buf.build/bufbuild/es
    out: gen/ts
    opt: target=ts

managed:
  enabled: true
  override:
    - file_option: go_package_prefix
      value: github.com/your-org/beacon/gen/go

Run buf generate to regenerate all three at once.

Generated code as first-class modules

Each generated directory is its own module with its own manifest — so the three workspace managers can find it:

gen/
├── go/
│   └── go.mod          ← module github.com/your-org/beacon/gen/go
│                          each consumer's go.mod has: replace ... => ../../gen/go
├── python/
│   └── pyproject.toml  ← listed in uv workspace members; consumers use { workspace = true }
└── ts/
    └── package.json    ← listed in pnpm-workspace.yaml; consumers use workspace:*

All three use the same pattern as other internal packages — workspace linking, not file: copies. gen/go/ is also listed in go.work for local dev, and each consumer's go.mod has a replace directive for CI.

# Taskfile.proto.yml
tasks:
  generate:
    sources:
      - proto/**/*.proto
    generates:
      - gen/go/**/*.go
      - gen/python/**/*_pb2.py
      - gen/ts/**/*_pb.ts
    cmds:
      - buf lint
      - buf generate
      - buf breaking --against '.git#branch=main'

Why this matters: the breaking change check (buf breaking) is the safety net. If a Go developer renames a proto field, buf breaking fails the build before the Python and TypeScript consumers break. The contract is enforced at the definition layer, not the consumer layer.

Tool 4: Per-Language Root Config — Three Files, One Repo

Each language has one config file at the repo root. No duplication across services, no "why is lint disabled in this one service" surprises.

`biome.json` — TypeScript

{
  "$schema": "https://biomejs.dev/schemas/1.9.0/schema.json",
  "linter": {
    "rules": {
      "correctness": {
        "noUnusedVariables": "error",
        "noUnusedImports": "error"
      },
      "style": { "useConst": "error" }
    }
  },
  "formatter": {
    "indentStyle": "space",
    "indentWidth": 2,
    "lineWidth": 100
  },
  "files": {
    "ignore": ["gen/ts/**", "**/node_modules/**", "**/.next/**"]
  }
}

root `pyproject.toml` — Python lint + type config

[tool.ruff]
line-length = 88
target-version = "py312"
src = ["src"]
exclude = ["gen/python"]

[tool.ruff.lint]
select = ["E", "W", "F", "I", "B", "UP"]
ignore = ["E501"]
fixable = ["ALL"]

[tool.ruff.lint.isort]
known-first-party = ["shared", "ml", "worker"]

[tool.mypy]
python_version = "3.12"
strict = true
mypy_path = "$MYPY_CONFIG_FILE_DIR/libs/shared/src"
exclude = ["gen/python"]

[[tool.mypy.overrides]]
module = ["*_pb2", "*_pb2_grpc"]
ignore_errors = true   # suppress errors in generated proto stubs

[dependency-groups]
dev = [
    "pytest>=8",
    "pytest-asyncio>=0.25",
    "mypy>=1.15",
    "ruff>=0.9",
]

Critical — mypy invocation: mypy reads config from the directory it's invoked from, not from the file being checked. Always run uv run mypy services/ml/ services/worker/ libs/ from the repo root. Running from inside services/ml/ silently ignores strict = true and falls back to defaults.

`.golangci.yml` — Go

version: "2"

run:
  timeout: 5m
  tests: true

linters:
  default: none
  enable:
    - errcheck
    - staticcheck
    - unused
    - ineffassign
    - govet
    - revive
    - gocritic
    - misspell
    - gosec
    - bodyclose
    - copyloopvar

  exclusions:
    generated: strict
    rules:
      - path: "gen/go/.*\\.pb\\.go$"
        linters: [revive, gocritic, godot, errcheck]
      - path: "_test\\.go$"
        linters: [gosec, errcheck]

formatters:
  enable:
    - gofumpt
    - goimports
  settings:
    goimports:
      local-prefixes:
        - github.com/your-org/beacon

`.editorconfig` — baseline cross-language consistency

Biome, Ruff, and golangci-lint enforce formatting within each language, but editors need .editorconfig for baseline consistency before any tool runs — charset, line endings, trailing newlines:

# .editorconfig
root = true

[*]
charset = utf-8
end_of_line = lf
insert_final_newline = true
trim_trailing_whitespace = true

[*.go]
indent_style = tab
indent_size = 4

[*.{ts,tsx,js,json,yaml,yml,toml}]
indent_style = space
indent_size = 2

[*.py]
indent_style = space
indent_size = 4

[*.proto]
indent_style = space
indent_size = 2

Why this matters: three config files, all at the repo root. Each tool finds its own by walking up the directory tree. You can change the Go linting rules for every Go service in one edit, the Python formatting rules for every Python service in one edit, and the TypeScript rules for every TypeScript package in one edit. No per-service drift.

Tool 5: Three Linters, One `task lint` Command

Each language uses its own best-in-class linter. Task unifies them:

task lint:all   # runs all three in parallel (deps: is parallel)
task ts:lint    # TypeScript only
task go:lint    # Go only
task py:lint    # Python only

Why not one linter for everything?

There is no cross-language linter that handles TypeScript + Python + Go well. The best tools are language-specific:

Biome — TypeScript: sub-millisecond, replaces ESLint + Prettier
Ruff — Python: 10–100x faster than Flake8, replaces isort + Black
golangci-lint — Go: aggregates 50+ linters behind one binary

Task makes them feel like one: task lint:all exits non-zero if any of them fails, regardless of language.

Build graph — how proto flows into all three

proto/analytics/*.proto
proto/events/*.proto
    ↓ buf generate
    ↓
gen/go/    gen/python/    gen/ts/
    ↓             ↓              ↓
pkg/shared   libs/shared    packages/sdk
    ↓             ↓              ↓
services/api  services/ml   frontend/web
services/ingest  services/worker

A change to proto/ forces all three build chains to run. Everything else is independent — a change to services/api/ doesn't rerun Python tests.

CI: three parallel language jobs

# .github/workflows/ci.yml
jobs:
  detect:
    runs-on: ubuntu-latest
    outputs:
      proto: ${{ steps.changes.outputs.proto }}
      ts: ${{ steps.changes.outputs.ts }}
      go: ${{ steps.changes.outputs.go }}
      py: ${{ steps.changes.outputs.py }}
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: dorny/paths-filter@v3
        id: changes
        with:
          filters: |
            proto:
              - 'proto/**'
            ts:
              - 'frontend/**'
              - 'packages/**'
              - 'gen/ts/**'
              - 'proto/**'   # proto change triggers TS too
            go:
              - 'services/api/**'
              - 'services/ingest/**'
              - 'pkg/**'
              - 'gen/go/**'
              - 'proto/**'
            py:
              - 'services/ml/**'
              - 'services/worker/**'
              - 'libs/**'
              - 'gen/python/**'
              - 'proto/**'

  proto:
    needs: detect
    if: needs.detect.outputs.proto == 'true'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: |
          npm install -g @bufbuild/buf
          buf lint && buf generate
          buf breaking --against '.git#branch=main'

  ts:
    needs: detect
    if: needs.detect.outputs.ts == 'true'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0   # required for turbo --affected
      - uses: pnpm/action-setup@v4
      - run: pnpm install --frozen-lockfile   # equivalent of uv sync --frozen
      - run: pnpm turbo run typecheck lint test --affected

  go:
    needs: detect
    if: needs.detect.outputs.go == 'true'
    runs-on: ubuntu-latest
    strategy:
      matrix:
        module: [services/api, services/ingest, pkg/shared, pkg/store]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-go@v5
        with:
          go-version-file: ${{ matrix.module }}/go.mod
          cache-dependency-path: ${{ matrix.module }}/go.sum
      # GOWORK=off is explicit — not relying on go.work being gitignored
      # This forces each module to prove it has complete go.mod declarations
      - run: cd ${{ matrix.module }} && GOWORK=off golangci-lint run ./...
      - run: cd ${{ matrix.module }} && GOWORK=off go test ./... -race

  py:
    needs: detect
    if: needs.detect.outputs.py == 'true'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v5
        with:
          enable-cache: true
      - run: uv sync --frozen --group dev
        # --frozen: fail if uv.lock is out of date (never silently update in CI)
        # --group dev: includes pytest, mypy, ruff
      - run: uv run ruff check services/ml/ services/worker/ libs/
      - run: uv run mypy services/ml/ services/worker/ libs/
        # always from repo root — mypy does not discover config per-file like Ruff
      - run: uv run pytest services/ml/ services/worker/ libs/ -x
      - run: uv cache prune --ci

A PR touching only services/ml/ skips the proto, ts, and go jobs entirely.

Tool 6: lefthook — One Hook to Rule All Three Languages

Each language article recommended its own hook tool (custom .githooks/ for TypeScript, pre-commit for Python, lefthook for Go). In a polyglot repo, you pick one. lefthook wins: it's a single binary, works with all languages, and runs hooks in parallel by default.

Install

lefthook is managed as a root-level dev dependency so everyone on the team gets it:

// package.json (root)
{
  "devDependencies": {
    "@biomejs/biome": "^1.9.0",
    "lefthook": "^1.11.0",
    "turbo": "^2.0.0"
  },
  "scripts": {
    "prepare": "lefthook install"
  }
}

pnpm install runs prepare automatically — lefthook hooks are wired up on first install.

`lefthook.yml`

# lefthook.yml
glob_matcher: doublestar  # ** matches 0 or more dirs (required for nested files)

pre-commit:
  parallel: true
  jobs:
    - name: ts-lint
      glob: "**/*.{ts,tsx}"
      run: pnpm biome check --no-errors-on-unmatched {staged_files}

    - name: py-lint
      glob: "**/*.py"
      run: uv run ruff check {staged_files}
      stage_fixed: true

    - name: py-format
      glob: "**/*.py"
      run: uv run ruff format {staged_files}
      stage_fixed: true

    - name: proto-lint
      glob: "**/*.proto"
      run: buf lint

pre-push:
  parallel: false
  jobs:
    - name: go-lint
      glob: "**/*.go"
      # golangci-lint cannot lint files across multiple packages (named files must all
      # be in one directory — see golangci-lint#3715). Run per-module instead:
      run: go work edit -json | jq -r '.Use[].DiskPath' |
             xargs -P4 -I{} sh -c 'cd {} && GOWORK=off golangci-lint run ./...'

    - name: go-mod-tidy
      glob: "go.{mod,sum}"
      # go mod tidy is per-module in a multi-module workspace; iterate over all modules:
      run: go work edit -json | jq -r '.Use[].DiskPath' |
             xargs -I{} sh -c 'cd {} && GOWORK=off go mod tidy && git diff --exit-code go.mod go.sum'

    - name: proto-breaking
      glob: "**/*.proto"
      # Guard against first push / no origin/main (new repos, shallow clones)
      run: |
        if git rev-parse origin/main >/dev/null 2>&1; then
          buf breaking --against '.git#branch=main'
        else
          echo "No origin/main — skipping breaking change check"
        fi

    - name: py-typecheck
      glob: "**/*.py"
      # mypy is 10-30s on real codebases — too slow for pre-commit, right for pre-push
      # pass whole dirs, not {staged_files}: mypy needs package-level context
      run: uv run mypy services/ml/ services/worker/ libs/ --config-file pyproject.toml

Key concepts:

glob_matcher: doublestar — without this, **/*.py won't match services/ml/src/ml/main.py (two directories deep). This single line makes nested file matching work correctly.
Each job is skipped entirely if no staged files match its glob — committing only TypeScript changes doesn't run go-lint at all.
stage_fixed: true — after Ruff auto-fixes a Python file, lefthook re-stages it so the fix is included in the commit.
parallel: true — all four language checkers run simultaneously. On a fast machine, a commit touching files in all three languages still completes in the time of the slowest single checker.
pre-push runs buf breaking — the breaking change check is too slow for pre-commit but important before pushing to a shared branch.

The key insight: you don't need different hook tools for different languages. lefthook routes by file glob. The Python and Go developers on your team never need to know that Biome exists; the TypeScript developers never need to know that Ruff exists. They all run pnpm install once and get the right checks for their files automatically.

The Full Picture: How These Tools Interact

Developer commits
    ↓
lefthook pre-commit (all four checks in parallel)
    ↓ Biome on staged .ts/.tsx files
    ↓ Ruff on staged .py files (+ auto-fix + re-stage)
    ↓ golangci-lint on staged .go files
    ↓ buf lint on staged .proto files
    ↓ (any failure → abort commit; all pass → continue)
    ↓
Three workspace managers resolve internal deps in parallel:
    ↓ pnpm: workspace:* symlinks for TypeScript
    ↓ uv: { workspace = true } editable installs for Python
    ↓ go work: local module overlay for Go (gitignored)
    ↓
task build:all
    ↓ proto:generate (buf → gen/go/ + gen/python/ + gen/ts/)
    ↓ ts:build (pnpm turbo — dep-ordered, cached in .turbo/)
    ↓ go:build (per-module, cached in .task/)
    ↓ py:build (uv sync, cached in .task/)
    ↓
CI: detect-changes → three parallel language jobs
    ↓ proto change → all three jobs + breaking check
    ↓ TypeScript: pnpm turbo --affected (fetch-depth: 0)
    ↓ Go: GOWORK=off → matrix per module → go test -race
    ↓ Python: uv run pytest (affected services)
    ↓ all jobs required → single ci-complete gate

Summary: Why This Stack

Problem	Tool	Mechanism
TypeScript dep linking	pnpm workspaces	`workspace:*` + symlinks
Python dep linking	uv workspaces	`{ workspace = true }` + editable installs
Go dep linking	go workspaces	`use` directives (gitignored)
Cross-language task orchestration	Task (Taskfile.yml)	`includes` + `sources`/`generates` caching
Cross-language contracts	buf + Protobuf	single `.proto` → 3 language stubs
TypeScript lint + format	Biome	root `biome.json`
Python lint + format	Ruff	root `pyproject.toml` `[tool.ruff]`
Go lint + format	golangci-lint + gofumpt	root `.golangci.yml`
Cross-language git hooks	lefthook	`glob_matcher: doublestar` + parallel jobs
CI: only run changed languages	dorny/paths-filter	per-language path filters + job conditions

The polyglot monorepo isn't one tool — it's a composition. Three workspace managers run in parallel. Task sits on top as the single command interface. buf provides the shared contract layer between all three languages. lefthook enforces quality at commit time regardless of which language a developer touches.

The complexity cost is real: you're maintaining three build toolchains instead of one. The payoff is equally real: one git blame, one pull request for a cross-language feature, one place to define the contract between your TypeScript frontend and your Go and Python backends.

Deploying a Python Service from the Polyglot Monorepo

Go compiles to a static binary — Docker is trivial. TypeScript builds to a dist/ directory — straightforward. Python is the tricky one: editable workspace installs point back at source paths and break inside a container without extra work.

The fix is uv sync --package ml --no-editable. The build context must be the repo root — not services/ml/ — because uv sync needs to resolve libs/shared, gen/python, and the root pyproject.toml.

Without a .dockerignore, Docker sends the entire monorepo as context including node_modules/, Go binaries, and the frontend build cache. Add this at the repo root:

# .dockerignore (at repo root — used by all Python service Dockerfiles)
frontend/
packages/
node_modules/
.turbo/
.task/
.next/
*.go
go.*
go.work
go.work.sum
.venv/

Then build from the repo root: docker build -f services/ml/Dockerfile .

# services/ml/Dockerfile
FROM python:3.12-slim AS builder
COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
WORKDIR /app

# Copy manifests first — dep install layer is cached separately from source
COPY pyproject.toml uv.lock ./
COPY libs/shared/pyproject.toml  libs/shared/pyproject.toml
COPY gen/python/pyproject.toml   gen/python/pyproject.toml
COPY services/ml/pyproject.toml  services/ml/pyproject.toml

RUN uv sync \
    --package ml \      # install only ml + its deps (libs/shared, gen/python, etc.)
    --no-dev \          # exclude pytest, mypy, ruff
    --frozen \          # fail if uv.lock is stale
    --no-editable       # copy source into site-packages — no source tree needed at runtime

# Copy source after dep install (changes here don't bust dep cache)
COPY libs/shared/src  libs/shared/src
COPY gen/python        gen/python
COPY services/ml/src  services/ml/src

FROM python:3.12-slim
COPY --from=builder /app/.venv /app/.venv
ENV PATH="/app/.venv/bin:$PATH"
CMD ["uvicorn", "ml.main:app", "--host", "0.0.0.0", "--port", "8080"]

--no-editable is the critical flag. Without it, libs/shared installs as an editable install pointing at libs/shared/src/ in the build context. The final image has a .venv with broken symlinks and imports fail at runtime. --no-editable copies the package source into .venv/lib/pythonX.Y/site-packages/ — self-contained.

What to Watch Out For

Fresh clone requires task go:init — go.work is gitignored; a fresh clone has no workspace file. Run task go:init once per clone (or once per machine). The status: guard makes it idempotent — it's safe to run every time.

gen/ts/ must be a workspace member, not file: — a file: reference copies the package into node_modules at install time. After buf generate, the copies are stale until pnpm install runs again. Use workspace:* (symlinked, always live).

Docker build context must be repo root — uv sync --package ml resolves workspace deps (libs/shared, gen/python). Build with docker build -f services/ml/Dockerfile . from the repo root. Use .dockerignore to exclude node_modules/, Go source, and the frontend build cache.

GOWORK=off in CI must be explicit — .gitignore prevents committing go.work but doesn't prevent git add -f. Set GOWORK=off in every CI Go step. Don't rely on the file not being present.

go.mod replace directives are required for CI — go.work resolves in-repo deps locally, but GOWORK=off requires each module's go.mod to have replace directives: replace github.com/your-org/beacon/gen/go => ../../gen/go. Without these, CI fails because the module can't find its in-repo dependencies. Run go mod tidy in each module after adding them.

golangci-lint cannot accept {staged_files} across packages — passing files from multiple packages produces "named files must all be in one directory." Run golangci-lint per-module in pre-push, not per-file in pre-commit.

pnpm and uv both need lock enforcement in CI — pnpm install --frozen-lockfile and uv sync --frozen are the equivalent safety checks. Omitting either means CI silently tests different dep versions than developers have locally.

sources lists must include lockfiles — Task caches by hashing declared sources. If uv.lock, pnpm-lock.yaml, or go.sum changes without any source file changing, Task serves a stale cached build. Add lockfiles to every sources list.

task dev log visibility — three servers writing to one terminal is unreadable. Run them in separate terminals or use a process manager (overmind, foreman, or pnpm turbo run dev for TypeScript). task ts:dev, task go:dev, task py:dev as separate commands is the pragmatic answer.

uv members and pnpm coexistence — uv will error if a members glob matches a directory that has no pyproject.toml. Be explicit: list Python-only directories rather than using broad globs like services/*.

mypy in pre-push, not pre-commit — mypy is 10–30s on a real Python codebase. Every commit, on any .py change. It belongs in pre-push alongside buf breaking, not blocking commits.

uv sync --frozen in CI — without --frozen, uv silently re-resolves if uv.lock is stale, meaning CI may test different dep versions than developers have locally.

--no-editable in Python Docker images — editable installs point at source paths in the build context. Use uv sync --package ml --no-editable in Dockerfiles so the .venv is self-contained.

jq is a prerequisite — Taskfile.go.yml uses go work edit -json | jq. Install with brew install jq or apt install jq. Add to your task prereqs check or README.

proto changes cascade + deployment order — buf breaking catches compile-time breakage, but not runtime version skew. In production, always deploy consumers (Python ML, TypeScript frontend) before producers (Go API) when adding fields. Never remove or rename fields in a single deployment — deprecate first, remove in a later release.

lefthook glob_matcher: doublestar — without this line, **/*.py won't match files more than one directory deep. Always verify with lefthook run pre-commit --all-files after first setup.

Generated code is committed — gen/go/, gen/python/, gen/ts/ are in git. This keeps CI simple and makes stub changes reviewable. Proto changes produce large diffs — that's the tradeoff.

Going Further

Dependency updates across three package managers — use Renovate (not Dependabot — Renovate handles pnpm, uv, and Go modules in a single config). For audits: pnpm audit, uv audit, govulncheck ./... per Go module.

Adding a new service — checklist:

Python service: add to pyproject.toml members, create pyproject.toml + Taskfile.yml + dir: include, add to lefthook.yml paths, add to CI py paths-filter
Go service: add to go.work use, add to Taskfile.go.yml init cmd + lint/test, add CI matrix entry
TypeScript package: add to pnpm-workspace.yaml, add Taskfile include

Cross-language integration tests — unit tests per language are not enough. Add a task test:integration that starts all three services (via Docker Compose or k3s) and runs contract tests against the live stack. Pact works across Go, Python, and TypeScript.

Environment config — one .env at repo root, read by all three languages:

TypeScript: dotenv or @t3-oss/env-nextjs
Python: pydantic-settings with env_file = ".env"
Go: godotenv.Load() or pass via Docker env

CODEOWNERS — proto/ should require review from both Go and Python leads. Use GitHub CODEOWNERS to enforce this on every proto PR.

Hot reload across proto changes — task dev doesn't re-run buf generate when .proto files change. Add a file-watcher task (e.g., via watchexec or task --watch) that runs buf generate and restarts affected dev servers on .proto changes.

Go Monorepo Magic: Organize, Build, and Ship Multi-Service Apps

M Hossein — Thu, 04 Jun 2026 07:49:00 +0000

What is a Monorepo?

A monorepo is a single Git repository that contains multiple distinct services or modules. The alternative is a polyrepo: one repo per service.

This repo is forge-monorepo. It contains:

forge-monorepo/
├── services/
│   ├── api/         ← HTTP API service
│   ├── worker/      ← Background job processor
│   └── notifier/    ← Notification service
├── pkg/
│   ├── shared/      ← Shared types + domain logic
│   ├── events/      ← Event definitions
│   └── proto/       ← Protobuf definitions
└── gen/
    └── go/          ← Generated proto stubs (own go.mod)

The core idea: code needed by multiple services lives in pkg/, and all services import it directly — no private module proxy, no vendoring ceremony.

Tool 1: Go Workspaces (`go work`) — The Foundation

Go workspaces are the monorepo primitive built into the Go toolchain since 1.18. They replace replace directives in go.mod for local multi-module development.

How it's declared

go.work at the repo root:

go 1.22

use (
    ./gen/go
    ./pkg/shared
    ./pkg/events
    ./services/api
    ./services/worker
    ./services/notifier
)

This tells the Go toolchain: treat every listed directory as a module, and resolve imports between them on disk. That's it — one file defines the entire workspace.

What this enables

When you run any go command inside the workspace, the toolchain:

Reads every go.mod in every listed directory
Resolves inter-module imports to local disk paths automatically
Lets every module keep its own go.mod and version independently

Internal imports

Each module has a normal go.mod with a full module path:

// pkg/shared/go.mod
module github.com/yourorg/forge/pkg/shared

go 1.22

Any other workspace module imports it by that exact path:

// services/api/main.go
import (
    "github.com/yourorg/forge/pkg/shared"
    "github.com/yourorg/forge/pkg/events"
)

No special syntax. The workspace resolves github.com/yourorg/forge/pkg/shared to ./pkg/shared on disk automatically.

Critical: do NOT commit `go.work`

go.work is a local development convenience. It is not a source of truth for module dependencies. Add both files to .gitignore:

# Go workspace (local dev only)
go.work
go.work.sum

# Task cache
.task/

CI validates each module independently with GOWORK=off — more on that in Tool 2.

Why `go work` over `replace` directives?

replace directives in go.mod are messy: they leak into the module graph, they break when you go get, and you have to touch every module's go.mod to wire things up. go work is additive — the workspace file lives outside the modules and doesn't pollute their dependency declarations.

Tool 2: Task (Taskfile.yml) — Task Orchestration and Caching

Go workspaces handle module resolution. Task handles build orchestration.

The problem Task solves

If you need to build 6 modules in the right order, you have to figure out the sequence yourself. And if nothing in pkg/shared changed, you shouldn't rebuild everything that depends on it.

Task solves both: explicit task dependency ordering + file-based output caching.

`Taskfile.yml` — the pipeline

version: '3'

includes:
  gen: ./gen/go/Taskfile.yml
  shared: ./pkg/shared/Taskfile.yml
  events: ./pkg/events/Taskfile.yml
  api: ./services/api/Taskfile.yml
  worker: ./services/worker/Taskfile.yml
  notifier: ./services/notifier/Taskfile.yml

tasks:
  build:all:
    desc: Build all modules in dependency order
    deps:
      - gen:build
    cmds:
      - task: shared:build
      - task: events:build
      - task: api:build
        async: true
      - task: worker:build
        async: true
      - task: notifier:build
        async: true

  test:all:
    desc: Test all modules
    cmds:
      - task: shared:test
      - task: events:test
      - task: api:test
      - task: worker:test
      - task: notifier:test

  lint:all:
    desc: Run golangci-lint across all modules
    cmds:
      - go work edit -json | jq -r '.Use[].DiskPath' | xargs -P4 -I{} sh -c 'cd {} && golangci-lint run ./...'

  tidy:all:
    desc: Run go mod tidy in every module
    cmds:
      - go work edit -json | jq -r '.Use[].DiskPath' | xargs -I{} sh -c 'cd {} && go mod tidy'

A per-module Taskfile (e.g., pkg/shared/Taskfile.yml) looks like:

version: '3'

tasks:
  build:
    sources:
      - '**/*.go'
    generates:
      - '.task/shared.built'
    cmds:
      - go build ./...
      - touch .task/shared.built

  test:
    sources:
      - '**/*.go'
    cmds:
      - go test ./...

Key concepts:

deps: — tasks listed here run in parallel before the current task starts
cmds: — commands run sequentially in order
sources / generates — Task hashes these paths. Cache hit → task skipped entirely
async: true — run the task concurrently with other commands in the same cmds: block

The build graph

gen/go (buf generate)
    ↓
pkg/shared (go build)
    ↓
pkg/events (go build)
    ↓
services/api     services/worker     services/notifier
  (go build)       (go build)           (go build)

Affected-only in CI

Task has no --affected flag. Implement it with a git diff script:

#!/usr/bin/env bash
# scripts/affected.sh
# Outputs a newline-separated list of changed module paths

BASE=${1:-origin/main}
CHANGED_FILES=$(git diff --name-only "$BASE"...HEAD)

declare -A AFFECTED

while IFS= read -r file; do
  # Match each workspace module by its directory prefix
  go work edit -json | jq -r '.Use[].DiskPath' | while read -r mod; do
    mod_clean="${mod#./}"
    if [[ "$file" == "$mod_clean"/* ]]; then
      echo "$mod_clean"
    fi
  done
done <<< "$CHANGED_FILES" | sort -u

In CI:

- name: Run affected tests
  run: |
    AFFECTED=$(bash scripts/affected.sh origin/main)
    for mod in $AFFECTED; do
      (cd "$mod" && GOWORK=off go test ./...)
    done

Tool 3: Shared Modules — The Internal Module Pattern

Each pkg/ module is a normal Go module with its own go.mod. It doesn't need to be published anywhere.

pkg/shared/
├── go.mod          ← module github.com/yourorg/forge/pkg/shared
├── types.go
├── domain.go
└── errors.go

Because pkg/shared is listed in go.work, any service imports it by its declared module path:

package main

import (
    "github.com/yourorg/forge/pkg/shared"
    "github.com/yourorg/forge/pkg/events"
)

func main() {
    user := shared.NewUser("alice")
    ev := events.NewUserCreated(user)
    _ = ev
}

The key insight: the workspace resolves that import to disk at development time. In CI, with GOWORK=off, each module must declare real require entries in its own go.mod. This forces you to be explicit about dependencies while keeping the local dev loop frictionless.

Proto and generated code

pkg/proto/ holds .proto files. gen/go/ holds the generated stubs as its own module:

// gen/go/go.mod
module github.com/yourorg/forge/gen/go

go 1.22

require google.golang.org/protobuf v1.34.1

Services import generated types the same way — no magic:

import forgepb "github.com/yourorg/forge/gen/go"

Tool 4: Root `.golangci.yml` — Shared Config

Centralize lint rules instead of duplicating them across every module.

.golangci.yml at repo root:

version: "2"

linters:
  enable:
    - errcheck
    - gosimple
    - govet
    - ineffassign
    - staticcheck
    - unused
    - gofumpt
    - goimports
    - revive
    - gosec
    - exhaustive
    - wrapcheck

linters-settings:
  gofumpt:
    extra-rules: true
  goimports:
    local-prefixes: github.com/yourorg/forge
  revive:
    rules:
      - name: exported
        severity: warning
      - name: error-return
        severity: error

issues:
  exclude-rules:
    - path: _test\.go
      linters:
        - gosec
        - errcheck

golangci-lint v2 walks upward from the directory where it's invoked to find the nearest config file. Running golangci-lint run ./... inside services/api/ picks up the root .golangci.yml automatically — no per-module config needed unless you want to override.

Why this matters: change one rule and it propagates to every service. No drift, no eslint.config.js archaeology.

Tool 5: golangci-lint + gofumpt — Unified Linting and Formatting

One .golangci.yml at the root. Each module runs golangci-lint run ./... but rules are defined once. One source of truth, many consumers.

golangci-lint v2

The version: "2" key at the top of .golangci.yml is required for the v2 config schema. The v1 schema is silently different — don't mix them.

Run across all modules in parallel:

go work edit -json | jq -r '.Use[].DiskPath' | \
  xargs -P4 -I{} sh -c 'cd {} && golangci-lint run ./...'

gofumpt

gofumpt is a stricter superset of gofmt. It enforces additional rules: blank lines between imports and first declaration, grouping of related declarations, trailing newlines. Configure it under formatters: in the v2 schema (or as a linter via gofumpt: true in linters-settings as shown above).

Install:

go install mvdan.cc/gofumpt@latest

Run standalone (useful in hooks):

gofumpt -l -w .

Why not just `gofmt`?

gofmt is the baseline. gofumpt is what you actually want in a team — it eliminates the formatting debates gofmt leaves open. Add it to the linter pipeline once and you never argue about blank lines again.

Tool 6: lefthook — Git Hooks

lefthook is a fast, language-agnostic Git hooks manager written in Go. It ships as a single binary, runs hooks in parallel by default, and integrates natively with staged files.

Install

go install github.com/evilmartians/lefthook@latest
# or: brew install lefthook

Initialize in the repo:

lefthook install

`lefthook.yml`

pre-commit:
  parallel: true
  commands:
    go-lint:
      glob: "*.go"
      run: golangci-lint run {staged_files}
      stage_fixed: true

    go-fmt:
      glob: "*.go"
      run: gofumpt -l -w {staged_files}
      stage_fixed: true

    proto-lint:
      glob: "*.proto"
      run: buf lint {staged_files}

    mod-tidy-check:
      run: |
        go work edit -json | jq -r '.Use[].DiskPath' | while read -r mod; do
          (cd "$mod" && go mod tidy && git diff --exit-code go.mod go.sum) || exit 1
        done

commit-msg:
  commands:
    conventional:
      run: |
        msg=$(cat {1})
        if ! echo "$msg" | grep -qE '^(feat|fix|chore|docs|refactor|test|ci)(\(.+\))?: .+'; then
          echo "Commit message must follow Conventional Commits format"
          exit 1
        fi

Key concepts:

{staged_files} — lefthook's template variable; expands to the list of files currently staged for this commit
stage_fixed: true — if gofumpt rewrites a file, lefthook re-stages it automatically. You don't have to git add again.
parallel: true — go-lint and go-fmt run concurrently; the hook only blocks as long as the slowest command

The key insight: it doesn't lint the whole repo on every commit — only the files currently staged. golangci-lint run {staged_files} is dramatically faster than golangci-lint run ./... across every module. This keeps commits fast while still enforcing quality.

The Full Picture

Developer commits
    ↓
lefthook pre-commit
    ↓ golangci-lint on staged .go files (fast, per-file)
    ↓ buf lint on staged .proto files
    ↓ go mod tidy check
    ↓ (fails → abort; passes → continue)
    ↓
go work resolves internal imports to local disk paths
    ↓
task build:all
    ↓ reads Taskfile.yml includes
    ↓ builds modules in order: gen/go → pkg/shared → pkg/events → services/* (parallel)
    ↓ caches outputs in .task/
    ↓
CI: GOWORK=off — each module tested independently
    ↓ git diff → affected modules → test only those + dependents
    ↓ golangci-lint matrix across modules
    ↓ .golangci.yml config inherited from repo root

Summary: Why This Stack

Problem	Tool	Mechanism
Share code without publishing	Go Workspaces	`go.work` + disk path resolution
Run tasks in dependency order	Task	`deps:` + `cmds:` ordering
Don't rebuild unchanged modules	Task cache	`sources`/`generates` hashing
CI: only changed modules	git diff script	`GOWORK=off` per-module testing
Consistent lint config	Root `.golangci.yml`	upward config file walk
One linter + formatter	golangci-lint + gofumpt	single `.golangci.yml`
Quality before commits	lefthook	`{staged_files}` pre-commit hook

Quick Start

# 1. Clone and initialize workspace
git clone https://github.com/yourorg/forge-monorepo
cd forge-monorepo
go work sync          # sync all module dependencies

# 2. Install tools
go install github.com/go-task/task/v3/cmd/task@latest
go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest
go install mvdan.cc/gofumpt@latest
go install github.com/evilmartians/lefthook@latest
brew install bufbuild/buf/buf

# 3. Install git hooks
lefthook install

# 4. Build everything
task build:all

# 5. Test everything
task test:all

What to Watch Out For

go.work in CI. Set GOWORK=off in your CI environment variables. Each module's go.mod must have correct require entries — the workspace won't paper over missing dependencies in CI.

Module path discipline. Your go.mod module paths must be globally valid (i.e., github.com/yourorg/forge/...), even if you never publish. The workspace resolves them locally, but the paths need to be stable and consistent across all modules.

golangci-lint v1 vs v2. The config schema changed significantly. If you're migrating from v1, the version: "2" key is mandatory and several linter names moved. Run golangci-lint migrate to auto-upgrade your config.

Task and go generate. Add a generate task that runs buf generate before any go build task that depends on generated types. Generated code in gen/go/ is a real module with its own go.mod — it needs to be built first, just like any other dependency.

Python Monorepo Magic: Organize, Build, and Ship Multi-Service Apps

M Hossein — Wed, 03 Jun 2026 07:10:00 +0000

What is a Monorepo?

A monorepo is a single Git repository that contains multiple distinct packages or applications. The alternative is a polyrepo: one repo per app or package.

This repo is nest-monorepo. It contains:

nest-monorepo/
├── apps/
│   ├── api/         ← FastAPI backend
│   ├── worker/      ← Celery async worker
│   └── cli/         ← Internal tooling CLI
├── packages/
│   ├── shared/      ← Pydantic schemas + SQLAlchemy models
│   ├── events/      ← Event type definitions
│   └── proto/       ← Protobuf definitions + generated stubs

The core idea: code that is needed by multiple apps lives in packages/, and all apps can import it directly — no PyPI publishing required.

Tool 1: uv Workspaces — The Foundation

uv is the package manager. Workspaces are its monorepo feature.

How it's declared

Root pyproject.toml:

[tool.uv.workspace]
members = ["apps/*", "packages/*"]

That's it. Three lines define the entire workspace. uv reads every pyproject.toml under those globs and treats each directory as a workspace member.

What this enables

When you run uv sync at the repo root, uv:

Reads every pyproject.toml in every workspace directory
Resolves all external dependencies into a single uv.lock file at the root
Installs internal packages as editable installs so they can import each other directly

The `{ workspace = true }` source

Look at apps/api/pyproject.toml:

[project]
name = "api"
version = "0.1.0"
requires-python = ">=3.12"
dependencies = ["shared", "events", "fastapi>=0.115"]

[tool.uv.sources]
shared = { workspace = true }
events = { workspace = true }

{ workspace = true } means: don't fetch this from PyPI — link it from this workspace instead. When api imports shared, Python resolves it to packages/shared/src/ on disk via an editable install. No publishing, no version pinning against PyPI, no symlink juggling.

This is the mechanism that makes internal sharing work.

Why uv over pip/poetry?

uv uses a global content-addressable cache at ~/.cache/uv. Every version of every package is stored once globally across all projects on your machine. Workspaces get hard links to the cache, not copies. This means:

10–100x faster installs than pip
A single uv.lock committed at the repo root — one lockfile, all packages
Strict resolution — packages can't accidentally import things they didn't declare

Note: uv.lock is committed to version control. Unlike go.work.sum, it is not gitignored. It is the canonical record of every resolved dependency across the entire workspace.

One virtualenv to rule them all

uv sync creates a single .venv at the repo root. All workspace members are installed into it as editable installs. There is no per-package virtualenv.

This has two important consequences:

No version skew. api and worker cannot depend on different versions of shared — they share the same installed environment.
IDE setup is trivial. Point your editor's Python interpreter at .venv/bin/python once, at the repo root. Every package is immediately importable — no per-project interpreter switching.

uv sync                  # creates .venv, installs everything
source .venv/bin/activate  # or let your IDE detect it
python -c "import shared"  # works from anywhere in the repo

Add .venv/ to .gitignore.

Tool 2: Task (Taskfile.yml) — Task Orchestration and Caching

uv workspaces handle dependencies. Task handles tasks (build, test, lint, etc.).

Installing Task

Task is a language-agnostic binary — it has nothing to do with Go and requires no Go installation.

# macOS / Linux via Homebrew (recommended)
brew install go-task

# Linux: direct install script (no Go needed)
sh -c "$(curl --location https://taskfile.dev/install.sh)" -- -d -b ~/.local/bin

# Or grab a binary directly from https://github.com/go-task/task/releases

Verify with task --version.

The problem Task solves

If you run uv run pytest across 6 packages, you have to figure out the order yourself. shared must be importable before events, which must be importable before api and worker. proto must run buf generate before anything imports the generated stubs. Do it wrong and you get import errors or stale generated code.

Also, if nothing in packages/shared changed, you shouldn't rebuild it.

Task solves both: explicit task ordering + file-based caching via sources/generates.

`Taskfile.yml` — the pipeline

version: '3'

includes:
  shared:
    taskfile: ./packages/shared/Taskfile.yml
    dir: ./packages/shared        # without dir:, commands run from repo root
  events:
    taskfile: ./packages/events/Taskfile.yml
    dir: ./packages/events
  proto:
    taskfile: ./packages/proto/Taskfile.yml
    dir: ./packages/proto
  api:
    taskfile: ./apps/api/Taskfile.yml
    dir: ./apps/api
  worker:
    taskfile: ./apps/worker/Taskfile.yml
    dir: ./apps/worker
  cli:
    taskfile: ./apps/cli/Taskfile.yml
    dir: ./apps/cli

tasks:
  build:all:
    desc: Build all packages in dependency order
    cmds:
      - task: shared:build
      - task: events:build
      - task: proto:generate
      - task: api:build
      - task: worker:build
      - task: cli:build

  test:all:
    desc: Run all tests
    deps: [shared:build, events:build, proto:generate]
    cmds:
      - task: api:test
      - task: worker:test
      - task: cli:test

  lint:all:
    cmds:
      - task: shared:lint
      - task: events:lint
      - task: proto:lint
      - task: api:lint
      - task: worker:lint
      - task: cli:lint

packages/proto/Taskfile.yml:

version: '3'

tasks:
  generate:
    desc: Generate Python stubs from .proto files
    sources:
      - '**/*.proto'
    generates:
      - 'generated/**/*.py'
    cmds:
      - buf generate

Key concepts:

sources — glob patterns of input files. Task fingerprints these.
generates — files produced by the task. Task checks if they still exist and match the fingerprint.
If inputs haven't changed and outputs still exist, Task skips the task entirely (cache hit). Results are stored in .task/.
deps: runs dependencies in parallel before the task executes.
cmds: runs commands sequentially, in order.

How the build graph flows

packages/shared (uv sync)
    ↓
packages/events (uv sync)     packages/proto (buf generate)
         ↓                              ↓
    apps/api                     (generates stubs)
    apps/worker                          ↓
    apps/cli               apps/api, apps/worker (import proto stubs)

Running individual apps

# Run just the API
task api:dev

# Run just the worker
task worker:dev

# Build a single package
task shared:build

Affected-only in CI

Task has no --affected flag. The naive approach — looping over changed package names — is wrong: if shared changes, api and worker (which depend on it) must also be tested. You need reverse-dependency propagation.

#!/usr/bin/env bash
# ci/affected.sh — tests changed packages AND their dependents
CHANGED=$(git diff --name-only origin/main...HEAD)

# Reverse dependency map: if X changes, also test these
declare -A RDEPS=(
  [shared]="api worker cli"
  [events]="api worker"
  [proto]="api worker"
)

PACKAGES=()
mark() {
  local pkg=$1
  [[ " ${PACKAGES[*]} " =~ " ${pkg} " ]] && return   # skip if already queued
  PACKAGES+=("$pkg")
  for dep in ${RDEPS[$pkg]:-}; do mark "$dep"; done   # propagate to dependents
}

echo "$CHANGED" | grep -q "^packages/shared/" && mark "shared"
echo "$CHANGED" | grep -q "^packages/events/" && mark "events"
echo "$CHANGED" | grep -q "^packages/proto/"  && mark "proto"
echo "$CHANGED" | grep -q "^apps/api/"        && mark "api"
echo "$CHANGED" | grep -q "^apps/worker/"     && mark "worker"
echo "$CHANGED" | grep -q "^apps/cli/"        && mark "cli"

for pkg in "${PACKAGES[@]}"; do task "${pkg}:test"; done

A change to packages/shared marks shared → then propagates to api, worker, cli. A change to apps/api only marks api.

The honest alternative: skip this complexity and run task test:all every time. Task's sources/generates caching skips packages whose inputs haven't changed. You get the same outcome with far less script maintenance.

The key insight: Task knows nothing about Python — it just tracks file fingerprints and runs shell commands in the order you declare. That's exactly the level of abstraction you want.

Tool 3: Shared Packages — The Internal Packages Pattern

Source-level packages (no build step)

Most packages in this repo point directly at Python source:

packages/shared/pyproject.toml:

[project]
name = "shared"
version = "0.1.0"
requires-python = ">=3.12"
dependencies = [
  "pydantic>=2.0",
  "sqlalchemy>=2.0",
]

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

packages/shared/src/shared/__init__.py exports the Pydantic models and SQLAlchemy base classes directly. No compilation, no dist/ folder. When apps/api imports shared, it reaches straight into packages/shared/src/ via the editable install.

This is the internal packages pattern — source is the distribution artifact.

Packages that do need a build step

packages/proto generates Python stubs from .proto files:

packages/proto/pyproject.toml:

[project]
name = "proto"
version = "0.1.0"
requires-python = ">=3.12"
dependencies = ["grpcio>=1.60", "protobuf>=4.0"]

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

packages/proto/Taskfile.yml:

tasks:
  generate:
    sources: ['**/*.proto']
    generates: ['generated/**/*.py']
    cmds:
      - buf generate   # buf handles all codegen — don't mix with raw protoc

Task's sources/generates fingerprinting ensures proto:generate runs before anything that imports the stubs — and skips if the .proto files haven't changed.

Consuming internal packages in apps

apps/api/pyproject.toml:

[project]
name = "api"
version = "0.1.0"
requires-python = ">=3.12"
dependencies = [
  "shared",
  "events",
  "proto",
  "fastapi>=0.115",
  "uvicorn>=0.30",
]

[tool.uv.sources]
shared = { workspace = true }
events = { workspace = true }
proto  = { workspace = true }

apps/worker/pyproject.toml declares the same [tool.uv.sources] block. Because uv resolves everything into a single uv.lock, both api and worker are guaranteed to use the exact same version of shared — no version skew between apps in the same repo.

Why this matters: you can refactor a Pydantic model in packages/shared and immediately see type errors in both apps/api and apps/worker without publishing anything. The workspace is the registry.

Tool 4: Root `pyproject.toml` — Shared Config

Rather than duplicating [tool.mypy] and [tool.pytest.ini_options] across 6 packages, this repo centralizes all tool configuration in the root pyproject.toml.

Root pyproject.toml:

[project]
name = "nest-monorepo"
version = "0.0.0"
requires-python = ">=3.12"

[tool.uv.workspace]
members = ["apps/*", "packages/*"]

# Dev tools live here — not in subpackage pyproject.toml files
[dependency-groups]
dev = [
    "pytest>=8",
    "pytest-asyncio>=0.25",   # asyncio_mode = "auto" requires this
    "pytest-cov>=5",
    "mypy>=1.15",
    "ruff>=0.9",
    "pydantic[mypy]>=2.0",    # provides the pydantic.mypy plugin
    "types-requests",
]

[tool.mypy]
python_version = "3.12"
strict = true
disallow_untyped_defs = true
no_implicit_optional = true
warn_redundant_casts = true
warn_unused_ignores = true
plugins = ["pydantic.mypy"]
# Paths where mypy finds internal package source
mypy_path = "packages/shared/src:packages/events/src:packages/proto/src"

[tool.pytest.ini_options]
testpaths = ["apps", "packages"]   # no top-level tests/ dir — each package has its own
asyncio_mode = "auto"
addopts = "-v --tb=short"

[tool.coverage.run]
branch = true
source = ["apps", "packages"]   # not "src" — there's no top-level src/

[tool.coverage.report]
fail_under = 80
show_missing = true

Install dev tools with:

uv sync --group dev

How mypy finds this config

mypy does not walk up the directory tree. Unlike Ruff, mypy reads config from the directory where it's invoked — not from the file being type-checked. If you run uv run mypy src/ from inside apps/api/, mypy finds no [tool.mypy] there, silently falls back to non-strict defaults, and your strict = true is ignored.

The correct pattern: always run mypy from the repo root, passing the directories to check:

uv run mypy apps/ packages/

This is different from Ruff, which discovers config per-file by walking upward. With mypy, the invocation location is what matters — not the file location.

There is no [tool.mypy] in subpackages. Do not add one. mypy does not merge configs — a subpackage [tool.mypy] completely replaces the root config rather than extending it.

Why this matters: change strict = true in one place and it applies to all 6 packages — but only when mypy is invoked from the root. Enforce this in your Taskfile and CI.

Tool 5: Ruff — Unified Linting and Formatting

Ruff replaces Flake8 + Black + isort with a single fast tool written in Rust. A single [tool.ruff] section in the root pyproject.toml applies to the entire repo.

Root pyproject.toml (continued):

[tool.ruff]
target-version = "py312"
line-length = 100
src = ["apps", "packages"]   # not "src" — tells Ruff where source roots are

[tool.ruff.lint]
select = [
  "E",   # pycodestyle errors
  "W",   # pycodestyle warnings
  "F",   # pyflakes
  "I",   # isort
  "UP",  # pyupgrade
  "B",   # flake8-bugbear
  "SIM", # flake8-simplify
]
ignore = ["E501"]  # line length handled by formatter

[tool.ruff.lint.per-file-ignores]
"**/tests/**" = ["S101"]  # allow assert in tests

[tool.ruff.lint.isort]
known-first-party = ["shared", "events", "proto"]   # without this, internal packages
                                                     # sort as third-party — wrong grouping

[tool.ruff.format]
quote-style = "double"
indent-style = "space"

Config discovery: how subpackages inherit

Ruff discovers config by walking up the directory tree from the file being linted until it finds a pyproject.toml, ruff.toml, or .ruff.toml. In nest-monorepo, every file in apps/ and packages/ walks up and hits the root pyproject.toml. One config, all packages.

Per-package overrides

If apps/cli has looser requirements (it's internal tooling), add an extend in apps/cli/pyproject.toml:

[tool.ruff]
extend = "../../pyproject.toml"

[tool.ruff.lint]
ignore = ["E501", "T201"]  # allow print() in CLI

extend inherits all parent settings and overrides only what you specify. This is the right pattern — not a full [tool.ruff] block that silently drops all parent rules.

Each package runs:

uv run ruff check src/
uv run ruff format src/

But the rules are defined once. This is the same pattern as the mypy config: one source of truth, many consumers.

The key insight: Ruff running at 10–100x the speed of Flake8 is not just a nice-to-have in a monorepo. When pre-commit runs on every commit across 6 packages, linting speed is the difference between a 2-second hook and a 30-second one.

Tool 6: pre-commit + Git Hooks — Enforcing Quality at Commit Time

Install pre-commit as a uv tool (not a project dependency):

uv tool install pre-commit
pre-commit install

.pre-commit-config.yaml at the repo root:

repos:
  - repo: https://github.com/astral-sh/ruff-pre-commit
    rev: v0.9.0
    hooks:
      - id: ruff
        args: [--fix]
      - id: ruff-format

  # mypy CANNOT use the standard pre-commit mirror hook here.
  # pre-commit/mirrors-mypy runs mypy in an isolated virtualenv that has no
  # access to your uv workspace's editable installs — `from shared import ...`
  # produces false ImportError failures. Run mypy via the local repo instead:
  - repo: local
    hooks:
      - id: mypy
        name: mypy
        language: system           # uses the active .venv, not an isolated env
        entry: uv run mypy
        args: [apps/, packages/, --config-file, pyproject.toml]
        types: [python]
        pass_filenames: false      # mypy needs whole-package context, not individual files

The key insight: pre-commit automatically passes only staged files to Ruff — no custom script needed. ruff check --fix runs only against the files you're about to commit. mypy, however, needs the full package context to resolve imports correctly, so pass_filenames: false is set and it receives the full apps/ packages/ instead.

The flow on git commit:

pre-commit intercepts the commit
ruff checks and auto-fixes staged .py files
ruff-format formats staged .py files
mypy type-checks all of apps/ and packages/ (fast due to incremental cache)
If any hook fails — commit is aborted, fixed files are left staged for review
If all hooks pass — commit proceeds

Installing consistently across the team

Add to root Taskfile.yml:

tasks:
  setup:
    desc: Bootstrap the repo for local development
    cmds:
      - uv sync
      - uv tool install pre-commit
      - pre-commit install

Every developer runs task setup once after cloning. No tribal knowledge required.

Why this matters: a linting problem caught at commit time costs 3 seconds to fix. The same problem caught in CI costs 5 minutes of context switching. The same problem caught in code review costs someone else's time too.

CI: GitHub Actions

# .github/workflows/ci.yml
name: CI
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: astral-sh/setup-uv@v5
        with:
          enable-cache: true          # caches ~/.cache/uv between runs

      - run: uv sync --frozen --group dev
        # --frozen: fail if uv.lock is out of date (never auto-update in CI)
        # --group dev: includes pytest, mypy, ruff

      - run: uv run ruff check apps/ packages/
      - run: uv run ruff format --check apps/ packages/

      - run: uv run mypy apps/ packages/
        # always run from repo root — mypy does not walk up per-file

      - run: uv run pytest apps/ packages/ -x --cov
        env:
          PYTHONDONTWRITEBYTECODE: "1"

      - run: uv cache prune --ci
        # trims downloaded wheels, keeps source-built cache for next run

The --frozen flag is the CI safety net. If a developer forgot to commit an updated uv.lock after adding a dependency, the CI job fails immediately rather than silently installing a different package version than the rest of the team has.

Deploying One App from a Monorepo

The hardest Python monorepo question isn't the build — it's Docker. You need an image for apps/api that includes its internal deps (packages/shared, packages/events) without shipping the entire repo or the dev toolchain.

uv makes this clean with --package:

# apps/api/Dockerfile
FROM python:3.12-slim AS builder

COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv

WORKDIR /app

# Copy the workspace manifests first (cache layer for dep installation)
COPY pyproject.toml uv.lock ./
COPY packages/shared/pyproject.toml packages/shared/pyproject.toml
COPY packages/events/pyproject.toml packages/events/pyproject.toml
COPY apps/api/pyproject.toml        apps/api/pyproject.toml

# Install only api's deps — no dev tools, no other apps
RUN uv sync \
    --package api \
    --no-dev \
    --frozen \
    --no-editable    # copies source into site-packages instead of editable links

# Now copy the actual source (separate layer so code changes don't bust dep cache)
COPY packages/shared/src packages/shared/src
COPY packages/events/src packages/events/src
COPY apps/api/src        apps/api/src

FROM python:3.12-slim
COPY --from=builder /app/.venv /app/.venv
COPY --from=builder /app/apps/api/src /app/src

ENV PATH="/app/.venv/bin:$PATH"
CMD ["uvicorn", "api.main:app", "--host", "0.0.0.0", "--port", "8000"]

Key flags:

--package api — installs only api and its transitive deps (including shared and events)
--no-dev — excludes pytest, mypy, ruff
--frozen — fails if uv.lock is stale
--no-editable — copies package source into .venv/lib/ so the final image doesn't need the source tree

Why this matters: without --no-editable, the editable installs point back at the source paths in the build context. The final image would need all of packages/shared/src/ mapped at the same absolute path. --no-editable severs that dependency — the .venv is self-contained.

The Full Picture: How These Tools Interact

Developer commits
    ↓
pre-commit hook
    ↓ ruff-check (staged files only, --fix)
    ↓ ruff-format (staged files only)
    ↓ (fails → abort commit; passes → continue)
    ↓
uv workspace resolves internal deps via { workspace = true } editable installs
    ↓
task build:all
    ↓ reads Taskfile.yml includes
    ↓ builds packages in order: shared → events → api/worker/cli
    ↓ caches outputs in .task/
    ↓
CI: git diff → affected packages → test only those + dependents
    ↓ diffs against main branch
    ↓ runs only impacted packages
    ↓ Ruff config inherited from root pyproject.toml
    ↓ mypy config inherited from root pyproject.toml

Summary: Why This Stack

Problem	Tool	Mechanism
Share code without publishing to PyPI	uv workspaces	`{ workspace = true }` + editable installs
Run tasks in dependency order	Task	explicit `cmds:` ordering + `deps:` for parallel
Don't rebuild unchanged packages	Task cache	`sources`/`generates` fingerprinting in `.task/`
Run tasks on only changed code in CI	git diff script	`git diff --name-only origin/main...HEAD`
Consistent mypy/pytest config	root `pyproject.toml`	single `[tool.mypy]`; always invoke from repo root
One linter/formatter config	Ruff root `[tool.ruff]`	config discovery walks up per-file
Per-package lint overrides	Ruff `extend`	inherits parent rules, adds/removes specific ones
Enforce quality before commits	pre-commit hooks	staged-file Ruff; mypy via `language: system`
Deploy one app without full repo	`uv sync --package api --no-editable`	installs app + internal deps, no dev tools

The monorepo isn't one tool — it's these tools composing together. uv handles what exists and how it's linked, Task handles when things run and in what order, and the tooling layer in pyproject.toml handles how everything is configured.

TypeScript Monorepo Magic: Organize, Build, and Ship Multi-Package Apps

M Hossein — Tue, 02 Jun 2026 12:16:54 +0000

What is a Monorepo?

A monorepo is a single Git repository that contains multiple distinct packages or applications. The alternative is a polyrepo: one repo per app or package.

This repo is shrimp-monorepo. It contains:

shrimp-monorepo/
├── apps/
│   ├── api/            ← Node.js backend (Elysia + gRPC)
│   ├── client-user/    ← React frontend for end users
│   └── client-admin/   ← React frontend for admins
├── packages/
│   ├── shared-types/   ← TypeScript types used everywhere
│   ├── ui/             ← Shared React components
│   ├── proto/          ← Protobuf definitions + generated code
│   ├── grpc-client/    ← gRPC client wrapper
│   └── db-schema/      ← Drizzle ORM schema
└── tooling/
    └── typescript/     ← Shared tsconfig files

The core idea: code that is needed by multiple apps lives in packages/, and all apps can import it directly — no npm publishing required.

Tool 1: pnpm Workspaces — The Foundation

pnpm is the package manager. Workspaces are its monorepo feature.

How it's declared

pnpm-workspace.yaml:

packages:
  - 'apps/*'
  - 'packages/*'
  - 'tooling/*'

This tells pnpm: treat every directory in these globs as a package. That's it — three lines define the entire workspace.

What this enables

When you run pnpm install at the repo root, pnpm:

Reads every package.json in every workspace directory
Hoists shared external dependencies into a single node_modules at the root
Creates symlinks for internal packages so they can import each other

The `workspace:*` protocol

Look at apps/api/package.json:

"dependencies": {
  "@shrimp/db-schema": "workspace:*",
  "@shrimp/proto": "workspace:*",
  "@shrimp/shared-types": "workspace:*"
}

workspace:* means: don't fetch this from npm — link it from this workspace instead. When api imports @shrimp/db-schema, Node resolves it to packages/db-schema/src/index.ts on disk via a symlink in node_modules/.pnpm.

This is what makes internal sharing work without publishing packages.

Why pnpm over npm/yarn?

pnpm uses a content-addressable store (the .pnpm-store/ directory in this repo). Every version of every package is stored once globally. Workspaces get hard links to the store, not copies. This means:

Faster installs
Significantly less disk space
Strict by default — packages can't accidentally import things they didn't declare

Tool 2: Turborepo — Task Orchestration and Caching

pnpm workspaces handle dependencies. Turbo handles tasks (build, test, lint, etc.).

The problem Turbo solves

If you run pnpm run build in a repo with 8 packages, you have to figure out the order yourself. proto must build before grpc-client, which must build before api. Do it wrong and you get stale or missing types.

Also, if nothing in packages/ui changed, you shouldn't need to rebuild it.

Turbo solves both: dependency-aware task ordering + task result caching.

`turbo.json` — the pipeline

{
  "tasks": {
    "build": {
      "dependsOn": ["^build"],
      "outputs": ["dist/**", ".output/**", "generated/**"]
    },
    "dev": {
      "cache": false,
      "persistent": true
    },
    "lint": {
      "dependsOn": ["^build"]
    },
    "typecheck": {
      "dependsOn": ["^build"]
    },
    "test:unit": {
      "dependsOn": ["^build"],
      "outputs": ["coverage/**"]
    }
  }
}

Key concepts:

"dependsOn": ["^build"] — the ^ prefix means build all packages that this package depends on first. So when building @shrimp/api, Turbo automatically builds @shrimp/proto, @shrimp/db-schema, and @shrimp/shared-types beforehand, in the right order.
"dependsOn": ["build"] (no ^) — run the same package's build task first. Used by test:e2e: build the app before running E2E tests.
"cache": false — never cache this task. dev is persistent/interactive, so caching makes no sense.
"persistent": true — this task runs forever (a dev server). Turbo knows not to treat it as something that finishes.
"outputs" — Turbo hashes these paths to know what "done" looks like. If inputs haven't changed and outputs still exist, Turbo skips the task entirely (cache hit).

How the build graph flows

proto:generate
    ↓
@shrimp/proto (build)
    ↓
@shrimp/grpc-client (build)
    ↓
apps/api (build)
apps/client-user (build)
apps/client-admin (build)

@shrimp/shared-types, @shrimp/db-schema, and @shrimp/ui also build in parallel before their consumers, since they have no inter-dependencies.

Filtering — run tasks for specific packages

The root package.json uses --filter for targeted dev:

"dev:user":  "turbo run dev --filter=@shrimp/client-user",
"dev:admin": "turbo run dev --filter=@shrimp/client-admin",
"dev:api":   "turbo run dev --filter=@shrimp/api"

--filter accepts package names, directory globs, or git-based expressions.

Affected-only in CI

The CI workflow uses the most powerful filter:

run: pnpm turbo run typecheck lint test:unit build --affected

--affected uses the git diff against the base branch to determine which packages changed, then runs tasks only on those packages (and their dependents). A PR that only touches packages/ui won't re-run api tests.

Tool 3: Shared Packages — How Internal Code is Shared

Source-level packages (no compilation step)

Most packages in this repo point directly at TypeScript source:

packages/shared-types/package.json:

{
  "name": "@shrimp/shared-types",
  "main": "./src/index.ts",
  "types": "./src/index.ts",
  "exports": {
    ".": "./src/index.ts"
  }
}

There is no build script. When @shrimp/api imports @shrimp/shared-types, it imports .ts files directly. The consuming app's bundler or TypeScript compiler handles the compilation.

This is sometimes called the "internal packages" pattern — no dist/ folder, no compile step, just source. It works because all consumers in the monorepo are TypeScript themselves.

Packages that do need a build step

@shrimp/proto generates TypeScript from .proto files:

"scripts": {
  "proto:generate": "pnpm exec protoc --ts_out ./generated ...",
  "build": "pnpm run proto:generate"
}

Turbo's "dependsOn": ["^build"] ensures proto:generate runs before anything that consumes @shrimp/proto.

Tool 4: Shared TypeScript Config — `tooling/typescript`

Rather than duplicating 25 lines of compilerOptions across 8 packages, this repo centralizes TypeScript config in tooling/typescript/.

tooling/typescript/tsconfig.base.json — strict settings shared by all packages:

{
  "compilerOptions": {
    "strict": true,
    "noUnusedLocals": true,
    "noUncheckedIndexedAccess": true,
    "moduleResolution": "bundler",
    ...
  }
}

tooling/typescript/tsconfig.react.json — extends base, adds JSX:

{
  "extends": "./tsconfig.base.json",
  "compilerOptions": { "jsx": "react-jsx" }
}

Each package inherits via extends:

apps/api/tsconfig.json:

{
  "extends": "../../tooling/typescript/tsconfig.base.json",
  "compilerOptions": {
    "outDir": "./dist",
    "rootDir": "./src"
  }
}

packages/ui/tsconfig.json:

{
  "extends": "../../tooling/typescript/tsconfig.react.json",
  "include": ["src/**/*.ts", "src/**/*.tsx", "tests/**/*.ts"]
}

Why this matters: change one setting in tsconfig.base.json and it propagates to every package in the repo. No drift, no "why is strict mode off in this one package" surprises.

Tool 5: Biome — Unified Linting and Formatting

Biome replaces ESLint + Prettier with a single fast tool. A single biome.json at the root applies to the entire repo.

biome.json key config:

{
  "linter": {
    "rules": {
      "correctness": {
        "noUnusedVariables": "error",
        "noUnusedImports": "error"
      },
      "style": { "useConst": "error" }
    }
  },
  "formatter": {
    "indentStyle": "tab",
    "indentWidth": 2,
    "lineWidth": 100
  }
}

Each package runs biome check . as its lint script, but the rules are defined once. This is the same pattern as TypeScript config inheritance: one source of truth, many consumers.

Tool 6: Git Hooks — Enforcing Quality at Commit Time

The repo uses a custom git hooks directory instead of the default .git/hooks. This means hooks can be committed and versioned.

Setup (runs on pnpm install via the prepare lifecycle):

"prepare": "git config core.hooksPath .githooks"

.githooks/pre-commit:

#!/usr/bin/env sh
pnpm precommit:staged

scripts/biome-staged.mjs — runs Biome only on staged files:

const staged = spawnSync("git", ["diff", "--cached", "--name-only", "-z", ...]);
// ... filters to .ts/.tsx/.js/.json files
spawnSync("pnpm", ["exec", "biome", "check", "--no-errors-on-unmatched", ...files]);

The key insight: it doesn't lint the whole repo on every commit — only the files currently staged. This keeps commits fast while still enforcing quality.

The Full Picture: How These Tools Interact

Developer commits
    ↓
git pre-commit hook (.githooks/pre-commit)
    ↓ runs biome-staged.mjs
    ↓ Biome checks only staged files
    ↓ (fails → abort commit; passes → continue)
    ↓
pnpm workspace resolves internal deps via workspace:* symlinks
    ↓
turbo run build
    ↓ reads turbo.json pipeline
    ↓ resolves dependency graph from package.json deps
    ↓ builds packages in order: proto → grpc-client → api/clients
    ↓ caches outputs in .turbo/
    ↓
CI: turbo run ... --affected
    ↓ diffs against main
    ↓ runs only impacted packages
    ↓ TypeScript config inherited from tooling/typescript/
    ↓ Biome config inherited from root biome.json

Summary: Why This Stack

Problem	Tool	Mechanism
Share code without publishing to npm	pnpm workspaces	`workspace:*` + symlinks
Run tasks in dependency order	Turbo	`"dependsOn": ["^build"]`
Don't rebuild unchanged packages	Turbo cache	`outputs` hashing
Run tasks on only changed code in CI	Turbo `--affected`	git diff
Consistent TypeScript config	`tooling/typescript`	`extends` inheritance
One linter/formatter config	Biome root `biome.json`	single config, all packages
Enforce quality before commits	Git hooks (`.githooks/`)	staged-file Biome check

The monorepo isn't one tool — it's these tools composing together. pnpm handles what exists, Turbo handles when things run, and the tooling layer handles how they're configured.

DEV Community: M Hossein

I switched on production evals for my LLM app — and they scored nothing

The Context & The Constraint

The Naive Approach (The MVP)

The Architectural Evolution (Iterative Refinement)

Subsystem 1: Tracing Without Leaking

Subsystem 2: The LLM Judge & Edge Constraints

Subsystem 3: The Prompt Versioning Illusion

The Takeaway

Stop Shipping Bloat: 7 Steps to 15x Faster, 90% Smaller Docker Builds

1. Leverage Multi-Stage Builds

2. Pick a Smaller Base & Pin Your Versions

3. Order Layers by Invalidation Frequency

The Antipattern vs. The Optimized Pattern

The Bloated Approach (What to Avoid)

The Optimized Approach (Production-Ready)

4. Don't Skip the .dockerignore

5. Chain Your RUN Commands

Summary Checklist

Multi-VPS Distributed n8n Cluster on Ubuntu 24.04 with an IPIP Tunnel

The Target Architecture

Step 1: Base Server Prep & Official Docker Installation

Installing Docker using the Modern Apt Sources Method

Step 2: Provisioning a Persistent Private IPIP Tunnel

Step 3: Deploying the Control Plane (VPS 1)

1. The Environment Setup (/home/n8n/.env)

2. The Infrastructure Deployment (/home/n8n/docker-compose.yml)

Step 4: Configure Nginx Reverse Proxy with WebSockets

Step 5: Deploying the Execution Worker (VPS 2)

1. Create /home/n8n/.env on VPS 2

2. Create /home/n8n/docker-compose.yml on VPS 2

Step 6: Server Hardening & Security Policies

1. Restrict Network Interfaces via UFW

2. Prevent Docker from Bypassing Your Firewall

3. Polish the SSH Configuration

Verification & Monitoring

Stop Guessing, Start Profiling: A Dev's Guide to Go Mechanics

The Problem: The Mysterious 2-Second Freeze

📊 Diagram 1: The Mark Assist Traffic Jam

The Right Way: Mechanical Sympathy

Fix 1: Stop Making Trash (Size-Classed Pools)

Fix 2: Protect the Database (The Cancellation Storm)

📊 Diagram 2: The Self-Inflicted DDoS

Fix 3: Don't Break the Chain (Failing Closed)

Fix 4: See the Ghost (Continuous Profiling)

Summary

The Blast Radius: Mentoring Senior Engineers into Systems Thinkers

The Scenario: The Siloed Senior Engineer

Phase 1: The Standard Management Reaction

Phase 2: The Force Multiplier Framework

1. The Blameless Post-Mortem (Reframing the Incident)

2. Vulnerability as Leadership (The Tribe Meeting)

3. Structured Cross-Pollination (The RFC Process)

4. Operational Empathy ("You Build It, You Run It")

The Payoff: Six Months Later

The Takeaway

Architecting Strict Sequential Ordering in a Concurrent World

Phase 1: The MVP & The Database Bottleneck

Phase 2: The Event-Driven Reality (Step-by-Step Fixes)

Step 1: The Ingestion Illusion & Operational Memory

Step 2: The Database Constraint & The Infinite Ledger

Step 3: Concurrency Anti-Patterns

Step 4: The Edge Case Reality

The Takeaway

Polyglot Monorepo Magic: TypeScript, Python, and Go in One Repo

What is a Polyglot Monorepo?

Tool 1: Three Workspace Managers — The Parallel Foundation

TypeScript: pnpm-workspace.yaml

Python: root pyproject.toml

Go: go.work

go.mod plumbing — how CI works without go.work

What uv sync and pnpm install and go work sync each do

Tool 2: Task (Taskfile.yml) — The Cross-Language Orchestrator

The root Taskfile.yml

Language-specific Taskfiles

CI: affected-only runs

Tool 3: buf + Protobuf — The Contract Layer

buf.yaml — workspace config at repo root

buf.gen.yaml — one command, three languages

Generated code as first-class modules

4. Don't Skip the `.dockerignore`

5. Chain Your `RUN` Commands

1. The Environment Setup (`/home/n8n/.env`)

2. The Infrastructure Deployment (`/home/n8n/docker-compose.yml`)

1. Create `/home/n8n/.env` on VPS 2

2. Create `/home/n8n/docker-compose.yml` on VPS 2

TypeScript: `pnpm-workspace.yaml`

Python: root `pyproject.toml`

Go: `go.work`

`go.mod` plumbing — how CI works without `go.work`

What `uv sync` and `pnpm install` and `go work sync` each do

The root `Taskfile.yml`

`buf.yaml` — workspace config at repo root

`buf.gen.yaml` — one command, three languages

`biome.json` — TypeScript

root `pyproject.toml` — Python lint + type config

`.golangci.yml` — Go

`.editorconfig` — baseline cross-language consistency

Tool 5: Three Linters, One `task lint` Command

`lefthook.yml`

Tool 1: Go Workspaces (`go work`) — The Foundation

Critical: do NOT commit `go.work`

Why `go work` over `replace` directives?

`Taskfile.yml` — the pipeline

Tool 4: Root `.golangci.yml` — Shared Config

Why not just `gofmt`?

`lefthook.yml`

The `{ workspace = true }` source

`Taskfile.yml` — the pipeline

Tool 4: Root `pyproject.toml` — Shared Config