DEV Community: Kuba

Concurrency Is Not Parallelism — And Go Knows the Difference

Kuba — Fri, 15 May 2026 16:58:38 +0000

I build backend systems for a living. Most of what I ship runs on NestJS — TypeScript, async/await, event-driven patterns, the usual. But over the last year, Go has quietly become my second language, and not by accident. I reached for it specifically because of how it handles concurrency.

This post is about that decision, and more importantly — about what's actually happening under the hood when you write go func(). Because most articles stop at "goroutines are lightweight threads," and that explanation is simultaneously true and almost useless for building mental models that actually help you write better code.

Let's go deeper.

Why Concurrency Matters (and Why Most Developers Underestimate It)

Here's a question worth sitting with: when your server handles 1,000 simultaneous HTTP requests, what is actually happening on the machine?

If you're coming from Node.js, your instinct might be "the event loop handles it — one thread, non-blocking I/O." If you're coming from Java or C#, you might think "a thread pool." If you're coming from Python, you might be somewhere between "GIL hell" and "asyncio."

All of these are concurrency models. They have different tradeoffs, different failure modes, and different mental overhead. Go's model is arguably the most honest about what's actually happening — and I mean that as a genuine compliment.

But before we get into Go, we need to agree on terminology, because this field is plagued by conflation.

Concurrency vs. Parallelism — The Distinction That Actually Matters

Rob Pike (one of Go's creators) gave a talk in 2012 that I consider required viewing for any backend developer: "Concurrency is not parallelism."

The core idea:

Concurrency is about structure. It's the composition of independently executing computations. It's a design property.
Parallelism is about execution. It's the simultaneous execution of multiple computations. It's a runtime property.

A concurrent program can run in parallel, but doesn't have to. A parallel program doesn't have to be well-structured concurrently.

Practical example: imagine you're cooking dinner. You put pasta water on to boil, then chop vegetables while you wait, then check the water, then continue chopping. You're doing one thing at a time — but you've structured the work concurrently. If you had two pairs of hands doing both simultaneously, that would be parallel.

Go lets you express concurrency clearly. Your OS + Go runtime decides whether to execute it in parallel.

Why I Use Go (and Where NestJS Falls Short)

I want to be honest here: NestJS is excellent. TypeScript's type system is genuinely great. The ecosystem is mature. For most CRUD-heavy B2B SaaS work, it's the right tool.

But there are specific scenarios where I've felt the friction:

1. Long-running background processing

In my billing platform (Billy), I have a webhook inbox processor — raw Stripe events land in a table, a scheduled job picks them up, groups them by webhookId, and processes them sequentially per group. In NestJS I'm using BullMQ for this, which means Redis as a dependency, queue configuration, worker threads, retry logic wired manually.

In Go, a similar pattern can be expressed with goroutines and channels in ~50 lines, with zero external dependencies. Not always the right tradeoff — but the simplicity gap is real.

2. CPU-bound work

Node.js has a single-threaded event loop. If you're doing heavy computation — parsing, hashing, data transformation — you block the loop, or you ship it to a worker thread with all the overhead and complexity that entails. Go goroutines can run on multiple OS threads simultaneously. CPU-bound work just... works.

3. CLI tools and infrastructure code

Go compiles to a single static binary. No runtime, no node_modules, no version management hell. For my HookScope CLI (distributed via Homebrew), Go was the only sensible choice. Try distributing a NestJS CLI tool to developers who don't have Node installed.

4. When you want to reason about what's actually happening

This is the subtle one. Go's concurrency model is explicit. When you write go f(), you're saying "run this concurrently." When you communicate via channels, you can see data flow. In Node, the event loop abstraction is powerful but opaque — you often can't tell why something is slow without profiling.

Go's explicitness is sometimes more work. It's also clarity.

The Go Concurrency Model: What's Actually Under the Hood

Okay, here's where we go deep. I want to explain the Go scheduler in a way that gives you real intuition, not just vocabulary.

Goroutines Are Not OS Threads

This is the most important thing to understand.

An OS thread is expensive:

Stack size: typically 1–8 MB by default
Creation time: involves a syscall, kernel context switch
Scheduling: managed by the OS scheduler, which knows nothing about your program

A goroutine is cheap:

Initial stack size: 2–8 KB (grows dynamically as needed, up to 1 GB by default)
Creation time: a few hundred nanoseconds
Scheduling: managed by the Go runtime scheduler, which does know about your program

This is why you can run hundreds of thousands of goroutines without breaking a sweat, while running that many OS threads would immediately destroy your machine.

// This is completely fine in Go
func main() {
    for i := 0; i < 100_000; i++ {
        go func(id int) {
            time.Sleep(10 * time.Second)
            fmt.Printf("goroutine %d done\n", id)
        }(i)
    }
    time.Sleep(15 * time.Second)
}

Trying to do this with OS threads in C or Java? Good luck. You'd hit OS limits around 10,000 threads on most systems, and memory usage would be catastrophic.

The M:N Threading Model

Go uses what's called an M:N threading model: M goroutines are multiplexed across N OS threads.

Goroutines (M):    G1  G2  G3  G4  G5  G6  ...
                    |   |   |    \  |  /
OS Threads (N):    T1  T2  T3   T4
                    |   |   |    |
CPU Cores (P):     C1  C2  C3  C4

The key insight: N (OS threads) typically equals the number of logical CPU cores (controlled by GOMAXPROCS, defaults to runtime.NumCPU()). The runtime maps goroutines to threads dynamically.

The GMP Scheduler

Go's scheduler is built around three entities: G, M, and P.

G — Goroutine

A G represents a goroutine. It contains:

The goroutine's stack
Its current status (running, runnable, waiting, dead)
The program counter (where it is in execution)
A pointer to the function it's running

M — Machine (OS Thread)

An M represents an OS thread. It's the actual execution context provided by the operating system. Each M is associated with at most one P at a time.

When M is doing a blocking syscall (file I/O, network wait), Go detaches it from the P and lets P grab another M to keep executing goroutines. This is crucial — it's how Go avoids stalling the entire program on a single blocking call.

P — Processor

P is the most subtle concept. A P represents a "logical processor" — a context required to execute Go code. Each P has its own local run queue of goroutines waiting to be executed.

The number of Ps = GOMAXPROCS (defaults to CPU core count). This is your true parallelism ceiling: no matter how many goroutines you have, at most GOMAXPROCS goroutines run simultaneously.

P1 [local queue: G3, G7, G11] → executing G1 on M1
P2 [local queue: G4, G8]      → executing G2 on M2
P3 [local queue: G5, G9, G12] → executing G6 on M3
P4 [local queue: G10]         → executing G13 on M4

Global run queue: [G14, G15, G16, ...]

Work Stealing

Here's one of the smartest parts of the Go scheduler: work stealing.

If P1's local queue is empty and P2's queue has goroutines sitting idle, P1 will steal half of P2's goroutines. This happens automatically, without any programmer involvement.

Why does this matter? It means you don't need to manually balance work across goroutines. The scheduler does it. CPU time is efficiently utilized without coordination from your application code.

// You don't need to think about which CPU core handles which work
// The scheduler handles it
results := make(chan int, numWorkers)
for i := 0; i < numWorkers; i++ {
    go func(id int) {
        results <- doWork(id) // Go figures out where this runs
    }(i)
}

Goroutine Preemption — The Evolution

Early Go (pre-1.14) used cooperative scheduling: goroutines would yield only at specific points — function calls, channel operations, syscalls. This meant a goroutine doing a tight CPU-bound loop would starve other goroutines on the same thread:

// Pre-1.14: this could starve other goroutines on the same P
func infiniteLoop() {
    for {
        // No function call, no yield point
        // This goroutine never gives up the thread
    }
}

Go 1.14 introduced asynchronous preemption: the runtime sends a signal to OS threads, which interrupts goroutines at (almost) any point. This is similar to how OS schedulers work, but implemented at the Go runtime level.

The result: no more goroutine starvation from CPU-bound loops. The scheduler is now truly preemptive.

Channels: Communicating Instead of Sharing

Go's philosophy on concurrency comes from Tony Hoare's Communicating Sequential Processes (CSP), formalized in 1978.

The Go proverb that captures it:

"Do not communicate by sharing memory; instead, share memory by communicating."

This is a direct inversion of how most concurrent programming is taught. Traditional approach: you have shared state, you protect it with mutexes. Go's approach: you send data through channels.

What a Channel Actually Is

A channel is a typed conduit. Sending to a channel and receiving from a channel are synchronization points.

ch := make(chan int)    // unbuffered channel
ch := make(chan int, 5) // buffered channel, capacity 5

Unbuffered channel: sender blocks until receiver is ready. Receiver blocks until sender sends. This is a rendezvous — both parties must be present.

Buffered channel: sender blocks only when buffer is full. Receiver blocks only when buffer is empty. Decoupled, but bounded.

Under the hood, a channel is a hchan struct in the Go runtime:

hchan {
    qcount   uint           // number of elements in queue
    dataqsiz uint           // capacity of circular queue
    buf      unsafe.Pointer // pointer to circular queue array
    elemsize uint16
    closed   uint32
    sendq    waitq          // list of blocked senders
    recvq    waitq          // list of blocked receivers
    lock     mutex
}

When you send to a full buffered channel:

Your goroutine is added to sendq
The goroutine is parked (removed from the run queue)
When a receiver drains an element, it checks sendq, wakes the blocked sender, copies the data directly from the sender's stack to the receiver's

This is the key insight: Go minimizes data copying and context switching by letting goroutines park/unpark efficiently at channel boundaries.

Patterns I Actually Use

Fan-out: distribute work across workers

func fanOut(jobs <-chan Job, numWorkers int) <-chan Result {
    results := make(chan Result, numWorkers)

    var wg sync.WaitGroup
    for i := 0; i < numWorkers; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            for job := range jobs {
                results <- process(job)
            }
        }()
    }

    go func() {
        wg.Wait()
        close(results)
    }()

    return results
}

Pipeline: chain stages of processing

func pipeline(input <-chan RawEvent) <-chan ProcessedEvent {
    validated := validate(input)     // stage 1
    enriched := enrich(validated)    // stage 2
    return transform(enriched)       // stage 3
}

// Each stage is a goroutine consuming from one channel
// and producing to another. Backpressure is built in.

Cancellation with context

func processWithTimeout(ctx context.Context, id string) error {
    ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
    defer cancel()

    resultCh := make(chan Result, 1)
    go func() {
        resultCh <- doExpensiveWork(id)
    }()

    select {
    case result := <-resultCh:
        return handleResult(result)
    case <-ctx.Done():
        return ctx.Err() // deadline exceeded or cancelled
    }
}

The select statement is Go's concurrency control flow. It blocks until one of the cases is ready. If multiple are ready, it picks one at random (genuinely random — this is specified in the language spec to prevent starvation).

The `sync` Package: When Channels Aren't the Answer

I want to push back on something: channels are not always the right tool.

Go's standard library has sync.Mutex, sync.RWMutex, sync.WaitGroup, sync.Once, and sync/atomic. These exist because sometimes shared state with a mutex is simpler and more appropriate than channels.

Rule of thumb I follow:

Channel: when you're passing ownership of data, or coordinating between goroutines
Mutex: when you're protecting shared state that multiple goroutines need to read/write
atomic: when you need lock-free operations on simple values (counters, flags)

// This is fine — a mutex is clearer here than a channel
type Cache struct {
    mu    sync.RWMutex
    items map[string]Item
}

func (c *Cache) Get(key string) (Item, bool) {
    c.mu.RLock()
    defer c.mu.RUnlock()
    item, ok := c.items[key]
    return item, ok
}

func (c *Cache) Set(key string, item Item) {
    c.mu.Lock()
    defer c.mu.Unlock()
    c.items[key] = item
}

Don't force channels where they don't fit. The goal is clarity.

Common Pitfalls I've Hit (So You Don't Have To)

1. Goroutine Leaks

This is the #1 production issue with goroutines. A goroutine that's blocked forever on a channel no one will ever send to — it sits in memory, forever.

// BUG: if no one sends to ch, this goroutine leaks
go func() {
    val := <-ch  // blocks forever if ch is abandoned
    process(val)
}()

Fix: always use context.Context for cancellation, and design your channels so they can be closed or drained. Use goleak in tests to catch leaks.

go func() {
    select {
    case val := <-ch:
        process(val)
    case <-ctx.Done():
        return // clean exit
    }
}()

2. Closing a Channel from the Receiver

Only the sender should close a channel. If the receiver closes it and the sender tries to send, you get a panic.

// WRONG
func receiver(ch chan int) {
    val := <-ch
    close(ch) // dangerous if sender hasn't stopped
}

// RIGHT: sender closes
func sender(ch chan int) {
    defer close(ch)
    for _, v := range data {
        ch <- v
    }
}

3. The Loop Variable Capture Bug

Classic Go footgun (fixed in Go 1.22, but worth knowing for older codebases):

// BUG in Go < 1.22: all goroutines capture the same `i`
for i := 0; i < 5; i++ {
    go func() {
        fmt.Println(i) // likely prints 5 five times
    }()
}

// FIX (pre-1.22): pass as parameter
for i := 0; i < 5; i++ {
    go func(id int) {
        fmt.Println(id) // correct
    }(i)
}

Go 1.22 changed loop variable semantics so each iteration gets its own variable. But if you're maintaining older code, know this bug exists.

4. Unbounded Goroutine Spawning

// BUG: spawns a goroutine per request with no limit
http.HandleFunc("/work", func(w http.ResponseWriter, r *http.Request) {
    go doHeavyWork() // what if 10,000 requests hit simultaneously?
})

Fix: use a worker pool with a bounded channel:

workerPool := make(chan struct{}, 100) // max 100 concurrent

http.HandleFunc("/work", func(w http.ResponseWriter, r *http.Request) {
    workerPool <- struct{}{}        // acquire slot (blocks if full)
    go func() {
        defer func() { <-workerPool }() // release slot
        doHeavyWork()
    }()
})

5. Race Conditions — Use the Race Detector

Go ships with a built-in race detector. Use it:

go test -race ./...
go run -race main.go

It adds overhead (~2–20x slowdown), but catches races that would otherwise appear as subtle bugs in production. I run it in CI on every push. Non-negotiable.

Practical Architecture: The Outbox Pattern in Go

Let me show you something concrete. In Billy, I have an outbound webhook system — when things happen, I need to notify customers' endpoints reliably. Classic Outbox Pattern.

Here's how I'd implement the dispatcher in Go:

type OutboxDispatcher struct {
    db      *sql.DB
    client  *http.Client
    workers int
}

func (d *OutboxDispatcher) Run(ctx context.Context) error {
    jobs := make(chan OutboxEvent, d.workers*2)
    results := make(chan dispatchResult, d.workers*2)

    // Start worker pool
    var wg sync.WaitGroup
    for i := 0; i < d.workers; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            for event := range jobs {
                select {
                case results <- d.dispatch(ctx, event):
                case <-ctx.Done():
                    return
                }
            }
        }()
    }

    // Result processor
    go func() {
        for result := range results {
            if result.err != nil {
                d.markFailed(result.event, result.err)
                continue
            }
            d.markDelivered(result.event)
        }
    }()

    // Polling loop
    ticker := time.NewTicker(5 * time.Second)
    defer ticker.Stop()

    for {
        select {
        case <-ticker.C:
            events, err := d.fetchPending(ctx)
            if err != nil {
                log.Printf("fetch error: %v", err)
                continue
            }
            for _, event := range events {
                select {
                case jobs <- event:
                case <-ctx.Done():
                    close(jobs)
                    wg.Wait()
                    close(results)
                    return ctx.Err()
                }
            }
        case <-ctx.Done():
            close(jobs)
            wg.Wait()
            close(results)
            return ctx.Err()
        }
    }
}

What this gives you: bounded concurrency, clean shutdown via context cancellation, backpressure via buffered channels, and zero external dependencies. No Redis, no BullMQ, no queue broker.

Is this appropriate for every project? No. At scale you'd want persistent queues, dead-letter queues, observability. But for a mid-sized SaaS bootstrapped by one developer? The simplicity-to-reliability ratio is hard to beat.

Benchmarks: Why This Model Is Fast

Let me give you some numbers to anchor the intuition.

Context switch cost:

OS thread context switch: ~1–10 microseconds
Goroutine context switch: ~100–300 nanoseconds (~10-50x faster)

Memory per unit:

OS thread: 1–8 MB stack (fixed at creation)
Goroutine: 2–8 KB initial stack (grows as needed)

Maximum concurrent units (typical server):

OS threads: ~10,000 before memory pressure becomes critical
Goroutines: 100,000–1,000,000 routinely

This is why Go HTTP servers can handle massive connection counts without a thread-per-connection model. Each connection gets a goroutine; the runtime handles the rest.

A Go HTTP server handles each connection in a goroutine:

// This is essentially what net/http does internally
func (srv *Server) Serve(l net.Listener) error {
    for {
        conn, err := l.Accept()
        if err != nil { return err }
        go srv.handleConn(conn) // one goroutine per connection
    }
}

With OS threads, this model breaks at ~10K connections. With goroutines, it's standard practice at 100K+.

Where Go Falls Short (Being Honest)

No language post should end without the limitations.

Generics are still maturing. Go 1.18 added generics, but the ecosystem adoption is uneven. Type inference has gaps. Error messages with generic types can be cryptic.

Error handling is verbose. The if err != nil pattern gets tiresome. There's no ? operator like Rust, no .unwrap_or. You write a lot of error-checking boilerplate. Go proposals for better error handling have been discussed for years without resolution.

No inheritance, limited OOP patterns. Go has interfaces and embedding, not inheritance. Coming from TypeScript/Java, this requires a genuine mental shift. Sometimes it's liberating; sometimes you want to express something naturally hierarchical and Go makes you work for it.

The ecosystem is younger. npm has millions of packages. Go's ecosystem is smaller. For niche integrations or specific tooling, you might be writing it yourself.

Debugging goroutine dumps is rough. When your program crashes with 10,000 goroutines, the stack trace is biblical. Tools exist (pprof, trace), but the learning curve is real.

Summary: The Mental Model That Actually Helps

Here's what I want you to take away:

Goroutines are not threads. They're lightweight, managed by the Go runtime, and can number in the hundreds of thousands.
The GMP scheduler is the engine. G (goroutines), M (OS threads), P (logical processors). P count = true parallelism. Work stealing keeps CPUs fed without programmer effort.
Channels are about coordination, not just communication. The blocking semantics of unbuffered channels are a synchronization primitive. Use them to express when goroutines should rendezvous.
Channels are not always the answer. sync.Mutex exists for a reason. Choose the tool that makes the code clearest.
Goroutine leaks are the #1 production bug. Always think about how goroutines exit. Always use context cancellation. Always run with -race.
The simplicity is intentional. Go has very few concurrency primitives — goroutines, channels, select, sync. This is not a limitation. It's a design philosophy: small surface area, deep composability.

For the kind of backend work I do — billing systems, webhook processors, CLI tools, API servers — Go's concurrency model is genuinely better suited than anything I've used in Node.js. Not because Node is bad, but because Go's model is explicit, performant, and scales from a single goroutine to a distributed system with the same mental model.

If you're building backend systems and you haven't spent serious time with Go's concurrency, I think it's worth your time. Not to replace your current stack, but to expand how you think about structure, scheduling, and what's possible.

Building Billy (B2B billing SaaS), HookScope (webhook inspector), and a few other things publicly on X (@kubabuilds). Posts about architecture, Go, and the reality of solo SaaS development.

GDPR for Developers: What the Regulation Actually Means in Code

Kuba — Tue, 12 May 2026 08:16:33 +0000

Most articles about GDPR are written by lawyers for lawyers. This one is written by a developer for developers — with real code, real mistakes, and real consequences.

I've built several B2B SaaS products in Europe. Billing platforms, webhook inspectors, hotel management systems. Every single one had to deal with GDPR in some form. And every single time I talked to other developers about it, the conversation went one of two ways:

Either "we just added a cookie banner and called it done" — or pure panic, because someone realized they'd been copying production data into staging for two years.

Neither response is correct. GDPR is not a checkbox. It's a set of engineering constraints that affects your schema design, your logging, your test data, your third-party integrations, and your deletion logic. If you treat it as a legal problem, you'll handle it wrong. It's an engineering problem.

Let me walk you through what it actually means in code.

What GDPR Actually Cares About

Before writing any code, you need to understand what the regulation is trying to enforce. Strip away the legal language and you get six core principles:

Lawfulness — you need a legal basis to process personal data (consent, contract, legitimate interest, etc.)
Purpose limitation — you can only use data for the reason you collected it
Data minimisation — collect only what you actually need
Accuracy — keep data correct and up to date
Storage limitation — don't keep data longer than necessary
Integrity and confidentiality — protect the data technically

As a developer, principles 3, 5, and 6 hit your code directly. The rest are mostly product and legal decisions — but you're the one implementing them.

Mistake #1: Your Schema Collects Too Much

The most common GDPR violation I see in codebases is schema bloat. Developers add columns "because they might be useful later." Under GDPR, that's a problem — you're collecting data without a documented purpose.

Look at this typical user table:

CREATE TABLE users (
  id UUID PRIMARY KEY,
  email VARCHAR NOT NULL,
  first_name VARCHAR,
  last_name VARCHAR,
  phone VARCHAR,
  date_of_birth DATE,
  gender VARCHAR,
  address TEXT,
  city VARCHAR,
  country VARCHAR,
  ip_address VARCHAR,
  user_agent TEXT,
  referral_source VARCHAR,
  linkedin_url VARCHAR,
  twitter_handle VARCHAR,
  annual_income INTEGER,
  created_at TIMESTAMP,
  updated_at TIMESTAMP
);

Half of those columns are probably unused. annual_income? linkedin_url? If you're not actively using these fields in your application logic, you have no legal basis to store them.

The fix: for every column, document the purpose. If you can't write one sentence explaining why you need it, drop it. This is called a Record of Processing Activities (ROPA) — GDPR Article 30 requires you to maintain one.

A practical way to do this in a codebase is to annotate your entity definitions:

export type UserEntityProps = {
  id: string;
  email: string;           // purpose: authentication, transactional emails
  firstName: string;       // purpose: personalization, invoicing
  lastName: string;        // purpose: invoicing
  createdAt: Date;
  updatedAt: Date;
  // REMOVED: phone — not used in any feature
  // REMOVED: dateOfBirth — collected at signup but never used
  // REMOVED: gender — collected "for analytics" but analytics never built
};

This looks obvious written down. But I've seen production databases with columns that hadn't been read by any query in over a year.

Mistake #2: Your Staging Database Has Real User Data

This is the one that gets companies fined.

Your production database has real personal data — emails, names, payment info, addresses. Your staging environment needs realistic data to test against. So you dump production and restore it to staging. Quick, easy, and a GDPR violation.

Staging environments typically have:

Weaker access controls
More people with access (junior devs, contractors, QA)
Logs that go to less secure places
Less monitoring

The moment real PII lands in your staging database, you've created an unlawful processing activity. You're using data beyond its original purpose, storing it in an environment that wasn't disclosed to users, and likely sharing it with people who have no business seeing it.

The fix has two parts.

First, never let production PII leave production without transformation. Before any data moves to a non-production environment, it must be anonymized or pseudonymized.

Second, automate this. Manual anonymization is error-prone. Someone forgets a table. Someone adds a new column with PII and nobody updates the anonymization script. Someone copies the DB "just this once" because they need to debug something urgently.

The correct solution is a tool that:

Understands your schema
Knows which columns contain PII (by name and by value pattern)
Applies masking rules automatically and consistently
Runs in your CI/CD pipeline so staging is refreshed automatically

I'll come back to this in detail later. First, let's talk about deletion.

Mistake #3: You Don't Actually Delete Data

GDPR Article 17 gives users the right to erasure — commonly called "the right to be forgotten." When a user requests deletion, you must delete their personal data.

Most applications implement this wrong. They set a deleted_at timestamp and call it done:

async deleteUser(userId: string): Promise<void> {
  await this.userRepository.update(userId, {
    deletedAt: new Date(),
  });
}

This is not deletion. The data is still there. You've just hidden it from your UI. The user's email, name, and everything else is sitting in your database, fully readable by anyone with database access.

True erasure means different things for different data types:

Structured PII in your own database — overwrite with anonymized values, don't just soft-delete:

async eraseUser(userId: string): Promise<void> {
  await this.userRepository.update(userId, {
    email: `deleted-${userId}@erased.invalid`,
    firstName: 'Deleted',
    lastName: 'User',
    phone: null,
    address: null,
    deletedAt: new Date(),
    erasedAt: new Date(),
  });
}

Notice: keep the row with a tombstone. You need to preserve the id for referential integrity and the erasedAt timestamp to prove compliance. You just remove all the personal data from it.

Backups — this is the hard one. Your production backup from last night contains the user's real data. GDPR allows a reasonable retention period for backups (typically 30-90 days, document your policy). After that window, the backup must be deleted too. You can't restore a 2-year-old backup and bring deleted users back to life.

Third-party services — if you sent the user's data to Mailchimp, Mixpanel, Intercom, or any other SaaS, you need to trigger deletion there too. Build a checklist, ideally automated:

async handleUserErasureRequest(userId: string): Promise<void> {
  const user = await this.userRepository.findById(userId);

  // 1. Erase in your own database
  await this.eraseUser(userId);

  // 2. Remove from email marketing
  await this.mailchimpService.deleteContact(user.email);

  // 3. Remove from analytics
  await this.mixpanelService.deleteUser(user.id);

  // 4. Remove from support tool
  await this.intercomService.deleteContact(user.email);

  // 5. Log the erasure for compliance
  await this.auditLogService.log({
    event: 'USER_ERASURE_COMPLETED',
    userId,
    thirdPartyServices: ['mailchimp', 'mixpanel', 'intercom'],
    completedAt: new Date(),
  });
}

That audit log entry is not optional — it's your proof that you complied with the request.

Mistake #4: No Audit Trail

GDPR requires you to be able to demonstrate compliance. "We handle data properly" is not enough — you need to prove it.

This means audit logging for every significant operation involving personal data:

Who accessed the data
What they did with it
When
From where (IP, system)

Most applications log errors. Few log data access. Build audit logging as a first-class concern, not an afterthought.

In a NestJS application, a clean approach is using a request context to capture the actor once per request and propagate it through the service layer:

// Captured once in the auth guard
@Injectable()
export class AuthGuard implements CanActivate {
  constructor(private readonly cls: ClsService) {}

  canActivate(context: ExecutionContext): boolean {
    const request = context.switchToHttp().getRequest();

    this.cls.set('userId', request.user.id);
    this.cls.set('ip', request.ip);
    this.cls.set('userAgent', request.headers['user-agent']);

    return true;
  }
}

// Used anywhere in the service layer
@Injectable()
export class AuditLogService {
  constructor(private readonly cls: ClsService) {}

  async log(event: string, metadata?: Record<string, unknown>): Promise<void> {
    await this.auditLogRepository.create({
      event,
      userId: this.cls.get('userId'),
      ip: this.cls.get('ip'),
      metadata,
      createdAt: new Date(),
    });
  }
}

Now any service can call auditLogService.log('USER_PROFILE_VIEWED') without manually threading the request context through every function call.

What events should you log? At minimum:

User login and logout
Password changes
Personal data exports (when a user downloads their data)
Erasure requests and completions
Admin accessing user data
Bulk data operations

Mistake #5: Storing Sensitive Data in Logs

Your application logs are not GDPR-compliant by default. Every time you log a user object for debugging, you're potentially writing PII to log files that:

Get shipped to Datadog, Papertrail, or another SaaS with their own retention policies
Are accessible to more people than your database
May be retained indefinitely

// BAD — logs full user object including PII
this.logger.debug('Processing order', { user, order });

// GOOD — log only non-PII identifiers
this.logger.debug('Processing order', { 
  userId: user.id, 
  orderId: order.id,
  plan: user.plan 
});

This sounds minor but it's a systematic issue. One console.log(req.body) in the wrong place can write email addresses, payment info, or passwords into your log aggregator.

The rule: never log objects that contain PII. Log IDs and non-sensitive fields only. If you need to debug a specific user issue, do it through controlled database queries with proper access controls — not by making their data appear in logs.

Mistake #6: Ignoring Data Retention

You don't need to keep data forever. Under GDPR's storage limitation principle, you should define how long you keep each type of data and actually enforce it.

Common retention periods (document these in your privacy policy):

Data type	Typical retention	Reason
Active user account	Until erasure request	Contract
Inactive account	2 years after last login	Legitimate interest
Payment records	7 years	Tax law (overrides GDPR)
Audit logs	1-3 years	Compliance
Marketing emails	Until unsubscribe	Consent
Server logs	30-90 days	Security

Enforce this in code, not in policy documents. A nightly job that runs retention cleanup:

@Cron('0 2 * * *') // 2am daily
async runRetentionCleanup(): Promise<void> {
  const twoYearsAgo = subYears(new Date(), 2);

  // Find inactive accounts
  const inactiveUsers = await this.userRepository.find({
    where: {
      lastLoginAt: LessThan(twoYearsAgo),
      erasedAt: null,
    },
  });

  for (const user of inactiveUsers) {
    // Don't delete — anonymize (preserve referential integrity)
    await this.eraseUser(user.id);
    await this.auditLogService.log('USER_AUTO_ERASED_RETENTION_POLICY', {
      userId: user.id,
      reason: 'inactive_2_years',
    });
  }
}

Handling Test Data Properly

Back to the staging problem. Here's what a proper test data pipeline looks like.

You need a process that runs automatically and produces anonymized data your team can actually use. The process has two phases: schema analysis and data transformation.

Schema analysis — inspect your database schema and identify PII columns. Detection works at two levels: column name patterns and value patterns.

Column name heuristics catch obvious cases:

const PII_COLUMN_PATTERNS = [
  /^email$/i,
  /^(first|last|full|display)_?name$/i,
  /^phone(_number)?$/i,
  /^(ip|ip_address|ipaddr)$/i,
  /^(address|street|city|zip|postal)(_.*)?$/i,
  /^(dob|date_of_birth|birth_date)$/i,
  /^(ssn|pesel|national_id)$/i,
  /^(iban|account_number)$/i,
  /^password(_hash)?$/i,
  /^(api_key|secret|token)$/i,
];

function detectPiiByColumnName(columnName: string): boolean {
  return PII_COLUMN_PATTERNS.some(pattern => pattern.test(columnName));
}

Value pattern sampling catches columns with non-obvious names:

const EMAIL_REGEX = /^[^\s@]+@[^\s@]+\.[^\s@]+$/;
const PHONE_REGEX = /^\+?[\d\s\-().]{7,}$/;
const IP_REGEX = /^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$/;

async function detectPiiByValues(
  sampleValues: string[]
): Promise<PiiType | null> {
  const nonNull = sampleValues.filter(Boolean);
  if (nonNull.length === 0) return null;

  const emailMatches = nonNull.filter(v => EMAIL_REGEX.test(v)).length;
  if (emailMatches / nonNull.length > 0.8) return 'email';

  const phoneMatches = nonNull.filter(v => PHONE_REGEX.test(v)).length;
  if (phoneMatches / nonNull.length > 0.8) return 'phone';

  const ipMatches = nonNull.filter(v => IP_REGEX.test(v)).length;
  if (ipMatches / nonNull.length > 0.8) return 'ip_address';

  return null;
}

Data transformation — apply the right strategy to each column based on its type and your configuration:

type MaskingStrategy = 
  | 'fake'      // replace with realistic fake value
  | 'mask'      // partially obscure (j**@g*****.com)
  | 'scramble'  // shuffle values between rows
  | 'preserve'  // keep original
  | 'nullify'   // set to NULL
  | 'fixed';    // always set to specific value

function applyStrategy(
  value: string | null,
  strategy: MaskingStrategy,
  fakerType?: string,
  fixedValue?: string,
): string | null {
  switch (strategy) {
    case 'fake':
      return generateFakeValue(fakerType ?? 'word');
    case 'mask':
      return maskValue(value);
    case 'preserve':
      return value;
    case 'nullify':
      return null;
    case 'fixed':
      return fixedValue ?? null;
    // 'scramble' is handled at the batch level, not per-row
  }
}

function maskValue(value: string | null): string | null {
  if (!value) return null;
  if (EMAIL_REGEX.test(value)) {
    const [local, domain] = value.split('@');
    return `${local[0]}**@${domain[0]}*****.${domain.split('.').pop()}`;
  }
  // Generic masking for other types
  return value[0] + '*'.repeat(value.length - 2) + value[value.length - 1];
}

The scramble strategy deserves special attention — it preserves statistical distribution while destroying the link between a value and a specific person. For salary data, for example, you want to keep the realistic range of values (€30k-€150k) but not associate €150k with the actual person who earns it:

function scrambleColumn(values: (string | null)[]): (string | null)[] {
  const nonNull = values.filter((v): v is string => v !== null);
  // Fisher-Yates shuffle
  for (let i = nonNull.length - 1; i > 0; i--) {
    const j = Math.floor(Math.random() * (i + 1));
    [nonNull[i], nonNull[j]] = [nonNull[j], nonNull[i]];
  }
  // Put back in positions where original wasn't null
  let idx = 0;
  return values.map(v => v === null ? null : nonNull[idx++]);
}

The critical constraint in all of this: foreign key integrity must be preserved. If orders.user_id references users.id, and you're subsetting 10% of your users, you can only include orders that belong to those users. This requires building a dependency graph of your schema and processing tables in topological order.

What Pseudonymization Actually Means

GDPR distinguishes between anonymization and pseudonymization. They're not the same thing and the difference matters legally.

Anonymized data — data that cannot be linked back to an individual, even with additional information. GDPR doesn't apply to truly anonymized data. Aggregated statistics ("we have 10,000 users in Germany") are anonymized.

Pseudonymized data — data where direct identifiers are replaced with artificial identifiers (like UUIDs), but where re-identification is theoretically possible if you have the mapping. GDPR still applies to pseudonymized data — it's just considered lower risk.

When you replace a user's email with a randomly generated UUID in your analytics pipeline, that's pseudonymization. The UUID is meaningless on its own, but your mapping table could reconstruct the original email.

Practically, for test data purposes, you want anonymization — fake data generated fresh, with no mapping back to real individuals. The fake strategy above produces anonymized data. The mask strategy produces pseudonymized data (the structure is preserved, re-identification is theoretically possible for emails with few characters).

Encryption: What to Actually Encrypt

"Encrypt sensitive data" is common advice. Less common is guidance on what that actually means in practice.

Encrypt at rest: enable full-disk encryption on your database server. If you're on AWS RDS, GCP Cloud SQL, or any major managed database — turn this on. It's usually one checkbox and it covers your entire database.

Encrypt in transit: TLS for all connections. Your application connecting to your database should use TLS. Your users connecting to your API should use HTTPS. No exceptions.

Column-level encryption for specific high-risk fields (credit card numbers, health data, government IDs). Use a proper encryption library, not a homebrew solution. Store the encryption key separately from the data — ideally in a key management service (AWS KMS, HashiCorp Vault).

import { createCipheriv, createDecipheriv, randomBytes } from 'crypto';

// Don't do this in production — use a proper KMS
const ENCRYPTION_KEY = Buffer.from(process.env.FIELD_ENCRYPTION_KEY, 'hex');
const ALGORITHM = 'aes-256-gcm';

function encrypt(plaintext: string): string {
  const iv = randomBytes(16);
  const cipher = createCipheriv(ALGORITHM, ENCRYPTION_KEY, iv);

  const encrypted = Buffer.concat([
    cipher.update(plaintext, 'utf8'),
    cipher.final(),
  ]);

  const authTag = cipher.getAuthTag();

  // Store iv + authTag + encrypted together
  return Buffer.concat([iv, authTag, encrypted]).toString('base64');
}

function decrypt(ciphertext: string): string {
  const buffer = Buffer.from(ciphertext, 'base64');

  const iv = buffer.subarray(0, 16);
  const authTag = buffer.subarray(16, 32);
  const encrypted = buffer.subarray(32);

  const decipher = createDecipheriv(ALGORITHM, ENCRYPTION_KEY, iv);
  decipher.setAuthTag(authTag);

  return decipher.update(encrypted) + decipher.final('utf8');
}

What not to bother with: encrypting non-sensitive fields. Encrypting your entire users table including created_at and country adds overhead with zero compliance benefit. Be surgical — encrypt what's actually sensitive.

The GDPR Audit Checklist for Your Next Code Review

Before shipping any feature that touches personal data, go through this list:

[ ] Does this feature collect new personal data? If yes, is the purpose documented?
[ ] Is there a legal basis for processing this data?
[ ] Is this data ever logged? If yes, is it stripped of PII?
[ ] How does a user request deletion of this data?
[ ] Does deletion cascade to third-party services?
[ ] Is there a retention period defined for this data?
[ ] If this data goes to staging — is the pipeline anonymized?
[ ] Does accessing this data generate an audit log entry?
[ ] Is PII encrypted at rest and in transit?
[ ] Is there a data minimization review — do you actually need all the fields you're collecting?

The Bottom Line

GDPR compliance is not a single feature you build. It's a set of constraints that runs through your entire architecture — your schema design, your logging, your test data pipeline, your deletion logic, your third-party integrations.

The developers who handle it well treat it like any other non-functional requirement: they make decisions about it upfront, build the right abstractions early, and enforce it systematically rather than scrambling when a user submits an erasure request.

The developers who handle it poorly add a cookie banner, copy production to staging, and hope nobody notices.

In Europe right now, regulators are increasingly focused on developer-level violations — not just consent banners, but actual data handling practices. GDPR fines hit €1.2 billion in 2024. The technical implementation is no longer someone else's problem.

Build it right from the start. It's genuinely easier than fixing it later.

How to Safely Import Thousands of Records from an External API in NestJS

Kuba — Thu, 16 Apr 2026 11:56:53 +0000

When building a SaaS product that integrates with a third-party API like Stripe, you will eventually face a problem that seems simple on the surface: how do you import existing data when a user connects their account?

In theory it sounds trivial. Fetch the data, save it to your database, done. In practice, you are dealing with paginated APIs, rate limits, foreign key dependencies, partial failures, and users staring at a spinner with no idea what is happening.

I ran into all of these problems while building Billy — a billing infrastructure layer for developers that lets you manage Stripe subscriptions, tokens, and seat-based billing through a simple SDK. When a developer connects their existing Stripe account, Billy needs to import all their products, prices, customers, and subscriptions automatically. Some accounts have tens of thousands of records. The naive approach falls apart immediately.

This article walks through the architecture I ended up with — from tracking sync state to paginating safely, handling failures gracefully, and surfacing progress to the user in real time. Everything is in NestJS with BullMQ and MikroORM, but the concepts apply to any stack.

Why the Naive Approach Fails

Let me start with what not to do, because understanding the failure modes makes the solution obvious.

The first instinct is something like this:

async function syncCustomers(stripeSecretKey: string) {
  const stripe = new Stripe(stripeSecretKey)
  const customers = await stripe.customers.list({ limit: 100 })

  for (const customer of customers.data) {
    await customerRepository.save(mapToCustomer(customer))
  }
}

This has at least five problems.

It only fetches the first page. Stripe paginates at 100 items maximum per request. If your user has 5,000 customers, this function imports 100 and silently drops the rest. No error, no warning.

It runs synchronously in the request cycle. If you call this from an HTTP endpoint, the request will time out for any account with significant data. Even if it doesn't time out, you are blocking a Node.js worker thread doing I/O for minutes.

It has no progress tracking. The user sees a loading spinner and has no idea if something is happening or if the app is broken. This is a terrible experience.

It is not idempotent. If something fails halfway through and you re-run it, you get duplicate records or errors from unique constraint violations.

It ignores dependency order. Subscriptions reference customers and prices. Prices reference products. If you import subscriptions before customers exist in your database, you get foreign key violations and the whole thing crashes.

Each of these is a real problem in production. Let me walk through how to solve all of them.

The Architecture Overview

The solution has four main components working together:

A sync log entity that tracks the state of each sync operation with full history
A BullMQ background job that runs the sync outside the HTTP request cycle
An ordered execution pipeline that respects data dependencies
Idempotent upsert operations that make re-runs safe

Let me go through each one in detail.

Designing the Sync Log

The first mistake I almost made was storing sync status as fields on the Project entity itself:

// Don't do this
type Project = {
  // ...
  syncStatus: 'never' | 'pending' | 'running' | 'completed' | 'failed'
  lastSyncedAt: Date | null
  syncError: string | null
  customersCount: number
  subscriptionsCount: number
}

This works for the simplest case but breaks down quickly. You lose history — if a sync fails and you re-run it, you overwrite the previous state. You can't track which specific step failed. You can't show granular progress. And your Project entity grows indefinitely as you add more sync types.

The better approach is a separate ProjectSyncLog entity where each sync type gets its own log entry:

type SyncType =
  | 'sync_products'
  | 'sync_prices'
  | 'sync_coupons'
  | 'sync_promotion_codes'
  | 'sync_customers'
  | 'sync_subscriptions'
  | 'full_sync'

type ProjectSyncLog = {
  id: string
  projectId: string
  type: SyncType
  status: 'pending' | 'running' | 'completed' | 'failed'
  currentStep: number
  totalSteps: number | null
  syncedCount: number
  failedCount: number
  error: string | null
  startedAt: Date
  completedAt: Date | null
}

A full sync creates six log entries — one for each type. If products and prices succeed but customers fail, you know exactly what happened and you can re-run just sync_customers without touching anything else.

The Project entity only keeps a lightweight summary for quick access:

type Project = {
  // ...
  lastSyncedAt: Date | null
  syncStatus: 'never' | 'pending' | 'running' | 'completed' | 'failed'
}

This separation of concerns makes the system much easier to reason about. The log is the source of truth for sync state. The project fields are just a convenience cache.

Running Sync as a Background Job

The sync must not run in the HTTP request cycle. When a user creates a project and connects their Stripe account, you want to respond immediately and do the heavy lifting in the background.

With BullMQ in NestJS this is straightforward:

// In CreateProjectHandler
@CommandHandler(CreateProjectCommand)
export class CreateProjectHandler implements ICommandHandler<CreateProjectCommand> {
  constructor(
    private readonly projectRepository: ProjectRepositoryPort,
    @InjectQueue('project-sync') private readonly syncQueue: Queue,
  ) {}

  async execute(command: CreateProjectCommand) {
    const project = Project.create(command.payload)
    await this.projectRepository.create(project)

    // Dispatch sync job and return immediately
    await this.syncQueue.add(
      'full-sync',
      {
        projectId: project.id.value,
        stripeSecretKey: project.stripeSecretKey,
      },
      {
        attempts: 3,
        backoff: {
          type: 'exponential',
          delay: 5000,
        },
      }
    )

    return { data: project.id.value }
  }
}

The user gets a response in milliseconds. The sync job runs whenever a worker picks it up.

The job processor handles the actual orchestration:

@Processor('project-sync')
export class ProjectSyncProcessor {
  constructor(
    private readonly commandBus: CommandBus,
    private readonly projectRepository: ProjectRepositoryPort,
  ) {}

  @Process('full-sync')
  async handleFullSync(job: Job<{ projectId: string; stripeSecretKey: string }>) {
    const { projectId, stripeSecretKey } = job.data

    await this.projectRepository.updateSyncStatus(projectId, 'running')

    const syncTypes: SyncType[] = [
      'sync_products',
      'sync_prices',
      'sync_coupons',
      'sync_promotion_codes',
      'sync_customers',
      'sync_subscriptions',
    ]

    let hasFailures = false

    for (const type of syncTypes) {
      try {
        await this.commandBus.execute(
          new RunSyncTypeCommand({ projectId, stripeSecretKey, type })
        )
      } catch (error) {
        hasFailures = true
        // Log the failure but continue with next sync type
        // Each type manages its own log entry
      }
    }

    const finalStatus = hasFailures ? 'failed' : 'completed'
    await this.projectRepository.updateSyncStatus(projectId, finalStatus)
  }
}

Notice the sequential for loop rather than Promise.all. This is intentional. Prices reference products by their Stripe product ID. Subscriptions reference both customers and prices. Running them in parallel will cause foreign key violations for any user who has these relationships in their Stripe data — which is basically everyone.

The dependency order is:

Products → Prices → Coupons → Promotion Codes → Customers → Subscriptions

Always sync in this order and you will never hit a dependency issue.

Paginating Through Stripe Without Loading Everything Into Memory

Stripe uses cursor-based pagination. Each list endpoint accepts limit (max 100) and starting_after (the ID of the last item from the previous page). You keep fetching until has_more is false.

The cleanest way to handle this in TypeScript is with an async generator:

async function* paginateStripe<T extends { id: string }>(
  fetchPage: (cursor?: string) => Promise<Stripe.ApiList<T>>
): AsyncGenerator<T> {
  let cursor: string | undefined = undefined

  while (true) {
    const page = await fetchPage(cursor)

    for (const item of page.data) {
      yield item
    }

    if (!page.has_more) break

    cursor = page.data[page.data.length - 1].id
  }
}

An async generator yields items one at a time as pages arrive. You never load the entire dataset into memory — you process each item as it comes. For a user with 50,000 customers, this is the difference between 50MB of memory usage and 500MB.

Usage is clean and readable:

for await (const customer of paginateStripe((cursor) =>
  stripe.customers.list({
    limit: 100,
    starting_after: cursor,
  })
)) {
  await this.upsertCustomer(customer, projectId)
  syncLog.syncedCount++
}

You can also add rate limit handling inside the generator. Stripe allows 100 read requests per second on most endpoints, but if you are running multiple syncs in parallel you may hit limits:

async function* paginateStripe<T extends { id: string }>(
  fetchPage: (cursor?: string) => Promise<Stripe.ApiList<T>>,
  delayMs = 50
): AsyncGenerator<T> {
  let cursor: string | undefined = undefined

  while (true) {
    const page = await fetchPage(cursor)

    for (const item of page.data) {
      yield item
    }

    if (!page.has_more) break

    cursor = page.data[page.data.length - 1].id

    // Small delay between pages to avoid rate limits
    await new Promise(resolve => setTimeout(resolve, delayMs))
  }
}

A 50ms delay between pages adds about 5 seconds per 1,000 items — acceptable for a background job, and it keeps you well within Stripe's rate limits.

Making Every Write Idempotent

Background jobs can fail and be retried. Network timeouts happen. Your database can be temporarily unavailable. If a sync crashes at customer 3,000 and restarts from the beginning, you need writes to be safe to repeat.

The solution is upsert operations keyed on the external ID — in this case the Stripe ID:

async upsertCustomer(stripeCustomer: Stripe.Customer, projectId: string): Promise<void> {
  const existing = await this.customerRepository.findByStripeId(stripeCustomer.id)

  if (existing) {
    existing.update({
      email: stripeCustomer.email ?? null,
      name: stripeCustomer.name ?? null,
      phone: stripeCustomer.phone ?? null,
      metadata: stripeCustomer.metadata ?? null,
      description: stripeCustomer.description ?? null,
    })
    await this.customerRepository.save(existing)
    return
  }

  const customer = Customer.create({
    projectId,
    stripeCustomerId: stripeCustomer.id,
    externalUserId: stripeCustomer.metadata?.billyExternalUserId ?? stripeCustomer.id,
    email: stripeCustomer.email ?? '',
    name: stripeCustomer.name ?? '',
    phone: stripeCustomer.phone ?? null,
    metadata: stripeCustomer.metadata ?? null,
    status: 'active',
  })

  await this.customerRepository.create(customer)
}

If the record exists, update it. If not, create it. Running this function ten times with the same input produces the same result as running it once. This is the idempotency guarantee you need.

One important detail: for customers coming from an existing Stripe account that was not previously connected to Billy, there is no billyExternalUserId in the metadata. In that case I fall back to using the Stripe customer ID as the external user ID. The developer can update this later once they integrate the SDK into their application.

The same pattern applies to products, prices, and subscriptions. Always check by Stripe ID first before creating.

Handling Failures Gracefully

Each sync type runs in its own try/catch block and maintains its own log entry. This isolation means a failure in one type does not cascade to others:

@CommandHandler(RunSyncTypeCommand)
export class RunSyncTypeHandler implements ICommandHandler<RunSyncTypeCommand> {
  async execute(command: RunSyncTypeCommand): Promise<void> {
    const { projectId, stripeSecretKey, type } = command.payload

    const syncLog = await this.syncLogRepository.create({
      projectId,
      type,
      status: 'running',
      syncedCount: 0,
      failedCount: 0,
      currentStep: 0,
      startedAt: new Date(),
    })

    try {
      switch (type) {
        case 'sync_products':
          await this.syncProducts(projectId, stripeSecretKey, syncLog)
          break
        case 'sync_prices':
          await this.syncPrices(projectId, stripeSecretKey, syncLog)
          break
        case 'sync_customers':
          await this.syncCustomers(projectId, stripeSecretKey, syncLog)
          break
        case 'sync_subscriptions':
          await this.syncSubscriptions(projectId, stripeSecretKey, syncLog)
          break
        // ... other types
      }

      syncLog.status = 'completed'
      syncLog.completedAt = new Date()
    } catch (error) {
      syncLog.status = 'failed'
      syncLog.error = error.message
      syncLog.completedAt = new Date()
      throw error // Re-throw so the processor knows this type failed
    } finally {
      await this.syncLogRepository.save(syncLog)
    }
  }
}

Individual item failures are handled differently from type-level failures. If one customer fails to import due to bad data, you probably want to log it and continue rather than abort the entire customer sync:

async syncCustomers(projectId: string, stripeSecretKey: string, syncLog: ProjectSyncLog) {
  const stripe = this.stripeClientFactory.create(stripeSecretKey)

  for await (const customer of paginateStripe((cursor) =>
    stripe.customers.list({ limit: 100, starting_after: cursor })
  )) {
    try {
      await this.upsertCustomer(customer, projectId)
      syncLog.syncedCount++
    } catch (error) {
      syncLog.failedCount++
      // Log the individual failure but keep going
      this.logger.warn(`Failed to import customer ${customer.id}: ${error.message}`)
    }

    // Persist progress every 100 items
    if ((syncLog.syncedCount + syncLog.failedCount) % 100 === 0) {
      syncLog.currentStep = syncLog.syncedCount
      await this.syncLogRepository.save(syncLog)
    }
  }
}

This way a user with 10 customers that have malformed metadata does not block the import of their other 4,990 customers. The failedCount field tells them exactly how many items had problems.

Surfacing Progress to the User

Real-time progress is not just a nice-to-have — it is the difference between a user thinking the feature is broken and a user trusting the system.

I poll the sync status endpoint from the frontend using React Query's refetchInterval. The key insight is that you only poll while a sync is running — once it completes or fails you stop polling automatically:

const { data: syncStatus } = useProjectSyncStatus(projectId, {
  refetchInterval: (query) => {
    const status = query.state.data?.syncLogs?.some(
      log => log.status === 'running' || log.status === 'pending'
    )
    return status ? 2000 : false
  }
})

The backend endpoint returns the current state of all sync logs:

@Get(':projectId/sync-status')
async getSyncStatus(@Param('projectId') projectId: string) {
  const syncLogs = await this.syncLogRepository.findByProjectId(projectId)
  const project = await this.projectRepository.findById(projectId)

  return {
    syncStatus: project.syncStatus,
    syncLogs: syncLogs.map(log => ({
      type: log.type,
      status: log.status,
      syncedCount: log.syncedCount,
      failedCount: log.failedCount,
      currentStep: log.currentStep,
      error: log.error,
    }))
  }
}

In the UI I show a list of sync operations with individual progress for each:
✓ Products 12 synced
✓ Prices 31 synced
⟳ Customers 1,247 / syncing...
Subscriptions waiting

This gives the user complete visibility. They know the system is working, they know how much data has been imported, and they can see immediately if something goes wrong.

Allowing Re-Sync

The final piece is letting users trigger a re-sync manually. This is important for two reasons: recovering from failures, and keeping data fresh if a user has added new records in Stripe after the initial import.

The re-sync endpoint accepts an optional type parameter:

@Post(':projectId/sync')
async triggerSync(
  @Param('projectId') projectId: string,
  @Body() dto: { type?: SyncType }
) {
  const project = await this.projectRepository.findById(projectId)

  await this.syncQueue.add('full-sync', {
    projectId,
    stripeSecretKey: project.stripeSecretKey,
    types: dto.type ? [dto.type] : undefined, // undefined = all types
  })

  return { queued: true }
}

If you pass a specific type, only that sync runs. If you pass nothing, the full sync runs. The idempotent upserts mean this is always safe to trigger — you will never end up with duplicate data.

The Result

When a developer connects their Stripe account to Billy, their entire billing history appears within minutes. Products, prices, customers, and subscriptions — all imported automatically, with real-time progress visible in the dashboard.

No manual migration scripts. No CSV exports. No "please re-create your products in our system". Just connect and go.

This turned out to be one of the highest-leverage features in the product. The biggest objection to adopting new billing infrastructure is always "I already have data in Stripe". With automatic sync, that objection disappears.

Key Takeaways

Use a separate log entity per sync operation rather than fields on the parent entity. You get history, granular failure tracking, and the ability to re-run individual steps.

Run sync types sequentially in dependency order. Products before prices before customers before subscriptions. Parallel execution will give you foreign key violations.

Use async generators to paginate external APIs. You process items as they arrive rather than loading everything into memory.

Make every write idempotent by upserting on the external ID. Jobs will be retried. Network will fail. Your writes need to be safe to repeat.

Persist progress every N items rather than every item. Writing to the database on every record will slow your sync significantly. Every 100 items is a reasonable balance.

Show real-time progress to the user. Polling with React Query's refetchInterval is simple and effective. Users who can see progress trust the system. Users staring at a spinner do not.

Isolate failures per sync type so a problem with one type does not block the others. Track individual item failures separately from type-level failures.

If you are building a SaaS and want billing infrastructure that handles subscriptions, tokens, and seat-based billing without rebuilding Stripe logic from scratch — I am building Billy for exactly this. Early access coming soon.

CQRS + Event Sourcing in NestJS: A Real-World Deep Dive

Kuba — Thu, 09 Apr 2026 11:42:22 +0000

If you've ever worked on a system where "update this record" becomes a landmine — bugs appearing from nowhere, audit logs you can't trust, state that's impossible to replay — you've felt the pain that CQRS and Event Sourcing were designed to solve.

This isn't a "what is CQRS" post. There are a hundred of those. This is a how to actually implement it in NestJS, including the parts nobody talks about: the projections, the event store, the versioning headaches, and what you gain (and lose) by going all-in.

I've been using this stack in NoteCore, a local-first developer knowledge base I'm building, and the patterns here come directly from production code.

Let's go.

The Problem With Traditional CRUD

Before diving into code, let's establish why this matters.

In classic CRUD, your write and read models are the same table:

// The classic way
async updateNote(id: string, dto: UpdateNoteDto) {
  return this.noteRepository.update(id, {
    title: "dto.title,"
    content: dto.content,
    updatedAt: new Date(),
  });
}

This looks fine until:

You need to know what changed (audit trail, undo/redo)
Your reads and writes have completely different shapes (dashboard aggregates vs. raw entities)
You're building a distributed system where multiple services need to react to changes
You want to replay history to rebuild state or add new projections retroactively

CQRS + Event Sourcing solves all of this — but it comes with a real cost in complexity. I'll be honest about both sides.

The Big Picture

Before any code, here's the mental model:

Command (intent)
    ↓
Command Handler (validates, loads aggregate, applies business logic)
    ↓
Domain Event (fact: "this happened")
    ↓
Event Store (append-only log of all events)
    ↓
Projections (rebuild read models from events)
    ↓
Query Handler (reads from projected read models)

Commands represent intent: CreateNote, ArchiveNote, AddTag.

Events are immutable facts: NoteCreated, NoteArchived, TagAdded.

The event store is your source of truth — not the DB table.

Projections are derived views built by replaying events.

The key insight: you never update state directly. You emit events, and state is always derived by replaying those events.

Project Structure

Here's how I structure a CQRS module:

src/
└── notes/
    ├── application/
    │   ├── commands/
    │   │   ├── create-note/
    │   │   │   ├── create-note.command.ts
    │   │   │   └── create-note.handler.ts
    │   │   └── archive-note/
    │   │       ├── archive-note.command.ts
    │   │       └── archive-note.handler.ts
    │   ├── queries/
    │   │   ├── get-note/
    │   │   │   ├── get-note.query.ts
    │   │   │   └── get-note.handler.ts
    │   │   └── list-notes/
    │   │       ├── list-notes.query.ts
    │   │       └── list-notes.handler.ts
    │   └── events/
    │       ├── note-created.event.ts
    │       └── note-archived.event.ts
    ├── domain/
    │   ├── note.aggregate.ts
    │   └── note.events.ts
    ├── infrastructure/
    │   ├── event-store/
    │   │   ├── event-store.service.ts
    │   │   └── stored-event.entity.ts
    │   └── projections/
    │       ├── note-projection.service.ts
    │       └── note-read.entity.ts
    └── notes.module.ts

This looks like a lot of files. It is. That's the cost. But every file has exactly one responsibility — and when something breaks, you know exactly where to look.

Part 1: The Aggregate

The aggregate is the heart of this pattern. It's where your business logic lives, and it's responsible for:

Validating commands
Emitting domain events
Rebuilding its own state from a stream of events

// domain/note.aggregate.ts
import { AggregateRoot } from '@nestjs/cqrs';
import { NoteCreatedEvent } from './events/note-created.event';
import { NoteArchivedEvent } from './events/note-archived.event';
import { TagAddedEvent } from './events/tag-added.event';

export class Note extends AggregateRoot {
  private id: string;
  private title: string;
  private content: string;
  private tags: string[] = [];
  private archived: boolean = false;
  private version: number = 0;

  // Factory method — prefer this over constructor for creation
  static create(id: string, title: string, content: string, authorId: string): Note {
    const note = new Note();
    note.apply(new NoteCreatedEvent(id, title, content, authorId));
    return note;
  }

  // Reconstitute from event stream (Event Sourcing)
  static reconstitute(events: StoredEvent[]): Note {
    const note = new Note();
    for (const event of events) {
      note.applyFromHistory(event);
    }
    return note;
  }

  archive(): void {
    if (this.archived) {
      throw new Error('Note is already archived');
    }
    this.apply(new NoteArchivedEvent(this.id));
  }

  addTag(tag: string): void {
    if (this.tags.includes(tag)) {
      throw new Error(`Tag "${tag}" already exists`);
    }
    this.apply(new TagAddedEvent(this.id, tag));
  }

  // Event handlers — these mutate state
  // NestJS CqrsModule calls these automatically via apply()
  onNoteCreatedEvent(event: NoteCreatedEvent): void {
    this.id = event.noteId;
    this.title = event.title;
    this.content = event.content;
    this.version++;
  }

  onNoteArchivedEvent(event: NoteArchivedEvent): void {
    this.archived = true;
    this.version++;
  }

  onTagAddedEvent(event: TagAddedEvent): void {
    this.tags.push(event.tag);
    this.version++;
  }

  getId(): string { return this.id; }
  getVersion(): number { return this.version; }
}

Notice the separation:

Business validation happens before apply() — in archive() and addTag()
State mutation happens in on* handlers — always in response to events
The aggregate can be rebuilt entirely from its event history via reconstitute()

This means your Note object's state at any point in time is a pure function of the events that happened to it. That's powerful.

Part 2: Domain Events

Events are simple value objects — immutable facts. They should carry enough data to rebuild state without needing to look anything up.

// domain/events/note-created.event.ts
export class NoteCreatedEvent {
  constructor(
    public readonly noteId: string,
    public readonly title: string,
    public readonly content: string,
    public readonly authorId: string,
    public readonly occurredAt: Date = new Date(),
  ) {}
}

// domain/events/note-archived.event.ts
export class NoteArchivedEvent {
  constructor(
    public readonly noteId: string,
    public readonly occurredAt: Date = new Date(),
  ) {}
}

// domain/events/tag-added.event.ts
export class TagAddedEvent {
  constructor(
    public readonly noteId: string,
    public readonly tag: string,
    public readonly occurredAt: Date = new Date(),
  ) {}
}

Golden rule: Once an event is persisted, it's immutable. You never update events. You never delete events. If you made a mistake, you emit a compensating event.

Part 3: The Event Store

This is the part most tutorials skip. Where do the events actually go?

You need an append-only store. Here's a PostgreSQL-backed implementation:

// infrastructure/event-store/stored-event.entity.ts
import { Entity, Column, PrimaryGeneratedColumn, CreateDateColumn, Index } from 'typeorm';

@Entity('event_store')
@Index(['aggregateId', 'version'], { unique: true })
export class StoredEvent {
  @PrimaryGeneratedColumn('uuid')
  id: string;

  @Column()
  @Index()
  aggregateId: string;

  @Column()
  aggregateType: string;

  @Column()
  eventType: string;

  @Column({ type: 'jsonb' })
  payload: Record<string, unknown>;

  @Column()
  version: number;

  @CreateDateColumn()
  occurredAt: Date;
}

// infrastructure/event-store/event-store.service.ts
import { Injectable, ConflictException } from '@nestjs/common';
import { InjectRepository } from '@nestjs/typeorm';
import { Repository, DataSource } from 'typeorm';
import { StoredEvent } from './stored-event.entity';

@Injectable()
export class EventStoreService {
  constructor(
    @InjectRepository(StoredEvent)
    private readonly eventRepo: Repository<StoredEvent>,
    private readonly dataSource: DataSource,
  ) {}

  async append(
    aggregateId: string,
    aggregateType: string,
    events: DomainEvent[],
    expectedVersion: number,
  ): Promise<void> {
    await this.dataSource.transaction(async (manager) => {
      // Optimistic concurrency check
      const currentVersion = await manager
        .createQueryBuilder(StoredEvent, 'e')
        .where('e.aggregateId = :aggregateId', { aggregateId })
        .select('MAX(e.version)', 'max')
        .getRawOne();

      const latestVersion = currentVersion?.max ?? -1;

      if (latestVersion !== expectedVersion) {
        throw new ConflictException(
          `Concurrency conflict: expected version ${expectedVersion}, got ${latestVersion}`,
        );
      }

      let version = expectedVersion + 1;

      for (const event of events) {
        await manager.save(StoredEvent, {
          aggregateId,
          aggregateType,
          eventType: event.constructor.name,
          payload: JSON.parse(JSON.stringify(event)),
          version: version++,
        });
      }
    });
  }

  async loadEvents(aggregateId: string): Promise<StoredEvent[]> {
    return this.eventRepo.find({
      where: { aggregateId },
      order: { version: 'ASC' },
    });
  }

  async loadEventsSince(aggregateId: string, version: number): Promise<StoredEvent[]> {
    return this.eventRepo
      .createQueryBuilder('e')
      .where('e.aggregateId = :aggregateId', { aggregateId })
      .andWhere('e.version > :version', { version })
      .orderBy('e.version', 'ASC')
      .getMany();
  }
}

The expectedVersion parameter is critical — it gives you optimistic concurrency. If two requests try to update the same aggregate simultaneously, only one will succeed. The other gets a ConflictException and can retry.

Part 4: Commands and Handlers

Commands express user intent. A command handler loads the aggregate, calls a method, persists the events, and publishes them.

// application/commands/create-note/create-note.command.ts
export class CreateNoteCommand {
  constructor(
    public readonly noteId: string,
    public readonly title: string,
    public readonly content: string,
    public readonly authorId: string,
  ) {}
}

// application/commands/create-note/create-note.handler.ts
import { CommandHandler, ICommandHandler, EventPublisher } from '@nestjs/cqrs';
import { CreateNoteCommand } from './create-note.command';
import { EventStoreService } from '../../infrastructure/event-store/event-store.service';
import { Note } from '../../domain/note.aggregate';

@CommandHandler(CreateNoteCommand)
export class CreateNoteHandler implements ICommandHandler<CreateNoteCommand> {
  constructor(
    private readonly eventStore: EventStoreService,
    private readonly publisher: EventPublisher,
  ) {}

  async execute(command: CreateNoteCommand): Promise<void> {
    const { noteId, title, content, authorId } = command;

    // Create the aggregate — events are buffered inside it
    const note = Note.create(noteId, title, content, authorId);

    // Merge the aggregate with EventPublisher so events get dispatched
    const mergedNote = this.publisher.mergeObjectContext(note);

    // Persist events to the store
    const uncommittedEvents = mergedNote.getUncommittedEvents();
    await this.eventStore.append(
      noteId,
      'Note',
      uncommittedEvents,
      -1, // -1 means "this aggregate should not exist yet"
    );

    // Dispatch events to NestJS event bus (for projections / sagas)
    mergedNote.commit();
  }
}

// application/commands/archive-note/archive-note.handler.ts
@CommandHandler(ArchiveNoteCommand)
export class ArchiveNoteHandler implements ICommandHandler<ArchiveNoteCommand> {
  constructor(
    private readonly eventStore: EventStoreService,
    private readonly publisher: EventPublisher,
    private readonly eventDeserializer: EventDeserializerService,
  ) {}

  async execute(command: ArchiveNoteCommand): Promise<void> {
    const { noteId } = command;

    // Load event stream and reconstitute the aggregate
    const storedEvents = await this.eventStore.loadEvents(noteId);

    if (storedEvents.length === 0) {
      throw new NotFoundException(`Note ${noteId} not found`);
    }

    const domainEvents = storedEvents.map((e) =>
      this.eventDeserializer.deserialize(e),
    );

    const note = Note.reconstitute(domainEvents);
    const mergedNote = this.publisher.mergeObjectContext(note);

    // Business logic — will throw if already archived
    mergedNote.archive();

    // Persist new events
    const uncommittedEvents = mergedNote.getUncommittedEvents();
    await this.eventStore.append(
      noteId,
      'Note',
      uncommittedEvents,
      note.getVersion(),
    );

    mergedNote.commit();
  }
}

Part 5: Projections (The Read Side)

Here's where CQRS really shines. Your read models are completely independent of your write model. They're optimized for queries, not for correctness.

// infrastructure/projections/note-read.entity.ts
import { Entity, Column, PrimaryColumn, UpdateDateColumn } from 'typeorm';

@Entity('notes_read')
export class NoteReadEntity {
  @PrimaryColumn()
  id: string;

  @Column()
  title: string;

  @Column({ type: 'text' })
  content: string;

  @Column()
  authorId: string;

  @Column({ type: 'text', array: true, default: '{}' })
  tags: string[];

  @Column({ default: false })
  archived: boolean;

  @Column()
  version: number;

  @UpdateDateColumn()
  lastProjectedAt: Date;
}

// infrastructure/projections/note-projection.service.ts
import { Injectable } from '@nestjs/common';
import { OnEvent } from '@nestjs/event-emitter';
import { InjectRepository } from '@nestjs/typeorm';
import { Repository } from 'typeorm';
import { NoteReadEntity } from './note-read.entity';
import { NoteCreatedEvent } from '../../domain/events/note-created.event';
import { NoteArchivedEvent } from '../../domain/events/note-archived.event';
import { TagAddedEvent } from '../../domain/events/tag-added.event';

@Injectable()
export class NoteProjectionService {
  constructor(
    @InjectRepository(NoteReadEntity)
    private readonly readRepo: Repository<NoteReadEntity>,
  ) {}

  @OnEvent(NoteCreatedEvent.name)
  async onNoteCreated(event: NoteCreatedEvent): Promise<void> {
    await this.readRepo.save({
      id: event.noteId,
      title: event.title,
      content: event.content,
      authorId: event.authorId,
      tags: [],
      archived: false,
      version: 1,
    });
  }

  @OnEvent(NoteArchivedEvent.name)
  async onNoteArchived(event: NoteArchivedEvent): Promise<void> {
    await this.readRepo.update(event.noteId, {
      archived: true,
    });
  }

  @OnEvent(TagAddedEvent.name)
  async onTagAdded(event: TagAddedEvent): Promise<void> {
    const note = await this.readRepo.findOneBy({ id: event.noteId });
    if (!note) return;

    await this.readRepo.update(event.noteId, {
      tags: [...note.tags, event.tag],
    });
  }
}

Your read model can be denormalized however you want. Want a full-text search index? Add it here. Want a separate notes_by_tag table? Add another projection. The event stream is the source of truth — projections are just cached views.

Part 6: Queries

Queries are the simple side of CQRS. They just read from the projection:

// application/queries/get-note/get-note.query.ts
export class GetNoteQuery {
  constructor(
    public readonly noteId: string,
    public readonly requesterId: string,
  ) {}
}

// application/queries/get-note/get-note.handler.ts
import { QueryHandler, IQueryHandler } from '@nestjs/cqrs';
import { InjectRepository } from '@nestjs/typeorm';
import { Repository } from 'typeorm';
import { GetNoteQuery } from './get-note.query';
import { NoteReadEntity } from '../../infrastructure/projections/note-read.entity';
import { NoteDto } from '../dtos/note.dto';

@QueryHandler(GetNoteQuery)
export class GetNoteHandler implements IQueryHandler<GetNoteQuery> {
  constructor(
    @InjectRepository(NoteReadEntity)
    private readonly readRepo: Repository<NoteReadEntity>,
  ) {}

  async execute(query: GetNoteQuery): Promise<NoteDto | null> {
    const note = await this.readRepo.findOneBy({ id: query.noteId });
    if (!note) return null;

    return {
      id: note.id,
      title: note.title,
      content: note.content,
      tags: note.tags,
      archived: note.archived,
    };
  }
}

Queries have zero business logic. They just map database rows to DTOs.

Part 7: The Module

Wiring it all together:

// notes.module.ts
import { Module } from '@nestjs/common';
import { CqrsModule } from '@nestjs/cqrs';
import { TypeOrmModule } from '@nestjs/typeorm';

// Entities
import { StoredEvent } from './infrastructure/event-store/stored-event.entity';
import { NoteReadEntity } from './infrastructure/projections/note-read.entity';

// Command Handlers
import { CreateNoteHandler } from './application/commands/create-note/create-note.handler';
import { ArchiveNoteHandler } from './application/commands/archive-note/archive-note.handler';

// Query Handlers
import { GetNoteHandler } from './application/queries/get-note/get-note.handler';
import { ListNotesHandler } from './application/queries/list-notes/list-notes.handler';

// Services
import { EventStoreService } from './infrastructure/event-store/event-store.service';
import { NoteProjectionService } from './infrastructure/projections/note-projection.service';
import { EventDeserializerService } from './infrastructure/event-store/event-deserializer.service';

const CommandHandlers = [CreateNoteHandler, ArchiveNoteHandler];
const QueryHandlers = [GetNoteHandler, ListNotesHandler];

@Module({
  imports: [
    CqrsModule,
    TypeOrmModule.forFeature([StoredEvent, NoteReadEntity]),
  ],
  providers: [
    ...CommandHandlers,
    ...QueryHandlers,
    EventStoreService,
    NoteProjectionService,
    EventDeserializerService,
  ],
})
export class NotesModule {}

Part 8: Projection Rebuilding

One of the killer features of Event Sourcing: you can rebuild any projection from scratch at any time.

// infrastructure/projections/projection-rebuilder.service.ts
import { Injectable, Logger } from '@nestjs/common';
import { InjectRepository } from '@nestjs/typeorm';
import { Repository, DataSource } from 'typeorm';
import { StoredEvent } from '../event-store/stored-event.entity';
import { NoteReadEntity } from './note-read.entity';
import { EventDeserializerService } from '../event-store/event-deserializer.service';
import { NoteProjectionService } from './note-projection.service';

@Injectable()
export class ProjectionRebuilderService {
  private readonly logger = new Logger(ProjectionRebuilderService.name);

  constructor(
    @InjectRepository(StoredEvent)
    private readonly eventRepo: Repository<StoredEvent>,
    @InjectRepository(NoteReadEntity)
    private readonly readRepo: Repository<NoteReadEntity>,
    private readonly deserializer: EventDeserializerService,
    private readonly projection: NoteProjectionService,
    private readonly dataSource: DataSource,
  ) {}

  async rebuildAll(): Promise<void> {
    this.logger.log('Starting full projection rebuild...');

    // Wipe the read model
    await this.readRepo.clear();

    // Load all events, ordered by aggregate and version
    const allEvents = await this.eventRepo.find({
      where: { aggregateType: 'Note' },
      order: { aggregateId: 'ASC', version: 'ASC' },
    });

    this.logger.log(`Replaying ${allEvents.length} events...`);

    for (const stored of allEvents) {
      const event = this.deserializer.deserialize(stored);
      // Manually invoke projection handlers (bypassing the event bus)
      await this.projection.handleEvent(event);
    }

    this.logger.log('Projection rebuild complete.');
  }
}

This is something you can't do with traditional CRUD. You want to add a new read model for full-text search? Write the projection, run the rebuilder against your event history, and it's done. No data migration. No backfills. Just replay.

Part 9: Event Versioning (The Hard Part)

Here's where most tutorials end and real life begins. Events live forever — but your code doesn't. What happens when NoteCreatedEvent needs a new field?

Strategy 1: Additive changes only

Add optional fields with defaults. Old events are still valid:

export class NoteCreatedEvent {
  constructor(
    public readonly noteId: string,
    public readonly title: string,
    public readonly content: string,
    public readonly authorId: string,
    public readonly workspaceId: string = 'default', // ← new, optional
    public readonly occurredAt: Date = new Date(),
  ) {}
}

Strategy 2: Upcasters for breaking changes

If you need to change the shape significantly, write an upcaster:

// infrastructure/event-store/upcasters/note-created-v1-to-v2.upcaster.ts
export class NoteCreatedV1ToV2Upcaster {
  upcast(oldPayload: NoteCreatedEventV1): NoteCreatedEventV2 {
    return {
      ...oldPayload,
      workspaceId: 'default', // backfill missing field
      _eventVersion: 2,
    };
  }
}

// infrastructure/event-store/event-deserializer.service.ts
@Injectable()
export class EventDeserializerService {
  private upcasters = new Map([
    ['NoteCreatedEvent_v1', new NoteCreatedV1ToV2Upcaster()],
  ]);

  deserialize(stored: StoredEvent): DomainEvent {
    let payload = stored.payload;

    // Apply upcasters if needed
    const upcasterKey = `${stored.eventType}_v${payload._eventVersion ?? 1}`;
    const upcaster = this.upcasters.get(upcasterKey);
    if (upcaster) {
      payload = upcaster.upcast(payload);
    }

    // Map to the correct event class
    switch (stored.eventType) {
      case 'NoteCreatedEvent':
        return new NoteCreatedEvent(
          payload.noteId,
          payload.title,
          payload.content,
          payload.authorId,
          payload.workspaceId,
        );
      case 'NoteArchivedEvent':
        return new NoteArchivedEvent(payload.noteId);
      default:
        throw new Error(`Unknown event type: ${stored.eventType}`);
    }
  }
}

Versioning is a first-class concern you need to think about before you start writing events to production.

Performance: Snapshots

Loading 10,000 events to reconstitute an aggregate is slow. Snapshots solve this:

// infrastructure/snapshots/snapshot.entity.ts
@Entity('aggregate_snapshots')
export class SnapshotEntity {
  @PrimaryGeneratedColumn('uuid')
  id: string;

  @Column()
  aggregateId: string;

  @Column({ type: 'jsonb' })
  state: Record<string, unknown>;

  @Column()
  version: number;

  @CreateDateColumn()
  createdAt: Date;
}

// In ArchiveNoteHandler (modified)
async execute(command: ArchiveNoteCommand): Promise<void> {
  let note: Note;
  let baseVersion: number;

  // Try loading from snapshot first
  const snapshot = await this.snapshotRepo.findOne({
    where: { aggregateId: command.noteId },
    order: { version: 'DESC' },
  });

  if (snapshot) {
    note = Note.fromSnapshot(snapshot.state);
    baseVersion = snapshot.version;

    // Load only events since the snapshot
    const newEvents = await this.eventStore.loadEventsSince(
      command.noteId,
      snapshot.version,
    );
    note.applyHistoricalEvents(newEvents.map(this.deserializer.deserialize));
  } else {
    const allEvents = await this.eventStore.loadEvents(command.noteId);
    note = Note.reconstitute(allEvents.map(this.deserializer.deserialize));
  }

  // ... rest of handler
}

Take a snapshot every N events (100–500 is a reasonable range).

When NOT to Use This

I'd be doing you a disservice if I didn't say this clearly:

Don't reach for CQRS + Event Sourcing as a default.

Use it when you have:

A genuine need for an audit trail (financial systems, compliance, undo/redo)
Different scaling requirements for reads vs. writes
Multiple consumers of your domain events (other services, analytics, notifications)
Enough domain complexity that the overhead is worth it

Don't use it for:

Simple CRUD apps (users table, settings, basic content management)
Small teams without prior ES experience — the learning curve is steep
Systems where eventual consistency is a hard problem (and you haven't thought it through)

The read model in CQRS is eventually consistent by default. After a command succeeds, the projection might lag by milliseconds. For most use cases this is fine. For others (like showing the user their freshly created note immediately after creation), you need to handle it explicitly — either by returning the data directly from the command handler, or by waiting for the projection to catch up.

The Full Flow: Putting It Together

Here's a controller that uses both sides:

// notes.controller.ts
import { Controller, Post, Get, Body, Param } from '@nestjs/common';
import { CommandBus, QueryBus } from '@nestjs/cqrs';
import { v4 as uuid } from 'uuid';
import { CreateNoteCommand } from './application/commands/create-note/create-note.command';
import { ArchiveNoteCommand } from './application/commands/archive-note/archive-note.command';
import { GetNoteQuery } from './application/queries/get-note/get-note.query';

@Controller('notes')
export class NotesController {
  constructor(
    private readonly commandBus: CommandBus,
    private readonly queryBus: QueryBus,
  ) {}

  @Post()
  async create(@Body() dto: CreateNoteDto) {
    const noteId = uuid();
    await this.commandBus.execute(
      new CreateNoteCommand(noteId, dto.title, dto.content, dto.authorId),
    );
    return { id: noteId };
  }

  @Post(':id/archive')
  async archive(@Param('id') id: string) {
    await this.commandBus.execute(new ArchiveNoteCommand(id));
  }

  @Get(':id')
  async getNote(@Param('id') id: string) {
    return this.queryBus.execute(new GetNoteQuery(id, 'user'));
  }
}

The controller knows nothing about aggregates, events, or projections. It dispatches commands and queries — that's it.

What I Learned Building NoteCore With This Stack

A few things I wish I'd known upfront:

1. Design your events around facts, not operations. NoteContentUpdated is better than UpdateNoteCommand. The event should describe what happened, not what was requested.

2. Keep aggregates small and focused. The bigger the aggregate, the more contention you'll have on optimistic locking. A Workspace that contains everything is a concurrency nightmare.

3. Eventual consistency is a UX problem, not just a technical one. Decide early how you'll handle it in your frontend — loading states, optimistic updates, or explicit polling.

4. The event deserializer is where you pay your technical debt. Keep it clean. It grows over time and becomes load-bearing infrastructure.

5. Projections can diverge. Have a rebuild mechanism ready from day one. You will need it.

Conclusion

CQRS and Event Sourcing aren't silver bullets. They're power tools — they make certain problems dramatically easier, and they add real complexity everywhere else.

The things you get for free: full audit history, time travel debugging, the ability to add new read models without touching existing code, horizontal scaling of reads and writes independently.

The things you pay for: more files, eventual consistency, event versioning, serialization concerns, and a steeper learning curve for new team members.

For NoteCore — where I need local-first sync, undo/redo, and the ability to add AI-powered features retroactively against the full history of user actions — it's absolutely the right call.

For a simple REST API that does CRUD on a few tables? Stick with the simple thing.

Building NoteCore in public — follow the journey on Twitter/X or check out HookScope if you work with webhooks.

If this was useful, a ❤️ helps other developers find it. Comments and questions welcome — especially if you've hit different tradeoffs in your own projects.

Clean Architecture in NestJS — A Practical Guide

Kuba — Mon, 06 Apr 2026 13:06:09 +0000

Building NestJS apps is easy. Building NestJS apps that are still easy to change six months later is a different story.

Clean Architecture gives you a way to structure your code so that business logic doesn't bleed into HTTP handlers, database queries don't dictate your domain model, and swapping infrastructure (say, Postgres → SQLite, or REST → gRPC) doesn't require rewriting half the app.

This guide is practical. We'll build a small feature — managing User entities — and walk through each layer with real code.

The core idea

Clean Architecture revolves around one rule: dependencies point inward.

┌──────────────────────────────────┐
│          Infrastructure          │  HTTP, DB, external APIs
│   ┌──────────────────────────┐   │
│   │      Application         │  Use cases / orchestration
│   │   ┌──────────────────┐   │   │
│   │   │     Domain       │  Entities, ports
│   │   └──────────────────┘   │   │
│   └──────────────────────────┘   │
└──────────────────────────────────┘

Domain — pure business logic, no framework dependencies
Application — use cases that orchestrate domain objects
Infrastructure — NestJS controllers, TypeORM repositories, anything "real"

The domain knows nothing about NestJS. The application knows nothing about Express. Infrastructure wires everything together.

Folder structure

src/
  users/
    domain/
      user.entity.ts
      user.repository.ts        ← port (interface)
    application/
      commands/
        create-user.command.ts
        create-user.handler.ts
      queries/
        get-user.query.ts
        get-user.handler.ts
    infrastructure/
      persistence/
        typeorm-user.repository.ts
        user.orm-entity.ts
      http/
        users.controller.ts
        dto/
          create-user.dto.ts
    users.module.ts

If you're using CQRS (and you should consider it for anything beyond CRUD), separate commands and queries from the start. Retrofitting this later is painful.

1. Domain layer

The domain layer holds your entities and the interfaces (ports) that infrastructure must implement.

domain/user.entity.ts

export class User {
  constructor(
    public readonly id: string,
    public readonly email: string,
    public readonly name: string,
    private readonly createdAt: Date,
  ) {}

  static create(email: string, name: string): User {
    return new User(
      crypto.randomUUID(),
      email,
      name,
      new Date(),
    );
  }

  isOlderThan(days: number): boolean {
    const cutoff = new Date();
    cutoff.setDate(cutoff.getDate() - days);
    return this.createdAt < cutoff;
  }
}

No decorators, no @Column(), no @Injectable(). Just a plain class with behavior. This is intentional — your domain entity should be testable with zero framework setup.

domain/user.repository.ts

export interface UserRepository {
  findById(id: string): Promise<User | null>;
  findByEmail(email: string): Promise<User | null>;
  save(user: User): Promise<void>;
}

export const USER_REPOSITORY = Symbol('USER_REPOSITORY');

This is your port. The domain defines what it needs; infrastructure provides it.

2. Application layer

Use cases live here. Each handler does one thing: it receives a command or query, calls domain logic, and persists or returns the result.

application/commands/create-user.command.ts

export class CreateUserCommand {
  constructor(
    public readonly email: string,
    public readonly name: string,
  ) {}
}

application/commands/create-user.handler.ts

import { CommandHandler, ICommandHandler } from '@nestjs/cqrs';
import { Inject } from '@nestjs/common';
import { CreateUserCommand } from './create-user.command';
import { User } from '../../domain/user.entity';
import { UserRepository, USER_REPOSITORY } from '../../domain/user.repository';

@CommandHandler(CreateUserCommand)
export class CreateUserHandler implements ICommandHandler<CreateUserCommand> {
  constructor(
    @Inject(USER_REPOSITORY)
    private readonly userRepository: UserRepository,
  ) {}

  async execute(command: CreateUserCommand): Promise<string> {
    const existing = await this.userRepository.findByEmail(command.email);
    if (existing) {
      throw new Error(`User with email ${command.email} already exists`);
    }

    const user = User.create(command.email, command.name);
    await this.userRepository.save(user);

    return user.id;
  }
}

Notice what's missing: no HTTP-specific concepts, no TypeORM, no NestJS response objects. This handler works identically whether triggered via REST, a CLI command, or a message queue consumer.

3. Infrastructure layer

Here's where NestJS, TypeORM, and the outside world come in.

infrastructure/persistence/user.orm-entity.ts

import { Entity, Column, PrimaryColumn, CreateDateColumn } from 'typeorm';

@Entity('users')
export class UserOrmEntity {
  @PrimaryColumn('uuid')
  id: string;

  @Column({ unique: true })
  email: string;

  @Column()
  name: string;

  @CreateDateColumn()
  createdAt: Date;
}

infrastructure/persistence/typeorm-user.repository.ts

import { Injectable } from '@nestjs/common';
import { InjectRepository } from '@nestjs/typeorm';
import { Repository } from 'typeorm';
import { UserRepository } from '../../domain/user.repository';
import { User } from '../../domain/user.entity';
import { UserOrmEntity } from './user.orm-entity';

@Injectable()
export class TypeOrmUserRepository implements UserRepository {
  constructor(
    @InjectRepository(UserOrmEntity)
    private readonly ormRepo: Repository<UserOrmEntity>,
  ) {}

  async findById(id: string): Promise<User | null> {
    const record = await this.ormRepo.findOneBy({ id });
    return record ? this.toDomain(record) : null;
  }

  async findByEmail(email: string): Promise<User | null> {
    const record = await this.ormRepo.findOneBy({ email });
    return record ? this.toDomain(record) : null;
  }

  async save(user: User): Promise<void> {
    await this.ormRepo.save(this.toOrm(user));
  }

  private toDomain(record: UserOrmEntity): User {
    return new User(record.id, record.email, record.name, record.createdAt);
  }

  private toOrm(user: User): UserOrmEntity {
    const entity = new UserOrmEntity();
    entity.id = user.id;
    entity.email = user.email;
    entity.name = user.name;
    return entity;
  }
}

The mapper (toDomain / toOrm) is the key. It keeps your ORM schema and your domain model independent — you can change one without touching the other.

infrastructure/http/users.controller.ts

import { Controller, Post, Body, HttpCode } from '@nestjs/common';
import { CommandBus } from '@nestjs/cqrs';
import { CreateUserDto } from './dto/create-user.dto';
import { CreateUserCommand } from '../../application/commands/create-user.command';

@Controller('users')
export class UsersController {
  constructor(private readonly commandBus: CommandBus) {}

  @Post()
  @HttpCode(201)
  async create(@Body() dto: CreateUserDto): Promise<{ id: string }> {
    const id = await this.commandBus.execute(
      new CreateUserCommand(dto.email, dto.name),
    );
    return { id };
  }
}

The controller is intentionally thin. It validates the request (via DTOs + class-validator), dispatches a command, and returns a response. No business logic here.

4. Wiring it all together

users.module.ts

import { Module } from '@nestjs/common';
import { CqrsModule } from '@nestjs/cqrs';
import { TypeOrmModule } from '@nestjs/typeorm';
import { UserOrmEntity } from './infrastructure/persistence/user.orm-entity';
import { TypeOrmUserRepository } from './infrastructure/persistence/typeorm-user.repository';
import { USER_REPOSITORY } from './domain/user.repository';
import { CreateUserHandler } from './application/commands/create-user.handler';
import { UsersController } from './infrastructure/http/users.controller';

const CommandHandlers = [CreateUserHandler];

@Module({
  imports: [CqrsModule, TypeOrmModule.forFeature([UserOrmEntity])],
  controllers: [UsersController],
  providers: [
    ...CommandHandlers,
    {
      provide: USER_REPOSITORY,
      useClass: TypeOrmUserRepository,
    },
  ],
})
export class UsersModule {}

The provide/useClass pair is where Dependency Inversion happens. The application layer depends on the USER_REPOSITORY token (a symbol), not on TypeORM directly. Swap useClass to an in-memory repository for tests — zero changes to handlers.

Testing the application layer in isolation

describe('CreateUserHandler', () => {
  let handler: CreateUserHandler;
  let repo: InMemoryUserRepository;

  beforeEach(() => {
    repo = new InMemoryUserRepository();
    handler = new CreateUserHandler(repo);
  });

  it('creates a user and returns its id', async () => {
    const id = await handler.execute(
      new CreateUserCommand('kuba@example.com', 'Kuba'),
    );
    expect(id).toBeDefined();
    const user = await repo.findByEmail('kuba@example.com');
    expect(user?.name).toBe('Kuba');
  });

  it('throws when email already taken', async () => {
    await handler.execute(new CreateUserCommand('kuba@example.com', 'Kuba'));
    await expect(
      handler.execute(new CreateUserCommand('kuba@example.com', 'Other')),
    ).rejects.toThrow('already exists');
  });
});

No NestJS test harness, no database, no HTTP — just fast, focused unit tests.

Common tradeoffs to be aware of

"This is a lot of boilerplate for CRUD." Yes. For a simple 3-field entity, the overhead is real. Clean Architecture pays off when your domain has actual rules — validation, state machines, business constraints. If it's just a thin wrapper over a DB table, CQRS + ports-and-adapters might be overkill.

"Do I need separate ORM and domain entities?" Only if they diverge. If your TypeORM entity maps 1:1 to your domain object with no behavior, merge them for now and split later when needed. The important thing is that your application layer never imports typeorm directly.

"What about aggregates and domain events?" Out of scope for this guide, but a natural next step. Once your User.create() starts needing to publish an event (e.g., to send a welcome email), domain events give you a clean way to do it without coupling to side effects.

Summary

Layer	Contains	Can import
Domain	Entities, value objects, repository interfaces	Nothing external
Application	Command/query handlers, use cases	Domain only
Infrastructure	Controllers, ORM repos, external clients	Application + Domain + NestJS

Start with the domain. Define your ports. Let infrastructure implement them. Keep your handlers framework-free. Test the application layer without spinning up a full NestJS app.

The payoff isn't apparent on day one — it becomes obvious when you need to refactor and the blast radius is small.

If you're building something and applying these patterns, I'd love to see what you're working on — drop a link in the comments.

I Built a Local-First AI Knowledge Base for Developers — Here's What Makes It Different

Kuba — Thu, 02 Apr 2026 10:24:52 +0000

Stop losing context between projects. NoteCore is your second brain, built for devs.

Every developer I know has the same problem.

You finish a project. Six months later you're starting something similar and you know you've solved this exact auth problem before. You know you wrote down why you chose CQRS over a simple service layer in that one architecture doc. You know there was a snippet somewhere that handled exactly this edge case.

But you can't find it. It's buried in Notion, or Obsidian, or a random .md file in a repo you barely remember the name of.

So you solve it again from scratch. You spend two hours re-deriving something you already figured out.

That's the problem NoteCore exists to solve.

What is NoteCore?

NoteCore is a local-first AI knowledge base built specifically for developers. It's not another note-taking app. It's a queryable second brain — you throw in code snippets, architecture decisions, documentation fragments, lessons learned, and you get back answers in natural language.

The stack: Electron + React + Vite on the desktop, a local NestJS + SQLite backend running inside the app, Voyage AI for embeddings, and Claude (claude-sonnet-4-6) for the AI layer. Everything runs on your machine. Your notes never leave unless you explicitly opt into cloud sync.

Let's walk through what actually makes it work.

Semantic Search with Voyage AI Embeddings

Full-text search is table stakes. It's also not good enough.

If you wrote a note titled "JWT refresh token rotation strategy" and later search for "how did I handle token expiry", a keyword search returns nothing. The words don't overlap. But the meaning does — and that's exactly what semantic search solves.

NoteCore embeds every note you save using Voyage AI's voyage-code-2 model, which is optimized specifically for code and technical content. When you write a query, it gets embedded using the same model and we run a cosine similarity search over your entire local vector store.

The result: you can ask questions the way you actually think, not the way you happened to word something six months ago.

// Simplified version of what happens on every search
const queryEmbedding = await voyageClient.embed({
  input: userQuery,
  model: 'voyage-code-2',
});

const results = await vectorStore.similaritySearch(
  queryEmbedding.embeddings[0],
  { topK: 10, threshold: 0.75 }
);

Embeddings are generated locally on save and stored in SQLite alongside the note content. No cloud round-trip on every search — it's fast.

Why Voyage AI over OpenAI embeddings?

Two reasons. First, voyage-code-2 is purpose-built for code — it understands that const handleAuth = async () => and "authentication handler function" are talking about the same thing. Generic text embedding models don't handle this nearly as well.

Second, Voyage AI's pricing model made more sense for a local-first product where we're not batching thousands of requests through a SaaS backend — we're embedding notes one at a time as they're saved.

AI Chat Over Your Notes (Claude)

Search gets you relevant notes. Chat gets you answers.

There's a big difference. When you search, you still have to read through results and connect the dots yourself. When you chat, NoteCore does that for you — it retrieves the most semantically relevant chunks from your knowledge base, injects them into context, and lets Claude synthesize an actual answer.

It works like this:

You ask: "What was my approach to handling optimistic updates in that React project?"
NoteCore runs a semantic search across your notes, pulls the top-k most relevant chunks
Those chunks get injected into a Claude prompt as context
Claude answers based on your actual notes, not generic knowledge

The key design decision here was keeping Claude grounded. We explicitly instruct it not to answer from general knowledge if the answer isn't in your notes. If you haven't written anything about a topic, it tells you that — it doesn't hallucinate an answer and make you think you already figured something out when you didn't.

const systemPrompt = `
You are a personal knowledge assistant. Answer the user's question 
using ONLY the notes provided below as context. 

If the answer is not present in the notes, say so clearly. 
Do not use general knowledge to fill in gaps.

Notes context:
${relevantChunks.map(c => c.content).join('\n\n---\n\n')}
`;

This constraint is actually the most important part of the feature. A second brain that makes things up is worse than useless — it gives you false confidence.

Conversation context

NoteCore maintains conversation history within a session, so you can do multi-turn reasoning over your notes:

"What was my auth approach in that project?"
"Why did I choose that over sessions?"
"Would that still work with the new architecture I'm planning?"

Each follow-up is aware of what came before. The full conversation history gets passed to Claude on each turn, so context accumulates naturally.

BYOK — Bring Your Own Key

This one was non-negotiable for me when designing NoteCore.

Most AI tools that embed an LLM work one of two ways: they either route everything through their own backend (meaning your notes hit their servers, they pay for inference and charge you a margin) or they lock you into a specific model with no flexibility.

NoteCore does neither. With BYOK, you connect your own API keys — Anthropic for Claude, Voyage AI for embeddings — and NoteCore calls the APIs directly from the local backend. We never see your requests, your notes, or your responses.

From the user's perspective it's simple: you paste your keys in settings, they're stored encrypted locally using electron-store with OS-level encryption, and that's it. From that point on, you're billed directly by Anthropic/Voyage at their standard rates. No markup, no middleman.

// Keys are stored encrypted, never in plaintext
await secureStore.set('anthropic_api_key', encrypted(apiKey));

// On each AI request, retrieved and used directly
const key = decrypt(await secureStore.get('anthropic_api_key'));
const client = new Anthropic({ apiKey: key });

Why does this matter? A few reasons:

Privacy. Your notes are your thoughts. Architecture decisions, half-formed ideas, things you're figuring out — none of that should sit on someone else's server.

Cost transparency. You see exactly what you're spending. No opaque "AI credits" that obscure actual API costs.

Model flexibility. When Anthropic ships a new model, you can switch immediately. You're not waiting for us to update our backend. You control which model version you're running.

Trust. Honestly, this one's simple. Developers don't trust SaaS products with their sensitive context by default. BYOK removes the trust requirement entirely.

Why NoteCore Is Different

There are other tools in this space. Obsidian, Notion AI, Mem, Reflect. Here's the honest comparison:

Obsidian is brilliant but its AI features are plugin-dependent, fragmented, and the semantic search story is weak without significant setup. It's also a general-purpose tool — not optimized for code and technical content.

Notion AI requires your notes to live in Notion's cloud. Full stop. If that's fine with you, great. It wasn't fine for me.

Mem and Reflect are cloud-first, subscription-based, and neither is particularly developer-focused. They're optimized for the knowledge worker, not the engineer who wants to query their own code snippets.

NoteCore's specific bets:

Local-first by default. Your data lives on your machine. Cloud sync is opt-in.
Developer-focused embeddings. voyage-code-2 understands code semantically in a way generic models don't.
BYOK as a first-class feature, not an afterthought. You own your API relationships.
Grounded AI. The chat feature answers from your notes, not from the internet or general LLM knowledge.
One-time payment. No subscription. Buy it, own it, use it forever.

The Architecture in Brief

For those who want to understand how the pieces fit:

┌─────────────────────────────────────┐
│           Electron Shell            │
│  ┌─────────────┐  ┌──────────────┐  │
│  │  React UI   │  │ Local NestJS │  │
│  │  (Vite)     │◄─►  + SQLite   │  │
│  └─────────────┘  └──────┬───────┘  │
└─────────────────────────┼───────────┘
                           │ BYOK keys
                    ┌──────▼────────┐
                    │  Voyage AI    │  (embeddings)
                    │  Anthropic    │  (chat)
                    └───────────────┘

The local NestJS backend runs as a child process inside Electron. The React frontend communicates with it over a local HTTP port. SQLite holds both note content and vector embeddings. Everything is single-tenant, single-machine by design.

There's also an optional cloud backend (NestJS + Postgres on Railway) for sync and the web dashboard, but that's entirely opt-in. The app works fully offline with zero cloud dependency.

Current Status

NoteCore is approaching beta. The local backend is complete across all core phases — note management, embedding pipeline, semantic search, AI chat, BYOK. Active work right now is on the onboarding flow, feature flag gating via Gatedly, and packaging via GitHub Releases.

If you're a developer who's felt the pain of losing your own context between projects — this is built for you.

Waitlist is open at notecore.app.

Drop your email. Beta is coming soon and early access will be limited.

Building NoteCore in public. Follow along on X for updates.

Customer Called Me raged at Midnight

Kuba — Tue, 24 Mar 2026 21:34:03 +0000

I Built a Webhook Inspector After a Customer Called Me at Midnight

Two weeks ago I was integrating Stripe webhooks for a client project.
Everything was working fine in development. I deployed to production,
went to sleep, and woke up to a message:

"Hey, payments aren't going through."

I spent 3 hours debugging. The webhook was firing. My server was
receiving it. But something in the payload had changed silently —
no announcement, no deprecation warning. Just a broken handler
and an angry customer.

That's when I decided to build HookScope.

The Problem Nobody Talks About

Webhooks sound simple. Provider sends a POST, you handle it. Easy.

Until you're integrating 5 different providers simultaneously.

Stripe has its signature format. GitHub has another. Shopify does
it differently. Clerk uses Svix under the hood. Przelewy24 — the
Polish payment provider — has its own completely custom verification
flow that I had to reverse-engineer from their docs.

Every provider:

Different header names
Different HMAC algorithms
Different payload formats
Different retry logic

And none of them tell you when they change something.

Two Weeks, One Tool

I gave myself two weeks to build something I'd actually want to use.

The core idea was simple: one place to inspect, debug, and monitor
all your webhooks — regardless of provider.

The technical challenge that took the longest wasn't the inspector
itself. It was combining the CLI tunnel with the dashboard in a way
that felt seamless.

You run one command:

hookscope forward 3000

And instantly your local server is receiving webhooks from Stripe,
GitHub, or any other provider — with full inspection in the dashboard,
signature verification handled automatically, and real-time updates
via WebSockets.

Getting that end-to-end flow to feel instant took most of the two weeks.

The Feature I'm Most Proud Of

Schema drift detection.

Every webhook payload gets flattened into a map of dot-notation
paths and types:

{
  "data.object.amount": "number",
  "data.object.currency": "string",
  "type": "string"
}

On every new request, HookScope diffs the current schema against
the stored baseline. If anything changes — a field removed, a type
changed, a new required field added — you get an alert immediately.

Before your handler crashes. Before a customer calls you.

And for paid users, HookScope automatically generates updated
TypeScript, Zod, Go, and Python types via AI. Schema changes,
types update automatically.

What I Launched With

After two weeks I had:

CLI tunnel written in Go (distributed via Homebrew)
Real-time webhook inspector with WebSocket updates
Automatic signature verification for 5 providers
Schema drift detection with alerts
Email, Slack and Discord notifications
Duplicate detection
Silence detection (alerts when webhooks stop coming)
AI-powered DTO generation

Was it perfect? No. But it solved the exact pain point that woke
me up at midnight two weeks earlier.

The Lesson

The best tools come from real pain.

I didn't build HookScope because it seemed like a good SaaS idea.
I built it because I needed it — and every developer integrating
webhooks needs it too, they just don't know it yet.

If you've ever been burned by a silent webhook change or spent
hours debugging a mysterious 400, give it a try.

It's free to start: app.hookscope.dev

The CLI installs in one command:

brew tap nowackikuba/hookscope
brew install hookscope
hookscope login
hookscope forward 3000

Building in public. Feedback welcome in the comments.

How Stripe Silently Broke My Production App (And How I Built a Fix)

Kuba — Mon, 23 Mar 2026 15:46:10 +0000

How Stripe Silently Broke My Production App (And How I Built a Fix)

It was a Tuesday afternoon. A client called — orders weren't going through.

I checked the logs. Stripe webhooks were coming in, the handler was running, but something was off. After an hour of debugging I found it: Stripe had quietly changed the structure of their payment_intent.succeeded payload. A field I was relying on had moved. No announcement. No deprecation warning. Just a silent change that crashed my production handler.

I fixed it in 20 minutes. But I lost 3 hours total — debugging, deploying, explaining to the client.

That's when I started thinking: why doesn't anyone alert me when a webhook payload changes?

The Problem with Webhooks

Webhooks are deceptively simple. Provider sends a POST request, you handle it. Easy.

Until it isn't.

The real problem is that you don't control the schema. Stripe, GitHub, Shopify — they can change their payload structure at any time. Sometimes they announce it. Often they don't. And you only find out when your handler crashes.

Here's what a typical debugging session looks like:

Something breaks in production
You check logs — no clear error
You manually inspect the last few webhook payloads
You realize a field changed, moved, or disappeared
You fix it, deploy, explain to stakeholders

This happens more than you'd think. Stripe changed their payload structure multiple times last year.

The Solution: Schema Drift Detection

The idea is simple: store a flattened schema for every webhook endpoint, and compare it against every new incoming request.

If the schema changes — alert the developer immediately, before it crashes production.

Step 1: Flatten the payload

Instead of storing the raw JSON, we flatten it into a map of dot-notation paths to primitive types:

function flattenSchema(
  obj: Record<string, unknown>,
  prefix = ''
): Record<string, string> {
  return Object.entries(obj).reduce((acc, [key, value]) => {
    const fullKey = prefix ? `${prefix}.${key}` : key;

    if (Array.isArray(value)) {
      acc[fullKey] = 'array';
      if (value.length > 0 && typeof value[0] === 'object' && value[0] !== null) {
        Object.assign(acc, flattenSchema(value[0] as Record<string, unknown>, `${fullKey}[]`));
      }
    } else if (value !== null && typeof value === 'object') {
      Object.assign(acc, flattenSchema(value as Record<string, unknown>, fullKey));
    } else {
      acc[fullKey] = value === null ? 'null' : typeof value;
    }

    return acc;
  }, {} as Record<string, string>);
}

For example, this Stripe payload:

{
  "type": "payment_intent.succeeded",
  "data": {
    "object": {
      "id": "pi_123",
      "amount": 2000,
      "currency": "usd"
    }
  }
}

Becomes:

{
  "type": "string",
  "data.object.id": "string",
  "data.object.amount": "number",
  "data.object.currency": "string"
}

This is our baseline schema. Lightweight, easy to store, easy to compare.

Step 2: Diff the schemas

When a new webhook arrives, we flatten its payload and compare it against the stored baseline:

interface SchemaDiff {
  added: string[];
  removed: string[];
  typeChanged: { field: string; from: string; to: string }[];
}

function diffSchemas(
  previous: Record<string, string>,
  current: Record<string, string>
): SchemaDiff {
  const added = Object.keys(current).filter(k => !(k in previous));
  const removed = Object.keys(previous).filter(k => !(k in current));
  const typeChanged = Object.keys(current)
    .filter(k => k in previous && previous[k] !== current[k])
    .map(k => ({ field: k, from: previous[k], to: current[k] }));

  return { added, removed, typeChanged };
}

function isSchemaDiffEmpty(diff: SchemaDiff): boolean {
  return (
    diff.added.length === 0 &&
    diff.removed.length === 0 &&
    diff.typeChanged.length === 0
  );
}

Step 3: Handle multiple event types

Here's a detail that matters: most providers send different event types to the same endpoint.

Stripe sends payment_intent.succeeded, customer.created, invoice.paid — all to the same webhook URL. You can't compare schemas across different event types or you'll get false positives on every request.

The solution: store schemas per endpoint AND per event type.

// schemas stored as: Record<eventType, Record<fieldPath, type>>
{
  "payment_intent.succeeded": {
    "data.object.amount": "number",
    "data.object.currency": "string"
  },
  "customer.created": {
    "data.object.email": "string",
    "data.object.name": "string"
  },
  "__default__": {
    // for webhooks without an event type field
  }
}

Step 4: Wire it up

In NestJS, this lives in a scanner service that runs as a BullMQ background job after every incoming webhook:

@Injectable()
export class SchemaDriftScanner implements ScanServicePort {
  async scan(context: ScanContext): Promise<void> {
    if (!context.payload) return;

    const flattenedPayload = flattenSchema(context.payload);
    const endpoint = await this.endpointRepository.findById(context.endpointId);
    if (!endpoint) return;

    const eventTypeKey = context.eventType ?? DEFAULT_EVENT_TYPE_KEY;
    const targetSchema = endpoint.schemas?.[eventTypeKey];

    if (!targetSchema) {
      // First request — save as baseline, no alert
      endpoint.saveSchema(flattenedPayload);
      await this.endpointRepository.save(endpoint);
      return;
    }

    const diff = diffSchemas(targetSchema, flattenedPayload);

    if (!isSchemaDiffEmpty(diff)) {
      // Schema changed — emit alert event
      this.eventBus.publish(new AlertDetectedEvent({
        type: 'schema_drift',
        endpointId: context.endpointId,
        userId: context.userId,
        eventType: context.eventType,
        metadata: AlertMetadata.schemaDrift(diff),
      }));

      // Update baseline
      endpoint.saveSchema(flattenedPayload);
      await this.endpointRepository.save(endpoint);
    }
  }
}

The full pipeline looks like this:

POST /webhooks/:endpointId
  → verify signature
  → save request to DB
  → emit WebSocket event (live inspector)
  → enqueue ScanWebhookJob

ScanWebhookJob (BullMQ)
  → SchemaDriftScanner.scan()
  → DuplicateScanner.scan()

AlertDetectedEvent
  → save to DB
  → send email/Slack/Discord notification

Taking It Further: Auto-Generated Types

Detecting drift is useful. But what if the tool could also fix it for you?

When a schema drift alert fires, we use the Claude API to automatically generate updated TypeScript, Zod, Go, and Python types from the new schema:

// Input: flattened schema
{
  "data.object.amount": "number",
  "data.object.currency": "string",
  "data.object.total": "number"  // new field
}

// Output: updated NestJS DTO
export class DataObjectDto {
  @IsNumber()
  amount: number;

  @IsString()
  currency: string;

  @IsNumber()
  total: number; // automatically added
}

No manual work. Schema changes, types update automatically.

The Result

After building this into HookScope, I tested it against my own Stripe integration. Within minutes of changing a test payload structure, I got an alert with the exact diff — and an auto-generated updated DTO ready to copy into my codebase.

That 3-hour debugging session? It would have been a 5-minute fix.

Key Takeaways

Flatten your schemas — dot-notation paths make diffing trivial
Store per event type — or you'll get false positives on every request
Run async — never block the webhook response for scanning
Fail safe — if scanning fails, the webhook should still be processed
Default key for unknown event types — __default__ as a fallback

The full implementation is live at app.hookscope.dev if you want to see it in action.

If you've ever been burned by a silent webhook schema change, I'd love to hear about it in the comments.

DEV Community: Kuba

Concurrency Is Not Parallelism — And Go Knows the Difference

Why Concurrency Matters (and Why Most Developers Underestimate It)

Concurrency vs. Parallelism — The Distinction That Actually Matters

Why I Use Go (and Where NestJS Falls Short)

1. Long-running background processing

2. CPU-bound work

3. CLI tools and infrastructure code

4. When you want to reason about what's actually happening

The Go Concurrency Model: What's Actually Under the Hood

Goroutines Are Not OS Threads

The M:N Threading Model

The GMP Scheduler

G — Goroutine

M — Machine (OS Thread)

P — Processor

Work Stealing

Goroutine Preemption — The Evolution

Channels: Communicating Instead of Sharing

What a Channel Actually Is

Patterns I Actually Use

The sync Package: When Channels Aren't the Answer

Common Pitfalls I've Hit (So You Don't Have To)

1. Goroutine Leaks

2. Closing a Channel from the Receiver

3. The Loop Variable Capture Bug

4. Unbounded Goroutine Spawning

5. Race Conditions — Use the Race Detector

Practical Architecture: The Outbox Pattern in Go

Benchmarks: Why This Model Is Fast

Where Go Falls Short (Being Honest)

Summary: The Mental Model That Actually Helps

GDPR for Developers: What the Regulation Actually Means in Code

What GDPR Actually Cares About

Mistake #1: Your Schema Collects Too Much

Mistake #2: Your Staging Database Has Real User Data

Mistake #3: You Don't Actually Delete Data

Mistake #4: No Audit Trail

Mistake #5: Storing Sensitive Data in Logs

Mistake #6: Ignoring Data Retention

Handling Test Data Properly

What Pseudonymization Actually Means

Encryption: What to Actually Encrypt

The GDPR Audit Checklist for Your Next Code Review

The Bottom Line

How to Safely Import Thousands of Records from an External API in NestJS

Why the Naive Approach Fails

The Architecture Overview

Designing the Sync Log

Running Sync as a Background Job

Paginating Through Stripe Without Loading Everything Into Memory

Making Every Write Idempotent

Handling Failures Gracefully

Surfacing Progress to the User

Allowing Re-Sync

The Result

Key Takeaways

CQRS + Event Sourcing in NestJS: A Real-World Deep Dive

The Problem With Traditional CRUD

The Big Picture

Project Structure

Part 1: The Aggregate

Part 2: Domain Events

Part 3: The Event Store

Part 4: Commands and Handlers

Part 5: Projections (The Read Side)

Part 6: Queries

Part 7: The Module

Part 8: Projection Rebuilding

Part 9: Event Versioning (The Hard Part)

Performance: Snapshots

When NOT to Use This

The Full Flow: Putting It Together

What I Learned Building NoteCore With This Stack

Conclusion

Clean Architecture in NestJS — A Practical Guide

The core idea

Folder structure

1. Domain layer

2. Application layer

The `sync` Package: When Channels Aren't the Answer