DEV Community: Hammad Khan

A Domain-Driven Notification Microservice — Patterns From Production

Hammad Khan — Fri, 22 May 2026 21:06:57 +0000

Notifications start small. "Send the user an email when their order ships." A function. A library. Done.

A year later, you have email, SMS, push, in-app, Slack, and Microsoft Teams. You have user preferences per channel. You have quiet hours, batching, throttling, and a "do not disturb" mode. You have unsubscribe links and bounce handling. You have analytics on open rates and template-level metrics. You have multi-language templates and timezone-aware scheduling.

What started as a function is now a system. If you keep it as a sprawling collection of sendEmail and sendSlack helpers across your codebase, that system will eat your engineering team alive.

This is the shape of the notification microservice I built (and have rebuilt twice). The pattern isn't novel — it's domain-driven design applied to notifications — but the specifics matter.

The core insight

A notification has three distinct concerns:

What happened in the business — "order placed," "user mentioned," "invoice overdue." This is a domain event.
What kind of message to send — "transactional email," "high-urgency push," "Slack mention." This is a delivery policy.
How to actually send it — "render this template, then call the email provider's API." This is a channel adapter.

Most codebases collapse all three into one function:

async function sendOrderShippedEmail(orderId: string, userId: string) {
  const user = await getUser(userId)
  const order = await getOrder(orderId)
  const html = renderTemplate('order-shipped', { user, order })
  await sendgrid.send({ to: user.email, subject: 'Your order shipped', html })
}

This function knows about the domain event, the message type, and the channel. Three concerns. One function. Each one will change for different reasons, and changes will ripple across all the call sites.

The DDD-shaped version separates them.

Domain events

The business code emits an event. It doesn't know or care how the notification gets delivered.

// In your order service
await eventBus.publish({
  type: 'order.shipped',
  occurredAt: new Date().toISOString(),
  data: {
    orderId: order.id,
    userId: order.userId,
    trackingNumber: order.trackingNumber,
  },
})

This is a fire-and-record operation. The order service is done. It moves on. The event lands in your event bus (BullMQ, Kafka, NATS, whatever).

The notification service consumes events. Its job is to translate "order.shipped" into "a transactional email to the user with this template."

Notification preferences

The user's preferences live in a separate domain. They might be stored in the notification service's database or in a profile service — doesn't matter, as long as they're queryable:

type UserNotificationPreferences = {
  userId: string
  channels: {
    email: { enabled: boolean; address: string }
    sms: { enabled: boolean; number?: string }
    push: { enabled: boolean; deviceTokens: string[] }
    slack: { enabled: boolean; userId?: string }
  }
  perEventType: Record<string, {
    channels: ('email' | 'sms' | 'push' | 'slack')[]
    enabled: boolean
  }>
  quietHours: { start: string; end: string; tz: string } | null
}

When the notification service receives order.shipped, it looks up the user's preferences. The user has email enabled, SMS enabled, push enabled — but for this event type (order.shipped), they've only chosen email. So the service sends one email.

This decoupling is crucial. The business code emits one event. The notification service decides what to do with it based on user preferences. The user can change their preferences without anyone touching the business code.

The dispatcher

The middle layer is a dispatcher that:

Consumes an event.
Looks up the user's preferences.
Decides which channels to deliver on.
For each channel, builds a delivery job and queues it.

type Channel = 'email' | 'sms' | 'push' | 'slack'

class NotificationDispatcher {
  async handle(event: DomainEvent) {
    const userId = (event.data as any).userId
    if (!userId) return

    const prefs = await this.prefsRepo.findByUserId(userId)
    if (!prefs) return

    const eventPrefs = prefs.perEventType[event.type]
    if (!eventPrefs?.enabled) return

    const now = new Date()
    const channels = this.filterByQuietHours(eventPrefs.channels, prefs, now)

    for (const channel of channels) {
      await this.deliveryQueue.add('deliver', {
        eventType: event.type,
        userId,
        channel,
        eventData: event.data,
        scheduledAt: now.toISOString(),
      })
    }
  }

  private filterByQuietHours(
    requested: Channel[],
    prefs: UserNotificationPreferences,
    now: Date
  ): Channel[] {
    if (!prefs.quietHours) return requested
    const isQuiet = isWithinQuietHours(now, prefs.quietHours)
    if (!isQuiet) return requested
    // During quiet hours, only allow non-disruptive channels (e.g., email/in-app)
    return requested.filter(c => c === 'email' || c === 'in-app')
  }
}

The dispatcher is pure routing logic. It doesn't render templates. It doesn't call any provider. It just figures out which channels to deliver on and queues delivery jobs.

Channel adapters

Each channel is a separate worker that consumes jobs from the delivery queue and dispatches to its specific channel.

class EmailDeliveryWorker {
  async handle(job: DeliveryJob) {
    if (job.channel !== 'email') return

    const user = await this.userRepo.findById(job.userId)
    const prefs = await this.prefsRepo.findByUserId(job.userId)
    if (!user || !prefs?.channels.email.enabled) return

    const template = await this.templateRepo.find(job.eventType, 'email')
    const rendered = await this.renderer.render(template, {
      user,
      ...job.eventData,
    })

    await this.emailProvider.send({
      to: prefs.channels.email.address,
      from: rendered.from,
      subject: rendered.subject,
      html: rendered.html,
      text: rendered.text,
      headers: {
        'X-Event-Type': job.eventType,
        'X-User-Id': job.userId,
        'List-Unsubscribe': `<${this.unsubscribeUrl(job.userId, job.eventType)}>`,
      },
    })

    await this.deliveryLog.record({
      userId: job.userId,
      channel: 'email',
      eventType: job.eventType,
      sentAt: new Date().toISOString(),
      provider: this.emailProvider.name,
    })
  }
}

The email worker knows about:

The user (to get their address).
The template store (to find the right template for this event type and channel).
The renderer (to fill in the template).
The email provider (to actually send).

It doesn't know about Slack, SMS, push, or any business logic. If I want to add a new channel, I add a new worker. The dispatcher already knows how to route to it (the channel is just a string in the job).

Templates as first-class entities

A template store is its own small domain. Each template has:

An event type it's for (order.shipped).
A channel it's for (email, sms).
A language (en, de, fr).
A version (so changes are auditable).
The actual template content (HTML, plain text, Markdown).

type Template = {
  id: string
  eventType: string
  channel: Channel
  language: string
  version: number
  subjectTemplate?: string       // for email
  bodyTemplate: string
  format: 'html' | 'markdown' | 'plain'
  createdAt: string
  active: boolean
}

The renderer takes a template and a context object and produces a rendered message. Use a templating library that has a sandboxed mode (Handlebars's strict mode, for example) — you do not want template authors writing arbitrary JS that gets executed during rendering.

class Renderer {
  async render(template: Template, context: Record<string, any>): Promise<Rendered> {
    const compiled = Handlebars.compile(template.bodyTemplate, { strict: true })
    const body = compiled(context)
    const subject = template.subjectTemplate
      ? Handlebars.compile(template.subjectTemplate, { strict: true })(context)
      : undefined
    return { body, subject, format: template.format }
  }
}

Templates being stored in a database (not in code) means non-engineers can edit them. We had marketing folks editing email copy via an admin panel without ever touching a deploy.

The unsubscribe surface

Every notification should be unsubscribable. The dispatcher checks enabled flags before queuing, but the user needs a way to flip those flags. Two patterns:

A preferences page in your app. Standard. Each event type has a checkbox per channel.
A one-click unsubscribe link in every notification. Required by law in many jurisdictions for email marketing, and good UX everywhere.

The unsubscribe link encodes the user ID, the event type, and a signed token. Clicking it flips the preference:

function unsubscribeUrl(userId: string, eventType: string): string {
  const token = sign({ userId, eventType, action: 'unsubscribe' })
  return `https://app.example.com/api/notify/unsubscribe?token=${token}`
}

The endpoint verifies the token, updates the preference, and shows a "you're unsubscribed" page. The same endpoint can power the List-Unsubscribe header for email clients that support one-click unsubscribe.

Observability

Each delivery generates two records:

Delivery log. "We attempted to send X to user Y on channel Z at time T using provider P."
Provider callback. "The provider says message M was delivered (or bounced, or opened, or clicked)."

Both feed into the same table, keyed by a message ID. The observability story collapses into "show me everything that happened for user Y this week," which is what support teams ask for.

Throttling and batching

Two problems show up at scale:

A user gets 50 notifications in 10 minutes because something noisy happened. You need to batch them into a digest.
A celebrity user's actions trigger 10,000 notifications to followers in a burst. You need to throttle.

The dispatcher is where this logic lives. Before queuing a delivery, check a sliding-window counter (Redis-backed):

async handle(event: DomainEvent) {
  // ...existing routing logic...

  for (const channel of channels) {
    const window = `notify:${userId}:${channel}:${eventType}`
    const count = await this.redis.incr(window)
    if (count === 1) await this.redis.expire(window, 60)  // 1-minute window

    if (count > MAX_PER_MINUTE) {
      // Throttled. Queue for batching instead.
      await this.batchQueue.add('batch', { userId, channel, eventType, eventData: event.data })
    } else {
      await this.deliveryQueue.add('deliver', { /* ... */ })
    }
  }
}

The batch worker runs periodically (every 5 minutes or whatever the digest schedule is), collects everything in the batch queue for a user, renders a "digest" template, and sends one combined notification.

What I'd warn the next team about

Don't put rendering in the dispatcher. Keep the dispatcher routing-only. The dispatcher decides which channels to use; the workers decide how to render and send. Mixing them couples your routing logic to your template engine in ways that hurt later.

Use job retries with caution. If the email provider returns a 503, retry. If it returns a 400 with "invalid recipient," do not retry — that's a permanent failure. Different error codes have different retry semantics; encode this in the worker.

Track template performance. Open rate per template per language is a real signal. Templates that score below 10% open rate are usually broken (bad subject line, bad timing, irrelevant content). Surface this in your admin UI.

Make the dead-letter queue visible. Failed deliveries should go somewhere humans can see them. We had three months of bounces piling up before anyone noticed; turned out a customer had typo'd their email and was getting nothing. A weekly dashboard of "delivery failures by user" caught it.

The takeaway

A notification microservice is one of the highest-leverage extractions you can do. The business code becomes simple (emit events). User preferences become a first-class concept. Templates become editable without deploys. New channels become new workers, not new branches in shared functions.

The pattern is more code than sendgrid.send(...). It's also the difference between "we have a notification feature" and "we have a notification platform." If you're shipping more than two notification types and you're tired of touching the same five functions every time product adds a new channel, this extraction pays for itself within a quarter.

JWT Hardening Checklist: Beyond 'Use HS256'

Hammad Khan — Sat, 16 May 2026 13:33:17 +0000

JWT Hardening Checklist: Beyond "Use HS256"

Every codebase that uses JWTs has the same comment somewhere: "Using HS256 for now, can switch to RS256 later." That comment was right when it was written and it's wrong now, because "later" has been five years and nobody's revisited it.

This is a checklist of the JWT issues I've found across production codebases, in rough order of severity. Most teams have two or three of them. A few teams have all of them.

1. Algorithm pinning

The famous bug: a JWT library that accepts "alg": "none" and skips signature verification. Auditors have been chasing this for a decade and yet libraries still ship with permissive defaults.

The fix is to pin the algorithm at the verification call:

import jwt from 'jsonwebtoken'

const decoded = jwt.verify(token, SECRET, { algorithms: ['HS256'] })

Note algorithms is an array, but you should pass exactly one. Multiple algorithms means the attacker chooses. They will choose none.

The Python equivalent (PyJWT):

decoded = jwt.decode(token, SECRET, algorithms=['HS256'])

The Ruby equivalent (jwt gem):

decoded = JWT.decode(token, SECRET, true, { algorithm: 'HS256' })

If your verify call doesn't pass algorithms explicitly, fix it. This is a five-character change and it closes a class of CVEs.

2. Symmetric vs asymmetric (HS256 vs RS256/ES256)

HS256 uses a shared secret. Whoever holds the secret can sign tokens. If you're signing tokens in one service and verifying them in another, both services need the secret. That means twice the attack surface for the secret, and rotation is twice the work.

RS256 uses an asymmetric keypair. The signer holds the private key. Verifiers hold the public key. Compromising a verifier doesn't let an attacker forge tokens (they only have the public key). For multi-service architectures, this is strictly better.

The migration is straightforward if you have the runway:

Add asymmetric key generation to your auth service.
Have your auth service start issuing JWTs signed with both algorithms (dual-sign during the rotation window — yes, you can include both signatures).
Update verifiers to accept either.
Once all verifiers are on the new code, drop the old HS256 signature.
Once all in-flight HS256 tokens have expired, drop the verify support.

If you're a single-service app and you don't see a multi-service future, HS256 is fine. Don't migrate for migration's sake.

3. Expiry — and short expiry

Tokens should expire. The lifetime should be measured in minutes, not hours.

const token = jwt.sign(payload, SECRET, {
  algorithm: 'HS256',
  expiresIn: '15m',
})

The argument I hear against short tokens: "the user will get logged out too often." This is what refresh tokens are for. Access tokens live 15 minutes. Refresh tokens live longer (hours to days) and live in HttpOnly cookies, where JS can't reach them. When the access token expires, the client calls a refresh endpoint, gets a new access token, and moves on.

If your tokens live 8 hours, an attacker who exfiltrates one has 8 hours of impersonation. If they live 15 minutes, they have 15 minutes. The math is straightforward.

4. Verify the issuer and audience

The iss (issuer) and aud (audience) claims are how you tell that a token was minted by the right authority and intended for your application:

const decoded = jwt.verify(token, SECRET, {
  algorithms: ['HS256'],
  issuer: 'https://auth.example.com',
  audience: 'api.example.com',
})

If you don't verify these, a token issued by your auth server for one of your services can be replayed against a different service. Inter-service token reuse is a real attack pattern.

5. Don't put PII or secrets in the payload

JWT payloads are base64-encoded, not encrypted. Anyone with the token can decode and read the contents. This includes:

The user's email or name (PII).
Role information (which the attacker now knows).
Application-internal IDs (which the attacker can guess at).

Keep the payload minimal: user ID, issued-at, expiry, issuer, audience, maybe a session ID. Everything else, look up server-side from the user ID.

If you absolutely need an encrypted payload, use JWE instead of JWT. But the better answer is usually "don't put the data there."

6. Refresh tokens belong in HttpOnly cookies

Storing tokens in localStorage is the default in many tutorials and the default in many bugs. JS code can read localStorage. XSS gets you the tokens.

The better pattern:

Access token: short-lived (15 min), held in memory by the SPA, sent in Authorization: Bearer headers.
Refresh token: longer-lived, set as HttpOnly; Secure; SameSite=Lax cookie. JS can't read it. The browser sends it automatically on the refresh endpoint.

The refresh endpoint is the only one that reads the refresh token. Everything else uses the access token.

// Refresh endpoint
app.post('/auth/refresh', async (req, res) => {
  const refresh = req.cookies['refresh_token']
  if (!refresh) return res.status(401).end()
  try {
    const decoded = jwt.verify(refresh, REFRESH_SECRET, {
      algorithms: ['HS256'],
      issuer: 'https://auth.example.com',
      audience: 'api.example.com',
    })
    if (await isRevoked(decoded.jti)) return res.status(401).end()
    const access = signAccessToken(decoded.sub)
    res.json({ access_token: access })
  } catch {
    res.status(401).end()
  }
})

7. Refresh token rotation

When the client uses a refresh token, give them a new refresh token in the response. Invalidate the old one. This is "refresh token rotation."

The win: if an attacker steals a refresh token and uses it, you'll notice when the legitimate client tries to use the same (now-invalidated) token next time. You can force-logout the entire session in response.

// On refresh, mint a new refresh too
app.post('/auth/refresh', async (req, res) => {
  const refresh = req.cookies['refresh_token']
  if (!refresh) return res.status(401).end()
  const decoded = jwt.verify(refresh, REFRESH_SECRET, { /* ... */ })
  if (await isRevoked(decoded.jti)) {
    // Refresh token reuse detected. Kill the entire session.
    await killSession(decoded.session_id)
    return res.status(401).end()
  }
  await markRevoked(decoded.jti)  // single-use
  const newRefresh = signRefreshToken(decoded.sub, decoded.session_id)
  const access = signAccessToken(decoded.sub)
  res.cookie('refresh_token', newRefresh, { httpOnly: true, secure: true, sameSite: 'lax' })
  res.json({ access_token: access })
})

The jti (JWT ID) claim is what makes this work. Each refresh token has a unique ID. You track which IDs have been used (in Redis, with a TTL slightly longer than the refresh token's expiry). Reuse of a used ID is a sign of compromise.

8. Revocation

Stateless tokens can't be revoked by changing a database row. They're valid until they expire. This is a fundamental design constraint, and you work around it two ways:

Short access tokens so the window of impersonation is small.
A revocation list keyed by jti or session ID for cases where you need immediate revocation (logout, password change, role downgrade).

The revocation list is a small Redis SET. On every token verify, check the SET. If the jti is in the SET, reject the token regardless of signature validity.

async function verifyWithRevocation(token: string): Promise<Decoded | null> {
  const decoded = jwt.verify(token, SECRET, { /* ... */ })
  if (await redis.sismember('revoked_jti', decoded.jti)) return null
  return decoded
}

The Redis hit costs about 0.1ms. The peace of mind is worth it.

9. Don't trust the `kid` claim blindly

If you support multiple signing keys (e.g., for rotation), tokens carry a kid (key ID) claim that says which key to use. A naive implementation does this:

function getKey(kid: string): string {
  return KEYS[kid]  // attacker controls kid; they can pick any key, including ones you've rotated out
}

The fix: maintain an allowlist of currently-valid kid values. Reject any token whose kid isn't in the allowlist, before you do anything else.

const VALID_KIDS = new Set(['2026-q2', '2026-q1'])
function getKey(kid: string): string {
  if (!VALID_KIDS.has(kid)) throw new Error('invalid_kid')
  return KEYS[kid]
}

For RS256/ES256 with JWKS endpoints, the library typically handles this, but verify your specific library's behavior.

10. Clock skew

The exp and nbf claims are time-based. If your servers' clocks drift, tokens issued by one server can be rejected by another. Allow a small clock skew:

jwt.verify(token, SECRET, {
  algorithms: ['HS256'],
  clockTolerance: 30,  // 30 seconds of skew allowed
})

Don't make this too generous. 30 seconds is reasonable. 5 minutes is "you have a different problem to solve, please go fix NTP."

The checklist, in summary

Pin the algorithm at verify time. Never accept alg=none.
Use asymmetric keys (RS256/ES256) for multi-service architectures.
Short access tokens (15 min). Long refresh tokens in HttpOnly cookies.
Verify iss and aud. Always.
Don't put PII in the payload.
Rotate refresh tokens on each use. Track jti for reuse detection.
Maintain a revocation list for immediate kills.
Allowlist kid values.
Allow small clock skew (30s).

Each item is a small piece of code. The aggregate is the difference between "we use JWTs" and "we use JWTs correctly." If your codebase fails three or more of these, it's worth a Tuesday afternoon.

Designing Idempotent Bulk Import Pipelines (E.164, VIN, and the Rest)

Hammad Khan — Wed, 13 May 2026 13:16:43 +0000

Designing Idempotent Bulk Import Pipelines (E.164, VIN, and the Rest)

Bulk imports are a special category of pain. You give the user a CSV uploader, they hand you a file, and your job is to take that file from "blob of text" to "thousands of clean rows in your database." Somewhere in the middle, every assumption you have about your data falls apart.

I rebuilt the bulk import system for a dealership SaaS recently. The redesign covered phone number normalization, VIN format validation, file-format consistency, and the structural decision that makes the whole thing reliable: idempotency.

If you've ever shipped a bulk importer, you've probably hit the same problems. Here's the shape of the design that finally worked.

The four problems

Any bulk import system has to solve four problems:

Validation. Every row has to satisfy schema constraints before it goes into the database. Some constraints are obvious (required fields), some are domain-specific (a VIN must be 17 characters in a specific alphabet).
Normalization. "077 12345" and "+447712345" and "01234 567 890" might all need to become +441234567890 before they hit the database. The user doesn't think about this; you have to.
Error reporting. When 200 of 5,000 rows fail validation, the user needs to know which 200 and why. A flat "import failed" tells them nothing.
Idempotency. When the upload crashes halfway through and the user clicks "Retry," the second run must not double-write the first 2,500 rows.

The fourth problem is the one teams skip. It's also the one that breaks production. Let me start there.

Idempotency: the key decision

The pattern that works is stable per-row keys with an upsert at the DB level.

For each row, compute a stable key from the row's natural identity. For dealership customers, that might be (dealer_id, email_lowercase). For vehicles, (dealer_id, vin). Whatever uniquely identifies the row in the customer's mental model.

function customerKey(row: CustomerRow, dealerId: string): string {
  return `${dealerId}:${row.email.trim().toLowerCase()}`
}

Then use that key as the basis for an upsert:

INSERT INTO customers (dealer_id, email, name, phone, ...)
VALUES ($1, $2, $3, $4, ...)
ON CONFLICT (dealer_id, email) DO UPDATE SET
  name = EXCLUDED.name,
  phone = EXCLUDED.phone,
  ...
  updated_at = NOW();

Now if the import crashes after row 2,500 and the user retries, the first 2,500 rows update themselves in-place (no-op if the data didn't change) and the remaining 2,500 get inserted. Same end state as if the import had run cleanly the first time.

This sounds obvious. Most bulk import code I see doesn't do it. Most code uses INSERT INTO ... VALUES (...) and crashes on the unique-constraint violation when the user retries.

Normalize once, at the entry point

The normalization rule: do it once, as close to the file parsing as possible. Don't sprinkle normalization throughout the codebase.

// import-normalize.ts
import { parsePhoneNumberFromString } from 'libphonenumber-js'

export function normalizePhone(raw: string, defaultCountry: string): string | null {
  if (!raw || !raw.trim()) return null
  // Auto-prepend country code if user typed a local number without +
  const candidate = raw.trim().startsWith('+') ? raw.trim() : raw.trim()
  const parsed = parsePhoneNumberFromString(candidate, defaultCountry as any)
  if (!parsed?.isValid()) return null
  return parsed.format('E.164')  // e.g. "+441234567890"
}

export function normalizeVin(raw: string): string | null {
  if (!raw) return null
  const cleaned = raw.trim().toUpperCase().replace(/[^A-HJ-NPR-Z0-9]/g, '')
  // VIN must be 17 chars; no I, O, or Q allowed
  if (cleaned.length !== 17) return null
  return cleaned
}

export function normalizeEmail(raw: string): string | null {
  if (!raw) return null
  const trimmed = raw.trim().toLowerCase()
  return /^[^\s@]+@[^\s@]+\.[^\s@]+$/.test(trimmed) ? trimmed : null
}

Three things in here that look small:

Each function returns null on invalid input, not a thrown exception. The caller decides what to do with invalid rows (skip them, surface an error). Throwing inside normalization makes the loop harder to reason about.
Phone normalization uses libphonenumber-js, not a regex. Phone numbers are too varied for regex to handle correctly. The library handles country codes, formatting, valid-number checks, and the auto-prepend logic.
VIN normalization knows about the VIN alphabet. The letters I, O, and Q are not valid VIN characters (they look too much like 1 and 0). Strip them out during cleanup, and reject if the result isn't 17 characters.

The validation pipeline

The pipeline as a whole looks like this:

type ImportResult = {
  validRows: Customer[]
  errors: Array<{ row: number; field: string; reason: string }>
}

async function processImport(file: Buffer, dealerId: string, dealerCountry: string): Promise<ImportResult> {
  const parsed = parseCsv(file)
  const errors: ImportResult['errors'] = []
  const validRows: Customer[] = []

  for (let i = 0; i < parsed.length; i++) {
    const row = parsed[i]
    const rowNum = i + 2  // 1-indexed, plus header

    const email = normalizeEmail(row.email)
    if (!email) {
      errors.push({ row: rowNum, field: 'email', reason: 'invalid email' })
      continue
    }

    const phone = normalizePhone(row.phone, dealerCountry)
    if (row.phone && !phone) {
      errors.push({ row: rowNum, field: 'phone', reason: 'invalid phone' })
      continue
    }

    const name = (row.name ?? '').trim()
    if (!name) {
      errors.push({ row: rowNum, field: 'name', reason: 'required' })
      continue
    }

    validRows.push({ dealerId, email, phone, name })
  }

  return { validRows, errors }
}

A few choices to point out:

continue on validation error, don't return. Process every row, accumulate every error. The user wants to see all 200 errors at once, not the first one.
Row numbers are 1-indexed plus the header. The user's spreadsheet starts at row 1 (header) and the first data row is row 2. Match the user's mental model.
Errors capture row, field, and reason. This is what gets surfaced back to the UI. "Row 47: phone — invalid phone" is what the user needs to fix the spreadsheet.

The atomic write

Now you've got validRows. The write step is a single transactional upsert:

async function writeRows(rows: Customer[]) {
  if (rows.length === 0) return { inserted: 0, updated: 0 }

  return await db.transaction(async (tx) => {
    const result = await tx
      .insertInto('customers')
      .values(rows)
      .onConflict((oc) => oc.columns(['dealer_id', 'email']).doUpdateSet({
        name: (eb) => eb.ref('excluded.name'),
        phone: (eb) => eb.ref('excluded.phone'),
        updated_at: () => sql`now()`,
      }))
      .returning(['id', 'xmax'])  // xmax = 0 means insert, otherwise update
      .execute()

    const inserted = result.filter(r => r.xmax === '0').length
    const updated = result.length - inserted
    return { inserted, updated }
  })
}

Three things:

Single transaction. Either all 4,800 valid rows make it in, or none do. No partial state for the user to discover later.
xmax trick to distinguish inserts from updates. Postgres-specific: when INSERT ... ON CONFLICT DO UPDATE performs an insert, the returning row's xmax is 0. On an update, xmax is non-zero. This lets you tell the user "inserted 3,200, updated 1,600."
updated_at on conflict. Always touch the timestamp, even if no other fields changed. The audit trail wants to know the row was re-imported.

For very large imports (50,000+ rows), this single transaction will become a problem (long-held locks, replication lag). The fix is to batch:

const BATCH_SIZE = 1000
for (let i = 0; i < rows.length; i += BATCH_SIZE) {
  const batch = rows.slice(i, i + BATCH_SIZE)
  await writeRows(batch)
}

Each batch is its own transaction. You lose all-or-nothing across batches, but you gain the ability to scale to large imports. For our dealership use case (rare imports of 5,000-50,000 rows), this trade was right.

What about the "supply-chain" failures?

The original code we replaced loaded its xlsx parser from a CDN. We had a separate write-up on why that's a bad idea (and a separate fix). For the bulk-import context specifically: the parser runs against user-uploaded files. If the parser is malicious, every uploaded file is exfiltrated. Vendor the parser, verify its hash in CI, and don't load it from a remote URL. (Details in the companion article on the xlsx CDN supply-chain hole.)

The user-facing report

The validation step returns errors. The UI needs to show them in a way the user can act on. Two patterns work:

Inline error report. After upload, show "5,200 rows imported, 200 errors. Download error report?" The download is a CSV with the user's original rows plus an error column. They fix the spreadsheet and re-upload.
Reject-on-any-error. If any row has an error, reject the whole upload, surface the errors, and don't write anything. This is right for cases where partial imports are dangerous (vehicle records that have FK dependencies on other tables, for example).

For our customer imports, option 1 was right. For our vehicle imports, option 2 was right. Pick deliberately. Document the choice. Don't let the default be "import what we can, drop the rest silently" — that's how dealers find missing customers six months later.

The full pipeline shape

To summarize the design:

Parse the file into raw rows. No transformation here.
Normalize each field (phone, email, VIN, etc.) to the canonical form.
Validate each row. Collect errors. Don't stop on the first failure.
Decide based on the policy: reject-on-error or skip-bad-rows-and-continue.
Upsert valid rows in a transaction (or batched transactions), using a stable natural key.
Report back to the user: counts, errors, downloadable diff.

Each step is a small, testable function. The pipeline as a whole is reproducible: same file, same dealer config, same result. Crashes mid-way are recoverable because of idempotency.

The takeaway

Bulk imports become reliable once you commit to two ideas: normalize at the boundary, and make every write idempotent via a stable key. Once those two are in place, retries are free, partial failures are recoverable, and "the import broke" stops being a thing your support team has to handle.

The hardest part of building one isn't the validation. It's having the conversation about what your stable key actually is — and being willing to make the user pick one if the data doesn't have one obvious.

From Hours to Seconds: Simplifying RuboCop with rubocop-hk

Hammad Khan — Mon, 25 Aug 2025 07:11:35 +0000

The two-hour setup that drove me crazy

I started what should have been a simple task last Thursday at 3 PM: setting up RuboCop for a new Ruby project. I was still changing configuration files, searching for "best RuboCop rules for Ruby 3.2," and arguing with myself about whether to use double or single quotes by 5 PM. Does this sound familiar?

At that point, I knew it was time to stop. If I'm spending two hours setting up a linter instead of writing code, something is wrong. That night, I began working on rubocop-hk, a pre-configured RuboCop gem that lets you start linting in just 30 seconds.

What is rubocop-hk?

rubocop-hk is a RuboCop configuration gem that works well right away without any changes. It's like RuboCop's opinionated cousin who has already made all the hard choices for you.

RuboCop's default setup takes a long time to customize, but rubocop-hk gives you a setup that has been tested in real-world Ruby projects and improved over time. You can now install it just like any other gem because it's on RubyGems.

What I like best is It's still RuboCop on the inside. You get all the power of RuboCop without having to worry about setting it up. It saves time by making the most boring part of setting up a project a one-minute task.

The problem it solves (or why I made it)

Let's be honest about what adding RuboCop to a project does:

Before rubocop-hk:

Put RuboCop in your Gemfile (5 minutes)
Run it and get 500 or more offenses (10 minutes of crying)
Begin making .rubocop.yml (15 minutes)
Search for "best RuboCop configuration" on Google (30 minutes)
Take rules from different blog posts (20 minutes)
Talk with your team for 45 minutes about how long the lines should be.
Finally, choose a configuration (15 minutes)
Remember that you forgot to set up Rails cops (start over)

Total time: more than two hours of setting up hell.

After rubocop-hk:

Put rubocop-hk in your Gemfile (30 seconds)
Make a small .rubocop.yml file (30 seconds)
Start coding in a consistent way (worth its weight in gold)

The real killer isn't just the first step. It's keeping things the same across a lot of projects. How many times have you copied .rubocop.yml files from one project to another, only to find that each one is a little different and none of them are quite right?

How to Install and Get Started

Are you ready to take back your afternoon? Here's how to get going:

Step 1: Put it in your Gemfile

group :development, :test do
  gem 'rubocop-hk'
end

Step 2: Install the bundle

$ bundle install

Step 3: Make your .rubocop.yml

inherit_gem:
  rubocop-hk: default.yml

# Optional: Add settings that are specific to your project
AllCops:
  NewCops: enable
  TargetRubyVersion: 3.2

Step 4: Start RuboCop

$ bundle exec rubocop

Looking at 23 files
.......................

23 files checked, no crimes found

That's all. That's really all there is to it. In less than a minute, you've set up RuboCop with settings that are ready for production.

Important Features and Benefits

What makes rubocop-hk special isn't what it adds; it's what it takes away from your workflow:

🎯 The Philosophy of No Configuration

The gem follows the Ruby community's best practices right out of the box. No more being stuck on a decision or making changes over and over. It just works.

⚡ Productivity Right Away

Stop arguing about whether to use %w[] or %w() for arrays of words. Based on popular Ruby style guides and how people use it in the real world, rubocop-hk has already made these choices.

🔧 Still Adjustable

Don't like a rule? You can change it in your .rubocop.yml. rubocop-hk gives you the base, but you are still in charge:

inherit_gem:
  rubocop-hk: default.yml

# Your group likes longer lines
Layout/LineLength:
  Max: 120

📦 Batteries Included

The gem has settings for:

Style cops (Ruby idioms that are always the same)
Layout cops (spacing and indentation)
Lint cops (find real bugs)
Metrics cops (code that is easy to understand and change)
Naming cops (making sure that variable and method names are always the same)

To see exactly what you're getting, go to GitHub and look at the full configuration.

Examples of How to Use It in Real Life

Example 1: Putting together a Rails project

inherit_gem:
  rubocop-hk: default.yml

require:
  - rubocop-rails

Rails:
  Enabled: true

AllCops:
  Exclude:
    - 'db/schema.rb'
    - 'vendor/**/*'
    - 'config/initializers/*'

Example 2: GitHub Actions CI/CD Pipeline

name: Linters
on: [push, pull_request]

jobs:
  rubocop:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: ruby/setup-ruby@v1
        with:
          bundler-cache: true
      - name: Run RuboCop
        run: bundle exec rubocop --parallel

Example 3: Slowly getting the team to use it

Start out strict and then loosen up when you need to:

inherit_gem:
  rubocop-hk: default.yml

# Turn off strict cops for a while while the team gets used to it
Metrics/MethodLength:
  Enabled: false

Style/Documentation:
  Enabled: false

Example 4: Fixing Old Code Automatically

# Fix automatically what's safe to repair
$ bundle exec rubocop -a

# Show what needs to be done by hand
$ bundle exec rubocop --only-recognized-offenses

Who Should Use This?

Developers Who Work Alone

Stop wasting time setting things up. Use rubocop-hk and concentrate on adding new features.

Teams that work on development

Stop arguing about the style guide. Make rubocop-hk your team's standard and get back to solving real problems.

Students in Bootcamp and Ruby Beginners

Learn how to write Ruby code well without having to deal with hundreds of cop configurations.

People who keep open source software up to date

Without having to keep up with complicated documentation, give contributors clear style rules. With rubocop-hk, all you have to do is point them to your .rubocop.yml.

Agencies and Consultants

Keep client projects consistent without having to move configuration files around.

Your turn: Start in 30 seconds

The time is running out. You could have already installed rubocop-hk and be writing better Ruby code by the time you finish reading this conclusion.

Get the gem: rubygems.org/gems/rubocop-hk
Star the repo: github.com/hammadxcm/rubocop-hk
Tell us about your experience: Let us know how long it took you to set up by leaving a comment below!

Do you have any questions? Did you find a bug? Do you have a suggestion for how to set it up? This gem was made for the community, by the community. If you have a problem, open an issue on GitHub.

Stop setting things up. Get started with coding. Your future self will be grateful.

What is your RuboCop setup horror story? Please leave a comment below and let me know how much time you've wasted setting up your linter! 👇

If rubocop-hk helped you save time, please like this post and tell your Ruby friends about it. Let's speed up Ruby development for everyone!

Mastering GraphQL Mutations with the graphql-client Ruby Gem

Hammad Khan — Fri, 05 Jan 2024 09:02:34 +0000

GraphQL has become a popular choice for building APIs due to its flexibility and efficiency. When working with GraphQL APIs in Ruby, the graphql-client gem proves to be a powerful tool. In this article, we will delve into the intricacies of using mutations with the graphql-client gem to modify data on the server side.

Introduction to GraphQL Mutations

In GraphQL, mutations are operations used to modify data on the server. While queries are used to fetch data, mutations enable us to create, update, or delete data. This distinction aligns with the principles of the GraphQL schema, where mutations are defined to perform write operations.

Setting Up the graphql-client Gem

Before diving into mutations, it’s crucial to have the graphql-client gem set up in your Ruby project. The gem facilitates easy communication with GraphQL APIs. Here is a basic setup snippet:

require 'graphql/client'
require 'graphql/client/http'

module Api
  HTTP = GraphQL::Client::HTTP.new(ENV['GRAPHQL_API_URL'])
  Schema = GraphQL::Client.load_schema(HTTP)
  Client = GraphQL::Client.new(schema: Schema, execute: HTTP)
end

Make sure to replace 'GRAPHQL_API_URL' it with the actual URL of your GraphQL endpoint.

Executing a Mutation

Executing a mutation with the graphql-client gem is similar to querying data. Let's take an example of a mutation that creates a country. In this case, we have a mutation called CreateCountryMutation:

CreateCountryMutation = Api::Client.parse(<<~'GRAPHQL')   
  mutation($name: String) {
    createCountry(name: $name) {
      id
    }
  }
GRAPHQL

This mutation expects a parameter name of type String and returns the ID of the created city.

To execute this mutation, you can use the following code snippet:

variables = { name: 'United States' }
result = Api::Client.query(CreateCountryMutation, variables: variables)


puts result.data.create_country.id

Replace 'United States' with the desired country name. The result object will contain the response from the server, and you can access the created country's ID using result.data.create_country.id.

Conclusion

The graphql-client gem simplifies the process of working with GraphQL APIs in Ruby, and mutations play a crucial role in modifying data. By understanding how to structure mutations and execute them using the graphql-client gem, you can seamlessly integrate write operations into your Ruby applications.

Explore more about GraphQL mutations, handle errors, and optimize your mutations for efficient data manipulation. The combination of GraphQL and the graphql-client gem opens up a world of possibilities for building robust and flexible APIs. Happy coding!

DEV Community: Hammad Khan

A Domain-Driven Notification Microservice — Patterns From Production

The core insight

Domain events

Notification preferences

The dispatcher

Channel adapters

Templates as first-class entities

The unsubscribe surface

Observability

Throttling and batching

What I'd warn the next team about

The takeaway

JWT Hardening Checklist: Beyond 'Use HS256'

JWT Hardening Checklist: Beyond "Use HS256"

1. Algorithm pinning

2. Symmetric vs asymmetric (HS256 vs RS256/ES256)

3. Expiry — and short expiry

4. Verify the issuer and audience

5. Don't put PII or secrets in the payload

6. Refresh tokens belong in HttpOnly cookies

7. Refresh token rotation

8. Revocation

9. Don't trust the kid claim blindly

10. Clock skew

The checklist, in summary

Designing Idempotent Bulk Import Pipelines (E.164, VIN, and the Rest)

Designing Idempotent Bulk Import Pipelines (E.164, VIN, and the Rest)

The four problems

Idempotency: the key decision

Normalize once, at the entry point

The validation pipeline

The atomic write

What about the "supply-chain" failures?

The user-facing report

The full pipeline shape

The takeaway

From Hours to Seconds: Simplifying RuboCop with rubocop-hk

The two-hour setup that drove me crazy

What is rubocop-hk?

The problem it solves (or why I made it)

How to Install and Get Started

Step 1: Put it in your Gemfile

Step 2: Install the bundle

Step 3: Make your .rubocop.yml

Step 4: Start RuboCop

Important Features and Benefits

Examples of How to Use It in Real Life

Example 1: Putting together a Rails project

Example 2: GitHub Actions CI/CD Pipeline

Example 3: Slowly getting the team to use it

Example 4: Fixing Old Code Automatically

Who Should Use This?

Your turn: Start in 30 seconds

Mastering GraphQL Mutations with the graphql-client Ruby Gem

Introduction to GraphQL Mutations

Setting Up the graphql-client Gem

Executing a Mutation

Conclusion

9. Don't trust the `kid` claim blindly