DEV Community: Elwin Ernst

Session anomaly detection in a multi-tenant auth service — with the real bugs I hit

Elwin Ernst — Fri, 28 Aug 2026 22:42:46 +0000

Account takeovers rarely announce themselves. What you see in the logs is a valid login. Correct email, correct password (or these days, correct passkey assertion). The attacker doesn't need to break the crypto — they got the credential from a paste dump, a phishing hit, or an old device that was never wiped.

The only thing that gives them away, most of the time, is context. The login is from a country the user has never been in. The IP is fresh. The browser fingerprint doesn't match. A lot of "how did that happen" incidents are provable in retrospect from these signals — if you were bothering to collect and score them.

This post walks through a small, pragmatic session anomaly detector I added to a multi-tenant NestJS auth service. Heuristics, not ML. The whole thing is ~180 lines of TypeScript plus a Postgres table.

Full implementation lives at ElwinErnst/auth-api.

What "anomaly" means here

Three signals, scored:

New IP — the source IP has never been seen for this user in the last 90 days. +30.
New country — the geo-resolved country has never been seen. +40.
New browser/OS fingerprint — coarse fingerprint (e.g. chrome|macintosh) never seen. +10.

Total: 0–80. Bucketed into:

< 40 → info (not stored, no signal)
40–69 → warning (stored, logged)
≥ 70 → critical (stored, logged as WARN, dashboard-worthy)

Deliberately excluded from this MVP: hour-of-day baseline. It's tempting — "logging in at 3am is weird if you always log in at 10" — but it requires a mature baseline to work. With five logins of history, quartiles are noise. When there are enough data points to make it meaningful, adding it is a follow-up.

Also deliberately excluded: any ML/LLM classifier. Rules produce boring, explainable, auditable results. An LLM classifier is the natural next layer — you use it to rank anomalies the rules already found, not to replace them.

Data model

One table for anomaly events. Sessions live in their own table (the one we already have for refresh token rotation), and we query them for the history baseline.

@Index(['userId', 'createdAt'])
@Entity('session_anomaly_events')
export class SessionAnomalyEvent {
  @PrimaryGeneratedColumn('uuid')
  id!: string;

  @Column({ type: 'uuid', name: 'user_id' })
  userId!: string;

  @Column({ type: 'uuid', name: 'tenant_id', nullable: true })
  tenantId!: string | null;

  @Column({ type: 'uuid', name: 'session_id', nullable: true })
  sessionId!: string | null;

  @Column({ type: 'text', nullable: true })
  ip!: string | null;

  @Column({ type: 'varchar', length: 3, nullable: true })
  country!: string | null;

  @Column({ type: 'varchar', length: 120, nullable: true })
  city!: string | null;

  @Column({ type: 'text', name: 'user_agent', nullable: true })
  userAgent!: string | null;

  @Column({ type: 'int' })
  score!: number;

  @Column({ type: 'jsonb', default: () => "'[]'" })
  flags!: string[];

  @Column({ type: 'varchar', length: 16 })
  severity!: 'info' | 'warning' | 'critical';

  @Column({ type: 'varchar', length: 40, name: 'login_kind' })
  loginKind!: 'password' | 'passkey';

  @CreateDateColumn({ type: 'timestamptz', name: 'created_at' })
  createdAt!: Date;
}

Notes worth calling out:

severity is stored even though it's derivable from score. Trades a few bytes for indexable filtering: "give me everything critical from the last week" is a WHERE severity = 'critical' AND created_at > ... instead of WHERE score >= 70 AND .... Cheap to have.
login_kind discriminates password vs. passkey. Passkey logins with new_ip + new_country are less alarming (users add passkeys from new devices routinely). Rules could weight them differently in v2.
session_id nullable because failed authentications will eventually flow through this table too (a login attempt that never made it to a session).

The core: `analyze()`

async analyze(input: AnalyzeInput): Promise<AnomalyResult> {
  const normalizedIp = normalizeIp(input.ip);
  const geo = normalizedIp ? geoip.lookup(normalizedIp) : null;
  const country = geo?.country ?? null;
  const uaFingerprint = fingerprintUserAgent(input.userAgent);

  const flags: string[] = [];
  let score = 0;

  const since = new Date(Date.now() - HISTORY_WINDOW_DAYS * 86400 * 1000);
  const history = await this.sessions.find({
    where: {
      userId: input.userId,
      createdAt: MoreThan(since),
      // Exclude the just-created session so we compare against previous
      // attempts, not against ourselves.
      ...(input.sessionId ? { id: Not(input.sessionId) } : {}),
    },
    order: { createdAt: 'DESC' },
    take: 50,
  });

  if (history.length === 0) {
    flags.push('first_login');
  } else {
    const seenIps = new Set(
      history.map((s) => normalizeIp(s.ip)).filter((v): v is string => !!v),
    );
    const seenCountries = new Set<string>();
    const seenUaFingerprints = new Set<string>();

    for (const s of history) {
      const ipNorm = normalizeIp(s.ip);
      if (ipNorm) {
        const g = geoip.lookup(ipNorm);
        if (g?.country) seenCountries.add(g.country);
      }
      const fp = fingerprintUserAgent(s.userAgent);
      if (fp) seenUaFingerprints.add(fp);
    }

    if (normalizedIp && !seenIps.has(normalizedIp)) {
      flags.push('new_ip');
      score += 30;
    }
    if (country && !seenCountries.has(country)) {
      flags.push('new_country');
      score += 40;
    }
    if (uaFingerprint && !seenUaFingerprints.has(uaFingerprint)) {
      flags.push('new_user_agent');
      score += 10;
    }
  }

  const severity =
    score >= 70 ? 'critical' : score >= 40 ? 'warning' : 'info';

  if (severity !== 'info') {
    await this.events.save(
      this.events.create({
        userId: input.userId,
        tenantId: input.tenantId,
        sessionId: input.sessionId,
        ip: normalizedIp,
        country,
        city: geo?.city ?? null,
        userAgent: input.userAgent,
        score,
        flags,
        severity,
        loginKind: input.loginKind,
      }),
    );
    this.logger.warn(
      `Session anomaly [${severity}] user=${input.userId} score=${score} flags=${flags.join(',')} ip=${normalizedIp} country=${country}`,
    );
  }

  return { score, flags, severity, country, city: geo?.city ?? null };
}

The two helpers keep the logic honest:

function normalizeIp(raw: string | null | undefined): string | null {
  if (!raw) return null;
  const trimmed = raw.split(',')[0]?.trim();
  if (!trimmed) return null;
  if (trimmed.startsWith('::ffff:')) return trimmed.slice(7);  // IPv4-mapped IPv6
  return trimmed;
}

function fingerprintUserAgent(ua: string | null | undefined): string | null {
  if (!ua) return null;
  const browser =
    ua.match(/(Chrome|Firefox|Safari|Edge|Opera|CriOS|FxiOS)/i)?.[1] ?? 'unknown';
  const os =
    ua.match(/(Windows|Mac OS X|Macintosh|Linux|Android|iPhone|iPad|iOS)/i)?.[1] ?? 'unknown';
  return `${browser.toLowerCase()}|${os.toLowerCase()}`;
}

The user-agent fingerprint is coarse on purpose. If we compared full UAs, every Chrome minor version bump would trip new_user_agent and drown the signal in noise. chrome|macintosh remains stable across Chrome 118, 119, 120, and flags the case that actually matters: same user, different platform.

Hooking it in

The analyzer runs at the very end of each successful login, both for the password path and for the WebAuthn / passkey path:

// auth.service.login (password)
const session = await this.sessionsService.createEmpty({ userId, tenantId, ... });

await this.anomalyService.analyze({
  userId: user.id,
  tenantId: membership.tenantId,
  sessionId: session.id,
  ip: context?.ip ?? null,
  userAgent: context?.userAgent ?? null,
  loginKind: 'password',
});

const tokenPair = await this.tokenService.generateTokenPair({ ... });

// passkeys.service.authenticationFinish (WebAuthn)
const session = await this.sessionsService.createEmpty({ ... });

await this.anomalyService.analyze({
  userId: passkey.userId,
  tenantId: tenant.id,
  sessionId: session.id,
  ip: context?.ip ?? null,
  userAgent: context?.userAgent ?? null,
  loginKind: 'passkey',
});

Deliberately not wired to block the login. This is a detection layer. A future iteration can add step-up MFA on critical — force a device confirmation or a magic link before the session becomes fully privileged — but that's product surface, not detection logic. Ship the detection first, then argue about the reaction.

The endpoint to read the flagged history:

@UseGuards(AccessJwtGuard)
@Get('/sessions/anomalies')
list(@CurrentAuth() auth: AccessTokenPayload, @Query('limit') limit?: string) {
  const parsed = limit ? Math.min(Math.max(Number(limit), 1), 200) : 50;
  return this.service.listForUser(auth.sub, parsed);
}

Four bugs I hit while building this — worth the specifics

Sharing these because "read the docs and it worked" is not what this actually looks like.

1. `geoip-lite@2.x` requires Node ≥24

I added yarn add geoip-lite and the local test passed on Node 26. Docker container (node:20-bookworm-slim) failed the build with:

error geoip-lite@2.0.3: The engine "node" is incompatible with this module.
Expected version ">=24.0.0". Got "20.20.2"

Fix: pinned to 1.4.10, which supports Node 12+. The 2.x branch is only a few weeks old and hasn't been the default for long enough to notice.

2. `import geoip from 'geoip-lite'` returned undefined

TypeScript compiled fine because allowSyntheticDefaultImports: true. At runtime the default import was undefined, because the project didn't have esModuleInterop: true — the two flags look similar but they're not the same.

Fix: import * as geoip from 'geoip-lite' (namespace import). Same result at compile time, works at runtime with a CommonJS module.

The error was subtle: TypeError: Cannot read properties of undefined (reading 'lookup') at the exact point we called geoip.lookup(ip). If you're getting that from any Node library, this is the first suspect.

3. Express ignored `X-Forwarded-For`

The smoke test looked like it should work: curl -H 'X-Forwarded-For: 8.8.8.8' .... But every session recorded the same internal Docker network IP. Because Express, by default, treats req.ip as the immediate hop unless you explicitly opt into trusting proxies.

Fix, in main.ts:

(app.getHttpAdapter().getInstance() as unknown as {
  set: (k: string, v: unknown) => void;
}).set('trust proxy', true);

One line. In a real deployment, you want to be more surgical about which proxies you trust (see Express docs on trust proxy), but for a demo behind a Docker gateway or a known reverse proxy, true is fine.

4. The analyzer saw the current session as history

This one took the longest to spot. Logins from 8.8.8.8 and 195.85.234.5 (Russia) were producing score = 0, severity = info, flags = []. The IPs were being stored correctly — I could see them in SELECT ip FROM sessions ORDER BY created_at DESC — but the analyzer decided nothing was new.

The reason: sessionsService.createEmpty() runs before anomalyService.analyze(). So when the analyzer queries sessions WHERE user_id = $1, the just-created session (with the new IP, new country, new UA) is in the result set. Every "seen IPs" contains the current IP. Every "seen countries" contains the current country. Nothing is ever new.

Fix:

where: {
  userId: input.userId,
  createdAt: MoreThan(since),
  ...(input.sessionId ? { id: Not(input.sessionId) } : {}),
},

Exclude the current session from history. After that, a login from a fresh IP produced score=40 severity=warning flags=[new_ip, new_user_agent], and a login from Japan produced score=70 severity=critical flags=[new_ip, new_country]. Exactly what you want.

Tradeoffs vs. the LLM/ML approach

An anomaly detection engineer at a bigger company will look at this and say: "why not train a model?" Answers:

Latency. This runs synchronously in the login path. Adding an LLM call at 300–1500 ms per request is a real UX regression. If you must do it, do it async and mark the session as "under review" until the score comes back.
Cost. At $1–$5 per million tokens, a classifier scoring every login for a moderately-sized SaaS becomes real money surprisingly fast.
Explainability. When your CISO asks "why did we flag this user," pointing at flags: ['new_ip', 'new_country'] and a hardcoded scoring rule is much more defensible than "the model said so."
Evaluation. Rules give you deterministic outputs. You can write unit tests that say "this input MUST produce this score." Try that with a classifier.

The right place for ML/LLM is ranking — take the top-N warnings/criticals produced by the rules, feed them to a classifier that decides "this looks like a legitimate travel + new phone" vs "this looks like a compromise," and page a human on the high-confidence-critical bucket.

What this doesn't cover (and shouldn't yet)

Impossible-travel detection (login from Buenos Aires at 10:00, login from Tokyo at 10:05). Nice to have; requires joining consecutive events, non-trivial in SQL but doable. Deferred.
Device binding (a token that only works from the device it was issued on). WebAuthn already gives you a version of this through credential IDs; adding it at the session cookie level is more invasive.
Block-on-critical. Detection ≠ prevention. Blocking on critical without a smooth step-up path locks out real users who took a real trip. Ship the reaction as a separate feature with its own UX flow.

Source

Analyzer + scoring: src/modules/session-anomaly/session-anomaly.service.ts
Entity: src/modules/session-anomaly/entities/session-anomaly-event.entity.ts
Login hook (password + passkey): both auth.service.ts and passkeys.service.ts in the same repo
Trust-proxy setup: src/main.ts
Meta-repo + docker-compose to run it locally: ElwinErnst/sytadel-suite

If you're building something similar and hit a bug I didn't cover, open an issue on the repo — I collect them for the follow-up post.

Automated secret rotation with a 24-hour grace window — done right in NestJS + Postgres

Elwin Ernst — Fri, 28 Aug 2026 13:00:37 +0000

Most secret rotation stories go like this: at 3am on a Sunday, someone finally rotates the API key that's been the same since 2019. The rotation itself takes a minute. The outage lasts nine hours because six different callers had that secret pinned in config and their deploys hadn't run.

The fix isn't more discipline. The fix is making rotation boring: automatic, on a schedule, with a grace window during which both the old and the new secret work. Callers can migrate at their own pace, monitoring tells you who still holds the old one, and after 24 hours the old one dies quietly.

This is the pattern I built into service_accounts in a multi-tenant NestJS auth service. Post walks through the schema, the verify path (which is where most implementations get subtle bugs), the cron, and the four things I'd tell my past self before starting.

Full code: ElwinErnst/auth-api, specifically the integrations/ module.

The mental model

Each service_account has:

A current secret hash — what new callers should use.
A previous secret hash + expires-at — what old callers can still use, for a bounded window.
Optionally, an auto-rotation policy: rotationIntervalDays (nullable) and a nextRotationAt timestamp.

Rotation itself is the same primitive whether it's triggered manually (an admin clicked "rotate") or by the cron (nextRotationAt reached). The only thing that changes is a trigger: 'manual' | 'scheduled' label in the log.

Every verify tries the current secret first. On failure, if previousSecretExpiresAt is still in the future, it tries the previous secret and logs a WARN so you can see who's still on the old credential. After the window expires, the previous hash becomes dead weight until the next rotation overwrites it.

That's the whole design. Everything else is code.

Schema additions

Starting from an existing service_accounts table (with secretHash, secretPreview, isActive, failedAuthAttempts, etc.), we add four columns:

@Column({ name: 'rotation_interval_days', type: 'int', nullable: true })
rotationIntervalDays!: number | null;

@Column({ name: 'next_rotation_at', type: 'timestamptz', nullable: true })
nextRotationAt!: Date | null;

@Column({ name: 'previous_secret_hash', type: 'text', nullable: true })
previousSecretHash!: string | null;

@Column({ name: 'previous_secret_expires_at', type: 'timestamptz', nullable: true })
previousSecretExpiresAt!: Date | null;

Two notes:

rotationIntervalDays is nullable. null = manual only. The most common state for a fresh service account: someone provisions it, we haven't decided on an auto policy yet. Nullable is cleaner than "0 means off," which is a footgun waiting to happen when someone types 0 meaning "immediately."
previous_secret_expires_at is stored separately rather than derived from updated_at + grace. Explicit is better here — the grace window might change in the future, and you don't want a config change to accidentally extend or shrink the acceptance window for previously-rotated secrets.

Rotation (shared for manual and scheduled)

private async rotateInternal(
  account: ServiceAccount,
  trigger: 'manual' | 'scheduled',
) {
  const previousHash = account.secretHash;

  const plainSecret = this.generateSecret();
  account.secretHash = await this.hashSecret(plainSecret);
  account.secretPreview = `••••${plainSecret.slice(-6)}`;
  account.lastUsedAt = null;

  // Keep the previous secret valid for GRACE_WINDOW_HOURS so callers
  // that still hold it have time to migrate without an outage.
  account.previousSecretHash = previousHash;
  account.previousSecretExpiresAt = new Date(
    Date.now() + GRACE_WINDOW_HOURS * 3600 * 1000,
  );

  // Reschedule next rotation if an auto policy is in place.
  if (account.rotationIntervalDays !== null) {
    account.nextRotationAt = new Date(
      Date.now() + account.rotationIntervalDays * 86400 * 1000,
    );
  }

  const saved = await this.serviceAccountsRepository.save(account);

  this.logger.log(
    `rotated service_account=${account.id} trigger=${trigger} grace_hours=${GRACE_WINDOW_HOURS}`,
  );

  return {
    serviceAccount: this.toServiceAccount(saved),
    clientSecret: plainSecret,
  };
}

The public surface is thin:

async rotateServiceAccountSecret(tenantId: string, serviceAccountId: string) {
  await this.assertAuthApiEnabled(tenantId);
  const account = await this.findServiceAccount(tenantId, serviceAccountId);
  return this.rotateInternal(account, 'manual');
}

async autoRotateDue(): Promise<{ rotated: number }> {
  const now = new Date();
  const due = await this.serviceAccountsRepository.find({
    where: {
      rotationIntervalDays: Not(null as unknown as number),
      nextRotationAt: LessThanOrEqual(now),
      isActive: true,
    },
    take: 100,
  });

  let rotated = 0;
  for (const account of due) {
    try {
      await this.rotateInternal(account, 'scheduled');
      rotated += 1;
    } catch (err) {
      this.logger.error(
        `auto-rotate failed for service_account=${account.id}: ${
          err instanceof Error ? err.message : String(err)
        }`,
      );
    }
  }
  return { rotated };
}

autoRotateDue intentionally logs and swallows per-account failures. If one account's rotation fails (a stale reference, a stray unique-index violation, whatever), the batch keeps going. The alternative — abort the whole batch on the first error — means one broken account blocks every other account's scheduled rotation.

The verify path (this is where it usually goes wrong)

Most implementations of "grace window" get this wrong. They either check the previous secret before the current (which lets the old secret keep working forever if you never cleared it), or they check both without distinguishing which one won (which loses the "who's still on the old credential" signal).

Here's the version that keeps both properties:

const secretOk = await this.verifySecret(dto.clientSecret, account.secretHash);
let usedPreviousSecret = false;

if (!secretOk) {
  // Grace window: accept the previous secret if it hasn't expired yet.
  const previousStillValid =
    account.previousSecretHash &&
    account.previousSecretExpiresAt &&
    account.previousSecretExpiresAt.getTime() > Date.now();

  if (previousStillValid) {
    const prevOk = await this.verifySecret(
      dto.clientSecret,
      account.previousSecretHash!,
    );
    if (prevOk) {
      usedPreviousSecret = true;
      this.logger.warn(
        `service_account=${account.id} authenticated with previous secret; grace_expires_at=${account.previousSecretExpiresAt!.toISOString()}`,
      );
    }
  }

  if (!usedPreviousSecret) {
    await this.registerTokenFailure(account);
    throw new UnauthorizedException('Invalid service account credentials');
  }
}

Properties this preserves:

Current secret is always tried first. Fast path for the majority of callers.
Previous is only attempted if current failed. No wasted crypto on the happy path.
Previous validity is time-bounded. After previousSecretExpiresAt, the check is skipped without even hashing.
Successful use of the previous secret emits a WARN log with the account ID and expiry. Ship that into your alerting and you know exactly which callers are behind.
Failure counters and lockouts still apply on total failure, not on falling back to previous. A caller using an old-but-still-valid secret is not a suspicious event; they've just been slow to redeploy.

The usedPreviousSecret flag is currently just for the log. In a future iteration you'd want to also expose it in the response (so a caller's SDK can auto-refresh its cached secret), but that's a separate design conversation.

The scheduler

@nestjs/schedule does the plumbing:

@Injectable()
export class SecretRotationCron {
  private readonly logger = new Logger(SecretRotationCron.name);

  constructor(private readonly integrations: IntegrationsService) {}

  @Cron(CronExpression.EVERY_10_MINUTES)
  async tick(): Promise<void> {
    try {
      const { rotated } = await this.integrations.autoRotateDue();
      if (rotated > 0) {
        this.logger.log(`tick rotated=${rotated}`);
      }
    } catch (err) {
      this.logger.error(
        `tick failed: ${err instanceof Error ? err.message : String(err)}`,
      );
    }
  }
}

Ten-minute interval is a deliberate call. Every 10 minutes gives you a maximum drift of 10 minutes between the scheduled nextRotationAt and the actual rotation. For a 30-day rotation cadence that's a 0.02% drift, which nobody cares about. For a 1-day (dev/testing) cadence it's 0.7%, still fine. If you drop the cadence to hourly rotation you'd want to tighten this.

The cron is idempotent. If a rotation runs and advances nextRotationAt, the next tick's WHERE nextRotationAt <= now() filter skips it. If you have two nodes running the cron (you shouldn't, but suppose), the second one will see the accounts as already-rotated and no-op. Postgres row-level locking on the save handles the concurrent-rotate race — you'd get two attempts for the same account, but the loser's write just overwrites with a slightly newer secret and the caller sees a valid credential.

Setting the policy

The policy is a per-account attribute, editable by tenant owners/admins:

@Patch('tenants/:tenantId/service-accounts/:serviceAccountId/rotation-policy')
@UseGuards(AccessJwtGuard, RolesGuard, TenantScopeGuard)
@Roles('OWNER', 'ADMIN')
setRotationPolicy(
  @Param('tenantId') tenantId: string,
  @Param('serviceAccountId') serviceAccountId: string,
  @Body() dto: SetRotationPolicyDto,
) {
  return this.integrationsService.setRotationPolicy(
    tenantId,
    serviceAccountId,
    dto.rotationIntervalDays ?? null,
  );
}

The DTO enforces sanity:

export class SetRotationPolicyDto {
  @IsOptional()
  @IsInt()
  @Min(1)
  @Max(365)
  rotationIntervalDays?: number | null;
}

Bounds are [1, 365]. Below 1 day: you're probably testing (and you can pass a value like 1 and manually delete next_rotation_at if you really need to test now). Above 365 days: at that point the rotation is nominal and you should probably not have "automatic" in the name.

When the policy changes, nextRotationAt is recomputed from now, not from the previous nextRotationAt. Setting a 30-day policy at 3pm on a Tuesday sets the next rotation for the same time 30 days later, regardless of what the previous schedule looked like.

What this doesn't cover (on purpose)

Notification to callers before rotation. You could email the tenant owner "rotation scheduled in 24h" — that's product surface, not infrastructure. Add it once you have a notification pipeline.
Automatic secret migration for known integrations. If your callers are all your own Kubernetes deployments, you could patch their secret manager on rotate. That's a big scope creep for a general auth service — leave it to whoever runs the caller.
Rotation of the JWT signing keys themselves. Different problem, different solution (key rollover with issuer-verified kid in the JWT header). Deferred; the service-account secret rotation covers 80% of the pain.
Metrics dashboards. The WARN log is the audit trail; sticking Prometheus counters on usedPreviousSecret events is a natural next step once the observability stack is in place.

Tradeoffs vs. other approaches

Manual-only rotation, no automation. What most SaaS auth systems do out of the box. Works if your users are disciplined; catastrophic when a secret leaks and needs immediate cycling but nobody remembers the rotation runbook.

Immediate rotation, no grace window. Simple. Every rotation is an outage until every caller updates its config. For a service account used by five backend services, that's five deploys before the world comes back. Not viable in practice.

HashiCorp Vault dynamic credentials. Real answer for production. Each caller gets a short-lived credential auto-generated on request; nothing to rotate because nothing lives long enough to become stale. Requires committing to Vault as infrastructure — big lift, big win at scale.

AWS Secrets Manager rotation. Similar pattern to what's here (schedule + Lambda hook), scoped to AWS. Fine if you're all-in on AWS. Not portable.

The "grace window in the app" approach here. Simple, portable, no new infrastructure. Doesn't scale to many callers per credential the way dynamic credentials do, but scales fine to the "service-to-service integrations for a SaaS" use case, and any team can operate it without a new runbook.

The four things worth flagging

nullable = manual-only for the interval, not 0 = off. Zero is a footgun.
Grace window as an explicit timestamp, not computed from updated_at. Configurable grace period can change without invalidating past rotations.
Try current first, previous second. Never the other way around; you lose the observability signal if the old one just keeps working silently.
Cron is idempotent by design. Query filter on nextRotationAt <= now() + advance-on-rotate means running the same rotation twice is a no-op.

Source

Service (rotate + verify + policy): src/modules/integrations/integrations.service.ts
Cron: src/modules/integrations/secret-rotation.cron.ts
Entity: src/modules/integrations/entities/service-account.entity.ts
Endpoint: PATCH /tenants/:tenantId/service-accounts/:serviceAccountId/rotation-policy
Meta-repo: ElwinErnst/sytadel-suite

If you build something similar or spot a bug in the grace-window logic, drop it in the repo issues.

Harden for the reader: three red flags an AppSec engineer hits on `git clone`

Elwin Ernst — Sun, 23 Aug 2026 19:13:50 +0000

There's a gap between software that works and software that survives being read by a security engineer. A stack can pass every test, boot cleanly, and demo flawlessly while quietly carrying the exact patterns a reviewer is trained to grep for in the first five minutes. For a security product especially, the reader is the user — the person deciding whether to trust you is going to open the repo, not just click the buttons.

So I did a pass over the suite with one question: if an AppSec engineer cloned this, what would make them wince? Three things did. None broke anything at runtime. All three are the kind of finding that turns "interesting" into "no thanks" in a hiring or procurement read. Here's each red flag and how it was closed.

Full code: ElwinErnst/sytadel-suite (auth-api, billing-api, vault-api, and the compose at the root).

Red flag 1 — `synchronize: true`

TypeORM's synchronize auto-derives the database schema from your entities on every boot. It's wonderful for the first week and a liability forever after: no migration history, no review of schema changes, and one renamed column away from data loss. Seeing it on in a repo says "nobody owns the schema." Worse, the default here was on:

// before — auth-api
synchronize: String(process.env.DB_SYNC ?? 'true') === 'true',

The fix is to make migrations the source of truth. DB_SYNC now defaults to false, and the app applies migrations on boot:

// after
synchronize: String(process.env.DB_SYNC ?? 'false') === 'true',
migrations: [join(__dirname, 'database', 'migrations', '*.js')],
migrationsRun: true,

Generating the baseline honestly matters: I ran migration:generate against an empty throwaway Postgres so the diff was the entire current schema (12 tables for auth, 4 for billing), then actually applied it to confirm it stands up. The data-source.ts used for the CLI globs every *.entity.ts — the old one listed a stale subset of four entities, which would have generated an incomplete baseline and silently missed tables.

Bonus find while I was in there. The DB module logged its own config on boot:

console.log('DB CONFIG =>', db);   // ← db includes the password. deleted.

A credential in stdout is a credential in your log aggregator, your CI output, and anywhere those get shipped. Gone.

Red flag 2 — `change-me-*` secrets hardcoded in compose

Eighteen occurrences of six distinct secrets, all reading like this:

AUTH_JWT_ACCESS_SECRET: change-me-access-secret

Two problems in one line. First, secrets living in a committed file with no way to override them — you can't inject real values without editing the repo. Second, they're shared: the same JWT secret is used by auth to sign and by zerotrust + billing to verify, copy-pasted across three service blocks, which means they can drift out of sync during an edit and break cross-service auth in a way that's miserable to debug.

The fix externalizes each logical secret to a single interpolation variable, reused everywhere it's shared:

# auth signs, zerotrust + billing verify — all reference the ONE variable,
# so they can't drift. Real value comes from a gitignored root .env;
# local/CI fall back to a clearly dev-only default.
AUTH_JWT_ACCESS_SECRET: ${JWT_ACCESS_SECRET:-dev-insecure-access-secret}
ZT_JWT_HS256_SECRET:    ${JWT_ACCESS_SECRET:-dev-insecure-access-secret}
BILLING_JWT_ACCESS_SECRET: ${JWT_ACCESS_SECRET:-dev-insecure-access-secret}

The dev-insecure-* default keeps docker compose up and CI working with zero setup — but the name shouts that it's not for production. For production, a separate overlay removes the fallbacks entirely and fails closed:

# docker-compose.prod.yml — no defaults; missing secret ⇒ compose refuses to start
AUTH_JWT_ACCESS_SECRET: ${JWT_ACCESS_SECRET:?set JWT_ACCESS_SECRET in .env for production}

docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d
# error: required variable JWT_ACCESS_SECRET is missing a value

Because auth-api references all six shared secrets, requiring them in that one service block gates the entire stack at docker compose config time — nothing starts on placeholder secrets. Now the reader sees: secrets are injectable, shared ones can't drift, and prod can't accidentally boot insecure.

Red flag 3 — anti-replay in an in-memory `Map`

The internal service-to-service calls (and vault's Zero Trust requests) are HMAC-signed with a nonce, and replays are rejected by remembering nonces for a short window. The remembering was a per-process Map:

private readonly replayCache = new Map<string, number>();

Correct for exactly one instance. Restart, and the memory is gone; scale to two instances, and a replay hitting the other instance sails through. For a Zero Trust product, "replay protection that only works single-process" is precisely the gap a reviewer probes.

Moved it to a replay_nonces table with an atomic check-and-record — a unique key plus insert-on-conflict, which is both correct across instances and free of the check-then-set race the Map had:

const result = await this.repo
  .createQueryBuilder()
  .insert().into(ReplayNonce)
  .values({ key, expiresAt })
  .orIgnore()                 // ON CONFLICT DO NOTHING
  .returning('key')
  .execute();
const inserted = result.raw as unknown[];
return Array.isArray(inserted) && inserted.length > 0;   // false ⇒ replay

The guard verifies the signature first and only then records the nonce — no polluting the store with unverified requests — and a per-minute job prunes expired rows. I chose Postgres over Redis deliberately: every service already runs Postgres, so a new stateful dependency for low-volume internal calls would be its own thing to justify to that same reviewer. Right-size the infra to the access pattern.

The through-line

None of these changed a feature. They changed what the code says about itself to someone qualified to read it. That framing — harden for the reader, not just the runtime — is a useful lens because it aligns with the actual threat model of a security product: the people evaluating you will read the source, and "it works" is table stakes, not the bar.

A few honest gotchas from doing it:

Docker died mid-migration. With no daemon to generate against, I hand-wrote one migration mirroring the generated style and let CI validate it on a fresh DB — the smoke boots the whole stack, applies migrations, and gates on a real login, so a broken migration fails there.
npm install in a yarn repo. Reflex cost me: it created a bogus package-lock.json and rewrote every URL in yarn.lock. Soft-reset, put the dep in package.json only, let the Dockerfile's non-frozen yarn install reconcile it. Use the repo's package manager, always.
Init-script ordering + grants (vault). Vault owns its schema through numbered SQL init scripts, not migrations. The new table had to be numbered before the runtime-role grant script so it's covered by GRANT … ON ALL TABLES, and deliberately left out of the append-only revokes since the app prunes it.

Every change above shipped the same way — branch → PR → the integration smoke on a fresh stack → merge — because the point of hardening you can't demonstrate is nothing. If a reviewer can't see it holds, it may as well not.

Source

ElwinErnst/sytadel-suite: migrations in auth-api/billing-api src/database/migrations/, secret interpolation in docker-compose.yml + docker-compose.prod.yml, and the persistent replay store in each service's common/replay/ (vault via infra/postgres/init/058_replay_nonces.sql). Related: the MCP server and agentic HITL access.

Adding WebAuthn to a multi-tenant NestJS + Next.js app: what nobody tells you

Elwin Ernst — Fri, 07 Aug 2026 21:59:12 +0000

Passwords in 2026 are the cheapest attacker surface you own. You rotate them, you salt them, you make them 12+ characters, you turn on rate limiting — and someone still phishes one out of a shared Notion doc and drains a tenant next Tuesday.

Passkeys (WebAuthn) fix the class of attacks. The credential can't leave the device, there's nothing to phish, and if a user loses the device they lose the passkey, not access to their whole password manager.

The docs make it sound like a weekend project. It's a weekend project if you already know the three or four things that trip you up. This post is those three or four things, with real code from a live multi-tenant NestJS + Next.js app.

Full implementation is public — ElwinErnst/auth-api for the backend, ElwinErnst/sytadel-suite for the meta-repo with the frontend and docker-compose.

What you're building

A user with a valid password + tenant should be able to:

Log in normally with password (unchanged).
From settings, register one or more passkeys (Face ID, Touch ID, Windows Hello, YubiKey, iCloud Keychain). Each with a friendly name.
Log in later using any registered passkey, without typing a password.
Rename or delete their passkeys.

Two decisions I made up front, because they change the whole shape:

Passkeys coexist with passwords, not replace them. That's what Auth0, Clerk, 1Password all do. Users hate having their only login method suddenly break, and you want a graceful ramp for existing accounts.
Multi-device is the norm, not the edge case. One user, N passkeys. Their MacBook Touch ID and their phone's Face ID are separate credentials, both valid, each with its own friendly name.

The endpoints end up looking like:

POST /passkeys/registration/begin      (JWT required)
POST /passkeys/registration/finish     (JWT required) - attestation verify + save
POST /passkeys/authentication/begin    (public)
POST /passkeys/authentication/finish   (public) - assertion verify + issue JWT
GET  /passkeys                          (JWT required) - my passkeys
PATCH /passkeys/:id                     (JWT required) - rename
DELETE /passkeys/:id                    (JWT required)

Four endpoints for the flow, three for management. The begin calls return WebAuthn options that the browser passes into navigator.credentials.create() or .get(). The finish calls take the browser's attestation/assertion, verify it server-side, and either persist the credential (registration) or issue a session (authentication).

The library

Don't hand-roll WebAuthn. The protocol has attestation formats, CBOR decoding, COSE keys, ES256/RS256/EdDSA signature verification, and a spec that keeps evolving. Every language ecosystem has one battle-tested library — in Node/TypeScript it's @simplewebauthn/server for the backend and @simplewebauthn/browser for the frontend. Both from the same author, both v13 at time of writing, both well-maintained.

Install:

# backend
yarn add @simplewebauthn/server

# frontend
npm install @simplewebauthn/browser

Everything below builds on those four exported functions:

generateRegistrationOptions() — server, returns options for navigator.credentials.create()
verifyRegistrationResponse() — server, verifies the attestation
generateAuthenticationOptions() — server, returns options for navigator.credentials.get()
verifyAuthenticationResponse() — server, verifies the assertion

Data model

Two tables. First the passkeys themselves:

@Index(['userId'])
@Entity('user_passkeys')
export class UserPasskey {
  @PrimaryGeneratedColumn('uuid')
  id!: string;

  @Column({ type: 'uuid', name: 'user_id' })
  userId!: string;

  @ManyToOne(() => User, { onDelete: 'CASCADE' })
  @JoinColumn({ name: 'user_id' })
  user!: User;

  @Index({ unique: true })
  @Column({ type: 'bytea', name: 'credential_id' })
  credentialId!: Buffer;

  @Column({ type: 'bytea', name: 'public_key' })
  publicKey!: Buffer;

  @Column({ type: 'bigint', default: 0 })
  counter!: string;

  @Column({ type: 'text', array: true, default: () => "'{}'" })
  transports!: string[];

  @Column({ type: 'varchar', length: 20, name: 'device_type' })
  deviceType!: 'singleDevice' | 'multiDevice';

  @Column({ type: 'boolean', name: 'backed_up', default: false })
  backedUp!: boolean;

  @Column({ type: 'varchar', length: 80, name: 'friendly_name' })
  friendlyName!: string;

  @Column({ type: 'timestamptz', name: 'last_used_at', nullable: true })
  lastUsedAt!: Date | null;

  @CreateDateColumn({ type: 'timestamptz', name: 'created_at' })
  createdAt!: Date;

  @UpdateDateColumn({ type: 'timestamptz', name: 'updated_at' })
  updatedAt!: Date;
}

The gotchas here are subtle but real:

credentialId is bytea, not text. WebAuthn credential IDs are raw bytes. You can encode them as base64url for transport, but store them as bytes so the unique index does the right thing regardless of encoding drift.
counter is bigint stored as string, but the WebAuthn library accepts a JavaScript number. Before verification, the service validates that the stored decimal is a non-negative safe integer. Values above Number.MAX_SAFE_INTEGER are rejected rather than rounded. Most authenticators use small counters (or zero for synced passkeys); this guard makes the library boundary explicit instead of pretending JavaScript can preserve a larger value.
transports, deviceType, backedUp all come from the browser's attestation response. You need deviceType and backedUp to render sensible UX later ("This passkey is synced across your Apple devices" vs "This YubiKey lives on one physical stick").
No unique constraint on (userId, friendlyName). Users can name two passkeys the same thing if they want. The unique constraint is on the credential ID, not the label.

Second table: challenges.

@Entity('webauthn_challenges')
export class WebauthnChallenge {
  @PrimaryGeneratedColumn('uuid')
  id!: string;

  @Column({ type: 'varchar', length: 16 })
  kind!: 'registration' | 'authentication';

  @Column({ type: 'uuid', name: 'user_id', nullable: true })
  userId!: string | null;

  @Column({ type: 'text' })
  challenge!: string;

  @Column({ type: 'timestamptz', name: 'expires_at' })
  expiresAt!: Date;

  @CreateDateColumn({ type: 'timestamptz', name: 'created_at' })
  createdAt!: Date;
}

Challenges are one-shot, short-lived (5 minutes here), and tied to a kind + userId so a registration challenge can't be replayed against an authentication endpoint. Finish runs in a database transaction and selects the challenge with a pessimistic row lock. Failed verification rolls back and keeps the challenge; successful verification deletes it in the same commit as the credential state change. This prevents an invalid response from burning a valid challenge and serializes concurrent finishes. Expired rows are cleaned up lazily on the next save. No Redis needed.

Registration: begin

async registrationBegin(userId: string) {
  const user = await this.usersService.findById(userId);
  if (!user) throw new UnauthorizedException('User not found');

  const existing = await this.passkeys.find({ where: { userId } });

  const options = await generateRegistrationOptions({
    rpName: this.webauthn.rpName,
    rpID: this.webauthn.rpID,
    userName: user.email,
    userDisplayName:
      [user.firstName, user.lastName].filter(Boolean).join(' ') || user.email,
    userID: Buffer.from(user.id),
    attestationType: 'none',
    excludeCredentials: existing.map((c) => ({
      id: c.credentialId.toString('base64url'),
      transports: c.transports as AuthenticatorTransportFuture[],
    })),
    authenticatorSelection: {
      residentKey: 'required',
      userVerification: 'required',
    },
  });

  await this.saveChallenge('registration', userId, options.challenge);
  return options;
}

Three details that matter:

excludeCredentials is not optional if you care about UX. If you don't pass the user's existing credentials, the browser will happily let them register the same authenticator twice. On second use they'll try one, it'll work, they'll try the other, it'll also work, and they'll never know they have two identical entries with different friendly names cluttering their list. Pass the exclude list and the browser will grey out already-registered authenticators.

attestationType: 'none' is almost always what you want. Full attestation gives you the manufacturer's certificate for the authenticator ("this really is a YubiKey 5C serial ABC123"), which sounds cool but adds complexity, storage, and legal considerations (some jurisdictions treat that as PII). none still verifies the passkey works — you just skip the "which brand of authenticator was this" question.

residentKey: 'required' guarantees discoverable credentials. Authentication intentionally sends an empty allowCredentials list to avoid exposing credential IDs, so every passkey created by this application must be discoverable. The tradeoff is deliberate: authenticators that cannot create resident credentials are unsupported. userVerification: 'required' also requires the authenticator's PIN, biometric, or equivalent local verification.

Registration: finish

async registrationFinish(userId, response, friendlyName) {
  return this.withChallenge('registration', userId, async (stored, manager) => {
    const verification = await verifyRegistrationResponse({
      response,
      expectedChallenge: stored.challenge,
      expectedOrigin: this.webauthn.origins,
      expectedRPID: this.webauthn.rpID,
      requireUserVerification: true,
    });
    if (!verification.verified || !verification.registrationInfo) {
      throw new UnauthorizedException('Passkey registration failed');
    }

    // Duplicate lookup and insert use this same transaction. withChallenge
    // deletes the locked challenge only after this callback succeeds.
    const passkeys = manager.getRepository(UserPasskey);
    const info = verification.registrationInfo;
    const credentialId = Buffer.from(info.credential.id, 'base64url');
    if (await passkeys.findOne({ where: { credentialId } })) {
      throw new BadRequestException('This passkey is already registered');
    }
    return passkeys.save(passkeys.create({
      userId, credentialId,
      publicKey: Buffer.from(info.credential.publicKey),
      counter: String(info.credential.counter ?? 0),
      transports: info.credential.transports ?? [],
      deviceType: info.credentialDeviceType,
      backedUp: info.credentialBackedUp,
      friendlyName: friendlyName.trim().slice(0, 80),
      lastUsedAt: null,
    }));
  });
}

The failure modes and how they're handled:

Expired or missing challenge → withChallenge throws 401. Users see "session expired, try again."
Attestation verification fails → 401. Includes the case where the browser sent an assertion for a different RP ID (someone tried to phish the flow from a lookalike domain).
Duplicate credential → 400. Belt-and-suspenders on top of the excludeCredentials UX hint; if a user forces registration of an existing credential, we refuse cleanly.

Note the credentialIdBuf handling. The library returns info.credential.id as a base64url string, but we store bytes. Convert on the way in, convert on the way out when we hand it back to excludeCredentials or allowCredentials. Get this wrong in one direction and the unique index silently allows duplicates; get it wrong in the other and existing passkeys stop matching.

Authentication: begin

async authenticationBegin(email: string, tenantSlug: string) {
  const user = await this.usersService.findByEmailWithMemberships(email);

  // Never disclose account existence or registered credential IDs. Because
  // registration requires discoverable credentials, no allow-list is needed.
  const options = await generateAuthenticationOptions({
    rpID: this.webauthn.rpID,
    allowCredentials: [],
    userVerification: 'required',
  });
  await this.saveChallenge('authentication', user?.id ?? null, options.challenge);
  return options;
}

The critical property here — user enumeration resistance — comes from what the endpoint returns for a nonexistent email.

The wrong version (that everyone writes first) throws 404 when the user is missing. Which means an attacker can throw a wordlist at your endpoint and learn every email that has an account.

The version above always returns the same public shape: allowCredentials is empty for both real and nonexistent users. Discoverable credentials let the browser find the passkey without receiving credential IDs. The HTTP status and response shape are the same; strict timing normalization is still outside this example.

The browser will then pop the passkey picker anyway; the user with real credentials will succeed, the phantom will see the picker do nothing. But no observer can tell from the outside which is which.

Small smoke test I ran to prove it:

$ curl -s -X POST /api/passkeys/authentication/begin \
    -d '{"email":"admin@test.com","tenantSlug":"sentinel-labs"}'
{"rpId":"localhost","challenge":"8Dh-...","allowCredentials":[],"timeout":60000,"userVerification":"required"}

$ curl -s -X POST /api/passkeys/authentication/begin \
    -d '{"email":"ghost@nowhere.com","tenantSlug":"sentinel-labs"}'
{"rpId":"localhost","challenge":"UJlC...","allowCredentials":[],"timeout":60000,"userVerification":"required"}

Same shape, different challenges, no clue which email exists.

Authentication: finish

async authenticationFinish(response, tenantSlug, context) {
  const credentialId = Buffer.from(response.id, 'base64url');
  const candidate = await this.passkeys.findOne({ where: { credentialId } });
  if (!candidate) throw new UnauthorizedException('Unknown passkey');

  const passkey = await this.withChallenge(
    'authentication', candidate.userId, async (stored, manager) => {
      const repository = manager.getRepository(UserPasskey);
      const locked = await repository.findOne({
        where: { id: candidate.id },
        lock: { mode: 'pessimistic_write' },
      });
      if (!locked) throw new UnauthorizedException('Unknown passkey');

      // Reject, never round, a bigint the library cannot represent.
      const storedCounter = this.parseAuthenticatorCounter(locked.counter);
      const verification = await verifyAuthenticationResponse({
        response,
        expectedChallenge: stored.challenge,
        expectedOrigin: this.webauthn.origins,
        expectedRPID: this.webauthn.rpID,
        credential: {
          id: response.id,
          publicKey: new Uint8Array(locked.publicKey),
          counter: storedCounter,
          transports: locked.transports,
        },
        requireUserVerification: true,
      });
      if (!verification.verified) {
        throw new UnauthorizedException('Passkey authentication failed');
      }

      const newCounter = verification.authenticationInfo.newCounter;
      if (!Number.isSafeInteger(newCounter) || newCounter < 0) {
        throw new UnauthorizedException('Unsupported authenticator counter');
      }
      if (newCounter !== 0 && newCounter <= storedCounter) {
        throw new UnauthorizedException('Authenticator counter did not advance');
      }
      locked.counter = String(newCounter);
      locked.lastUsedAt = new Date();
      return repository.save(locked);
    },
  );

  // The production method continues here: resolve tenant and membership,
  // create the normal session, run anomaly analysis, then mint tokens.
}

Three things earn their weight here:

Counter validation can flag a cloned authenticator. For authenticators that maintain a non-zero signature counter, a value that does not advance is a clone or replay signal and authentication is rejected. Synced passkeys commonly report zero, so the counter is not a universal clone detector. Both stored and returned values must be safe integers because SimpleWebAuthn's API uses JavaScript numbers.

The credential object is what the library uses to verify the signature. The public key comes from your DB, the counter comes from your DB. If you pass zeros or the wrong values, verification fails silently (no error, just verified: false). If your DB has drift here, nothing about the error message will tell you why.

Everything after verified: true is a normal login. Once WebAuthn has proven "this is really the user who owns credential X," you're back in familiar auth territory: resolve tenant, load membership, mint a session, hand back a token pair. This is why the passkey module depends on the existing TokenService and SessionsService — no need to duplicate that logic.

The frontend piece

Server-only won't work here. navigator.credentials.create() and .get() are browser APIs — they must run in a client component. In a Next.js App Router world that means client components + server actions, with the WebAuthn calls sandwiched between:

'use client';

import { startAuthentication } from '@simplewebauthn/browser';
import { passkeyLoginBeginAction, passkeyLoginFinishAction } from './passkey-actions';

export function PasskeyLoginButton() {
  const [email, setEmail] = useState('');
  const [tenantSlug, setTenantSlug] = useState('sentinel-labs');

  async function onSignIn() {
    // 1. Ask server for options
    const options = await passkeyLoginBeginAction({ email, tenantSlug });

    // 2. Browser talks to authenticator (Touch ID prompt appears here)
    const response = await startAuthentication({ optionsJSON: options });

    // 3. Send assertion to server; server verifies + sets session cookies
    const result = await passkeyLoginFinishAction({ response, tenantSlug });
    if (result.ok) router.push('/app');
  }

  return <button onClick={onSignIn}>Sign in with passkey</button>;
}

The server actions do the auth-api round-trip and, on success, set HttpOnly session cookies before returning { ok: true }. The pattern is nice because it keeps tokens off the client — the browser never sees the JWT, only the Set-Cookie header.

One footgun worth calling out: startAuthentication and startRegistration in v13 take { optionsJSON: options } — not options directly. It's a wrapper introduced to give the library room to add more parameters. If you copy older docs (or an LLM autocompletes from them), you get an inscrutable type error.

Security properties, in one place

Verification boundary

The service-level test suite covers the security invariants around this transaction: failed verification does not request challenge deletion, credential/counter state is saved before the challenge is deleted, and two concurrent finish calls are serialized so only one succeeds. Those tests use a transaction-aware repository harness rather than a real PostgreSQL process. A PostgreSQL integration test should still be added in deployment CI to prove the configured driver and isolation level preserve the expected pessimistic_write behavior under real concurrent connections.

What this design actually gives you:

No user enumeration. authentication/begin returns the same shape for real and fake emails.
Phishing resistance. WebAuthn ties the credential to the exact origin. A lookalike domain (sytadεl-labs.com) will not produce a valid assertion, because the RP ID is baked into the signature.
Replay resistance per session. A row lock serializes finish attempts; the challenge is deleted only in the transaction that successfully verifies and updates credential state.
Cloned-authenticator detection. Counter regression is rejected.
Credential-level revocation. Delete one passkey from settings, only that device stops working. The password and other passkeys stay live.
No secret material stored server-side. All you have is the public key. A DB dump reveals nothing an attacker can use to authenticate.

What it deliberately doesn't do:

Doesn't kill passwords. Coexistence, not replacement. That's a UX call — you can make it stricter (requireUserVerification: 'required' + phase out password support once N% of users have a passkey), but rushing this locks people out.
Doesn't verify user identity end-to-end. The passkey proves possession of a specific device, not that the human at the keyboard is your user. Combine with device attestation or step-up flows if you need higher assurance for sensitive operations.

Tradeoffs vs the alternatives

TOTP (Google Authenticator / Authy). Better than passwords, worse than passkeys. Still phishable — the six-digit code can be socially engineered onto a fake login page. Also requires the user to reach for their phone every login. Good as a "step-up" second factor, not great as a primary.

Magic links. Zero friction, zero security. Anyone with access to the user's inbox owns the account. Fine for low-value applications, unacceptable for anything a security team needs to defend.

SMS. Please don't. SIM swaps are cheap, SS7 is a joke, and telcos have breached SMS databases multiple times. NIST retired SMS as a valid second factor in 2020.

Passwords + hardware token (U2F). A subset of what passkeys give you, with worse UX. If you're building today, WebAuthn/passkeys are the strictly-better version of everything U2F does.

Passkeys as the only login method. The direction the industry is moving, but too aggressive right now. Users lose devices, IT admins need break-glass access, and if your only auth path is a passkey and the authenticator dies, you're staring at an account recovery flow you probably haven't built yet. Ship coexistence first, tighten later.

Source

Backend: auth/auth-api/src/modules/passkeys/ in ElwinErnst/auth-api
Frontend: sytadel-app/src/features/auth/ in ElwinErnst/sytadel-suite
Runnable end-to-end: docker compose up --build from the meta-repo, then http://localhost:3003/login

If you spot a hole in this design or hit an edge case I didn't cover, drop it in the repo issues or in the comments — I want to hear it.

Building a tamper-evident audit log in NestJS with hash chains

Elwin Ernst — Wed, 05 Aug 2026 00:29:55 +0000

Most audit logs are a lie by omission.

You write a row that says "user X deleted document Y at 12:03." A week later, an incident review pulls it up, everyone nods, case closed. What nobody checks is whether the row was still there yesterday, whether the timestamp was the original one, and whether someone with database access quietly edited it after the fact.

If your compliance story depends on that log being trustworthy — SOC 2, HIPAA, GDPR Article 30, or a customer's InfoSec review — you need more than "we insert rows and hope."

This post walks through a small, boring, effective pattern I use for that: a hash-chained append-only audit log, implemented in NestJS + Postgres, with per-tenant chains and a verification routine that fails loudly if anyone tampers.

Full code lives in ElwinErnst/securechain-vault inside the Sytadel Suite.

The threat model

Assume an attacker who already has SQL access to the audit table. That's the person you're trying to catch — a rogue admin, a compromised backend, a poorly-scoped read-write DB user. If your logs can't survive them, they're theater.

Three concrete tamper actions to detect:

Modify a row — change the outcome from FAILURE to SUCCESS, rewrite the user ID, blank the metadata.
Delete a row — remove the trace of a specific action.
Reorder or insert rows — inject a fake event that "explains" a later observed state.

A regular audit table catches none of these. A hash chain catches all three.

The design: hash chain

Every row stores three cryptographic hashes:

eventHash — SHA-256 of the row's own payload (action, user, resource, HTTP context, metadata). If any field changes, this breaks.
prevHash — the chainHash of the immediately previous event in the chain. This is what stitches events together.
chainHash — SHA-256 of prevHash + "|" + eventHash. The cumulative fingerprint up to and including this event.

Plus a monotonic seq counter per chain and a scope (tenant ID or GLOBAL), so each tenant gets its own independent chain.

event 1: eventHash=H1, prevHash=null, chainHash=sha256("|" + H1) = C1
event 2: eventHash=H2, prevHash=C1,   chainHash=sha256(C1 + "|" + H2) = C2
event 3: eventHash=H3, prevHash=C2,   chainHash=sha256(C2 + "|" + H3) = C3

Tampering properties:

Modify event 2 → H2 changes → C2 changes → event 3's prevHash no longer matches → chain breaks from event 3 onward.
Delete event 2 → event 3's prevHash still points to C2 that no longer exists as any row's chainHash → verification fails.
Insert a fake event between 2 and 3 → (scope, seq) unique constraint blocks it, or if you renumber, both prevHash links break.
Reorder → same as above.

The only way to rewrite one event silently is to rewrite every event after it — and you'd still leave the last chainHash different from what any external witness saw.

The data model

@Index(['scope', 'seq'], { unique: true })
@Index(['tenantId', 'createdAt'])
@Entity({ name: 'audit_logs' })
export class AuditLogEntity {
  @PrimaryGeneratedColumn('uuid')
  id!: string;

  @Column({ type: 'uuid', name: 'tenant_id', nullable: true })
  tenantId!: string | null;

  @Column({ type: 'uuid', name: 'user_id', nullable: true })
  userId!: string | null;

  // scope = tenantId or 'GLOBAL' — per-tenant chains + one global chain
  @Column({ type: 'varchar', length: 64 })
  scope!: string;

  // Monotonic sequence within the scope
  @Column({ type: 'bigint' })
  seq!: string;

  @Column({ type: 'varchar', length: 80 })
  action!: string;

  @Column({ type: 'varchar', length: 60, name: 'resource_type' })
  resourceType!: string;

  @Column({ type: 'varchar', length: 120, name: 'resource_id', nullable: true })
  resourceId!: string | null;

  @Column({ type: 'varchar', length: 10 })
  outcome!: 'SUCCESS' | 'FAILURE';

  @Column({ type: 'int', name: 'http_status' })
  httpStatus!: number;

  @Column({ type: 'varchar', length: 10, name: 'http_method' })
  httpMethod!: string;

  @Column({ type: 'varchar', length: 255, name: 'http_path' })
  httpPath!: string;

  @Column({ type: 'inet', nullable: true })
  ip!: string | null;

  @Column({ type: 'varchar', length: 255, name: 'user_agent', nullable: true })
  userAgent!: string | null;

  @Column({ type: 'jsonb', nullable: true })
  metadata!: Record<string, unknown> | null;

  @Column({ type: 'char', length: 64, name: 'event_hash' })
  eventHash!: string;

  @Column({ type: 'char', length: 64, name: 'prev_hash', nullable: true })
  prevHash!: string | null;

  @Column({ type: 'char', length: 64, name: 'chain_hash' })
  chainHash!: string;

  @CreateDateColumn({ type: 'timestamptz', name: 'created_at' })
  createdAt!: Date;
}

Two details that matter:

(scope, seq) is unique. This is the constraint that blocks silent insertion of fake rows. Postgres will refuse a second row with seq = 5 in the same scope.
bigint for seq, stored as string in TypeORM. JavaScript's Number loses precision past 2^53. For long-lived audit tables you want the full 64-bit range.

Deterministic serialization

Hashing arbitrary JavaScript objects is a trap: JSON.stringify({a: 1, b: 2}) and JSON.stringify({b: 2, a: 1}) produce different strings. The hash of your event would depend on how V8 happened to order the keys today, which is not a property you want to bet compliance on.

So the first primitive is a deterministic stringify that sorts keys recursively:

function isPlainObject(value: unknown): value is Record<string, unknown> {
  return value != null && typeof value === 'object' && !Array.isArray(value);
}

export function stableStringify(input: unknown): string {
  if (Array.isArray(input)) {
    return `[${input.map(stableStringify).join(',')}]`;
  }
  if (isPlainObject(input)) {
    const keys = Object.keys(input).sort();
    const props = keys.map((key) => `"${key}":${stableStringify(input[key])}`);
    return `{${props.join(',')}}`;
  }
  return JSON.stringify(input);
}

export function sha256Hex(s: string): string {
  return createHash('sha256').update(s).digest('hex');
}

Twenty lines, no dependencies, does exactly one job.

Writing an event: `createChained`

The write path has to do three things atomically: pick the next seq, read the previous chainHash, and insert the new row. If two requests race, they must not both grab seq = 5.

async createChained(input: CreateAuditLogInput): Promise<AuditLogEntity> {
  const scope = input.tenantId ?? 'GLOBAL';

  return this.dataSource.transaction(async (manager) => {
    const r = manager.getRepository(AuditLogEntity);

    // Lock the last event of this scope for write.
    // This serializes writes within one scope but does NOT
    // serialize across scopes — each tenant has its own chain.
    const last = await r
      .createQueryBuilder('a')
      .setLock('pessimistic_write')
      .where('a.scope = :scope', { scope })
      .orderBy('a.seq', 'DESC')
      .limit(1)
      .getOne();

    const nextSeq = last ? (BigInt(last.seq) + 1n).toString() : '1';
    const prevHash = last?.chainHash ?? null;

    const eventPayload = {
      scope, seq: nextSeq,
      tenantId: input.tenantId, userId: input.userId,
      action: input.action,
      resourceType: input.resourceType, resourceId: input.resourceId,
      outcome: input.outcome, httpStatus: input.httpStatus,
      httpMethod: input.httpMethod, httpPath: input.httpPath,
      ip: input.ip, userAgent: input.userAgent,
      metadata: input.metadata,
    };

    const eventHash = sha256Hex(stableStringify(eventPayload));
    const chainHash = sha256Hex(`${prevHash ?? ''}|${eventHash}`);

    const row = r.create({
      ...input, scope, seq: nextSeq,
      prevHash, eventHash, chainHash,
    });

    return r.save(row);
  });
}

Two things worth calling out:

The lock is per-scope, not global. Because scope = tenantId, tenant A's writes never block tenant B's writes. This is the difference between "audit log scales with tenants" and "audit log is your bottleneck." A multi-tenant SaaS with N active tenants can write N chains in parallel.

The eventPayload is a subset of the row. We deliberately don't hash id (assigned by Postgres later) or createdAt (also assigned later). If you hash things the database owns, you can't precompute the hash before the insert. Keep the hashed payload to fields the application controls.

Automatic cross-cutting logging

Writing createChained calls in every controller is how audit logs stop happening. NestJS interceptors solve this by wrapping every request that has an @Audit(...) decorator:

@Injectable()
export class AuditInterceptor implements NestInterceptor {
  constructor(
    private readonly reflector: Reflector,
    private readonly audit: AuditService,
  ) {}

  intercept(context: ExecutionContext, next: CallHandler): Observable<unknown> {
    const req = context.switchToHttp().getRequest<RequestWithContext>();
    const res = context.switchToHttp().getResponse<Response>();

    const meta = this.reflector.get<AuditMeta>(AUDIT_META_KEY, context.getHandler());

    const base = {
      tenantId: req.tenantContext?.tenantId ?? null,
      userId: req.user?.id ?? null,
      action: meta?.action ?? `${req.method} ${req.path}`,
      resourceType: meta?.resourceType ?? 'http',
      resourceId: resolveResourceId(req, meta),
      httpMethod: req.method,
      httpPath: req.path,
      ip: resolveIp(req),
      userAgent: req.headers['user-agent'] ?? null,
    };

    return next.handle().pipe(
      mergeMap((data) =>
        from(this.audit.createChained({
          ...base,
          outcome: 'SUCCESS',
          httpStatus: res.statusCode,
          metadata: meta?.metadata ?? null,
        })).pipe(mergeMap(() => [data])),
      ),
      catchError((err) =>
        from(this.audit.createChained({
          ...base,
          outcome: 'FAILURE',
          httpStatus: err?.getStatus?.() ?? 500,
          metadata: { errorName: err?.name, errorMessage: err?.message },
        })).pipe(mergeMap(() => throwError(() => err))),
      ),
    );
  }
}

A controller opts in with:

@Delete(':id')
@Audit({ action: 'document.delete', resourceType: 'document', resourceIdParam: 'id' })
async deleteDocument(@Param('id') id: string) { ... }

Success and failure both get logged, with HTTP status, error name, and error message on failure. No controller ever calls the audit service directly.

Verification: reading the chain back

Writing is only half. The whole point is being able to say "yes, this log is still intact" and prove it. A verification routine walks the chain in order and re-computes everything:

async verifyChain(scope: string): Promise<VerifyResult> {
  const events = await this.repo
    .createQueryBuilder('a')
    .where('a.scope = :scope', { scope })
    .orderBy('a.seq', 'ASC')
    .getMany();

  let prevHash: string | null = null;
  let expectedSeq = 1n;

  for (const e of events) {
    // 1. seq must be strictly monotonic starting at 1
    if (BigInt(e.seq) !== expectedSeq) {
      return { ok: false, brokenAt: e.seq, reason: 'seq gap or reorder' };
    }

    // 2. eventHash must match the recomputed hash of the payload
    const recomputed = sha256Hex(stableStringify({
      scope: e.scope, seq: e.seq,
      tenantId: e.tenantId, userId: e.userId,
      action: e.action,
      resourceType: e.resourceType, resourceId: e.resourceId,
      outcome: e.outcome, httpStatus: e.httpStatus,
      httpMethod: e.httpMethod, httpPath: e.httpPath,
      ip: e.ip, userAgent: e.userAgent,
      metadata: e.metadata,
    }));
    if (recomputed !== e.eventHash) {
      return { ok: false, brokenAt: e.seq, reason: 'row was modified' };
    }

    // 3. prevHash must match the previous row's chainHash
    if (e.prevHash !== prevHash) {
      return { ok: false, brokenAt: e.seq, reason: 'prevHash mismatch' };
    }

    // 4. chainHash must match sha256(prevHash + "|" + eventHash)
    const expectedChain = sha256Hex(`${prevHash ?? ''}|${e.eventHash}`);
    if (e.chainHash !== expectedChain) {
      return { ok: false, brokenAt: e.seq, reason: 'chainHash mismatch' };
    }

    prevHash = e.chainHash;
    expectedSeq++;
  }

  return { ok: true, verified: events.length, headChainHash: prevHash };
}

The headChainHash at the end is the single value you can print, sign, tweet, or ship to an external witness. If tomorrow anyone modifies a single field of any row, the recomputed headChainHash will be different from the one you published today.

Benchmarks

Setup: NestJS 10 + Postgres 16 in Docker Compose, MacBook Pro M2. Single vault-api instance, single Postgres. Benchmark script calls GET /vault/vaults (an audited endpoint) via the Zero Trust gateway so each request round-trips through auth + gateway + vault + audit write.

Sequential write, single scope (concurrency = 1, N = 1,000):

Metric	Value
Throughput	205.8 writes/sec
Latency p50	4.7 ms
Latency p95	6.3 ms
Latency p99	8.6 ms
Latency max	12.4 ms

Storage:

Metric	Value
Rows	3,258
Total table size (heap + indexes)	3,728 kB
Heap size	2,712 kB
Bytes per row (heap)	~852 bytes

Chain verification (N = 3,238 in one scope):

Phase	Time
Fetch all rows from Postgres	85–107 ms
Recompute + verify chain	13–17 ms
End-to-end verify	~100 ms

The verify step alone processes ~250k rows/sec on a single thread — the bottleneck is fetching from Postgres, not the crypto. A 1M-row chain verifies in under 30 seconds on this hardware.

The concurrent-write bug (and the fix)

The sequential numbers hide a subtle race. Running the same bench with concurrency = 20 against a single scope surfaced it immediately: ~55% of requests failed with duplicate key value violates unique constraint "audit_logs_scope_seq" (Postgres error 23505), and the vault returned HTTP 500.

Root cause: under READ COMMITTED isolation, two transactions can both read seq = N before either has inserted seq = N+1. The SELECT ... FOR UPDATE LIMIT 1 locks the existing row it found, not the nonexistent row about to be inserted. Both transactions then try to insert seq = N+1 and the unique constraint aborts the second one.

Three ways to fix it, in increasing order of correctness:

App-side retry with backoff on 23505. Simple, works, but you're papering over the race instead of preventing it.
pg_advisory_xact_lock(hashtext(scope)) at the start of the transaction. Serializes writes per-scope explicitly. Cheap, deterministic. The lock releases automatically at COMMIT.
SERIALIZABLE transaction isolation. Postgres aborts conflicting transactions cleanly and you retry. Most correct, slightly more overhead across the whole app.

The fix I shipped (commit 556f7c3) uses the advisory lock. hashtext(scope) returns int32, which pg_advisory_xact_lock accepts, and the lock scopes to the transaction — no cleanup, no leaks.

Re-running the same concurrency = 20 bench after the fix:

	Before	After
Requests succeeded	228 / 500	500 / 500
23505 errors	272	0
Throughput	167 req/sec	497 req/sec
Latency p95	31 ms	60 ms

Sequential throughput is essentially unchanged (198 vs 205 req/sec, +1 ms p95 for the extra advisory-lock round-trip). Chain verification still passes end-to-end after the fix.

The p95 went up under concurrency because requests now wait for the lock instead of failing fast — that's the tradeoff you want. Serialized-and-slow beats fast-and-broken every time in an audit context.

For an audit log, per-scope serialization is not a scaling loss — it's the semantics you want anyway (one chain = one linear history). The original implementation just picked the wrong primitive to enforce it.

Cross-scope scaling

Because scope = tenantId, different tenants never contend on the same lock. Multi-tenant throughput scales close to linearly with the number of active tenants until you hit Postgres write ceiling. A single-instance Postgres 16 on modest hardware sustains a few thousand writes/sec across many chains without breaking a sweat.

Tamper detection, live

To prove the chain actually catches modifications, I ran a quick experiment against the same 3,238-row table:

# 1. Modify one row (seq=100 flipped from SUCCESS to FAILURE)
UPDATE audit_logs
   SET outcome = 'FAILURE'
 WHERE scope = 'f4a...' AND seq = 100;

# 2. Run verify
$ node verify-chain.mjs
Fetched 3238 rows in 85 ms

== VERIFY RESULT ==
rows verified  : 3238
ok             : false
broken at seq  : 100
reason         : eventHash mismatch (row modified)
head chainHash : caae1db44dcd4391...

# 3. Revert
UPDATE audit_logs
   SET outcome = 'SUCCESS'
 WHERE scope = 'f4a...' AND seq = 100;

# 4. Verify again
$ node verify-chain.mjs
rows verified  : 3238
ok             : true
head chainHash : 80f73959d1ba21bc...

Two things worth noting:

The verify pinpoints the exact row where the chain broke, not just "something's wrong."
After reverting, the headChainHash is the same value it had before the tamper (80f73959d1ba21bc...). If you published that hash yesterday to an external witness — a tweet, a Slack, a customer email — you can now prove today that the log is byte-identical to yesterday's.

Tradeoffs vs. other approaches

Merkle tree. Verification is O(log N) instead of O(N), and you can prove that a specific event is or isn't in the tree without walking everything. Worth it if you need selective inclusion proofs (e.g. showing an auditor "here's your event, here's the proof it's in the log we published") without exposing the rest of the log. Extra machinery, more code paths to get wrong.

Anchoring in a blockchain. Publish periodic chainHash snapshots to Bitcoin (via OpenTimestamps) or Ethereum. You get an external, adversarial witness — nobody can rewrite Bitcoin to cover up a log tamper. Downside: cost, latency, and operational complexity. Good for milestone checkpoints (daily/weekly), overkill for every event.

HSM-signed logs. Sign each event with a hardware key. You get non-repudiation on top of integrity — you can prove who wrote the log, not just that it's intact. Requires HSM infrastructure, adds latency to every write.

Just a hash chain in Postgres, like this post. Simple, fast, verifiable, zero external dependencies. Doesn't survive an attacker who also controls the process that writes the log — they can forge a consistent chain. Pair it with periodic blockchain anchoring if that's your threat model.

For most SaaS products, the "boring hash chain in Postgres" is exactly the right ceiling: it detects the realistic tamper (someone with DB access) at zero extra ops cost, and it composes cleanly with blockchain anchoring later if you need it.

Source

Full implementation: ElwinErnst/securechain-vault, inside the Sytadel Suite meta-repo (docker-compose + smoke tests + all four backend services + Astro landing).

Relevant files:

vault-api/src/common/utils/audit-hash.util.ts — stableStringify + sha256Hex
vault-api/src/modules/audit/audit.service.ts — createChained
vault-api/src/common/interceptors/audit.interceptor.ts — automatic cross-cutting logging
vault-api/src/database/entities/audit-log.entity.ts — the row shape

If you build something similar or spot a hole in this design, I want to hear it. The comments here (and the repo issues) are open.

DEV Community: Elwin Ernst

Session anomaly detection in a multi-tenant auth service — with the real bugs I hit

What "anomaly" means here

Data model

The core: analyze()

Hooking it in

Four bugs I hit while building this — worth the specifics

1. geoip-lite@2.x requires Node ≥24

2. import geoip from 'geoip-lite' returned undefined

3. Express ignored X-Forwarded-For

4. The analyzer saw the current session as history

Tradeoffs vs. the LLM/ML approach

What this doesn't cover (and shouldn't yet)

Source

Automated secret rotation with a 24-hour grace window — done right in NestJS + Postgres

The mental model

Schema additions

Rotation (shared for manual and scheduled)

The verify path (this is where it usually goes wrong)

The scheduler

Setting the policy

What this doesn't cover (on purpose)

Tradeoffs vs. other approaches

The four things worth flagging

Source

Harden for the reader: three red flags an AppSec engineer hits on `git clone`

Red flag 1 — synchronize: true

Red flag 2 — change-me-* secrets hardcoded in compose

Red flag 3 — anti-replay in an in-memory Map

The through-line

Source

Adding WebAuthn to a multi-tenant NestJS + Next.js app: what nobody tells you

What you're building

The library

Data model

Registration: begin

Registration: finish

Authentication: begin

Authentication: finish

The frontend piece

Security properties, in one place

Verification boundary

Tradeoffs vs the alternatives

Source

Building a tamper-evident audit log in NestJS with hash chains

The threat model

The design: hash chain

The data model

Deterministic serialization

Writing an event: createChained

Automatic cross-cutting logging

Verification: reading the chain back

Benchmarks

The concurrent-write bug (and the fix)

Cross-scope scaling

Tamper detection, live

Tradeoffs vs. other approaches

Source

The core: `analyze()`

1. `geoip-lite@2.x` requires Node ≥24

2. `import geoip from 'geoip-lite'` returned undefined

3. Express ignored `X-Forwarded-For`

Red flag 1 — `synchronize: true`

Red flag 2 — `change-me-*` secrets hardcoded in compose

Red flag 3 — anti-replay in an in-memory `Map`

Writing an event: `createChained`