OmniJudge Post-Mortem: 1-ULP Drift, Prisma Traps, and Alpine musl

Vineet Wagh — Thu, 01 Oct 2026 06:23:29 +0000

Evaluating hackathons at scale is deceptively difficult.

When 40 software projects are evaluated across 4 specialized tracks by 30 volunteer judges using a 4-criteria weighted rubric, naive arithmetic means fail catastrophically:

Hawk vs. Dove Skew: Strict judges award scores between 2.0 and 3.5. Lenient judges award scores between 4.0 and 5.0. A project evaluated by two hawks loses to a mediocre project evaluated by two doves if raw scores are averaged.
Incomplete Block Design (IBD): Judges cannot evaluate all 40 projects without severe cognitive fatigue. Each judge inspects an incomplete cohort of 8 to 12 projects, meaning the Law of Large Numbers does not balance out calibration variance.

We built OmniJudge with Modified Z-Score (MAD) normalization, server-enforced track jurisdictions, sealed wire redaction, and append-only auditability.

THE OMNIJUDGE ENGINE PIPELINE:
[40 Projects / 30 Judges]
   │
   ▼
[Track Jurisdiction & COI Defense]
   │ ──► Hour 60 Bug: Unassigned Judge WHERE 1=1 (Patched)
   ▼
[MAD Normalization Engine]
   │ ──► NaN on Zero Variance & 1-ULP 10^15 Float Drift (Patched)
   ▼
[4-Stage Deterministic Tie-Breaker]
   │ ──► Unreviewed Project Score Inversion (Patched)
   ▼
[RFC 4180 CSV Streamer]
   │ ──► CWE-1236 Formula Injection Sanitizer (Patched)

During the 72-hour build, three silent failures threatened the integrity of the platform:

The IEEE-754 1-ULP Drift: Microscopic floating-point discrepancies in rubric composite score computations caused standard zero-variance checks to fail, inflating normalized scores by a factor of 3 × 10¹⁵.
The Prisma Parameter-Drop Trap: An ORM query compiler behavior at Hour 60 inadvertently dropped SQL WHERE clauses for unassigned judges, granting unauthorized god-mode visibility into all 40 submissions.
The Unreviewed Score Inversion: Mathematical normalization placed unreviewed projects (baseline z = 0.0000) above legitimate submissions that received negative normalized scores from rigorous judges.

Here is the raw engineering post-mortem covering what broke, why it broke, and how we hardened it.

1. The Architecture Decision That Shaped Everything: Killing WebSockets for Air-Gapped Invariants

At Hour 20, we had an ambitious Figma-style "live collaborative judging room." Judges on the same track saw real-time peer cursors, instant score ripples, and synchronized notes over WebSockets backed by Redis Pub/Sub.

At Hour 36, we ruthlessly deleted the entire WebSocket engine and excised Redis from our stack.

Two non-negotiable architectural constraints forced this decision:

1.1 Air-Gapped Single-Container Invariants

Automated hackathon evaluators (run.py) test applications in isolated Docker environments (often with --network none). Requiring a Redis sidecar violated our single-container Alpine Linux runtime mandate.

A single-process Next.js + SQLite binary boots in 800ms, has zero external daemon dependencies, and runs reliably in any air-gapped or network-restricted container.

1.2 Cognitive Anchoring Bias (Decision Theory)

In behavioral economics (Kahneman & Tversky), peer awareness during evaluation introduces severe anchoring cascades. If Judge Alpha sees Judge Beta award a 5.0 in real-time, Judge Alpha's independent scoring is subconsciously anchored upward.

Modified Z-Score normalization mathematically requires statistical independence across reviewer observations. Real-time collaborative WebSockets actively undermined the mathematical validity of our judging engine.

The Replacement Architecture

Instead of stateful WebSockets, we implemented:

Atomic HTTP Transactions: Scoring updates occur over atomic HTTP POST requests wrapped in prisma.$transaction.
Append-Only SQLite Audit Trail: Every score write, vote, and setting toggle writes an immutable before/after diff to the AuditLog table (never updated, never deleted).
Detached Asynchronous Webhook Dispatcher: For external integrations, we engineered an asynchronous HMAC-SHA256 webhook dispatcher (src/lib/webhooks.ts) bounded by a strict 4-second AbortController timeout to prevent SSRF and event loop blocking:

// src/lib/webhooks.ts
export async function dispatchWebhookEvent<T = unknown>(
  event: WebhookEventName,
  data: T
): Promise<{ dispatchedCount: number; errors: string[] }> {
  // Queries active subscribers and dispatches asynchronously
  // Enforces 4,000ms AbortController timeout to prevent SSRF and thread-hangs
  const controller = new AbortController();
  const timeoutId = setTimeout(() => controller.abort(), 4000);

  const res = await fetch(sub.url, {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'X-OmniJudge-Signature': signWebhookPayload(serialized, sub.secret),
      'X-OmniJudge-Event': event,
      'User-Agent': 'OmniJudge-Webhook-Engine/1.0',
    },
    body: serialized,
    signal: controller.signal,
  });
  clearTimeout(timeoutId);
}

The system achieved zero external dependencies, 100% offline air-gapped compliance, and imperviousness to external subscriber latency.

2. The Docker Gotcha That Cost Us Hours: Alpine musl, OpenSSL 3, and Ephemeral Container Boots

Single-container evaluation (docker run -p 8080:8080) sounds trivial until you combine Node.js 20 on Alpine Linux, Prisma ORM, SQLite volume mounts, and cross-platform developer workstations.

What should have taken 20 minutes cost over 6 hours of forensic debugging across commits b03425c, 38eabd1, and cfdea6c.

THE DOCKER TRAP TAXONOMY:
1. Windows Host (CRLF) ──► entrypoint.sh ──► Alpine /bin/sh: not found (\r trap)
2. node:20-alpine (musl) ──► Prisma Engine ──► Expected linux-musl-openssl-3.0.x
3. Empty Volume Mount ──► SQLite file:/data/dogfood.db ──► SQLITE_CANTOPEN crash
4. Ephemeral Evaluator ──► prisma migrate deploy ──► Migration history desync

Gotcha A: The Alpine musl libc & OpenSSL 3 BinaryTarget Mismatch (Commit `b03425c`)

To minimize image size, we selected node:20-alpine. However, Alpine uses musl libc instead of GNU glibc. Prisma's query engine is a native Rust binary compiled specifically for the host C library.

When the container booted, Next.js initialized, but the first database query crashed instantly:

PrismaClientInitializationError: Unable to load query engine library.
Expected linux-musl-openssl-3.0.x, found nothing in /app/node_modules/.prisma/client

Locally on Windows, Prisma downloaded windows-x64. In Debian, it expects debian-openssl-3.0.x. In Alpine 3.19+, it requires linux-musl-openssl-3.0.x AND a runtime OpenSSL library.

The Fix:

In prisma/schema.prisma, explicitly declare the Alpine target:

generator client {
  provider      = "prisma-client-js"
  binaryTargets = ["native", "linux-musl-openssl-3.0.x"]
}

In Dockerfile, install OpenSSL in both builder and runner stages:

RUN apk add --no-cache openssl

Gotcha B: The Windows CRLF Line-Ending Trap

Editing entrypoint.sh on Windows checked out CRLF (\r\n) line endings. When copied into Alpine Linux, /bin/sh tried to parse the shebang line as #!/bin/sh\r. Because no binary named /bin/sh\r exists, the Linux kernel threw an infamously misleading error:

/bin/sh: ./entrypoint.sh: not found

The file existed, was marked executable (chmod +x), and was readable, yet the kernel claimed it didn't exist!

The Fix:
In Dockerfile, add an explicit carriage-return strip:

COPY entrypoint.sh ./entrypoint.sh
RUN sed -i 's/\r$//' ./entrypoint.sh && chmod +x ./entrypoint.sh

Gotcha C: The Missing `/data` Mount Void (`SQLITE_CANTOPEN`, Commit `38eabd1`)

The platform persists state to DATABASE_URL="file:/data/dogfood.db".

When running docker run -v dogfood_data:/data, Docker mounts the volume. But during cold acceptance tests where no host directory was pre-created, the container started without a /data folder on disk. SQLite does not recursively create missing parent directories; it failed with SQLITE_CANTOPEN (code 14).

The Fix:
In entrypoint.sh, ensure directory creation prior to invoking Prisma:

# Ensure data directory exists for SQLite database
mkdir -p /data

Gotcha D: `prisma migrate deploy` vs. `prisma db push` (Commit `cfdea6c`)

In standard CI/CD, database migrations use npx prisma migrate deploy. However, in automated evaluator environments (run.py), containers are spun up from scratch, mounted to ephemeral storage, and seeded with synthetic datasets (fixtures.json).

migrate deploy frequently failed with drift detection errors or complained about missing _prisma_migrations records on freshly initialized SQLite files.

The Fix:
We switched the boot sequence to prisma db push:

echo "[1/3] Synchronizing database schema..."
npx prisma db push --accept-data-loss --skip-generate

echo "[2/3] Seeding database from fixtures..."
npx tsx src/lib/seed.ts

echo "[3/3] Starting Next.js on port 8080..."
exec node server.js

prisma db push synchronizes the schema in-memory directly against SQLite in under 200ms, completely avoiding migration state lockups and providing an air-gapped, zero-drift boot sequence.

3. The Scoring Algorithm We Almost Shipped Before Realizing It Was Broken

Why Classical Z-Score Collapsed

The standard Z-score relies on sample mean μ and sample standard deviation σ:

z_i = (x_i - μ) / σ

In our acceptance fixtures (Hack_docs/fixtures.json), judge jdg_30 (Rafa Okonkwo) gave every assigned project an identical score of 4.0:

X = [4.0, 4.0, 4.0, 4.0] ⟹ μ = 4.0, σ = 0.0

The standard Z-score evaluated to:

z_i = (4.0 - 4.0) / 0.0 = 0 / 0 = NaN

In JavaScript, NaN poisons Array.prototype.sort(), causing total corruption of the leaderboard array and throwing exceptions during CSV serialization.

Transitioning to Modified Z-Score (MAD)

To achieve robustness against extreme outliers (a 50% breakdown point), we implemented the Boris Iglewicz and David Hoaglin Modified Z-Score:

modified_z_i = 0.6745 × (x_i - median) / MAD

Where:

median(X) is the median score awarded by the judge.
MAD is the Median Absolute Deviation: MAD = median(|x_i - median|).

Mathematical Derivation of the Constant 0.6745

The constant 0.6745 is the exact mathematical factor required to make the Modified Z-Score asymptotically equivalent to the standard Z-Score under a standard normal distribution N(0, 1).

For a standard normal random variable Z ~ N(0, 1), the median is 0. By definition, the population MAD is the value mad satisfying:

P(|Z| ≤ mad) = 0.50
2Φ(mad) - 1 = 0.50 ⟹ Φ(mad) = 0.75
mad = Φ⁻¹(0.75) ≈ 0.67448975...

For normally distributed data with standard deviation σ:

E[MAD] = Φ⁻¹(0.75) × σ ≈ 0.6745 × σ ⟹ σ ≈ MAD / 0.6745 ≈ 1.4826 × MAD

Multiplying (x_i - median) by 0.6745 / MAD ensures that when scores follow a Gaussian distribution, the Modified Z-Score is on the exact same scale as the classical Z-Score.

The IEEE-754 1-ULP Drift Catastrophe (3 × 10¹⁵ Blowup)

While MAD resolved NaN on theoretical zero-variance sets, a far more sinister bug emerged in production during floating-point composite evaluations.

When criteria weights (e.g., 1.5, 1.0, 1.0, 0.5) were applied to criterion scores, decimal fractions could not be represented exactly in IEEE-754 64-bit binary floating-point representation.

For two projects evaluated identically by a judge, weighted composite computations yielded:

Project A Composite: 4.0000000000000000
Project B Composite: 4.0000000000000009 (differing by 1 Unit in the Last Place, ULP ≈ 2.22 × 10⁻¹⁶)

When computing MAD:

Deviations: [0.0, 8.88 × 10⁻¹⁶, 0.0, ...]
Computed MAD: MAD = 2.220446049250313 × 10⁻¹⁶

A naive zero check if (mad === 0) evaluated to false! The engine proceeded to compute:

modified_z = 0.6745 × (4.0000000000000009 - 4.0) / (2.22 × 10⁻¹⁶) ≈ +3.0375 × 10¹⁵

A microscopic rounding error in the 16th decimal place exploded the project's normalized score to over 3 quadrillion, rocketing a mediocre project to the global #1 rank!

The Production Fix (`src/lib/normalization.ts`)

We neutralized this failure mode by introducing an explicit floating-point epsilon threshold (EPSILON = 1e-9):

// src/lib/normalization.ts
export const EPSILON = 1e-9;

export function normaliseJudgeScores(
  scores: number[],
  options?: NormaliseOptions
): number[] {
  if (scores.length === 0) return [];
  if (!scores.every((s) => typeof s === 'number' && Number.isFinite(s))) {
    return scores.map(() => 0);
  }

  const sorted = [...scores].sort((a, b) => a - b);
  const mid = Math.floor(sorted.length / 2);

  const median =
    sorted.length % 2 !== 0
      ? sorted[mid]
      : (sorted[mid - 1] + sorted[mid]) / 2;

  const deviations = scores.map((s) => Math.abs(s - median));
  const sortedDevs = [...deviations].sort((a, b) => a - b);

  const mad =
    sortedDevs.length % 2 !== 0
      ? sortedDevs[mid]
      : (sortedDevs[mid - 1] + sortedDevs[mid]) / 2;

  // Zero-variance & floating-point precision guard:
  // Catches identical scores, single review panels, and IEEE-754 1-ULP drift.
  // When MAD < EPSILON, return neutral zeros — no signal, no 10^15 score blowup.
  if (mad < EPSILON) {
    return scores.map(() => 0);
  }

  const minSpread = options?.minSpread;
  const scale = minSpread !== undefined ? Math.max(mad, minSpread / 0.6745) : mad;
  const shrink = options?.shrink ? scores.length / (scores.length + 3) : 1;

  return scores.map((s) => (shrink * 0.6745 * (s - median)) / scale);
}

If a judge awards identical scores (or scores differing only by IEEE-754 precision noise), MAD < 10⁻⁹. The engine assigns neutral 0.0 modified Z-scores. The judge provides zero discriminating signal, completely protecting the ranking engine from division-by-zero crashes and quadrillion-scale score explosions.

4. The Part of the Spec We Thought Was Simple Until We Built It

Two spec requirements appeared straightforward on paper but contained severe edge cases:

4.1 Spec Trap 1: "Export evaluation results to CSV"

A. CWE-1236 CSV Formula Injection Defense

Adversarial participants submitted project titles starting with =cmd|'/C calc'!A0 or @SUM(A1:A10). When opened in Microsoft Excel, these execute arbitrary commands.

However, naive sanitization that prefixes - with ' breaks negative numbers (e.g., a legitimate negative normalized score like -0.6745 becomes '-0.6745, corrupting numerical calculations).

We solved this with regex-based semantic sanitization in escapeCsvField:

// src/app/api/export.csv/route.ts
function escapeCsvField(value: string | number): string {
  let str = String(value ?? '');

  // CSV Formula Injection Defense (CWE-1236):
  // Neutralize formula triggers on text strings while preserving valid negative floats
  if (/^[=+\-@\t\r]/.test(str) && typeof value === 'string' && isNaN(Number(str))) {
    str = `'${str}`;
  }

  // RFC 4180 Escaping: Wrap fields containing commas, quotes, or newlines in quotes
  if (str.includes(',') || str.includes('"') || str.includes('\n') || str.includes('\r')) {
    return `"${str.replace(/"/g, '""')}"`;
  }
  return str;
}

B. The Unreviewed Project Score Inversion Trap

An unreviewed project has a baseline normalized score of 0.0000.
An evaluated project that received strict scores from a hawk judge might have a normalized score of -0.4500.

If sorted purely by score, unreviewed projects outrank evaluated projects:

0.0000 (Unreviewed Project B) > -0.4500 (Evaluated Project A)

We introduced a status partition invariant:
Evaluated projects (reviewCount > 0) strictly outrank unreviewed projects (reviewCount === 0), regardless of whether the normalized score is negative.

C. Deterministic 4-Stage Tie-Breaking

Ties are resolved deterministically through a 4-tier comparator (src/lib/ranking.ts):

Tier 1: Evaluated projects strictly outrank unreviewed (reviewCount > 0).
Tier 2: Normalized score DESC with ε = 10⁻⁹ floating-point tolerance.
Tier 3: Raw weighted composite score DESC with ε = 10⁻⁹ tolerance.
Tier 4: Lexicographical project ID ASC.

4.2 Spec Trap 2: "Scope judges to assigned tracks" (The Hour 60 Prisma Parameter-Drop Bug)

At Hour 60, during adversarial role-probing (tests/test_phase3_adversarial.py), an unassigned judge (assignedTracks = [], trackIds = []) logged in.

In src/app/judge/page.tsx, the project query was written as:

// THE FATAL FLAW AT HOUR 60:
const trackIds = assignedTracks.map((t) => t.id);

const projects = await prisma.project.findMany({
  where: {
    trackId: trackIds.length > 0 ? { in: trackIds } : undefined, // <-- THE TRAP
  },
  include: { team: true, track: true },
});

The developer intended: "If the judge has assigned tracks, filter by them; if not, pass undefined so Prisma knows there are no tracks."

The Prisma Query Compiler AST Trap

In Prisma Client, passing undefined to a where property instructs the query compiler to strip that field from the SQL query AST entirely.

Prisma did not compile trackId: undefined into WHERE trackId IN (NULL) or WHERE 1=0. It compiled it to:

-- Generated SQL when trackIds.length === 0 and trackId is undefined:
SELECT `id`, `title`, `summary`, `trackId`, `teamId` 
FROM `Project`;
-- (WHERE clause completely omitted! Effectively WHERE 1=1)

An unassigned judge with zero clearance was granted god-mode visibility into all 40 submissions across every track!

The Fix: Relational Defense in Depth (Commit `a059d80` and `121138c`)

// src/app/judge/page.tsx
const trackIds = assignedTracks.map((t) => t.id);
const projectsWhere: {
  trackId: { in: string[] };
  teamId?: { not: string };
} = {
  // Always pass { in: trackIds }. In SQLite/Prisma, [] evaluates to 0 records
  trackId: { in: trackIds },
};

// Check if judge is affiliated with any team (Conflict of Interest defense)
if (judgeTeamMember?.teamId) {
  projectsWhere.teamId = { not: judgeTeamMember.teamId };
}

const projects = await prisma.project.findMany({
  where: projectsWhere,
  include: { team: true, track: true },
  orderBy: { id: 'asc' },
});

And in POST /api/judge/scores/route.ts:

// Enforce Conflict of Interest Bar
const isTeamMember = await prisma.teamMember.findFirst({
  where: {
    userId: session.id,
    teamId: project.teamId,
  },
});
if (isTeamMember) {
  return NextResponse.json(
    { error: 'Forbidden: Conflict of interest — judges cannot evaluate their own team project' },
    { status: 403 }
  );
}

5. Verification Credentials & Reproducibility Matrix

The architecture was verified across five automated testing suites. Every test executed cleanly with zero failures.

Test Suite	File Path	Assertions	Result	Focus Area
Official Acceptance Suite	`Hack_docs/run.py`	7 / 7 checks	PASS (100%)	Full end-to-end container evaluation (T1 & T2 specs)
Adversarial Regression	`tests/test_phase3_adversarial.py`	47 / 47 checks	PASS (100%)	IDOR probing, token spoofing, role boundary testing
Tier 2 Exhaustive Audit	`tests/test_t2_exhaustive_audit.ts`	17 / 17 checks	PASS (100%)	MAD math, zero-variance fixtures, CWE-1236 injection
Cryptographic Certificates	`tests/test_t4_certificates.ts`	16 / 16 checks	PASS (100%)	Canonical JSON, HMAC-SHA256 anti-tamper verification
Webhooks & Ingest Engine	`tests/test_t4_webhooks_and_import.ts`	11 / 11 checks	PASS (100%)	Async webhook dispatch, Zod bulk transaction rollback

Reproduction Commands

Any judge can reproduce our audit in under 60 seconds directly from PowerShell:

# 1. Run Tier 2 Exhaustive Audit Suite (17/17 assertions):
npx tsx tests/test_t2_exhaustive_audit.ts

# 2. Run Cryptographic Certificate Suite (16/16 assertions):
npx tsx tests/test_t4_certificates.ts

# 3. Run Webhooks & Bulk Import Suite (11/11 assertions):
npx tsx tests/test_t4_webhooks_and_import.ts

# 4. Official Acceptance Suite:
python Hack_docs/run.py .dogfood.toml

6. Lessons Learned & System Architecture Principles

Never Trust ORM undefined Semantics: Modern ORMs like Prisma treat undefined as "omit this constraint," not "assert null." Always provide explicit values ({ in: array }) or build defensive schema guards.
Never Compare Floating-Point Variances with Zero: In weighted numerical systems, exact decimal values do not exist in binary IEEE-754 floats. A delta of 2.22 × 10⁻¹⁶ divided into a difference will inflate numbers to 10¹⁵. Always clamp with an epsilon (EPSILON = 1e-9).
Simplicity Over Distributed Complexity: Real-time WebSockets and Redis pub/sub add operational fragility and cognitive bias. Replacing them with atomic transactions, append-only audit logs, and non-blocking asynchronous webhooks gave us an air-gapped engine with zero failure states.
Treat CSV as an Attack Surface: Any user-supplied text field exported to a CSV is an arbitrary code execution vector (CWE-1236). Neutralize formula characters at the serialization boundary.

Full open-source codebase, OpenAPI specifications, and test suites are available on GitHub:
👉 https://github.com/Vineetw07/OmniJudge

DEV Community: Vineet Wagh