DEV Community: Iurii Rogulia

AI Coding for Senior Developers: What to Delegate (and Not)

Iurii Rogulia — Thu, 02 Jul 2026 10:00:44 +0000

It's Tuesday morning. I have a task in the backlog: add a new billing event type to an existing webhook pipeline. The schema exists, the handler pattern exists, three similar event types are already wired up. I open the agent, point it at the relevant files, and describe what I need. Twelve minutes later the implementation is done, the tests pass, and I'm reading through the diff before committing.

That's most of it. No drama, no revelation. This is what a productive morning looks like now.

But those twelve minutes involve a specific kind of attention that I didn't have before working with AI tools regularly — a trained sense for where to slow down and where to let the agent run. That sense is what I want to write about here. Not the philosophy of it (I covered why AI amplifies rather than equalizes in the previous article). The operational shape of an actual workday.

Where I Delegate Without Hesitation

"Trust completely" is a bad frame because I still read what the model produces. A better frame is: there are tasks where the right answer is well-defined enough that I can evaluate the output quickly without deep scrutiny. I delegate these freely.

Boilerplate that follows an established pattern. If I need a new CRUD endpoint and ten similar endpoints already exist in the codebase, I point the agent at two of them and ask for the new one in the same style. The output is usually correct on the first pass. I read it, I verify it follows the same error-handling conventions, and I move on. The value here isn't that the model is smarter than me — it's that it can hold the existing pattern in context and reproduce it accurately without me having to do so consciously.

Tests for behavior I've already specified in prose. If I've written a description of what a function should do — what inputs it accepts, what edge cases it handles, what it should return or throw — the model writes tests that match that description well. This is different from asking the model to write tests for code it just wrote, which produces a different and much lower-quality result. I described that pattern in detail in the first article. The distinction matters: tests from my specification test behavior; tests from the model's own code test the implementation.

Type definitions and interfaces. Describing a data structure in plain language and asking for the TypeScript interface is something the model does better than I would if I were typing from scratch. The output is complete, well-named, and includes JSDoc when I ask. I review it, adjust names where I have preferences, and move on.

Regular expressions. These used to take me twenty minutes even for moderately complex cases — writing the pattern, testing against edge cases in a REPL, adjusting. Now: I describe the matching requirements including the edge cases I care about, the model produces the regex, and I verify it against the inputs I actually have. Elapsed time went from roughly twenty minutes to roughly two. The model is also better than I am at producing readable regexes with named groups, because it has pattern-matched on a huge number of examples.

Format conversions and data transformations. Parsing an API response into a different shape, writing a one-off CSV-to-JSON converter, building a jq pipeline for a log file. These are well-defined inputs-to-outputs with no ambiguity about what "correct" means. I describe the input format, describe the output format, and the model produces the code.

Bash one-liners and scripting. I know enough Bash to get things done. I don't enjoy writing Bash. The model does fine here and I review the output before running anything that touches production.

Documentation for code I've already written. I write comments where decisions need explanation; the model fills in JSDoc blocks and inline clarifications for things that are genuinely readable but underdocumented. The result is acceptable as a first draft; I adjust the parts that don't sound like how I would write.

Where I Use AI But Read Every Line

These are the high-stakes areas — tasks where I still use AI assistance because the drafting speed is real, but where I treat the output as a pull request from a developer I don't fully trust yet. Every line gets read. Important parts get tested in isolation before the code goes anywhere near production.

SQL queries, especially with JOINs and multi-tenancy. I build on an existing Drizzle ORM schema and the model can produce correct queries from my description. For a SaaS product with proper multi-tenancy, this is the area where I see the most subtle AI-generated mistakes. But "correct" for a query means more than "returns results." It means filtered by the right tenant, using an index that actually exists, returning only the columns the caller needs, and behaving predictably under the data distribution I have in production. For any query that touches more than one table or runs in a hot path, I read the output carefully and run EXPLAIN ANALYZE on the actual production database before deploying. The model cannot know what my indexes are or what my data looks like. I do.

Anything that touches money. Stripe charge creation, invoice line item calculation, VAT application, refund logic. The model drafts; I review as if I were a senior engineer who doesn't trust the author. Every conditional branch. Every rounding behavior. Every failure path. If a payment goes wrong, the damage is real and often irreversible. Speed on this code is not a priority.

User input handling. Any code that accepts data from outside the system: form submissions, webhook payloads, API parameters. I check that validation covers the edge cases I can think of, that error messages don't leak internal structure, and that no input can reach a database or a file system without going through the validation layer first.

Auth checks inside functions. Not the architectural question of where auth should happen — that's a decision I make and document before the model writes any code. But when a function has a guard clause that checks permissions or ownership, I read it carefully. Subtle errors here — checking the wrong field, failing open rather than closed, not accounting for a null case — are the kind of thing that's invisible in tests and visible only when someone exploits it.

Migrations. The model generates them quickly and the syntax is usually correct. I read every migration twice: once for what it does, once for what it might do to existing data. I run migrations on a copy of production data before running them on production. The model has no knowledge of the actual rows in the database.

Where I Don't Delegate

slug="fractional-cto"
text="If AI tools are producing code faster than your team can evaluate it safely, I help senior engineering leaders build the review discipline and guardrails that keep it in check."
/>

This list is short because the previous article covered the principles. Here I want to state these as practices, not arguments.

Architectural decisions. Where modules live, how the application is layered, what the module boundary is between two services. I make these decisions and communicate them to the model explicitly before it writes anything. If I don't, the model will make them for me, locally and inconsistently. It will also make them differently in different sessions. I use a CLAUDE.md file in every project to capture these decisions — more on that in the next article in this series.

Security topology. Where authentication happens. Which routes are public. What an unauthenticated request is allowed to touch. The model can implement the mechanism; I decide where it lives. This is not a task that can be delegated safely because its correctness depends on understanding the full request path, and the model doesn't have that understanding unless I've given it explicitly — and even then I verify.

Schema design. The model drafts DDL quickly. The decision about which columns are nullable, which indexes serve the queries that will actually run, whether to denormalize this field or keep it normalized — those require knowing the query patterns, the data growth projections, and the operational constraints. I make these decisions and then ask the model to generate the DDL.

Trade-off decisions. The model is good at presenting options. "Here are three ways to implement this caching layer, with trade-offs." I use that output as a starting point. But the actual choice requires context the model doesn't have: what the operational team can support, what the cost ceiling is, what technical debt already exists in this area, what the team's experience is with each option. The model doesn't know. I tell it what I've decided, not the other way around.

Production incident debugging. When something is actively broken, I want a direct path from observation to diagnosis. The model introduces latency into that path because it generates hypotheses that sound plausible but are based on general patterns, not the specific state of the system I'm looking at. In a production incident I want logs, metrics, and my own knowledge of the system — not a confident-sounding guess about what might be wrong. I turn the agent off and debug directly.

Signals That Make Me Stop

This is the part that took time to develop and that I don't see written about often. Mid-session, in real time, there are specific outputs that cause me to stop and reconsider before accepting anything else from that session.

The model creates a new file instead of modifying an existing one. If I asked it to extend an existing module and it produced a new file alongside it, it didn't see the existing code or decided to work around it. Either way, the output probably duplicates something. I reject it and redirect with an explicit pointer to the existing file.

A new dependency appears. I have not approved. I did not ask for a library. The model added one because it was convenient for the task. I stop here every time. Adding a dependency is a decision that affects the entire project — security surface, bundle size, maintenance burden. It is never implicit.

The model reinvented a pattern that already exists in the codebase. This happens when I didn't give it enough context about what's already there. The result is a second version of something that should be unified, and now I have inconsistency I didn't start with. I reject the output and provide the existing pattern explicitly.

Confident use of an API method I don't recognize. Sometimes it's a hallucinated method name. Sometimes it's a real method that was added in a newer version than I'm running. Sometimes it's correct. I check every time without exception, because the failure mode for a hallucinated API call is a runtime error in a code path that looks fine statically.

// Model produced this. I didn't recognize `.parseAsync` with this signature.
const result = await schema.parseAsync(input, { strict: true });
// Checked: the `strict` option doesn't exist on Zod's parseAsync.
// The code would have worked silently, ignoring the unknown option.

A try/catch that swallows errors. The model adds these because it has seen a lot of code that wraps things in try/catch. When the catch block logs and returns null — or worse, returns a fallback value and continues — it hides failures rather than surfacing them. I always check what catch blocks do. An error that disappears into a log is an incident waiting to happen.

A test that checks the return type rather than the return value.

// This is not a test. This is noise.
it("returns a number", () => {
  expect(typeof calculateTax(100, "FI")).toBe("number");
});

// This is a test.
it("applies Finnish VAT at 25.5% to the base amount", () => {
  expect(calculateTax(100, "FI")).toBe(125.5);
});

I wrote about this in the first article. When I see the first pattern in model output, I reject the test and write a description of the actual behavior I want to verify.

Magic numbers without explanation. setTimeout(fn, 5000) — why 5000? maxRetries = 3 — where did 3 come from? The model uses reasonable-looking defaults without documenting the reasoning. I either add the comment myself or ask the model to explain and then document it, because whoever reads this code next (usually me, six months later) will have no idea why that number is there.

What I Do Differently Now

Working with AI tools daily has introduced habits that didn't exist in my workflow before.

I write specifications in prose before asking for code. The quality of what the model produces is a direct function of the quality of the specification it receives. I described the mechanism in detail in the previous article. Practically, this means I spend five to ten minutes writing a description of what a function needs to do — its inputs, outputs, constraints, and failure modes — before I prompt for implementation. This has had an unexpected benefit beyond AI: it surfaces ambiguities before I'm in the middle of writing code.

I use AI to review my own code. After writing something non-trivial, I paste it into a session and ask the agent to find problems with it. This catches things I've become blind to after staring at the code for an hour: unchecked error paths, a null case I didn't handle, an assumption that's not documented. It is not a substitute for a human review. It is a useful additional pass that costs two minutes.

I refactor more often. The cost of refactoring went down because the mechanical parts — renaming, restructuring, updating call sites — are faster with AI assistance. So I do it sooner, at lower threshold. I used to tolerate more awkwardness in a module because cleaning it up wasn't worth the time. Now the threshold for "this is worth fixing" is lower.

I use AI to explore alternatives before committing. "Here's my current approach. What are two or three other ways to solve this, with trade-offs?" The model is good at this and it's made me less likely to commit to the first approach that compiles. I don't take its recommendation — I use the options as a starting point for my own assessment.

I maintain a CLAUDE.md in every project. A document that captures the architectural decisions, the patterns, the conventions, and the explicit constraints the model should follow. The details of what goes in it and how to write it are what the next article in this series covers.

I watch for cognitive offload in myself. This is the subtler discipline and the one I think most people working with AI underestimate. The risk isn't that I delegate execution to the model — that's the whole point. The risk is that I start delegating thinking without noticing: accepting a suggestion because the explanation sounded confident rather than because I evaluated the substance, skipping a verification step because I'm tired and the output looks right, treating "the model generated this" as evidence that it was considered. Leverage and dependency look identical in the moment; the difference is whether I'd still be able to make the decision without the model's draft in front of me. When I notice the dependency pattern in myself — and it happens — I stop the session, take a break, and come back when I can do the evaluation properly. This is the new failure mode AI introduces for experienced developers, and it doesn't trigger any of the signals listed above. It comes from inside.

What Hasn't Changed

This is the honest part of the balance sheet.

I am only marginally faster at understanding a new codebase. This is more nuanced than I used to think. The model genuinely reduces the cost of orientation: it maps structure, surfaces entry points, identifies the libraries in use, and narrates what individual files do. That part is real and useful, and it's saved me hours on unfamiliar repositories — the search cost of "where does this even start" has collapsed. But orientation and comprehension are different things. Understanding the codebase well enough to change it safely — knowing why something was built this way, what invariants are being protected, what assumptions the original author was making — still requires reading the code and thinking about it. The shortcut is to the map, not to the territory.

I am not faster at estimating the complexity of tasks. The model is an optimist. When I ask it how long something will take, or when I ask it to assess the risk of a change, it systematically underestimates. My own estimates, based on experience with what actually goes wrong, are more accurate. I've stopped asking.

Production incidents don't go faster. As noted above: I debug those the old way.

Conversations with clients haven't changed. Understanding what a business actually needs, translating that into a technical approach, deciding what not to build — none of that is faster.

Code reading as a deep skill is still slow and manual. The model can help narrate code, but narration and understanding are different things.

The Honest Assessment

AI is now part of my toolchain the same way a compiler, a debugger, and version control are. Each of those tools changed how professional developers work when they were introduced. None of them changed what the work actually requires — they changed the mechanics of doing it. AI is doing the same thing, at larger scale, in more parts of the workflow.

The tool is productive where I have clear constraints and well-specified intent. It is a source of plausible-looking problems where I don't. The discipline is knowing the difference, in real time, for each task I'm about to hand off. That's a skill that develops with use and with being wrong often enough to notice the pattern.

After 25 years of building systems, the part of my job that AI has not changed is the part that was never about typing: deciding what to build, deciding what not to build, and knowing which decisions can be taken back if they turn out to be wrong.

If working with AI tools makes your development process feel chaotic rather than faster, that's usually a process problem, not a tool problem. My fractional CTO work often starts with sorting out exactly this — establishing the structure that makes AI-assisted development reliable rather than risky. If you've inherited a codebase that was built with AI without that structure in place, that's what my rescue projects service is for.

For a deeper look at what a well-maintained AI workflow looks like in practice — the CLAUDE.md file, system prompts, and mid-session stop signals — read Prompts That Keep an AI Agent From Wrecking Your Codebase.

PDF Integrity Report: May 2026

Iurii Rogulia — Thu, 02 Jul 2026 10:00:42 +0000

Originally published at htpbe.tech. The version on htpbe.tech stays in sync with the latest detection algorithm — refer to it for the canonical text.

Every month we look at aggregate, anonymized data from checks processed by HTPBE and write up what the structural signals tell us about the state of PDF tampering. No file contents, no personally identifiable information — only the structural and metadata patterns the algorithm uses to classify documents.

This report is about proportions and movement, not raw counts. What share of documents came back flagged, which signals fired more or less often than the month before, which origins shifted, and what the recurring tampering shapes looked like. Those are the numbers that mean something; an absolute file count for a single month is noise by comparison.

The Shape of the Verdicts

The flagged share climbed again — from just under half in March to roughly seven in ten in May. The bigger story is within the flagged set. For the first time, "certain" verdicts — where multiple unambiguous structural signals converge — overtook "high-confidence" ones as the single largest bucket.

Verdict	Direction vs. prior months
Certain modification	▲ now the largest single bucket
High-confidence modification	▲ slightly, but overtaken by "certain"
Not flagged	▼ shrinking share

Two forces are behind that. First, the traffic mix tilted hard toward the API this month, and API callers skew toward documents that are already modified — developers testing an integration with known-bad files, and forger-bridge traffic uploading a fake to see whether it gets caught. That population produces stacked, unambiguous evidence, which lands in the "certain" tier. Second, detection coverage widened (see the version count below), so files that once scraped a "high" now cross into "certain" because a newer check adds the converging second signal.

Read the flagged share as a statement about who is submitting documents, not as a population-wide fraud rate.

Signals That Moved

Among flagged documents, the evidence mix shifted in a consistent direction: the classical first-order signals held their lead, while the newer second-order signals kept gaining share.

Up month over month:

Generator-fingerprint contradictions — the declared producer says one thing, the binary structure says another. Growing fastest, because it catches files where the forger spoofed the producer string but left the structural fingerprint of the tool they actually used.
Multi-source page-template assembly — pages stamped from one source spliced onto pages from another. A new dedicated check this month moved this from "occasionally caught" to "routinely caught."
Producer-identity spoofing on re-distilled files — also a May-shipped check; immediately started firing on files laundered through a re-distill step.
Missing creation date — roughly a quarter of all files now arrive with no creation timestamp at all, a share that has crept up every month. A missing creation date strips out one of the cleaner forensic anchors, and its rise is itself a signal: someone is scrubbing it.

Flat or down:

Date-field inconsistencies — still the most common single finding by share, but no longer growing; the easy timestamp tells are increasingly being cleaned by forgers before submission.
Post-signature modification — down in share, mostly because signed documents were a thin slice of the month.

The trend we have flagged all year held: a forger who learned to scrub creation dates and avoid an incremental-update trail does not necessarily know to reconcile the structural fingerprint of the tool they rebuilt the file with against the producer string they spoofed. The second-order signals are where those cases get caught.

Incremental Updates: Almost Without Exception

The cleanest signal we track, and it got cleaner. Files carrying incremental updates were flagged in virtually every case this month — the highest rate in the four months we have published, continuing a monotonic climb. The average revision chain on those files sat around three appends.

The mechanism is unchanged: incremental updates let content be appended after the original write. Legitimate workflows produce them — signature application, annotation, form-fill — but on the population reaching the tool, those clean cases have shrunk to a rounding error. When an incremental update shows up on a document submitted for fraud detection, it is now almost synonymous with post-creation editing.

Representative Cases

These are composite, anonymized illustrations of the recurring shapes the engine resolved this month — not specific files. Each maps to the structural markers that actually drove the verdict.

The spreadsheet rebuild (verdict: certain). A "bank statement" arrives looking clean to the eye. Structurally, the producer field names a spreadsheet-export pipeline rather than the bank's core system, and the modification timestamp trails the creation timestamp by weeks. Two converging signals — producer mismatch plus a date gap — and the file is flagged with high certainty. This is the single most common shape in the flagged set, month after month.

The signature that held while the bytes moved (verdict: certain). A signed contract shows a green "signed by" badge in the viewer. Structurally, an incremental update was appended after the signature's byte range — a figure changed on page three, saved as a new revision the signature never covered. The signature stays technically valid; the document is not what was signed. This is the case digital-signature validation alone cannot catch, and structural analysis does.

The two-source invoice (verdict: certain). An invoice looks like one coherent document. Structurally, page one carries the font subsets and object fingerprint of an institutional generator, while page two was spliced in from a different source — a different font-subset prefix, a stamp-coverage discontinuity at the page boundary. Multi-source page-template assembly: the body is genuine, one page was swapped.

The borrowed identity (verdict: high → certain). A file declares "Adobe Acrobat" in its producer string but is missing the XMP toolkit marker and document-instance identifiers a genuine Acrobat save always writes, and its structural fingerprint matches a re-distill pipeline. Producer-identity spoofing — the May-shipped check that turns "claims to be Adobe" into a flag when the structure says otherwise.

Document Origin

The origin mix shifted: scanned documents rose sharply, to nearly a quarter of submissions — overtaking consumer-software exports — while institutional documents remained the plurality.

Origin classification	Direction
Institutional (server-side / enterprise generators)	plurality, steady
Scanned ("Cannot Verify")	▲ sharp rise, now ~a quarter
Consumer software ("Cannot Verify")	▼ slipped below scanned
Online editor / unknown / other	small shares

Scans and consumer-software exports fall into a "Cannot Verify" bucket where the structural layer deliberately returns a conservative inconclusive verdict rather than an intact-or-modified call — forcing a binary verdict on those formats would generate false positives in both directions. The rise in scanned share is worth watching: re-scanning a tampered printout is a known way to launder edits out of the structural record, which is exactly why a scan can never earn an "intact" verdict here.

Digital Signatures

Signed documents remained a thin slice of the month, too small to quote a meaningful rate. The pattern that did appear is the one we keep reporting: a signature that is valid in the viewer does not guarantee the bytes were not altered, because incremental updates appended after signing fall outside the signed scope. Checking integrity at the structural layer, not the signature-validation layer, is what catches that — see "the signature that held while the bytes moved" above.

Algorithm Development

May was, by version count, the busiest month since launch — twenty-nine versions shipped, up from April's eighteen. The work split three ways, as always:

New detection categories — multi-source page-template assembly, producer-identity spoofing on re-distilled files, and refinements to drawing-operator and content-stream consistency checks. The first two show up directly in the "signals that moved" section above.
False-positive reductions — roughly half the releases narrowed heuristics misfiring on legitimate document classes: professional export pipelines, multi-tool re-export chains, certain office-suite and print-to-PDF outputs, signed-document workflows.
Clearer inconclusive verdicts — scans, consumer-software exports and HTML-to-PDF output continued to be routed into an explicit inconclusive verdict rather than forced into intact-or-modified.

Wider coverage is itself part of why the flagged share rose: a share of the documents now flagged would have come back intact under the early-May algorithm.

The Software Ecosystem

The recurring fingerprints held. Online manipulation services as intermediate steps — a service in the producer field with a different application in the creator field, the signature of a compress / merge / page-extract step between creation and submission. Design-tool origin — vector- and consumer-design applications appearing where a system-generated producer belongs, on documents that purport to be business records. Programmatic manipulation libraries — where the signal is no longer the spoofable producer string but the structural fingerprint the library leaves at the binary level. May's producer-identity-spoofing check was built for exactly that last category.

PDF Version Landscape

Concentration tightened: PDF 1.7 alone accounted for over half the sample, with 1.6, 1.4, 1.5 and 1.3 splitting most of the rest. PDF 2.0, despite nearly a decade of availability, stayed a rounding-error share.

Summary

May 2026, in relative terms:

The flagged share climbed past seven in ten — but read it as a traffic-mix effect (API-dominated, skewed toward already-modified files), not a population fraud rate.
"Certain" verdicts overtook "high-confidence" ones for the first time — converging, unambiguous evidence is becoming the norm in the flagged set.
Incremental-update files were flagged almost without exception — the cleanest single signal we track, climbing every month.
Second-order signals — generator-fingerprint contradictions, multi-source assembly, producer-identity spoofing — kept gaining share on the classical date and incremental-update tells.
Scanned documents rose sharply, to nearly a quarter of submissions; missing creation dates continued their slow climb.
Twenty-nine algorithm versions shipped, the most in any month so far.

Every pattern here comes from the same forensic engine teams run on their own intake stream through the PDF tamper detection API. If you want to run a single document through the same analysis by hand, the free checker does it in the browser.

This report covers checks processed by HTPBE in May 2026. File contents are not stored or analyzed; only structural metadata signals are retained. All figures are aggregate and anonymized.

Why AI Coding Widens the Senior–Junior Developer Gap

Iurii Rogulia — Wed, 01 Jul 2026 10:00:46 +0000

Two things happened in the same week.

A founder sent me a repository. Thirty thousand lines, built in three months with AI assistance. It compiled. The tests were green. In the demo it looked polished. In production it had five separate authentication flows, a test suite that verified return types rather than business logic, and a database schema that disagreed with the ORM in three different ways. I wrote about the patterns in detail in the previous article in this series. The short version: the code was locally coherent and globally incoherent, because no one had been in the role of architect.

Two days later, a senior developer I respect sent me a message. He'd integrated AI coding tools into his workflow six months earlier. He said his output had roughly doubled on everything that wasn't architecture-level work. He was shipping faster, making fewer typos in tedious boilerplate, and spending more time on the parts of the job he found interesting.

Same technology. Opposite results. That gap is not random.

Wind Does Not Have a Direction

There's an old observation about fire and wind: wind doesn't create fire, and it doesn't choose sides. It intensifies whatever is already burning. A small flame, it extinguishes. A large fire, it feeds until it becomes something much larger.

AI-assisted coding works the same way.

The tools don't have preferences. They don't know whether your database migration is safe to run or whether your authentication boundary is in the right place. They produce plausible-looking output based on patterns in their training data, and they do this with consistent confidence regardless of whether the output is correct. The tool is powerful. The question is: powerful in what direction?

That depends entirely on what was already there before you opened the chat.

This is not a comfortable observation because it contradicts the popular narrative. The narrative says AI democratizes software development — that it gives less experienced developers access to patterns and capabilities that previously required years of practice. This is true at the surface level and misleading at the level that matters. There is a real difference between generating a plausible-looking piece of code and knowing whether that code should exist at all, whether it fits the system around it, and whether it will hold under conditions the demo never tests.

AI closes the gap on the first kind of knowledge. It widens the gap on the second.

What Happens Without an Architect in the Loop

I don't think the people who produce the codebases I described in the previous article are careless or uninformed. They asked the model to help them build something, and the model helped. The problem is structural: the model has no persistent understanding of the system as a whole. Each response optimizes locally. It solves the current problem without regard for whether that solution creates a new problem two files away, or whether a nearly identical solution already exists from a different prompt three weeks earlier.

A senior developer doesn't distinguish themselves by holding more in their head than a junior. Distributed systems have long since exceeded the cognitive capacity of any single person — the idea that good engineering is about maximizing mental retention is a romanticization that doesn't survive contact with real systems. What a senior does instead is build processes that make the dependency on memory unnecessary: Architecture Decision Records, explicit invariants in code, contract tests, CI gates that encode architectural constraints, schema-as-code, runbooks for failure modes. The system is made legible not by one person's heroic memory but by accumulated structure that anyone can read.

This is worth saying clearly because it changes how you think about AI in the picture. AI accelerates coding velocity. The senior's job is to make sure the system is protected from its own AI-assisted speed — through the same formal processes that make it resilient to any other kind of acceleration.

This is not a skill that comes from knowing a framework or a programming language. It comes from having seen a lot of systems, having watched them fail, and having developed a sense — call it taste, or judgment — for when something is wrong even before you can articulate precisely why. This judgment is not a substitute for process. It is what tells you which processes need to exist, where the invariants are, and which parts of the system deserve explicit protection.

That judgment is what a junior developer doesn't have yet. Not because they're incapable of it but because it requires time and failure to develop. There's no shortcut. And when you give a developer without that judgment an AI assistant that produces confident-looking code at high volume, what you get is more code faster, along with all the problems that come from code written without the judgment to evaluate it.

The junior's fundamental problem with AI is not that they accept bad suggestions. It's that they often can't distinguish the bad suggestions from the good ones. When a model proposes five different ways to approach authentication and the junior picks one, the criterion for picking is not "which of these is architecturally sound" — it's "which of these seems most familiar, or most recently discussed, or most confidently explained." The model's tone doesn't change between good suggestions and bad ones. It sounds equally sure of itself when it's right and when it's leading you into a pattern that will be painful to undo.

What Happens When the Senior Is in the Loop

slug="fractional-cto"
text="Need a senior engineer in the loop without a full-time hire? Fractional CTO work is exactly this — technical leadership that keeps AI-assisted velocity from producing architectural chaos."
/>

The senior developer I mentioned at the start roughly doubled his output on routine work. I believe that number because it matches what I experience myself. But the mechanism is worth examining carefully, because it's not what most people assume.

It's not that AI writes the code and the senior reviews it. Or rather — that's the surface description, but it leaves out the most important part. The senior rejects far more than they accept. For every suggestion that gets merged, there are several that get discarded, requested to be rewritten, or corrected before they land. That filtering work is invisible. People talk about how AI accelerated their development; they rarely talk about how often they told it no, or how many iterations it took to get something they trusted.

The senior also knows exactly which parts of the system they will never delegate. I have a short list of things I will not let a model make decisions about without heavy supervision:

Database schema changes. Not because models can't generate migrations — they can, quickly and with correct syntax — but because a migration that looks right can have consequences that only show up after it runs, and the model cannot predict those consequences without knowledge of the actual production data distribution. I generate migrations with AI assistance and read every line twice before running them anywhere.

Security boundaries. Where does authentication happen? Who is allowed to see what? What does an unauthenticated request touch? These are architectural decisions that need to be made deliberately, by a person who understands the full system, and written down so they can be audited. A model will write an auth check if you ask for one. Whether that check is in the right place, at the right layer, for the right reason — that requires judgment the model doesn't have.

Anything that touches money. Payment flow, billing logic, invoice calculation. The model can write a Stripe integration faster than I can. I'll still read every line, test every edge case, and be the one who decides how failures are handled.

Operational and runtime correctness — and this one is less obvious than the others. AI generates code that works in ideal conditions. What it does not reliably handle is behavior under pressure: race conditions, idempotency and retry storms in webhooks and queues, deadlocks, failure semantics when an external API dies mid-pipeline. The model produces code that passes tests in a clean environment; operational correctness is about what happens in the unclean one. Plausible-looking code that fails under load takes root in production before it becomes visible — which is what makes it the most expensive kind of wrong.

Outside these areas, I use AI extensively. Migrations for things that aren't schema changes. CRUD endpoints that follow established patterns. Type definitions and interfaces. Test cases for behavior I've already specified in prose — and that qualifier matters: AI writing tests for code it just wrote is a different thing entirely, and it produces a different kind of failure; I covered that in detail in the previous article. Regular expressions. Boilerplate for new services that should look like existing services. These are the places where AI genuinely multiplies output, because they're places where the right answer is relatively well-defined and the main cost is typing.

What this means in practice: I'm much faster on the parts of a project that are fundamentally mechanical, and about the same speed on the parts that require judgment. The ratio of judgment work to mechanical work shifts — there's more of the former as a proportion of my time. I find this more interesting, not less.

A Concrete Illustration

Consider something simple: prompting for a database query.

A junior developer, new to a codebase, might write something like:

Write me a query to get all orders for a user

The model returns a query. Probably a correct one. The junior adds it to the codebase.

A senior developer, working in the same codebase, might write something like:

I need to fetch all open orders for a given user, using the existing
Drizzle ORM schema in schema/orders.ts. Orders are soft-deleted (deleted_at column),
must be filtered by tenant_id for multi-tenancy, and we need the customer's
name from a JOIN on the customers table. Return only order_id, created_at,
status, and customer_name. The function should be in lib/queries/orders.ts
next to the existing getOrderById function, following the same pattern.
Include proper typing with InferSelectModel.

The model returns a query that fits the actual codebase. The senior reads it, adjusts two things, and merges it.

Both developers used AI. The difference is not in the tool. It's in the question asked. The senior's question encodes architectural context the junior didn't know to include: the soft-delete pattern, the multi-tenancy constraint, the existing code location, the existing naming conventions. That context comes from understanding the system. AI can't supply it. You have to bring it.

The bottleneck in software development before AI was implementation speed — who could write code fastest. The bottleneck in the AI era is specification precision. The real advantage a senior brings is not the ability to write code quickly by hand, but the ability to formalize intent: to state constraints, describe invariants, design interfaces, and define failure semantics before a line of code is written. AI writes code from a specification; the quality of that code is now a function of the quality of the specification it received. That's a different skill from typing faster, and it takes the same years to develop.

This is what I mean when I say AI amplifies what you already have. The senior gets a useful response on the first prompt because the prompt reflects real understanding. The junior gets a response that will need to be rewritten — or worse, won't be recognized as wrong until it causes a production incident.

The Illusion of Equal Access

There's a version of the AI democratization story that I think is genuinely harmful, not because it's dishonest, but because it gives founders and hiring managers the wrong mental model.

The story goes: AI makes anyone capable of writing production-grade software. Hire cheaper developers and give them AI tools. Get the same output at lower cost.

What actually happens: you get output that looks like production-grade software. It passes a code review from someone who doesn't know what to look for. It deploys. Then real users arrive, and it starts to fail in the ways I described in the previous article — auth that breaks when sessions expire, tests that pass but don't catch logic errors, dependencies that quietly conflict, schema drift that surfaces only when you try to add a feature.

Companies that have tried this are now, depending on how far along they are, either discovering these problems in production or paying to have them fixed. A few of them have hired me for that.

The senior developer is expensive not because the market is irrational. It's because the value is real, and AI has not made it less real. If anything, AI has made the judgment gap more visible, because junior developers can now produce large codebases quickly, which means the consequences of poor judgment arrive sooner and at greater scale.

What AI Actually Changes for Senior Developers

To be direct about this: AI has made my working life better and my output higher. I don't think this is controversial if you're honest about the conditions.

Speed on routine work: roughly 2–3× on the kinds of tasks that involve following established patterns — CRUD handlers, type definitions, boilerplate for new services, test cases against a specification I've already written.

Speed on unfamiliar technology: significantly higher, sometimes 5× or more. If I'm working with a library I haven't used before, I can have a working integration faster than if I were reading documentation alone, because the model can show me idiomatic usage in context. I still read the documentation. I still verify that what the model produced is actually idiomatic. But the feedback loop is faster.

There is also an effect I didn't fully anticipate: AI accelerates not just feature delivery but architectural entropy — the accumulation of inconsistencies, duplicated abstractions, and divergence between layers. Commit speed scales; so does the rate of debt accumulation. I use AI and still control entropy through the same processes I always have: ADRs, regular refactoring cycles, code review that looks across files rather than at functions in isolation. The velocity is real. So is the discipline required to keep it from compounding.

What doesn't change: the time I spend on architecture decisions, on debugging production incidents, on reviewing what I've built against what I intended to build. These don't go faster. The judgment work is the same.

What gets worse if I'm not careful: consistency across a large codebase, when AI writes in different sessions without shared context about previous decisions. I use a CLAUDE.md file in every project I work on with AI tools — a document that describes the patterns, conventions, and constraints that the model should follow. Without it, the model optimizes locally and introduces drift. With it, the output is much more consistent. Even then, I audit for drift regularly. The tools can introduce subtle inconsistencies that only show up when you read across files.

There is a structural reason this problem doesn't go away with better models or larger context windows. LLMs optimize within a session; software architecture lives between sessions — across months and years of decisions, reversals, and accumulated constraints. That is a different category of problem, not a scaling problem. A junior developer who has noticed the inconsistency problem often concludes that a longer context window will fix it. It won't. Architectural consistency lives in code, documentation, and process — not in the model's session memory. The senior knows to put it there.

One honest disclaimer: everything I've described is grounded in SaaS and backend systems — which is where I work. In realtime systems, embedded software, kernels, HPC, lock-free concurrency, distributed consensus, or safety-critical code, "plausible-looking" output is not merely expensive — it can be directly dangerous. The human judgment required shifts even further toward "don't delegate anything load-bearing." I won't write about those domains because I don't practice in them, but if you do, the argument in this article applies more strongly, not less.

What This Means for Businesses Hiring Right Now

I'll be brief here because it's not primarily a technical point.

Companies that are replacing senior engineering judgment with AI tools and cheaper developers will accumulate technical debt at scale. The codebases will be larger and will arrive faster, and the problems in them will be harder to fix precisely because there's more code to sort through. The inbox I have for rescue projects and fractional CTO work is, in part, a record of this trend playing out in real businesses over the last two years.

Companies that hired one or two senior engineers and gave them AI tools have gotten real acceleration — not because of the AI alone, but because the AI is working in a context where someone with judgment is deciding what to build, how to build it, and what to discard.

This is not a prediction. It's already visible in the pattern of what work comes through my door, and in how those businesses differ when I first see them.

The Honest Summary

I am not making an argument against AI-assisted development. I use it daily. I think it's made me materially faster and, in some ways, better — it surfaces options I might not have considered, it catches the kinds of errors that come from typing too fast, and it handles the mechanical parts of coding with a patience I don't always have.

I am making an argument about the conditions under which it works. Wind does not decide what burns. The fire was already there — or it wasn't. The wind only reveals which.

After 25 years of building systems, the thing I can do that AI cannot is not merely write better code. It is deciding what must be made explicit: which constraints belong in tests, which decisions belong in ADRs, which failure modes deserve runbooks, which boundaries must not drift, and which simplification will save a month of future work rather than create one. AI makes the expression of that understanding faster. It doesn't substitute for it.

If you're working with AI and it feels like a superpower, you're probably already the fire. If it feels like you're just approving everything it suggests and hoping it works — that's worth thinking about carefully.

If what I've described sounds like the codebase you're currently maintaining — the one that was built fast with AI and is now difficult to change — that's what my rescue projects service is for. If you're a senior developer or technical leader looking for someone to work alongside on complex systems, my fractional CTO work might be a better fit.

For the concrete patterns that come out of AI-only development — five auth flows, wrong tests, security shortcuts — see Vibe-Coded Codebase Problems. For the practical tools that prevent them — CLAUDE.md, system prompts, stop signals — read Prompts That Keep an AI Agent From Wrecking Your Codebase. And for the vatnode.dev codebase specifically — built with AI assistance under senior oversight — the difference in structural coherence is visible in the code.

Bank Statement Fraud in Lending: What Gets Altered and How to Catch It

Iurii Rogulia — Wed, 01 Jul 2026 10:00:36 +0000

Originally published at htpbe.tech. The version on htpbe.tech stays in sync with the latest detection algorithm — refer to it for the canonical text.

A borrower submits a three-month bank statement PDF. The layout matches their bank’s template exactly. The running balance climbs steadily through the period, peaking just above the income threshold required for approval. There are no overdrafts. The font is correct. The logo renders cleanly. Your underwriter approves the file.

None of that means the document was unaltered.

According to fraud analytics firm SEON, bank statements are the most commonly falsified document in lending applications — cited in over 59% of fraudulent loan applications. The editing happens after the borrower downloads the legitimate PDF from their bank portal, before they upload it to your application form. The tools required cost nothing and require no technical skill.

What Borrowers Actually Change

The three most common alterations are running balance inflation, inserted deposits, and hidden overdrafts.

Running balance inflation is the simplest. The borrower opens the PDF in Adobe Acrobat, selects the balance figures, and types over them. No arithmetic is recalculated — the inserted numbers are static text objects with no relationship to the surrounding transaction data. This is invisible to visual review but immediately detectable to any system that checks whether the edit session was recorded in the file structure.

Inserted deposits involve adding a transaction line — typically a salary credit or a one-off transfer — to push the average monthly income above a threshold. Again, Adobe Acrobat or an online editor like iLovePDF or Smallpdf handles this in seconds. The inserted line sits alongside real transactions in the visible content layer, but the edit leaves a different fingerprint in the structural layer.

Hidden overdrafts are less common but more technically deliberate. A borrower who has run negative balances removes those rows or replaces negative figures with small positives. This can involve reformatting an entire section, and consumer PDF editors typically re-embed fonts or rewrite object streams in ways that differ structurally from the bank’s original rendering pipeline.

Why Visual Review Fails

A reviewer comparing a statement on screen cannot see the xref chain, the producer field shift between bank portal and applicant upload, or the modification timestamp arriving four days after the creation timestamp. Visual review catches sloppy fraud: wrong font, misaligned columns, a logo that renders at the wrong resolution. It does not catch competent fraud performed with mainstream consumer tools.

The structural evidence of an edit lives in the file’s metadata and revision history, not on the rendered page.

What Structural Forensics Actually Catches

Three structural signals cover the large majority of bank statement fraud attempts.

The xref incremental update trail. Every time a PDF is modified and saved, the changes are appended to the file and a new cross-reference table is written. A bank statement generated by a bank portal and saved without intermediate processing typically has one xref entry. The exceptions matter: some retail banks save incrementally on first export, mail-handling gateways and DMS pipelines can rewrite metadata in transit, and mobile banking apps occasionally re-save the PDF on share. So xref_count > 1 is a strong correlate of post-creation modification, not a deterministic marker — it earns a closer look, not an automatic rejection.

Producer mismatch. The creator field identifies the software that originally produced the document. The producer field identifies the software that last saved it. A bank’s web portal typically generates PDFs using server-side engines: iText, Aspose.PDF, PrinceXML, or similar. When the producer field shows Adobe Acrobat 24.2 or iLovePDF while the creator shows Temenos or FIS, the document was re-saved in a tool that is not part of any documented bank statement pipeline. HTPBE_EDITING_TOOL_FINGERPRINT in the API response is a high-confidence anomaly — the right action is to flag for review, not auto-reject, because the customer might have opened the file in Preview to compress before email.

Modification date gap. The PDF specification requires that generators set both a CreationDate and a ModDate. On a clean bank export these timestamps are usually equal or within seconds of each other. On an edited document the gap reveals when the editing occurred — a statement dated Monday with a modification timestamp of Thursday means someone opened and saved the file in between. The size of the gap, the day of the week, and the relationship to the application submission date all become usable signals.

What an API Response Looks Like

Here is a representative HTPBE response on a bank statement where the borrower used iLovePDF to remove two overdraft entries and adjust the closing balance:

{
  "id": "ck_4e2f9a1b-7c3d-4f8e-b5a2-9d1e6c0f8b4a",
  "status": "modified",
  "modification_confidence": "high",
  "modification_markers": ["HTPBE_EDITING_TOOL_FINGERPRINT", "HTPBE_MULTIPLE_REVISION_LAYERS"],
  "creator": "Temenos T24",
  "producer": "iLovePDF",
  "xref_count": 3,
  "has_digital_signature": false,
  "creation_date": 1751760000,
  "modification_date": 1752105600
}

creator: "Temenos T24" is a core banking platform used by mid-tier banks across Europe and Australia. producer: "iLovePDF" is a consumer online PDF editor. iLovePDF, PDF24, Smallpdf, and similar consumer editors are not part of any documented bank statement pipeline — their producer string in a bank statement is a high-confidence anomaly. The xref_count of 3 indicates three write sessions: the original generation and two subsequent saves. The modification timestamp trails the creation timestamp by four days.

The verdict is modified with modification_confidence: "high". That gives an underwriter the signals to decide whether to route for manual review, request a fresh download from the bank portal, or proceed.

The `inconclusive` Signal in Lending Context

Not every bank statement returns modified or intact. Some legitimate statements — particularly from smaller credit unions or non-standard banking apps — are generated in consumer software or exported via generic PDF print drivers, and return inconclusive. That is not a failure verdict: it means the document was produced in software that does not identify itself as an institutional banking system, so structural integrity cannot be checked against a known baseline.

In a lending context the meaning depends on the claimed institution. Major retail banks — Chase, Barclays, Commonwealth Bank, TD — all generate statements from server-side institutional PDF engines, so inconclusive on a statement claimed to be from one of them is itself a signal worth escalating. The same verdict on a statement from a smaller institution where consumer-style export is normal can be treated as routine.

Where It Fits in Your Underwriting Stack

Plaid and similar open banking connectors pull transaction data directly from the bank’s API. They are the gold standard for income source-of-truth checks when the applicant has an account at a supported institution. They bypass the document layer entirely — and that is also their limit. Plaid supports a fraction of global financial institutions, and BNPL and MCA lenders frequently serve applicants whose banks are not connected, who have international income, or who hold multiple accounts. Those applicants submit PDF statements because open banking cannot reach them.

Identity fraud platforms like Persona and Alloy confirm that the person submitting the application is who they claim to be — face matching, ID document checks, watchlist screening. They do not analyze the structural integrity of submitted financial documents.

The structural PDF forensics for alternative lenders layer fills the gap between document submission and open banking coverage. It does not replace Plaid when Plaid is available — it operates on the submitted PDF regardless of whether the borrower’s institution is connected to any open banking network.

For a full breakdown of how document fraud shows up in income source-of-truth check workflows, see the bank statement fraud detection guide.

Integrating the Check at Application Intake

The check runs at document upload. Before the statement reaches income parsing, before a human underwriter touches it, before a credit model sees any figures from it.

curl -X POST https://api.htpbe.tech/v1/analyze \
  -H "Authorization: Bearer $HTPBE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://your-storage.example.com/statements/applicant-7821.pdf"}'

The response is synchronous for most documents. For large statements, poll GET /api/v1/result/{id} — typical analysis time is under three seconds. Store the check_id against the application record. If the credit decision is later disputed, the forensic report is retrievable as a permanent audit trail showing exactly which structural signals triggered the hold.

The full API reference and test scenarios are available at the self-serve API.

False Positives and Calibration

The harder question for lending is not "how many forgeries does this catch" but "how often does it flag a clean document". A high false-positive rate is expensive: delayed approvals, customer friction, and a swelling manual-review queue. Calibration matters more than any individual forensic example.

Real sources of false positives on bank statements include:

A retail bank that saves incrementally on its first export, producing xref_count > 1 on an otherwise untouched file.
A mail or document gateway that adds its own metadata layer at delivery, shifting the producer field away from the original generator.
A customer who downloads the PDF, opens it in Preview or Adobe Reader to compress for email, and re-saves before uploading.
A statement generated through a third-party aggregator pipeline — Plaid's downstream PDF render, a Yodlee export — where the producer field is the aggregator, not the bank.
DMS or ECM pre-processing on the borrower's accountant's side, common for self-employed applicants who route financial documents through bookkeeping software.

None of these are fraud, but each can produce a single modification marker in isolation. The calibration approach is to combine signals rather than treat any one as a verdict. A modified outcome with one marker on a re-compressed file is a different population from modified with HTPBE_EDITING_TOOL_FINGERPRINT plus xref_count = 3 plus a four-day ModDate gap. Underwriting policy should weight the full response — status, modification_confidence, modification_markers, the producer string itself, and the claimed institution — rather than firing on marker presence alone. A practical default: manual review on inconclusive for institutions with a known institutional baseline, and on modified only when confidence is high or two or more markers are present.

What This Does Not Catch

Structural forensics detects modifications to existing documents. Two scenarios fall outside its scope.

Statements fabricated from scratch in the same software the bank uses. If a borrower somehow builds a document using the exact same PDF generation stack as their bank — same library, same parameters — the structural layer will appear consistent. This attack requires technical knowledge most borrowers do not have, and is far less common than editing an existing export. For this pattern, cross-referencing with open banking data or direct account fraud detection remains the correct control.

Statements generated in consumer software legitimately. Some smaller institutions and fintech neobanks generate statements via generic export tools, mobile print drivers, or third-party aggregators. For these, inconclusive is the expected and correct verdict. The signal is only meaningful when the claimed institution has a known institutional PDF generation profile.

Understanding these limits matters. The forensics layer is not a fraud oracle — it is a structural signal that increases confidence and reduces the human review burden on documents that carry structural evidence of alteration.

For pay stub fraud detection in the same workflow, the patterns are similar — see the fake pay stub detection guide for how income document fraud across document types can be detected with the same API call. For the broader stack view, the KYC vs. document forensics breakdown covers where this layer sits alongside identity checks.

Frequently Asked Questions

How can I tell if a bank statement PDF is fake?

Visual review catches sloppy fakes — wrong fonts, misaligned columns, implausible totals. Competent edits made with Adobe Acrobat or online editors look identical to the original on screen. The reliable signal lives in the file structure: the producer field, the cross-reference (xref) chain, and the gap between CreationDate and ModDate. A statement with producer: "iLovePDF" on a major retail bank’s document is a structural anomaly worth escalating.

What is producer mismatch in PDF forensics?

The producer field identifies the software that last saved a PDF. The creator field identifies the software that originally generated it. When a bank emits a statement through its server-side engine (Temenos, Aspose, FIS) and the file later carries producer: "Microsoft Excel" or an online editor signature, those two fields disagree in a way no documented bank distribution pipeline produces. That is producer mismatch.

Can structural forensics catch a fabricated bank statement?

Partially. A statement built from scratch in Word or Excel returns inconclusive, not modified — there is no prior structure to compare against. For a document claiming to be from a top-tier retail bank, inconclusive is itself a signal: real bank portals do not emit PDFs with a consumer-software producer. Fabrication from inside the bank’s own infrastructure (rare, requires platform access) is out of structural scope and needs an open-banking or out-of-band check.

Does Plaid replace bank statement forensics?

No — they cover different applicants. Plaid pulls transaction data directly when the borrower’s institution is connected and the borrower consents. BNPL, MCA, and lenders serving international or unsupported institutions still receive PDF statements, and that is where structural forensics applies. The two layers are complementary.

Vibe Coding Problems: What AI-Generated Code Gets Wrong

Iurii Rogulia — Tue, 30 Jun 2026 10:00:39 +0000

A few months ago, a founder sent me a repository and said: "The developer used AI to build it. It works in demos but breaks in production and we can't figure out why."

I've seen this enough times now that I have a checklist before I open the code. Not because I want to be right, but because the patterns are so consistent it would be irresponsible to pretend otherwise. The codebase compiles. The tests pass. The structure looks vaguely like something a senior developer would produce. And yet it is still broken — just in ways that don't show up until you have real users, real data, or someone reading it carefully.

This is not an argument against AI-assisted development. I use it myself. This is about what happens when an LLM drives the whole project with nobody steering — no architect in the loop, no one reviewing whether the fifth approach to authentication should have replaced the first four. The output looks like code. It is not, in any meaningful sense, a designed system.

Five Overlapping Ways to Do the Same Thing

The most reliable tell is duplication — not the obvious kind, but the kind that happens when a model adds a feature, is asked about it again later in a different context, and adds it again without checking what already exists.

Authentication is the canonical example. In a recent rescue project — a codebase under 30k lines — I counted five distinct authentication flows: one in middleware, one in individual page components, one duplicated across API routes, one wrapped in a client-side hook that did its own token check, and one experimental Passport.js setup in a folder no route imported anymore. Each was added at a different point in the conversation history. None was deliberately redundant. The model never went back and removed what it had made obsolete.

The same happens with validation. There's a Zod schema at the API boundary, a manual check function three files away, and a third set of checks inside the database utility — written at different times, with slightly different rules. Which one is authoritative? Nobody knows, including the person who asked the AI to write them.

This matters in production because when you fix a bug in one layer, the other layers still have the old behaviour. You patch the API validation and the database utility silently rejects inputs that should now be valid. Or the opposite. The system is not one thing — it's an archaeological site of previous attempts.

Tests That Assert the Wrong Thing

A passing test suite is not evidence that the code is correct. In a vibe-coded repo, it is often evidence of the opposite.

Here's the shape of the problem:

it("calculates the order total", () => {
  const total = calculateOrderTotal(items);
  expect(typeof total).toBe("number");
});

That test passes. It will always pass, because calculateOrderTotal does return a number — it just returns the wrong number. The VAT is not applied. Discounts are double-counted. The shipping is hardcoded to zero. None of this is caught because the test was written to verify that the function runs without throwing, not to verify that it produces a correct result.

When a model writes tests for code it just wrote, it tends to test the implementation rather than the behaviour. It knows what the function does and it asserts that. What it doesn't know — because nobody told it — is what the function should do in edge cases, error conditions, or inputs that don't look like the happy path.

The result is a CI pipeline that's permanently green on a system with incorrect business logic. The tests are not lying. They are asking the wrong question.

Everything in `package.json`

Open the dependencies and you will find a history of the model's training data.

There is axios for HTTP. There is also node-fetch. There is also the native fetch with a polyfill that's not needed in the Node version being used. There are three date libraries: moment (because tutorials used it for years), date-fns (because someone told the model moment was deprecated), and dayjs (because another prompt mentioned it was lighter). They are all installed. Two of them are used in different files. One is not used at all but was added to package.json during a refactor that never happened.

The same with utilities. lodash and underscore. uuid and nanoid and crypto.randomUUID() all used in different places for the same purpose. A PDF library installed alongside a different PDF library, each used in one file.

Unused dependencies are not just an aesthetic problem. They expand the attack surface for dependency vulnerabilities. They bloat the build. They make it harder to understand what the system actually uses. And they are almost impossible to clean up confidently without reading every import in the codebase — because there is no documentation saying why any of them were added.

The proportion is consistent enough to be predictable. The last three vibe-coded repos I audited had between 25% and 35% of their direct dependencies entirely unused. In one of them, removing the unused entries cut the production Docker image by roughly 40%.

Security Patterns Copied from Tutorials

The model has processed an enormous amount of tutorial code. Tutorials are written to be easy to follow, which means they cut security corners that a production system cannot afford.

I find access tokens stored in localStorage because that is how most OAuth tutorials store them — it is easy to read from JavaScript, which is exactly the problem. I find CORS configured as Access-Control-Allow-Origin: * on an API that handles authenticated requests, because that is the configuration that makes the demo work without the CORS error. I find SQL built by string concatenation:

const result = await db.query(`SELECT * FROM orders WHERE user_id = '${userId}'`);

This is an injection vulnerability. The model has seen this pattern in thousands of tutorials and Stack Overflow answers where the point was to demonstrate something else, not to demonstrate safe query construction.

The one that I find most alarming, because it is the hardest to reverse: API keys committed into the codebase, sometimes in client-side bundles. The model puts configuration where it has seen configuration put before — directly in the code, or in a .env file that was never added to .gitignore. The key is now in the git history. Rotating it is mandatory. If it is a third-party service key, you must assume it was seen.

I am not describing carelessness. I am describing what happens when a system that has learned from millions of examples of code-written-for-explanation is asked to produce code-that-must-actually-be-secure. These are different tasks. The model was not told they were different.

The Schema That Disagrees With the Code

Database drift is the slowest-moving problem and often the most expensive to fix.

The migration files exist. They do not reflect the current state of the database, because migrations were generated at different stages of development and some were edited directly rather than through a new migration. Others were run manually on the production database by someone who was in a hurry. The migration history and the actual schema are no longer the same document.

The ORM models have columns that do not exist. The database has columns that the ORM does not know about. Foreign keys are defined in the code but not enforced at the database level — or enforced in the database but not declared in the ORM, so queries that cross that boundary fail in unpredictable ways. Indexes were added at the model's suggestion but are on columns that are never actually queried, while the columns that appear in every WHERE clause have no index.

The practical consequence: you cannot do a fresh setup from the migration history alone. The schema and the code are permanently entangled with the specific database instance that was used during development. Moving to a new environment requires manual reconstruction. Rolling back a bad migration is not possible.

A README That Describes a Different Product

The documentation was written by the same model that built the app — and it imagines what the finished version would look like.

The README describes role-based permissions that exist only as placeholder comments. The API reference lists endpoints that were planned but never added. The "quick start" fails on step three because a dependency was renamed and the docs weren't. Two of the environment variables it lists have different names than the actual code reads.

This is not the usual kind of documentation drift, where someone wrote accurate docs and the code moved on. The model produces confident-sounding documentation based on the intended design, not the actual implementation. It describes the system it was asked to build, not the system it built. A new developer setting up the project is working from a description of software that does not exist.

Why This Happens

None of this is random. It follows directly from how large language models work.

The model's training data is dominated by tutorial code, Stack Overflow answers, and open-source repositories in various states of completion. Tutorial code is written to illustrate a point, not to be maintained. Stack Overflow answers solve the specific question asked, without regard for what surrounds them. Open-source repositories include everything from exemplary production code to abandoned weekend projects — and the model cannot easily distinguish between them.

More structurally: the model optimises for plausible-looking output. Each response is evaluated on whether it seems correct, not on whether it integrates cleanly with the thirty responses that preceded it. There is no persistent memory of previous architectural decisions. When you ask it to add authentication at the start of a project and ask again halfway through, it adds authentication again — and there is no mechanism that asks whether the first attempt should be removed.

There is also no one in the role of architect. An experienced developer making these decisions would delete the previous approach before adding a new one. They would read the test and ask whether it is actually testing the right thing. They would notice that three HTTP clients are installed and remove two of them. The model does none of this unless explicitly prompted, and even then it only does it locally — for the current file or function, not the system as a whole.

The result is not incompetent code. It is code that is locally coherent but globally incoherent — each piece makes sense in isolation, and the whole doesn't hold together.

There is also a quieter cause that is harder to write about because it concerns hiring decisions rather than technology. Many of the projects I see arrive in this state because the company used AI assistance to substitute for a senior engineering review that nobody on the team was qualified to do. The model was not asked to draft a feature that a senior engineer would then evaluate — it was the only entity in the loop with any opinion about software design. That arrangement produces the codebases described in this article more reliably than any specific prompt or tool choice. The fix is not better prompts. The fix is putting an experienced developer back into the decision path, even part-time.

What Cleanup Actually Looks Like

slug="rescue-projects"
text="Inherited a vibe-coded codebase? I audit, stabilise, and fix the patterns described above — without a rewrite that costs you twice."
/>

The sequence I follow when I take one of these projects on:

Audit first. Read the codebase before touching it. Map what's actually there: every auth mechanism, every validation layer, every dependency that's imported somewhere. The goal is to understand the actual system, not the intended one. This is also what I do in a technical due diligence engagement before a client commits to acquiring or building on top of an existing codebase.

Dead code removal. Once you have the map, remove what is not used. This is tedious and requires confidence — you need to be sure a dependency is genuinely unused before removing it. Tools like depcheck help, but manual review is necessary for anything that touches shared utilities.

Real tests. Not tests that assert the code runs — tests that assert it produces correct results for known inputs, including edge cases and error conditions. This usually means reading the actual business logic and asking: what is this function supposed to do, and what are the cases it can fail? Write those tests before refactoring anything.

Dependency cull. Consolidate to one HTTP client, one date library, one ID generation utility. Document why each remaining dependency exists.

Security pass. Tokens out of localStorage. CORS locked to actual origins. SQL through parameterised queries or an ORM with no raw interpolation. Secrets out of the repository and into an actual secrets manager. If anything was committed, rotate it.

This is not a rewrite. The underlying application logic is usually salvageable — the business rules, the data model, most of the UI. What needs replacing is the infrastructure that holds it together.

I emphasise this because the first instinct of most founders, when they realise what they have, is to throw it away and start again. That instinct is almost always wrong. A rewrite means rebuilding everything the AI happened to get right alongside everything it got wrong, paying for it a second time, and arriving — months later — at a system that has not yet been tested against real users. Stabilising what exists is usually two to four times cheaper than starting over, and you keep the parts that already work. The codebase is in a worse state than it looks. It is rarely in a worse state than starting from zero.

items={[
{
q: "What is a vibe-coded codebase?",
a: "A codebase built primarily by prompting an LLM with no experienced developer reviewing the output, making architectural decisions, or enforcing consistency. The code compiles and demos work, but the system is globally incoherent: duplicate auth flows, tests that assert the wrong thing, security patterns copied from tutorials, and a schema that has drifted from the migration history.",
},
{
q: "Should I rewrite a vibe-coded codebase or fix it?",
a: "Almost always fix it. The underlying business logic and data model are usually salvageable. A rewrite means rebuilding everything the AI happened to get right alongside everything it got wrong, paying for it twice, and arriving months later at a system with no real-user battle-testing. Stabilising what exists is typically two to four times cheaper than starting over.",
},
{
q: "What are the most common security problems in AI-generated code?",
a: "Three patterns I find in nearly every vibe-coded codebase: access tokens stored in localStorage (readable by any JavaScript on the page); CORS configured as Access-Control-Allow-Origin: * on authenticated APIs; and SQL built by string concatenation instead of parameterised queries. API keys committed to git history are also common — if found, rotate them immediately.",
},
{
q: "How do I audit a codebase for vibe-coded problems?",
a: "Read before touching anything. Map every authentication mechanism, every validation layer, and every dependency that is actually imported somewhere. Use depcheck to identify unused packages, then manually verify anything that touches shared utilities. Look for duplicate implementations of the same concept. The goal is to understand the actual system before assuming the README describes it correctly.",
},
{
q: "How do I prevent these problems when using AI to write code?",
a: "Put an experienced developer back into the decision path, even part-time. An LLM should draft features that a senior engineer evaluates and integrates, not drive the whole project. Practically: use a CLAUDE.md or equivalent to enforce architectural rules at the prompt level, run lint gates on every commit, and never let the model add a dependency or auth flow without a human reviewing whether the existing one should be removed first.",
},
]}
/>

If what I've described sounds like the codebase on your laptop — or the one a developer just handed you — that's what my rescue projects service is for. I'll tell you honestly what's there and what it'll take to fix it.

For the broader argument about why AI amplifies existing skill gaps rather than closing them, see AI Coding Is Wind. For the practical tools that prevent these patterns from forming in the first place — CLAUDE.md, lint gates, stop prompts — read Prompts That Keep an AI Agent From Wrecking Your Codebase.

PDF Digital Signature Chain: Detect PDFs Signed Then Modified

Iurii Rogulia — Tue, 30 Jun 2026 10:00:37 +0000

Originally published at htpbe.tech. The version on htpbe.tech stays in sync with the latest detection algorithm — refer to it for the canonical text.

The PDF digital signature chain is a cryptographic lock: it computes a hash over a specific byte range of the file and stores that hash inside the document. If any byte within that range changes, the signature becomes invalid. This is what makes digitally signed PDFs trustworthy — in theory.

In practice, the PDF specification allows content to be appended after a signature without invalidating it. This is intentional: it supports legitimate workflows such as adding a second signature, appending a certification, or adding long-term validation (LTV) data. But it also creates a bypass: an attacker can append modified content after the signature, and the signature itself remains cryptographically valid. The signed portion did not change. The document did.

This is how a PDF gets signed then modified without triggering a certificate-level alert — and it is one of two high-stakes tampering patterns that HTPBE detects at certain confidence.

What a PDF digital signature actually covers

When a signing platform — DocuSign, Adobe Sign, HelloSign, or any PDF-signing library — signs a document, it records a /ByteRange entry in the signature dictionary. This entry specifies exactly which bytes of the file the cryptographic hash covers:

/ByteRange [0 840 1256 4390]

This means bytes 0–840 and bytes 1256–5646 are signed. The gap (bytes 841–1255) is reserved for the signature value itself — it cannot be signed because it contains the signature.

After signing, the file ends at byte 5646. If content is appended, the file grows beyond that boundary. The signature covers bytes 0–5646. Everything appended after byte 5646 is unsigned.

An e-signature platform showing “signature valid” on such a document is not lying: the signed byte range is intact. But the document you are reading is larger than what the signer approved.

How to detect a PDF signed then modified: two patterns at `certain` confidence

HTPBE raises two markers at certain confidence for signature-related tampering. Certain is the highest confidence level in the system. Unlike high-confidence markers — which indicate strong but circumstantial evidence — certain markers are binary. Either the bytes changed or they did not. Either the signature object exists or evidence of its removal is in the file.

Pattern 1: Modifications after digital signature

The detection reads the signature dictionary’s /ByteRange, calculates the covered range, and compares it against the actual file size. If there are xref entries pointing to objects outside the signed byte range, the document was modified after signing.

Structural evidence that confirms this:

The file has an incremental update (a new xref section appended after the last signature event)
The new xref section contains object references that fall outside the /ByteRange of the existing signature
The signature itself remains internally valid — only the unsigned append is flagged

This pattern appears when someone takes a signed contract, appends a modified page (or replaces a referenced object through an incremental update), and re-distributes the file. The signing platform’s signature UI may still show “valid” because the platform validates the byte range it knows about, not the full document.

The API response for this pattern:

{
  "status": "modified",
  "modification_confidence": "certain",
  "modification_markers": ["Modifications after digital signature"],
  "has_digital_signature": true,
  "modifications_after_signature": true,
  "signature_removed": false,
  "xref_count": 3
}

Pattern 2: Signature removed

The second pattern is removal. A signed PDF has its signature object stripped and the file is saved as a new document. The goal is to produce a file that does not show a signature at all — removing the constraint of an immutable signed payload.

This is harder to clean up completely. The PDF specification’s incremental-update model means prior xref entries leave traces in the file structure. When a signature object is removed, the object slot persists in the xref table as a free entry. Additionally, HTPBE’s ghost-info scan inspects object streams (ObjStm) for residual metadata — Info dictionaries, annotation references, and signature field stubs that were present before removal but not cleanly erased.

The structural evidence for removal:

A /SigFlags entry in the AcroForm dictionary (or a residual reference to one) indicates the document had a signature form field
The corresponding signature object is absent or marked free in the xref
Object streams preserve the original Info dictionary with signing-tool metadata that contradicts the document’s current state

API response for signature removal:

{
  "status": "modified",
  "modification_confidence": "certain",
  "modification_markers": ["Signature removed from document"],
  "has_digital_signature": false,
  "modifications_after_signature": false,
  "signature_removed": true,
  "xref_count": 2
}

Why these are `certain`, not `high`

Most forensic markers in this system produce high confidence because they rely on inferential evidence: a timestamp inconsistency, a producer mismatch, an unexpected xref count. These signals are reliable but not deterministic. A single high marker is suspicious. Multiple high markers together constitute a strong case.

certain confidence works differently. The underlying evidence is structural and self-consistent:

For modifications after signing: the signature’s byte range is embedded in the document. The file size is a fact. Whether xref entries exist outside the signed range is an arithmetic check. There is no ambiguity.

For signature removal: the PDF xref model records object lifecycle events. A free-entry slot where a signature object previously lived, combined with residual metadata that references it, is evidence that does not require interpretation. The object existed and was removed.

Neither pattern requires comparing the document against an external original. The file carries its own forensic record.

Integrating PDF digital signature chain fraud detection into a contract workflow

The check is a single API call. Upload the PDF to accessible storage (S3, Cloudflare R2, Google Cloud Storage — presigned URLs work), then submit the URL:

curl -X POST https://api.htpbe.tech/v1/analyze \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://your-storage.example.com/contracts/executed-agreement-2024.pdf"}'

The response comes back immediately with a check ID. Retrieve the full result:

curl https://api.htpbe.tech/v1/result/{check_id} \
  -H "Authorization: Bearer YOUR_API_KEY"

For a contract workflow, the fields to branch on are modifications_after_signature and signature_removed. Both being false on a signed document (where has_digital_signature: true) means the signed byte range covers the full document and no removal evidence was found:

interface SignatureCheckResult {
  status: 'intact' | 'modified' | 'inconclusive';
  modification_confidence: 'certain' | 'high' | 'none' | null;
  modification_markers: string[];
  has_digital_signature: boolean;
  modifications_after_signature: boolean;
  signature_removed: boolean;
  xref_count: number;
}

function assessSignedContract(result: SignatureCheckResult): 'valid' | 'tampered' | 'unsigned' | 'review' {
  // Certain-confidence tampering: reject immediately
  if (result.modifications_after_signature || result.signature_removed) {
    return 'tampered';
  }

  // Signed and structurally intact
  if (result.has_digital_signature && result.status === 'intact') {
    return 'valid';
  }

  // No signature present — escalate for manual review in a workflow
  // that expects signed documents
  if (!result.has_digital_signature) {
    return 'unsigned';
  }

  // Modified (high confidence) but no signature-specific markers
  // Producer mismatch, incremental updates without signature bypass, etc.
  return 'review';
}

Testing without real contracts

Test keys accept mock URLs that return deterministic responses. For signature-chain testing:

# Returns modifications_after_signature: true
https://api.htpbe.tech/v1/test/modified-medium.pdf

# Returns signature_removed: true
https://api.htpbe.tech/v1/test/signature-removed.pdf

Use these in your CI test suite to cover both certain-confidence branches without consuming production quota.

How this complements DocuSign and Adobe Sign validation

Platforms like DocuSign and Adobe Sign validate the signing certificate chain and timestamp integrity at the time of signing. That is a different layer from what HTPBE checks.

DocuSign’s validation answers: “Was this document signed with a valid certificate at the claimed time?”

HTPBE’s analysis answers: “Has this document been modified since it was signed, or has the signature been removed?”

These are complementary checks. A document can pass DocuSign certificate validation while still containing unsigned content appended after the fact — because DocuSign validates the byte range it signed, not the file as you receive it now.

The practical scenario where both layers matter: an executed agreement is downloaded from a signing platform, modified by one party (additional clause appended, a payment term changed via an incremental update), and re-submitted as the “final executed version.” DocuSign’s certificate is intact. HTPBE surfaces the post-sign modification.

Limitations

Certificate trust validation is out of scope. HTPBE does not validate that the signing certificate comes from a trusted CA, is within its validity period, or chains to a known root. That is the e-signature platform’s job. HTPBE operates on structural evidence, not PKI.

LTV annotations are not flagged. Long-Term Validation (LTV) data — certificate revocation records, timestamp tokens — is legitimately appended to signed PDFs after the signing event. HTPBE accounts for this and does not flag LTV appends as modifications after signature.

Counter-signatures on the same byte range. A workflow where a second party adds their signature to an already-signed document creates a valid multi-signature PDF with an incremental update. HTPBE distinguishes this from content modification because the appended xref section contains a new signature object, not arbitrary content changes. This is treated as intact.

Clean removal with full file rewrite. If an attacker takes a signed PDF and rewrites the entire file from scratch — rebuilding the xref table from scratch, regenerating all object streams, and removing all structural traces of the signature — the ghost-info scan may not surface residual evidence. This requires deliberate, expert-level counter-forensic work and is not the profile of typical contract fraud.

Who should use this check

This check is most relevant for teams that receive executed contracts, certified documents, or signed legal agreements from counterparties and need to check post-receipt integrity.

Specific workflows where it applies:

Legal operations and contract management platforms — checking that executed agreements received from opposing counsel or vendors match what was sent for signing. See how HTPBE integrates into legal contract workflows.

Financial services compliance — loan agreements, insurance policy documents, and disclosure forms often carry digital signatures. A compliance audit that includes structural fraud detection of signed documents catches tampering that certificate validation alone misses.

M&A and due diligence workflows — document rooms accumulate hundreds of executed agreements. Running batch checks through the PDF signature fraud detection API surfaces any that were modified after signing before they influence deal terms.

E-signature platform integrations — if you build on top of DocuSign or Adobe Sign and store executed documents in your own system, adding a structural check at the point of storage creates an immutable forensic record for each document from the moment it enters your custody.

The check is fast enough to run synchronously at document intake: average analysis time is under three seconds for contracts up to 10 MB.

Health Check Endpoint in Node.js: Liveness vs Readiness

Iurii Rogulia — Mon, 29 Jun 2026 10:00:38 +0000

Your load balancer is routing traffic to a server whose database connection pool is exhausted. Docker restarted a container that never finished its startup migrations. Kubernetes replaced a healthy pod because the liveness probe hit an endpoint that returned 503 on a transient Redis timeout.

All of these happen because /health returned the wrong thing — or because nobody designed it carefully enough.

A health check endpoint is not a ping. It is the interface between your application and the infrastructure that decides whether your application lives or dies, receives traffic or gets replaced. Getting this interface wrong costs you incidents. Getting it right costs you an afternoon.

Liveness vs Readiness vs Startup

Kubernetes formalised a distinction that applies to any containerised workload:

Liveness probe — "Is this process still running correctly, or has it become a zombie?" If the liveness probe fails, the container is restarted. The question is about process health, not dependency health. If your app is live but Postgres is down, you do not want the container restarted — you want it to stop receiving traffic until Postgres recovers.

Readiness probe — "Is this instance ready to serve requests right now?" If the readiness probe fails, the container is removed from the load balancer rotation. Traffic stops coming to it. The container keeps running. When the probe passes again, traffic resumes. This is the correct mechanism for handling a temporarily unavailable database or Redis.

Startup probe — "Has the application finished initialising?" Some apps take 10–30 seconds on boot — running migrations, warming caches, establishing connection pools. The startup probe gives you time to do this without triggering false liveness failures. Once the startup probe passes, liveness and readiness probes take over.

In Docker Compose or Docker Swarm without Kubernetes, you get a single HEALTHCHECK directive. The semantics are simpler: healthy or unhealthy. If a container is unhealthy for a certain number of consecutive checks, Docker will restart it — there is no "remove from load balancer rotation" equivalent. This is an important difference: in Docker Compose, a failed healthcheck always means a restart, not a graceful drain. The consequence: if you check external dependencies (Postgres, Redis) in your Docker healthcheck and those dependencies go down, your container will be restarted — even though a restart cannot fix a database outage. Design accordingly: set retries high (3–5) and interval long (30s+) to tolerate transient failures without triggering unnecessary restarts. Save the aggressive dependency checking for your monitoring system.

The practical mapping:

Probe	What it checks	Failure action
Liveness	Process is alive, not deadlocked	Restart container
Readiness	Dependencies reachable, app ready for load	Remove from load balancer rotation
Startup	App initialisation complete	Delay liveness/readiness probes

What to Check

A useful health endpoint checks the things your app needs to serve requests correctly. For a typical Node.js API backed by Postgres and Redis:

Database connectivity — a lightweight query that exercises the connection pool. Not a SELECT 1 to the database server directly, but through your ORM/connection pool, so you detect pool exhaustion and misconfigured connection strings, not just network reachability.

Redis connectivity — a PING command. If Redis is down and you depend on it for caching or session state, you are degraded. If you depend on it for rate limiting that gates all requests, you may be unhealthy.

Background job queue health — include this in readiness only if serving HTTP traffic directly depends on queue capacity. For example: if your API enqueues jobs and immediately returns a response that assumes the job will be processed, a backed-up or stuck queue is a readiness concern. If jobs run in the background independently of request handling, queue health belongs in /metrics, not /health — a flooded queue should trigger an alert, not take your API offline. In vatnode, queue health lives in metrics: the API can accept Stripe webhooks and return 200 even if workers are temporarily stuck; the jobs will drain once workers recover.

Disk space — optional but useful in containerised environments where logs or temporary files can fill a volume. A simple df check can prevent a class of incidents where the container fills its storage and starts failing writes silently.

What not to check in a health endpoint:

External third-party APIs (Stripe, Mailgun, VIES). Their transient failures should not take your service offline. Handle them with circuit breakers at the call site.
Business logic assertions. Health is an infrastructure concern, not a data consistency check.
Anything that takes more than a second to complete under normal conditions.

The Response Format

A health endpoint that just returns 200 OK with no body is better than nothing, but barely. A useful response tells you what is healthy, so that when something breaks you know what to investigate.

Here is the structure I use:

// types/health.ts
export type HealthStatus = "ok" | "degraded" | "unhealthy";

export interface ComponentHealth {
  status: HealthStatus;
  latencyMs: number;
  error?: string;
}

export interface HealthResponse {
  status: HealthStatus;
  version: string;
  uptime: number; // seconds
  checks: {
    database: ComponentHealth;
    redis: ComponentHealth;
    queue?: ComponentHealth;
  };
}

The status at the top level is a rollup. If all checks pass, it is ok. If non-critical checks fail (Redis is down but the app can serve cached data from memory), it is degraded. If critical checks fail (database unreachable), it is unhealthy.

The version field is your current deployment version or git SHA. This is extremely useful when debugging — it lets you verify that the instance you are looking at is actually running the code you just deployed.

The uptime field catches restart loops. An instance with 30 seconds of uptime that is supposed to have been running for days has restarted recently.

Implementing It in Hono

slug="mvp-development"
text="Healthchecks, structured logging, and graceful degradation are part of every production system I deliver. If you need a developer who treats reliability as a first-class requirement, let's talk."
/>

// routes/health.ts
import { Hono } from "hono";
import { sql } from "drizzle-orm";
import { db } from "@/lib/db";
import { redis } from "@/lib/redis";
import { orderQueue } from "@/lib/queues/order-queue";
import type { HealthResponse, HealthStatus, ComponentHealth } from "@/types/health";

const health = new Hono();

const PROCESS_START = Date.now();

async function checkDatabase(): Promise<ComponentHealth> {
  const start = Date.now();
  try {
    await Promise.race([
      db.execute(sql`SELECT 1`),
      new Promise((_, reject) => setTimeout(() => reject(new Error("timeout")), 2000)),
    ]);
    return { status: "ok", latencyMs: Date.now() - start };
  } catch (err) {
    return {
      status: "unhealthy",
      latencyMs: Date.now() - start,
      error: err instanceof Error ? err.message : "unknown error",
    };
  }
}

async function checkRedis(): Promise<ComponentHealth> {
  const start = Date.now();
  try {
    const response = await Promise.race([
      redis.ping(),
      new Promise<never>((_, reject) => setTimeout(() => reject(new Error("timeout")), 1000)),
    ]);
    if (response !== "PONG") throw new Error(`unexpected response: ${response}`);
    return { status: "ok", latencyMs: Date.now() - start };
  } catch (err) {
    return {
      status: "degraded", // Redis down = degraded, not unhealthy (depends on your app)
      latencyMs: Date.now() - start,
      error: err instanceof Error ? err.message : "unknown error",
    };
  }
}

async function checkQueue(): Promise<ComponentHealth> {
  const start = Date.now();
  try {
    const counts = await Promise.race([
      orderQueue.getJobCounts("waiting", "active", "failed"),
      new Promise<never>((_, reject) => setTimeout(() => reject(new Error("timeout")), 1500)),
    ]);
    // Thresholds are system-specific — calibrate against your normal throughput.
    // A high-volume queue may have 1000 waiting jobs as steady state;
    // a low-volume queue may flag 10 failed jobs as a problem.
    const isHealthy = counts.failed < 50 && counts.waiting < 1000;
    return {
      status: isHealthy ? "ok" : "degraded",
      latencyMs: Date.now() - start,
    };
  } catch (err) {
    return {
      status: "degraded",
      latencyMs: Date.now() - start,
      error: err instanceof Error ? err.message : "unknown error",
    };
  }
}

// This is the readiness-style check — full dependency verification.
// Wire it to /health/ready (Kubernetes) or /health (Caddy/monitoring).
// Docker's container healthcheck should call /health/live instead.
health.get("/health/ready", async (c) => {
  const [database, redis, queue] = await Promise.allSettled([
    checkDatabase(),
    checkRedis(),
    checkQueue(),
  ]);

  const db_result =
    database.status === "fulfilled"
      ? database.value
      : { status: "unhealthy" as HealthStatus, latencyMs: 0, error: "check threw" };

  const redis_result =
    redis.status === "fulfilled"
      ? redis.value
      : { status: "degraded" as HealthStatus, latencyMs: 0, error: "check threw" };

  const queue_result =
    queue.status === "fulfilled"
      ? queue.value
      : { status: "degraded" as HealthStatus, latencyMs: 0, error: "check threw" };

  // Determine overall status
  let overallStatus: HealthStatus = "ok";
  if (db_result.status === "unhealthy") {
    overallStatus = "unhealthy";
  } else if (redis_result.status !== "ok" || queue_result.status !== "ok") {
    overallStatus = "degraded";
  }

  const body: HealthResponse = {
    status: overallStatus,
    version: process.env.APP_VERSION ?? "unknown",
    uptime: Math.floor((Date.now() - PROCESS_START) / 1000),
    checks: {
      database: db_result,
      redis: redis_result,
      queue: queue_result,
    },
  };

  // 200 for ok and degraded — load balancer should still route traffic
  // 503 for unhealthy — remove from rotation
  const httpStatus = overallStatus === "unhealthy" ? 503 : 200;

  return c.json(body, httpStatus);
});

export { health };

HTTP Status Codes: Why 200 for Degraded

This surprises people: returning 200 for a degraded service is intentional — but it depends on what your load balancer does with the response.

The reasoning: if you return 503 when Redis is temporarily unavailable and your app can still serve most requests from Postgres, you just took yourself out of the load balancer rotation. All your instances are likely degraded at the same time if Redis is down — so you just took down your entire service over a non-fatal condition.

The general rule:

200 — route traffic here (ok or degraded, app is serving requests)
503 — do not route traffic here (unhealthy, app cannot serve requests)

Your monitoring system reads the response body and alerts on degraded status. The load balancer cares about the HTTP status; your alerting cares about the body. Keep these concerns separate.

A few important caveats:

Check what your load balancer actually does. Some older or simpler load balancers (HAProxy with basic config, certain managed cloud LBs) only look at status codes, not the body. The 200-for-degraded pattern assumes your LB and monitoring are separate consumers with separate configurations. If your LB is the only health check consumer, you may need to return 503 for degraded too.

Consider a separate internal port. A cleaner architecture separates concerns by port: the external port serves user traffic, an internal-only port (e.g., 9090) serves GET /health with full details for the load balancer and monitoring system. This way the detailed health response never touches the public network, and you avoid the content-negotiation problem entirely. Not always worth the operational overhead, but worth knowing the pattern exists.

The Timeout Circuit Breaker

The single most important constraint on a health endpoint: it must never hang.

If your database connection pool is exhausted, a SELECT 1 query will sit in the queue waiting for a connection to become available. Without a timeout, your health check hangs for 30 seconds, the load balancer times out waiting for a response, and it marks your instance unhealthy — not because it is, but because the health check itself blocked.

Every check in the example above uses Promise.race with a timeout. The thresholds I use:

Database: 2 seconds (a SELECT 1 over a local connection should complete in under 5ms; if it takes 2 seconds something is wrong)
Redis: 1 second (PING should be sub-millisecond)
BullMQ queue counts: 1.5 seconds (reads from Redis, same logic)

One important caveat: Promise.race bounds the health endpoint's response time, but it does not cancel the underlying operation. If db.execute(sql\SELECT 1)loses the race, it keeps running inside the connection pool — the database driver has no way to know you stopped waiting. This means a pool under pressure can accumulate abandoned queries alongside new ones. Where possible, use driver-level timeouts in addition toPromise.race: most Postgres clients support a query_timeoutorstatement_timeout setting that cancels the query at the server level.

The total health check should complete in well under 3 seconds. Configure your Docker or Kubernetes timeout to 5–10 seconds to give it headroom without making your orchestrator wait indefinitely.

Docker and Kubernetes Configuration

In Docker Compose:

services:
  api:
    image: myapp:latest
    healthcheck:
      # Use /health/live, not the full /health — Docker healthcheck triggers restarts,
      # not graceful drains. Checking Postgres/Redis here means a DB outage restarts your container.
      test: ["CMD", "wget", "--spider", "-q", "http://localhost:3001/health/live"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 20s

Using wget --spider (instead of wget -qO-) makes the intent explicit: exit code 0 on 2xx, non-zero on anything else. The -qO- variant prints the body to stdout, which is noise in Docker logs and makes the behavior less obvious.

The start_period gives your app time to finish initialising — running migrations, establishing connection pools — before the health check starts counting failures. Without it, your container can fail health checks during startup and get restarted in a loop before it has had a chance to boot.

In Kubernetes, split liveness and readiness:

# kubernetes/deployment.yaml
spec:
  containers:
    - name: api
      image: myapp:latest
      livenessProbe:
        httpGet:
          path: /health/live
          port: 3001
        initialDelaySeconds: 10
        periodSeconds: 30
        timeoutSeconds: 5
        failureThreshold: 3
      readinessProbe:
        httpGet:
          path: /health/ready
          port: 3001
        initialDelaySeconds: 5
        periodSeconds: 10
        timeoutSeconds: 5
        failureThreshold: 2
      startupProbe:
        httpGet:
          path: /health/live
          port: 3001
        initialDelaySeconds: 5
        periodSeconds: 5
        failureThreshold: 12 # Allow 60 seconds for startup

With separate probes, implement separate routes:

// /health/live — only checks if the process itself is functional
// No external dependencies; if this fails, restart the container
health.get("/health/live", (c) => {
  return c.json({ status: "ok", uptime: Math.floor((Date.now() - PROCESS_START) / 1000) });
});

// /health/ready — checks if the instance can serve traffic
// Uses the full check above
health.get("/health/ready", async (c) => {
  // ... full dependency checks, returns 503 if unhealthy
});

If you are running Docker Compose without Kubernetes, use a minimal /health/live endpoint for Docker's container healthcheck. Use the full /health or /health/ready endpoint for external monitoring, Caddy upstream checks, or manual diagnostics — not for Docker restarts. The Docker daemon will restart containers that fail their healthcheck after retries consecutive failures, so what Docker calls should be process-level only.

Caching the Result and Managing Load

/health can become your most frequently called endpoint. A load balancer checking every 10 seconds, a monitoring system checking every minute, an autoscaler reading it constantly — all hitting the same Postgres and Redis that serve your actual users.

If each health check fires a SELECT 1 to the database, you are generating steady background load that compounds with your regular query traffic. At low volume this is irrelevant; on a high-traffic system with a loaded database, it matters.

The fix: cache the health check result for a few seconds. A 5-second TTL means a load balancer checking every 10 seconds hits your real dependencies at most twice per minute per instance, instead of six times per minute.

let cachedResult: { body: HealthResponse; status: number; ts: number } | null = null;
const CACHE_TTL_MS = 5000;

health.get("/health/ready", async (c) => {
  if (cachedResult && Date.now() - cachedResult.ts < CACHE_TTL_MS) {
    return c.json(cachedResult.body, cachedResult.status as 200 | 503);
  }

  // ... run checks ...

  cachedResult = { body, status: httpStatus, ts: Date.now() };
  return c.json(body, httpStatus as 200 | 503);
});

The tradeoff: a failure that occurs between cache refreshes will not be visible for up to 5 seconds. For most systems this is acceptable — load balancers already have failureThreshold and interval buffers built in. If you need sub-second failure detection, you probably need a more sophisticated monitoring pipeline, not a faster health endpoint.

One caveat: cache ok and degraded results freely, but consider a shorter TTL (or no cache) for unhealthy. A single transient Postgres timeout that resolves in 200ms should not cause 5 seconds of 503 responses. A simple approach: set the TTL based on the result status — 5 seconds for ok, 2 seconds for degraded, 0 for unhealthy.

A related point: if your monitoring system needs rich queue metrics, failed job counts, and latency histograms — that data belongs in a metrics endpoint (/metrics in Prometheus format, or a dedicated internal route), not in /health. The healthcheck tells the orchestrator whether to route traffic. Metrics tell your team what is happening inside the system. These are different questions with different consumers.

What Not to Expose Publicly

The full JSON response with component statuses and latencies is useful for your monitoring system and your team. It should not be publicly accessible without authentication.

A response body like this:

{
  "checks": {
    "database": { "status": "ok", "latencyMs": 3 },
    "redis": { "status": "degraded", "error": "connect ECONNREFUSED 10.0.0.5:6379" }
  }
}

tells an attacker your internal Redis IP address and that it is currently unreachable. The error strings from failed checks often contain connection strings, hostnames, and infrastructure details.

Options:

Restrict by network — serve the full response only to requests from internal networks or specific IPs. Your load balancer and monitoring system are internal; public-facing traffic is not.
Two endpoints — a public /health that returns only { "status": "ok" } or { "status": "unhealthy" } without details, and an authenticated /health/details that returns the full response.
Strip errors in production — include error fields only when NODE_ENV !== "production", or behind an ?verbose=1 query param gated by an internal header.

On vatnode, there are two health routes in production: /health is publicly reachable (UptimeRobot and Caddy need it) but returns a sanitized body — status rollup and latencies, no error strings or infrastructure details. /health/details is on an internal network interface behind IP allowlisting and returns the full response including error messages from failed checks. UptimeRobot gets enough signal to alert on downtime; the team gets full diagnostics when they need them.

Graceful Degradation and Not Restarting Too Fast

One pattern that burns people: a liveness probe that checks external dependencies and triggers container restarts when a downstream service is flaky.

Postgres goes down for 30 seconds. Your liveness probe checks Postgres. It fails 3 times in a row. Kubernetes restarts your pod. The new pod starts up, Postgres is still recovering, the liveness probe fails again. Kubernetes restarts again. Now you have a restart loop on top of a database outage, and your application is never in a stable state long enough to serve the requests that do not need Postgres.

The fix: liveness probes check only the process itself (is the Node.js event loop responsive?). Readiness probes check external dependencies. When Postgres recovers, the readiness probe passes, traffic resumes, and no container was ever unnecessarily restarted.

In Docker Compose, you only have one healthcheck, and its failure semantics are different from Kubernetes: an unhealthy container gets restarted, period. There is no "remove from rotation without restarting" equivalent. This makes the Docker case harder: if you include Postgres in your healthcheck, a database outage will trigger container restarts — which is exactly what the Kubernetes section warns against.

The pragmatic approach for Docker Compose: keep the healthcheck strictly process-level — no external dependencies at all. Point it at /health/live, which checks only that the Node.js event loop is responsive. Set retries high (3–5) and interval long (30s+) to absorb transient failures, and use your external monitoring system (Caddy upstream check, UptimeRobot, Grafana) for richer dependency checks. Accept that Docker's healthcheck is a liveness probe in all but name, and build your graceful degradation logic inside the application (circuit breakers, fallback paths) rather than relying on the orchestrator to make the distinction for you.

What Good Looks Like

On vatnode, three consumers read health endpoints with different intentions: Docker calls /health/live every 30 seconds to decide whether to restart the container; Caddy's upstream health check calls /health (readiness-style, full dependency check) every 30 seconds to decide whether to route traffic to this upstream; UptimeRobot calls /health every 5 minutes for external availability monitoring. Each consumer gets the endpoint that matches its semantics. Under normal conditions, the /health response looks like this:

{
  "status": "ok",
  "version": "a3f2c19",
  "uptime": 847293,
  "checks": {
    "database": { "status": "ok", "latencyMs": 2 },
    "redis": { "status": "ok", "latencyMs": 1 },
    "queue": { "status": "ok", "latencyMs": 4 }
  }
}

Uptime of 847293 seconds means roughly 9.8 days since last restart. Database latency of 2ms means the connection pool is healthy and the query is executing normally. This response takes me 3 seconds to read and tells me the system is fine.

When something is wrong, the response tells me where to look immediately — without SSHing into the server, without tailing logs, without waiting for a monitoring alert to fire.

If you're building production infrastructure that needs to stay reliable — whether that's a SaaS API, an e-commerce backend, or a worker-heavy integration platform — healthcheck design is one of those details that seems boring until the 2 AM incident when it's the only tool you have.

I've run these patterns in production across several systems, from vatnode.dev to pikkuna.fi. If you need a senior developer who can own production reliability end-to-end — get in touch. I'm available for freelance projects and long-term engagements.

Related:

Background Jobs in Node.js: BullMQ, pg-boss, or Just a Cron? — BullMQ queue health in practice
Self-Hosting a Production API on a €6/month VPS — Docker Compose healthcheck and Caddy upstream health checks in context
Vatnode VAT Validation SaaS — production system where these health patterns run

External documentation:

PDF Fraud Detection in Loan Origination

Iurii Rogulia — Mon, 29 Jun 2026 10:00:36 +0000

Originally published at htpbe.tech. The version on htpbe.tech stays in sync with the latest detection algorithm — refer to it for the canonical text.

A borrower submits a loan application through your LOS. Bank statements, a W-2, two pay stubs, a tax return. Everything looks right — the account numbers, the employer name, the income figure that places the borrower within your debt-to-income threshold.

Three weeks later, the loan closes. Eighteen months after that, it defaults. When you pull the origination file in a repurchase review, someone finally opens the PDF in the right tool and sees it: the producer field on the bank statement shows “iLovePDF”. The modification date is three days after the creation date. The balance figures were edited after the bank generated the file.

The fraud was in the PDF the entire time. Your LOS never checked.

PDF fraud detection in loan origination closes this gap. A structural forensics check at document intake examines the file itself — not just what it says, but whether its internal history is consistent with the system that claims to have generated it.

The document fraud surface in loan origination

Every loan application that requires income or asset fraud detection is a document fraud surface. The documents involved follow a predictable pattern:

Bank statements — downloaded from the borrower’s online banking portal as a PDF. Real statements are generated by institutional document systems: Chase, Wells Fargo, Bank of America, HSBC. Their producer signatures are consistent and identifiable. An edited statement carries the producer of whichever tool modified it last.

W-2s — issued by employers and generated either by payroll software (ADP, Paychex, Gusto) or filed and downloaded through tax preparation platforms (TurboTax, H&R Block, IRS e-file). A W-2 whose claimed issuer is a national employer but whose producer is a consumer PDF editor should not exist in a normal workflow.

Pay stubs — generated by payroll platforms (ADP Workforce Now, Paychex Flex, Gusto, Rippling). Each has a distinct producer signature. A pay stub that claims ADP origin but carries a different producer was modified between generation and submission.

Tax returns — either IRS-issued transcripts or preparer-generated documents from TurboTax, H&R Block, or a CPA’s practice management software. The producer field narrows the expected origin significantly.

Asset letters and employment fraud detection letters — generated on institutional letterhead, typically by the issuer’s document management system or a staffing platform. These documents should show producer strings matching enterprise software, not consumer tools.

The modification pattern is the same across all of these: the borrower takes a legitimate PDF, opens it in an editor, changes a number — income, balance, employment date, contribution amount — and submits the modified file. The edit takes five minutes. Without structural forensics, detection is nearly impossible.

Why your LOS does not catch this

Encompass, Blend, Byte, and SimpleNexus are routing and workflow platforms. They intake documents, attach them to loan files, route them to the appropriate stage, and hold them for underwriter review. That is what they were built to do.

Document fraud detection in a mortgage LOS is not part of that design. These platforms do not inspect whether the PDF is structurally consistent with the system that allegedly generated it — that was never in their scope.

The OCR extraction layer that sits in front of many LOS platforms — Ocrolus, FormFree, Finicity — has a different role. These tools read the document’s content: the income figures, the account balances, the employment dates. They extract numbers from the page and compare them against stated income or bank transaction data.

Content extraction and structural forensics are different checks. OCR reads what the document says. Structural forensics reads whether the document’s internal history is consistent with the system that claims to have generated it. A skillfully edited bank statement can pass OCR extraction cleanly — the numbers on the page are internally consistent, the transactions add up, the balance matches the running total. The forgery is invisible at the content layer. It is only visible in the file structure.

Neither your LOS nor your OCR vendor closes this gap. Both process the document as presented. Neither examines whether the document was modified before it arrived.

What the structural signals actually look like

A real bank statement from Wells Fargo, Chase, or Bank of America carries a producer string generated by the bank’s document management system. The creation date reflects when the statement was generated. The modification date is absent or matches the creation date within seconds. The xref table has one entry — the original generation, nothing else.

An edited version of that statement shows a different picture. Here is what HTPBE returns on a typical altered bank statement submitted in a mortgage application:

{
  "id": "ck_4e2a1b9f-7c3d-4f8e-b2a1-5d0c9e3a7f2b",
  "status": "modified",
  "modification_confidence": "high",
  "modification_markers": ["HTPBE_EDITING_TOOL_FINGERPRINT", "HTPBE_MULTIPLE_REVISION_LAYERS"],
  "creator": "Wells Fargo Document Services",
  "producer": "Smallpdf",
  "xref_count": 3,
  "has_digital_signature": false,
  "creation_date": 1743465600,
  "modification_date": 1743724800
}

creator: "Wells Fargo Document Services" alongside producer: "Smallpdf" is not a combination that occurs in any legitimate document workflow. Wells Fargo generates the statement; nothing in a normal mortgage process re-saves it in Smallpdf. The creation-to-modification gap of three days confirms the edit window. Three xref entries — the original generation and two subsequent edit sessions — tell you how many times it was touched.

For a W-2 that should have originated from ADP Workforce Now and was instead edited in Adobe Acrobat Reader:

{
  "id": "ck_8f1b3d2e-5a4c-4e7f-c3b2-9e1d4a6b8c0f",
  "status": "modified",
  "modification_confidence": "high",
  "modification_markers": ["HTPBE_EDITING_TOOL_FINGERPRINT", "HTPBE_DATES_DISAGREE"],
  "creator": "ADP Workforce Now",
  "producer": "Adobe Acrobat DC",
  "xref_count": 2,
  "has_digital_signature": false,
  "creation_date": 1735689600,
  "modification_date": 1743120000
}

HTPBE_DATES_DISAGREE with a 3-month gap between creation and modification on a W-2 is not a timing artifact. W-2s are generated in January and submitted to lenders in January or February of the same year. A modification date in late March on a document with a January creation date means someone opened and re-saved it well after initial generation.

What `inconclusive` means in a mortgage context

inconclusive is the verdict returned when a document was created with consumer software — Microsoft Word, Google Docs, LibreOffice, Canva — that does not leave the structural markers present in institutionally-generated documents.

For mortgage documents, inconclusive on a document that claims institutional origin is itself a red flag.

A bank statement that returns inconclusive with producer: "Microsoft Word" was not generated by a bank. Banks do not produce statements in Microsoft Word. The document was built from scratch in consumer software, which is not the same as editing an existing statement but is equally disqualifying for underwriting purposes.

The routing rule for mortgage document intake:

intact — structural signals consistent with claimed origin, proceed to underwriting queue
modified — post-creation edit detected, route to pre-underwriting review with named markers
inconclusive on a bank statement, W-2, pay stub, or tax return — document origin inconsistent with institutional claim, treat as modified and hold for review

inconclusive on a borrower-authored document — a personal letter of explanation, a gift letter — is expected and acceptable. The signal only matters when the document claims to come from an institutional source.

Integration at document intake

The check belongs at the moment the PDF arrives in your intake pipeline — before it is attached to the loan file, before it is queued for the underwriter, before OCR extraction runs. At that point, the document URL is available, the file is in storage, and the check takes under three seconds.

import httpx
import os

HTPBE_API_KEY = os.environ["HTPBE_API_KEY"]
BASE_URL = "https://api.htpbe.tech/v1"
HEADERS = {"Authorization": f"Bearer {HTPBE_API_KEY}"}

# Document types that claim institutional origin in a mortgage application
INSTITUTIONAL_DOC_TYPES = {
    "bank_statement",
    "w2",
    "pay_stub",
    "tax_return",
    "asset_letter",
    "employment_letter",
}


def check_loan_document(pdf_url: str, doc_type: str) -> dict:
    """
    Run structural forensics on a loan application document.
    Returns a routing decision and audit metadata.

    doc_type: "bank_statement" | "w2" | "pay_stub" | "tax_return" |
              "asset_letter" | "employment_letter" | "personal_letter"
    """
    submit = httpx.post(
        f"{BASE_URL}/analyze",
        headers=HEADERS,
        json={"url": pdf_url},
        timeout=30,
    )
    submit.raise_for_status()
    check_id = submit.json()["id"]

    result = httpx.get(
        f"{BASE_URL}/result/{check_id}",
        headers=HEADERS,
        timeout=30,
    )
    result.raise_for_status()
    data = result.json()

    status = data["status"]
    markers = data.get("modification_markers", [])
    producer = data.get("producer", "")
    creator = data.get("creator", "")

    # Modified — post-creation edit confirmed
    if status == "modified":
        return {
            "action": "hold",
            "queue": "pre_underwriting_review",
            "check_id": check_id,
            "reason": f"Structural modification detected: {', '.join(markers)}",
            "creator": creator,
            "producer": producer,
        }

    # Inconclusive on a document claiming institutional origin
    if status == "inconclusive" and doc_type in INSTITUTIONAL_DOC_TYPES:
        return {
            "action": "hold",
            "queue": "pre_underwriting_review",
            "check_id": check_id,
            "reason": (
                f"{doc_type} produced by consumer software ({producer}), "
                "inconsistent with institutional origin"
            ),
            "producer": producer,
        }

    # Intact, or inconclusive on a personal document
    return {
        "action": "proceed",
        "queue": "underwriting",
        "check_id": check_id,
    }

The check_id is stored against the document record in the loan file. If the loan is selected for a QC audit, repurchase review, or regulatory examination, the forensic report is retrievable at any point via GET /api/v1/result/{check_id}. The report includes the verdict, the named markers, the producer and creator strings, and the timestamp of the check — a complete audit trail attached to the document without requiring the original file.

For LOS platforms that process documents via webhook (Blend’s document event hooks, Encompass’s pipeline triggers), the check runs asynchronously on document receipt. The routing decision is applied before the document advances to the next workflow stage.

What the underwriter sees

The structural verdict surfaces alongside the document in the loan file. It is not a score or a probability — it is a named set of signals that the underwriter can read and act on.

For a held document, the underwriter sees:

The verdict: modified
The named markers: HTPBE_EDITING_TOOL_FINGERPRINT, HTPBE_MULTIPLE_REVISION_LAYERS
The creator: Chase Document Management
The producer: iLovePDF
The check ID linking to the full forensic report

This is not a black box. The underwriter can explain the hold, escalate it for borrower clarification, or reject the document based on a documented structural finding. That documented finding matters for the compliance angle.

Compliance: TRID, HMDA, and adverse action documentation

TRID requires that lenders maintain a documented basis for decisions in the loan origination process. HMDA requires that adverse action be supported by identifiable reasons. When a lender rejects a document — or takes adverse action on an application that included fraudulent documents — the regulatory expectation is that the basis for that decision can be stated.

“The bank statement appeared altered” is a subjective finding. “The bank statement returned a modified verdict with markers HTPBE_EDITING_TOOL_FINGERPRINT and HTPBE_MULTIPLE_REVISION_LAYERS — the document’s creator field shows Chase Document Management and the producer field shows iLovePDF, with a modification date three days after creation” is a documented, machine-generated structural finding.

Named structural markers from HTPBE translate directly into adverse action documentation. The check ID links to a permanent, retrievable record of the finding. The audit trail exists from the moment the document was checked — without any manual step from the underwriter or compliance team.

What this does not catch

Structural forensics has a defined scope. Two patterns fall outside it:

Documents fabricated from scratch in the correct software. If a fraudster creates a bank statement using the same document system a real bank uses — or registers a business with a payroll provider and generates a real ADP pay stub with inflated figures — the structural signals will be consistent with a legitimate document. The content is false; the structure is clean. Forensic PDF analysis cannot detect this pattern. Income fraud detection against source data (Plaid, The Work Number, IRS income source-of-truth check) is the appropriate control for fabricated-from-source fraud.

Encrypted or password-protected PDFs. A PDF with strong encryption cannot be analyzed for structural signals. The check returns inconclusive by necessity. For loan document intake, receiving an encrypted document from a borrower without prior arrangement is itself unusual and worth flagging.

For the fraud pattern that accounts for the majority of LOS document fraud — taking a legitimate PDF and editing it with available tools — structural forensics catches it consistently. That is because the tools available for editing PDFs (Adobe Acrobat, Smallpdf, iLovePDF, PDF24, Microsoft Word’s PDF export) all leave recoverable traces in the file structure.

Where to go from here

The check runs against any document URL your LOS has access to — a presigned S3 URL, a Cloudflare R2 link, a Blob URL. There is no file upload to a third-party service in the critical path.

Teams integrating into Encompass, Blend, or a custom LOS pipeline can start with the web tool at htpbe.tech — free to try, with 5 free checks on signup, then pay-per-check (credit packs from $5) or a subscription from $15/mo — to check a sample of application documents from recent closed loans before committing to an API build. The results on a closed-loan sample frequently surface modifications that were missed at origination.

For teams ready to build, API access starts at $15/month with test keys available on all plans for integration testing before live documents are involved.

For the full mortgage use case — including pay stub fraud detection patterns, asset letter signals, and employment letter checks — see mortgage document fraud detection and fintech lending fraud detection. For pay stub specifics, see fake pay stub detection.

PDF Font Subset Divergence: Forensic Tampering Detection

Iurii Rogulia — Sun, 28 Jun 2026 11:00:37 +0000

Originally published at htpbe.tech. The version on htpbe.tech stays in sync with the latest detection algorithm — refer to it for the canonical text.

Inside every PDF that embeds fonts is a six-character prefix that most people never notice. For forensic analysis, it is one of the most precise signals a document carries: it tells you, per page, whether that font was embedded during the same rendering session as the rest of the document.

When those prefixes diverge across pages, the document was not produced in a single pass. Prefix divergence is structural evidence of page assembly from multiple sources — visible in the file itself, without any reference copy.

What font subsetting is

PDF renderers do not embed the full font file for every typeface they use. Embedding a full font — even a modest one — adds hundreds of kilobytes to a file for no practical benefit when only a fraction of its glyphs appear in the document. Instead, the renderer embeds a subset: only the glyphs actually used on that page or in that rendering session.

The PDF specification requires that embedded font subsets be tagged with a six-character random uppercase prefix, followed by a + and the font name:

ABCDEF+Arial
XKZWQP+TimesNewRoman
MJVHRT+SourceSansPro

The prefix is generated fresh at save time by the rendering engine. It is not derived from the font content, the document content, or any deterministic hash. It is random and local to that rendering session.

This has a structural consequence: all fonts embedded during the same rendering session carry a prefix generated by the same process in the same execution context. When a renderer produces a multi-page document in a single call, all font subsets originate from one session. Their prefixes may differ — each subset gets its own random tag — but they are all generated by the same process in a consistent environment.

The forensic signal

When pages in a PDF carry font subsets with prefix patterns that are inconsistent with a single-session origin, it is a signal that the pages were rendered or assembled separately.

This is not about comparing two specific prefix strings — the prefixes are random and have no comparable value. The signal comes from structural inconsistency: fonts that should share a rendering context do not, or page-level font data shows patterns consistent with independent session origins.

Three concrete scenarios produce this pattern:

Multi-call AI document generation. Language model APIs that render PDF output page-by-page — sending each page as a separate generation request — produce independent font sessions per page. Each page’s embedded fonts carry subsets from a distinct rendering context. A three-page document generated this way contains three separate font-subsetting environments. The prefix patterns do not align across pages the way they would in a document rendered in a single pass.

Page insertion from a foreign source. A common fraud pattern in document tampering is inserting a page from one PDF into another. A bank statement with a page replaced from a different statement, or a contract with a signature page substituted. The inserted page was embedded by a different renderer, in a different session, at a different time. Its font subsets carry a different context signature than the surrounding pages.

Template reuse with copy-pasted page objects. Some document assembly tools construct PDFs by duplicating page objects from template files rather than re-rendering content. The duplicated pages bring their original font subsets with them. When the assembly tool adds new pages alongside these imported objects, the new pages carry fresh font subsets from the assembler, while the imported pages retain subsets from the original template’s renderer.

What this exposes in practice

Font subset divergence is most relevant for two categories of documents.

The first is AI-rendered financial summaries and reports. Automated document generation pipelines that use LLM-based rendering (common in fintech for generating statements, summaries, or reports programmatically) often operate page-by-page. When a received document claims to be a system-generated output but its pages show independent font session signatures, the claimed origin is inconsistent with single-pass institutional generation. This does not prove fraud — it identifies a structural anomaly that warrants scrutiny.

The second is manually assembled multi-source documents. HR and lending workflows regularly encounter documents where an applicant has taken pages from different source documents and combined them into a single PDF. A pay stub with a page from a different employer’s statement; a lease agreement with a different tenant’s financial page inserted. Font subset divergence surfaces the page-assembly boundary directly.

How HTPBE surfaces this

HTPBE’s multi-session detection layer scans embedded font subsets across all pages and compares the rendering context signatures. When divergence is detected that is inconsistent with a single-session origin, the analysis adds a modification marker to the result.

A response for a document with font subset divergence looks like this:

{
  "id": "ck_7e3b1a9d-4f2c-4d8a-c3e1-5b9f2a0e7d4c",
  "status": "modified",
  "modification_confidence": "high",
  "modification_markers": [
    "HTPBE_PAGES_FROM_MULTIPLE_SOURCES",
    "HTPBE_MULTIPLE_REVISION_LAYERS"
  ],
  "xref_count": 2,
  "has_digital_signature": false,
  "creator": "Adobe Acrobat",
  "producer": "Adobe Acrobat",
  "creation_date": 1746057600,
  "modification_date": 1747872000,
  "page_count": 4
}

The HTPBE_PAGES_FROM_MULTIPLE_SOURCES marker indicates that pages in this document were assembled from rendering contexts that are structurally inconsistent with single-pass generation. Combined with HTPBE_MULTIPLE_REVISION_LAYERS (two xref entries, meaning the file was opened and saved after initial creation), this result is consistent with page insertion after the original document was produced.

To submit a document for analysis:

curl -X POST https://api.htpbe.tech/v1/analyze \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://your-storage.example.com/submissions/application-pack.pdf"}'

Retrieve the result:

curl https://api.htpbe.tech/v1/result/ck_7e3b1a9d-4f2c-4d8a-c3e1-5b9f2a0e7d4c \
  -H "Authorization: Bearer YOUR_API_KEY"

The full result object includes page count, xref structure, digital signature state, and the complete modification_markers array. Marker descriptions are stable strings — safe to store and use in downstream routing logic.

Limitations

Font subset divergence is not a certain-confidence marker. It is high confidence: a strong structural signal, but not cryptographically provable in the way that signature tampering is.

Several legitimate production workflows produce divergent font subsets:

Document assembly pipelines. Enterprise content management systems sometimes compose final PDFs from independently rendered components — a cover page from one service, body pages from another, appendices from a third. This is architecturally legitimate and produces the same structural pattern as page insertion fraud.

Print-and-scan-and-append workflows. Some legal and compliance workflows scan physical pages and append them to electronically generated PDFs. The scanned pages carry no font subsets (raster content has no embedded fonts), so they introduce a different kind of structural discontinuity rather than prefix divergence — but the broader pattern of mixed rendering contexts is common in legitimate document production.

Certain PDF/A archival tools. Some PDF/A conversion and compliance tools re-embed or re-subset fonts during archival processing. This can cause legitimate documents to show divergent prefix contexts if the conversion tool processed pages independently.

When HTPBE_PAGES_FROM_MULTIPLE_SOURCES appears alongside other markers — HTPBE_MULTIPLE_REVISION_LAYERS, HTPBE_EDITING_TOOL_FINGERPRINT, HTPBE_DATES_DISAGREE — the cumulative signal is substantially stronger than any individual marker. When it appears in isolation, it warrants investigation rather than automatic rejection.

Integrating font-level forensics into document workflows

For backend workflows that need to handle this marker explicitly:

interface HTPBEResult {
  id: string;
  status: 'intact' | 'modified' | 'inconclusive';
  modification_confidence: 'certain' | 'high' | 'none' | null;
  modification_markers: string[];
  xref_count: number;
  page_count: number;
  creator: string | null;
  producer: string | null;
}

function evaluateDocumentResult(
  result: HTPBEResult,
  claimsInstitutionalOrigin: boolean
): 'accept' | 'reject' | 'review' {
  if (result.status === 'modified') {
    // Certain-confidence markers: auto-reject
    const certainMarkers = [
      'HTPBE_POST_SIGNATURE_EDIT',
      'HTPBE_SIGNATURE_REMOVED',
      'HTPBE_DATES_DISAGREE',
    ];
    const hasCertain = result.modification_markers.some((m) =>
      certainMarkers.includes(m)
    );
    if (hasCertain) return 'reject';

    // High-confidence: reject if document claims institutional origin,
    // otherwise route to review
    return claimsInstitutionalOrigin ? 'reject' : 'review';
  }

  if (result.status === 'inconclusive' && claimsInstitutionalOrigin) {
    // Consumer software origin for a document claiming bank/gov origin
    return 'review';
  }

  return 'accept';
}

For workflows that need to flag assembled or AI-generated multi-page documents, HTPBE_PAGES_FROM_MULTIPLE_SOURCES is one of several markers the aggregate verdict weighs together — the API returns the combined result, so you act on the verdict rather than re-deriving the signal yourself.

What font subset divergence does not catch

A forger who renders an entire fraudulent document in a single rendering session — fabricating all pages together with one tool call — produces consistent font subsets. This marker targets assembly-based fraud and multi-session generation, not single-session fabrication from scratch.

For single-session fabrication, other layers are more relevant: producer/creator mismatches against known institutional generators, timestamp anomalies, and structural patterns in the xref chain. The full forensic analysis runs all of these concurrently; font subset divergence is one input to the aggregate verdict, not the only one. The PDF xref table forensics post covers how the update chain independently surfaces modification history — a complementary signal to font-level analysis.

Who this matters to

Developers building document intake pipelines for lending, HR, insurance, or legal tech platforms can use this marker to triage documents that warrant manual inspection. It is not a binary reject signal in isolation — it is a routing signal that sends specific document patterns to a review queue rather than auto-accept.

Security researchers and forensic analysts can use the modification_markers array to reconstruct a document’s assembly history with greater precision than xref counts alone. Font-level session data provides page-granularity information about where document boundaries likely exist.

The HTPBE API returns this and all other forensic markers in a single response. Plans start at $15/month for 30 documents — or use the web tool to run an immediate check: free to try, with 5 free checks on signup, then pay-per-check (credit packs from $5).

PostgreSQL Production Checklist: UUIDs, RLS, Indexes, Pooling

Iurii Rogulia — Sun, 28 Jun 2026 10:00:37 +0000

PostgreSQL is the only database I trust for production SaaS. Not because the alternatives are bad — but because Postgres rewards the time you invest in learning it. After using it across vatnode.dev, pikkuna.fi, pi-pi.ee, and htpbe.tech, I have a set of patterns I reach for every time without thinking. This is that collection.

This is not a tutorial. It's opinionated, and I'll explain the "why" behind each decision. If you know basic SQL and want to know what experienced practitioners actually do — this is the article I wish existed when I started.

UUIDs vs Serial IDs

I use two ID strategies, depending on the table's purpose.

For user-facing entities — users, orders, invoices, API keys — I use UUID v7 via the uuidv7 package. Two reasons: enumeration prevention and index performance. If your order ID is 12345, a determined attacker can iterate through orders. With a UUID, that's not feasible. And unlike UUID v4 (fully random), UUID v7 is time-sorted — new rows always land at the end of the B-tree, not at a random position. This matters at scale: on a 5M-row table, UUID v4 produces a 285MB index; UUID v7 produces a 118MB index with 3x lower INSERT latency. I covered the full breakdown in UUID v7, ULID, and NanoID compared.

Important: PostgreSQL's built-in gen_random_uuid() generates UUID v4, not v7. Generate v7 in application code and pass it explicitly.

CREATE TABLE orders (
  id UUID PRIMARY KEY,
  user_id UUID NOT NULL REFERENCES users(id),
  created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
-- No DEFAULT on id — the application generates UUID v7 before INSERT

For internal join tables — user_roles, order_items, tag_assignments — I use BIGSERIAL. These IDs never surface in URLs or APIs, so enumeration isn't a concern. BIGSERIAL is faster to index, smaller on disk, and slightly better for sequential inserts.

In Drizzle ORM:

import { pgTable, uuid, bigserial, integer, text, timestamp } from "drizzle-orm/pg-core";
import { uuidv7 } from "uuidv7";

export const orders = pgTable("orders", {
  id: uuid("id")
    .primaryKey()
    .$defaultFn(() => uuidv7()),
  userId: uuid("user_id")
    .notNull()
    .references(() => users.id),
  createdAt: timestamp("created_at", { withTimezone: true }).notNull().defaultNow(),
});

export const orderItems = pgTable("order_items", {
  id: bigserial("id", { mode: "number" }).primaryKey(),
  orderId: uuid("order_id")
    .notNull()
    .references(() => orders.id),
  productId: uuid("product_id").notNull(),
  quantity: integer("quantity").notNull(),
});

Timestamps: Always UTC, Always Two Columns

Every table I create gets created_at and updated_at. No exceptions. I've debugged too many issues caused by "we didn't think we'd need that timestamp" to ever skip it.

CREATE TABLE subscriptions (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id UUID NOT NULL REFERENCES users(id),
  plan TEXT NOT NULL,
  created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

Always TIMESTAMPTZ (timestamp with time zone), never TIMESTAMP. Postgres stores TIMESTAMPTZ as UTC internally and converts on read based on the session timezone. If you store TIMESTAMP and later need to handle users across time zones, you're retrofitting a problem that didn't have to exist.

For updated_at, I use a trigger rather than relying on the ORM to set it. ORMs miss updates done via raw SQL, admin tools, or migration scripts.

CREATE OR REPLACE FUNCTION update_updated_at()
RETURNS TRIGGER AS $$
BEGIN
  NEW.updated_at = NOW();
  RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER set_updated_at
BEFORE UPDATE ON subscriptions
FOR EACH ROW EXECUTE FUNCTION update_updated_at();

Soft Deletes: Useful, With One Critical Index

Soft deletes — marking a row as deleted with a deleted_at timestamp instead of removing it — are useful when you need an audit trail, or when "deleted" means something recoverable (cancelled subscription, archived document).

ALTER TABLE documents ADD COLUMN deleted_at TIMESTAMPTZ;

The query pattern for fetching live records:

SELECT * FROM documents WHERE deleted_at IS NULL;

The gotcha: if you add a standard index on deleted_at, most queries will still do a full table scan — NULL values are spread throughout the table. What you actually want is a partial index — an index that only covers the rows you'll actually query:

-- Standard index — doesn't help much for WHERE deleted_at IS NULL queries
CREATE INDEX idx_documents_deleted_at ON documents(deleted_at);

-- Partial index — only indexes live rows. Much smaller, much faster.
CREATE INDEX idx_documents_active ON documents(user_id, created_at)
  WHERE deleted_at IS NULL;

The partial index only stores rows where deleted_at IS NULL. For a table with 10% deleted records, that's 10% smaller. More importantly, every query for live records uses this index directly.

When NOT to use soft deletes: high-volume tables where deleted rows accumulate fast (event logs, audit trails), and tables where regulatory compliance requires actual deletion (GDPR right to erasure). For GDPR, you need hard deletes. Trying to justify soft deletes as a GDPR exemption is a conversation you do not want to have with a data protection authority.

JSONB: Where It Helps and Where It Hurts

Postgres JSONB is genuinely useful in specific cases. I reach for it in three situations:

Flexible attributes where the shape isn't fully known at design time — product metadata, user preferences, feature flags per tenant.

Audit logs where you want to store the full before/after state of a row without designing a separate schema for every table.

External API responses — store the raw Stripe event, the PostNord shipment response, whatever third-party payload you received. Useful for debugging and for replaying events without re-fetching from the source. In pikkuna.fi, I store every API response from PostNord and Zoho in a raw_payload JSONB column alongside the parsed fields.

Where it hurts: anything you filter or sort on regularly. If you're querying WHERE metadata->>'status' = 'active', you're fighting the type system and paying in query planning complexity. Extract frequently-queried fields into proper typed columns.

// Good use of JSONB — audit log with flexible payload
export const auditLog = pgTable("audit_log", {
  id: bigserial("id", { mode: "number" }).primaryKey(),
  userId: uuid("user_id").references(() => users.id),
  action: text("action").notNull(),
  tableName: text("table_name").notNull(),
  recordId: uuid("record_id").notNull(),
  before: jsonb("before"),
  after: jsonb("after"),
  createdAt: timestamp("created_at", { withTimezone: true }).notNull().defaultNow(),
});

// Bad use of JSONB — filtering on JSON fields you should have typed
// SELECT * FROM products WHERE attributes->>'category' = 'electronics'
// Just add a category column.

If you find yourself writing ->>'field' in WHERE clauses more than once, that field belongs in a proper column.

Row-Level Security for Multi-Tenant SaaS

slug="mvp-development"
text="Database architecture — schema design, RLS, indexes, migrations, and connection pooling — is part of every production SaaS I build. Not an afterthought."
/>

Row-level security (RLS) is a Postgres feature that enforces data access rules at the database layer. For multi-tenant SaaS — where every user should see only their own data — it's the most reliable isolation layer you can add.

Without RLS, your application code is the only thing standing between a user and another user's data. One missed WHERE user_id = $1 clause and data leaks. RLS makes isolation structural.

Basic pattern:

-- Enable RLS on the table
ALTER TABLE documents ENABLE ROW LEVEL SECURITY;

-- Policy: users can only see their own documents
CREATE POLICY documents_user_isolation ON documents
  FOR ALL
  USING (user_id = current_setting('app.current_user_id')::UUID);

In your application, set the context before queries in the same transaction:

// lib/db-context.ts
import { db } from "@/lib/db";
import { sql } from "drizzle-orm";

export async function withUserContext<T>(
  userId: string,
  fn: (tx: typeof db) => Promise<T>
): Promise<T> {
  return db.transaction(async (tx) => {
    await tx.execute(sql`SELECT set_config('app.current_user_id', ${userId}, true)`);
    return fn(tx);
  });
}

The true parameter in set_config makes the setting transaction-local — it resets after the transaction ends. This is important: if you use a connection pool, you don't want one user's context bleeding into the next query on the same connection.

I use Supabase-style RLS on some projects (auth.uid()) and the set_config pattern on others depending on whether I control the auth layer. Both work — the key is consistency.

Indexes: What Actually Matters

Over-indexing is as harmful as under-indexing. Indexes slow down writes, consume storage, and confuse the query planner when you have too many of them. Here's what I actually add.

Partial indexes — covered above for soft deletes. Also useful for status-filtered queries:

-- Only index rows that are in an actionable state
CREATE INDEX idx_jobs_pending ON jobs(created_at)
  WHERE status = 'pending';

Covering indexes — include columns in the index so Postgres can answer the query from the index alone, without hitting the table:

-- A query that fetches user_id and status for orders in a date range
-- can be answered entirely from this index
CREATE INDEX idx_orders_covering ON orders(created_at DESC)
  INCLUDE (user_id, status);

Composite index column order — start with columns used in equality filters, then range or sort columns. Selectivity matters, but query shape matters more: a column with high cardinality that only appears in a range predicate should not go first if a lower-cardinality equality column appears in every query.

-- Bad order: status has low cardinality (3 possible values)
CREATE INDEX idx_bad ON subscriptions(status, user_id);

-- Good order: user_id is highly selective
CREATE INDEX idx_good ON subscriptions(user_id, status);

When NOT to index: tables under 10,000 rows (sequential scan is often faster), columns that change frequently (every update rewrites the index entry), and columns you only filter on in bulk admin queries (use a separate read replica or just accept the scan).

CTEs: Readability and the Optimization Fence

Common table expressions (WITH queries) are the most underused tool for readable SQL. I reach for them whenever a query would otherwise require nested subqueries.

-- Without CTE — hard to read
SELECT u.email, COUNT(o.id) as order_count
FROM users u
LEFT JOIN orders o ON o.user_id = u.id
  AND o.created_at > NOW() - INTERVAL '30 days'
  AND o.status = 'paid'
WHERE u.created_at > NOW() - INTERVAL '90 days'
GROUP BY u.email;

-- With CTE — each step is named and readable
WITH recent_users AS (
  SELECT id, email
  FROM users
  WHERE created_at > NOW() - INTERVAL '90 days'
),
recent_paid_orders AS (
  SELECT user_id, COUNT(*) as order_count
  FROM orders
  WHERE created_at > NOW() - INTERVAL '30 days'
    AND status = 'paid'
  GROUP BY user_id
)
SELECT u.email, COALESCE(o.order_count, 0) as order_count
FROM recent_users u
LEFT JOIN recent_paid_orders o ON o.user_id = u.id;

One thing to know: since Postgres 12, CTEs are not automatically optimization fences. Before Postgres 12, the planner always materialized each CTE (evaluated it once and stored the result). Since 12, the planner can inline CTEs into the main query and optimize across them. If you explicitly need materialization (e.g., to prevent re-evaluation of a volatile function), use WITH cte AS MATERIALIZED (...).

This matters when optimizing slow queries — don't assume a CTE is cached or is being optimized in isolation.

Transactions and `FOR UPDATE`: Preventing Race Conditions

Whenever two concurrent operations read the same row and both decide to act on it, you have a race condition. Classic examples: inventory decrement, seat booking, credit balance deduction.

The naive pattern:

// Race condition — two concurrent requests both read stock = 1
const product = await db.select().from(products).where(eq(products.id, productId));
if (product.stock > 0) {
  await db
    .update(products)
    .set({ stock: product.stock - 1 })
    .where(eq(products.id, productId));
  // Both requests pass the check, both decrement. Stock goes to -1.
}

The fix is SELECT ... FOR UPDATE inside a transaction. This locks the row until the transaction commits, forcing concurrent requests to wait:

// lib/inventory.ts
import { db } from "@/lib/db";
import { products } from "@/lib/db/schema";
import { eq, sql } from "drizzle-orm";

export async function decrementStock(productId: string): Promise<boolean> {
  return db.transaction(async (tx) => {
    // Lock the row — concurrent requests will wait here
    const [product] = await tx
      .select({ stock: products.stock })
      .from(products)
      .where(eq(products.id, productId))
      .for("update");

    if (!product || product.stock <= 0) {
      return false; // Out of stock
    }

    await tx
      .update(products)
      .set({ stock: sql`${products.stock} - 1` })
      .where(eq(products.id, productId));

    return true;
  });
}

The atomic stock - 1 expression in the UPDATE is also intentional — it applies the decrement relative to the current value at write time, not relative to the value you read earlier. This is safe even without FOR UPDATE, but combining both gives you correctness under high concurrency.

Connection Pooling: The Math You Need to Do

Postgres has a hard limit on simultaneous connections. The default is 100. Each connection consumes ~5–10MB of memory. On a $20/month VPS, you do not have room for 100 open connections.

The problem with serverless and edge deployments is that each function invocation can open its own connection. 50 concurrent requests to a Next.js API route = 50 connections. Under traffic, this exhausts the pool and requests start failing with too many connections.

The solution is PgBouncer or an equivalent pooler (Supabase uses PgBouncer under the hood; Neon has its own). The connection pool sits between your application and Postgres, maintaining a small number of real connections and multiplexing many application connections through them.

A rough sanity check for initial sizing:

Available connections ≈ (RAM in MB / 10) - 5 (reserved for superuser)
Pool size per service ≈ available connections / number of services

For a 1GB VPS with Postgres and one application this gives you a starting point of ~80 application connections. Actual limits depend on work_mem, shared_buffers, autovacuum workers, and query complexity — treat this as a sanity check, not a formula to follow blindly.

PgBouncer configuration I use in transaction pooling mode (recommended for most SaaS apps):

[databases]
myapp = host=localhost port=5432 dbname=myapp

[pgbouncer]
pool_mode = transaction
max_client_conn = 1000
default_pool_size = 25
reserve_pool_size = 5
reserve_pool_timeout = 3
server_idle_timeout = 600

Transaction pooling mode returns connections to the pool after each transaction, not after the session ends. This is the most efficient mode. The tradeoff: you cannot use session-level features like SET (session-scoped config), prepared statements across requests, or LISTEN/NOTIFY across a transaction boundary. For RLS with set_config(..., true), this matters — use transaction-scoped settings, which transaction pooling handles correctly.

Migrations: Never Touch the Schema Manually

This sounds obvious and people still do it. I've inherited production databases where no one knows exactly what schema is running, because "someone made a quick change in the admin panel six months ago."

The rule is absolute: every schema change goes through a migration file, committed to version control, applied by a migration tool. No exceptions for "just a quick index" or "just adding one column."

I use Drizzle ORM's migration workflow:

# Generate migration from schema changes
npx drizzle-kit generate

# Apply migrations (run in CI/CD, not manually)
npx drizzle-kit migrate

The migration file is a plain SQL file, committed alongside the code change that needs it. Every deployment applies pending migrations before the new code starts.

On rollback strategy: Postgres DDL is transactional. If you add a column, index, and constraint in one migration, and the constraint fails, the whole migration rolls back. This is a feature — design your migrations to be atomic. For large tables, be careful with operations that hold ACCESS EXCLUSIVE locks (adding NOT NULL constraints without a default, rebuilding indexes). Use CREATE INDEX CONCURRENTLY and ALTER TABLE ... SET NOT NULL with a check constraint added in an earlier migration.

One pattern I always follow: never delete a column in the same migration that removes its usage from the application code. Deploy the code change first (which stops writing to the column), then remove the column in a subsequent migration. This avoids downtime from deployment ordering.

What This Looks Like Across Projects

In vatnode.dev — RLS for tenant isolation on validation logs, partial indexes on deleted_at and status, UUID v4 for all API-exposed IDs (vatnode predates my switch to v7; new projects use v7), PgBouncer in front of Postgres. The schema has 11 tables; migrations are applied automatically on each Coolify deployment.

In pikkuna.fi — JSONB for storing raw PostNord and Zoho API responses alongside structured order data, FOR UPDATE transactions for order status transitions, updated_at triggers on every table. 100% automated order pipeline means concurrent webhooks hit the same order rows — without locking, you get races.

In pi-pi.ee — soft deletes on products and categories (store operators want to recover things), covering indexes on the orders table for the admin dashboard queries, all timestamps UTC.

Good PostgreSQL architecture is mostly boring: stable IDs, explicit ownership, predictable migrations, and indexes that match the queries you actually run. The value isn't in any individual pattern — it's in never having to debug the class of problem each one prevents.

These patterns are not exotic. They're the defaults I reach for because skipping them has caused me real problems in the past. UUID enumeration, missing audit trails, oversized connection pools, schema drift — each is a class of incident that becomes impossible once the pattern is in place.

If you're building a SaaS or API-backed product and want a database foundation that holds under production load — get in touch. I'm available for freelance projects and longer-term engagements.

Related reading:

UUID v7, ULID, and NanoID: Which One Should You Use? — deeper look at ID strategies and insert performance
Stripe Webhooks Done Right: Production Architecture — idempotency patterns that depend on the PostgreSQL foundation above
Production SaaS Checklist: Launch in 8 Weeks With Next.js — the full checklist that includes database setup
Vatnode VAT Validation SaaS — 11-table schema running these patterns in production
Drizzle ORM documentation
PostgreSQL official documentation — Row Security Policies
PgBouncer documentation

H-1B Salary Slip Fraud Detection: Altered Indian Payslips at the Offer Stage

Iurii Rogulia — Sat, 27 Jun 2026 10:41:33 +0000

Originally published at htpbe.tech. The version on htpbe.tech stays in sync with the latest detection algorithm — refer to it for the canonical text.

A mid-sized US fintech company extends an offer to a senior engineer based in Bangalore. The candidate submits three months of payslips from their current employer — a well-known Indian IT services firm — showing a monthly CTC of ₹18 lakhs. The offer is set accordingly. Three weeks later, during background screening, the BGV vendor contacts the employer and receives no response. The candidate explains that Indian companies routinely ignore BGV inquiries and submits an employment confirmation letter instead.

The actual salary on those payslips was ₹11 lakhs. The candidate had opened the PDFs in a browser-based editor, changed the compensation figures, and re-exported them. Everything else — the employer letterhead, the PF deductions, the company logo — was authentic.

This is the dominant salary fraud vector in Indian-to-US hiring pipelines. H-1B salary slip fraud detection is not part of most offer workflows.

Why H-1B Salary Slip Fraud Detection Fails in Standard BGV Workflows

Indian background screening has a structural gap that enables document fraud: employer non-response rates on employment and compensation fraud detection routinely run between 15% and 30%, according to data from major BGV vendors. Indian employers — particularly mid-size IT services companies and startups — frequently do not maintain dedicated BGV response teams. HR contacts leave. Emails go unanswered for weeks.

When a BGV vendor cannot confirm compensation, the candidate is typically asked to provide supporting documentation: payslips, an offer letter, or a compensation certificate. This is a reasonable fallback. It is also the point where document fraud enters the workflow.

The candidate already has their real payslips. They know the exact format, the correct deductions line, the PF and ESI structure. Editing the compensation figure and re-exporting takes under ten minutes. The resulting document looks identical to a genuine payslip because it is a genuine payslip — with one number changed.

US immigration counsel reviewing documents for H-1B prevailing wage compliance, and HR teams calibrating offers against claimed compensation, work from those figures. If the underlying documents have been altered, the entire wage analysis is based on false data.

What the Altered Payslip Actually Contains

Indian payroll is dominated by a small number of software platforms: Keka, GreytHR, Darwinbox, Zoho Payroll, and SAP SuccessFactors for larger enterprises. Each platform generates PDFs with a recognizable structural signature — a specific Producer value, a single-pass xref structure, and metadata that reflects an automated document generation event rather than a manual save.

When a candidate downloads their genuine payslip and opens it in a PDF editor — Adobe Acrobat, Foxit, Smallpdf, ILovePDF, or even a browser’s built-in save-as function — the editing software appends its own changes to the file. In most cases it writes its own name into the Producer field. It adds a new cross-reference entry recording the edit session. It updates the modification timestamp.

The result is a file that claims to have been generated by Keka or GreytHR — because the original producer metadata is still present in the file’s early sections — but also carries a second production record from the editing tool. The modification timestamp is later than the document’s claimed issuance date. The edit history shows at least two sessions: the original payroll generation and the candidate’s edit.

None of this is visible when you open the PDF. It is preserved in the file’s internal structure.

What HTPBE Detects

When a salary slip with this edit pattern is submitted through HTPBE, the analysis reads those structural layers and returns a verdict based on what the file itself recorded.

A realistic API response for an altered Indian payslip:

{
  "id": "ck_7e2a91f4-bc34-4d7e-a108-3f5c8e012bd9",
  "status": "modified",
  "modification_confidence": "high",
  "modification_markers": [
    "HTPBE_EDITING_TOOL_FINGERPRINT",
    "HTPBE_MULTIPLE_REVISION_LAYERS",
    "HTPBE_DATES_DISAGREE"
  ],
  "creator": "Keka HR",
  "producer": "Smallpdf",
  "creation_date": 1738368000,
  "modification_date": 1745712000,
  "xref_count": 3
}

Three signals are present in this response:

HTPBE_EDITING_TOOL_FINGERPRINT — The file’s internal records name two different software systems. Keka HR generated the original document. Smallpdf processed it afterward. Payroll software does not send finished payslips through Smallpdf. The mismatch indicates the file was reopened in a different tool after initial generation.

HTPBE_MULTIPLE_REVISION_LAYERS — Three xref entries exist in the file. The first corresponds to the original payroll generation. The later entries record editing sessions. A genuine payslip from an automated payroll system has exactly one.

HTPBE_DATES_DISAGREE — The modification timestamp is nearly three months after the creation date. A payslip generated once by a payroll system has matching creation and modification times. A gap of months indicates the file was opened and resaved long after it was originally issued.

Together, these markers produce a modified verdict. No comparison against an original is needed — the file’s own structure records what happened.

The INCONCLUSIVE Case: Small Employers and Word-Generated Slips

Not all Indian employers use structured payroll software. Many small and mid-size employers — boutique agencies, early-stage startups, family-owned businesses — generate payslips in Microsoft Word or Google Docs, export them to PDF, and send them by email. These documents have no institutional payroll metadata. They originate from consumer software.

For these files, HTPBE returns inconclusive.

inconclusive is not a fraud flag. It means the document was created with consumer software, and there is no structural basis for checking whether it was modified after creation.

The correct interpretation depends on the context:

If the candidate works at a known large employer — Infosys, Wipro, TCS, HCL, Cognizant, or any company with structured payroll infrastructure — an inconclusive verdict on their payslip is a significant flag. Those companies use enterprise payroll systems. Their payslips do not originate in Word. Escalate to BGV.

If the candidate works at a small employer or consultancy where Word-generated slips are common, inconclusive alone does not indicate fraud. Combine it with other signals: Is the document’s visual layout consistent? Are the PF/ESI deduction calculations correct? Does the employer size match the claimed compensation band? Is there corroborating documentation?

The distinction matters in practice. Rejecting every inconclusive document in an Indian hiring pipeline would produce a high false-positive rate for candidates from smaller employers. The more defensible approach is to treat inconclusive as a signal that feeds a risk-scoring model, not as an automatic rejection.

How to Integrate This Into Your Offer and fraud-detection workflow

The right time to run this check is at the offer stage, before the compensation figure is finalized and before H-1B sponsorship documentation is started.

Step 1: Collect payslips as a URL or upload them to your document storage.
HTPBE analyzes PDFs by URL. When candidates submit payslips through your ATS portal or onboarding platform, generate a presigned URL from your document storage (S3, Azure Blob, Google Cloud Storage) and pass it to the payslip fraud detection API.

Step 2: Submit to HTPBE and route on verdict.

curl -X POST https://api.htpbe.tech/v1/analyze \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://your-storage.example.com/candidates/12345/payslip-jan.pdf"}'

Step 3: Route based on the response.

def route_payslip(result: dict, employer_is_large: bool) -> str:
    status = result["status"]

    if status == "modified":
        # Structural evidence of post-creation edit — escalate immediately
        return "escalate_to_hr"

    if status == "inconclusive" and employer_is_large:
        # Large employer should not produce Word-origin payslips — escalate to BGV
        return "escalate_to_bgv"

    if status == "inconclusive":
        # Small employer context — flag for manual review, not auto-reject
        return "manual_review"

    # intact
    return "proceed"

Store the HTPBE check ID alongside the candidate record. If a hiring decision is later challenged, the forensic report is retrievable from GET /api/v1/result/{check_id} and provides an auditable basis for the decision.

The Cross-Border Pattern: UK Tier 2 and Australian 482 Visas

The same structural fraud pattern appears in UK and Australian immigration workflows. Indian nationals applying for UK Skilled Worker visas (formerly Tier 2) and Australian Temporary Skill Shortage (subclass 482) visas must provide proof of current overseas earnings as part of the sponsorship and salary assessment process.

The incentive is identical: higher claimed current earnings support a stronger case for the role’s salary level and the visa’s skills threshold. The method is identical: genuine payslip, edited compensation figure, re-exported PDF.

UK Home Office and Australian Department of Home Affairs guidance both require sponsors to check that salary evidence is genuine. The structural forensic checks that detect this fraud pattern in US H-1B workflows apply directly to Tier 2 and 482 applications. If your organization sponsors visas across multiple jurisdictions, a single API integration covers all three workflows.

Who This Is For

US HR and talent acquisition teams processing Indian candidates for H-1B sponsorship or US-based offers where salary history informs the offer. Run HTPBE at the document collection stage, before offer letters are finalized and before immigration counsel is engaged.

Background fraud detection vendors covering Indian employment. When primary-source fraud detection fails and the candidate provides supporting payslip documentation, forensic analysis is the check that BGV currently lacks. The fake salary slip detection page covers the full detection pattern for payslip fraud across geographies.

Immigration compliance teams and counsel reviewing compensation documentation for prevailing wage determinations and visa petitions. A modified verdict on a submitted payslip is a material fact for the record. Document it before the petition is filed, not after.

HR platforms and ATS vendors looking to add document integrity checks to their candidate fraud detection layer. The HR and hiring solutions page covers how HTPBE integrates into recruiting workflows at the platform level.

The API is available on the Starter plan at $15/month. For BGV vendors and ATS platforms processing large volumes, the Growth and Pro plans support up to 1,500 checks per month.

BullMQ vs pg-boss vs Cron: Node.js Background Jobs Compared

Iurii Rogulia — Sat, 27 Jun 2026 10:03:48 +0000

Your order confirmation email is sometimes delayed. Your nightly report occasionally runs twice. Your integration with the third-party API times out and nobody notices until a customer complains three days later.

These are background job problems. And the solution is almost always "you need a proper job queue" — but that phrase hides a decision with real infrastructure consequences. Redis-backed or Postgres-backed? A cron schedule or event-driven workers? Retries with backoff or fire-and-forget?

I have dealt with this across several production systems: on vatnode.dev, where BullMQ workers handle subscription logic triggered by Stripe webhooks, and on pikkuna.fi, where the full order pipeline — CRM update, shipment creation, invoice generation, email — runs through an async worker chain. Here is how I make that decision.

Three Levels of Background Work

Before comparing tools, it helps to be precise about what "background job" actually means in your case.

Level 1: Scheduled tasks, single process, low stakes. You need to send a weekly digest email, refresh a cache at 6 AM, or clean up expired sessions every hour. The work is triggered by time, not events. If it fails, it will run again in an hour. You are running one app instance.

This is what node-cron or an OS-level cron is for. Bringing in Redis and a queue worker for this is over-engineering.

Level 2: Event-driven work that needs retries, deduplication, or concurrency control. A Stripe webhook fires, you need to create an order, update a CRM, and send a confirmation email. Any of these steps can fail. You might be running multiple app instances. You need to guarantee delivery and avoid duplicates.

This is what a job queue is for. Either BullMQ or pg-boss applies here.

Level 3: High-throughput processing pipelines. You are ingesting events at thousands per minute, running parallel transformations, or building complex job dependency graphs (fan-out, fan-in, rate-limited sub-queues). Sub-second job pickup latency matters.

This is where BullMQ (Redis-backed) becomes the right tool over Postgres alternatives.

Most applications are Level 1 or Level 2. I will focus there.

Simple Cron: When It Is Actually Fine

node-cron and similar packages are underrated for genuinely simple use cases. The implementation is minimal:

// lib/scheduler.ts
import cron from "node-cron";
import { db } from "@/lib/db";
import { sessions } from "@/lib/db/schema";
import { lt } from "drizzle-orm";

export function startScheduler(): void {
  // Run every day at 3 AM
  cron.schedule("0 3 * * *", async () => {
    try {
      const deleted = await db
        .delete(sessions)
        .where(lt(sessions.expiresAt, new Date()))
        .returning({ id: sessions.id });

      console.log(`Cleaned up ${deleted.length} expired sessions`);
    } catch (err) {
      console.error("Session cleanup failed:", err);
    }
  });
}

Call startScheduler() from your app entrypoint and you are done.

These are the reasons to move to a queue:

No distribution. If you deploy two instances, both run the cron — your cleanup job runs twice. For idempotent cleanup that's harmless; for an email send, it isn't.
No retries. If the database is temporarily unavailable, the job throws and nothing re-runs it until the next scheduled interval.
No visibility. You cannot see which jobs ran, which failed, or how long they took.

These are not reasons to avoid cron — they are the checklist for deciding whether cron is sufficient. If none of these matter for your use case, keep it simple.

BullMQ: The Redis-Backed Standard

slug="mvp-development"
text="Need a production SaaS with reliable background job processing built in? I architect the full stack — queues, workers, retries, and observability — as part of every MVP I deliver."
/>

BullMQ is the most widely used job queue in the Node.js ecosystem. It uses Redis as its backing store and provides retries with configurable backoff, job prioritization, concurrency control, delayed jobs, repeating jobs, job dependencies, and a solid UI via Bull Board.

Here is the basic setup. A queue for enqueueing, a worker for processing:

// lib/queues/order-queue.ts
import { Queue, Worker, Job } from "bullmq";
import { redis } from "@/lib/redis"; // ioredis instance

interface OrderJobData {
  orderId: string;
  customerId: string;
}

// The queue — used by your API/webhook handlers to enqueue work
export const orderQueue = new Queue<OrderJobData>("order-processing", {
  connection: redis,
  defaultJobOptions: {
    attempts: 4,
    backoff: {
      type: "exponential",
      delay: 3000, // 3s, 6s, 12s, 24s
    },
    removeOnComplete: { count: 500 },
    removeOnFail: false, // Keep failed jobs for inspection
  },
});

// The worker — runs in a separate process or alongside your app
export const orderWorker = new Worker<OrderJobData>(
  "order-processing",
  async (job: Job<OrderJobData>) => {
    const { orderId, customerId } = job.data;

    await job.updateProgress(10);
    await syncOrderToCrm(orderId);

    await job.updateProgress(40);
    await createShipment(orderId);

    await job.updateProgress(70);
    await generateInvoice(orderId, customerId);

    await job.updateProgress(90);
    await sendConfirmationEmail(customerId, orderId);

    await job.updateProgress(100);
    return { processed: true };
  },
  {
    connection: redis,
    concurrency: 3, // Process up to 3 orders simultaneously
  }
);

orderWorker.on("failed", (job, err) => {
  console.error(`Order job ${job?.id} failed after all retries:`, err);
  // Alert your team — Telegram, Slack, PagerDuty, whatever you use
});

Enqueueing from a Stripe webhook handler is a single line:

await orderQueue.add(
  "process-order",
  { orderId, customerId },
  { jobId: `order-${orderId}` } // Deduplicate by orderId in the active window
);

The jobId is how BullMQ handles deduplication within active jobs — if a job with that ID is already waiting or active, the new enqueue is silently ignored. This is useful for webhook retries, but note it does not cover completed jobs. For that you still need a database-level idempotency check.

What BullMQ does well:

Job prioritization (numeric priority field on each job)
Delayed jobs — schedule a follow-up email 3 days after sign-up
Repeating jobs with cron expressions — a proper replacement for node-cron that works across multiple instances
Job dependencies with FlowProducer — fan-out, fan-in, pipelines
Rich UI with Bull Board
Large community, mature documentation

What to watch out for:

Redis is a separate infrastructure component to manage. If Redis goes down, job processing stops — jobs in the queue are still there, but no new ones are picked up and none are processed until Redis recovers. For most applications this is acceptable; for a payment pipeline it means you need Redis HA (Redis Sentinel or Cluster, or a managed service like Upstash).

The other thing: BullMQ workers are long-running processes. In a Next.js deployment, you need a separate worker process. On vatnode, the Turborepo monorepo has a dedicated apps/worker package that runs alongside the Next.js app in the same container.

pg-boss: Postgres as Your Queue

pg-boss takes a different approach: it uses Postgres as the queue backing store, maintaining a pgboss schema with job tables, state transitions, and indexes. No Redis required.

The key insight: if you already use Postgres, you can enqueue a job in the same database transaction as your business logic. This gives you ACID guarantees that Redis cannot match.

Consider this scenario: you receive a payment confirmation and need to (1) update the order status and (2) enqueue a job to send a confirmation email. With Redis-backed queues:

// Without transactional enqueue — there's a window where the DB write succeeds
// but the queue enqueue fails, and the email never gets sent
await db.update(orders).set({ status: "paid" }).where(eq(orders.id, orderId));
await orderQueue.add("send-confirmation", { orderId }); // This can fail independently

With pg-boss, both operations happen in the same Postgres transaction:

// lib/queues/pg-boss-queue.ts
import PgBoss from "pg-boss";

const boss = new PgBoss(process.env.DATABASE_URL!);
await boss.start();

// In your payment handler — inside a transaction
await db.transaction(async (tx) => {
  await tx.update(orders).set({ status: "paid" }).where(eq(orders.id, orderId));

  // Enqueue using the same Postgres connection — atomic with the DB write.
  // Pass the raw pg client from the active transaction, not the ORM wrapper
  // (e.g. tx.client with node-postgres, or the equivalent your adapter exposes).
  await boss.sendOnce(
    "send-confirmation",
    { orderId },
    { key: `confirmation-${orderId}` }, // Deduplication key
    tx.client // raw pg client from the active transaction
  );
});

If the transaction rolls back, the job is also rolled back. The email cannot be sent for an order that does not exist in the database. This is the killer feature of Postgres-backed queues.

Setting up a worker in pg-boss:

await boss.work<{ orderId: string }>(
  "send-confirmation",
  { teamSize: 2, teamConcurrency: 2 },
  async (job) => {
    const { orderId } = job.data;
    await sendConfirmationEmail(orderId);
  }
);

What pg-boss does well:

Transactional job enqueue — the main reason to choose it
No additional infrastructure if you already have Postgres
At-least-once delivery with configurable retry policy
Job deduplication with sendOnce and a key
Scheduled and recurring jobs with cron syntax
Reasonable throughput for most web applications

What to watch out for:

Postgres is not Redis. Job pickup latency is in the tens of milliseconds, not sub-millisecond. For typical web application workloads — order processing, webhook handling, scheduled emails — pg-boss handles volume comfortably; for sustained high-throughput pipelines, BullMQ is the safer choice. The UI tooling is sparse compared to BullMQ. If your team needs dashboards and visibility, you will need to build something yourself or query the pgboss tables directly.

Also: pg-boss adds tables to your Postgres instance. The schema is well-isolated (its own schema), but it is another thing in your database. Migrations and schema management become part of your normal deployment process.

Dead Letter Queues, Retries, and Deduplication

These three features separate a reliable job queue from a fragile one.

Retries with backoff — both BullMQ and pg-boss support exponential backoff. The key thing to configure is the maximum number of attempts and the delay policy:

// BullMQ
{ attempts: 5, backoff: { type: "exponential", delay: 2000 } }
// Attempt delays: 2s, 4s, 8s, 16s, 32s

// pg-boss
boss.send("my-job", data, { retryLimit: 4, retryDelay: 2, retryBackoff: true });
// retryDelay is in seconds; retryBackoff enables exponential scaling

Do not set retries to unlimited. A job that fails 50 times is either broken or pointing at a broken dependency — you want it to stop and alert you.

Dead letter queues — in BullMQ, jobs that exhaust all retries move to the failed state. They stay there (if removeOnFail: false) and you can inspect them via Bull Board or query Redis directly. In pg-boss, failed jobs stay in the job table with a failed state and a output column containing the error.

In both cases: set up an alert on failed jobs. In production I use a worker.on('failed') listener that sends a Telegram message with the job ID and error. Five minutes of debugging a failed job is much better than discovering it three days later when a customer complains.

Deduplication — BullMQ uses jobId for deduplication within active/waiting jobs. pg-boss uses sendOnce with a user-defined key. The important distinction: BullMQ's jobId deduplication expires once the job completes. pg-boss's sendOnce key deduplicates within a configurable retention window. For most webhook-driven workflows, database-level idempotency on top of queue-level deduplication is the right approach regardless of which queue you use.

Monitoring: What Actually Matters

Queue depth, failed job count, and job processing duration are the three numbers that tell you whether your background processing is healthy.

// BullMQ — get queue metrics
const counts = await orderQueue.getJobCounts("waiting", "active", "completed", "failed", "delayed");

// { waiting: 0, active: 2, completed: 1847, failed: 3, delayed: 0 }

For pg-boss:

-- Check queue state directly in Postgres
SELECT name, state, count(*)
FROM pgboss.job
WHERE createdon > NOW() - INTERVAL '24 hours'
GROUP BY name, state
ORDER BY name, state;

The metric that surprises people most is job age — how long a job sits in the waiting state before a worker picks it up. If this grows, you either have too few workers or your workers are blocked on slow downstream calls. Increase concurrency in your worker config or add more worker instances.

I track these metrics with a simple Prometheus exporter that scrapes BullMQ counts every 30 seconds and pushes them to Grafana. The setup is a hundred lines of code and has caught two incidents before they became customer-visible problems.

Serverless Alternatives

If you are on Vercel or another serverless platform, long-running worker processes are not an option. Two tools worth knowing:

Vercel Cron — HTTP endpoints called on a schedule. Simple, integrated, works with the App Router. Fine for Level 1 scheduled tasks. Not suitable for event-driven work or complex retry logic.

Inngest — event-driven background functions with retries, delays, and step functions. Designed for serverless. The DX is excellent; the trade-off is a third-party dependency and pricing that scales with invocations.

I do not use either in my current production stack because I run Node.js on VPS infrastructure with Docker, where long-running workers are straightforward. But if your deployment target is serverless and you need more than basic cron, Inngest is the most production-ready option I have evaluated.

The Decision Matrix

Situation	Recommendation
Single instance, scheduled tasks, low stakes	`node-cron` or OS cron
Multi-instance, scheduled tasks	BullMQ repeatable jobs (replaces cron)
Event-driven work, already using Postgres, want ACID enqueue	pg-boss
Event-driven work, already using Redis, need rich UI	BullMQ
High throughput (thousands of jobs/minute)	BullMQ
Serverless deployment	Inngest or Vercel Cron
Complex job graphs (fan-out, dependencies)	BullMQ FlowProducer

The pg-boss vs BullMQ decision usually comes down to whether you already have Redis in your stack. If you do — BullMQ is the obvious choice. If you do not, and your throughput is under a few hundred jobs per minute, pg-boss lets you skip a Redis dependency entirely without meaningful trade-offs for typical web application workloads.

On vatnode, I chose BullMQ because Redis was already there for rate limiting and caching. Adding BullMQ cost zero new infrastructure. On a greenfield project without Redis, I would evaluate pg-boss seriously.

What This Looks Like in Production

On vatnode, BullMQ workers handle subscription lifecycle jobs triggered by Stripe webhooks — plan activations, usage resets, cancellation workflows. Worker concurrency is set to 5, which is enough to handle burst traffic during billing cycles without saturating the Postgres connection pool. Failed jobs go to the failed state and trigger a Telegram alert; I have had fewer than 10 permanent failures in the past six months, all due to temporary Stripe API unavailability.

On pikkuna.fi, the order processing chain runs through a BullMQ worker that calls Zoho CRM, PostNord, Netvisor, and Mailgun in sequence — a pattern I also apply to any API integration work where third-party calls need retry logic and visibility. The full chain — from Stripe webhook to sent confirmation email — completes in under 2 minutes. Before this architecture, intermittent failures in one integration would silently break the rest of the chain with no visibility. Now each step is retried independently and failed steps appear in the monitoring dashboard.

items={[
{
q: "BullMQ vs pg-boss — which should I choose?",
a: "The decision usually comes down to your existing stack. If you already have Redis for rate limiting or caching, BullMQ is the obvious choice — zero new infrastructure, rich UI via Bull Board, high throughput. If you have Postgres but no Redis, pg-boss lets you skip a Redis dependency entirely with reasonable throughput for typical web application workloads. The killer feature of pg-boss is transactional job enqueue: the job and your DB write happen atomically.",
},
{
q: "How does BullMQ handle failed jobs?",
a: "Failed jobs move to the 'failed' state after exhausting all retry attempts. If you set removeOnFail: false, they stay in Redis and can be inspected via Bull Board or queried directly. Set up a worker.on('failed') listener that alerts your team — Telegram, Slack, or PagerDuty. A job that fails 50 times is either broken or pointing at a broken dependency: cap your retry attempts and alert rather than looping indefinitely.",
},
{
q: "Can I use BullMQ or pg-boss on Vercel?",
a: "No — both require long-running worker processes, which serverless functions do not support. On Vercel, use Vercel Cron for simple scheduled tasks or Inngest for event-driven background work with retries and step functions. Inngest is the most production-ready serverless queue option I have evaluated.",
},
{
q: "How do I prevent duplicate jobs from Stripe webhook retries?",
a: "In BullMQ, set a deterministic jobId (e.g. 'order-' + orderId) — if a job with that ID is already waiting or active, the new enqueue is ignored. In pg-boss, use sendOnce() with a deduplication key. Both approaches cover active-window deduplication. For completed jobs, add a database-level idempotency check: query for an existing processed record before doing any work.",
},
{
q: "How do I run BullMQ workers alongside a Next.js app?",
a: "In a standalone Next.js deployment, workers need a separate process. In my vatnode.dev Turborepo monorepo, there is a dedicated apps/worker package that runs in the same Docker container alongside the Next.js app. The worker process starts independently and shares the Redis connection. For simple setups, you can also start workers from a separate entry point file invoked with node.",
},
]}
/>

If you are building a SaaS or e-commerce platform and running into the limits of synchronous request handling or simple cron jobs, you will hit exactly these decisions. I have implemented BullMQ-based pipelines across several production systems and can help you design an architecture that matches your actual throughput and reliability requirements.

Background jobs are not about moving work out of the request. They are about making failure visible and recoverable.

If you need a senior developer who can own background job infrastructure end-to-end — get in touch. I am available for freelance projects and long-term engagements.

Related:

Stripe Webhooks Done Right: Production Architecture — how BullMQ fits into the webhook processing pipeline
Vatnode VAT Validation SaaS — BullMQ workers in production
Pikkuna E-commerce Platform — async order processing chain

External documentation:

DEV Community: Iurii Rogulia

AI Coding for Senior Developers: What to Delegate (and Not)

Where I Delegate Without Hesitation

Where I Use AI But Read Every Line

Where I Don't Delegate

Signals That Make Me Stop

What I Do Differently Now

What Hasn't Changed

The Honest Assessment

PDF Integrity Report: May 2026

The Shape of the Verdicts

Signals That Moved

Incremental Updates: Almost Without Exception

Representative Cases

Document Origin

Digital Signatures

Algorithm Development

The Software Ecosystem

PDF Version Landscape

Summary

Why AI Coding Widens the Senior–Junior Developer Gap

Wind Does Not Have a Direction

What Happens Without an Architect in the Loop

What Happens When the Senior Is in the Loop

A Concrete Illustration

The Illusion of Equal Access

What AI Actually Changes for Senior Developers

What This Means for Businesses Hiring Right Now

The Honest Summary

Bank Statement Fraud in Lending: What Gets Altered and How to Catch It

What Borrowers Actually Change

Why Visual Review Fails

What Structural Forensics Actually Catches

What an API Response Looks Like

The inconclusive Signal in Lending Context

Where It Fits in Your Underwriting Stack

Integrating the Check at Application Intake

False Positives and Calibration

What This Does Not Catch

Frequently Asked Questions

Vibe Coding Problems: What AI-Generated Code Gets Wrong

Five Overlapping Ways to Do the Same Thing

Tests That Assert the Wrong Thing

Everything in package.json

Security Patterns Copied from Tutorials

The Schema That Disagrees With the Code

A README That Describes a Different Product

Why This Happens

What Cleanup Actually Looks Like

PDF Digital Signature Chain: Detect PDFs Signed Then Modified

What a PDF digital signature actually covers

How to detect a PDF signed then modified: two patterns at certain confidence

Pattern 1: Modifications after digital signature

Pattern 2: Signature removed

Why these are certain, not high

Integrating PDF digital signature chain fraud detection into a contract workflow

Testing without real contracts

How this complements DocuSign and Adobe Sign validation

Limitations

Who should use this check

Health Check Endpoint in Node.js: Liveness vs Readiness

Liveness vs Readiness vs Startup

What to Check

The Response Format

Implementing It in Hono

HTTP Status Codes: Why 200 for Degraded

The Timeout Circuit Breaker

Docker and Kubernetes Configuration

Caching the Result and Managing Load

What Not to Expose Publicly

Graceful Degradation and Not Restarting Too Fast

What Good Looks Like

PDF Fraud Detection in Loan Origination

The document fraud surface in loan origination

Why your LOS does not catch this

What the structural signals actually look like

What inconclusive means in a mortgage context

Integration at document intake

What the underwriter sees

Compliance: TRID, HMDA, and adverse action documentation

The `inconclusive` Signal in Lending Context

Everything in `package.json`

How to detect a PDF signed then modified: two patterns at `certain` confidence

Why these are `certain`, not `high`

What `inconclusive` means in a mortgage context

Transactions and `FOR UPDATE`: Preventing Race Conditions