DEV Community: 137Foundry

Why Free-Tier SaaS Signups Are an Engineering Risk, Not Just an IT One

137Foundry — Sat, 22 Aug 2026 11:33:07 +0000

Shadow IT conversations tend to default to marketing tools and sales automation, the kind of thing a non-technical team signs up for with a work email and a credit card. That framing misses a meaningful chunk of the actual risk. Engineers create shadow IT too, and the version engineering teams produce often carries more direct access to production systems than anything marketing ever touches.

This isn't a story about developers being careless. It's a story about how the exact instincts that make good engineers effective, moving fast, trying tools before committing, working around a slow internal process, produce a specific and underappreciated category of risk.

The Shape of Engineering-Originated Shadow IT

It rarely looks like a rogue application running in production. It looks like a personal API key generated for a debugging session that never got revoked. It looks like a staging environment spun up on a free-tier cloud account to test something quickly, still running eight months later with a stale copy of production-adjacent data. It looks like a browser extension that reads clipboard content, installed to speed up a repetitive task, now sitting in a browser that also has access to internal admin panels.

None of these are dramatic on their own. Collectively, across a team of a dozen engineers over a couple of years, they add up to a meaningful, mostly invisible attack surface that nobody signed off on and nobody is actively monitoring.

Personal API Keys Are the Most Common Pattern

Almost every engineer has, at some point, generated a personal API key for a third-party service to unblock themselves quickly, intending to formalize it later. Later rarely comes. The key stays tied to a personal account, outside any centralized credential rotation policy, often with broader scopes than the actual task required because requesting a narrower scope took an extra five minutes nobody had at the time.

The fix isn't telling engineers to stop doing this, since the underlying instinct, unblock yourself now, is exactly what makes a good engineer productive. The fix is making the sanctioned path just as fast. A self-serve credential request flow with automatic expiration and scoped permissions removes the reason to reach for a personal key in the first place. GitHub and similar platforms have pushed hard on short-lived, scoped tokens over long-lived personal access tokens for exactly this reason, and the pattern is worth copying internally even for services that don't enforce it themselves.

Forgotten Staging Environments Are a Slow-Motion Liability

A staging or demo environment spun up on a personal cloud account for a quick proof of concept tends to outlive its original purpose by months or years. It was fast to create and nobody owns decommissioning it, so it just sits there, often running an outdated version of whatever software it was testing, sometimes still connected to a data source nobody remembers granting access to.

These environments are attractive targets precisely because they're unmonitored. A production system gets patched, logged, and reviewed. A forgotten staging box on someone's personal account gets none of that, while potentially holding real credentials, real API connections, or a snapshot of real data pulled in for testing convenience and never scrubbed.

Browser Extensions Deserve More Scrutiny Than They Get

Developer-focused browser extensions, JSON formatters, API testing helpers, productivity tools, request broad permissions almost by default, and engineers install them freely because they genuinely help. The risk isn't that any specific popular extension is malicious. It's that browser extension permissions are usually all-or-nothing, and a tool installed for one narrow purpose often ends up with read access to every page the browser visits, internal admin dashboards included.

A reasonable middle ground is maintaining an approved extension list for anything touching a browser that also has access to production tooling, reviewed periodically rather than locked down entirely. OWASP maintains general guidance on browser extension security considerations that's useful background for building that review process, even though the guidance predates the current wave of AI-assisted coding extensions specifically.

Why This Gets Missed in Standard Shadow IT Audits

A typical shadow IT discovery process leans on expense reports and SSO logs, both of which are tuned to catch paid, business-function tools. Free-tier developer tools, personal cloud accounts, and browser extensions mostly fall outside both of those data sources. No corporate card gets charged, and a lot of developer tooling never touches the company's SSO provider at all.

Catching this category requires a different discovery method: a direct, specific conversation with the engineering team about personal API keys, forgotten test environments, and browser extension usage, framed the same way any good shadow IT amnesty period should be framed, no penalty, just visibility. Engineers are usually candid about this once they understand the goal is inventory, not blame.

What a Reasonable Policy Actually Looks Like

The instinct to lock everything down is understandable and almost always counterproductive. A policy that blocks all free-tier signups and personal API keys outright just pushes the same behavior further out of sight, exactly the dynamic that makes shadow IT worse instead of better. A workable policy looks more like this: fast, self-serve paths for the common cases (a scoped API key, a short-lived test environment with automatic teardown), paired with a periodic, blame-free check-in on what's actually running.

Automatic expiration does more work here than any policy document. A test environment that tears itself down after fourteen days of inactivity can't become a two-year-old forgotten liability, no matter how busy the team gets. Cloudflare and most major cloud providers support this kind of lifecycle policy natively; the barrier to using it is almost always that nobody set the default, not that the capability doesn't exist.

Connecting This Back to the Broader Shadow IT Problem

The marketing-tool version of shadow IT and the engineering-tool version of shadow IT are usually treated as separate problems by separate teams, which is part of why the engineering side gets less attention. They're the same underlying problem: a slow or invisible approved path pushes people toward a faster unofficial one, and the unofficial one accumulates risk nobody is tracking.

The engineers at 137Foundry have seen this pattern often enough in client codebases to treat it as a standard part of any technical audit, not a special case. A broader look at discovery methods, risk categorization, and governance that applies across both the business and engineering sides of this problem is available in 137Foundry's guide on identifying and reining in shadow IT.

A Simple Checklist for the Next Team Retro

If a full audit feels like too much to take on right now, a short list of questions raised at the next team retro will surface most of the same risk far more cheaply. Ask whether anyone is using a personal API key for a service the team relies on regularly. Ask whether there's a test or staging environment nobody has touched in the last month. Ask which browser extensions the team relies on for daily development work, and whether anyone has actually looked at what permissions those extensions request.

None of these questions require a security specialist to ask, and none of them take more than a few minutes to answer honestly in a retro setting where the tone is already collaborative rather than evaluative. The value isn't in getting a perfect inventory from one conversation. It's in normalizing the topic enough that engineers mention these things as they come up, rather than only when someone finally runs a formal audit two years later and finds a surprising amount of forgotten infrastructure still technically live.

Why This Matters More As Teams Adopt AI Coding Tools

The rise of AI coding assistants has added a new wrinkle to this problem. Many of these tools request broad codebase access, sometimes including the ability to read environment variables or connect to external APIs on the developer's behalf, and adoption often happens organically, one engineer trying a tool before the rest of the team even knows it exists. That's the exact shadow IT pattern described throughout this piece, just with a newer category of tool.

The same principles apply: fast, sanctioned evaluation paths beat outright bans, scoped and short-lived credentials beat broad standing access, and periodic blame-free check-ins catch what expense reports and SSO logs miss. Treating AI coding tool adoption as a special, unprecedented case tends to produce worse outcomes than just applying the shadow IT playbook that already exists for everything else engineers bring in on their own initiative.

The Bottom Line for Engineering Leads

If your last shadow IT conversation focused entirely on marketing and sales tooling, there's a real chance a meaningful category of risk, personal API keys, forgotten test environments, unreviewed browser extensions, never came up at all. It's worth a direct, low-stakes conversation with the engineering team specifically, separate from whatever process handles the rest of the company, because the discovery methods that work for one side mostly miss the other.

How to Audit SSO Logs to Find Unsanctioned Apps

137Foundry — Sat, 22 Aug 2026 11:33:06 +0000

If your company has a single sign-on provider, there's a good chance someone on the security or IT team assumes the SSO dashboard is a reasonably complete picture of what employees use. It almost never is, and the gap between what SSO shows and what's actually running is one of the most useful places to start a shadow IT discovery effort, because the data is already sitting there waiting to be queried.

This is a practical walkthrough for engineers who've been asked to run that audit, or who just want a faster way to answer "what's actually connected to our identity provider" than clicking through an admin console one app at a time.

Why SSO Logs Are a Better Starting Point Than a Survey

Surveys depend on people remembering and being willing to disclose every tool they use. SSO logs depend on nothing except the tool having been connected through the identity provider at least once. That makes the data far more reliable for anything that did go through SSO, even informally, even if the person who set it up never told anyone.

The catch is coverage. Anything authenticated with a personal email address, or with local credentials that never touched SSO at all, won't show up in this data no matter how carefully you query it. Treat an SSO log audit as one input among several, not a complete inventory on its own.

Pulling the Raw Data

Most identity providers expose an events or audit log API that includes application authentication events, not just user logins. Okta and similar platforms typically let you query these events over a rolling window, commonly ninety days by default, sometimes longer depending on your plan tier. Pull the full window rather than relying on the dashboard's summary view. Summary views often deduplicate or roll up in ways that hide low-frequency apps, which are frequently the exact ones nobody remembers signing up for.

GET /api/v1/logs?since=2026-05-01T00:00:00.000Z&filter=eventType eq "user.authentication.sso"

The exact query syntax varies by provider, but the shape is consistent: filter authentication events by type, page through results, and collect the target application for each event alongside the authenticating user.

Building a Deduplicated App List

Once you have raw events, group by target application and count distinct users and total events per app. This single table answers most of the questions that matter: which apps have broad adoption, which have a single user (often a strong shadow IT signal on its own), and which haven't been touched in months but are still technically connected.

from collections import defaultdict

app_users = defaultdict(set)
app_events = defaultdict(int)

for event in sso_events:
    app = event["target_app"]
    app_users[app].add(event["user_id"])
    app_events[app] += 1

for app, users in sorted(app_users.items(), key=lambda x: -len(x[1])):
    print(f"{app}: {len(users)} users, {app_events[app]} events")

Sort by user count ascending instead of descending for the discovery pass specifically. The high-adoption apps at the top of a descending sort are almost always already known and approved. The single-user, low-event apps at the bottom are where unsanctioned tools tend to surface.

Cross-Referencing Against Your Known-Good List

The output of the previous step is a raw app list, not a verdict. Cross-reference it against whatever your company already treats as the approved tool list, however informal that list currently is. Anything present in the SSO data but absent from the approved list goes into a review queue, not an automatic ban queue. Some of these will turn out to be legitimate tools that were approved through a channel that never made it onto the master list.

approved = load_approved_app_list()
review_queue = [app for app in app_users if app not in approved]

This is also the point where it's worth checking whether an app on the review queue supports SSO enforcement, not just SSO login. Some apps allow SSO as an optional login method while still permitting a separate password-based login path, which means the SSO log audit will systematically undercount usage for that specific app.

Handling Apps That Don't Use SSO at All

SSO log analysis has a structural blind spot: it can only see what went through SSO. For a fuller picture, pair this method with an expense report scan and a short, blame-free survey. Between the three, you'll catch nearly everything: SSO logs catch anything connected through the identity provider, expense reports catch recurring paid subscriptions regardless of login method, and the survey catches free tools that show up in neither data source.

CISA has published broader guidance on asset discovery methodology that's worth reading if you're building this into a recurring process rather than a one-time pass, and the OWASP project index covers authentication and session handling considerations relevant to any tool you're evaluating for SSO enforcement going forward.

Turning a One-Time Query Into a Recurring Job

A single SSO log audit is useful, but the value compounds when it's automated and run on a schedule. A weekly or biweekly scheduled job that diffs the current app list against the previous run's list will flag new apps almost as soon as they appear, long before they've accumulated enough usage to be an entrenched dependency that's painful to migrate away from.

current_apps = set(app_users.keys())
new_apps = current_apps - previous_run_apps
if new_apps:
    notify_security_channel(new_apps)

This kind of lightweight automation turns shadow IT discovery from an annual fire drill into an ambient part of how the security team operates, which is a much cheaper way to run it than a full audit every twelve months.

Common Pitfalls When Running This Analysis

A few mistakes show up repeatedly when engineers run this kind of audit for the first time. The most common is trusting the dashboard's built-in "unique apps" count instead of pulling raw events. Most admin consoles roll up subdomains or slightly different app registrations under a single umbrella entry, which quietly hides the exact granularity you're trying to surface. Always work from the raw event stream, even though it's more tedious to parse.

A second pitfall is treating a low event count as low risk automatically. A tool authenticated only twice in ninety days might be dormant and harmless, or it might be a one-time data export that already happened and can't be undone by revoking access now. Event frequency tells you about usage patterns, not about the risk already incurred. Check what data an app can access before assuming a low-frequency connection is low priority.

A third pitfall is forgetting service accounts and API integrations in the audit scope. Not every SSO event maps to a human clicking a login button. Some map to a scheduled integration authenticating on a service account's behalf, and those often carry broader data access than any individual employee login would. Filter your query results by event subtype early, or you'll spend time investigating machine-to-machine connections that need a different kind of review than a marketing tool a person signed up for.

Building a Dashboard Instead of Running Ad Hoc Queries

Once the audit script proves useful, it's worth the extra afternoon to wire it into a lightweight internal dashboard rather than re-running the same query manually every month. A simple table showing app name, user count, event count, and days since last first-seen, refreshed weekly, gives whoever owns tool governance a live view instead of a point-in-time snapshot that goes stale the moment it's generated.

This doesn't need to be elaborate. A scheduled job writing results to a shared spreadsheet or a small internal database, paired with a Slack notification when a genuinely new app appears, covers most of the value a dedicated security tool would provide, at a fraction of the engineering cost. The goal is visibility that persists, not a one-time report that gets read once and then forgotten in a shared drive.

What to Do Once You Have the List

Finding unsanctioned apps is the easy part. Deciding what happens to each one, and doing it in a way that doesn't push the next round of tool adoption further underground, is the harder part and the one that determines whether this exercise actually reduces risk over time. 137Foundry has written a longer breakdown of the discovery-to-governance process, including how to run the decision phase without turning it into a witch hunt, in its guide on identifying and reining in shadow IT.

The short version: SSO log analysis gets you a fast, reliable, low-effort starting point. What you build on top of it, the review cadence, the ownership, the faster approval path for legitimate requests, is what actually keeps the problem from coming back six months later.

None of this requires a security team of any particular size. A single engineer with API access to the identity provider and a couple of hours can produce a more accurate picture than most companies have ever had, and turning that one-time script into a scheduled job is a small additional step that pays for itself the first time it catches a new unsanctioned app before it becomes an entrenched dependency nobody wants to migrate away from.

Stale Data Is a UX Bug, Not Just a Data Engineering Problem

137Foundry — Fri, 21 Aug 2026 11:31:42 +0000

When a button doesn't respond to a click, that's a UX bug and everyone treats it like one. When a dashboard shows a number that's quietly six hours out of date, that same team often files it under "data engineering," not "user experience," even though the user's actual experience is identical: they trusted something the interface presented as current, and it wasn't.

That framing gap matters more than it seems like it should, because it decides who owns the fix and how urgently it gets treated.

The Interface Doesn't Communicate Staleness

Most dashboards render the same way whether the underlying data is thirty seconds old or thirty hours old. There's no visual difference between a fresh number and a stale one unless someone deliberately built one in. A user has no way to distinguish trustworthy data from questionable data just by looking at it, so they default to trusting all of it equally.

This is functionally the same failure mode as a UI that doesn't distinguish a disabled button from an enabled one. The interface is lying by omission, not because anyone intended to mislead, but because staleness was never treated as a state worth designing for.

Silent Failures Are the Expensive Kind

A pipeline that crashes loudly produces an obvious incident: an error, a page, a fix. A pipeline that keeps running but falls behind produces something worse, a system that looks healthy from every angle a status page checks while quietly delivering wrong answers to real decisions. Data quality research consistently treats these silent degradations as more costly than loud failures precisely because nobody's watching for them the same way.

The cost isn't hypothetical. A stale inventory count that says "in stock" for something that sold out three hours ago costs a sale and a customer's trust in a way that's directly measurable, even though the underlying cause never triggered a single alert.

Treat "As Of" Timestamps Like Any Other UI State

Good frontend practice already has a pattern for this: loading states, empty states, error states, each gets deliberate design attention because an undesigned state is still a state the user experiences, just a worse one. Web performance and UX guidance treats perceived state honesty as a core usability concern for exactly this reason, an interface that hides uncertainty from the user isn't simpler, it's just less honest.

Staleness deserves the same treatment. A visible "updated 4 hours ago" label costs almost nothing to add and changes the entire trust calculus for anyone reading the number next to it. It turns an invisible assumption into an explicit, checkable fact.

Instrument It Like Any Other User-Facing Metric

The reason staleness rarely gets this treatment is that it's rarely measured as a first-class signal. Teams instrument job success and job duration religiously, but the age of the data a job actually produced is often left as an afterthought, something you'd have to reconstruct from logs after the fact rather than something the system reports directly.

OpenTelemetry tracing can carry this signal end to end, tagging each stage of a pipeline with timing data that ultimately answers the question a user is implicitly asking every time they look at a chart: how current is this, really. Once that answer exists as a real, queryable number, exposing it in the UI is a small step.

A Small Design Pattern That Fixes Most of This

The fix doesn't require a redesign. A small, consistently placed freshness indicator, something as simple as "as of 2:14 PM" rendered next to the metric it describes, does most of the work. The key detail is that it has to be unavoidable, not tucked into a tooltip or a settings page nobody opens. If a user can look at a number without seeing how old it is, the indicator isn't doing its job yet.

Color coding helps here the same way it helps with any other status communication: green for within the expected freshness window, amber for approaching it, red for past it. This mirrors exactly how teams already communicate uptime or error budgets to non-technical stakeholders, and there's no reason data freshness should get less thoughtful treatment than those other reliability signals get by default.

A Comparison That Makes the Point Clearer

Think about how differently teams treat a slow API response versus stale data. A slow response gets a loading spinner, a skeleton screen, maybe a timeout with a retry button, because the industry collectively agreed years ago that hiding latency from the user is bad practice. Stale data gets none of that same treatment by default, even though it's arguably the more dangerous failure mode, since a slow response is visibly slow while stale data looks exactly like fresh data until someone checks.

That inconsistency isn't really about technical difficulty. Building a loading spinner and building a freshness badge require roughly the same amount of frontend work. It's about which failure mode the industry has collectively decided is worth designing for by default, and staleness simply hasn't gotten the same attention loading states have, despite causing comparable or worse damage when it goes unaddressed.

Why Engineers Resist Building This

The honest reason this pattern is rare isn't technical difficulty, it's that admitting a number might be stale feels like admitting the pipeline isn't good enough. That's the wrong frame. Every pipeline has some latency, and pretending otherwise doesn't make the latency disappear, it just moves the discovery of it from a planned design decision to an unplanned support ticket. A team that's honest about freshness in the interface looks more trustworthy over time than one that implicitly claims everything is instant and occasionally gets caught being wrong.

What This Looks Like Once It's Actually Adopted

Teams that make this shift tend to notice a change in how support tickets read. Instead of "the dashboard is showing wrong numbers," which requires an engineer to first figure out whether the data pipeline is broken or the number is just old, tickets start reading "the dashboard says it's twenty minutes stale, is that expected right now," which is a question support can often answer on their own by checking the same status the freshness indicator is drawing from. That shift alone removes a meaningful chunk of unnecessary engineering escalations, because the ambiguity that used to require investigation is now visible on the screen itself.

It also changes how engineers experience their own alerts. A freshness breach that's already visible to users, because the same indicator that alerted engineering is also showing a warning state on the dashboard, feels different to respond to than a silent pipeline failure nobody outside the team knows about yet. The pressure is honest instead of hidden, and honest pressure is generally easier to work under than the anxiety of racing to fix something before anyone notices it was ever broken.

Where Ownership Should Actually Sit

None of this means data engineering shouldn't fix the underlying lag. It means product and design should co-own how staleness gets communicated to the person looking at the screen, the same way they'd co-own how a loading spinner or an error banner gets communicated. Splitting the problem that way gets both the pipeline fix and the interface honesty addressed, instead of one team quietly carrying a UX problem they don't have the mandate to fix.

We go deeper on the engineering side of this, defining a real freshness target and instrumenting against it, in our guide on designing a data freshness SLA for automation pipelines. Between the interface and the pipeline, closing this gap is the kind of unglamorous, cross-team work 137Foundry treats as core to shipping dashboards people can actually trust.

How to Build a Freshness Check That Runs Before Your Dashboard Loads

137Foundry — Fri, 21 Aug 2026 11:31:40 +0000

Most dashboards render whatever data happens to be in the table, with no check on whether that data is actually current. Here's a practical pattern for adding a freshness check that runs before the dashboard renders, so users get an explicit signal instead of a silent assumption.

Step 1: Store a Last-Updated Timestamp Alongside the Data

The pipeline that populates the dashboard's tables should also write a small metadata row: which table it updated, and the timestamp of the newest record it processed. This doesn't need its own service. A single row in a lightweight metadata table, updated at the end of every successful pipeline run, is enough.

def record_freshness(table_name, newest_record_ts):
    db.execute(
        "INSERT INTO freshness_log (table_name, updated_at, newest_record_ts) "
        "VALUES (%s, NOW(), %s) "
        "ON CONFLICT (table_name) DO UPDATE SET "
        "updated_at = NOW(), newest_record_ts = EXCLUDED.newest_record_ts",
        (table_name, newest_record_ts)
    )

Step 2: Query Freshness Before the Dashboard Query Runs

On page load, query the freshness table first, before running the expensive dashboard query itself. This is a cheap, single-row lookup, so it adds negligible latency, and it gives you the information you need to decide what to show the user.

def get_freshness(table_name):
    row = db.query_one(
        "SELECT newest_record_ts FROM freshness_log WHERE table_name = %s",
        (table_name,)
    )
    return row["newest_record_ts"] if row else None

Step 3: Compare Against Your Defined SLA Window

Compute how old the data actually is, and compare it against whatever freshness window the dashboard is supposed to guarantee. This threshold should come from an explicit decision about what the data is used for, not an arbitrary round number.

from datetime import datetime, timezone

def freshness_status(newest_record_ts, sla_minutes):
    age = (datetime.now(timezone.utc) - newest_record_ts).total_seconds() / 60
    if age <= sla_minutes:
        return "fresh"
    elif age <= sla_minutes * 1.5:
        return "warning"
    return "stale"

Step 4: Surface the Result in the UI, Not Just the Logs

The whole point of this check is that a human sees the result, not just a monitoring system. Render a small, unavoidable indicator, "updated 4 minutes ago" for a fresh state, a visible warning banner for a stale one, so the user makes their own judgment call about whether to trust what they're looking at instead of assuming it by default.

Step 5: Pipe the Same Signal Into Your Alerting Stack

The same freshness check that powers the UI indicator should feed your monitoring system too, so engineering finds out about a stale pipeline the same moment a user would, or ideally before. A metrics backend like Prometheus can scrape this value on an interval and alert when it crosses your threshold, and a dashboard built in Grafana can plot it over time so a slow creep toward staleness is visible as a trend, not just a binary pass or fail.

Step 6: Test It With a Deliberately Stale Fixture

Write a test that seeds the freshness table with an old timestamp and confirms the dashboard actually shows the warning state, not just that the happy path works. Python's standard library documentation covers the unittest.mock patterns useful for freezing time in a test like this, since freshness checks are inherently time-dependent and easy to get wrong without deliberately testing the boundary.

Step 7: Handle the Case Where Freshness Data Itself Is Missing

Don't assume the freshness table will always have a row. A brand-new table, a renamed pipeline, or a one-off backfill job that skips the metadata write will all leave get_freshness returning nothing. Decide explicitly what the dashboard shows in that case, ideally an honest "freshness unknown" state rather than silently falling back to treating missing data as fresh. Failing open on a freshness check defeats the entire purpose of building one.

def render_freshness_badge(table_name, sla_minutes):
    ts = get_freshness(table_name)
    if ts is None:
        return "freshness unknown"
    return freshness_status(ts, sla_minutes)

Step 8: Account for Multiple Tables Feeding One Dashboard

Most real dashboards aren't backed by a single table. A page showing order volume alongside inventory levels is really two freshness stories layered on top of each other, and they can diverge, one feed running perfectly on time while the other has quietly stalled. Track freshness per source table, not once per dashboard, and roll them up so the UI shows the staleness of whichever underlying feed is worst, rather than an average that hides the one that's actually broken.

def dashboard_freshness_status(table_names, sla_minutes):
    statuses = [freshness_status(get_freshness(t), sla_minutes) for t in table_names]
    if "stale" in statuses:
        return "stale"
    if "warning" in statuses:
        return "warning"
    return "fresh"

Step 9: Cache the Freshness Lookup Carefully

The freshness query is cheap, but on a high-traffic dashboard it's still an extra database round trip on every page load. Cache the result for a short window, ten to thirty seconds is usually plenty, rather than caching it for as long as the SLA window itself, which would let the cached "fresh" answer outlive the actual freshness it's supposed to represent. Getting this cache duration wrong in the generous direction quietly reintroduces exactly the problem this whole feature exists to solve.

from functools import lru_cache
import time

_cache = {}

def get_freshness_cached(table_name, ttl_seconds=15):
    now = time.time()
    cached = _cache.get(table_name)
    if cached and now - cached[1] < ttl_seconds:
        return cached[0]
    value = get_freshness(table_name)
    _cache[table_name] = (value, now)
    return value

Step 10: Roll It Out to One Dashboard Before Templating It

Resist the urge to build a generic freshness component for every dashboard in the company on day one. Ship it on the single dashboard that's caused the most confusion, watch how the warning and stale states actually get used in practice, and adjust the thresholds based on real feedback before extracting it into a shared component. A freshness indicator that's technically correct but tuned wrong, warning constantly on a dashboard where five-minute staleness is genuinely fine, teaches users to ignore it, which defeats the point just as thoroughly as not building it at all.

A Note on Framework-Specific Implementation

The examples above are deliberately framework-agnostic pseudocode. In a real app, the freshness lookup fits naturally into whatever data-loading pattern the frontend already uses, a server-side loader function, a REST endpoint the dashboard polls, or a resolver in a GraphQL schema. The specific plumbing matters less than the underlying discipline: compute freshness from real data, not from job status, cache it briefly rather than for the full SLA window, and fail toward an honest "unknown" state rather than a false "fresh" one when the freshness table itself is empty or unreachable.

What This Buys You

Once this check exists, "is this data current" stops being a question someone has to ask in a channel and becomes something the dashboard answers for itself, every time it loads. It's a small amount of engineering work that removes a recurring category of trust problem. If you want the fuller picture on setting the SLA number this check is actually measured against, and how to alert before you breach it rather than after, we cover that in our guide on designing a data freshness SLA for automation pipelines, the kind of end-to-end reliability work 137Foundry's engineering team builds for clients regularly.

How to Add Request Deduplication to a Node.js API in an Afternoon

137Foundry — Thu, 20 Aug 2026 11:20:19 +0000

You don't need a redesign to make an existing Node.js API safe against duplicate requests. If you've got a database with unique constraints and a few hours, you can add a working idempotency layer to one endpoint today and roll it out to the rest as time allows. Here's the whole process, step by step, using nothing more exotic than Express and whatever relational database you're already running.

Step 1: Decide Which Endpoints Actually Need This

Not every endpoint is worth protecting. A GET request is naturally safe to repeat. A POST that only reads data or triggers something harmless to run twice doesn't need dedup logic either. The endpoints that matter are the ones with real side effects: creating an order, charging a card, sending a notification, provisioning a resource. Start with the one endpoint where a duplicate would actually cause a visible problem, ideally your highest-traffic mutating endpoint, rather than trying to cover everything on day one.

Step 2: Add the Dedup Table

A single table handles this for most APIs:

CREATE TABLE idempotency_keys (
  key TEXT PRIMARY KEY,
  request_hash TEXT NOT NULL,
  response_status INT,
  response_body JSONB,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

The key column is the client-generated idempotency key, and its primary key constraint is what makes the whole thing race-safe. request_hash lets you detect when a client reuses a key with a different payload, which should be treated as a conflict rather than a legitimate retry. response_status and response_body let you replay the exact original response on a repeat request instead of just returning a generic "already processed" message.

Step 3: Write the Middleware

const crypto = require('crypto');

async function idempotencyMiddleware(req, res, next) {
  const key = req.header('Idempotency-Key');
  if (!key) return next();

  const requestHash = crypto
    .createHash('sha256')
    .update(JSON.stringify(req.body))
    .digest('hex');

  try {
    await db.query(
      'INSERT INTO idempotency_keys (key, request_hash) VALUES ($1, $2)',
      [key, requestHash]
    );
  } catch (err) {
    if (err.code === '23505') {
      const existing = await db.query(
        'SELECT request_hash, response_status, response_body FROM idempotency_keys WHERE key = $1',
        [key]
      );
      const row = existing.rows[0];

      if (row.request_hash !== requestHash) {
        return res.status(409).json({ error: 'idempotency_key_conflict' });
      }
      if (row.response_status) {
        return res.status(row.response_status).json(row.response_body);
      }
      return res.status(409).json({ error: 'request_in_progress' });
    }
    throw err;
  }

  req.idempotencyKey = key;
  next();
}

The insert happens before any real work runs. If it succeeds, this is a genuinely new request and processing continues normally. If it fails with a unique violation (23505 is PostgreSQL's constraint violation code), one of two things is true: this exact request was already made, and you should replay the stored response, or a different request reused the same key, and you should reject it with a 409. If the row exists but has no stored response yet, another request with the same key is currently mid-flight, which the client should treat as a retryable in-progress state.

Step 4: Store the Response After Processing

The middleware alone doesn't complete the pattern. After your route handler finishes, you need to write the response back to the same row:

async function finalizeIdempotency(req, status, body) {
  if (!req.idempotencyKey) return;
  await db.query(
    'UPDATE idempotency_keys SET response_status = $1, response_body = $2 WHERE key = $3',
    [status, body, req.idempotencyKey]
  );
}

Call this right before sending the response in your route handler, so the stored row reflects exactly what the client received the first time. This is the piece that makes a retry return the original result instead of just a generic acknowledgment.

Step 5: Wire It Into One Route First

app.post('/orders', idempotencyMiddleware, async (req, res) => {
  const order = await createOrder(req.body);
  await finalizeIdempotency(req, 201, order);
  res.status(201).json(order);
});

Keep the first rollout narrow. One route, tested thoroughly, gives you confidence in the pattern before you copy it across the rest of your mutating endpoints, and it lets you catch integration issues, like a client library that doesn't consistently send the header, without that mistake reaching every endpoint at once.

Step 6: Test the Race Condition, Not Just the Happy Path

A sequential test, send a request, then send the identical request again and check that the response matches, confirms the basic mechanism works. It doesn't confirm the part that actually matters under load: two requests with the same key arriving close enough together to race.

const [res1, res2] = await Promise.all([
  request(app).post('/orders').set('Idempotency-Key', 'test-key-1').send(payload),
  request(app).post('/orders').set('Idempotency-Key', 'test-key-1').send(payload),
]);

const successCount = [res1, res2].filter(r => r.status === 201).length;
expect(successCount).toBe(1);

If both requests return 201, the unique constraint isn't actually preventing concurrent processing, which usually means the insert is happening after the real work instead of before it. This is the single most valuable test in the whole implementation, because it's the one that catches the mistake that looks correct in every other test.

Optional: Adding a Fast Path With Redis

If your API handles enough traffic that a relational lookup on every request becomes a noticeable cost, a common pattern is to put a fast, in-memory check in front of the database. Redis works well here: attempt a SET key value NX (set-if-not-exists) before touching the database at all. If the Redis set succeeds, proceed to the database insert as the durable record. If it fails, you can often skip the database round trip entirely and return a cached response straight from Redis. This isn't a replacement for the database's unique constraint, since Redis alone doesn't give you the same durability guarantee across a restart, but it's a reasonable optimization once the relational check starts showing up in your latency numbers.

A Mistake This Pattern Specifically Prevents

The most common mistake teams make when they build idempotency logic without a database constraint is checking for an existing key, then doing the work, then writing the key, as three separate steps in application code. That ordering looks correct in a single-threaded test and fails the moment two requests with the same key run concurrently, which is a textbook race condition: both checks run before either write completes, and both proceed to do the real work. The insert-first ordering in Step 3 above isn't a stylistic preference. It's the specific detail that closes this exact race, because the database's constraint check and the write happen as one atomic operation instead of two separate steps your application code has to coordinate.

Step 7: Roll Out to Remaining Endpoints

Once the pattern is proven on one route, adding it to others is mostly copy and wire, since the middleware and the finalize call don't change per endpoint. The main judgment call at each new endpoint is whether it genuinely needs protection, per Step 1, rather than adding the overhead everywhere by default.

How Long This Actually Takes

The steps above are genuinely small in isolation: one migration, one middleware function, one finalize call, one route wired up, one concurrency test. Most of the real time in an afternoon rollout goes into deciding which endpoint to start with and verifying the concurrency test actually fails before your fix and passes after it, not into writing the code itself. If you're doing this for the first time, budget an hour for the schema and middleware, another hour for wiring and manual testing against one route, and the rest for the concurrency test and a second pass at whichever endpoint you pick next.

What You Haven't Solved Yet

This afternoon's work covers the core client-initiated case. It doesn't cover webhook deliveries, which typically arrive with their own event ID rather than a client-supplied key, though the same table and constraint pattern applies with a small adjustment. It also doesn't cover retention: without a cleanup job, the idempotency_keys table grows indefinitely. A scheduled job that deletes rows older than your realistic retry window, a day or two is generous for most clients, keeps lookups fast long term.

For the fuller version of this pattern, including how to scope keys correctly across different operations, handle the conflict case in more detail, and decide on a retention window, 137Foundry has a longer breakdown at How to Design an Idempotency Key Strategy So Retried API Requests Never Double-Process. If you'd rather have someone build this into an existing API rather than doing it endpoint by endpoint yourself, 137Foundry works on exactly this kind of backend reliability work. The PostgreSQL unique constraint used here is standard across most relational databases, so the same approach ports cleanly if you're on a different engine than the one in these examples.

Why Your Payment Webhook Handler Needs an Idempotency Table, Not Just a Try/Catch

137Foundry — Thu, 20 Aug 2026 11:20:18 +0000

A lot of payment webhook handlers look reasonable on the surface. They validate the signature, parse the payload, wrap the database write in a try/catch, and log an error if anything goes wrong. It works fine in every test anyone runs against it. Then, months later, a customer opens a ticket saying they were charged for the same order twice, and nobody on the team can immediately explain how, because nothing in the code looks obviously broken.

The answer is almost always the same: the handler assumed every incoming request represented a new event. Nothing in a try/catch block checks whether this exact event has already been processed. It only catches errors that happen while processing it, which is a completely different problem.

Try/Catch Solves the Wrong Failure Mode

A try/catch block protects against exceptions: a malformed payload, a database connection drop, a null field where you expected a value. Those are real failure modes worth handling, and every handler should have them. But a duplicate webhook delivery isn't an exception. It's a perfectly valid, correctly formatted request that happens to represent something you already did. The handler runs successfully both times. There's no error to catch, because nothing failed. The code just does the same real-world action twice, cleanly, with no exception anywhere in the stack trace to point you at the problem.

This is why the bug is so easy to miss in code review. Reviewers scan for missing error handling, and this handler has plenty of it. The gap isn't in what happens when something goes wrong. It's in the assumption that every request is new.

Where the Duplicate Deliveries Actually Come From

Payment processors and most other webhook senders guarantee at-least-once delivery, not exactly-once, and they're explicit about this in their documentation because it's a deliberate design tradeoff, not an oversight. If your endpoint doesn't return a success status within their timeout window, or returns any error status, the sender will redeliver the same event, sometimes more than once. That timeout window is often short, in the 10 to 30 second range, which means anything that makes your handler briefly slow, a database migration running in the background, a cold start, elevated load, increases your odds of triggering a redelivery you'll then have to handle correctly.

None of this is misbehavior on the sender's side. Redelivery on timeout is the correct, documented response to an ambiguous outcome, since the sender genuinely doesn't know whether your handler processed the event or not. The responsibility for handling that ambiguity sits with your endpoint, not with the sender's retry policy.

What an Idempotency Table Actually Looks Like

The fix is a dedicated table, separate from your business data, with the webhook's event ID as a unique-constrained column. Most payment processors and webhook senders already assign a unique ID to every event, including redeliveries of the same logical event, so you don't need to generate your own key. Before doing any real work, check whether that event ID already exists in the table. If it does, return success immediately and stop. If it doesn't, insert the ID as the first write, then do the real processing.

The order matters. Inserting the event ID first, before the business logic runs, and relying on the unique constraint to reject a second concurrent insert, closes the race condition where two near-simultaneous redeliveries both check for the ID before either has recorded it. Checking for existence, then doing the work, then recording the ID afterward, leaves a window where two redeliveries arriving close together both pass the check and both do the real work.

A Minimal Schema

CREATE TABLE processed_webhook_events (
  event_id TEXT PRIMARY KEY,
  event_type TEXT NOT NULL,
  processed_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

The primary key constraint does the heavy lifting here. An INSERT against this table either succeeds, meaning you're the first (and only) handler processing this event, or it fails with a constraint violation, meaning someone else already claimed it. Catching that specific constraint violation and treating it as "already handled, return success" rather than an application error is the one piece of exception handling that's actually relevant to this problem, as opposed to the general try/catch around the rest of the handler. This pattern works the same way on any relational database that enforces uniqueness atomically, PostgreSQL included, which is part of why it's such a portable fix across different tech stacks.

What This Actually Costs You

The objection teams sometimes raise is that this adds a new table, a new write, and a new failure surface to a handler that already works most of the time. That's true, and it's worth being honest about the tradeoff rather than pretending the fix is free. The counterpoint is that the cost is fixed and small, one row, one indexed lookup, while the cost of not having it scales with your traffic and your webhook sender's retry behavior, neither of which you control. A handler that processes a few hundred events a day might go months without a visible duplicate. One that processes tens of thousands a day, across a sender that occasionally has its own network issues, will see duplicates often enough that the missing table becomes a recurring support burden rather than a rare edge case.

Handling Slow Processing Without Missing the Timeout Window

Even with the idempotency table in place, a handler that does all its real work synchronously, updating the order, sending a confirmation email, syncing a CRM, before responding, is still vulnerable to triggering a redelivery through simple slowness. The fix is to separate the fast, synchronous part (validate the signature, check and record the event ID) from the slower work, which runs in a background job after the response has already gone out. This shrinks your acknowledgment window down to essentially one database write, which is difficult for anything short of a full outage to push past a sender's timeout.

Systems like Redis work well as the queue for that background job if you don't already have one, since the durability requirement is modest: you just need the job to survive a brief restart, not to guarantee delivery across a distributed cluster.

Testing This Properly

The test that matters here isn't "does the handler process a valid webhook correctly." It's "does the handler behave correctly when the exact same payload arrives twice, including when both arrive close enough together to race." A test that fires the same event synchronously, once, then again, checking that the second call is a no-op, covers the sequential case. A test that fires both concurrently and asserts exactly one insert succeeded covers the race condition, and it's the test most teams skip because it takes more setup than a sequential call.

Watching for the Table Itself Becoming a Bottleneck

Once the table is in place, it's worth keeping an eye on its growth and lookup performance, since it's now in the hot path of every incoming webhook. An index on the event ID column, which the primary key constraint already provides, keeps lookups fast well past the point most teams will ever reach in practice. Pruning records older than your sender's realistic redelivery window, most senders redeliver within a few days at most, keeps the table from growing indefinitely and keeps those lookups fast for years rather than months.

The Broader Pattern

This same table-plus-unique-constraint approach applies beyond payment webhooks specifically, to any endpoint where a client-controlled or sender-controlled identifier can stand in for "have I already done this." 137Foundry's engineering blog has a deeper walkthrough of the client-side version of this pattern, including how to scope idempotency keys, handle the conflict case where a key gets reused with a different payload, and design the retention window, in How to Design an Idempotency Key Strategy So Retried API Requests Never Double-Process. The webhook case is the same underlying mechanism applied to inbound events instead of outbound API calls, and once you've built it once, adding it to a second endpoint is a small amount of incremental work rather than a new design problem.

The Takeaway

A try/catch block and an idempotency table solve different problems, and a payment handler needs both. The try/catch protects against real failures during processing. The idempotency table protects against the far more common case where nothing failed at all, and the handler simply ran twice for the same event because that's exactly what at-least-once delivery, correctly implemented on the sender's side, is supposed to do under the conditions the specification allows for, as documented for systems like Apache Kafka that formalize the same delivery guarantee.

How to Add a Second-Pass AI Review Step Without Slowing Your Team Down

137Foundry — Wed, 19 Aug 2026 11:26:39 +0000

A second AI model reviewing the first model's output sounds redundant on its face, machine checking machine, but it catches a real and specific slice of problems: obvious logic errors, missing edge case handling, and inconsistencies between what a PR description claims and what the diff actually does. The objection most teams raise isn't whether it helps, it's whether adding another step to the review pipeline is worth the added latency. Done right, it barely adds any.

Step 1: Scope It to What It's Actually Good At

An AI review pass is weak at the failure mode that matters most, missing unstated context about your specific system, because it has the same blind spot to context that isn't written down as the model that generated the code in the first place. Where it's genuinely useful: catching inconsistencies a tired human reviewer skims past, flagging code that doesn't match its own PR description, and surfacing edge cases (null handling, boundary conditions, unhandled error paths) mechanically and consistently. Scoping the AI review step to exactly that narrower job, rather than treating it as a substitute for human review, is what keeps it fast and useful instead of becoming its own bottleneck.

Step 2: Run It in Parallel, Not in Sequence

The biggest latency mistake teams make is running the AI review pass as a blocking step before a human ever looks at the PR. That turns one review into two sequential reviews, doubling the wait. The better pattern: trigger the AI review pass automatically the moment a PR opens, running in parallel with (not before) a human reviewer picking it up. By the time a human opens the PR, the AI review's findings are already posted as comments, ready to be triaged alongside the human's own read, rather than adding a wait step in front of it.

Step 3: Make Findings Advisory, Not Blocking

An AI review comment that must be resolved before merge, the same way a human's blocking review comment works, creates exactly the kind of friction that erodes team buy-in fast, especially when a chunk of AI-flagged findings turn out to be false positives or genuinely low-priority. Treating AI review output as advisory, visible, easy to dismiss with a one-line reason, but not a merge gate, keeps the signal available without turning every minor false positive into a blocked PR waiting on a human to argue with a bot.

Step 4: Tune the Prompt to Your Codebase's Actual Risk Areas

A generic "review this code" prompt produces generic output. A prompt scoped to your team's actual risk areas, "flag anything touching authentication, payment processing, or database migrations for extra scrutiny; ignore purely cosmetic changes," produces a review pass that's actually useful to triage quickly, because the noise-to-signal ratio drops sharply when the review step already knows what your team cares about most. This is the same context-completeness problem covered in more depth in our guide on reviewing AI-generated code without rubber-stamping it, applied to the review tooling itself rather than just the code generation step.

Step 5: Feed It the Diff, Not Just the Final State

An AI review pass that only sees the final state of changed files misses context a diff provides directly, specifically, what changed relative to what existed before, which is often more informative than the final code alone for spotting a regression or an unintentional behavior change. Configuring the review step to receive the actual diff, with enough surrounding context to understand what was modified and why, produces meaningfully better findings than pointing it at the changed files in isolation.

Step 6: Measure False Positive Rate and Actually Adjust

Teams that adopt an AI review step and never revisit its tuning tend to see engagement drop off within a few weeks, as reviewers learn which categories of finding are usually noise and start ignoring the tool's output wholesale, including the findings that were actually useful. Periodically checking which categories of AI-flagged issue get dismissed most often, and either tightening the prompt or disabling that specific category, keeps the signal-to-noise ratio high enough that the team keeps actually reading the output instead of reflexively dismissing it.

Step 7: Give It a Different Model Than the One That Wrote the Code

Using the same model, or the same underlying provider and configuration, to both generate code and review it risks correlated blind spots, the reviewing pass may be prone to missing exactly the categories of mistake the generating pass is prone to making, since they share the same training characteristics and failure tendencies. Where practical, running the review pass with a different model than the one used for generation reduces this correlation, similar to the reasoning behind why a second human reviewer who didn't write the code catches things the author doesn't. This isn't always feasible depending on your tooling setup, but it's worth weighing when choosing which models to standardize on for which stage of the pipeline.

Step 8: Route Findings to the Right Level of Attention

Not every AI-flagged finding deserves the same visibility. A minor style inconsistency and a possible SQL injection vector shouldn't compete for the same reviewer attention at the same priority level. Configuring the review step to tag findings by rough severity, even a simple high, medium, low split, lets a reviewer triage quickly: address high-severity findings before merge, skim medium ones, and batch low-severity style notes for a later cleanup pass rather than letting them clutter the PR's primary review thread.

What This Looks Like End to End

A workflow that keeps this fast in practice: PR opens, AI review pass triggers automatically and posts findings as PR comments within a couple minutes, a human reviewer picks up the PR on their normal timeline and reads both the diff and the AI comments together, findings the human agrees with get addressed, findings that are false positives get a one-line dismissal, and the PR merges once the human approves, with the AI pass never having blocked anything. Total added latency in this setup: close to zero, since the AI review runs concurrently with, not before, the human review cycle it's meant to support.

Where This Fits in a Broader AI-Assisted Development Practice

An AI review pass isn't a substitute for the review discipline described in our main guide on this topic, it's one more input into it. The failure modes that matter most, missing unstated context, fabricated API calls that look structurally plausible, silently narrowed scope, still require a human who knows the system to catch reliably. What an AI review pass adds is a fast, cheap first filter that surfaces the mechanical issues before a human's attention gets spent on them, freeing that attention for the harder judgment calls a second AI model genuinely can't make.

137Foundry's AI automation service helps engineering teams design review pipelines like this one, tuned to an actual codebase's risk profile rather than a generic template. You can see our full range of services or read more about how we work.

For a broader technical reference on automated code review tooling generally, GitHub's documentation on pull request workflows covers the underlying automation hooks most teams build this kind of pipeline on top of, and Google's engineering practices for code review remains a solid baseline for what a review step, human or AI-assisted, should actually be optimizing for. OWASP's guidance on secure code review is a useful third reference specifically for tuning the prompt's risk-area list described in Step 4.

Why AI Coding Assistants Confidently Call Methods That Don't Exist

137Foundry — Wed, 19 Aug 2026 11:26:14 +0000

You review a PR, the code calls client.batchUpdateWithRetry(), it reads naturally, fits the surrounding code's style, and you move on. Later, the build fails, or worse, it doesn't fail because a linter or type checker didn't catch it, and someone discovers at runtime that no such method exists on that client. The AI assistant that generated it wasn't being careless. It was doing exactly what it's built to do, and that's worth understanding to know where to watch for it.

Why This Happens at All

Large language models generate code by predicting what token is statistically likely to come next, given everything that came before, including the surrounding code, the library being used, and patterns from a huge volume of training data covering how that library (or libraries like it) are typically used. For extremely common libraries with widespread, consistent usage patterns in training data, this produces remarkably accurate output most of the time. The failure shows up specifically when a library is less common, has gone through breaking API changes across versions, or when the model is filling in a gap with the most statistically plausible completion rather than a verified fact.

The result reads as confident because the model isn't distinguishing between "I've seen this exact method used this way many times" and "this is the most plausible-sounding completion given the pattern." Both produce equally fluent, equally confidently formatted output. Nothing about the code's surface signals which case you're in.

Where This Shows Up Most

A few situations raise the odds of this particular failure mode meaningfully:

Less popular or newer libraries. A library with a smaller footprint in public training data gives the model less signal to draw on, and it's more likely to blend patterns from similar-but-different libraries into a plausible-sounding but incorrect API surface.

Libraries that changed their API across major versions. If a library renamed a method, changed a parameter's type, or deprecated an option between versions, a model trained on a mix of old and new documentation and code examples can generate a call that was valid in an older version but not the one actually installed, or vice versa.

Internal or private libraries. A model has no training exposure at all to your team's internal libraries unless that context is provided directly. Any call to an internal API is either grounded in what's visible in the current codebase and prompt, or it's a plausible-sounding guess extrapolated from how a similar public library would typically work.

Configuration options and parameter names specifically. Method names are relatively easy to verify by checking whether the call even resolves. Individual parameter names and configuration keys are easier to get subtly wrong in a way that still type-checks or compiles if the language and framework are permissive about extra or misnamed options, silently ignoring a config value instead of erroring on it.

Why This Is Hard to Catch by Reading Alone

The core problem is that fabricated API calls are indistinguishable from correct ones by tone or structure. A hallucinated method name follows the exact same naming conventions as real methods in that library, because the model learned those conventions from real usage. Nothing about the code itself signals "verify this one specifically." That's precisely why this failure mode survives a normal skim-and-approve review pass, and why it needs a specific, deliberate countermeasure rather than general carefulness.

What Actually Catches It

Type checking and compilation are the first, cheapest filter. In a statically typed language, a call to a genuinely nonexistent method fails to compile, which catches the most blatant cases for free. This doesn't cover every case, since a permissive dynamic language, or a dynamically dispatched call, can silently accept a nonexistent method call until it's actually invoked at runtime.

Tracing the call against real documentation, not memory, is the reliable manual check. If a reviewer is even slightly unsure whether a specific method, parameter, or config key genuinely exists as written, opening the actual current documentation for the installed version takes under a minute and resolves the question definitively, versus trusting that it looks right.

Integration tests that actually exercise the call path catch what static analysis misses. A test that mocks the client entirely never discovers that a method doesn't exist on the real object; a test that exercises the real client (against a sandbox or test environment where feasible) will fail immediately if the call is fabricated.

Pinning and reading the exact installed version matters more with AI-generated code than it used to. If a codebase has multiple versions of a library referenced across its history, or the assistant's training data spans several versions, explicitly telling the assistant which version is installed, and verifying generated calls against that specific version's docs, closes a meaningful chunk of this gap.

A Related Case: Correct Method, Wrong Assumptions About Behavior

A quieter variant of this same problem is a method call that genuinely exists and compiles fine, but where the generated code makes an incorrect assumption about its behavior, whether it's synchronous or async, whether it throws or returns an error value, whether it mutates its input or returns a new copy. This doesn't fail to compile in most languages, and it doesn't always fail tests either, if the tests were generated from the same incorrect assumption. It shows up as a subtle behavioral bug, often around error handling or ordering, that's harder to catch than a flatly nonexistent method because the code runs without obvious complaint most of the time.

The same countermeasure applies here as with fabricated calls: when a reviewer isn't certain about a method's exact behavior, particularly around error handling and async semantics, checking the real documentation takes less time than debugging the resulting production issue later. This is worth treating as a habit specifically for any call the reviewer hasn't personally used many times before, regardless of whether the method itself is real.

Why Newer Codebases and Fast-Moving Libraries Carry More Risk

Libraries under active development, shipping breaking changes across minor or major versions frequently, are a specific risk multiplier here. A model's training data reflects a snapshot in time, and a library that's changed its API meaningfully since that snapshot leaves a gap between what the model confidently generates and what the currently installed version actually supports. Teams working with fast-moving frameworks or in-house libraries under active iteration should treat generated calls into those specific libraries with more scrutiny than calls into a stable, slow-changing dependency with a long, consistent history.

This Is One Piece of a Larger Review Problem

Fabricated API calls are one specific, well-documented failure mode among several that make AI-generated code need a different review approach than human-written code, alongside plausible-but-wrong logic and silently missing context a prompt didn't provide. Our broader guide on reviewing AI-generated code without rubber-stamping it covers the full set of failure modes and a review checklist built around them.

137Foundry helps engineering teams build review and tooling practices tuned to how AI-generated code actually fails, not how human-written code fails. See our full services or read more about how we work.

For more on the underlying model behavior, Wikipedia's entry on hallucination in AI systems covers the broader phenomenon this is a specific instance of, and most language model providers, including Anthropic, publish their own guidance on where model-generated output needs independent verification rather than direct trust. MDN's web documentation and equivalent official docs for whatever library you're using remain the fastest, most reliable way to verify a specific method or parameter actually exists as generated, faster in practice than trying to reason about whether it looks plausible.

Why Deprecating an API Endpoint Takes Longer Than Building It Did

137Foundry — Tue, 18 Aug 2026 11:31:57 +0000

Building a new endpoint is a known quantity: design the contract, implement it, test it, ship it, usually measured in days. Retiring an old one that's genuinely still in use by unknown parties can drag on for months, and the gap between those two timelines catches a lot of engineering teams off guard, especially ones used to the tighter feedback loop of building rather than the slower, more social process of removing.

The Asymmetry Between Building and Removing

When you build something, you control every dependency, because nothing depends on it yet. When you remove something, you don't control the dependencies at all, because by definition other systems have had time to build against it. Building is a one-directional decision made by one team. Deprecating is a negotiation with every consumer that's ever integrated, whether or not you know who they all are.

This asymmetry is the core reason deprecation timelines run longer than build timelines, and it's not a process failure when it happens. It's the natural consequence of an interface having real, external adoption, which is usually a genuinely good sign the endpoint succeeded in the first place, not a problem to be embarrassed about.

You Don't Actually Know Who's Using It

For a genuinely public API, usage logs tell you request volume, not who's behind those requests or how load-bearing the integration is for them. An endpoint with low but steady traffic might be a low-priority integration someone could migrate off in a day, or it might be a business-critical system for one specific partner who simply doesn't call it often. Request volume alone can't distinguish between the two.

This uncertainty is why responsible deprecation tends to start with active monitoring well before any announcement: instrumenting the endpoint to log enough detail, API keys, user agents, referring applications, to actually identify who's calling it, so outreach can be targeted rather than a blind broadcast into the unknown. Application performance monitoring tools, like the observability tooling documented on OpenTelemetry, make it straightforward to tag requests with enough metadata to build this picture without a custom logging solution built from scratch.

Signaling Deprecation in the Response Itself

Beyond documentation and outreach, the endpoint's own responses can carry the deprecation signal directly. A Deprecation and Sunset header pair, both defined in draft IETF standards (the Sunset header draft is the relevant reference), lets consumers detect a deprecation programmatically, in their own monitoring, rather than depending entirely on someone having read a changelog or an email months earlier. Automated tooling on the consumer's side can flag these headers and alert their own team, which catches integrations run by developers who've since left the consuming company and whose replacements never read the original announcement.

The Announcement Is the Easy Part

Publishing a deprecation notice, adding a header, updating documentation, all take an afternoon. What actually takes time is everything after: waiting for consumers to notice, waiting for them to prioritize the migration against their own competing work, and following up with the ones who haven't moved as the deadline approaches.

Consumers, especially external ones, generally don't prioritize a migration until it's urgent for them, which means the visible activity on a deprecation often clusters right before the actual sunset date rather than spreading evenly across the deprecation window. This is normal, not a sign the deprecation window was too short, and it's worth planning for rather than being surprised by.

Silent Failures Are Worse Than Loud Ones

A deprecated endpoint that quietly starts returning subtly wrong data, rather than an obvious error, is far more dangerous than one that fails loudly. A hard error gets noticed and fixed immediately. A quiet degradation can run in production for weeks before anyone traces a downstream problem back to it. This is part of why a genuinely safe deprecation strategy keeps the old endpoint fully functional and correct, unchanged, all the way through the sunset date, rather than degrading it gradually as an incentive to migrate faster.

Learning From How Major Platforms Handle This

Large platforms with an enormous, unknowable long tail of consumers tend to run deprecation windows measured in years rather than months for anything with wide adoption. Google's API deprecation policy is a useful reference point for how a platform with millions of unknown consumers approaches the tradeoff between moving the platform forward and not breaking existing integrations. Most teams don't operate at that scale, but the underlying principle scales down cleanly: the size and diversity of your consumer base should directly inform how conservative your deprecation timeline needs to be, not a fixed number chosen in advance regardless of actual adoption.

The Long Tail Problem

Most consumers of a popular endpoint migrate within the first half of a reasonable deprecation window once they're aware of it. The remaining stragglers, the ones who missed the announcement, the ones for whom migration isn't a priority, the ones running unmaintained integrations nobody currently owns on their end, are disproportionately time-consuming per integration relative to how few of them remain. This long tail is exactly why deprecation timelines tend to feel like they're dragging near the end even after the bulk of traffic has already moved.

What Actually Shortens a Deprecation Timeline

A few things genuinely help without cutting corners on safety: instrumenting usage early enough to identify consumers well before the announcement goes out, rather than starting that discovery process only after deciding to deprecate; providing a genuinely easy migration path, ideally with example code in the same languages consumers are actually using, rather than just documentation describing the difference conceptually; and direct, personal outreach for any consumer identifiable as high-volume or business-critical, rather than relying solely on a changelog entry they may never see.

None of these compress the fundamental asymmetry between building and removing. They reduce how much of the long tail turns into genuinely stuck stragglers by the time the sunset date arrives, which is realistically the best outcome any deprecation process can aim for rather than eliminating the long tail entirely.

Setting Expectations Before Starting

Teams that plan a deprecation assuming it'll take roughly as long as the feature took to build tend to be caught off guard when it runs several times longer. Planning from the start with a realistic timeline, informed by how many known consumers exist and how business-critical the endpoint is to them, avoids the awkward position of extending a deprecation window after already announcing a hard date, which erodes trust in every future deprecation announcement from that same API, not just the one currently running late.

A well-run versioning strategy makes this whole process more predictable, since a deprecation happening within an established, expected cadence is a very different experience for consumers than a surprise removal with no precedent. There's more on building that foundation in our guide on how to version an API without breaking existing clients. This migration work is a recurring part of what our team handles for clients retiring legacy interfaces, and the timeline reality described here is consistent across nearly every engagement, regardless of how simple the endpoint itself originally was.

How to Set Up Contract Testing for a REST API With OpenAPI

137Foundry — Tue, 18 Aug 2026 11:31:55 +0000

Manual code review catches most obvious breaking changes to an API. It doesn't catch all of them, especially under deadline pressure, and it definitely doesn't scale as an API and its number of endpoints grows. Contract testing closes that gap by turning the API's documented contract into something a test suite checks automatically, on every change, without relying on a human remembering to look.

This matters most for teams shipping frequently, where the volume of changes makes exhaustive manual review impractical, and for teams with external integration partners, where a missed breaking change costs someone else's engineering time to diagnose, not just your own.

Step 1: Write the OpenAPI Spec as the Source of Truth

Before contract testing means anything, the API needs an accurate OpenAPI specification describing every endpoint, request shape, response shape, and status code. If the spec is out of date or was written once and never maintained, contract tests built against it will validate against the wrong thing, which is worse than not testing at all because it creates false confidence.

Treating the spec as a living document, updated as part of the same pull request that changes the actual API behavior, is the precondition everything else here depends on. Some teams generate the spec directly from code annotations to keep the two from drifting apart; others maintain it by hand but require it in the same PR as any endpoint change. Either approach works as long as spec and implementation genuinely stay in sync.

Step 2: Validate Responses Against the Spec in CI

The first and most straightforward layer of contract testing: for every endpoint, run a set of test requests and validate that the actual response matches the OpenAPI-defined schema for that endpoint, in an automated CI step that runs on every pull request. Libraries exist for most major languages that take an OpenAPI spec and a live response and report exactly where they diverge, field by field.

This catches the most common category of accidental breaking change: a field removed, a type changed, a previously optional field made required, all without anyone needing to manually diff two versions of the spec by eye. OpenAPI's schema definitions build on JSON Schema under the hood, so most of the validation libraries and tooling in this space work across both, which is worth knowing if a team already has JSON Schema tooling in place from another part of the stack.

Step 3: Add Consumer-Driven Contracts for Known Integration Partners

Schema validation catches structural drift. It doesn't catch a client's specific behavioral dependency, like relying on array ordering or an exact status code, that isn't part of the formal schema at all. For a known set of integration partners, tools like Pact let each consumer define their actual expectations as a contract, which then gets checked against the provider's API automatically, catching exactly the kind of implicit-dependency breakage that pure schema validation misses.

This step requires more coordination than schema validation alone, since it needs consumer teams to actively participate in defining their contracts. It's worth the investment for a small number of important, known integration partners. It's less practical for a public API with an unbounded, unknown consumer base, where schema validation and careful changelog discipline do more of the work instead.

Step 3.5: Don't Skip Error Responses in Schema Validation

It's easy to write contract tests that only cover the happy path, the 200-level responses for valid input, and skip validating error responses against the schema entirely. This is a mistake, since error response shape is just as much a client-facing contract as a successful one, and error handling code in client applications is often less well-tested than the happy path, which makes it more fragile when an error response's shape shifts unexpectedly. Explicitly test that a 400, a 404, and a 422 each return the documented error shape, not just that they return the right status code.

Step 4: Version Contract Tests Alongside API Versions

If the API supports multiple concurrent versions during a deprecation window, contract tests need to validate each supported version separately, not just the current one. It's easy to build a CI pipeline that only tests the latest version, which means a change that accidentally breaks an older, still-supported version can slip through entirely undetected until an actual client still on that older version hits it in production, often weeks or months after the change originally shipped.

Step 5: Fail the Build, Don't Just Warn

Contract test failures should block a merge, not just log a warning that's easy to scroll past. A contract violation caught in CI and treated as a hard failure costs a few minutes of a developer's time to investigate right then. The same violation reaching production, undetected, costs considerably more later on, in both engineering time spent tracing the root cause and the trust of whichever client integration broke because of it.

Handling False Positives Without Losing Trust in the Suite

A contract test suite that produces frequent false positives, failing on changes that are actually fine, trains developers to ignore or bypass it, which defeats the entire point. This usually happens when the OpenAPI spec itself is stricter or looser than the API actually needs to be, rather than a problem with the testing approach. Keeping the spec genuinely accurate, revisiting it when a test fails for a reason that turns out not to matter, is ongoing maintenance, not a one-time setup task.

A test suite developers trust gets acted on immediately when it fails. One that's earned a reputation for crying wolf gets its failures dismissed, quietly reintroducing exactly the risk the whole setup was meant to catch.

A Realistic Rollout Order

Trying to set up full consumer-driven contract testing for every integration partner on day one is more setup than most teams need immediately, and it tends to stall the whole effort under its own weight. A more realistic order: get accurate OpenAPI schema validation running in CI first, since it catches the majority of accidental breaking changes with the least setup cost and the fastest path to a working, trusted safety net. Add consumer-driven contracts later for the handful of integration partners where the coordination cost is clearly worth it, typically the highest-volume or most business-critical ones, once the schema-validation layer is already stable and trusted.

Where This Fits Into a Broader Versioning Strategy

Contract testing is the enforcement mechanism for a versioning strategy, not a replacement for one. Knowing what counts as a breaking change, deciding how to expose versions, and setting real deprecation windows are the policy decisions; contract testing is what makes sure the policy actually holds under the pressure of a fast-moving codebase rather than degrading the first time someone's in a hurry.

137Foundry's web development team builds this kind of testing infrastructure as part of API projects specifically because policy without enforcement tends to erode within a few release cycles. There's more on the versioning side of this in our guide on how to version an API without breaking existing clients, which covers the decisions this testing layer exists to enforce.

How to Build a Lightweight Job Orchestrator Without Adopting a Heavy Scheduler

137Foundry — Mon, 17 Aug 2026 11:20:41 +0000

Somewhere between "a handful of cron jobs" and "we need Airflow" is a large middle ground most small-to-mid automation setups actually live in: a few dozen jobs, some with dependencies on each other, that need to run reliably without the operational overhead of standing up and maintaining a full orchestration platform. This is a walkthrough of building that middle ground yourself, in an afternoon, with tools you probably already have.

The instinct to reach straight for a full-featured scheduler is understandable, these tools are mature and well documented, but they also come with real operational weight: a scheduler service to run and monitor, a web UI to secure, and a learning curve for anyone new to the team who needs to understand DAG definitions before they can safely touch a job. For a dozen jobs with straightforward dependencies, that weight often isn't worth carrying yet.

Step 1: Decide what you actually need orchestration for

Before writing any code, get specific about what's failing today with plain cron. Usually it's one of three things: jobs that need to run in a specific order (job B shouldn't start until job A finishes successfully), jobs that need retry logic beyond what cron gives you for free, or visibility into which jobs ran, when, and whether they succeeded. Not all three problems require the same fix, and conflating them is how teams end up reaching for a full scheduler to solve a visibility problem that a much smaller tool would have handled.

Step 2: Model dependencies as an explicit list, not implicit timing

The most common cron anti-pattern is encoding a dependency as a time offset: job A runs at 2:00am, job B runs at 2:15am because it "should" be done by then. This works until job A runs long one night and job B starts against incomplete data, silently.

Replace implicit timing with an explicit dependency list in a small config file or database table: each job declares which jobs must complete successfully before it starts. A runner script checks this list before triggering each job, rather than relying on wall-clock offsets to approximate an ordering guarantee that was never actually enforced.

JOBS = {
    "extract_orders": {"depends_on": []},
    "transform_orders": {"depends_on": ["extract_orders"]},
    "load_warehouse": {"depends_on": ["transform_orders"]},
}

Step 3: Track run state in a table, not in memory

A minimal job_runs table (job name, started_at, finished_at, status, error) is the backbone of the whole system. Every run of every job writes a row here on start and updates it on completion or failure. This single table answers "did job A finish successfully today" for the dependency check in step 2, and it's also your monitoring surface, since a dashboard query against this table shows run history without any additional tooling.

PostgreSQL or any relational database you already run is sufficient for this. There's no need for a dedicated state store when the job volume is in the dozens rather than the thousands.

Step 4: Write a runner that checks dependencies before triggering

A single script, triggered on a short interval (every minute is common), checks each job's dependency list against the job_runs table. If all dependencies have a successful run for the current cycle and the job itself hasn't already run, trigger it. This replaces the manual "if it's 2:15, assume 2:00 finished" logic with an actual check against real completion state.

This runner is the entire orchestration layer. It's a few hundred lines of code, not a platform. For teams that eventually outgrow this approach, tools like Apache Airflow formalize exactly this pattern (dependency graphs, run state, retries) at much greater scale and complexity, which is worth knowing about even if it's not where you start.

Step 5: Add retries at the runner level, not inside each job

Rather than duplicating retry logic inside every individual job script, handle retries in the runner: if a job's status comes back as failed, the runner decides whether to retry based on a per-job retry count and backoff policy, then updates the job_runs row accordingly. Centralizing this in one place means changing the retry policy doesn't require touching every job script individually.

For jobs that use a message-passing pattern rather than direct invocation, Redis is a common, lightweight choice for the queue itself, avoiding the need for a heavier message broker when job volume doesn't justify one.

Photo by Brett Sayles on Pexels

Step 6: Build the smallest possible dashboard

A single page that queries the job_runs table and shows the last run status of every job, sorted by most recently failed, is usually enough. It doesn't need to be fancy. The goal is that anyone on the team can glance at it and answer "is everything running" without needing to know which log file to grep.

Handling the job that fails partway through a dependency chain

One case worth planning for explicitly: what happens when a job in the middle of a dependency chain fails. If transform_orders fails, the runner shouldn't trigger load_warehouse, since its dependency check will correctly see that transform_orders doesn't have a successful run for the current cycle. That part works automatically once the dependency check in step 4 is in place.

What needs explicit handling is what happens to extract_orders, the job upstream of the failure. It already ran successfully. Does it need to run again once transform_orders is fixed and retried, or can the orchestrator reuse the existing extracted data? This is a decision specific to each job, and it's worth documenting per job rather than assuming one answer covers the whole pipeline. Jobs that extract from a source that changes quickly (near-real-time data) usually need to re-run from the top. Jobs extracting from a stable source (a daily export that doesn't change after it lands) can often safely resume from the point of failure without re-running everything upstream.

A note on idempotency across the whole chain

None of this holds up if the individual jobs aren't safe to re-run. A transform_orders job that appends rows every time it runs, rather than upserting based on a stable key, will produce duplicate data the moment the orchestrator retries it after a failure. Every job in the chain needs to be safe to run twice with the same input, which usually means writes go through an upsert or ON CONFLICT DO NOTHING pattern rather than a plain append. This is worth verifying for every job before relying on the orchestrator's retry behavior, since a retry that silently duplicates data is often worse than the original failure it was trying to recover from.

When this stops being enough

This pattern holds up well for teams running somewhere under a hundred distinct jobs with straightforward dependency chains. Once you need dynamic DAGs generated at runtime, complex branching logic, or you're running enough jobs that a single runner script becomes a bottleneck, that's the point to seriously evaluate a dedicated orchestrator instead of continuing to extend the homegrown version. Building it yourself first, though, means you'll know exactly which of Airflow's features you actually need, instead of adopting the whole platform and discovering later that you use ten percent of it.

The failure handling side of this deserves its own attention beyond what fits in a job orchestrator. 137Foundry has a detailed guide on building a dead letter queue for jobs in a system like this one, covering what to capture when a job in the dependency chain fails permanently and how to review and safely reprocess it without breaking the jobs downstream of it.

How to Decide When a Failed Automation Job Needs a Human, Not Another Retry

137Foundry — Mon, 17 Aug 2026 11:20:40 +0000

Retry logic answers one question well: "should this run again." It's much worse at answering a different, more important question: "should a person look at this before it runs again." Most teams only build for the first question, which is why a job can retry the same doomed operation forty times before anyone notices it was never going to succeed.

This matters more as automation scales. A team running three jobs can get away with someone occasionally glancing at logs and catching the doomed retries by eye. A team running fifty jobs across a dozen integrations cannot, and the gap between "we have retries" and "we correctly route failures to the right response" becomes the difference between a system that mostly runs itself and one that quietly accumulates unresolved problems nobody has bandwidth to notice.

The default assumption is almost always "retry"

When an automation job fails, the reflexive fix is to add a retry. It's a reasonable default, most failures genuinely are transient, and automatically retrying a timeout or a brief network blip is the right call almost every time. The problem is that this default gets applied uniformly, to failures that are transient and failures that never will be, because the retry logic itself usually can't tell the difference.

A record with a malformed field, a reference to something that was deleted upstream, or a value that violates a downstream constraint isn't going to succeed on attempt six just because it failed on attempts one through five. Retrying it isn't buying time for the problem to resolve, because there's no version of "the problem resolves on its own" available. It just delays the moment someone actually looks at it, while burning compute and generating log noise on every attempt.

Photo by Tima Miroshnichenko on Pexels

Three questions that separate "retry" from "escalate"

Has this exact failure happened before, recently, on this same record? If the same record has failed the same way three times in the last hour, the fourth attempt is not meaningfully more likely to succeed than the third was. That's the clearest signal that the retry loop has stopped doing useful work.

Is the failure the kind that resolves with time, or the kind that requires a change? A connection timeout resolves with time, the network recovers, the downstream service comes back. A schema validation failure requires a change, either to the record or to the validation rule, and no amount of waiting fixes it. Classifying errors this way at the point they're caught, rather than treating every exception identically, is most of the work here.

Would a human looking at this right now actually be able to do something about it? If the answer is "no, they'd just see the same generic error and requeue it," escalating too early creates its own kind of alert fatigue, the same problem retrying forever creates, just moved earlier in the pipeline. The escalation needs to come with enough context (payload, error, attempt history) that a human reviewing it can actually make a decision, not just acknowledge that something is broken.

What "escalate" should actually mean

Escalating shouldn't mean paging someone at 2am for every quarantined record. It should mean moving the record out of the automatic retry loop and into a queue that's reviewed on a cadence, with enough detail attached that review is fast. PagerDuty and similar incident tools are the right layer for "this needs attention right now," reserved for backlog growth or systemic failure patterns, not for every individual record that stops retrying.

The distinction matters because pairing every single quarantined record with an urgent page trains the team to ignore the pages, which defeats the purpose of having them. Wikipedia's overview of incident management covers the broader discipline of separating "needs immediate response" from "needs eventual review," which is exactly the split that matters here.

A worked example

Consider a job that syncs customer records from a CRM into a billing system. Two kinds of failure show up in production: a timeout connecting to the CRM's API during its maintenance window, and a record whose email field is missing entirely, which the billing system's API rejects outright.

The first case is exactly what retries are for. The CRM's maintenance window ends, the same request that failed at 2:00am succeeds cleanly at 2:15am, and the retry loop closes out the failure without anyone needing to look at it. The second case is different in kind, not just severity. No amount of retrying fixes a missing email field. The record needs a human to either backfill the missing data at the source or decide the record should be excluded from billing entirely. Retrying it five times just delays that decision while generating five identical log entries.

The job's error handling needs to tell these two cases apart at the moment it catches the exception, not leave that determination to whoever eventually reads the logs. A timeout exception type gets the retry treatment. A validation exception type skips straight to the review queue.

Building the classification into the job, not the postmortem

The most effective version of this isn't a manual triage step applied after the fact, it's error classification built into the job itself at the point of failure. Wrap validation and business-rule failures in a distinct exception type from transient infrastructure failures, and let the retry logic branch on that type rather than treating every caught exception the same way. It's a small amount of extra structure that pays for itself the first time a genuinely permanent failure stops silently retrying for six hours before anyone notices.

The Google SRE book's chapter on handling overload makes a related point about designing systems that fail predictably rather than degrading in ways nobody planned for, and the same reasoning applies at the level of an individual job's error handling, not just system-wide load management.

What good escalation actually costs a team

Teams sometimes avoid building this classification because it feels like extra engineering effort for a problem that "mostly works" with generic retries. The honest accounting looks different once you count the cost on the other side: engineering time spent manually reconciling silently-dropped records, customer trust lost when a support ticket reveals data went missing for a week, and the alert fatigue that comes from every genuinely urgent failure being buried in a stream of retries that were never going to succeed.

Building the classification is a modest, one-time investment, usually a few distinct exception types and a branch in the retry decision. Living without it is a recurring cost that shows up every time a permanent failure gets treated like a transient one, which tends to happen more often than teams expect once a pipeline has been running for a year or two and accumulated a long tail of edge cases nobody anticipated at launch.

Where the retried-forever failures actually end up

Once a job correctly identifies that a failure needs a human rather than another retry, it needs somewhere durable to put that record while it waits for review. 137Foundry's automation team has written a detailed guide on building exactly that: a dead letter queue that captures what failed, why, and how many times, so the record isn't lost and isn't retried into oblivion either. It's the natural next step once a job stops treating every failure as identical.