DEV Community: Sonia Bobrik

On December 6, Wall Street Moves Midnight to 8 p.m.

Sonia Bobrik — Sun, 11 Oct 2026 13:56:29 +0000

For as long as anyone has built software around U.S. stocks, the market has handed engineers a quiet gift every weeknight: eight hours between the end of post-market trading at 8 p.m. Eastern and the 4 a.m. pre-market open, when the exchanges went dark and nobody minded a restart. Investor relations teams are already rethinking release schedules now that corporate news is losing its overnight buffer, yet the first people to feel the change will be the engineers whose batch jobs, deploys and reconciliations live in that same gap. On Sunday, December 6, 2026, U.S. equities are scheduled to start trading 23 hours a day, five days a week, and the single hour that survives comes with a new answer to an old question: when does a day end?

The Trade Date Now Starts the Evening Before

Under the FINRA definitions scheduled to take effect that day, a trade date stops lining up with a calendar day. It opens at 8:00 p.m. Eastern and closes at 8:00 p.m. the following day, so anything executed after 8 p.m. belongs to tomorrow, even when tomorrow is a Saturday or a holiday. The examples in Nasdaq's post-trade FAQ show what that looks like on the wire. A fill at 9:30 p.m. on Monday, December 7 goes out with Tuesday's date in FIX tag 75 (TradeDate) and settles on Wednesday, while tag 60 (TransactTime) keeps the real moment of execution. Nasdaq will even compute tag 336 (TradingSessionID) from that timestamp so clearing can tell overnight trades apart.

Put differently, the trade date on a confirmation and the date on a New York wall clock will disagree for four hours every night, by design. The vocabulary disagrees too: Nasdaq's rule filing calls the new hours the Night Session, NYSE Arca and Cboe call theirs the Overnight Trading Session, and Nasdaq markets the whole program as Global Trading Hours. Normalize session names at the edge of your system, the same way you normalize currency codes.

The UTC Shortcut That Only Works in Summer

If your code derives a trade date from a timestamp, two shortcuts will pass review and then fail. Taking the calendar date in Eastern time is wrong for everything between 8 p.m. and midnight. Taking the date in UTC, which is exactly what you get by slicing the date off a FIX TransactTime, is sneakier. During daylight saving time, 8 p.m. EDT is precisely midnight UTC, so the UTC date matches the new trade date from March to November and every test built from summer data passes. Clocks fall back on November 1. From then on, 8 p.m. Eastern is 1 a.m. UTC, and every post-market trade between 7 and 8 p.m. gets stamped with tomorrow's date. The overnight session launches five weeks into standard time.

The fix is boring, which is the point:

from datetime import date, datetime, time, timedelta
from zoneinfo import ZoneInfo

ET = ZoneInfo("America/New_York")
ROLLOVER = time(20, 0)  # 8:00 p.m. Eastern opens the next trade date

def trade_date(executed_at: datetime) -> date:
    if executed_at.tzinfo is None:
        raise ValueError("refusing to guess the zone of a naive timestamp")
    local = executed_at.astimezone(ET)
    if local.time() >= ROLLOVER:
        return local.date() + timedelta(days=1)
    return local.date()

UTC = ZoneInfo("UTC")
monday_930pm_est = datetime(2026, 12, 8, 2, 30, tzinfo=UTC)
monday_730pm_est = datetime(2026, 12, 8, 0, 30, tzinfo=UTC)  # already Tuesday in UTC
assert trade_date(monday_930pm_est) == date(2026, 12, 8)  # trades as Tuesday
assert trade_date(monday_730pm_est) == date(2026, 12, 7)  # still Monday's trade

Notice what the function does not need: a holiday calendar. Under the new definitions the trade date is plain clock arithmetic. The calendar comes back for settlement, since T+1 still skips weekends and market holidays.

Your Maintenance Window Is Now One Hour Long

The SIPs that publish consolidated quotes and trades, along with the FINRA/Nasdaq trade reporting facilities, will run from 9 p.m. Sunday to 8 p.m. Friday. The only weekday gap is the hour between 8 and 9 p.m., Monday through Thursday, reserved for processing and the switch to the next trade date. Apart from the odd market holiday, the stretch from Friday evening to Sunday evening becomes the only window long enough for a schema migration or a nervous upgrade.

The hour itself is stricter than the old evening. Firms using Nasdaq's post-trade risk service currently have until 8:20 p.m. to act on held trades. That grace period disappears, and the clearing firm's default, allow or inhibit, applies to every held trade at 8:00 p.m. sharp.

Then there is the job somebody schedules this month. In October, 8:15 p.m. Eastern is 00:15 UTC, so 15 0 * * * looks like it lands safely inside the pause. After November 1 it fires at 7:15 p.m., in the middle of live post-market trading. Shift it to 15 1 * * * for winter and, on March 14, 2027, it starts firing at 9:15 p.m., a quarter hour into the night session. The day-of-week field drifts as well, because Tuesday at 00:15 UTC is still Monday evening in New York. On Kubernetes, set .spec.timeZone to America/New_York (the field has been stable since v1.27) and write the schedule in Eastern time. Don't smuggle CRON_TZ= into the schedule string instead; the API server rejects it with a validation error.

The Night a Venue Cancels Your Fills

Overnight trading in U.S. stocks isn't new, and its biggest stress test went badly. On August 5, 2024, as Japan's Nikkei had its worst day since 1987, Blue Ocean, which Bloomberg described as the largest dedicated overnight venue in the U.S., buckled under the volume, halted and canceled every trade executed between 1:45 and 3:06 a.m. Eastern. Brokers that relied on it, Robinhood and a string of Korean firms among them, had to unwind trades their customers had already seen as filled. Korean financial authorities later counted roughly 630 billion won, about $435 million, in canceled transactions across some 90,000 accounts.

Exchanges bring sturdier plumbing to the night session, but the night still behaves differently from the day, and every difference is a place where a state machine can go wrong. Here is what I would check before December 6:

Derive trade dates from execution time, never from arrival time. Unreported, corrected or canceled trades from one session can arrive as prior-day messages once the next night session opens at 9 p.m., after the new trade date has already begun.
Make every fill reversible. Keep executions in an append-only ledger and model busts as compensating entries, so a 3 a.m. cancellation becomes an event you replay instead of a row someone deletes by hand.
Expect a mass cancel at 4 a.m. Nasdaq cancels night-session orders that are still open when the session ends, and it won't accept market orders overnight. An order manager that only tracks the cancels it sent will show customers ghost orders.
Teach your halt handling a new code. Stocks facing splits, large dividends, mergers, spin-offs or symbol changes will be halted before the night session and, for Nasdaq listings, reopened through an 8 a.m. halt cross. Nasdaq tags these halts M1, and a halt by the listing market applies on every exchange.
Keep "last trade" and "last sale" apart. Trades outside 9:30 to 4 carry the Form T sale condition, so they add to consolidated volume but can't update the official last sale. A chart that plots one while a risk check reads the other will disagree all night.
Check which feed your vendor consumes. Nasdaq plans to publish night-session activity on a separate market-data feed, with new Plus products covering the whole day.
Book a test slot. Industry test sessions are set for October 16 and 30, November 6 and 20, and December 4, and Cboe plans to let firms trade production test symbols in its overnight session from November 9.

Somebody's Morning Is Your Overnight

The hours that feel like the dead of night in New York are a workday elsewhere. As Euronews noted in its report on Nasdaq's 23-hour schedule, the 9 p.m. to 4 a.m. Eastern session runs from 3 a.m. to 10 a.m. Central European Time. In Seoul the same window falls between 11 a.m. and 6 p.m. Those are the people who will be trading, and they hint at who should be on call: a follow-the-sun rotation beats paging a New York engineer at 2 a.m. to interpret a halt message.

None of this calls for heroics. Most of it is work teams postponed because the old schedule forgave it: a timestamp stored without a zone, a cron line written in UTC, a ledger that deletes rows, a halt handler that only knows the codes it has already seen. The overnight session takes that forgiveness away. Replay a Sunday night and a Monday evening through staging now, while 8 p.m. is still a quiet hour, so that the first Tuesday that begins on a Monday evening is one you planned for.

Your Fire Escape Runs Through the Fire

Sonia Bobrik — Sun, 11 Oct 2026 13:53:34 +0000

For two hours on February 28, 2017, Amazon could not mark S3 as down on its own health dashboard, because the dashboard's admin console depended on S3. Every postmortem contains two stories, and that detail belongs to the second one. The first story explains what broke, and it gets the headlines. The second explains how the team got back, and it keeps revealing that the tools needed to end an outage were standing on top of the thing that failed. A recent essay on dependency blindness and the quiet ways invisible systems break flags the habit behind it as a warning sign: recovery plans that "depend more on memory and intuition than on clear system design." The biggest outages of the past decade keep proving that warning right.

The Status Page That Lived on the Patient

The S3 story is worth retelling in full. At 9:37 a.m. Pacific, an engineer on the S3 team, following an established playbook, ran a command meant to remove a small number of servers from a subsystem used by the billing process. One input was entered incorrectly. A far larger set of servers disappeared, including the ones behind two subsystems that the rest of S3 in us-east-1 relied on: the index, which manages the metadata and location of every object in the region, and placement, which allocates storage for new ones. Both needed a full restart.

That is where the first invisible system surfaced. As Amazon's own summary of the S3 disruption admits, the company had "not completely restarted the index subsystem or the placement subsystem in our larger regions for many years." S3 had grown enormously in the meantime, so restarting those systems and running the safety checks that validate their metadata took longer than expected. The index was fully back at 1:18 p.m. and placement at 1:54 p.m. Meanwhile, the service statuses on that dashboard couldn't be updated until 11:37 a.m., so the largest cloud provider in the world reported on its own outage through Twitter and banner text.

The typo started that morning, but it didn't decide its shape. A cold-start path nobody had walked at current scale decided how long it lasted, and a status page living on the patient decided how little customers could see while it did.

The Safety Feature That Hid the Building

Facebook's outage on October 4, 2021 followed the same script at a higher volume. During routine maintenance, a command meant to assess the availability of global backbone capacity took down every connection in the backbone instead. An audit tool existed to block exactly that kind of command, and a bug in it let the command through.

Then a safety mechanism finished the job. Facebook's DNS servers are built to stop advertising their BGP routes when they can't reach the data centers, which is normally a sensible way for an unhealthy location to step aside. With the backbone gone, all of them stepped aside at once, and Facebook, Instagram and WhatsApp vanished from the internet for roughly six hours.

Meta's engineering write-up of the outage is unusually candid about the way back. Engineers could not reach the data centers "through our normal means because their networks were down," and "the total loss of DNS broke many of the internal tools we'd normally use to investigate and resolve outages like this." So people were sent on site, into facilities designed to be hard to get into, full of hardware and routers designed to be hard to modify even with physical access. Every lock that keeps an attacker out is also a lock your own engineers will have to open on the worst day of the year.

The same post holds the hopeful half of the story. Once the network was back, Meta couldn't simply switch everything on, because its data centers had dropped tens of megawatts of power draw, and reversing that suddenly could put "everything from electrical systems to caches at risk." It restored traffic in stages and credited the "storm" drills it had been running for a long time, exercises that take a service, a data center or an entire region offline on purpose. The way back had been rehearsed, and it showed.

Fixing the Bug Is the Short Part

Newer incidents let you measure the gap between repairing a cause and recovering a system. On October 19 and 20, 2025, a latent race condition in DynamoDB's automated DNS management left the service's regional endpoint in us-east-1 with an empty DNS record. The trouble started at 11:48 p.m. Pacific, and all DNS information was restored by 2:25 a.m. The event itself didn't end until 2:20 that afternoon.

Those remaining twelve hours were recovery, as everything built on DynamoDB tried to come back, and one of the first systems in line was a component few customers had heard of. EC2's DropletWorkflow Manager holds a lease on every physical server, or droplet, that hosts instances, and its checks on those leases depend on DynamoDB. While DynamoDB was unreachable, the leases expired one by one, and a droplet without a lease can't receive new instances. When DynamoDB returned at 2:25 a.m., the manager began re-establishing leases across the entire EC2 fleet, the work took so long that it timed out and was queued again, and the system fell into what AWS calls "congestive collapse." The summary also contains a sentence worth reading twice: the situation "had no established operational recovery procedure."

Google Cloud hit the same wall in June 2025. A policy change with unintended blank fields sent its Service Control binaries into a global crash loop, and an emergency switch that disabled the failing code path was fully rolled out within 40 minutes. In us-central1, though, full recovery took up to about two hours and forty minutes, because the restarting tasks stampeded the Spanner table they depend on. Google's report says plainly that Service Control "did not have the appropriate randomized exponential backoff." GitHub met a version of the same problem on August 17, 2026, when errors in some Copilot services set off a client-side retry loop that added traffic during recovery, and the team had to tame it before it could safely restore traffic.

The purest example arrived in May 2026. Coinbase ran its exchange matching engine as a five-node Raft cluster inside an AWS cluster placement group, a feature that packs instances close together within one availability zone for low latency. After several chillers failed in one AWS data hall, a thermal shutdown followed, and when AWS terminated instances in that group, three of the five nodes went down and the cluster lost quorum. According to Coinbase's postmortem, getting it back took a new node group outside the impaired placement group and "an emergency code change shipped during the incident to remove a startup assumption that all five cluster nodes were resolvable." The startup path assumed a healthy cluster. It was code that only matters after a disaster, built on a premise that only holds when there isn't one.

Why the Way Back Rots

None of these teams was careless. They ran into a property of recovery code that has nothing to do with skill: it is the least-executed code you own. The happy path runs millions of times a day, so a broken assumption in it gets caught within minutes. The cold start, the failover script, the break-glass account and the status-page login may run once every few years, and the system keeps changing underneath them the whole time.

That change comes from three directions, and each one looks like progress. Growth changes the physics, so a restart that took minutes at launch takes hours at today's scale, as S3 discovered. Consolidation changes the topology, because putting chat, docs, dashboards, paging and deploys behind one identity provider in one cloud is efficient right up until that provider or that cloud is what's down. Hardening changes the locks, so the controls that keep attackers away from production also slow down your own responders. Each decision is reasonable on its own, and none of them shows up on a dashboard, because dashboards measure a running system, not the path for bringing a dead one back. Recovery paths rarely break in one dramatic moment. They drift, one sensible decision at a time, until they route straight through the failure they were built to escape.

Write a Bill of Materials for the Way Back

We already know how to make hidden dependencies visible. Teams generate software bills of materials so nobody has to guess what a build contains. The way back deserves the same treatment: a short document listing everything a recovery needs, built by asking one question of every item, "what does this need in order to work, and could that be down at the same time?" Start with these:

The status page. If it shares a cloud account, a DNS provider or a login with your product, assume it goes dark with your product, the way AWS's dashboard did in 2017.
Access. Can someone with authority reach production when SSO, the VPN or the office network is down? Break-glass credentials belong outside the identity provider they exist to bypass, and they need testing on a schedule, because an untested break-glass account is only a rumor.
Runbooks. If they live in a wiki behind the same login and network as production, keep an exported copy somewhere that shares neither.
Communication. Pick an incident channel that depends on neither your own infrastructure nor your main identity provider, and make sure everyone on call knows where it is before they need it.
Cold starts. Record when each critical system last started from zero, not one instance at a time, and how long it took. If the honest answer is "never," every recovery estimate you have is a guess.
Startup assumptions. Read your boot and init code for anything that expects a full cluster, a reachable config service or a fixed peer count. Those are Coinbase's five resolvable nodes waiting to happen.
Retries. Every client and worker that retries should back off exponentially with jitter, or your recovery will arrive as a stampede.

Expect at least one loop on the first pass, such as a secrets vault hosted on the cluster it holds the keys for, or a deploy pipeline that needs the service it is supposed to redeploy. Finding it on paper is cheap. Finding it during an incident is how a short outage turns into a long one.

Rehearse Without Your Tools

Chaos engineering usually asks whether a system survives a failure. The postmortems above raise a narrower and more uncomfortable question: can your team recover when its own tools are part of the failure? Meta's storm drills point toward the answer. For your next game day, take something away from the responders instead of from production. Declare the identity provider down, move the team off its usual chat workspace, or require one service to come up cold in a fresh environment with nothing cached. Then listen for the moment someone says, "wait, how do we..." Each of those moments is an invisible dependency surfacing on a quiet afternoon instead of in the middle of the night with customers watching. Add it to the bill of materials, fix the worst one, and run the drill again next quarter.

The Fire Code Principle

Buildings learned this long before software did, mostly from fires that taught it the hard way. Fire codes insist that exits keep working after everything else in the building has failed. Exit signs and emergency lighting carry their own backup power, stairwells are enclosed and separated from the rest of the structure, and everyone knows not to take the elevator. Underneath all of it sits one rule: the way out must not depend on the systems that are on fire.

Software has no fire marshal, so the inspection falls to us. Walk your exit route while the building is quiet, and make sure it doesn't run through the fire.

Hype Is Technical Debt, and Crypto Always Collects

Sonia Bobrik — Sun, 11 Oct 2026 13:52:56 +0000

Every long-lived codebase has a file nobody wants to open, usually the one with a "TODO: clean up after launch" comment older than half the team. Most crypto projects have a marketing twin of that file: the pinned launch-week thread promising mainnet by Q3, a tier-one exchange listing and partnerships that "can't be announced yet." That thread kept coming back to me while I listened to Sofia and Nika of the crypto marketing agency Techwaves share their expert view of the current state of the crypto market with host Polar Mil on The Seasonal Tokens Podcast. Along the way they get into the mistakes crypto projects make most often and how people who market crypto for a living think about hype.

The episode aired in May 2023, half a year after FTX imploded, and the problems it digs into have not aged a day. The guests come at them from the marketing desk. This post comes at them from the repo, because from where developers sit, hype behaves exactly like debt, and we already know how debt ends.

The Metaphor Was Born in Finance Software

The term "technical debt" comes from Ward Cunningham's 1992 OOPSLA experience report on WyCash, a system for managing cash portfolios. It is fitting that the idea maps so cleanly back onto markets. Cunningham's argument was simple: shipping code you know is not quite right is a loan. Borrowing a little lets you move faster, provided you repay it soon by reworking the code. The trouble starts when nobody ever does.

McKinsey's research on tech debt splits the concept into the two parts of any loan. The principal is the work you will eventually have to do anyway. The interest is the complexity tax every project pays in the meantime. The numbers are sobering: in a survey of 50 CIOs, respondents reported that 10 to 20 percent of the technology budget meant for new products was being diverted to tech-debt issues, and they put the debt itself at 20 to 40 percent of the value of their entire technology estate.

Now run the same split on a token launch. The principal of hype debt is the gap between what you have announced and what you have shipped. The interest is everything that gap costs while it stays open: a support channel that asks "wen mainnet" every morning, engineers pulled off the roadmap to write reassurance threads, and every new holder who bought the promise rather than the product. Technical debt charges interest in engineering hours. Hype debt charges it in trust, the one asset no protocol can mint.

Not All Hype Debt Is Reckless

Martin Fowler's technical debt quadrant sorts debt along two axes, deliberate versus inadvertent and reckless versus prudent, and it transfers to marketing almost one to one. Prudent and deliberate is announcing a testnet window to recruit early node operators, knowing you will have to hit it or explain in public why you missed. Reckless and deliberate is promising exchange listings you do not control and hinting at price targets you have no business discussing. Reckless and inadvertent is paying an influencer without knowing what disclosure the law expects. Prudent and inadvertent is discovering after launch that your messaging attracted traders instead of users, and rewriting it before the mismatch hardens into your reputation.

McKinsey makes a point about software that applies here too: the goal is not zero debt. A well-placed announcement can be worth what it costs. The goal is to know which quadrant you are borrowing from, and to write the loan down.

How the Bill Arrives

Crypto collects hype debt through three channels, and none of them sends a polite reminder first.

The first is the market. A liquidity crunch is the moment the market stops accepting IOUs. On October 10 and 11, 2025, just days after Bitcoin set a record above $126,000, a sudden tariff shock set off more than $19 billion in liquidations, the largest such event the asset class has ever recorded. Major coins dropped by double digits, and plenty of smaller tokens fell much further. When leverage unwinds, a token is worth roughly what it actually does, and narrative premium is the first thing sold.

The second is your own community. Unpaid hype debt turns your Discord into an incident channel and your roadmap into a list of apologies. Every hour your engineers spend on damage control is interest in its purest form.

The third is the law. When the SEC charged Kim Kardashian for touting EthereumMax's token on Instagram, the case came down to a single post she had been paid $250,000 to publish without disclosing the payment. She agreed to pay $1.26 million, without admitting or denying the findings, and not to promote crypto asset securities for three years. The post even carried an #ad hashtag, which shows how little a hashtag covers: the regulator's position was that the nature, source and amount of compensation all had to be disclosed. Enforcement priorities shift with every administration, but the anti-touting provision of US securities law dates back to 1933. If your growth plan includes paid promoters, their disclosures are your problem too.

Ship Evidence, Not Adjectives

Builders hold one big advantage in the hype economy: in crypto, almost any claim can be made checkable. Contracts can be verified on a block explorer, benchmarks can live in the repo as scripts anyone can rerun, audits can be pinned to the exact commit they reviewed, and usage can be read straight from the chain. A project that markets with artifacts instead of adjectives pays down principal with every post.

The most practical tool I know for this is a promise ledger: a public board, or even a pinned GitHub issue, where every external commitment gets a row with the date it was made, an owner, a status and a link to the evidence once it ships. Think of it as a bug tracker for marketing. Nobody gets to make a promise without adding a row, and the row stays visible until it is closed.

Before any announcement goes out, run it through a short pre-flight check:

Every claim has an artifact. A commit hash, a contract address, a testnet URL or a reproducible benchmark. If the evidence does not exist yet, the claim waits.
Dates come with their blockers. "Mainnet in Q3" is a mood. "Mainnet once the bridge audit, currently in review, is complete" is a plan people can follow.
Updates talk about users, not price. Describe what people can do now that they could not do last month, and leave the chart out of it.
Paid promotion is disclosed in full. Who paid, how much and for what, in writing, before anything goes live.
The changelog ships with the post. If an update cannot link to release notes, it is not an update yet.

Read the Repo Before the Chart

The podcast's own tagline is "crypto investing, not gambling," and hype debt is roughly where that line runs. A project carrying heavy hype debt asks you to bet on its ability to repay. A project that ships evidence asks you to inspect something real. Developers are unusually well equipped to tell the difference: compare a roadmap from a year ago with the commit history today, check whether the audited code matches what is actually deployed, and count how many public promises are still open. None of this predicts price. It does tell you whether you are investing in a product or underwriting a narrative.

Pay It Down Before the Market Does It for You

Bear markets work like forced refactors. They strip out whatever was held together by momentum and keep whatever was actually built. The projects that make it to the next cycle are rarely the loudest ones from the last. They are the ones whose announcements read like release notes, whose promise ledgers stayed short, and whose developers spent the drawdown shipping instead of apologizing.

So treat every public commitment the way you would treat a shortcut in code: take it on purpose, write it down and pay it back on schedule. And before you hit post, ask one question. Would you be comfortable seeing this tweet quoted in your postmortem?

Every Free Tier Is a Loan: Capital Discipline for Developers

Sonia Bobrik — Sat, 10 Oct 2026 19:46:34 +0000

For roughly a decade, the cheapest input in software was somebody else's money. Venture capital quietly paid for generous free tiers, "unlimited" flat-rate plans and permissively licensed infrastructure, and most of us built on top of all of it without ever reading the terms as terms. A sharp analysis of how capital discipline separates durable companies from fragile ones argues that markets now reward the businesses that can roll their obligations over at today's prices without tearing up the plan. Developers should take that personally, because our stacks are full of loans nobody remembers signing, and plenty of them are maturing right now.

The Refinancing Wall Is Not Just a Bond-Market Problem

In finance, a refinancing wall is what you hit when a big pile of cheap debt comes due at roughly the same time and has to be replaced with new debt at much higher rates. The OECD's Global Debt Report 2026 describes exactly that pressure: companies borrowed more from markets in 2025 than ever before in real terms, about $13.7 trillion, and because the debt maturing next is mostly low-cost paper issued before rates jumped, interest bills keep ratcheting up with every rollover.

Your vendors live inside that report. A developer-tools company that raised at a 2021 valuation and subsidized growth with nearly free capital is now being asked by its own investors and lenders for margins, not just signups. That pressure doesn't stay on its balance sheet. It flows downstream, straight onto your pricing page.

Here is the uncomfortable mapping. A free tier is a loan at a 0% teaser rate: the vendor lends you service below cost in exchange for future lock-in. A flat-rate plan with fuzzy limits is an interest-only mortgage. A permissive license on a project controlled by a single venture-backed company is a variable-rate note whose terms the lender can rewrite. None of these are bad deals. They're just deals, and deals have maturity dates.

The Wall Has Been Maturing in Public

If this sounds abstract, scroll back through a few years of changelogs. Heroku retired its free dynos in late 2022. HashiCorp moved Terraform to a source-available license in 2023, and the community answered with the OpenTofu fork. In February 2024, AWS started charging $0.005 an hour for every public IPv4 address, roughly $44 a year for something that used to cost nothing, multiplied across every load balancer, NAT gateway and forgotten test instance. A few weeks later Redis dropped its BSD license, and within about a week the Linux Foundation had launched Valkey, a fork with the major cloud providers behind it. After Broadcom closed its VMware acquisition, perpetual licenses gave way to bundled subscriptions. AT&T publicly cited a projected 1,050% increase in its bill, and even the two-to-threefold uplift Gartner described as more common is the kind of number that rewrites a budget.

The pattern isn't anecdotal. Vertice's SaaS inflation index put average year-over-year software price increases at roughly 12% in late 2025, about four times the US consumer inflation rate. Every event above was a rate reset. The teams that got hurt weren't the ones who used the cheap tier. They were the ones with no plan for the day it ended.

The Biggest Teaser Rate in Your Stack Right Now

For many developers in 2026, that teaser rate lives inside the editor. In June 2025, Cursor switched its $20 Pro plan from a fixed allowance of 500 requests to a usage pool billed at model API prices, and the backlash ended in a public apology and refunds. This spring GitHub made a calmer version of the same move. In the post announcing that GitHub Copilot is moving to usage-based billing, the company's chief product officer acknowledged that a one-line question and an hours-long agent run had cost the user the same, that GitHub had been eating much of the rising inference bill, and that the old premium-request model could not continue. Since June 1, most Copilot usage draws down credits based on the tokens it actually consumes.

Base plan prices didn't move, and code completions are still included. That is exactly what a polite refinancing looks like: the same monthly payment, new terms on everything that actually costs money.

Now read that with a capital allocator's eyes. If your team's workflow depends on agents that run for an hour at a time, the price of that hour was never the price you were paying for it. A 2027 engineering budget that assumes today's AI spend is modeling a teaser rate as if it were fixed.

Read Your Stack Like a Credit Analyst

Credit analysts don't ask whether a company is good. They ask what happens when its debt reprices. You can run the same review on your dependencies in an afternoon, and it starts with a plain spreadsheet: every service, license and tool that something important relies on, with five questions answered for each.

What is the real coupon? Price the dependency as if the subsidy vanished tomorrow: the self-hosted cost, a competitor's list price, or the raw API rate hiding behind the flat fee. The gap between that number and what you pay today is your exposure.
Who controls the covenant? A project governed by a foundation with many corporate contributors behaves like a fixed-rate loan. A project where one company owns every copyright can be relicensed whenever its board needs a better margin.
When is the likely maturity date? Resets have tells: a funding round that's years old, a new CFO, an acquisition, layoffs, a "pricing update" banner, or a free tier that shrinks a little every quarter.
Is there a refinancing path? A protocol-compatible fork, an open standard, or a second provider you could genuinely switch to. If the only alternative is a rewrite, you aren't diversified, you're hoping.
How correlated is the exposure? One vendor running your CI, your code review and your coding agents is a single counterparty. When it reprices, everything reprices at once.

Refinance Before You're Forced To

The point isn't to avoid vendors. Self-hosting everything is just another way to overpay, in engineering hours instead of invoices. The point is to make sure no single reset can blow up a roadmap.

Put seams where the repricing risk is concentrated, not everywhere. A thin internal interface in front of your model provider costs a few days and turns a price hike into a config change. Three layers of abstraction around your ORM "just in case" cost months and protect you from nothing. Prefer standards over products at the boundaries, too: SQL, S3-compatible storage, OCI images and OpenTelemetry are the fixed-rate mortgages of software. Teams that only used core Redis commands found Valkey close to a drop-in swap, while teams leaning on proprietary modules had a much harder conversation.

Rehearse the exit. A migration path that has never run is a theory. Once a quarter, point your test suite at the alternative: the fork, the second model provider, the self-hosted build. A few minutes of CI time is a cheap insurance premium.

Meter your own usage before the vendor does. Put token spend, egress and per-seat costs on the same dashboard as latency. When metered billing lands, you'll already know which workflows are cheap and which are quietly expensive.

Why the Person With the Map Wins

The capital discipline argument from the top of this post has a second half that's easy to skim past: credibility lowers the cost of capital, because lenders charge extra for uncertainty. Engineering budgets behave the same way. The developer who walks into planning with a one-page map of the stack's teaser rates (what each dependency costs today, what it would cost at market, when it's likely to reset, and what leaving would take) is doing capital allocation, whether or not anyone calls it that. That's the person who gets handed the next big architectural decision, because their numbers survive contact with finance.

Cheap money built half of the modern toolchain. It isn't coming back, but the toolchain is still running, and it's still quietly accruing interest. The only real question is whether you find the reset date on your own calendar or on an invoice.

Money Moves in Seconds Now. Your Billing Code Is the Bottleneck.

Sonia Bobrik — Sat, 10 Oct 2026 19:39:46 +0000

A transfer sent over the Federal Reserve's instant payment rail lands in the recipient's account within seconds, even at 3 a.m. on a Sunday. The invoice behind that transfer may have needed the better part of two months to get there, crawling through a month-end batch job, a shared accounts-payable inbox, two approval queues and a reconciliation spreadsheet that someone updates on Fridays. That gap is exactly why the argument that cash velocity, not revenue, has become the real test of business strength deserves a developer's attention and not only a CFO's. How quickly earned money turns into money a company can actually spend is now decided less by banks and more by software, and a lot of that software is written by people like us.

The Banks Stopped Being the Excuse

For most of the history of business payments, slow money had an alibi. ACH moved in batches that typically settled a business day or two later, wires paused for weekends and holidays, and a paper check could spend a week traveling and clearing. Nobody outside a bank could do much about any of that, so engineering teams treated the payment timeline like the weather.

That alibi has expired. On July 20, 2023, the Fed launched FedNow, its round-the-clock instant payments service, which lets banks and credit unions of any size move money for their customers at any hour, on any day of the year. Jerome Powell's launch statement even used our scenario as a selling point: a company getting immediate access to its funds when an invoice is paid. Europe went a step further and turned speed into an obligation. Under the Instant Payments Regulation, banks in the euro area have had to accept instant euro transfers since January 2025 and send them since October 2025, at a price no higher than a regular transfer and with a free check that the payee's name matches the account number. The European Central Bank's guide to the Instant Payments Regulation lays out the staggered deadlines, which reach the EU countries outside the euro area in 2027. Add the UK's Faster Payments, India's UPI and Brazil's Pix, and instant transfers are an everyday habit for hundreds of millions of people.

So the interbank leg, the part everyone used to blame, can now take seconds. What stays slow is everything that happens before someone presses "send," and everything that happens after the money lands but before anyone knows what it was for. Both of those are software.

Follow One Invoice and Watch Where the Days Go

Pick an ordinary B2B invoice from last quarter and replay its life as a timeline. The client signed off on the work on the 3rd, but invoices come out of a month-end batch job, so the bill didn't exist until the 1st of the following month. It traveled as a PDF attached to an email, landed in a shared accounts-payable mailbox and waited for someone to key it into the customer's ERP. It bounced once because the purchase order number was missing. It waited a week for an approver who was on holiday. It was slotted into the customer's weekly payment run. When the money finally arrived, the reference field read something like "INV2231 2232 PART," and someone on your side lost a morning working out which invoices it covered and why it was short.

Not one of those delays happened inside a payment network. Each lives in a system somebody designed: a cron schedule, an email template, an import script, a validation rule, a matching spreadsheet. The money itself was in motion for a few seconds. Everything around the money took seven weeks.

Payment Messages Finally Have Room to Explain Themselves

The second shift is quieter, and it matters even more to anyone who writes integration code. For decades most payments carried only a few cramped lines of free text about their purpose, which is why matching money to invoices turned into a manual craft. That is changing. The Fed moved Fedwire, its wholesale wire system, to the ISO 20022 message standard in a single-day cutover on July 14, 2025, and Swift ended the period in which banks could still use legacy MT messages for cross-border payment instructions on November 22, 2025. FedNow and Europe's instant payment schemes were built on ISO 20022 from day one.

The part developers should care about is structured remittance data. An ISO 20022 payment can reference specific invoices in fields software can parse, instead of a string a human has to decode. If your invoices carry a unique, machine-readable reference, and your payment links and bank instructions pass it through untouched, matching a payment to an invoice stops being detective work and becomes a database join. Finance teams call the result automated cash application. Engineers can simply call it deleting a manual step.

The Invoice Is Becoming an API Payload

The document is changing as well. France's e-invoicing mandate went live on September 1, 2026: every VAT-registered business established there must now be able to receive electronic invoices through an approved platform, large and mid-sized companies must also issue them, and smaller firms join on the issuing side in September 2027. Germany has required businesses to accept e-invoices since January 2025, and the EU's VAT in the Digital Age package, adopted in March 2025, makes e-invoicing mandatory for cross-border B2B trade inside the EU from July 2030.

A PDF is a picture of data. A structured invoice is the data itself, and that changes what the receiving side can automate: validate it on arrival, match it to a purchase order, route it to the right approver and schedule the payment without anyone retyping a line. Each of those steps used to be a queue measured in days. If you build billing, procurement or accounting software, this is the moment when "export to PDF" stops being a feature and starts being technical debt.

What to Build Before the Next Billing Cycle

None of this needs a banking license. It needs someone to treat the path from "work accepted" to "cash applied" as a product surface rather than a back-office chore. These changes offer the best ratio of engineering effort to days saved:

Invoice on the event, not the calendar. Generate the bill when a milestone is accepted or a usage period closes. A month-end batch can quietly add almost a month of dead time before the customer even sees a number.
Give every invoice a machine-matchable identity. Use a unique reference that survives the trip through the payment message, or assign each customer a virtual account number, so incoming money identifies itself.
Put a pay-now path inside the invoice. A payment link or bank instruction that works over instant rails removes the wait between "approved" and "paid." FedNow and The Clearing House's RTP network both support request-for-payment messages, so ask your bank or payment provider what it exposes.
Ingest bank data as events. Pull intraday reports and payment notifications (camt.052 and camt.054 in ISO 20022 terms, or your provider's webhooks) instead of a once-a-day CSV, and apply cash automatically whenever the reference matches.
Validate before you send. Block your own invoice if the purchase order number, tax ID or billing contact is missing. A bounce your code catches costs seconds; a bounce the customer's AP team catches can cost a week.
Make disputes loud. A rejected or disputed invoice should open a ticket with an owner and a deadline instead of sitting unread in a shared inbox.

Find Your Slowest Stage With One Query

Days sales outstanding tells you how long collection takes overall, but not which stage to fix. For that you need timestamps. If your billing system writes a row to an invoice_events table each time an invoice changes state (work_accepted, invoice_sent, invoice_approved, payment_scheduled, payment_received, cash_applied), this PostgreSQL query shows how long invoices sit in each stage and what share of all waiting time each stage owns:

WITH steps AS (
  SELECT
    invoice_id,
    stage,
    occurred_at,
    LEAD(occurred_at) OVER (
      PARTITION BY invoice_id ORDER BY occurred_at
    ) AS left_at
  FROM invoice_events
  WHERE occurred_at >= now() - interval '120 days'
)
SELECT
  stage,
  COUNT(*) AS invoices,
  ROUND((percentile_cont(0.5) WITHIN GROUP (
    ORDER BY EXTRACT(EPOCH FROM left_at - occurred_at)
  ) / 86400)::numeric, 1) AS median_days,
  ROUND(100 * SUM(EXTRACT(EPOCH FROM left_at - occurred_at))
    / SUM(SUM(EXTRACT(EPOCH FROM left_at - occurred_at))) OVER ()) AS pct_of_total_wait
FROM steps
WHERE left_at IS NOT NULL
GROUP BY stage
ORDER BY pct_of_total_wait DESC;

Each row describes the stage an invoice was sitting in and how long it stayed there before the next event, so pct_of_total_wait is the column to sort your backlog by. When invoice_sent dominates, the delay looks like the customer's fault, yet the cure is often on your side: a missing PO field, an invoice delivered to the wrong mailbox or a format their system can't ingest. When payment_received is large, your own reconciliation is the bottleneck, and the machine-readable references described above are the fix.

Speed Is a Design Decision Now

When money needed days to cross the banking system, cash velocity looked like something the banks controlled. Now that money can move in seconds, a seven-week gap between delivering value and being able to spend the proceeds is mostly the sum of small choices: one batch schedule, one PDF template, one manual matching rule at a time. Companies that close that gap fund their next hire, their next experiment and their next bad quarter with their own money instead of someone else's. And the people best equipped to close it are the ones who already spend their days finding the slowest stage in a system and engineering it away.

The Ship in Your Dependency Graph

Sonia Bobrik — Sat, 10 Oct 2026 19:38:01 +0000

On March 14, 2024, four subsea cables off Côte d'Ivoire failed almost simultaneously, and Microsoft started warning customers in its South Africa North and South Africa West regions about rising latency and dropped packets. Nobody had pushed a bad config, and no certificate had expired. Some operators failed over automatically while others spent hours buying capacity on cables that still worked, but the broken glass itself had to wait for the small, aging fleet of repair ships that quietly keeps the internet running, and that wait is the one part of our stack almost nobody writes into a design doc. If your product crosses an ocean, there is a ship somewhere in your dependency graph, and it is worth knowing how that ship schedules its work.

Four Cables, One Failure Domain

WACS, MainOne, SAT-3 and ACE look like four-way redundancy on a network diagram. On the seabed they were neighbors. MainOne's preliminary analysis ruled out human activity, given the depth of the fault, and pointed to movement of the seabed itself, which means a single geological event took out every "independent" path at once. Engineers meet this pattern constantly at smaller scale: two availability zones fed by one substation, two DNS providers managed through one registrar account, two "diverse" fiber routes squeezed through the same bridge. Redundancy is not a count of paths. It is a claim that their failures are uncorrelated, and geography is extremely good at falsifying that claim.

The seabed correlates failures in time, too. Research published in Nature Communications followed underwater avalanches of sand and mud, known as turbidity currents, down the Congo Canyon off West Africa. One flow in January 2020 ran for more than 1,130 kilometers, accelerating from about 5 to 8 meters per second, and snapped two seabed telecom cables, slowing the internet from Nigeria to South Africa. Those cables had gone roughly 18 years without such a break, and then they broke again in March 2020, April 2021 and January 2022. The trigger was not an earthquake but the biggest Congo River floods in decades, with the destructive flows arriving weeks to months after the flood peak, often at spring tides. Some cables even survived flows of similar speed that snapped their neighbors, most likely because erosion was patchy and concentrated around steep steps in the canyon floor. Read that with on-call eyes: failures arrive in bursts after an observable precursor, and small differences in placement decide which replica lives.

Mean Time to Repair Is a Queue

Logical recovery happens in routers. Physical recovery happens on a ship, and ships are a scarce shared resource with a scheduler. In July 2026, the final report of an international advisory body on submarine cable resilience, set up by the ITU (the UN agency whose X.509 standard defines the certificates in your TLS stack) and the International Cable Protection Committee, put hard numbers on that scheduler. Between 2012 and 2024, the average time to begin a repair more than doubled, from under 20 days to over 50, even though the yearly number of repairs and of available ships stayed flat; permitting delays take most of the blame. About 70 vessels can technically fix a cable, yet only 20 to 25 are dedicated to repair work or on long-term charter, and a significant share were built in the 1980s or 1990s. Asia-Pacific and the North Atlantic each host roughly 40% of the active repair fleet, while Africa, the South Atlantic and the Pacific islands share about 13%. A fault in the North Atlantic is often handled within 3 to 5 days, while a comparable one off West Africa can take 12 to 18.

Africa shows the sharpest version of this. The whole continent has one permanently stationed, dedicated repair ship, Orange Marine's Léon Thévenin, built in 1983 and based in Cape Town. In 2024 it left port on March 19 for the Côte d'Ivoire breaks, where it took SAT-3 while another vessel, CS Sovereign, was assigned the other three cables. It was back in Cape Town by April 25. On May 12, Seacom and EASSy broke off the KwaZulu-Natal coast, cutting all subsea capacity between South Africa and East Africa, and on May 14 the same ship sailed again. One server, two coasts, strictly serialized. Orange Marine has since ordered two new ships to replace it and a 1987-built vessel in Italy, with delivery planned for 2028 and 2029.

If you have ever watched a worker pool saturate, you know the curve. In the textbook single-server queue, the average wait equals utilization divided by one minus utilization, measured in service times: at 50% busy a new job waits about one repair, and at 90% busy it waits nine. Cable repairs are long jobs, since every splice keeps the ship holding position for a day or more and needs a weather window to do it. That is why correlated faults hurt twice. They arrive together, and then they queue together.

What a Cut Looks Like From Your Side

A cable cut rarely shows up as a red outage banner. It shows up as gray failure: health checks pass, error rates look tolerable, and users in one geography quietly have a terrible day. The physics explains why. Light in fiber covers roughly 200,000 km per second, so every extra 1,000 km of detour adds about 10 ms of round-trip time. A single TCP connection can move at most one window of data per round trip, so doubling the RTT roughly halves what that connection can push, before you even count congestion on the detour. NetBlocks' research director made exactly that point during the March 2024 outage: as networks route around damage, they can drain capacity that other countries depend on, so the first problem may be physical while the ones that follow are technical. Meanwhile your client timeouts, tuned to last month's p99, start firing, retries pile onto an already congested path, and a distant event on the ocean floor becomes a load spike you generated yourself.

A Pre-Mortem for Anything That Crosses an Ocean

You cannot buy a cable ship, but you can stop assuming repair capacity is infinite. A few habits make the difference:

Map physical paths, not logical ones. Run mtr -rwc 100 <host> from where your users actually are, compare the hops with a public submarine cable map, and ask your providers which systems carry your traffic.
Budget timeouts for the detour. Derive deadlines from a latency budget that survives an extra 100 to 150 ms of RTT instead of copying last month's p99.
Make retries polite. Use exponential backoff with jitter and a retry budget, so a reroute does not turn into a denial-of-service attack you launched on yourself.
Rehearse the slow path. On a staging host, sudo tc qdisc add dev eth0 root netem delay 150ms 20ms loss 1% simulates a long detour; watch what breaks first, then clean up with sudo tc qdisc del dev eth0 root.
Alert on latency steps by region. A sudden, flat jump in RTT from one geography is the signature of a reroute, and error-rate alerts alone will sleep through it. Public trackers such as Cloudflare Radar usually show within hours whether the problem is yours or the whole region's.
Write the runbook in weeks, not hours. Decide in advance what you will degrade for users behind a cut, such as upload sizes, media quality or sync frequency, because the fix might be a ship on the other side of a continent.

The Slowest Component Wins

Our services recover in milliseconds, routing converges in minutes, and the seabed heals as fast as an old ship can clear its queue. That last number has been growing for more than a decade, and the incidents above show it does not care how many lines your architecture diagram has. Treat repair capacity like any other shared dependency you do not control: know where you lean on it, make the slow path survivable, and design as if one day your ticket will be third in line.

The Signature Was Valid. The Software Wasn't.

Sonia Bobrik — Sat, 10 Oct 2026 19:37:19 +0000

In September 2015, WeChat shipped to the App Store with code that nobody at Tencent had written. So did a ride-hailing app, a music service, a business-card scanner and a long tail of other iPhone apps. Nobody had broken into those companies or touched their repositories. Somewhere on each of those teams, a developer frustrated by painfully slow downloads from Apple's servers had installed Xcode from a fast local cloud-storage share or a colleague's copy, and that copy carried a modified linker that stitched extra code into every app it built. The source was clean, the binaries were signed, and Apple's review approved them. That is precisely the blind spot that this essay on the hidden life of software provenance describes as operating on trust alone, and in most build pipelines it is wider than anyone likes to admit. This post is about the gap between the code you review and the bytes you actually ship, why a valid signature does not close it, and the cheap habits that do.

Ken Thompson Saw It Coming in 1983

When Ken Thompson accepted the Turing Award in 1983, he spent his lecture describing a C compiler with two tricks. It recognized when it was compiling the Unix login program and slipped in a backdoor password. It also recognized when it was compiling a new version of itself and quietly re-inserted both tricks. Once that binary existed, the compiler's source could be scrubbed perfectly clean and the backdoor would survive every rebuild. Thompson later confirmed that the trojan really did land in the login binary of a machine belonging to a Unix support group, although the poisoned compiler itself was never distributed. The lecture, published in 1984 as "Reflections on Trusting Trust," ends on a conclusion that has aged uncomfortably well: reading source code proves nothing about a binary unless you also trust everything that produced it.

For more than thirty years, that attack lived mostly in lecture halls. XcodeGhost moved it into production. The idea was not even new in 2015: months earlier, documents leaked by Edward Snowden had described CIA-backed researchers claiming they had built a modified Xcode able to plant surveillance backdoors in any app compiled with it. What was new was the scale. Palo Alto Networks counted just five malicious apps that had ever made it into the App Store before that autumn, which is why BBC News reported XcodeGhost as the first large-scale attack on Apple's App Store, and estimates of infected apps eventually passed 4,000. Apple's remedy was telling: rebuild with a clean copy of Xcode, and verify that copy with Gatekeeper before trusting it again.

Signatures Answer "Who," and Attackers Love That

Code signing was supposed to settle questions like these. Two incidents show why it cannot do the job alone.

Between June and November 2018, attackers who had gotten inside ASUS pushed a backdoored version of its Live Update utility through the company's official update servers, signed with a legitimate ASUS certificate. Kaspersky, which uncovered the operation in January 2019 and named it ShadowHammer, found that more than 57,000 of its own users had installed the update and estimated the total reach at about a million people. The attackers only cared about roughly 600 of those machines, identified by MAC address hashes hardcoded into the malware. Every computer that checked the update concluded it genuinely came from ASUS, and every one of them was right.

In 2023, the VoIP vendor 3CX, which says its software has more than 12 million users, shipped trojanized desktop apps for Windows and macOS. They were produced in 3CX's own compromised build environments, carried valid signatures and were served from 3CX's own website. Mandiant traced the intrusion to an employee who had installed X_TRADER, a futures-trading application retired in 2020 but still available for download in 2022, which had itself been trojanized. It was the first time Mandiant had seen one software supply chain attack directly produce another.

A signature proves that someone holding a key approved a particular set of bytes. It says nothing about what happened between the commit and the key. If an attacker lives in that stretch, your signing step stops being a safeguard and becomes their laundering service.

Tags Lie, Hashes Don't

The same failure shows up in tools developers touch every day. In March 2025, an attacker compromised tj-actions/changed-files, a GitHub Action used by more than 23,000 repositories, and retroactively moved its existing version tags to point at a single malicious commit. Workflows referencing a friendly tag such as @v35 silently started running code that dumped CI secrets into build logs, which on public repositories anyone can read. Nobody merged a pull request and nobody bumped a version; the tag simply started meaning something else. Workflows pinned to a full commit SHA kept running exactly what they had always run.

The most encouraging story in this genre is also the most boring. From the end of January 2021, attackers periodically altered Codecov's Bash Uploader, a script that CI pipelines everywhere downloaded and executed on every run, so that it could ship data from those CI environments to a server outside Codecov's control. It was caught on April 1 by one customer who did something almost nobody does: compared the SHA256 of the freshly downloaded script with the checksum published on GitHub. The values did not match. Two months of silent exfiltration ended because one person ran one command.

What Provenance Actually Adds

Provenance turns the invisible stretch between commit and artifact into a claim someone else can check. In the SLSA framework (Supply-chain Levels for Software Artifacts, pronounced "salsa"), a provenance statement says, in effect: this exact artifact, identified by its digest, was produced by this builder, from this repository at this commit, by this workflow. Sigstore made signing such statements nearly free. Instead of guarding a long-lived private key that can be stolen, a CI job proves its identity through OIDC, receives a certificate that expires within minutes, and records the signature in a public transparency log where later tampering would stand out.

None of this has to be built by hand anymore. GitHub's guide to artifact attestations shows that a workflow needs only a few permissions, including id-token: write and attestations: write, plus one actions/attest step pointed at its output; consumers can then run gh attestation verify against the file and get a clear yes or no. That alone reaches SLSA Build Level 2. GitHub's immutable releases go further and lock a release tag to a single commit, which is exactly the guarantee tj-actions users assumed they already had. On the npm side, packages published with npm publish --provenance link back to the commit and workflow that built them, and npm audit signatures (npm 9.5 or later) reports how many of your installed dependencies carry verified registry signatures and attestations.

Be honest about the limits. Provenance does not prove code is safe, and a malicious maintainer can publish malware with immaculate provenance. What it does is make the unexpected visible. A package that has shipped with attestations for two years and suddenly arrives without them, a binary built by a workflow you have never seen, an artifact whose digest matches no attested build: those are exactly the anomalies that XcodeGhost, ShadowHammer and the 3CX attack hid behind. Reproducible builds take the idea to its conclusion. Independent machines compile the same source and must produce bit-for-bit identical output, so no single build environment has to be trusted at all, which is why projects such as Debian and the Tor Browser have invested years in getting there.

A Provenance Audit You Can Run This Week

You do not need a dedicated security team to start. Most of the protection comes from a handful of small, unglamorous changes:

Pin third-party GitHub Actions to full commit SHAs and keep the human-readable version in a trailing comment, or rely on publishers that use immutable releases.
Stop piping downloaded scripts straight into a shell in CI. Fetch the file, compare its SHA256 with a value committed to your repository, and only then run it.
Run npm audit signatures in CI and treat a dependency that used to ship attestations and suddenly does not as an incident until proven otherwise.
Attest your own release artifacts and put the exact gh attestation verify command in your README, so users can check your builds instead of taking them on faith.
Install compilers and SDKs only from the vendor's origin, verify their signatures or checksums, and hold the toolchains baked into your CI images to the same rule.
Build one artifact twice on two different machines and diff the hashes. The first mismatch you chase down will teach you more about your pipeline than a quarter of architecture meetings.

The Question Worth Asking

Thompson needed three pages to show that source code is a promise, not proof. Four decades later, the tools for checking that promise are free and mostly boring: a pinned hash, an attestation, a one-line verify command in a README. What has changed is the question worth asking about everything that runs in your pipeline. It is no longer "is this signed?" It is "can I show where this came from, and would I notice if that changed tomorrow?" A signature tells you whom to blame. Provenance tells you what to believe.

Passkeys Can't Be Phished. Your Fallbacks Can.

Sonia Bobrik — Sat, 10 Oct 2026 19:33:54 +0000

On July 20, 2022, at least 76 Cloudflare employees got the same text message within a single minute, each one pointing to a convincing copy of the company's Okta login page. Three of them typed in their usernames and passwords, which at most companies would be the first line of an incident report. At Cloudflare the attack went nowhere, because the second factor was a FIDO2 security key, and a key like that will not sign a login for a domain it was never registered with. If you want the plain-language case for making that kind of protection your default, this practical upgrade path to passkeys and phishing-resistant MFA walks through it step by step, from password managers to recovery codes. This post picks up where that story usually ends: after the passkey button ships, when the other doors into the same account are still standing open.

Why the Key Said No

Cloudflare's own write-up of the attack is worth reading for the mechanics alone. The lookalike domain was registered less than 40 minutes before the first text went out. Whatever a victim typed was relayed to the attacker over Telegram in real time, and the page then asked for a one-time code, so both could be replayed on the real login page before the code expired. Anyone who made it past that step was offered a download of AnyDesk, the remote access tool. Twilio, hit by a campaign with very similar fingerprints, had disclosed a breach the day before Cloudflare published.

Against SMS codes or an authenticator app, a relay like that wins. A six-digit number has no idea which website it is being typed into, and neither does a distracted person holding a phone. A WebAuthn credential does know. The browser puts the page's origin into the data the authenticator signs, and the credential itself is scoped to a single relying party, so a key registered for the real login domain has nothing to offer a lookalike. Nobody has to spot the fake URL, because the protocol checks it for them. Cloudflare said that, like Google before it, it had not seen a successful phishing attack since hardware keys became mandatory. It also made a point of not punishing the three employees who were fooled, which is the right instinct. If a system fails whenever a human has a bad day, the system is the bug.

Attackers Don't Break the Lock. They Find Another Door.

About a year later, a similar text-message lure reached Retool, and this time there was no hardware key at the end of the chain. One employee signed in on a fake identity portal. Then came a phone call from someone posing as IT, using what Retool describes as a deepfaked voice, and the employee read out one more one-time code. That code let the attacker add their own device to the employee's Okta account, which meant fresh MFA codes on demand from then on. The employee's Google Authenticator was also syncing codes to the cloud, so the hijacked Google account unlocked every other code as well. Retool's own conclusion was that its multi-factor setup had quietly collapsed into a single factor. By the time the company revoked every internal session, 27 cloud customer accounts had been taken over.

Look at which step did the real damage. It wasn't the phished password, and it wasn't really the phished code either. It was what one phishable code was allowed to do: register a brand-new authenticator owned by the attacker. That single permission turned a moment of access into persistence.

Then, in August 2025, Proofpoint published research by Yaniv Miron showing that accounts with FIDO credentials aren't automatically safe either. A dedicated Evilginx phishlet tells Microsoft Entra ID that the victim is browsing with Safari on Windows, a combination Entra ID doesn't support for FIDO sign-in. Entra ID does exactly what it was built to do: it shows an error and offers other methods. The victim picks the authenticator app, enters a code into the proxied page, and the attacker leaves with a session cookie. No cryptography was broken. The server simply believed a user agent string written by the attacker. Proofpoint said it hadn't seen the technique in the wild yet and that criminals still mostly go after accounts with weaker protection. That is the trend to plan for, because every passkey you ship makes the remaining fallbacks a more attractive target.

Your Real Security Level Is Your Weakest Path

Put those three stories side by side and a rule falls out that is easy to state and uncomfortable to apply: an account is only as phishing-resistant as the weakest path that can create a session for it or change its credentials. That covers login fallbacks, the flow that registers a new passkey or authenticator app, recovery by email or SMS, magic links, long-lived refresh tokens and whatever your support team can do from an admin panel. Most passkey rollouts add one strong path and leave every old one in place. That raises the average while leaving the minimum exactly where it was, and attackers only ever shop at the minimum.

Designing the Doors Around the Passkey

You can't delete every fallback, because people lose phones and drop laptops in lakes. What you can do is make each remaining door as hard to open as the account deserves:

Let the account decide, not the client. If an account has a passkey, a sign-in that reports "no WebAuthn here" should be steered to another phishing-resistant option, such as the cross-device flow where the user scans a QR code with their phone, or to a slower recovery path. It should not get a menu of codes to type. In a proxied session the attacker writes every header, the user agent included.
Gate enrollment like a wire transfer. Registering a new passkey, adding an authenticator app or changing the phone number should require the strongest factor already on the account. Retool's attacker turned one spoken code into lasting access because enrollment accepted a weak factor.
Treat a downgrade as an event. When an account that has used a passkey for months suddenly signs in with an SMS code from a new device, notify every enrolled device, require step-up before sensitive actions and consider a limited session until the user confirms from a trusted device.
Put time on recovery. A recovery flow that finishes in seconds is just a login flow with fewer checks. A waiting period, alerts to every channel on file and a one-tap "this wasn't me" give the real owner a chance to cancel.
Ask for a second passkey early. Most recovery requests happen because someone lost their only credential. A second passkey, ideally a hardware key or one stored in a different ecosystem, keeps people away from the weak door in the first place.
Verify the origin against a fixed allowlist. A page on any of your subdomains can request an assertion scoped to the parent domain, so the origin inside clientDataJSON is what separates your login page from, say, a user-uploads subdomain. Libraries compare it only with the values you pass in, so hardcode them rather than reading them from request headers.

Make the Strong Door the Easy One

None of this holds if the passkey path is a chore, because people route around friction and the fallback quietly becomes the main entrance. The encouraging part is that passkeys are now faster than what they replace. In Microsoft's World Passkey Day update, the company reported that passkey sign-ins succeed about 98% of the time compared with 32% for passwords, and finish eight times faster than a password plus MFA. Microsoft also stopped presenting every sign-in method up front. Its sign-in screen now leads with the best method an account already has, a change that cut password use by more than 20% in its experiments. That is the list above applied to UX: defaults decide which door people walk through.

On the web, the cheapest way to make passkeys the default is conditional mediation, which puts saved passkeys into the browser's autofill right next to saved passwords:

// Pair with: <input name="username" autocomplete="username webauthn">
async function offerPasskeyAutofill(challenge) {
  if (!window.PublicKeyCredential?.isConditionalMediationAvailable) return;
  if (!(await PublicKeyCredential.isConditionalMediationAvailable())) return;

  const assertion = await navigator.credentials.get({
    mediation: "conditional",
    publicKey: { challenge, rpId: "example.com", userVerification: "preferred" },
  });

  // Send the assertion to your server, which checks the challenge, origin and RP ID.
}

Call it when the page loads so the request is already waiting. Users who have a passkey see it the moment they focus the username field, and nobody gets pushed through a separate ceremony they don't understand.

Audit the Doors Before Someone Else Does

Here is a one-hour exercise for your next sprint. List every way a session can come into existence for one of your users and every way their credentials can change. Next to each entry, write down how it authenticates, then circle the weakest one. That line, not the passkey button on your login page, is your actual security level, and it is the line the next phishing kit will be written for. Cloudflare's three employees were fooled and the attack still failed, because the key was the only second factor its systems would accept. Build accounts where that is true, and a bad day for one of your users stays a bad day instead of turning into your breach notice.

If you've already shipped passkeys, how did you handle fallback and recovery? Tell me in the comments what worked and what you had to roll back.

Crypto's $19 Billion Night Was a Distributed Systems Bug

Sonia Bobrik — Sat, 10 Oct 2026 19:33:24 +0000

On October 10, 2025, crypto exchanges force-closed more than $19 billion of leveraged positions belonging to roughly 1.6 million traders, the largest liquidation event the market has ever recorded. Most coverage told it as a macro story about a tariff threat, a nervous Friday, and too much leverage. Strip out the tickers, though, and the timeline reads like an incident report any backend team would recognize: a positive feedback loop, a source of truth that lied under load, and an emergency mechanism that made things worse the first time it actually ran. A good lens for taking it apart is this research-driven guide to avoiding the most common crypto mistakes, which treats market, counterparty, and technical risk as separate layers. That night all three failed at once, and the seams between them failed hardest.

The Move Was Ordinary. The Cascade Was Not.

The trigger was a post from President Trump announcing an additional 100% tariff on Chinese imports plus export controls on critical software. Bitcoin, which had set a record above $125,000 earlier that week, fell more than 12%. That is a brutal afternoon, but not an unprecedented one, and analysts at 21Shares later noted that Bitcoin's drawdown was smaller than in earlier major deleveraging events. The record lived in the plumbing. Bloomberg's reporting on the liquidation wave described more than $7 billion of positions being sold in under an hour, and CoinGlass warned that even the record tally was probably an undercount, because Binance publishes only one liquidation order per second.

Think about what that means operationally. During the most violent minutes the market has ever seen, its busiest venue was emitting sampled telemetry. Anyone who has debugged an outage with dashboards that only show averages knows the result: the worst minute of the year is also the least visible one. It skews the leaderboard too. Hyperliquid, where every liquidation lands on a public chain, topped CoinGlass's charts with about $10.3 billion, a comparison that quietly flatters venues that report less.

Mechanically, a liquidation cascade is the textbook positive feedback loop. Falling prices push leveraged positions below their maintenance margin, the liquidation engine dumps that collateral into the order book, the selling pushes prices lower, and the next tier of positions crosses its threshold. It is a retry storm with money attached: every failure adds load to the exact component that caused it.

Bug One: The Source of Truth Was the Thing on Fire

The most instructive failure happened inside Binance's unified margin accounts, which let traders post assets such as Ethena's USDe and Binance's own staking tokens, wBETH and BNSOL, as collateral. The exchange valued that collateral largely from its own order books. As market makers pulled their quotes and transfers on and off the platform degraded, USDe printed as low as $0.65 on Binance while staying within a fraction of a percent of a dollar on other venues and on-chain. wBETH, a token backed by staked ether, briefly traded around $430 while ETH itself was above $3,800. Nothing had happened to the underlying assets. Something had happened to Binance's view of them, and positions that had lost no real value anywhere else were liquidated anyway.

Every backend engineer has met this bug. It is the health check that asks the overloaded node whether it is healthy, or the cache that becomes the source of truth in the middle of a stampede. On a normal day, arbitrageurs behave like a reconciliation job, buying the cheap copy on one venue and redeeming it elsewhere until prices converge. That night the reconciliation job could not run, because moving funds on and off the exchange was exactly what had degraded. In some thin markets the price became whatever happened to be left in the book, and Binance later blamed several of the wildest wicks on stale limit orders dating back to 2019.

The part that stings is the timing. On October 6, Binance had announced that from October 14 it would start factoring staking conversion ratios into how it priced wBETH and BNSOL. The fix was already sitting in the release train when the incident arrived. Afterward the exchange paid about $283 million in compensation, mixed redemption prices into its index, and added a soft price floor for USDe. The general lesson is the one you would apply to any critical read path: use independent sources, and treat disagreement between them as an incident rather than as a price.

type Quote = { source: string; price: number };

// Never let the component under stress be the only witness to its own health.
export function markPrice(quotes: Quote[], maxDeviation = 0.02) {
  if (quotes.length < 3) {
    return { price: null, pauseLiquidations: true, reason: "too few independent sources" };
  }

  const sorted = quotes.map((q) => q.price).sort((a, b) => a - b);
  const median = sorted[Math.floor(sorted.length / 2)];
  const divergent = quotes.filter((q) => Math.abs(q.price - median) / median > maxDeviation);

  // One venue printing $0.65 while the rest print $1.00 is an incident, not a price.
  if (divergent.length > 0) {
    const names = divergent.map((q) => q.source).join(", ");
    return { price: median, pauseLiquidations: true, reason: `divergent: ${names}` };
  }

  return { price: median, pauseLiquidations: false, reason: "ok" };
}

Pausing has failure modes of its own. Someone who can push a single venue far enough might try to freeze liquidations on purpose, so a real system pairs a check like this with alerting, time limits, and a human decision. But a paused liquidation can be resumed. A wrongful one can only be apologized for.

Bug Two: The Safety Valve Had Never Been Opened

Perpetual futures venues keep one last mechanism for days when liquidations cannot keep up: auto-deleveraging, or ADL. When a losing account's margin is exhausted and the insurance backstop cannot absorb the hole, the venue closes positions on the winning side, starting with the most profitable and most leveraged, so the books still balance. On Hyperliquid, that Friday was the first time cross-margin ADL had fired in more than two years of operation. A later paper by Tarun Chitra of Gauntlet found that it closed about $2.1 billion of positions in twelve minutes.

The trouble is who stands at the front of that queue. Many of the winning positions belonged to market makers and funds whose profitable shorts were hedges for exposure held somewhere else. ADL closed one leg and left the other standing, so books that had been neutral turned directional in the middle of a crash. Some firms rushed to reopen shorts to get back to neutral, adding fresh selling to a falling market, while others pulled back from quoting altogether. It is the infrastructure equivalent of a load shedder that evicts your healthiest tenants first.

The deeper lesson is about code paths. Logic that runs once every two years is the least exercised code in your system, and nobody on call has ever watched it run in production. If your design leans on an emergency path, that path deserves game days and replays against realistic data, not just a paragraph in the docs.

Eight Months Earlier, the Same Lesson Cost $1.5 Billion

October was a lesson about trusting the wrong source of data. February had already taught the same lesson about trusting the wrong source of code. On February 21, 2025, attackers drained more than 400,000 ETH from a Bybit cold wallet, and five days later the FBI attributed the roughly $1.5 billion theft to North Korean operators it tracks as TraderTraitor. Chainalysis later put total crypto theft for all of 2025 at $3.4 billion, which means this single incident accounted for more than 40% of the year.

Bybit's own infrastructure was never breached. The wallet was a Safe multisig, and investigators traced the compromise to a Safe developer's laptop that had been infected through social engineering more than two weeks earlier. Using stolen cloud credentials, the attackers modified JavaScript served from the S3 bucket behind the Safe web interface. The injected code activated only for Bybit's cold wallet and one test address, showed the signers what looked like a routine transfer, and passed their hardware wallets a different payload: a delegatecall to a pre-deployed contract that overwrote the wallet's implementation pointer. About two minutes after the theft, the original files were put back.

Two truths fall out of that. First, the presentation layer belongs to your trusted computing base. The hardware wallets did their job perfectly, which is to sign exactly what they are handed. Second, N-of-M approval only means something when the N paths are independent. Three signatures produced through one compromised browser bundle are, for security purposes, one signature.

Three Layers, One Graph

The guide linked at the top is right that most people stuff crypto risk into a single bucket labeled "price." October 10 shows why separating the layers is only half the work. A market shock became a counterparty failure, one exchange mispricing its own collateral, through a technical design choice, and a second technical mechanism then turned it into a liquidity problem for everyone else. The layers are coupled, and the couplings are where the money disappears. Whether you hold crypto, build on it, or run systems that merely rhyme with it, the same review questions apply:

Name your source of truth. For every number that can trigger an irreversible action, such as a liquidation, a signature, or a withdrawal, write down where it comes from and ask whether that source is the thing most likely to be on fire when you need it.
Count independence, not replicas. Three signers on one front end, five oracles fed by one venue, and two "separate" wallets behind one cloud account each count as one.
Rehearse the rare path. Auto-deleveraging, emergency pauses, key rotation, and withdrawal halts deserve game days precisely because they almost never run.
Verify what you sign out of band. Decode the payload on a separate machine, recompute the transaction hash, compare it with the hardware wallet's screen, and refuse any delegatecall to a contract that is not on an explicit allowlist.
Assume your telemetry lies under load. If a dashboard samples one event per second, the worst minute of the year will be the one you can see least.
Size for the cascade, not the move. A 12% drop with a feedback loop behind it does not behave like a 12% drop.

The Boring Part Is the Point

A year on, the uncomfortable thing about October 10 is how little of it was exotic. There was no zero-day, no broken cryptography, no consensus failure, and the major chains kept producing blocks the entire time. What broke was the ordinary scaffolding around them: a price feed that read from its own wreckage, a reconciliation path that went dark when it was needed most, a safety valve nobody had seen open, and a dashboard that sampled its way through the storm. The Bybit heist had the same shape, a web front end trusted far more than it deserved.

That is good news for developers, because these are failure modes we already know how to engineer against. The next trigger might be a tariff, a hack, or a stablecoin wobble at 3 a.m. on a Sunday. Whether it becomes another $19 billion night will depend on the same decisions that decide whether a bad deploy becomes an outage: where your truth comes from, how independent your safeguards really are, and whether anyone has run the emergency path before the emergency.

The 21st Field: What Security Vendors' Worst Days Teach Developers

Sonia Bobrik — Sat, 03 Oct 2026 19:25:35 +0000

At the end of February 2024, CrowdStrike shipped a new template for its Falcon sensor that declared 21 input fields, while the code feeding it supplied only 20. Nobody noticed for almost five months, because every rule written against the template used a wildcard for the last slot. Then, at 04:09 UTC on July 19, a rule arrived that actually checked field 21, and a kernel driver on millions of Windows machines tried to read a 21st value from an array that held 20. The financial aftermath of that morning, and of the Okta, F5 and SonicWall breaches on either side of it, is traced in When the Security Vendor Becomes the Incident, a case study of the bills that investors, customers and courts sent afterward. The engineering story underneath those headlines is just as instructive, because nearly every root cause in it has a twin in ordinary application code.

Security products are the code we trust most and inspect least. They run with kernel or administrator rights, update on someone else's schedule and sit in the path of every boot, login and packet. Read their post-mortems back to back and three patterns keep returning. None of them is exotic, and all three probably exist in the repository you will open tomorrow.

Configuration Is Code That Skipped the Pipeline

CrowdStrike customers could stage and pin new sensor versions. Rapid Response Content, the behavioral rules the sensor pulls down between releases, was treated as data, and data went to every online host at once. Two safety nets failed on the way. The Content Validator checked the new rules against the template's definition of 21 fields, so it faithfully confirmed the wrong contract, and the interpreter had no runtime bounds check to catch what the validator missed. For many machines, recovery meant booting into Safe Mode or the Windows Recovery Environment, often typing in a BitLocker recovery key, and deleting one channel file by hand.

Microsoft's own estimate put the affected devices at less than one percent of all Windows machines. Airlines still grounded flights and hospitals still postponed non-urgent care, because, as Microsoft noted, CrowdStrike is used by enterprises that run many critical services. Fleet percentage is a comforting number and an almost useless one: blast radius depends on which machines fail, not how many. Microsoft's longer answer has been architectural. It has been working with CrowdStrike and other vendors on a Windows endpoint security platform that lets antivirus and EDR products run in user mode, outside the kernel, and announced a private preview for partners in mid-2025.

If that sounds like a kernel-driver problem, Cloudflare's outage on November 18, 2025 says otherwise. At 11:05 UTC, a deliberate ClickHouse permissions change made a metadata query, one that never filtered by database name, start returning duplicate rows. That query builds the feature file for Cloudflare's Bot Management model, which is regenerated every five minutes and pushed to the whole network. The file doubled in size and blew through a hard cap of 200 features, a limit set well above normal use because the proxy preallocates memory for them. The new Rust proxy, FL2, checked the count, got an error back and called unwrap() on it, and the resulting panic surfaced as HTTP 5xx errors across Cloudflare's network. Customers still on the older proxy got no 5xx errors; instead every request received a bot score of zero, so sites with bot-blocking rules began turning away real visitors. Because only part of the database cluster had the change at first, good and bad files alternated, the network kept failing and recovering, and the team's first suspicion was a massive DDoS attack. Cloudflare called it its worst outage since 2019.

Put the two incidents side by side and the lesson gets sharper. CrowdStrike's interpreter had no bounds check and read past the end of an array. Cloudflare's code had the check, in a memory-safe language, and the network went down anyway, because the only response it had to a failed check was to crash. Validation is half the job. The other half is deciding what happens to a bad file once you catch it, and "keep serving with the last good version" beats "panic" almost every time. Sixteen months apart, both failures had the same shape: a file produced by the vendor's own tooling and trusted because it came from inside, a consumer that assumed the file's shape, and a distribution system built for speed.

The first item on Cloudflare's remediation list is to ingest its own generated configuration files with the same suspicion it applies to user input. Feature flags, WAF rules, ML feature lists and policy bundles all belong in that bucket, and plenty of teams still ship them through a faster, less guarded lane than code.

The Front Door Was Somebody's Browser Profile

Okta's October 2023 breach did not begin with an exploit. A service account for its customer support system had its username and password saved in an employee's personal Google profile, signed into Chrome on a company laptop, and Okta concluded that the credential most likely leaked through that personal account or device. Inside the support system sat HAR files, the recordings of browser sessions that customers upload so engineers can reproduce a bug. Some still held live session tokens, and the attacker used them to hijack the sessions of five customers. Okta's fixes read like a checklist for any SaaS team: personal profiles blocked in managed Chrome, and administrator session tokens bound to network location so a stolen one stops working from somewhere else. Within a week of Okta's disclosure, Cloudflare, one of the targeted customers, open-sourced a HAR sanitizer that strips session cookies and JSON Web Tokens in the browser before a file is shared.

LastPass learned the same lesson at a higher price. After a 2022 intrusion into its development environment, attackers went after the home computer of a senior DevOps engineer, one of only four engineers who could reach the decryption keys for the company's production backups. A vulnerable third-party media package on that machine gave them remote code execution. A keylogger captured the master password as it was typed, after multi-factor authentication had already succeeded, and the corporate vault behind it held the keys to backups stored in Amazon S3. Those backups held copies of customer vault data.

Neither attack broke any cryptography. Both walked through convenience: browser sync, a home media app, a debugging file with a live cookie inside. If an engineer can reach production from the same machine that runs weekend side projects, that machine is part of your threat model whether or not your architecture diagram shows it.

Your Vendor's Build Room Is Part of Your Attack Surface

F5 found intruders in its network on August 9, 2025. When it disclosed the breach in October, it said a nation-state actor had held long-term, persistent access to its BIG-IP product development environment and engineering knowledge management platform, and had taken portions of BIG-IP source code along with details of vulnerabilities its engineers were still working on. F5 reported no evidence of tampering with its build and release pipelines. The risk was quieter than a poisoned update: an adversary holding the source and the bug backlog can find and weaponize flaws faster than customers can patch them.

CISA's Emergency Directive 26-01 turned that risk into concrete work for federal agencies: inventory every BIG-IP product, check whether management interfaces are reachable from the public internet, install F5's updates within about a week, and disconnect devices that have reached end of support. Nothing on that list requires access to F5's systems. It is the half of a vendor breach that customers control.

SonicWall's 2025 incident makes the same point from the storage side. In September, the company found that attackers had accessed firewall configuration files stored in its MySonicWall cloud backup service, and in November it attributed the intrusion to a state-sponsored actor. The credentials inside the files were encrypted, yet the files still described how each firewall was set up, which SonicWall itself warned could make targeted attacks easier, and its remediation guidance had customers reset admin passwords, VPN pre-shared keys and TOTP bindings anyway. A configuration backup is a secrets file with a friendlier extension.

What to Change Before Your Next Deploy

None of these vendors were careless amateurs, which is exactly why their post-mortems are worth stealing from. Here is what they add up to for a team shipping its own software:

Map your privileged dependencies. List every agent, driver, browser extension and appliance that runs with kernel, root or admin rights and updates itself, and write down who controls its update schedule.
Turn on update rings. If a vendor lets you stage content updates as well as agent versions, put a few non-critical machines in the first ring and let them soak before the rest follow.
Distrust your own config. Validate schema, field counts and size where a file is consumed, keep the last known-good version, fall back to it instead of crashing, and give every fast-moving feature a kill switch.
Price out the bad-boot day. Know where recovery keys live, test out-of-band console access, and time how long it takes to fix ten machines by hand.
Scrub debug artifacts. Strip cookies, tokens and authorization headers from HAR files and logs before they reach any support portal, and keep session tokens short-lived and bound to a device or network where your stack allows it.
Shrink the management plane. Keep admin interfaces off the public internet and patch edge appliances on a clock measured in days, not quarters.
Rotate after every vendor breach. Include tokens, API keys and config secrets you believe are idle, since those are the ones nobody remembers to change.

Find Your 21st Field First

Every company in these stories sells protection, and each one became, for a while, the thing its customers needed protecting from. That is not an argument for ripping out endpoint agents or identity providers, since a fleet without them is worse off. It is an argument for designing as if the most trusted code in your stack will someday misbehave: bounded inputs, a safe answer for when a bound is crossed, staged rollouts, scrubbed debug files, short-lived credentials and a recovery path someone has actually walked. Somewhere in your own system a template expects 21 fields and a caller sends 20. It is far cheaper to find it on a quiet Tuesday in a canary ring than at 04:09 on a Friday, on every machine you own.

The Most Expensive Code You Ship Is the Admin Panel Nobody Reviews

Sonia Bobrik — Sat, 03 Oct 2026 18:51:31 +0000

Every product has a second application hiding behind it. It has no designer, no load tests and no on-call rotation, and its security model often amounts to a boolean called is_admin. Support agents use it to look up customers, reset passwords and issue refunds, a few service accounts talk to it overnight, and nobody has read its pull requests with real curiosity since the sprint it was built in. Trace the costliest incidents of the past few years back to their first step and a surprising number start in exactly this kind of software, while the damage rarely stays inside IT: a recent analysis of the financial impact of a cyberattack shows how quickly an intrusion becomes a cash flow and credit problem for the entire company. This post looks at the other end of that chain, the unglamorous internal tooling where the money starts leaking, and at what developers can change in it before the next quarter closes.

Follow the Receipts

Start with the cleanest example. In May 2025 Coinbase disclosed that criminals had bribed support contractors and employees outside the United States to pull customer records out of the company's own support tools. There was no exploit and no malware. The insiders used access they legitimately had to collect names, home addresses, government ID images, masked bank details and account balances for 69,461 customers, and the buyers used that data to phone victims while posing as Coinbase. The company refused a $20 million extortion demand, then booked $307 million in expenses tied to the incident in a single quarter. The detail worth sitting with is that Coinbase's own monitoring had flagged support staff accessing data without a business need months before the extortion email arrived. The signal existed. A console that never asks why a record is being opened can only report abuse after it has happened.

The help desk is the same story told over the phone. According to Bloomberg's reporting on the MGM Resorts breach, the 2023 attack that disrupted MGM's resorts and casinos across the US began when attackers talked their way past the company's IT service desk. MGM later estimated a hit of roughly $100 million to its adjusted property EBITDAR, plus under $10 million in one-time costs. Clorox describes a similar opening move in the $380 million lawsuit it filed against Cognizant in July 2025, alleging that outsourced help desk agents reset passwords and MFA for an attacker without checking who was calling. Cognizant rejects the claims and says it was hired for a narrow scope of work. CISA and the FBI have described the same playbook in their joint advisory on Scattered Spider: impersonate an employee, call the help desk, walk away with a fresh password and a new MFA device. Whoever wins the lawsuit, the attack path was a reset workflow, and somebody designed that workflow.

Then there are the accounts that aren't people. In 2023 an attacker entered Okta's customer support system with a service account that could view and update support cases. Okta's own root cause analysis found that the account's username and password had been saved to an employee's personal Google profile, signed in on a company laptop. Inside the support system sat HAR files that customers had uploaded for troubleshooting, session tokens included. The attacker reached files belonging to 134 customers and used them to hijack sessions at several of them. Okta's stock closed down about 11.5% on the day of disclosure, and more than $2 billion of market value was gone within two trading sessions.

Finally, the integrations. In 2025 attackers spent months inside a Salesloft GitHub account, moved into the AWS environment behind its Drift chat product and left with the OAuth tokens that Drift customers had granted to connect it with Salesforce. Google's threat intelligence team said it was aware of at least 700 affected organizations. Because many of them used the integration for customer support, much of what leaked came from support tickets, and support tickets are where people paste things they shouldn't. Salesloft said the attackers pulled secrets such as AWS access keys, passwords and Snowflake tokens out of that data.

Why Attackers Shop in the Back Office

None of this is bad luck. Internal tools concentrate privilege by design: they exist to do what the product deliberately won't let a user do, such as view anyone's data, change anyone's email, reset anyone's MFA or move anyone's money. They are run by big, frequently outsourced teams, which means more people who can be tricked, exhausted or bought. In the help desk's case, they are authenticated by a human listening to a voice on a phone line. And they rarely get the threat modeling that customer-facing code receives, because "only staff can reach it" feels like a security control when it is really a fact about the network.

The economics are moving in the wrong direction. IBM's analysis of its 2026 Cost of a Data Breach findings puts the global average breach at a record $4.99 million and notes that fewer than half of organizations are actively securing non-human identities, the service accounts and tokens at the heart of the Okta and Drift stories. IBM's release on the same report adds that one in four malicious breaches is now AI-enabled, mostly through deepfake impersonation and AI-enabled malware, at an average cost of about $6 million. Put plainly, the caller asking your help desk for an MFA reset may not be a person at all, and "the agent recognized the voice" no longer counts as verification.

Treat the Back Office Like Production

The fixes are not exotic. Most of them are ordinary engineering work that never reached a roadmap because the tool was "internal":

Make every lookup carry a reason. Tie record access to an open ticket or case ID, mask sensitive fields by default and log every reveal. Bulk harvesting becomes slow, noisy and attributable.
Put budgets on human queries. Rate limits shouldn't stop at the public API. Alert when an agent opens far more accounts than their queue explains, or starts browsing customers nobody assigned to them.
Never let a voice authorize a reset. MFA resets, email and phone changes and payout address updates should require a push to an already enrolled device, a callback to a number on file or a second approver, and the UI should make skipping that step impossible rather than merely discouraged.
Give machine credentials an expiry date. Service accounts and OAuth grants need narrow scopes, short lifetimes, a named owner and a place in an inventory. A token that has worked for two years without anyone touching it is a finding, not a convenience.
Scrub secrets at the door. Strip cookies and authorization headers from HAR files on upload, scan ticket bodies for keys and expire attachments when a case closes.
Log for the worst day. Keep audit trails the support organization cannot edit, store them where a stolen session cannot reach, and make "who viewed this customer last month" a query that returns in seconds.

None of these items requires a new vendor. They require someone to open the admin panel's repository with the same suspicion they bring to the login page.

Pricing the Fix

The hard part is rarely the code. It is convincing a roadmap that internal tooling deserves engineering time when no customer will ever notice the change. Money is the argument that lands, and the incidents above make it unusually easy to price. Ticket-scoped access, per-agent query budgets and a hardened reset flow are measured in sprints, not years. Set that against $307 million in one quarter at Coinbase or the $380 million claim Clorox took to court. IBM's data supplies a second lever: detection and escalation together with lost business make up 63% of breach costs, so an audit trail that answers questions in seconds is not paperwork. It is a direct cut to the most expensive line items.

Phrase the risk the way finance already thinks. "The support console has broad read access" gets a polite nod and is forgotten by Friday. "One bribed contractor could export our customer list, and the last company this happened to spent $307 million cleaning up" gets a ticket scheduled. One sentence like that usually does more than a twelve-page threat model.

The attackers in these stories did not need a zero-day. They needed a search box that asked no questions, a reset button that trusted a voice, a token nobody remembered issuing and a ticket queue full of pasted secrets. Every one of those was built on purpose by someone shipping under a deadline, which means the people best placed to take them apart are the developers reading this, ideally before an extortion email does the code review for them.

Your Podcast Habit Is Read-Only. Here's How to Give It a Write Path

Sonia Bobrik — Sat, 03 Oct 2026 18:51:00 +0000

Plenty of developers keep a podcast queue longer than their sprint backlog, and they burn through it the same way: earbuds in, playback at 1.5x, one eye on Slack. Here is a test worth running on yourself. Take any episode you finished last week, whether it was a three-hour deep dive into database internals or the podcast episode you saved for your commute, and try to explain its three strongest ideas to a teammate without pressing play again. Most people recover one idea and a warm sense that it was good. That is not laziness. It is what happens when you treat audio as a read-only stream: bytes flow in, nothing gets written to disk, and the buffer is cleared by Thursday.

Why Listening Feels Like Learning

Good podcasts are engineered to feel effortless. A skilled host asks the question you were about to ask, the guest answers with a crisp story, and the whole thing lands with the satisfying click of an idea that suddenly seems obvious. That click is the trap. Fluency feels like understanding, and the two are not the same thing.

Harvard researchers measured this gap directly. For two weeks of an introductory physics course, students alternated between polished lectures from an experienced instructor and active-learning sessions that covered identical material. As the Harvard Gazette's write-up of the study explains, students felt they had learned more from the lectures, yet they scored higher on the tests that followed the active sessions. Lead author Louis Deslauriers put it bluntly: "Deep learning is hard work." The extra effort of the active sessions felt like confusion, so students mistook it for learning less.

A well-produced podcast is the most fluent lecture ever invented, and it carries a handicap that a page of text does not: it is strictly linear. When a guest explains that they partitioned by tenant, pushed hot writes onto a queue and rebuilt reads from an event log, the next sentence arrives before you have finished drawing the diagram in your head. With a blog post you would scroll back and stare at that paragraph. With audio, the stream keeps moving and your half-built mental model quietly gets overwritten.

Speed Is Not the Real Problem

The usual suspect is playback speed, and the evidence mostly clears it. In a study published in Applied Cognitive Psychology, a UCLA team had 231 students watch lecture videos at normal speed, 1.5x, 2x or 2.5x, with no pausing and no notes. Comprehension at 1.5x and 2x matched normal speed, both right away and a week later; only 2.5x clearly hurt. Eighty-five percent of the students the team surveyed already sped up their lectures. The most useful finding was a different one: watching twice at 2x beat watching once at normal speed, but only when the second pass was saved for later, right before the test, instead of coming straight after the first.

That study used video rather than pure audio, so treat it as a strong hint rather than a law. The hint is still valuable. Speeding up is fine. What matters is what you do with the time you save, and the best use is a spaced second pass over the parts that mattered.

Add a Write Path: Retrieval Beats Replay

If listening is a read operation, the write operation is retrieval: pulling an idea back out of your own head without looking. Psychologists have studied this for decades, and the results are lopsided. At Purdue, Jeffrey Karpicke and Janell Blunt had 200 students read science texts and then either build detailed concept maps or simply set the text aside and write down everything they could recall. As The New York Times reported on the experiment, the recall group retained about 50 percent more a week later. Just as telling, students were poor judges of which method was actually working for them.

There is also a clock running. Hermann Ebbinghaus showed back in the 1880s that unrehearsed memories fade fastest in the first hours and days after you meet them. An idea you have not retrieved by the end of the day is an idea you will probably have to hear again.

A Post-Episode Protocol That Takes Ten Minutes

None of this requires a note-taking system with seventeen plugins. It requires a small habit loop wrapped around the listening you already do:

Ask a question before you press play. "What would change how we do code review?" turns a passive stream into a search query, and your attention starts filtering for answers.
Bookmark instead of pausing. Most modern players let you mark a timestamp; Podcast Addict, for example, supports bookmarks with notes, so you can flag the moment a guest explains their migration plan and keep walking.
Do a two-minute brain dump within the hour. Close the app, write the three ideas you remember, then compare them with the show notes. The gaps tell you exactly which segment to replay.
Turn one idea into code within 48 hours. Heard about property-based testing? Write one property for a parser you own. Heard about feature flags? Put one risky change behind a flag. A twenty-line spike teaches more than twenty pages of notes.
Schedule the second pass. A week later, reread your dump and replay only the bookmarked segments, at 2x if you like.
Teach it to someone. A three-sentence summary in your team channel forces you to compress the idea, and compression is where understanding gets tested.

If the blank page stalls you, give the brain dump a fixed shape: one claim that surprised you, one you would argue with, and one thing you will try this week. The argument is the most valuable line, because disagreeing with a guest in writing is retrieval with a sharper edge.

Fewer Shows, Better Listening

There is also a curation problem. A queue of forty unplayed episodes is not a learning plan; it is a to-do list that quietly generates guilt. Pick two or three shows whose ideas you would actually act on, and let everything else be entertainment. That is a perfectly good use of a podcast. Folding laundry to the story of a legendary outage counts as fun, and fun is allowed.

Just be honest about which mode you are in. Learning mode needs at least part of your attention, which is why a dense conversation about distributed consensus will not survive being played under a debugging session. If you catch yourself rewinding the same thirty seconds three times, pause the episode, not the work you are paid for.

The Episode Is the Input, Not the Output

Podcasts are one of the best deals in a developer's learning budget: free, portable, and full of practitioners describing failures nobody puts in the docs. But the value does not transfer just because the audio played. It transfers when you retrieve, write and build. So switch off continuous playback in your player and let the silence after an episode become your cue. Spend two minutes writing down what you remember, pick one thing to try, and put it on tomorrow's list. Your queue will get shorter, and your head will finally start keeping what passes through it.