DEV Community: Sonia Bobrik

The 21st Field: What Security Vendors' Worst Days Teach Developers

Sonia Bobrik — Sat, 03 Oct 2026 19:25:35 +0000

At the end of February 2024, CrowdStrike shipped a new template for its Falcon sensor that declared 21 input fields, while the code feeding it supplied only 20. Nobody noticed for almost five months, because every rule written against the template used a wildcard for the last slot. Then, at 04:09 UTC on July 19, a rule arrived that actually checked field 21, and a kernel driver on millions of Windows machines tried to read a 21st value from an array that held 20. The financial aftermath of that morning, and of the Okta, F5 and SonicWall breaches on either side of it, is traced in When the Security Vendor Becomes the Incident, a case study of the bills that investors, customers and courts sent afterward. The engineering story underneath those headlines is just as instructive, because nearly every root cause in it has a twin in ordinary application code.

Security products are the code we trust most and inspect least. They run with kernel or administrator rights, update on someone else's schedule and sit in the path of every boot, login and packet. Read their post-mortems back to back and three patterns keep returning. None of them is exotic, and all three probably exist in the repository you will open tomorrow.

Configuration Is Code That Skipped the Pipeline

CrowdStrike customers could stage and pin new sensor versions. Rapid Response Content, the behavioral rules the sensor pulls down between releases, was treated as data, and data went to every online host at once. Two safety nets failed on the way. The Content Validator checked the new rules against the template's definition of 21 fields, so it faithfully confirmed the wrong contract, and the interpreter had no runtime bounds check to catch what the validator missed. For many machines, recovery meant booting into Safe Mode or the Windows Recovery Environment, often typing in a BitLocker recovery key, and deleting one channel file by hand.

Microsoft's own estimate put the affected devices at less than one percent of all Windows machines. Airlines still grounded flights and hospitals still postponed non-urgent care, because, as Microsoft noted, CrowdStrike is used by enterprises that run many critical services. Fleet percentage is a comforting number and an almost useless one: blast radius depends on which machines fail, not how many. Microsoft's longer answer has been architectural. It has been working with CrowdStrike and other vendors on a Windows endpoint security platform that lets antivirus and EDR products run in user mode, outside the kernel, and announced a private preview for partners in mid-2025.

If that sounds like a kernel-driver problem, Cloudflare's outage on November 18, 2025 says otherwise. At 11:05 UTC, a deliberate ClickHouse permissions change made a metadata query, one that never filtered by database name, start returning duplicate rows. That query builds the feature file for Cloudflare's Bot Management model, which is regenerated every five minutes and pushed to the whole network. The file doubled in size and blew through a hard cap of 200 features, a limit set well above normal use because the proxy preallocates memory for them. The new Rust proxy, FL2, checked the count, got an error back and called unwrap() on it, and the resulting panic surfaced as HTTP 5xx errors across Cloudflare's network. Customers still on the older proxy got no 5xx errors; instead every request received a bot score of zero, so sites with bot-blocking rules began turning away real visitors. Because only part of the database cluster had the change at first, good and bad files alternated, the network kept failing and recovering, and the team's first suspicion was a massive DDoS attack. Cloudflare called it its worst outage since 2019.

Put the two incidents side by side and the lesson gets sharper. CrowdStrike's interpreter had no bounds check and read past the end of an array. Cloudflare's code had the check, in a memory-safe language, and the network went down anyway, because the only response it had to a failed check was to crash. Validation is half the job. The other half is deciding what happens to a bad file once you catch it, and "keep serving with the last good version" beats "panic" almost every time. Sixteen months apart, both failures had the same shape: a file produced by the vendor's own tooling and trusted because it came from inside, a consumer that assumed the file's shape, and a distribution system built for speed.

The first item on Cloudflare's remediation list is to ingest its own generated configuration files with the same suspicion it applies to user input. Feature flags, WAF rules, ML feature lists and policy bundles all belong in that bucket, and plenty of teams still ship them through a faster, less guarded lane than code.

The Front Door Was Somebody's Browser Profile

Okta's October 2023 breach did not begin with an exploit. A service account for its customer support system had its username and password saved in an employee's personal Google profile, signed into Chrome on a company laptop, and Okta concluded that the credential most likely leaked through that personal account or device. Inside the support system sat HAR files, the recordings of browser sessions that customers upload so engineers can reproduce a bug. Some still held live session tokens, and the attacker used them to hijack the sessions of five customers. Okta's fixes read like a checklist for any SaaS team: personal profiles blocked in managed Chrome, and administrator session tokens bound to network location so a stolen one stops working from somewhere else. Within a week of Okta's disclosure, Cloudflare, one of the targeted customers, open-sourced a HAR sanitizer that strips session cookies and JSON Web Tokens in the browser before a file is shared.

LastPass learned the same lesson at a higher price. After a 2022 intrusion into its development environment, attackers went after the home computer of a senior DevOps engineer, one of only four engineers who could reach the decryption keys for the company's production backups. A vulnerable third-party media package on that machine gave them remote code execution. A keylogger captured the master password as it was typed, after multi-factor authentication had already succeeded, and the corporate vault behind it held the keys to backups stored in Amazon S3. Those backups held copies of customer vault data.

Neither attack broke any cryptography. Both walked through convenience: browser sync, a home media app, a debugging file with a live cookie inside. If an engineer can reach production from the same machine that runs weekend side projects, that machine is part of your threat model whether or not your architecture diagram shows it.

Your Vendor's Build Room Is Part of Your Attack Surface

F5 found intruders in its network on August 9, 2025. When it disclosed the breach in October, it said a nation-state actor had held long-term, persistent access to its BIG-IP product development environment and engineering knowledge management platform, and had taken portions of BIG-IP source code along with details of vulnerabilities its engineers were still working on. F5 reported no evidence of tampering with its build and release pipelines. The risk was quieter than a poisoned update: an adversary holding the source and the bug backlog can find and weaponize flaws faster than customers can patch them.

CISA's Emergency Directive 26-01 turned that risk into concrete work for federal agencies: inventory every BIG-IP product, check whether management interfaces are reachable from the public internet, install F5's updates within about a week, and disconnect devices that have reached end of support. Nothing on that list requires access to F5's systems. It is the half of a vendor breach that customers control.

SonicWall's 2025 incident makes the same point from the storage side. In September, the company found that attackers had accessed firewall configuration files stored in its MySonicWall cloud backup service, and in November it attributed the intrusion to a state-sponsored actor. The credentials inside the files were encrypted, yet the files still described how each firewall was set up, which SonicWall itself warned could make targeted attacks easier, and its remediation guidance had customers reset admin passwords, VPN pre-shared keys and TOTP bindings anyway. A configuration backup is a secrets file with a friendlier extension.

What to Change Before Your Next Deploy

None of these vendors were careless amateurs, which is exactly why their post-mortems are worth stealing from. Here is what they add up to for a team shipping its own software:

Map your privileged dependencies. List every agent, driver, browser extension and appliance that runs with kernel, root or admin rights and updates itself, and write down who controls its update schedule.
Turn on update rings. If a vendor lets you stage content updates as well as agent versions, put a few non-critical machines in the first ring and let them soak before the rest follow.
Distrust your own config. Validate schema, field counts and size where a file is consumed, keep the last known-good version, fall back to it instead of crashing, and give every fast-moving feature a kill switch.
Price out the bad-boot day. Know where recovery keys live, test out-of-band console access, and time how long it takes to fix ten machines by hand.
Scrub debug artifacts. Strip cookies, tokens and authorization headers from HAR files and logs before they reach any support portal, and keep session tokens short-lived and bound to a device or network where your stack allows it.
Shrink the management plane. Keep admin interfaces off the public internet and patch edge appliances on a clock measured in days, not quarters.
Rotate after every vendor breach. Include tokens, API keys and config secrets you believe are idle, since those are the ones nobody remembers to change.

Find Your 21st Field First

Every company in these stories sells protection, and each one became, for a while, the thing its customers needed protecting from. That is not an argument for ripping out endpoint agents or identity providers, since a fleet without them is worse off. It is an argument for designing as if the most trusted code in your stack will someday misbehave: bounded inputs, a safe answer for when a bound is crossed, staged rollouts, scrubbed debug files, short-lived credentials and a recovery path someone has actually walked. Somewhere in your own system a template expects 21 fields and a caller sends 20. It is far cheaper to find it on a quiet Tuesday in a canary ring than at 04:09 on a Friday, on every machine you own.

The Most Expensive Code You Ship Is the Admin Panel Nobody Reviews

Sonia Bobrik — Sat, 03 Oct 2026 18:51:31 +0000

Every product has a second application hiding behind it. It has no designer, no load tests and no on-call rotation, and its security model often amounts to a boolean called is_admin. Support agents use it to look up customers, reset passwords and issue refunds, a few service accounts talk to it overnight, and nobody has read its pull requests with real curiosity since the sprint it was built in. Trace the costliest incidents of the past few years back to their first step and a surprising number start in exactly this kind of software, while the damage rarely stays inside IT: a recent analysis of the financial impact of a cyberattack shows how quickly an intrusion becomes a cash flow and credit problem for the entire company. This post looks at the other end of that chain, the unglamorous internal tooling where the money starts leaking, and at what developers can change in it before the next quarter closes.

Follow the Receipts

Start with the cleanest example. In May 2025 Coinbase disclosed that criminals had bribed support contractors and employees outside the United States to pull customer records out of the company's own support tools. There was no exploit and no malware. The insiders used access they legitimately had to collect names, home addresses, government ID images, masked bank details and account balances for 69,461 customers, and the buyers used that data to phone victims while posing as Coinbase. The company refused a $20 million extortion demand, then booked $307 million in expenses tied to the incident in a single quarter. The detail worth sitting with is that Coinbase's own monitoring had flagged support staff accessing data without a business need months before the extortion email arrived. The signal existed. A console that never asks why a record is being opened can only report abuse after it has happened.

The help desk is the same story told over the phone. According to Bloomberg's reporting on the MGM Resorts breach, the 2023 attack that disrupted MGM's resorts and casinos across the US began when attackers talked their way past the company's IT service desk. MGM later estimated a hit of roughly $100 million to its adjusted property EBITDAR, plus under $10 million in one-time costs. Clorox describes a similar opening move in the $380 million lawsuit it filed against Cognizant in July 2025, alleging that outsourced help desk agents reset passwords and MFA for an attacker without checking who was calling. Cognizant rejects the claims and says it was hired for a narrow scope of work. CISA and the FBI have described the same playbook in their joint advisory on Scattered Spider: impersonate an employee, call the help desk, walk away with a fresh password and a new MFA device. Whoever wins the lawsuit, the attack path was a reset workflow, and somebody designed that workflow.

Then there are the accounts that aren't people. In 2023 an attacker entered Okta's customer support system with a service account that could view and update support cases. Okta's own root cause analysis found that the account's username and password had been saved to an employee's personal Google profile, signed in on a company laptop. Inside the support system sat HAR files that customers had uploaded for troubleshooting, session tokens included. The attacker reached files belonging to 134 customers and used them to hijack sessions at several of them. Okta's stock closed down about 11.5% on the day of disclosure, and more than $2 billion of market value was gone within two trading sessions.

Finally, the integrations. In 2025 attackers spent months inside a Salesloft GitHub account, moved into the AWS environment behind its Drift chat product and left with the OAuth tokens that Drift customers had granted to connect it with Salesforce. Google's threat intelligence team said it was aware of at least 700 affected organizations. Because many of them used the integration for customer support, much of what leaked came from support tickets, and support tickets are where people paste things they shouldn't. Salesloft said the attackers pulled secrets such as AWS access keys, passwords and Snowflake tokens out of that data.

Why Attackers Shop in the Back Office

None of this is bad luck. Internal tools concentrate privilege by design: they exist to do what the product deliberately won't let a user do, such as view anyone's data, change anyone's email, reset anyone's MFA or move anyone's money. They are run by big, frequently outsourced teams, which means more people who can be tricked, exhausted or bought. In the help desk's case, they are authenticated by a human listening to a voice on a phone line. And they rarely get the threat modeling that customer-facing code receives, because "only staff can reach it" feels like a security control when it is really a fact about the network.

The economics are moving in the wrong direction. IBM's analysis of its 2026 Cost of a Data Breach findings puts the global average breach at a record $4.99 million and notes that fewer than half of organizations are actively securing non-human identities, the service accounts and tokens at the heart of the Okta and Drift stories. IBM's release on the same report adds that one in four malicious breaches is now AI-enabled, mostly through deepfake impersonation and AI-enabled malware, at an average cost of about $6 million. Put plainly, the caller asking your help desk for an MFA reset may not be a person at all, and "the agent recognized the voice" no longer counts as verification.

Treat the Back Office Like Production

The fixes are not exotic. Most of them are ordinary engineering work that never reached a roadmap because the tool was "internal":

Make every lookup carry a reason. Tie record access to an open ticket or case ID, mask sensitive fields by default and log every reveal. Bulk harvesting becomes slow, noisy and attributable.
Put budgets on human queries. Rate limits shouldn't stop at the public API. Alert when an agent opens far more accounts than their queue explains, or starts browsing customers nobody assigned to them.
Never let a voice authorize a reset. MFA resets, email and phone changes and payout address updates should require a push to an already enrolled device, a callback to a number on file or a second approver, and the UI should make skipping that step impossible rather than merely discouraged.
Give machine credentials an expiry date. Service accounts and OAuth grants need narrow scopes, short lifetimes, a named owner and a place in an inventory. A token that has worked for two years without anyone touching it is a finding, not a convenience.
Scrub secrets at the door. Strip cookies and authorization headers from HAR files on upload, scan ticket bodies for keys and expire attachments when a case closes.
Log for the worst day. Keep audit trails the support organization cannot edit, store them where a stolen session cannot reach, and make "who viewed this customer last month" a query that returns in seconds.

None of these items requires a new vendor. They require someone to open the admin panel's repository with the same suspicion they bring to the login page.

Pricing the Fix

The hard part is rarely the code. It is convincing a roadmap that internal tooling deserves engineering time when no customer will ever notice the change. Money is the argument that lands, and the incidents above make it unusually easy to price. Ticket-scoped access, per-agent query budgets and a hardened reset flow are measured in sprints, not years. Set that against $307 million in one quarter at Coinbase or the $380 million claim Clorox took to court. IBM's data supplies a second lever: detection and escalation together with lost business make up 63% of breach costs, so an audit trail that answers questions in seconds is not paperwork. It is a direct cut to the most expensive line items.

Phrase the risk the way finance already thinks. "The support console has broad read access" gets a polite nod and is forgotten by Friday. "One bribed contractor could export our customer list, and the last company this happened to spent $307 million cleaning up" gets a ticket scheduled. One sentence like that usually does more than a twelve-page threat model.

The attackers in these stories did not need a zero-day. They needed a search box that asked no questions, a reset button that trusted a voice, a token nobody remembered issuing and a ticket queue full of pasted secrets. Every one of those was built on purpose by someone shipping under a deadline, which means the people best placed to take them apart are the developers reading this, ideally before an extortion email does the code review for them.

Your Podcast Habit Is Read-Only. Here's How to Give It a Write Path

Sonia Bobrik — Sat, 03 Oct 2026 18:51:00 +0000

Plenty of developers keep a podcast queue longer than their sprint backlog, and they burn through it the same way: earbuds in, playback at 1.5x, one eye on Slack. Here is a test worth running on yourself. Take any episode you finished last week, whether it was a three-hour deep dive into database internals or the podcast episode you saved for your commute, and try to explain its three strongest ideas to a teammate without pressing play again. Most people recover one idea and a warm sense that it was good. That is not laziness. It is what happens when you treat audio as a read-only stream: bytes flow in, nothing gets written to disk, and the buffer is cleared by Thursday.

Why Listening Feels Like Learning

Good podcasts are engineered to feel effortless. A skilled host asks the question you were about to ask, the guest answers with a crisp story, and the whole thing lands with the satisfying click of an idea that suddenly seems obvious. That click is the trap. Fluency feels like understanding, and the two are not the same thing.

Harvard researchers measured this gap directly. For two weeks of an introductory physics course, students alternated between polished lectures from an experienced instructor and active-learning sessions that covered identical material. As the Harvard Gazette's write-up of the study explains, students felt they had learned more from the lectures, yet they scored higher on the tests that followed the active sessions. Lead author Louis Deslauriers put it bluntly: "Deep learning is hard work." The extra effort of the active sessions felt like confusion, so students mistook it for learning less.

A well-produced podcast is the most fluent lecture ever invented, and it carries a handicap that a page of text does not: it is strictly linear. When a guest explains that they partitioned by tenant, pushed hot writes onto a queue and rebuilt reads from an event log, the next sentence arrives before you have finished drawing the diagram in your head. With a blog post you would scroll back and stare at that paragraph. With audio, the stream keeps moving and your half-built mental model quietly gets overwritten.

Speed Is Not the Real Problem

The usual suspect is playback speed, and the evidence mostly clears it. In a study published in Applied Cognitive Psychology, a UCLA team had 231 students watch lecture videos at normal speed, 1.5x, 2x or 2.5x, with no pausing and no notes. Comprehension at 1.5x and 2x matched normal speed, both right away and a week later; only 2.5x clearly hurt. Eighty-five percent of the students the team surveyed already sped up their lectures. The most useful finding was a different one: watching twice at 2x beat watching once at normal speed, but only when the second pass was saved for later, right before the test, instead of coming straight after the first.

That study used video rather than pure audio, so treat it as a strong hint rather than a law. The hint is still valuable. Speeding up is fine. What matters is what you do with the time you save, and the best use is a spaced second pass over the parts that mattered.

Add a Write Path: Retrieval Beats Replay

If listening is a read operation, the write operation is retrieval: pulling an idea back out of your own head without looking. Psychologists have studied this for decades, and the results are lopsided. At Purdue, Jeffrey Karpicke and Janell Blunt had 200 students read science texts and then either build detailed concept maps or simply set the text aside and write down everything they could recall. As The New York Times reported on the experiment, the recall group retained about 50 percent more a week later. Just as telling, students were poor judges of which method was actually working for them.

There is also a clock running. Hermann Ebbinghaus showed back in the 1880s that unrehearsed memories fade fastest in the first hours and days after you meet them. An idea you have not retrieved by the end of the day is an idea you will probably have to hear again.

A Post-Episode Protocol That Takes Ten Minutes

None of this requires a note-taking system with seventeen plugins. It requires a small habit loop wrapped around the listening you already do:

Ask a question before you press play. "What would change how we do code review?" turns a passive stream into a search query, and your attention starts filtering for answers.
Bookmark instead of pausing. Most modern players let you mark a timestamp; Podcast Addict, for example, supports bookmarks with notes, so you can flag the moment a guest explains their migration plan and keep walking.
Do a two-minute brain dump within the hour. Close the app, write the three ideas you remember, then compare them with the show notes. The gaps tell you exactly which segment to replay.
Turn one idea into code within 48 hours. Heard about property-based testing? Write one property for a parser you own. Heard about feature flags? Put one risky change behind a flag. A twenty-line spike teaches more than twenty pages of notes.
Schedule the second pass. A week later, reread your dump and replay only the bookmarked segments, at 2x if you like.
Teach it to someone. A three-sentence summary in your team channel forces you to compress the idea, and compression is where understanding gets tested.

If the blank page stalls you, give the brain dump a fixed shape: one claim that surprised you, one you would argue with, and one thing you will try this week. The argument is the most valuable line, because disagreeing with a guest in writing is retrieval with a sharper edge.

Fewer Shows, Better Listening

There is also a curation problem. A queue of forty unplayed episodes is not a learning plan; it is a to-do list that quietly generates guilt. Pick two or three shows whose ideas you would actually act on, and let everything else be entertainment. That is a perfectly good use of a podcast. Folding laundry to the story of a legendary outage counts as fun, and fun is allowed.

Just be honest about which mode you are in. Learning mode needs at least part of your attention, which is why a dense conversation about distributed consensus will not survive being played under a debugging session. If you catch yourself rewinding the same thirty seconds three times, pause the episode, not the work you are paid for.

The Episode Is the Input, Not the Output

Podcasts are one of the best deals in a developer's learning budget: free, portable, and full of practitioners describing failures nobody puts in the docs. But the value does not transfer just because the audio played. It transfers when you retrieve, write and build. So switch off continuous playback in your player and let the silence after an episode become your cue. Spend two minutes writing down what you remember, pick one thing to try, and put it on tomorrow's list. Your queue will get shorter, and your head will finally start keeping what passes through it.

97 Emails Before the Bell: Knight Capital and the Price of Waiting

Sonia Bobrik — Sat, 03 Oct 2026 18:42:37 +0000

At about 8:01 a.m. on August 1, 2012, an internal system at Knight Capital started emailing a group of employees about an error in the firm's order router. By the 9:30 opening bell it had sent 97 of those messages, and nobody acted on any of them. Within 45 minutes of the open, Knight had piled up billions of dollars in positions it never wanted, and unwinding them cost more than $460 million. Most retellings file this under "bad deploy." I'd file it next to a sharp essay on Startups.com arguing that the most expensive delay in business is no longer operational but financial, because the regulator's findings keep returning to the same thing that essay does: the distance between the moment evidence appears and the moment someone acts on it. Engineers build a big share of that distance, which means we can also shrink it.

The Morning, Reconstructed

The SEC's findings against Knight Capital read like an incident timeline, so here is the condensed version. Knight was preparing for a new NYSE program for retail orders, and starting July 27 it rolled the new code out, in stages, to the eight servers behind its automated router, SMARS. A technician missed one. Nobody double-checked the rollout, and no written procedure said anyone had to.

That alone might have been survivable. But the new code recycled a flag that had once switched on Power Peg, a function Knight had stopped using in 2003 but left sitting on its servers. In 2005, Knight moved the logic that counted filled shares to a different place in the code, and Power Peg was never retested, because it was never supposed to run again. So on August 1, orders carrying the recycled flag reached the forgotten server, woke the dormant code, and it began firing child orders with no way of knowing that the parent orders had already been filled.

The result, per the regulator: 212 customer orders turned into more than 4 million executions in 154 stocks, over 397 million shares, a net long position of roughly $3.5 billion and a net short position of roughly $3.15 billion. Then the response made it worse. With no incident procedure to lean on, the team removed the new code from the seven servers where it had been installed correctly, and flagged orders on those machines started triggering Power Peg too. The loss left Knight with serious capital problems, its parent company merged with GETCO less than a year later, and the SEC eventually added a $12 million penalty.

Do the Division

Spread $460 million across 45 minutes and you get about $10 million a minute, or roughly $170,000 for every second the router stayed connected. The bug decided how fast the money left. The delay decided how long it kept leaving.

That distinction matters because defects are probabilistic. However good your tests are, you will ship some. The time between "the system knows" and "a human acts" is not probabilistic. It is designed, usually by accident. At Knight it was enormous: 89 minutes of emails before the open, then 45 minutes of live trading while people hunted for the cause.

A slower clock was running too. About ten months earlier, in October 2011, a Knight desk kept generating quotes from leftover test data after a weekend disaster recovery test and lost nearly $7.5 million. The firm patched that specific failure and moved on, without asking the broader question of what would stop any of its systems from sending erroneous orders to the market. Power Peg itself had been sitting in production unused for nine years and untested for seven. Evidence doesn't always arrive as a burst of alerts. Sometimes it arrives as a smaller, cheaper incident that everyone agrees to call a one-off.

Evidence Without an Owner Is Just a Log Line

Knight wasn't short on data that morning. Senior staff could see positions piling up in an internal account from the moment the market opened. But the monitoring tool they relied on depended entirely on someone watching it, didn't display the relevant limits, and lagged under heavy volume. The account soaking up the trades carried a $2 million limit that wasn't connected to anything capable of stopping new orders. Evidence existed at almost every layer. Neither an automatic brake nor a clear rule for when a human should pull one existed at any of them.

The essay linked above has a precise name for this: accountability delay, the stretch between spotting a problem and assigning someone who owns the decision. On most engineering teams it lives in familiar places. A notifications channel everyone muted months ago. A dashboard someone glances at during standup. A threshold that turns a cell red instead of turning something off.

Your Feature Flags Are Inventory

The cheapest-looking decision in this story was reusing an old flag instead of creating a new one. It is also the one that let dead code wake up. Pete Hodgson's guide to feature toggles on Martin Fowler's site, a go-to reference on the subject, uses Knight as its cautionary tale and frames the fix in terms any CFO would recognize: toggles are inventory, inventory carries a cost, and disciplined teams keep their stock low. The tactics it describes are refreshingly blunt. Open a removal ticket the day a flag is born. Give every flag an expiry date. Add a "time bomb" test that fails, or even stops the app from starting, once a flag outlives that date. Cap the total number of flags, so adding one means retiring another.

Dead code deserves the same accounting. Knight left Power Peg on production servers for nine years. Deleting code that never runs feels like effort with no payoff, and that is exactly how inventory piles up, in a warehouse and in a repository.

The Same Gap, Pointed the Other Way

Delay doesn't only hurt when something is breaking. In 2012, the same year as Knight's disastrous morning, a Microsoft employee suggested changing how Bing displayed ad headlines. Building it would take an engineer a few days, but it was one idea among hundreds, it was marked low priority, and it waited more than six months. When an engineer finally shipped it as a cheap A/B test, revenue looked so abnormally high within hours that it set off an alert that usually means a bug. It wasn't a bug. As Ron Kohavi and Stefan Thomke later recounted in Harvard Business Review, the change lifted revenue by 12 percent, worth more than $100 million a year in the US alone, and became the best revenue-generating idea in Bing's history.

On a back-of-the-envelope basis, if the effect had held, the half year in the backlog cost something like $50 million in US revenue. Put the two stories side by side and the symmetry is hard to miss. One company couldn't stop a bad action fast enough. The other took half a year to start a good one. The detail that separates them is small: Bing's strange number fired an alert built to be investigated, and someone investigated it. Knight's emails were never designed as alerts, and staff generally didn't read them.

Both problems share a cure. When an action is cheap, safe, and reversible, nobody needs courage to take it, and the gap between evidence and action shrinks on its own. A kill switch and an experiment flag are the same machinery pointed in opposite directions.

Shrinking the Gap on Your Own Team

Give every automated message an owner, or delete it. If no named rotation is on the hook for a notification, it's noise, and noise is exactly where the one message that matters goes to hide.
Wire limits to brakes, not just dashboards. A threshold that only changes a color depends on someone watching the right screen at the right second.
Pre-authorize the stop. Decide on a calm day who can disable a feature, pause a job, or pull a service out of rotation without asking anyone, and write it down. The SEC specifically faulted Knight for lacking clear guidance on when to disconnect a malfunctioning system.
Verify rollouts across the whole fleet. A post-deploy check that every node reports the same version costs almost nothing, and the regulator found that a simple written double-check of the rollout could have caught the missed server and averted the whole event.
Never recycle a flag. New behavior gets a new name, an owner, and an expiry date. Old code paths get deleted, not parked.
Rehearse the rollback before you need it. Under pressure people reach for the most plausible move, and at Knight the most plausible move widened the damage. Drills make the right move feel obvious.

Run a Time-to-Off Drill

If you do one thing after reading this, measure what I call your time-to-off. Pick a non-critical feature in production, warn the team that a drill is coming, start a stopwatch, and turn the feature off for every user. Stop the clock when telemetry confirms it is actually off, not when someone types "done" in chat. Then study what slowed you down. Did anyone need approval? Did it require a deploy? Did anyone know where the switch lived, and was that person on vacation?

Then run the mirror image: find the oldest "low priority" idea in your backlog that could ship behind a flag and be tested within a week, and test it.

Knight's problem that morning was not a lack of information. It had 97 emails' worth before the market even opened. What it lacked was a short, owned, rehearsed path from knowing to doing. That path is cheap to build on a quiet afternoon and impossible to build at $10 million a minute. If you run the drill, share your number in the comments, especially if it surprised you.

We Version-Control Everything Except Our Own Memory

Sonia Bobrik — Sat, 03 Oct 2026 18:40:46 +0000

You know the moment. A five-year-old answer describes your exact bug, sends you to the official docs for the details, and the link lands on a glossy product page that has never heard of the API you need. No redirect, no archived copy, no apology. That small dead end is a symptom of something much bigger, and a recent deep dive into how the technology industry is erasing its own history traces the pattern in uncomfortable detail, from acquisitions that swallow engineering blogs to platform migrations that drop a decade of posts on the floor. The part that should sting for developers is who does the erasing. The web rarely forgets because of hackers or hardware failure. It forgets because of ordinary engineering decisions, shipped by people like us, often in a pull request titled "cleanup."

Most Pages Don't Die. Somebody Deletes Them.

In 2024, researchers pulled just under a million pages from Common Crawl snapshots taken between 2013 and 2023 and tried to load every one of them again. According to Pew Research Center's study of disappearing online content, 38% of the pages captured in 2013 were gone a decade later. Freshness offered little protection: 8% of the pages captured in 2023 had already vanished by that October.

The detail engineers should stare at is where the losses happened. In most cases the page disappeared from a website that was still online. The server answered; the page had simply been removed or moved without a forwarding address. Redirects are carrying more weight than most teams realize, too, since about a third of the links on news pages already bounce to a different URL than the one first published. Every one of those hops is a forwarding rule that somebody has to remember to keep alive. Money doesn't solve it either: the most-trafficked news sites were just as likely to contain broken links as the smallest ones. And Pew counted a page as dead only when it returned one of nine error codes that leave no room for doubt, so the real picture is probably worse.

The Link Still Loads. That's the Scary Part.

A 404 at least tells you the truth. The nastier failure is content drift, when the URL resolves but whatever you linked to has changed shape underneath you. Jonathan Zittrain, who teaches law and computer science at Harvard, made this point in his Atlantic essay on why the internet is rotting, drawing on research his team did with a dataset of New York Times articles published between 1996 and mid-2019. Those articles held more than two million outbound links. A quarter of the deep links were completely dead, and age made it brutal: 6% of links from 2018 had rotted, against 72% of links from 1998. When the team checked a sample of 4,500 links that still worked, 13% pointed to content that had drifted significantly from what the reporter originally cited.

Software teams manufacture drift every day without noticing. A GitHub link to a file on main promises that the highlighted lines will stay put, and the next refactor breaks that promise without a sound. A link to the "latest" docs changes meaning with every release. A README that says "see the wiki" outlives the wiki. None of these throw an error. They just start telling the reader something that is no longer true.

Now the Archive Itself Is Getting Locked Out

For most of the web's life, the safety net under all this decay has been the Internet Archive. Its Wayback Machine passed one trillion archived web pages in October 2025, the largest record of the web anyone has ever assembled. Right around the time that counter rolled over, the doors started closing.

In August 2025, Reddit limited the Wayback Machine to little more than its homepage, saying AI companies had been scraping Reddit data through the archive. At the end of 2025, The New York Times added the archive's crawler to its robots.txt and later confirmed it was hard-blocking it. The Guardian excluded itself from the archive's APIs and filtered its article pages out of the Wayback Machine. When Nieman Lab analyzed robots.txt files from a database of 1,167 news sites, it found 241 sites in nine countries disallowing at least one Internet Archive bot, and a follow-up in May 2026 counted more than 340 local US outlets limiting access. No publisher confirmed to Nieman Lab that an AI company had actually scraped its content through the Wayback Machine.

Read that with an engineer's eye and it looks like a textbook incident: a defense aimed at one threat takes down a shared dependency that everyone else relies on. It is the robots.txt version of firewalling your own backups. There's a bitter footnote, too. The newspaper whose outbound links became a landmark dataset for measuring link rot is now keeping its own pages out of the biggest archive built to fight it.

The Quietest Loss: Questions We Stopped Asking in Public

A stranger kind of forgetting is happening inside our own profession. Stack Overflow peaked in early 2014 with more than 200,000 questions a month. In December 2025 it received 3,862, a 78% drop from a year earlier. The slide started years before chatbots, but the cliff came after them. Developers didn't stop hitting bugs; they started solving them in private conversations with AI assistants built into their editors.

Those answers might be excellent. But a private chat has no URL, no public edit history, no accepted answer, no maintainer dropping in to say the behavior changed in v3, and no crawler will ever see it. The fix for that cursed driver bug now lives in one person's chat sidebar. When the next developer hits the same wall a few years from now, there will be nothing to find. We are trading a messy public commons for millions of private, unlinkable notebooks.

Deletion Is a Breaking Change

We have already run the experiment on what happens when shared history vanishes overnight. In March 2016, a developer unpublished left-pad, an 11-line npm package, and builds across the JavaScript world fell over because projects as large as Babel and React pulled it in somewhere down their dependency trees. npm restored the package and tightened its rules so that packages older than 24 hours could no longer be unpublished on a whim.

Nine years later, Google repeated the lesson at web scale. It announced that goo.gl short links would stop resolving on August 25, 2025, after years of steering developers toward Firebase Dynamic Links as the replacement, a service retired on that very same day. Under pressure, Google reversed course for links that still got clicks, but anything with no activity in late 2024 was switched off. Rarely visited old links are precisely what a historical record is made of.

A Preservation Checklist for People Who Ship Code

Tim Berners-Lee wrote in 1998 that cool URIs don't change. Nearly three decades later, it is still the most ignored style guide in our industry. Fixing that doesn't require a grant or a committee, only the habits we already apply to code, pointed at knowledge:

Give every moved page a forwarding address. Ship permanent 301s when docs or posts move, keep the redirect map in version control next to the code, and add a CI job that replays your legacy URLs and fails the build on any 404.
Link to commits, not branches. On any GitHub file view, pressing y swaps the address for a permalink pinned to the commit SHA, so the lines you highlighted can't wander away from your explanation.
Snapshot your sources, not just your code. When a design doc or blog post leans on an outside page, capture it with the Wayback Machine's Save Page Now and keep a local copy in a self-hosted ArchiveBox, since a single archive can be blocked; reach for Perma.cc when the citation is legal or academic.
Keep shorteners out of anything meant to last. READMEs, error messages, printed docs, and conference slides deserve full URLs on a domain you control.
Deprecate instead of deleting. Version your docs and leave old pages online with a banner pointing to the current release.
Write down the why. Commit messages, architecture decision records, and PR descriptions outlive the people who remember the context, so when you squash-merge, carry the PR description into the commit body.
Publish the answers you find in private. If an AI session or a 2 a.m. debugging marathon cracks something hard, spend ten minutes turning it into a post, an issue comment, or an FAQ entry.
Send your public repos to Software Heritage. Its Save Code Now form archives a repository along with its full history, and the archive already holds more than 22 billion unique source files.

Memory Is a Feature

On October 24, 2026, the Wayback Machine turns 25. It went public in 2001 with more than ten billion pages and now holds over a trillion, yet it only works if the rest of the web keeps its doors open and its URLs stable. The irony of our field is hard to miss. We built Git so that no change is ever truly lost, we argue about semantic versioning with theological intensity, and then we let whole decades of documentation evaporate in a CMS migration nobody thought to review.

Pick one item from that checklist and do it this week. Somewhere in 2031, a developer will land on your page instead of a 404 and never know how close it came.

When Your Software Only Works While Someone Is Watching

Sonia Bobrik — Sat, 03 Oct 2026 18:39:54 +0000

Every developer has met a Heisenbug, the defect that vanishes the moment you attach a debugger. Its evil twin gets far less attention: the feature that only works while someone important is looking. A recent breakdown of the true cost of a staged demo traces that pattern through a gravity-powered truck, an edited AI video and a self-driving clip that resurfaced in a deposition, then closes with advice for the people sitting in the audience. This piece is for the people on the other side of the stage, the ones asked to make it look finished by Thursday. Because when engineers stage a demo, we rarely reach for a camera. We reach for a conditional.

The golden path had an expiry date

On January 9, 2007, Steve Jobs walked onto the Macworld stage holding a phone that, by any honest engineering standard, did not work yet. As Fred Vogelstein later reconstructed for The New York Times Magazine, the prototype could handle a snippet of a song or video but not reliably a whole clip, and it behaved if you sent an email before opening the browser, but not necessarily the other way round. Engineers spent hours hunting for a route through the software that wouldn't fall over, then scripted the keynote around it. They called that route the golden path. AT&T hauled a portable cell tower to the venue. And because the radio might crash and reboot somewhere in a 90-minute keynote, the team, with Jobs's blessing, made a call every developer will recognize. As Andy Grignon, who ran the iPhone's radios, put it: "we just hard-coded it to always show five bars."

The engineers watched from about the fifth row, each downing a shot of Scotch when the segment they owned survived. By the finale the flask was empty.

That was staging, plainly. So why does nobody call it fraud? Two reasons, and both matter to anyone who ships software. First, the gap had a deadline the team controlled: the iPhone went on sale on June 29, roughly six months later, and the people who faked the signal bars were the same people responsible for making them true. Second, the fakes concealed instability, not absence. The phone genuinely placed calls on stage; the hard-coded bars hid a radio that couldn't yet be trusted for an hour and a half. A forgivable staged demo is a loan with a repayment date. An unforgivable one is a capability that was never coming.

The demo that ran itself on 11 million cars

Now take the presenter out entirely. In September 2015, the U.S. Environmental Protection Agency revealed that Volkswagen diesels carried software able to recognize an emissions test and change how the engine behaved while the test was running. If you've never looked at how the trick worked, the BBC's plain-language explainer on the Volkswagen scandal is worth five minutes of your time. In engineering terms, it was a golden path with nobody holding the phone: on the test rig the emissions controls did their full job, while in normal driving the cars could emit up to 40 times the permitted level of nitrogen oxides. Volkswagen eventually admitted the software sat in about 11 million vehicles worldwide.

What finally exposed it wasn't a firmware teardown. It was a road trip. In 2013, researchers at West Virginia University took a $70,000 grant from the International Council on Clean Transportation, borrowed a Jetta, a Passat and a BMW X5, strapped portable emissions analyzers to them and drove real roads, including a run from San Diego to Seattle. Two of the Volkswagens came back far over U.S. limits. Nobody had to outsmart the code. They simply walked the demo off its golden path.

The bill matched the scale. By 2020, Volkswagen put its cost in fines and settlements at €31.3 billion. And the conditional had authors. In August 2017, a federal judge in Detroit sentenced James Liang, a VW engineer whom prosecutors called a "pivotal figure" in designing the cheating systems, to 40 months in prison and a $200,000 fine, ten times what the government had asked for. His lawyers argued he had followed orders out of loyalty to his employer. It didn't save him. When a demo has no presenter, the code becomes the presenter, and code keeps a commit history.

The leaderboard is the new keynote

For AI teams, the stage has moved to benchmarks. In April 2025, Meta's Llama 4 Maverick debuted in second place on LM Arena, the leaderboard where human raters vote between anonymous chatbot answers. Developers who downloaded the weights quickly noticed the public model didn't behave like the one on the board. Meta's own launch materials had disclosed that the Arena entry was an experimental chat variant, and as TechCrunch reported when the discrepancy surfaced, the leaderboard version was far chattier and heavier on emojis than anything developers could actually run. LM Arena published more than 2,000 of the head-to-head battles, said Meta hadn't been clear enough that its entry was a customized model, and rewrote its leaderboard rules. When the unmodified release was added to the board, it landed in 32nd place.

The interest kept compounding. In January 2026, Yann LeCun, by then on his way out of Meta, told the Financial Times that the Llama 4 results were "fudged a little bit," with different models used for different benchmarks. By his account, Mark Zuckerberg stopped trusting the people behind the release and sidelined the whole generative AI group. The leaderboard spot lasted days. The explanation took nine months.

Meta didn't hide a switch inside a model; it submitted a different model. But for developers deciding what to build on, the effect rhymed with Volkswagen: the thing that was measured was not the thing that shipped. An evaluation is a demo that runs while you sleep, and Goodhart's law applies in full. Once a score becomes the target, every team under pressure learns to perform for the scorer.

Where heisenfeatures hide in ordinary codebases

Most of us will never touch emissions firmware or a frontier model. The same shape still turns up in mundane code, usually committed with good intentions the night before a sales call:

Identity checks inside business logic. A branch like if (account.isProspect) that routes the demo tenant to a warmer cache, a dedicated replica or precomputed results. The audience gets the fast path; paying customers get the other one.
Fixtures wearing a live badge. A dashboard that animates beautifully because it replays a JSON file, sitting under a label that says "real-time."
Time that lies. A setTimeout and a spinner standing in for a pipeline that doesn't exist yet, or the reverse: a cached answer returned instantly so an expensive model looks cheap.
Benchmark-shaped behavior. Prompts, retries or model routing that quietly change when an input matches a known eval format or dataset.
A kinder environment. Demo builds with rate limits disabled, retries cranked up and error toasts swallowed, so the failures every real user sees simply never render.

None of these begins as fraud. Each becomes dangerous when it outlives the meeting it was written for, because six months later nobody remembers which numbers on the screen were real.

Building software that behaves the same with or without an audience

Treat observation invariance as a property you test, the way you test idempotency. Run your demo script against the production build with production configuration, then diff the outputs and latencies against what the demo environment shows. If the demo only works on a different code path, you haven't found a feature. You've found a liability with a nice UI.

When you genuinely need to simulate something, push the fake to the edges of the system. A mock service with "simulated" in its name and a visible banner in the interface is honest staging. A conditional buried in core logic that checks who is watching is a heisenfeature. Fixtures can be audited at a glance; branches hide in plain sight.

Give every fake an owner and an expiry date, the way the iPhone's golden path quietly expired on June 29. A demo flag that reaches production configuration should fail the build, and a demo-only marker that outlives its deadline should fail review. Then lead the audience off the path on purpose: let the prospect type their own input, publish your evaluation harness, and report scores only for the exact artifact you ship, with the same weights, the same config and the same hardware tier.

And remember the Detroit courtroom. The person who writes the conditional owns it, no matter who asked for it. If a commit message would sound bad read aloud by a prosecutor, the code probably shouldn't merge.

The audience that never sees the golden path

One question belongs above every demo you will ever give: what would this system do if nobody were watching? That is the only version your users ever meet. They don't follow the golden path. They open the browser before the email, play the whole clip, and use your product on one bar of real signal at two in the morning. Build for them, and the demo stops being a performance. It becomes a preview.

Your Protocol Has Consensus. The Committee Reviewing It Doesn't.

Sonia Bobrik — Sat, 03 Oct 2026 18:27:37 +0000

Somewhere right now, five people at an asset manager are discussing a DeFi protocol whose builders will never hear the verdict. The risk lead read the docs. Compliance found a blog post from last year. Legal skimmed an audit of a version that is no longer deployed, the portfolio manager remembers a founder's thread, and finance pulled numbers off a dashboard. Twenty minutes in, they realize they do not agree on what the protocol actually is, so the item gets parked and nobody sends the team a reason. That quiet failure sits at the center of a recent interview with the TechWaves PR founder on why institutional DeFi has a communication problem on DefiLlama Research, where Sofia Bobrik argues that institutional deals rarely stall because nobody understood the product. They stall because several people understood it differently. If you write code for a protocol that wants institutional capital, that is an engineering problem wearing a business suit, and you already know its name.

Even Lamport Needed a Story

The name is consensus failure, and computer scientists have been studying it for more than forty years. In 1982, Leslie Lamport, Robert Shostak and Marshall Pease described a group of generals surrounding a city who must agree on a single plan while communicating only through messengers, knowing that some of their own may be traitors. The headline result is famous: with ordinary messages, agreement is possible only if more than two-thirds of the generals are loyal, a threshold that still echoes in the supermajority Ethereum validators need to finalize a checkpoint.

The backstory is less famous and more useful. In Lamport's notes on the Byzantine Generals Problem, he explains that the story was a deliberate choice. He had watched Dijkstra's dining philosophers collect far more attention than he felt it merited, simply because it was framed as a story, so he dressed his own result in one. He even changed the title after a colleague objected that the original version cast the generals as Albanian. One of the foundational results of distributed computing spread because its author treated packaging as part of the work. Correct ideas do not distribute themselves, not even Lamport's.

A Buying Committee Is a Network Without a Consensus Protocol

Now map the paper onto an institutional deal. Every stakeholder is a node. Your docs, website, deck, audit reports, dashboard listings and the founder's public posts are the messages. Each node processes whichever messages happened to reach it, forms a view, and then the group tries to agree. Gartner's survey on why B2B buying teams struggle to reach consensus found that buying groups now span five to 16 people across as many as four functions, and that 74% of them show unhealthy conflict during the decision. Groups that did reach consensus were two and a half times as likely to describe the resulting deal as high quality.

Those figures cover B2B purchasing in general, where buyers usually know which category they are shopping in. DeFi starts from a harder position. As the interview notes, a finance team often cannot tell whether it is looking at an infrastructure layer or a trading venue, a settlement rail or a liquidity protocol, a yield strategy or a tokenized asset platform. When each node returns a different answer to that question, there is no quorum. Nobody votes no. The proposal simply times out, and a timeout sends no error message back to the caller.

You Might Be the Byzantine General

Here is the uncomfortable part. In Lamport's model, one of the most damaging faults is a commander who sends different orders to different lieutenants, which leaves perfectly honest lieutenants unable to agree. Plenty of protocol teams play that role without meaning to. The deck says institutional lending infrastructure. The docs say permissionless money market. A podcast clip mentions a 24-hour timelock while the contract enforces 48. The audit everyone forwards covers v2, and v3 has been live for months. Each message was true when someone wrote it. Together they are conflicting orders.

Sales instinct then makes things worse. The obvious move is to tailor the story for each reader: yield for the portfolio manager, controls for compliance, architecture for the CTO. Gartner's data points the other way. Content tuned to individual stakeholders had a 59% negative impact on buying-group consensus, which the firm attributes to confirmation bias, while content built around the group's shared goals improved consensus by 20%. In protocol terms, you were sending every node a different value and hoping they would converge. The interview proposes a better target metric, "consistency of interpretation": every reader ends up with the same model of what you are, even when they arrived through different documents.

Visibility Is Just a Star Count

The same interview calls visibility a vanity metric in deep tech, and developers already own the perfect analogy. Stars tell you how many people noticed a repository. They say nothing about how many understood it well enough to run it in production or describe it accurately to a colleague. A viral thread about your protocol is a star count. What moves an institutional allocation is how many people inside one organization can restate what you do, correctly and in compatible terms, while you are not in the room.

Run a Differential Reading Test

Compiler engineers rely on a technique called differential testing: run the same input through several implementations and treat any disagreement in their output as a bug. Your public materials can be tested the same way. Hand your docs, site and deck to three or four people who think like the committee, such as a risk analyst, a lawyer, a finance generalist and an engineer from outside crypto. Ask each of them to answer the same questions in writing and in their own words, then diff the answers. These are the questions every committee ends up asking, and they closely track the risk questions raised in the interview:

What is it? One sentence, using a category the reader's firm already has a box for.
Who can change the rules, and how fast? Admin roles, multisig thresholds, timelock delays and upgrade paths.
What happens on the worst day? Liquidations, oracle failures, withdrawal queues and the exact conditions for pausing.
What lives off-chain? Keepers, oracle operators, custodians, KYC providers, legal wrappers and anything else a contract cannot guarantee.
What is still unsolved? The honest limits of your protocol and of the category as a whole.

For a cheap first pass, prompt an LLM to play each role with nothing but your public materials as context and compare the summaries. It will never replace a real compliance officer, but it is good at finding the paragraph that can be read two ways. Pay the most attention to divergence on the last question, because that is where risk communication either holds a group together or splits it.

Hidden Risk Is a Fork Waiting to Happen

Teams often downplay risk to look stronger. In consensus terms, that is exactly backwards. Institutional reviewers already know DeFi carries contract, oracle, liquidity, governance, custody and regulatory risk, and the interview is blunt that pretending otherwise costs a founder credibility on the spot. If you do not describe your risks, someone on the committee will discover one alone, and at that moment one node holds information the others lack. That is a fork, and forks are expensive to reconcile. Publishing your risks in one canonical place gives every reviewer the same starting state, and it signals the kind of literacy institutions look for: the assumptions you make, what you have done to mitigate them, and what remains open.

Make Narrative Drift Fail the Build

Most conflicting orders are not lies. They are stale replicas. Phil Karlton's old joke holds that the two hard things in computer science are cache invalidation and naming things, and institutional communication breaks on precisely those two: naming, when nobody can agree which category you belong in, and cache invalidation, when last year's deck keeps circulating long after the protocol changed. Treat the facts reviewers check the way you already treat config. Keep timelock delays, signer thresholds, oracle sources and caps in one versioned file, render your docs from it, and let CI compare it against the chain. If governance runs through OpenZeppelin's TimelockController, the check takes a few lines of viem:

import { readFileSync } from "node:fs";
import { createPublicClient, http, parseAbi } from "viem";
import { mainnet } from "viem/chains";

// docs/facts.json is the single source of truth the docs are rendered from
const facts = JSON.parse(readFileSync("docs/facts.json", "utf8"));
const client = createPublicClient({ chain: mainnet, transport: http(process.env.RPC_URL) });

const onchainDelay = await client.readContract({
  address: facts.timelock.address,
  abi: parseAbi(["function getMinDelay() view returns (uint256)"]),
  functionName: "getMinDelay",
});

if (onchainDelay !== BigInt(facts.timelock.minDelaySeconds)) {
  console.error(`Narrative drift: docs say ${facts.timelock.minDelaySeconds}s, chain says ${onchainDelay}s`);
  process.exit(1);
}

The same pattern covers a Safe's getThreshold(), oracle feed addresses and supply caps. Then deal with the cache: share decks as links to one hosted copy instead of attachments, so a forwarded link always resolves to the current version, and stamp every public document with the date its facts were last verified.

Instrument the Silence

A deal that dies in committee leaves no stack trace, so add your own instrumentation. Open every first institutional call with one question: before we begin, how would you describe what we do to your risk team? The answer is a free readout of the state your materials left behind. Write it down. When answers from unrelated firms start to converge, your messages are consistent, and the interview describes the payoff in practical terms: the first serious conversation stops being a tutorial on the basics and starts from an informed position. Engineers would call that a warm cache.

Ship Agreement, Not Just Code

Your protocol already solves a harder agreement problem than any committee will ever face, with adversarial validators and real money at stake. The off-chain version deserves the same rigor: one source of truth, identical messages to every node, honest disclosure of known faults, and tests that catch divergence before a reviewer does. Lamport understood that a correct result still needs a vehicle that carries it intact from one mind to the next. For a DeFi protocol, that vehicle is everything you publish. Every inconsistency in it is a bug, and unlike a bug in your contracts, you can patch this one without a governance vote.

Your Founder's Personal Brand Should Work Like a Bootloader

Sonia Bobrik — Sat, 03 Oct 2026 18:26:52 +0000

Engineering teams will spend an entire quarter removing a single point of failure from a payments service, then never ask the same question about the person whose face is on every conference banner the company pays for. The upside of executive visibility is real, and this guide to building a C-level personal brand that elevates the whole company covers it well, from aligning a leader's public voice with strategy to tracking what that voice does for partnerships, hiring, and funding. But one number rarely shows up in advice like this, and engineers already have a name for it: the bus factor. If the person carrying your company's reputation went silent tomorrow, how much of that reputation would still be standing a year from now?

The model I keep returning to comes from systems design: a founder's personal brand should work like a bootloader. It does the fragile, high-trust work of getting a company running, then hands control to things that don't depend on it anymore: the product, the team, and a body of public knowledge that belongs to the company. Most executive brands never make that handoff, and nobody notices until someone leaves.

A bus factor of one, with a LinkedIn account

In most startups this concentration isn't a strategy. It's an accident of timing. The founder was the first salesperson, the first recruiter, and the first person anyone outside the building had heard of. Investors met them before they met the product, and early hires joined because of a talk they gave or a thread they wrote. None of that is a bug at the start. Routing everything through one trusted node is the fastest way to bootstrap trust when nothing else exists yet, the same way a monolith is the fastest way to ship version one.

The trouble starts when nobody plans the migration. Trust that lives in a single person's feed can't be load-balanced, can't fail over, and leaves when they do. The most dramatic stress test in recent memory came in November 2023, when OpenAI's board removed Sam Altman as CEO. By the following Monday, around 700 of the company's roughly 770 employees had signed a letter threatening to quit unless he was reinstated, and within days he was. Whatever you think of the governance fight, it was a live demonstration of how much talent, investor confidence, and company identity can end up routed through one person.

Executives are not permanent infrastructure

We design for nodes disappearing because nodes disappear. Executives do too, and lately at a record pace. Fortune reported that CEO departures set a new January record in 2025, and by the end of the year Challenger, Gray & Christmas had counted 446 CEO exits at US public companies, the highest annual total since the firm began tracking in 2002. Leaders get recruited away, burn out, retire, get replaced by boards, or simply want a different job. That isn't a scandal. It's churn, and churn is something engineers know how to design for.

In that Fortune piece, Andrew Challenger advises companies to keep revisiting their succession plans. Here's the gap most of those plans share: they cover the job title but not the public voice. A new CEO inherits the org chart on day one. The audience the previous one built doesn't transfer with the badge.

Stack Overflow booted from two blogs

If you want to see the pattern done right, look at a site you've probably used this week. In 2008, Joel Spolsky and Jeff Atwood ran two of the most popular programming blogs around, Joel on Software and Coding Horror, and they used both audiences on purpose. Coding Horror readers voted on the name Stack Overflow. Atwood's subscribers got the first private beta invitations. The founders even recorded their weekly planning calls as a podcast, so the community could watch the product take shape before launch.

Then they did something much rarer than building an audience: they shipped a product whose core mechanic gave reputation away. Every upvote and accepted answer moved credibility from the people who started the site to the people who used it, and reputation points unlocked real privileges like voting, commenting, and editing other people's posts. For years, the most famous name on the site wasn't a founder or an employee at all. Jon Skeet, its all-time top answerer, was a Google engineer who wrote answers on his commute.

Atwood left in early 2012, writing that he trusted the team and the community to carry the network forward. Spolsky handed over the CEO role in 2019. In 2021, Prosus agreed to buy Stack Overflow for about $1.8 billion, and Spolsky noted that the same team would keep running it independently. The founders' brands booted the system. The community became the kernel. (Its more recent fight with AI assistants is a real story too, but it's a story about the product, not about who held the microphone.)

Designing the handoff on purpose

None of this means executives should go quiet. A bootloader runs again on every restart, and companies restart constantly: a funding round, a pivot, a crisis, a launch. The goal is a system that stays up between boots.

The cheapest habit is to co-sign the work. When a CTO publishes a deep technical post, the engineers who built the thing belong on the byline, not in a thank-you line at the bottom. Readers learn new names, and the next post can come from one of them.

The second habit is to turn opinions into documents. A position that lives only in a founder's feed walks out the door with the founder. A position written into a handbook, an engineering principles page, or a public decision record stays. GitLab is close to a reference implementation: it spent years running on a public handbook, so when co-founder Sid Sijbrandij handed the CEO role to Bill Staples in December 2024 and moved to executive chair, much of the company's operating philosophy was already written down where anyone could read it.

The third is to rotate the microphone. Let the executive introduce the conference session and a staff engineer give the talk. Send the principal engineer on the podcast. Let the people who were on call sign the incident review.

Finally, measure the handoff. The guide linked above suggests tracking partnership inquiries, investor meeting requests, media mentions, speaking invitations, and talent referrals. Add one column to that spreadsheet: who each signal was actually about. If the founder's share isn't shrinking over time, the founder's brand is growing, but the company's isn't.

A five-minute bus-factor audit

You can estimate your company's reputational bus factor without any tooling. Answer these honestly:

The silence test. If the CEO stopped posting for six months, what would dry up first: hiring, inbound sales, or investor interest?
The name test. Could a stranger find public work by at least three people at your company who aren't executives?
The candidate test. When new hires explain why they applied, do they name a person, or a product, a practice, or a piece of writing?
The document test. Are your core technical and product positions written down somewhere the founder doesn't personally own?
The stage test. Besides the CEO, who has represented the company at a conference, on a podcast, or in a widely shared post in the last twelve months?

If most answers point at one person, your bus factor is one. That's normal at seed stage and a liability at Series B.

If you're not the CEO

Two takeaways for everyone else. First, an executive who is serious about elevating the company needs more credible voices than their own, which makes this one of the rare places where helping your employer and building your own career are the same move. Offer to write the migration retrospective. Pitch the talk. Ask for the byline.

Second, run the audit when you're the one choosing an employer. A company whose entire public identity lives in one account can look very different after a single resignation, and that's worth knowing before you sign an offer.

Aim for a boring restart

The best founder brands end up a lot like bootloaders: essential, slightly mythical, and invisible once the system is running. They do the early work of convincing the world to trust something new, then hand control to a product, a team, and a body of written knowledge that keeps earning that trust on its own. If your company would still be recognizable with its founder offline for a year, the brand did its job. If it wouldn't, that's not a reason to post less. It's a reason to start the handoff now, while the founder is still around to run it.

What's the best founder-to-company handoff you've seen, and the worst? I'd love to hear about both in the comments.

A Developer's Guide to Staying Curious Without Missing Deadlines

Sonia Bobrik — Sat, 03 Oct 2026 18:26:22 +0000

Every team has a module nobody wants to touch and a handful of habits nobody remembers choosing. The tests pass, the deploys are boring, and the roadmap ships roughly on time, which feels like success right up until a new hire asks why the build takes eleven minutes and nobody has a good answer. In optimization terms, that team is parked on a local maximum: a hill that looks like the summit only because nobody has walked far enough to see the next one. A short essay on why exploring new ideas changes everything makes a point that sounds obvious until you try to live by it: curiosity pays off when trying new things is balanced by commitment to the ones worth keeping. For developers, that balance is more than a life lesson. It is a well-studied algorithmic problem, and treating it like one can change how you learn, debug, and lead.

Your Week Is a Multi-Armed Bandit

Picture a row of slot machines, each with an unknown payout. Every pull forces a choice: play the machine that has paid well so far, or gamble on one you barely know. Computer scientists call this the multi-armed bandit problem, and its core tension between exploring and exploiting turns up everywhere from clinical trial design to the recommendation engine picking your next video.

A developer's week works the same way. Exploiting means reaching for the stack, patterns, and shortcuts you already trust. Exploring means reading unfamiliar source code, trying a paradigm that feels clumsy at first, or chasing down why a system behaves the way it does. Pure exploitation is efficient and slowly fatal, because you get faster and faster at an approach whose ceiling you never get to see. Pure exploration fails differently, as anyone who has rewritten a side project in a fourth framework without ever shipping it can confirm.

One of the simplest bandit strategies, epsilon-greedy, splits the difference. Most of the time you pick the option that has performed best; with a small probability called epsilon, you try something else at random. The detail that matters for us hides in the theory: when payouts drift over time, a situation known as a non-stationary problem, exploration should never stop, because yesterday's best machine quietly stops being the best. Few environments drift faster than software. The tools that dominated ten years ago guarantee nothing about the next ten, and the only way to notice the drift is to keep pulling a few unfamiliar levers.

Bell Labs Ran This Experiment Decades Ago

Richard Hamming, the mathematician whose name lives on in Hamming codes, described his own version of this strategy in a 1986 talk called "You and Your Research." Nudged by colleagues, he reserved Friday afternoons for what he called Great Thoughts Time, setting routine work aside to ask questions like how computers would change science. Those afternoons led him to predict that most experiments would eventually run on computers instead of in laboratories, a forecast the company's vice presidents brushed off and one he lived to see vindicated. Half a day out of a five-day week is about ten percent, so Hamming was effectively running with an epsilon of 0.1.

He also noticed something about office doors. Colleagues who kept theirs shut got more done in the short run, yet years later many seemed unsure which problems deserved their effort. Those who left the door open put up with interruptions but kept picking up clues about what mattered. Swap the door for a Slack status and noise-canceling headphones, and the observation holds up uncomfortably well.

What Curiosity Does Inside Your Head

There is a biological reason these tangents tend to stick. In a 2014 study at UC Davis, volunteers rated how curious they were about the answers to more than a hundred trivia questions, then lay in an fMRI scanner while those questions reappeared. Before each answer was revealed, they briefly saw a photo of a stranger's face that had nothing to do with the question. As Scientific American's report on how curiosity prepares the brain for learning describes, people remembered the answers to intriguing questions better, and they also remembered the unrelated faces better, an effect that still held a day later. Anticipation switched on the brain's dopamine-linked reward circuitry, and the strength of its interaction with the hippocampus, a region central to forming memories, predicted who would recall those incidental faces. Charan Ranganath, the neuroscientist whose lab ran the study, likened curiosity to "an itch that you have to scratch."

The lesson for developers is that curiosity is more than a pleasant mood. It is a learning state you can switch on deliberately. Carnegie Mellon behavioral economist George Loewenstein proposed in 1994 that curiosity appears when we notice a gap between what we know and what we want to know, and you can manufacture that gap on demand. Before running a query, guess how many rows it will return. Before opening the profiler, predict which function will dominate the flame graph. Before reading the docs for a new API, sketch how you would have designed it. Being wrong is the whole point, because a failed prediction opens exactly the kind of gap your brain seems built to close and remember.

Curiosity Is a Debugging Tool, Not a Distraction

Debugging is where the explore-exploit trade-off gets personal. The moment you form a theory about a bug, every log line starts to look like evidence for it. That is confirmation bias, and it is how a two-hour fix becomes a two-day hunt. In Francesca Gino's research on the business case for curiosity, published in Harvard Business Review, the Harvard Business School professor explains that triggered curiosity makes people less prone to confirmation bias, more deliberate in their decisions, and more inventive in the solutions they reach. A curious debugger treats the first hypothesis as a guess to be broken, not a position to be defended.

Curiosity also changes what "fixed" means. Once the bug is gone, ask one more question: why did this ever work? Code that ran fine for months and then failed is pointing at an assumption, a dependency, or a boundary that nobody wrote down. Running git log -S with a suspicious string shows every commit that added or removed it, and git bisect binary-searches your history for the change that broke things. Use these tools to recover intent, not only to find culprits. Chesterton's old parable about the fence across a road applies neatly here: the strange check that looks pointless was usually added by someone with a reason, and learning that reason before you delete it is the cheapest humility lesson software offers.

How to Explore Without Wrecking Your Sprint

Curiosity needs structure, or it loses to the next deadline every single time. These habits keep your exploration rate above zero without turning the week into one long side quest:

Pick an epsilon and put it on the calendar. Ten percent is a sane default: one protected afternoon a week or roughly forty-five minutes a day. Curiosity without a time slot usually gets spent on whatever is loudest.
Keep a question backlog. When a "wait, why does that happen?" moment interrupts focused work, jot it in a running file and return to the task. Triage the list during your exploration block, so focus and curiosity stop fighting over the same hour.
Read one layer down. When a library surprises you, step into its source with the debugger instead of searching for a workaround. Most framework magic turns out to be ordinary code with a good name.
Timebox the rabbit holes. Open a branch prefixed with spike/, set a ninety-minute timer, and make the deliverable a short written note rather than mergeable code. The note is what turns an excursion into an asset.
Explore next door before you explore another planet. A 2012 study led by cognitive scientist Celeste Kidd found that infants pay the most attention to events that are neither too predictable nor too surprising, and research on adults points the same way. If you write TypeScript all day, a week with Rust's type system will stretch you in productive ways, while a language with nothing familiar may produce more frustration than insight.
Ask "why does this work?" in code review. Questions about passing code surface hidden assumptions long before failing code does.

Make Questions Cheap for Your Team

Personal habits only go so far when asking questions feels risky. In a survey Gino conducted of more than 3,000 employees across many industries, only about a quarter said they regularly felt curious at work, and roughly 70 percent said they faced barriers to asking more questions. Engineering teams are not exempt. Anyone who has watched a junior developer stay silent through an entire design review has seen that tax being paid.

Leads and senior engineers set the exchange rate for questions. Say "I don't know, let's find out" in public, and say it often. In code review, turn uncertainty into a question instead of a verdict. In incident reviews, treat "why didn't we see this coming?" as a design problem rather than a search for someone to blame. Gino also recommends dedicating whole days to asking "Why?", "What if…?" and "How might we…?" On a dev team that might be a monthly hour where anyone can nominate a mystery in the codebase and the group digs in together. The product of that hour is shared understanding, which no amount of refactoring can generate on its own.

The Compound Interest of Small Detours

Curiosity rarely pays out on the day you spend it. The afternoon you spend learning to read EXPLAIN ANALYZE output feels unproductive until the night a slow endpoint stalls checkout and you are the only one who knows where to look. Hamming argued that knowledge compounds like interest, and exploration follows the same math, except the returns arrive as options. Each small detour adds another road to your mental map, and when requirements shift, the developer with the bigger map simply sees more exits.

So here is an experiment for this week. Find one question you skipped recently because you were busy, give it thirty uninterrupted minutes, and write down what you learned, even if the answer turns out to be dull. Then share it in the comments: which rabbit hole paid off the most for you?

Review Your Web3 Announcements Like Pull Requests

Sonia Bobrik — Sat, 03 Oct 2026 18:25:41 +0000

Most Web3 teams run every line of Solidity through tests, an audit, and at least two reviewers, then publish the launch thread from someone's phone at 2 a.m. The contract gets a threat model. The sentences around it get vibes. One agency's checklist of expert tips for successful Web3 PR ends on a principle that sounds like etiquette: balance vision with transparency. In the biggest markets you are likely to ship to, that principle now carries the force of law, and regulators read your copy the way auditors read your code. This post is about treating announcements, docs, and landing pages as what they have quietly become: production code with legal side effects.

The #ad That Cost $1.26 Million

On October 3, 2022, the SEC announced that Kim Kardashian would pay $1.26 million to settle charges over a single Instagram post about EMAX, a token sold by a project called EthereumMax. She had been paid $250,000 for that post, and it linked to a site with instructions for buying the token. The post even carried an #ad tag. It didn't matter. As the SEC's announcement of the Kardashian settlement made clear, anyone paid to promote a crypto asset security has to disclose the nature, the source, and the amount of what they received. A hashtag doesn't tell anyone who paid or how much. She also agreed to stay away from promoting crypto asset securities for three years.

Washington's approach to crypto enforcement has shifted a lot since then, and plenty of tokens may never be treated as securities. The rule she broke, though, is an anti-touting provision from 1933 that hasn't moved an inch. Stablecoin teams got a rulebook of their own in July 2025, when the GENIUS Act made it unlawful to market a payment stablecoin in a way that suggests it is legal tender, guaranteed by the U.S. government, or federally insured, and barred calling a token a payment stablecoin at all unless it complies with the law. For anyone writing a landing page, words like "insured," "guaranteed," and "government-backed" stopped being tone choices. They became liabilities.

Europe Wrote the Style Guide Into Law

MiCA's rules for token offers read like a review checklist someone drafted for your marketing channel. Under Article 7, marketing around a public offer or a listing must be clearly identifiable as marketing, fair, clear, not misleading, and consistent with the white paper. It has to point to where the white paper lives, down to a phone number and an email address, and carry a standard disclaimer saying no EU authority has reviewed or approved it. The clause that breaks the usual hype cycle comes last: where a white paper is required, no marketing may go out before it is published. The three-week teaser campaign that runs ahead of the docs is simply not an option.

Service providers have their own deadline behind them. The transitional window that let existing firms keep operating under national rules closed across the EU on July 1, 2026, so MiCA authorization is now the price of entry. That is exactly why ESMA warned about a "halo effect" back in July 2025: firms waving their license around in marketing so that unregulated products, such as crypto lending, feel just as protected. Its expectations are unusually concrete for a regulator. Label every product as regulated or unregulated, keep the two in separate parts of the website, and make users acknowledge a pop-up before they reach anything outside MiCA. Read that as an engineer and you'll see a routing rule, a UI component, and a feature flag.

In the UK, a Compliance Failure Can Look Like a CSS Bug

Crypto promotions aimed at UK consumers have sat inside the financial promotions regime since October 8, 2023. The FCA's rules for marketing cryptoassets ban "refer a friend" and new-joiner bonuses, demand clear risk warnings, and impose a 24-hour cooling-off period on first-time investors. A few weeks after the regime went live, the regulator listed the most common failures it was seeing. Next to claims about safety, security, or ease of use with no mention of risk sat an item every front-end developer should recognize: warnings that weren't prominent enough because of small fonts, hard-to-read colors, or poor placement on the page.

That is not a legal subtlety. It's a contrast ratio. In its first year, the regime produced 1,702 consumer alerts about illegal crypto promotions, the takedown of more than 900 scam crypto websites, and 56 apps pulled from UK app stores. In October 2025 the FCA opened its first court case against an offshore crypto exchange that kept promoting to UK users, and in April 2026 it led a coordinated push by seventeen regulators against unauthorized finfluencer promotions. The scope reaches well past banner ads. The FCA has warned that memes can count as financial promotions and has flagged private channels like Discord as places where promotions now spread, and communicating an unlawful promotion is a criminal offense. If your community lives in a Discord server, your announcements channel is a marketing surface.

Put Every Public Claim Through the Pipeline

None of this needs a legal department the size of your engineering team. It needs the controls you already trust for code, pointed at words instead. A small protocol team can wire up most of it in an afternoon:

One source of truth for every number. APY ranges, token supply, fees, audit status, and regulatory status live in a single versioned claims.yml that the docs, the landing page, and the press kit all import. MiCA's consistency requirement becomes a build property instead of a hope, and you stop publishing three different TVL figures in the same week.
CODEOWNERS for copy. Keep announcements, the white paper, and site text in the repo, and add a rule like /announcements/ @your-org/legal @your-org/protocol-leads so nothing merges without someone who can say "we can't promise that."
Risk warnings as tested components. Build the warning once, give it design tokens that clear WCAG's 4.5:1 contrast minimum, and add a visual regression test that fails when someone shrinks it, greys it out, or pushes it below the fold.
Regulatory status as data. Tag each product page regulated, unregulated, or geo-restricted, and let routing, labels, and the acknowledgment modal read that field instead of somebody's memory.
A paid-promotion register. Every compensated post, from KOL threads to ambassador shout-outs, gets a row: who was paid, how much, and where the disclosure appears.
A lint step for promises. Crude, cheap, and more useful than it looks.

#!/usr/bin/env bash
# claims-lint.sh: fail the build when copy promises what the protocol can't.
# Point the paths at wherever your public copy lives.
PATTERN='guaranteed (return|yield|profit|apy|apr)|risk[- ]?free|can.{0,3}t lose|(fdic|federally)[- ]insured|government[- ]backed|fixed (apy|apr|yield)|passive income'

grep -RniE "$PATTERN" announcements/ docs/ site/content/
status=$?

if [ "$status" -eq 0 ]; then
  echo "Flagged phrasing above: rewrite it or get a sign-off from legal."
  exit 1
elif [ "$status" -gt 1 ]; then
  echo "claims-lint could not read the content folders." >&2
  exit 2
fi
echo "No flagged phrases found."

It will throw false positives, and that's fine. An honest "not FDIC insured" disclosure trips the same rule as the claim it negates, so treat a red build here as a request for review rather than a verdict, and allowlist your standard disclosures once they settle. What matters is that risky phrasing can no longer reach production without a second pair of eyes, exactly like an unchecked external call.

Compliance and Good PR Want the Same Thing

Here's what makes this an easy sell to your growth team: every control above also makes you more credible to the people whose coverage you actually want. Journalists who have covered a few crypto collapses tend to read superlatives as a warning sign. A launch post where every number traces back to one file, every product carries an honest label, and every paid voice is disclosed reads like a team with nothing to hide. The regulator and the reporter are asking for the same thing: say less than you could, and make all of it checkable.

The teams still standing next cycle won't be the ones with the loudest threads. They'll be the ones where every public sentence has a reviewer, a source, and a commit hash.

Your Scoring System Is Also a Training Set for the People You Score

Sonia Bobrik — Thu, 10 Sep 2026 01:11:30 +0000

Every system that assigns a number to human output eventually teaches people how to produce that number. This is not cynicism about human nature, it is a measurable property of deployed software, and the finance industry has been running the longest and best-documented version of the experiment. The story of how executives quietly rewrote their vocabulary once algorithms started grading earnings calls is worth reading as an engineering postmortem rather than a market curiosity, because the failure mode it describes — a model that works beautifully until its subjects learn the rules — shows up in code review bots, abuse classifiers, fraud scores, and LLM evaluation harnesses with the exact same shape. If you ship anything that scores text, you are already inside this loop. The only question is whether you have instrumented it.

The Dictionary That Stopped Working Once It Was Public

The origin story is a measurement bug. Early attempts to score financial documents borrowed general-purpose psychology word lists, which counted words like liability, cost, and tax as negative. In a bank's annual report those are neutral bookkeeping vocabulary, so the scores were noise dressed up as signal. In 2011, Tim Loughran and Bill McDonald published a domain-specific negative word list that fixed the mislabeling, and the corrected tone measure predicted future earnings and stock returns. A working signal, cleanly validated.

Then the list was published. Within a few years, companies whose filings attracted heavy automated download traffic began stripping out precisely the tokens on that list. Not negative language in general. The specific vocabulary the dominant scorer penalized.

There is a rule in here that every engineer should internalize: a published rubric is a published exploit. The moment you document your linting heuristics, your spam features, your ranking factors, or your code-review checklist, you have handed the measured population a spec. They will build to it, usually without any intent to deceive, because building to the spec is what conscientious people do when a spec exists and consequences attach to it.

Bigger Models Raise the Price of Gaming, They Do Not End It

The standard response is to swap the lexicon for something that reads meaning instead of counting tokens. That works, for a while. When the BERT architecture was released in 2018, contextual embeddings made simple word substitution insufficient, because the model could tell the difference between a hedge and a synonym. The finance research found a second, measurable behavioral shift right after that release. Sophistication moved the equilibrium. It did not remove it.

The useful mental model is a cost curve, not immunity. Every upgrade to your scorer raises the effort required to satisfy it without genuinely changing the underlying thing you care about. Sometimes that effort is so high that gaming and honest improvement converge, which is the ideal outcome. Often it just filters out the unsophisticated and rewards whoever can afford better tooling.

Machines Are Now a Large Share of Your Readers

This is no longer a finance-only phenomenon, because the audience composition of the open web has changed. Cloudflare's breakdown of who is actually crawling websites found GPTBot's share of crawler traffic climbing from 2.2% to 7.7% in a single year, a 305% jump in raw requests, while several traditional indexers stayed flat. Your API docs, your changelogs, your error messages, and your incident write-ups are being parsed by systems that will summarize them for humans who never load your page.

That changes what "writing for your users" means. Documentation that is technically accurate but structurally hostile to parsing gets summarized badly. Documentation optimized purely for extraction gets thin and repetitive. Both failure modes are real, and the second one is the earnings-call trap arriving in your repo.

Six Practices That Keep a Scorer Honest

Score residuals, not levels. What predicts anything is deviation from what the subject's circumstances already explain. Levels are easy to shift; residuals are expensive to fake.
Keep a private holdout rubric. Publish the criteria that describe genuine quality, retain a scoring variant nobody outside the team has seen, and use divergence between the two as your alarm.
Monitor feature distributions, not just accuracy. Accuracy on stale labels stays flat while the input distribution rots underneath it. Track the mean and variance of every feature your model weights heavily.
Instrument the unscripted channel. In earnings calls, prepared remarks got sanitized while the analyst Q&A stayed comparatively honest. Every system has an equivalent: freeform commit messages, on-call chatter, support transcripts.
Treat each scorer release as an intervention. Version the model, timestamp the deploy, and check for behavioral breaks afterward. You are not just observing a population, you are perturbing it.
Budget for relabeling from day one. A scoring model has a half-life. Pretending otherwise means you will discover the decay through a business incident instead of a dashboard.

Instrument the Exposure Split

The single sharpest technique from the finance literature is a natural experiment: compare subjects heavily exposed to machine scrutiny against subjects barely exposed at all. If both cohorts drift together, that is real change in the world. If only the exposed cohort drifts, and it drifts specifically on the features your scorer rewards, you are watching adaptation.

That translates into maybe forty lines of production code:

def adaptation_signal(df, feature_cols, exposure_col, period_col):
    """Divergence in feature TRENDS between high- and low-exposure cohorts.
    Large positive values = the scored population is moving on the
    exact dimensions your model rewards. That's the alarm."""
    hi_cut = df[exposure_col].quantile(0.75)
    lo_cut = df[exposure_col].quantile(0.25)
    high = df[df[exposure_col] >= hi_cut]
    low  = df[df[exposure_col] <= lo_cut]

    signals = {}
    for f in feature_cols:
        h_trend = high.groupby(period_col)[f].mean().diff().mean()
        l_trend = low.groupby(period_col)[f].mean().diff().mean()
        signals[f] = h_trend - l_trend
    return dict(sorted(signals.items(), key=lambda kv: -abs(kv[1])))

Run it weekly against whatever your system scores. Alert on the top features by absolute divergence. It will not tell you why the gap opened, but it will tell you where to look, which is more than most teams have.

The Cost That Never Reaches the Dashboard

There is a second-order effect worth naming, because it is the part that damages products rather than models. Text optimized against a parser is text drained of information. When every quarterly update, every postmortem, and every release note is written in the same carefully neutral register, the artifacts get longer, smoother, and less useful to the humans they were originally written for. The information does not vanish. It migrates into residuals, tone, timing, and the things people say when they forget the transcript exists — signals that only well-equipped observers can extract. Everyone else reads polished prose and learns nothing.

That is the real bill. Not a degraded AUC on some internal benchmark, but an organization that has slowly optimized its own writing into noise while its dashboards report improvement.

Build Like Your Scorer Will Be Read

Assume your criteria will leak. Assume the population you measure is intelligent, motivated, and reading carefully. Design so that the cheapest path to a high score is the path you actually wanted, keep a private check on the gap between measured and genuine quality, and set a review date for every model the moment you deploy it. The teams that get burned are not the ones whose metrics get gamed. Every metric gets gamed. They are the ones who never built the instrument that would have shown it happening.

Your Vulnerability Gate Is Quietly Failing Open

Sonia Bobrik — Thu, 10 Sep 2026 01:11:06 +0000

Somewhere in your repository there is a line of configuration nobody has read in two years — a --severity HIGH,CRITICAL flag, a Dependabot threshold, a policy file that blocks a merge when a transitive dependency crosses CVSS 7.0. That line encodes an assumption about the outside world that stopped being reliably true in April 2026, and the backstory matters before you touch it: the shared naming system your entire toolchain gates on came within hours of going dark, as this reconstruction of how the world nearly lost its only shared catalog of software flaws documents in uncomfortable detail, and while the identifiers themselves survived the funding scare, the free enrichment layer that made those identifiers machine-actionable did not survive intact. The CVE ID is still there. The severity score bolted onto it increasingly is not.

The Number You Gate On Has an Owner

Almost every developer-facing scanner — Trivy, Grype, npm audit, Snyk's free tier, your registry's built-in checker — resolves a CVE ID into a severity by asking someone else. Historically that someone was the National Vulnerability Database, which took the raw record a CVE Numbering Authority published and added the parts machines need: a CVSS vector, a CWE classification, and CPE strings describing which product versions are actually affected.

That arrangement broke under arithmetic. NIST spelled it out plainly when it announced the new prioritisation model: CVE submissions grew 263% between 2020 and 2025, and the analysis capacity behind the database never grew to match. The fix was triage rather than expansion. Every unenriched record published before March 1, 2026 was moved into a status called Not Scheduled — roughly 29,000 of them by most counts — and going forward only vulnerabilities that intersect a narrow set of criteria get full treatment. NIST published the new criteria in its own operations notice, and the shape of them is worth internalising: known exploitation, software the US federal government runs, and critical software as defined by Executive Order 14028. Independent estimates put the share of incoming CVEs that will clear that bar at somewhere between 15% and 20%.

The volume side of the equation is not slowing down either. The CVE program published 48,185 records in 2025, up 20.6% on the 40,009 published in 2024, with 484 numbering authorities feeding the pipeline as of January 2026 and first-quarter 2026 submissions running roughly a third ahead of the year before. FIRST forecast another record year. Whatever your scanner does with an unenriched record, it is going to be doing it a lot.

Fail Open, Fail Closed, or Fail Silently

Here is the concrete failure mode. A CVE lands with no NVD-supplied CVSS vector. Your scanner asks for a severity and gets nothing back. What happens next depends entirely on implementation details you almost certainly never audited.

Some tools fall back to a severity supplied by the CNA at publication time, which is often present and often disagrees with what NVD would have assigned. Some fall back to an ecosystem source like the GitHub Advisory Database, which curates its own severities and covers open-source packages well but hardware and enterprise appliances poorly. Some simply emit UNKNOWN. And a gate configured as "block if severity is HIGH or CRITICAL" treats UNKNOWN as not high — which means the build goes green and the merge lands. That is failing open, and it is failing open silently, which is the worst variant because nothing in your logs looks wrong.

The inverse configuration is not better. Teams that flip to "block on anything unresolved" discover their pipeline red-lining on hundreds of records with no context attached, and within about two sprints somebody adds a blanket ignore rule that never gets removed. Alert fatigue is not a personality flaw; it is the predictable output of a gate that cannot distinguish signal from absence of data.

Rewire Around Exploitation, Not Around Severity

The useful correction is not finding a replacement severity feed. It is admitting that a static base score was always a weak proxy for "should this block my release," and replacing it with signals that answer the question directly.

Make known exploitation a hard gate. CISA's Known Exploited Vulnerabilities catalog lists flaws with confirmed in-the-wild abuse, ships as CSV and JSON, and was reaffirmed as the anchor for federal patching timelines under Binding Operational Directive 26-04 in June 2026. It is small, it is free, and a hit on it justifies waking someone up.
Add EPSS as a soft gate. The Exploit Prediction Scoring System publishes daily probabilities that a given CVE will be exploited in the next 30 days. FIRST's own analysis puts the share of published vulnerabilities ever exploited at roughly 2–7%, so an EPSS threshold cuts triage volume by an order of magnitude without discarding much real risk.
Treat missing severity as its own state. UNKNOWN is not zero and it is not critical. Route it to a review queue with a service-level target rather than letting a boolean comparison decide.
Prefer package-level data for package-level problems. OSV records express affected version ranges per ecosystem, which is the question a lockfile actually poses. CPE strings were designed for enterprise asset inventories and map onto package.json badly.
Record which source produced each verdict. When a build fails, the log should say whether the score came from a CNA, from NVD, from an ecosystem advisory, or from a fallback default. Without that, you cannot debug the gate at all.

Fragmentation Is Now Your Problem Too

The redundancy built after the 2025 scare is genuine progress and it also generates work. ENISA's European Union Vulnerability Database went live in May 2025 and assigns its own EUVD identifiers on top of CVE IDs, built on CSAF and driven partly by Cyber Resilience Act obligations. CIRCL's Global CVE system opened to the public in January 2026 with a federated model where numbering authorities issue identifiers without waiting on central approval. CISA's Vulnrichment effort distributes enrichment across authorised data publishers. GitHub keeps its GHSA namespace.

None of these replace CVE. All of them mean the same defect can now appear in four catalogs, at four different times, carrying four different severity judgements and four different claims about which versions are affected. If your tooling ingests more than one source — and increasingly it will, whether or not you chose that — you need a canonical identifier map and an explicit precedence rule for conflicts. Otherwise you get duplicate findings, contradictory dashboards, and engineers who learn to distrust the scanner entirely.

What Actually Changed

For twenty-five years, vulnerability triage was something a developer could outsource to a public good. A free service turned an identifier into a decision, and CI configs were written on the quiet assumption that the service would always be there and always be complete. Both halves of that assumption are now formally retired — not by a catastrophe, but by a published policy change from the organisation that maintained it, arrived at honestly because the volume made anything else impossible.

The catalog survived. The convenience did not. Prioritisation has moved from an external dependency back into your codebase, where it now belongs. Open the policy file, check what it does when the score is missing, and fix that first.