DEV Community: Doogal Simpson

Why QA Engineers Are Your Team's Invisible Safety Net

Doogal Simpson — Wed, 02 Sep 2026 16:59:12 +0000

TL;DR: QA engineers operate in a tough paradox: when they do their job perfectly, their work is invisible. However, their role is incredibly challenging, requiring them to constantly deliver difficult feedback to defensive developers. To build resilient software, we must recognize QA as a collaborative safety net rather than an obstacle.

Think about the air conditioning in a large office building. When it works perfectly, nobody thinks about it. You sit at your desk, comfortable, completely focused on your work. But the moment it breaks? Everyone is furious, demanding to know who is responsible.

Quality Assurance (QA) engineers live in that exact same reality. They are the invisible infrastructure of a software team. When they do their job exceptionally well, the product simply works, and they get zero credit. But when something slips through to production, all eyes turn to them.

Why does QA feel invisible until everything breaks?

When quality assurance operates at peak efficiency, the end-user experiences a seamless product, which can mistakenly lead management to believe the QA role is redundant. The true value of QA is only realized when they are gone and critical bugs slip directly into production.

Imagine a hypothetical scenario where a team is under pressure to ship a new subscription billing system. To save time, management decides to bypass the QA phase and deploy directly. Within hours, edge-case database locks begin failing, charging users twice.

Without QA, we lose our safety net. A great tester doesn't just click buttons; they map out the weird, dark corners of your logic that you didn't have time to think about. When everything runs smoothly, it isn't because the code was perfect from day one—it is because QA caught the cracks before the concrete dried.

How do we handle the friction of QA bug reports?

The natural friction between developers and QA exists because a tester's job is to deliver bad news to engineers who are emotionally invested in their work. Reframing bug reports as a collaborative tool rather than a personal critique is key to a healthy engineering culture.

Let's be honest: nobody likes being told their baby is ugly. When you spend three days writing an elegant algorithm, hearing 'this doesn't work' can immediately make you defensive. But we have to separate our self-worth from our code. QA is not trying to catch you failing; they are trying to prevent the team from failing in front of the customer.

Here is how we can reframe these interactions to build a better working relationship:

Scenario	The Defensive Reaction	The Collaborative Response
QA finds an edge-case race condition	'That is an unrealistic scenario; nobody will do that.'	'Great catch. Let's look at how we can handle that lock state safely.'
QA rejects a ticket due to UI misalignment	'It is close enough. We have more important things to ship.'	'Thanks for spotting that mismatch. Let me tweak the CSS to match the designs.'
QA asks for more API documentation	'The code is self-documenting; just read the PR.'	'I will update the Swagger spec to make this easier for you to test.'

How can developers build a stronger partnership with QA?

Developers can build a better partnership by involving QA early in the development cycle, sharing context before writing code, and respecting their expertise. Treating QA as peer engineers rather than a final gatekeeper reduces friction and improves software quality.

If you only hand things over to QA ten minutes before a release deadline, you are setting everyone up for failure. Instead, try these three practices:

Shift Left: Invite your QA engineers to the initial design and RFC reviews. They will spot logical flaws in your planning before you write a single line of code.
Write Testing Notes: When you submit a pull request, don't just write 'done.' Explain how you built it, what components were affected, and where you think the weak points might be.
Celebrate the Saves: When a QA engineer catches a critical bug before a release, celebrate it publicly in your team chat. Normalize the idea that catching bugs is a massive win for the entire team.

FAQ

Why can't automated tests completely replace manual QA?

Automated tests only verify that the software behaves exactly as the developer programmed it to behave. They cannot evaluate the actual user experience, spot logical inconsistencies in the product design, or perform the creative, exploratory testing that a skilled human QA engineer does.

How should we handle disagreements over what is considered a bug?

When a developer and a tester disagree on a bug, the focus should shift away from technicalities and toward the user. Ask: 'Does this behavior harm the user experience or trust?' If the answer is yes, it is a bug. If it remains ambiguous, bring in the product manager to clarify the requirements.

How does involving QA early actually save development time?

Finding a bug during the design phase costs almost nothing to fix. Finding a bug during development takes hours of rewrite. Finding a bug in production can cost thousands of dollars and damage company reputation. Involving QA early prevents bugs before they are even built.

Handling Partial Failures in Distributed Systems

Doogal Simpson — Tue, 01 Sep 2026 15:01:39 +0000

Quick Answer: To handle partial failures in distributed systems, avoid complex distributed transactions. Instead, design for the "unhappy path" using an orchestration pattern (like the Saga pattern). This ensures that if a downstream service fails after a payment is processed, compensating actions automatically roll back the transaction and refund the user.

Let's be honest: a lot of the code running in production right now is built on a prayer. I've watched too many developers spend weeks writing elegant features, only to completely ignore what happens when the network inevitably hiccups. They design exclusively for the "happy path" and treat system failures as a minor afterthought.

But here is the reality of distributed systems: the network is unreliable, APIs crash, and validation fails. If you do not actively write code to handle these failures, you will eventually leave your data in a corrupted state, which usually means taking a customer's money and giving them nothing in return.

Why does the "happy path" fallacy break distributed systems?

The happy path fallacy is the assumption that every API call, database write, and network request in a sequence will succeed. When developers write linear code without handling intermediate failures, it leads to inconsistent system states—like charging a customer without delivering their purchase.

I cannot tell you the number of times I've seen systems written in such a way where the code takes the money, sits in a middle server, and then calls a downstream server to deliver the product. If that downstream call fails, you are stuck in the absolute worst-case scenario: you have the customer's money, but they have no product.

Some developers might joke that keeping the money without delivering the product is a win, but I guarantee you it is the fastest way to get your engineering team into deep trouble.

// The dangerous "Happy Path" approach I see all the time
async function checkout(order) {
  // 1. Take the money
  await paymentService.charge(order.userId, order.amount);

  // 2. Network drop? Validation failure?
  // If this fails, the user is charged but gets no inventory!
  await inventoryService.reserve(order.items);
}

Why should I avoid distributed transactions (2PC)?

Distributed transactions, such as Two-Phase Commit (2PC), force services to lock databases until all nodes agree on a change. This tight coupling destroys system throughput, increases latency, and introduces a single point of failure that defeats the purpose of microservices.

When developers realize their linear code is fragile, their first instinct is often to reach for distributed transactions. I strongly advise against this. Distributed transactions require database locks across service boundaries. If one of your services experiences a network lag or goes down mid-transaction, your entire system grinds to a halt while holding those locks open. It simply does not scale.

How do I design a resilient rollback workflow?

Rather than relying on database locks, use an orchestration-based Saga pattern to manage distributed state. An orchestrator tracks each step of a multi-service workflow and automatically triggers compensating actions (like refunds) if a downstream step fails.

Instead of trying to prevent failures with database locks, I build systems that accept failure as an inevitability. I recommend using an orchestration system. An orchestrator is a resilient state machine that coordinates the workflow steps. If a downstream step fails, the orchestrator detects the failure and executes compensating actions—like issuing a refund—to roll back the entire workflow to a clean state.

Strategy	Latency	Coupling	Failure Handling	Best For
Two-Phase Commit (2PC)	High	Tight (database level)	Automated rollback, but blocks resources	Monoliths, single DBs
Choreographed Saga	Low	Loose (event-driven)	Complex to trace, relies on event chains	Simple, decentralized workflows
Orchestrated Saga	Medium	Loose (central controller)	Explicit compensating steps (rollbacks)	Complex business transactions

FAQ

How do I test unhappy paths in my local development environment?

I suggest using chaos engineering tools or configuring mock downstream APIs to return random timeouts, 500 Internal Server Error responses, or validation failures. If your orchestrator does not automatically trigger a compensating rollback during these simulated network drops, your recovery logic is broken.

What is a compensating transaction?

A compensating transaction is an explicit action designed to undo the effects of a previous, successful step in a workflow. For example, if step one charges a card and step two fails to allocate warehouse stock, the compensating transaction is an API call that issues a refund to the customer's card.

What tools should I use for workflow orchestration?

I highly recommend looking at dedicated workflow engines like Temporal, Camunda, or AWS Step Functions. These platforms allow you to define resilient state machines that handle retries, timeouts, and compensating rollbacks out of the box, saving you from writing custom, fragile error-handling boilerplate.

Stop Terminal Prompting: A Markdown-First AI Workflow

Doogal Simpson — Mon, 31 Aug 2026 08:27:52 +0000

Quick Answer: Relying on interactive terminal prompts for slow AI agents forces us into constant, exhausting context-switching. To reclaim your flow state, don't code in the terminal. Instead, draft your thoughts, specs, and feedback in a local Markdown file first, using it as an offline scratchpad to keep your thinking deliberate.

Is anyone else's brain just completely fried?

Seriously. If you've been spending most of your week working with autonomous developer agents like Claude Code or Aider, you know exactly what I'm talking about.

You kick off a task, the agent spins up, and it's doing its thing. While it's spending five or ten minutes running, you think, "Well, I should probably be doing something productive." So you open a new tab and start on another task.

Before you know it, you've got six or seven different things on the go at once. It's like spinning plates, and the constant context-switching is absolutely exhausting. We've traded our old "compiling" coffee break for a frantic, reactive feedback loop. Here's how I broke out of that trap.

Why do AI agents make us feel so exhausted?

Working with autonomous agents is draining because their slow runtimes tempt us to multitask, which completely destroys our flow state. Instead of thinking deeply, we get stuck in a reactive loop of chasing terminal outputs across six different tabs.

Imagine you're building a complex schema migration. In the past, you wrote the migration, ran the test suite, and had a clear, linear path of thought. With an agent, you write a quick prompt, and then you wait. That waiting period is where the trouble starts.

Because the terminal is interactive, we treat it like a real-time chat app. We feel the urge to respond immediately to whatever partial solution or gibberish the agent spits back. This constant disruption prevents us from entering a true flow state. We stop being software architects and become reactive prompt managers.

How does a "Markdown-first" workflow solve agent fatigue?

A Markdown-first workflow replaces frantic terminal prompting with a slow, local scratchpad inside your IDE. Instead of reacting instantly to whatever the agent spits out, you brain-dump your thoughts into a file first, letting you process everything at your own pace.

You treat the markdown file as a buffer. You write down what you want to achieve, analyze what the agent did, and draft your corrections offline before the agent even sees them.

Here's an example of the scratchpad structure I keep open in my editor:

# Refactoring JWT Auth Middleware

## Agent Output Analysis
- The agent successfully updated the token validation.
- It missed handling the expired token database cleanup.

## Next Prompt Instructions
1. Keep the current middleware logic.
2. Add an automated cron cleanup function in `cron/cleanup.go`.

By keeping your thoughts in a file, you don't lose your place when you inevitably have to step away or wait for a long agent execution run.

How do you transition from terminal prompting to Markdown files?

Stop typing complex prompts directly into your CLI. Write them in a local scratchpad.md file first, then feed that file to your agent when you're ready, keeping your focus inside your IDE and away from terminal-induced panic.

This approach fundamentally changes how you interact with AI. Here's how the two workflows stack up:

Feature	Terminal-First Prompting	Markdown-First Prompting
Pacing	Fast, reactive, and urgent	Deliberate, offline, and controlled
Cognitive Load	High (constant context-switching)	Low (single source of truth)
Focus	Fragile (distracted by execution logs)	Deep (focused on system architecture)
Output Quality	Messy, iterative trial-and-error	Structured, high-context instructions

Writing things down gives your brain the space to actually think about the engineering problem rather than just reacting to the screen.

FAQ

How do I feed the Markdown file back to the AI agent?

Most modern agent CLIs allow you to pass files as context. You can either reference the markdown file directly in your command (like claude-code -f scratchpad.md), or simply copy and paste your structured "Next Instructions" section straight into the prompt when the agent finishes its run.

Doesn't writing Markdown slow down the development process?

Yes, it slows you down, and that's precisely the point. By forcing yourself to write a couple of paragraphs explaining your thoughts before running a command, you avoid sending the agent down hallucinated rabbit holes that take thirty minutes to debug.

Should I commit these Markdown scratchpads to git?

I don't recommend it. I add *.scratchpad.md or a dedicated .prompts/ directory to my global .gitignore. These files are meant to be temporary, highly personal cognitive buffers to help you think through a task, not permanent project documentation.

How to Build a Scale-Proof Distributed Counter

Doogal Simpson — Mon, 31 Aug 2026 08:24:32 +0000

To build a scale-proof like counter, you must decouple user actions from database writes. Start with optimistic UI updates for instant feedback, log the event to a user-specific store, and process counts asynchronously using in-memory distributed buffers before periodically flushing aggregated totals to a persistent database.

If I'm building a weekend side project, a single SQL UPDATE statement is fine. But at YouTube scale—where millions of users hit the like button simultaneously across the globe—that simple write becomes a massive architectural bottleneck.

Here is how I would design a distributed system that handles this kind of write-heavy throughput without melting the database.

Why can't I update the database immediately on every click?

Direct database updates fail at scale because high-volume concurrent writes to the same database row cause severe lock contention. To handle millions of writes per second, I must shield the database using an asynchronous, distributed processing layer.

Let's say a viral video goes live. Within seconds, ten thousand users hit the like button. If I try to execute UPDATE videos SET likes = likes + 1 WHERE id = 123 ten thousand times simultaneously, the database has to lock that specific row for each transaction. This creates a massive queue, drives CPU usage to 100%, and eventually crashes the database.

To solve this, I decouple the user interface from the system of record. When you click that like button, the UI updates instantly. The application doesn't wait for a database roundtrip; it optimistically assumes success and increments the counter locally on your screen.

How do I design a system to handle millions of likes at scale?

To scale high-throughput counters, I split the operation into two parallel pipelines: a user-centric ledger and a video-centric aggregated counter. This decouples the identity tracking ("who liked what") from the high-frequency numerical aggregation ("how many likes total").

Once the client emits the request, I split the backend work into two independent flows:

The User-Centric Flow: I log the specific relationship (e.g., "User X liked Video Y"). This data is sharded horizontally by User ID, making it highly distributed and easy to scale. It ensures that if you refresh the page, I can look up your history and show that you already liked the video.
The Video-Centric Flow: I emit an asynchronous event—"User X liked Video Y"—into an ingestion pipeline to update the overall count. This pipeline is built to ingest a wild, constant stream of events without slowing down.

Pipeline Component	Data Responsibility	Scale Strategy	Storage Medium
User-Centric	Tracks specific relationships ("User X liked Video Y").	Sharded horizontally by User ID.	Relational DB or Key-Value Store.
Video-Centric	Tracks overall aggregations ("Video Y has N likes").	Distributed in-memory counters & batching.	In-memory cache (Redis/Custom) -> Persistent DB.

How do I implement distributed in-memory counters?

Distributed in-memory counters ingest a stream of event messages across multiple parallel processes to track counts in local memory. A periodic background worker then sweeps these in-memory nodes, aggregates the counts, and flushes them to persistent storage in a single, batched database transaction.

Instead of hitting the database for every single event, I route the incoming stream of like events across multiple memory processes. Let's say I have five instances of a counter service running. When a "like" event occurs, it is routed to one of these instances, which increments an in-memory counter for that video.

Every few seconds, a background process runs to reconcile these numbers. It reads all the distributed, in-memory counters across the processes, sums them up, and flushes the aggregated count to the persistent database. If a video receives 10,000 likes in three seconds across the fleet, the database experiences only a single write operation of +10,000 instead of 10,000 individual writes.

This is how I balance real-time user experiences with backend storage limits.

FAQ

What is an optimistic UI update and why is it used for likes?

An optimistic UI update is a frontend pattern where the client-side UI immediately reflects a successful state (e.g., highlighting the like button and incrementing the count) before the server confirms the action. I use it to ensure a fast, responsive user experience despite the asynchronous nature of the backend queue.

How do distributed counters handle eventual consistency?

Because distributed counters rely on in-memory buffering and periodic database flushing, the public-facing like count is eventually consistent. While you will immediately see your own like, other users will see the total count update in small jumps every few seconds as background processes reconcile and write the batched counts to persistent storage.

What happens to the counter if an in-memory node crashes?

If a node crashes before flushing, some in-flight counts can be lost. To prevent this, I run durable, append-only event streams (like Apache Kafka) behind the ingestion layer. If a worker node goes down, a new node can replay unprocessed events from the stream to reconstruct the correct counts before writing them to the database.

Why AI Coding Can't Speed Up B2B Sales Cycles

Doogal Simpson — Sun, 30 Aug 2026 13:04:11 +0000

Modern development tools and AI have supercharged our ability to ship code, but they haven't changed the speed of human decision-making. High-ticket B2B sales still require weeks or months to build trust, meaning your overall business velocity is still gated by human relationships, not your deployment pipeline.

Imagine you just built a revolutionary B2B product. Using LLMs and modern frameworks, you and a small team designed, coded, and deployed the entire system in a single weekend. It is blazing fast, highly scalable, and ready for production.

But the moment you try to sell it to an enterprise client, your momentum hits a brick wall.

A colleague of mine recently summarized this paradox perfectly during a lunch-and-learn: "Computers got faster; humans didn't."

Even if AI allows us to build software at lightspeed, we are still selling that software to human beings. And humans do not operate at the speed of silicon.

Why hasn't AI shortened the B2B sales cycle?

AI cannot automate the human trust required to close high-value business deals. When an enterprise buyer signs off on a software contract worth tens of thousands of dollars, they are risking their own professional reputation, which requires human-to-human relationship building.

If you are selling a $10-a-month browser extension, self-serve checkout works fine. But when the price tag jumps to five or six figures, the buying dynamic changes entirely. The enterprise buyer has a set of existential questions that no marketing page or AI agent can fully resolve:

Will this integration break our existing legacy systems?
If this software goes down at 3:00 AM on a holiday, who is going to help us fix it?
Is this startup still going to exist in two years?
Will adopting this tool get me fired if it fails?

Answering these questions takes time. It requires multiple meetings, product demonstrations, and technical deep dives. Trust is a slow-burning currency that must be earned; it cannot be generated by a prompt.

What is the real bottleneck in B2B software adoption?

The primary bottleneck in enterprise sales is not code production, but institutional hurdles like security reviews, legal redlining, and political alignment. While engineers can push updates in seconds, legal teams and information security (InfoSec) departments still operate on human schedules.

To understand where the time actually goes, look at the friction points in a typical enterprise procurement lifecycle:

Process Phase	Operational Velocity	Primary Bottleneck	AI Impact
Code Generation	Extremely High	None	High (Massive speedup)
Security & Compliance	Low	InfoSec questionnaires, SOC2 audits	Low (Still requires human sign-off)
Legal & Procurement	Low	Master Service Agreements, legal redlining	Minimal (Lawyers must review liability)
Trust & Relationships	Extremely Low	Executive consensus, political risk	None (Purely human relationship)

An LLM can generate a compliance policy in seconds, but it cannot force a corporate legal team to approve a liability clause any faster. The bottleneck has simply shifted entirely from engineering to operations.

How can engineering teams adapt to slow sales cycles?

Engineers must shift their focus from raw feature velocity to building "trust-enabling" technical assets that make the human sales process frictionless. This means prioritizing robust security, clear documentation, and dead-simple proof-of-concept setups over endless new feature development.

If your sales team takes six months to close a deal, shipping ten new features this week won't help them close it any faster. Instead, use your engineering cycles to remove the technical friction points that slow down human decision-makers:

Build a Self-Service Sandbox: Allow prospective buyers to play with your product in a safe, isolated environment without needing a complex setup process.
Automate Compliance Readiness: Make it easy for your sales team to hand over SOC2 reports, penetration testing results, and architecture diagrams.
Simplify Data Portability: Build clear import and export pipelines. Buyers are more likely to trust you if they know they can easily get their data back out if the relationship doesn't work.

At the end of the day, shipping code is only half the battle. If we want our businesses to move faster, we have to stop treating sales as an afterthought and start engineering our products to build trust by design.

FAQ

Can AI agents eventually automate the B2B sales process?

While AI can automate lead generation, personalized outreach, and initial scheduling, it cannot automate the final stages of B2B sales. High-value transactions require accountability, and buyers will always want a human account executive to hold responsible for delivery and support.

How does the human bottleneck impact startup runway?

Startups often burn through their capital because founders assume fast development cycles will translate to fast revenue. If you build your product in three months, you must still budget for a six-to-nine-month enterprise sales cycle before that product generates significant cash flow.

Should developers join sales calls to speed up the process?

Yes. Having a technical engineer on a call can dramatically cut down the sales cycle. Developers can answer complex architecture and security questions on the spot, bypassing weeks of back-and-forth email chains between the buyer's IT department and your sales team.

Fix AI Agent Jargon with Simplified Technical English

Doogal Simpson — Thu, 27 Aug 2026 12:36:00 +0000

Tired of Claude Code generating bizarre, overly dramatic jargon like "load-bearing spine"? You can fix this by enforcing Simplified Technical English (STE) in your system instructions or .claudemd files. This 1970s aerospace standard restricts vocabulary, forcing your AI agent to communicate in clear, direct, and highly actionable prose.

"The load-bearing spine has hit a ceiling, and that is a significant foot gun with a large blast radius."

If you have spent any time recently working with AI coding agents, you have probably stared at your terminal reading absolute gibberish like this, wondering: What on earth are you trying to tell me?

I asked a straightforward technical question, and instead of a direct answer, I got a theatrical performance. It is incredibly tiring to translate AI metaphors back into plain English just to figure out which line of code actually broke.

Fortunately, there is a remarkably elegant fix for this. The solution does not involve complex prompt engineering; instead, it leverages a fifty-year-old aerospace standard: Simplified Technical English (STE).

Why does Claude Code output weird technical jargon?

AI models generate overly dramatic jargon because they are trained on vast internet corpuses where technical writing is often cluttered, metaphorical, and performative. To sound authoritative, the model indexes on complex vocabulary and metaphorical hand-waving instead of simple, direct statements.

Imagine a scenario where your team is debugging a database lock. A human engineer would say, "The transaction is blocked." An AI model, eager to please and sound sophisticated, might describe it as a "temporal execution bottleneck causing systemic architectural paralysis."

This happens because reinforcement learning from human feedback (RLHF) often rewards models for sounding smart and comprehensive. Without strict stylistic constraints, the agent defaults to verbose, metaphorical explanations that add cognitive load rather than solving your problem.

What is Simplified Technical English (STE) and how does it help?

Simplified Technical English (originally ASD-STE100) is an international writing standard developed in the 1970s for aerospace maintenance manuals to ensure complex procedures were clear and unambiguous. It reduces grammatical complexity and restricts vocabulary so that there is only one word for one meaning.

In the 1970s, aerospace engineers realized that overly complex manuals were leading to maintenance errors. They created STE to strip away ambiguity. By applying these same constraints to your AI agent, you force it to stop using metaphors and stick to direct, literal descriptions.

Here is how standard AI-speak compares to Simplified Technical English:

AI-Generated Jargon	Simplified Technical English (STE) Equivalent	Why it works better
"The load-bearing spine has hit a performance ceiling."	"The main database query is slow because CPU usage is at 100%."	It identifies the exact component and issue without metaphors.
"This architectural change has a massive blast radius."	"This change affects multiple dependent services."	It provides a clear risk assessment without dramatic language.
"We need to orchestrate a paradigm shift in our caching layer."	"We must change how the cache works."	It is direct, simple, and actionable.

How do I configure my AI agent to use Simplified Technical English?

You can enforce STE by injecting style rules directly into your agent's system configuration files, such as .claudemd, developer_instructions.md, or your global AI memory settings. Once these rules are in place, the model will automatically filter its reasoning into clean, structured prose.

If you are using Claude Code or similar terminal-based agents, create or edit your project's instructions file and append this brief system prompt:

# Rules for Output Style:
- Write all responses in Simplified Technical English (STE).
- Use short, active sentences (under 20 words).
- Avoid metaphors, idioms, and dramatic jargon (e.g., "blast radius", "foot gun").
- Use only one verb per sentence to describe actions.
- Do not use abstract nouns when concrete nouns are available.

By establishing these boundaries, you bypass the theatrical fluff. The agent stops trying to write a software engineering blog post in your terminal and starts acting like a highly efficient technical documentation tool.

FAQ

Does forcing STE limit the AI's ability to solve complex technical problems?

No. Restricting the output language to STE does not degrade the model's reasoning capabilities. It only changes how the model formats and explains the solution, stripping away confusing analogies while keeping the technical logic intact.

Where should I put these STE instructions for tools like Claude or Cursor?

You should place these instructions in your project's root configuration files. For Claude Code, use .claudemd or your custom developer instructions. For Cursor or VS Code Copilot, paste the rules into your system prompt or .cursorrules file.

What are the core rules of Simplified Technical English I should enforce?

The core rules are to keep sentences under 20 words, restrict verbs to active voice, use only one meaning per word (e.g., use "show" instead of "demonstrate" or "reveal"), and strictly ban metaphors and slang.

How YouTube Scales Watch History Using Bigtable

Doogal Simpson — Wed, 26 Aug 2026 14:27:35 +0000

TL;DR: YouTube scales watch history for billions of users using Google Bigtable. Instead of running expensive SQL query sorts on the fly, they design a massive, sorted key-value store where the row keys are structured as UserID plus a Reverse Timestamp. This allows YouTube to fetch a user's most recent videos instantly via a high-performance prefix scan.

If you ask me, one of the coolest things about YouTube is clicking on your history page and seeing your recently watched videos load instantly. It feels like a trivial feature. But when I see developers talk about building this at scale—dealing with billions of active users writing and reading at the exact same time—they often overlook how quickly a traditional database will choke. You can't just throw a standard SQL database at this problem unless you want to melt your servers and leave your users staring at a loading spinner.

Why does SQL fail at YouTube-scale watch history?

When I see engineers struggle with scaling history feeds, they usually try to optimize SQL queries that just aren't meant for this volume. Relational databases fail at this scale because sorting billions of rows on the fly using indexes is incredibly CPU-heavy. Under massive concurrent write-and-read loads, keeping index trees balanced while scanning them causes severe database locking.

Let's say we have a traditional SQL table tracking watch events. Every time a user watches a video, we append a row. When they want to see their last ten videos, the database has to search the index, sort those specific records by timestamp in descending order, and return the top ten.

Doing this for billions of users watching videos simultaneously is like trying to rebuild an airplane engine mid-flight. The database simply cannot keep up with the index maintenance and the sorting overhead. If you try to run this under real-world load, your database will quickly fall over.

What is Google Bigtable and how does it store data?

Google Bigtable is a distributed, NoSQL database designed to handle massive workloads by functioning like a giant, lexicographically sorted map. Instead of supporting complex relational joins, it stores data as simple key-value pairs sorted alphabetically by their row keys.

From what I've seen, if you want absolute speed at scale, you have to sacrifice relational flexibility—there's no way around it. Bigtable doesn't do joins, and it doesn't do complex query execution plans. Instead, it acts as a massive, distributed, sorted map.

Because the keys are pre-sorted, if you scan a range of keys, Bigtable reads them sequentially off the storage medium. This means if we can design our keys correctly, we can guarantee that a user's history is stored together, pre-sorted, and ready to read in a single disk seek.

What is a reverse timestamp and how does it solve sorting?

To me, the cleverest part of this entire design is how they avoid sorting altogether using a reverse timestamp. A reverse timestamp is simply the maximum possible epoch integer value minus the current epoch timestamp, which naturally forces newer records to the top of an alphabetical index.

Let's look at how the math works. In a normal timeline, larger numbers represent more recent events. But in an alphabetically sorted map, we want the most recent events to appear first. By subtracting the current time from a fixed maximum, a newer, larger timestamp yields a smaller reverse timestamp.

When Bigtable sorts these keys alphabetically, the smallest keys naturally rise to the top—meaning your absolute newest watch history is always at the very beginning of your data range. Here is how you generate this key structure:

function generateHistoryKey(userId, timestampMs) {
  const MAX_EPOCH = 9223372036854775807n; // 64-bit max integer
  const reverseTimestamp = MAX_EPOCH - BigInt(timestampMs);
  return `user#${userId}#${reverseTimestamp}`;
}

How does a prefix scan retrieve watch history instantly?

A prefix scan allows the database to locate the starting boundary of a user's data instantly and read only the first few sequential rows. Because the row keys are pre-sorted in reverse chronological order, the database simply reads the first 10 rows and terminates the query.

Let's look at how Bigtable stores this on disk for a hypothetical user, user_123:

Row Key	Video ID	Actual Time
`user_123#9223372036854575807`	`vid_999`	3 seconds ago (Most Recent)
`user_123#9223372036854675807`	`vid_888`	2 minutes ago
`user_123#9223372036854775807`	`vid_777`	1 hour ago

To fetch the history, the application queries Bigtable for keys starting with the prefix user_123# with a limit of 10. Bigtable jumps directly to the first key starting with user_123#, reads the first 10 rows sequentially, and returns them. No sorting, no heavy index scanning, and absolutely no loading spinners.

FAQ

Can you use reverse timestamps in relational databases like PostgreSQL?

While you can physically store reverse timestamps in a relational database, it is generally unnecessary. Relational databases use B-Tree indexes which can be scanned backward just as quickly as forward. Reverse timestamps are a design pattern optimized specifically for distributed, NoSQL key-value stores that rely heavily on forward-only prefix scanning.

What happens when the maximum epoch timestamp is reached?

If you use a standard 64-bit integer as your maximum epoch value, the maximum timestamp is so far in the future (the year 292,277,026,596) that the Earth's sun will have run out of hydrogen before your keys roll over. For all practical software applications, the limit is functionally infinite.

Why not use an in-memory cache like Redis for each user's history?

While Redis is incredibly fast, keeping the entire watch history of billions of users in-memory is prohibitively expensive. Bigtable balances the high write-throughput of an in-memory system with the cost-effective storage of persistent disk, pulling data blocks into memory only when they are actively queried.

Why You Should Wait 3 Months for Open-Source AI

Doogal Simpson — Wed, 26 Aug 2026 14:23:22 +0000

Stop chasing the bleeding edge of proprietary AI models. By waiting just three months to adopt new technology, you allow cheaper, highly stable open-source alternatives to catch up in intelligence. This pragmatic lag avoids buggy releases, lowers infrastructure costs, and frees you from supporting restrictive vendor ecosystems.

I’ve realized recently that I tend to play video games about two years after they actually launch. It’s not some grand, calculated plan—I’m just busy, and I don't get around to them right away. But this accidental delay has some massive advantages. By the time I buy a game, it's usually on sale, the game-breaking bugs have been patched, and the exhausting hype cycle is long dead. I get a stable, polished experience for a fraction of the cost.

It turns out, this exact same patient approach is the smartest way to handle AI models.

Should you always use the latest frontier AI models?

No, you should not default to the newest frontier models for production applications. While bleeding-edge models offer high benchmarks, they are expensive, prone to unannounced behavioral shifts, and lock you into proprietary ecosystems.

Imagine you are building an automated classification system. If you jump on the newest proprietary model on day one, you are paying peak API prices. You are also at the mercy of rate limits, sudden API deprecations, and undocumented model updates that can silently break your prompts.

By stepping back from the absolute leading edge, you bypass this chaotic testing phase. You let other teams spend their time and budget finding the bugs, while you focus on building stable, predictable software.

What are the benefits of waiting for open-source AI models?

Waiting a few months allows the open-source community to release models that match the raw intelligence of yesterday's proprietary giants. This short delay grants you total control over your data privacy, hosting costs, and infrastructure.

Instead of paying a premium to a closed-source provider, a short wait allows you to run an equivalent open-weights model on your own terms. Here is how the trade-offs look when you step off the daily AI treadmill:

Metric	Day 1 (Frontier)	Month 3 (Open Source)
Pricing	Premium API rates per token	Pennies via self-hosting or commodity providers
Stability	High risk of model drift, bugs, and downtime	Stable, frozen weights under your direct control
Autonomy	Tied to a single vendor's API and data policies	Freedom to host anywhere and protect user privacy

Is it really okay to be three months behind the AI curve?

Yes, staying three months behind the curve is completely fine because the performance gap between frontier models and open-source models has shrunk to almost nothing. In that tiny window, you gain pricing leverage and the freedom to avoid supporting organizations whose policies you dislike.

Frontier labs move fast, but they also bring a lot of baggage—including shifting usage terms, restrictive licensing, and high costs. If you have valid gripes with how these closed-source companies operate, you don't have to support them. By lagging just slightly behind the frontier, you can bypass them entirely. You get the same level of intelligence on your own hardware, free from vendor politics and restrictive commercial terms.

FAQ

Do I lose a competitive advantage by not using the newest AI models?

Rarely. Most end-users care about speed, reliability, and accuracy, not which specific model runs under the hood. A slightly older, highly optimized open-source model running with low latency often provides a far better user experience than a slow, bleeding-edge proprietary model.

How quickly do open-source models catch up to proprietary ones?

The gap has closed dramatically. Open-source models like Llama and Mistral regularly match the benchmark performance of proprietary frontier models within a three-to-six-month window, making long-term reliance on expensive closed APIs unnecessary.

Why is using a slightly older AI model more stable for production?

Older models have thoroughly documented edge cases, established community wrappers, and predictable performance. Because the weights are frozen and often run on your own infrastructure, you eliminate the risk of "model drift," where a remote vendor updates a model and suddenly breaks your application's prompts.

How AI Automates Fast Decisions in Software Engineering

Doogal Simpson — Mon, 24 Aug 2026 15:13:07 +0000

AI is shifting the software engineering landscape by taking over "fast decisions"—the automatic, high-context but low-complexity tasks like running tests and catching syntax errors. This frees up developers to focus entirely on "slow decisions" like system architecture, API boundaries, and product design.

I’ve been thinking a lot about the book Thinking, Fast and Slow lately. The core point of the book is that your brain operates in two distinct modes. When you're walking down the street, your brain is in a fast, automatic mode. You're navigating around other people on autopilot, making decisions without even realizing you're making them. But then you have slow decisions—like choosing which barbecue to buy for your garden. That takes deliberate, conscious thought.

When I think about how we interact with AI, this is exactly where I see things heading. AI is starting to pick up the fast, automatic decisions we make in our day-to-day engineering jobs, and that is going to fundamentally change how we spend our time.

How does Kahneman’s theory apply to writing code?

In software engineering, fast decisions are the low-friction, repetitive tasks we execute on autopilot, like running a test suite or fixing a syntax error. Slow decisions are the deliberate, strategic choices, like architecting a system or deciding on a test coverage strategy. AI is stepping in to run the fast decisions so we can focus entirely on the slow ones.

For example, when I am writing code, deciding whether to run the unit tests isn't really a decision I need to make. Of course I should run them. That is a fast decision. But deciding how those tests should be written, what scenarios we need to cover, and how the overall system should behave—those are slow decisions.

If we can offload the automatic "yes, run the tests" layer of our workflow, we reclaim the mental space needed for the hard, analytical design work that actually moves a project forward.

What are the "fast decisions" we can offload to AI?

Fast decisions are deterministic tasks that require local codebase context but very little creative reasoning. When I use a coding agent, I want it to handle things like running unit tests after a change, fixing formatting issues, or cleaning up import statements. These are things we can easily automate once an agent has enough context about our workspace.

I like to split these developer decisions into two distinct categories based on how much deep thought they require:

Decision Type	Developer Mode	Task Examples	AI Agent Feasibility
Fast Decisions	Autopilot (System 1)	Running unit tests, fixing syntax issues, formatting code, basic refactoring.	High (Can be fully automated)
Slow Decisions	Deliberate (System 2)	Designing system architecture, defining test coverage strategy, API design.	Low (Requires human reasoning)

How does delegating these decisions change our day-to-day jobs?

Delegating fast decisions to AI keeps us out of the constant loop of minor distractions, allowing us to stay in flow state longer. It means we stop wasting cognitive energy on routine tasks and spend it instead on deep architectural work.

Let's say you are building a payment processing service. In a traditional workflow, your day is fragmented by micro-decisions. You write some logic, manually trigger a test, fix a linter warning, check a configuration file, and then try to get back to your main train of thought. Every minor system step demands a small action from you.

If you hand those fast decisions to a coding agent, the agent runs the tests, formats the files, and flags obvious errors in the background. You don't have to decide to do those things; they just happen. This frees you up to think about the actual business logic, concurrency issues, and edge cases. It changes your job entirely.

FAQ

Can AI handle slow decisions in our codebases?

No, because slow decisions require deep context, trade-off analysis, and human-centric design. AI can offer suggestions, but deciding how a system should scale or what security model to use still requires human judgement and deep technical expertise.

Will this automation make junior developers obsolete?

I don't think so, but it will shift what we expect of them. Instead of spending their first year learning to write boilerplate and debug configuration files, they will need to learn system design and how to orchestrate AI tools much earlier in their careers.

How do I prepare my codebase for AI automation?

The best way is to make your fast decisions as deterministic as possible. If you have a clean project structure, fast test runs, and clear linting rules, it becomes much easier for an AI agent to step in and make those routine decisions for you.

Why Toxic Comments are a Systems Scaling Metric

Doogal Simpson — Sat, 22 Aug 2026 13:25:39 +0000

Receiving negative comments is an inevitable statistical side effect of scaling your reach. Based on the 90-9-1 engagement rule, an insulting comment is a 1-in-40,000 event that only triggers when your content or software expands past your core audience to fresh, unprimed users. It is a lagging indicator of growth.

Every software engineer understands the concept of scaling pain. When you run a service with a hundred users, edge-case bugs rarely trigger. But scale that service to millions of users, and those "one-in-a-million" hardware failures or race conditions start happening every single Tuesday.

Recently, a particularly toxic comment landed on one of my videos. While it is easy to take these things personally, looking at it through an engineering lens reveals a fascinating statistical truth: hostile feedback is not a personal failure, but a classic scaling problem.

What is the 90-9-1 rule in online community engagement?

The 90-9-1 rule dictates that 90% of users consume content silently, 9% actively curate it (likes/shares), and only 1% create content (comments). This power-law distribution governs almost all digital platforms and open-source repositories.

Imagine you are monitoring traffic on a microservice. Most requests are read-only (the 90% who view), a small fraction write metadata (the 9% who like or share), and a tiny sliver perform heavy write operations (the 1% who comment).

If your system only handles 1,000 views, you might only see 10 comments in total. At this volume, the probability of encountering extreme outliers is incredibly low. But as your traffic grows, that 1% "write" path begins to generate a statistically representative sample of the entire internet—including the outliers.

What is the probability of receiving a toxic comment in tech?

In technical niches, the probability of receiving an insulting comment is roughly 1-in-40,000 (or 0.0025%) per new visitor. This rate is determined by compounding the 1% comment rate with a 5% negative comment rate, and a further 5% insult rate within that negative subset.

Not all negative comments are created equal. In software, we generally enjoy a more constructive environment compared to highly polarized niches like politics. Here is how the nested probabilities break down when your reach expands:

Category	Funnel Percentage	Cumulative Probability
Total Viewers	100%	1.0000
Commenters	1% of viewers	0.0100
Negative Comments	5% of commenters	0.0005
Insulting/Toxic Comments	5% of negative comments	0.000025 (1 in 40,000)

When we filter the negative feedback further, we find a spectrum ranging from constructive criticism (e.g., "this sorting algorithm is suboptimal") to friendly banter (e.g., "found the Vim user") to flat-out insults. The purely toxic insults make up about 5% of that negative 5% slice.

Why do toxic comments indicate your system is scaling?

Toxic comments act as a lagging indicator of reach because they only occur when your distribution engine pushes content to "fresh" user segments. Your core, recurring audience has already warm-cached your value, whereas new cohorts represent unprimed traffic.

Think of your regular audience as a local cache. They know your style, understand your context, and have high cache hit rates for your content's intent. They are highly unlikely to leave hostile feedback because they have self-selected to be there.

But when a video or repository starts trending, the algorithm serves it to "cold" users who have no prior context. These are fresh database hits. As these fresh viewers flood in, the 1-in-40,000 probability event is bound to fire. If you get an insulting comment, it means your distribution pipeline is working well enough to reach outside your established network.

FAQ: Frequently Asked Questions about Online Engagement Stats

How should developers handle negative feedback on open-source projects?

Separate constructive criticism from noise. If a comment points out a performance bottleneck or security flaw, treat it as a bug report. If it is purely abusive, treat it as spam, moderate it immediately, and do not waste cognitive bandwidth on it.

Does the 90-9-1 rule apply to GitHub repository engagement?

Yes, the same power law applies. The vast majority of developers use a library silently (90%), a smaller portion star or fork it (9%), and a tiny minority open issues, submit pull requests, or participate in discussions (1%).

What is the difference between constructive criticism and toxic comments in tech?

Constructive criticism targets the code, architecture, or logic, offering a path toward improvement. Toxic comments target the person, utilizing insults or disparagement without offering any objective engineering value.

Why Scaling Software Is an Organizational Problem

Doogal Simpson — Fri, 21 Aug 2026 14:41:07 +0000

TL;DR: I've realized that scaling software development is fundamentally an organizational coordination problem, not a technical one. Just like multiple authors trying to write cohesive chapters of a single book without a shared plot or language, developers must constantly align on architectures and data contracts to prevent the system from fracturing.

I've spent enough time staring at fractured codebases to know the exact moment a project starts to unravel. It usually looks like a strictly typed Scala microservice trying to talk to a wild-west Node.js service over a fragile, undocumented HTTP interface. I look at this chaos and wonder: why is it so incredibly hard for us to build software that actually works together? Writing code in isolation is dead simple; coordinating hundreds of developers to write the same system is where the wheels fall off.

Why is scaling software development so difficult for large engineering teams?

From what I've observed, scaling software fails because it shifts the challenge from technical execution to a massive organizational coordination problem. As engineering teams grow, maintaining consistent architecture, data contracts, and design patterns across independent modules becomes exponentially harder.

I look at it through an analogy: imagine you are writing a book with another person. You split the work in half: you write the first half, and they write the second. When you compile the drafts, you discover they wrote their half in French, while you wrote yours in English.

Even if you both agree on English, you might find that you wrote a hard sci-fi novel while your partner wrote high fantasy. Now multiply this by tens or hundreds of developers. Everyone is writing different "chapters" of the same system. Without continuous, deliberate alignment, you end up with a fragmented product that fails to make sense end-to-end.

How do development teams maintain consistency across microservices and modules?

I recommend maintaining consistency by establishing strict, automated boundaries such as API contracts, shared schemas, and standardized local environments. These tools act as the shared editorial guidelines for our collaborative book, ensuring all contributors stay on the same page.

When I analyze scaling problems, I find it useful to map these collaborative challenges directly to their technical equivalents. Here is how I categorize these friction points and the technical solutions that actually solve them:

Collaborative Book Problem	Software Engineering Equivalent	Technical Solution
Writing in different languages	Fragmented tech stacks and incompatible frameworks	Standardized "golden paths" and approved language lists
Conflicting genres (Sci-Fi vs. Fantasy)	Mismatched architectural paradigms (OOP vs. Functional)	Linters, style guides, and shared RFC processes
Inconsistent character behaviors	Broken API contracts and unexpected payload changes	Schema registries (OpenAPI/Protobuf) and contract testing

What is the role of API contracts in team coordination?

I view API contracts as the shared plot points that ensure different authors' chapters connect seamlessly. By defining exactly how data enters and exits a service, teams can build their components independently without breaking the broader system.

Without these explicit contracts, you get the software equivalent of a character suddenly changing names or teleporting across the world between chapters. When one team modifies a database schema or changes an API payload format without notifying downstream teams, the entire system breaks.

I always advise enforcing API schemas and running consumer-driven contract tests in your CI/CD pipelines. This ensures that even if developers are writing their chapters in complete isolation, the final product remains cohesive.

FAQ

How do you prevent "too many cooks in the kitchen" in a large codebase?

I suggest defining clear domain boundaries using Domain-Driven Design (DDD) principles. Assigning clear, single-team ownership to specific modules or services reduces the coordination overhead and prevents conflicting changes to the same parts of the codebase.

Why do microservices often fail to solve coordination problems?

I've seen microservices fail because they split the codebase physically but do not inherently solve team alignment. Without robust API schemas, automated contract testing, and clear communication channels, microservices simply transform local codebase confusion into complex, distributed system failures.

What is a "golden path" in software engineering?

In my view, a golden path is an opinionated, pre-packaged set of templates, tools, and best practices provided to developers to build and deploy software. It acts as the formatting template for our collaborative book, making it easy for developers to do the right thing by default.

How Adaptive Bitrate Streaming Works

Doogal Simpson — Wed, 19 Aug 2026 14:47:11 +0000

Ever wonder why a streaming video suddenly drops to blurry 360p and then instantly snaps back to crisp 1080p? This happens because modern video platforms use Adaptive Bitrate Streaming (ABS). The video is sliced into 2-to-6-second chunks at multiple resolutions, and your player dynamically switches between them based on real-time network speeds.

I have noticed a funny thing when watching tech tutorials online. I will be staring at a screen, and suddenly the code turns into a pixelated mess. A few seconds later, without me touching a single setting or pausing, the image snaps back into crisp, high-definition focus.

It is easy to assume the creator recorded their screen poorly, but that is not what is happening. I want to talk about the clever web engineering behind this phenomenon: Adaptive Bitrate Streaming (ABS). It is the invisible system powering platforms like YouTube, Netflix, and Twitch.

How does adaptive bitrate streaming actually work?

When I look at how Adaptive Bitrate Streaming (ABS) functions, it boils down to slicing a video into multiple resolution tiers and chopping each tier into short, uniform segments. A master manifest file indexes these segments, allowing the player to dynamically request the optimal quality chunk based on active network speed.

If I upload a video to a modern streaming platform, a background processing pipeline immediately goes to work. First, the platform transcodes my original video file into several distinct resolutions—typically ranging from 360p up to 4K.

Then, the platform chops each of these resolution tracks into identical, bite-sized segments (usually between two and six seconds long). These chunks align perfectly. If I look at chunk #5 across all resolutions, they all represent the exact same two-second window of time. Finally, the system writes an index file—the manifest—so the playback client knows exactly where to find every single chunk.

Stage	Action	Purpose
Transcoding	Convert original file into multiple resolutions (e.g., 360p, 720p, 1080p, 4K)	Prepares alternative assets for varying network speeds
Chunking	Chop each resolution track into identical 2-6 second segments	Enables granular, mid-playback quality switching
Indexing	Create a manifest file mapping all chunks and resolutions	Gives the client-side player a roadmap of available options
Delivery	Playback engine requests chunks dynamically based on telemetry	Maintains uninterrupted streaming over fluctuating connections

Why do videos get pixelated instead of pausing to buffer?

Rather than forcing you to wait while a massive high-definition chunk buffers over a slow connection, the player will proactively grab the next 2-second chunk at a lower resolution. This strategy prioritizes uninterrupted, continuous playback over visual fidelity, keeping you in the flow of the video.

I like to visualize this process as a bucket brigade trying to keep a pool filled. The pool is your playback buffer, and the buckets are the video chunks.

If my network bandwidth suddenly drops, the brigade can no longer carry the heavy 4K buckets fast enough to keep the pool full. If I keep trying to haul them, the pool runs dry, and the video freezes (classic buffering). To avoid this, the player quickly swaps out the heavy buckets for lighter, low-resolution ones. They do not look as pretty, but they arrive fast enough to prevent playback from stalling. Once my network recovers, the player can start fetching the heavy 4K buckets again.

How does the client player decide which video chunk to download next?

The client-side player is constantly measuring the exact download time of the preceding chunks to calculate active network bandwidth. If a chunk takes longer to arrive than its actual play duration, the player's internal logic immediately downgrades the resolution request for the next chunk.

The video player is essentially running continuous telemetry on itself. When I hit play, the engine first fetches the manifest file. Because it does not have network telemetry yet, it usually plays it safe and starts by downloading a low- or mid-resolution chunk.

If that first chunk downloads in a fraction of a second, the player realizes, "I have plenty of bandwidth," and immediately requests the ultra-high-res version for the second chunk. If a family member starts downloading a massive game update on the same Wi-Fi network, the player registers the delay on the third chunk and swiftly requests a lower-quality chunk for the fourth. It is a continuous loop of feedback and adjustment.

Frequently Asked Questions

What is a manifest file in video streaming?

When I talk about a manifest file (like HLS's .m3u8 or DASH's .mpd), I mean a plain-text playlist that acts as a map. It indexes the URLs of all the individual video chunks, organized by resolution and bitrate, so the player knows exactly where to fetch them.

Why do videos often start in low quality before sharpening up?

I notice this a lot on slower connections: players often default to a low-resolution chunk for the very first segment to minimize start-up latency. The goal is to start playing the video instantly, and once the player measures the actual download speed, it scales up to the highest quality.

What is the difference between HLS and DASH streaming?

I view these as two sides of the same coin. Apple developed HLS (HTTP Live Streaming), which is native to iOS and Safari, whereas MPEG-DASH is an open international standard. Both accomplish adaptive streaming using the same chunk-swapping architecture but employ different file formats and manifest structures.