DEV Community: Ed

Your Terminal Is an AI Workspace Now

Ed — Tue, 17 Mar 2026 11:23:23 +0000

In 2025, something quietly changed about the terminal. It didn’t happen all at once. Claude Code arrived, then Cursor’s terminal mode, then AI agents running as background workers, streaming output continuously. Token-saving proxies started intercepting shell commands. Suddenly the terminal wasn’t just where you typed commands - it was where AI lived.

I’d been on iTerm2 since 2015. Back then, switching from Linux to macOS, it was the first thing every senior engineer told me to install. For a decade it was invisible infrastructure - just there, doing its job. I didn’t think about it.

AI made me think about it again. The lags. The memory. The realization that I couldn’t script it properly. And then, in a move that felt almost too on-brand, I had Claude Code do the migration for me.

This is that story - and what an AI-optimised terminal actually looks like.

Subscribe now

2015: When iTerm2 Was Still The Answer

[

](https://substackcdn.com/image/fetch/$s_!VHoO!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5b80be49-8580-4ebb-90c8-055743a22cfc_666x457.jpeg)

When I made the jump from Linux to macOS in 2015, the terminal was the first thing I felt homesick about. Linux gave you a raw, fast shell. macOS’s default Terminal.app felt like a toy.

iTerm2 fixed that. Split panes, saved profiles, a hotkey window that dropped down on a keystroke, shell integration that tracked your working directory across tabs. For a developer coming from Linux, it was a genuine power tool. I set it up once and stopped thinking about it. That’s the highest praise you can give infrastructure.

It worked invisibly for nearly a decade. Then three things started going wrong.

The terminal started lagging. Long output - build logs, test runs, streaming responses - would cause the window to stutter. Scrolling froze for a beat. The cursor fell behind what I was typing. This wasn’t dramatic, just irritating - until I started running Claude Code sessions. AI generates output fast, and iTerm2’s CPU-based renderer couldn’t keep up. My laptop fans would spin up during what was supposed to be a quiet coding session.

My machine got sluggish. Nothing I could point to directly, just swap pressure, slower context switches, the general feeling of running out of headroom. When I looked more carefully, iTerm2 was quietly eating 300-400MB across my open panes - on top of Cursor, Claude Code, Docker, and a local Redis already competing for memory.

I couldn’t script it. I wanted to automate my window layout, send commands programmatically from scripts, have AI tools interact with the terminal as an API rather than just a UI. iTerm2 has AppleScript. That’s its answer to programmability. No socket, no remote protocol, no way to open a tab from a shell script without writing brittle AppleScript glue.

None of these were dealbreakers on their own. Together, they were death by a thousand cuts - and all three had the same root cause: AI had moved into my terminal, and iTerm2 wasn’t built for a roommate.

It’s not that iTerm2 got worse. It’s that the job description changed. In 2015, a terminal needed to display text reliably, manage tabs and splits, and stay out of the way. In 2026, it also needs to handle continuous high-volume output without stuttering, run lean alongside a full AI toolchain, and expose enough programmability for AI tools to treat it as a first-class interface. iTerm2 was built for the first set of requirements. Kitty was built closer to the second.

The Migration That Migrated Itself

Here is the prompt I gave Claude Code: “Install Kitty and set it up the same way I have iTerm2 set up. Make it default for all editors and the system.”

That’s it. What happened next took under 10 minutes of AI execution time.

Claude Code read my iTerm2 preferences using defaults read - the macOS command that dumps application settings from their binary plist format. From that output it extracted my font (Monaco 12), my color scheme (a custom light theme), window dimensions, and every keybinding I had configured.

It installed Kitty via Homebrew and wrote a kitty.conf from scratch - ported colors to Kitty’s plain text format, mapped my split-pane shortcuts (Cmd+D for vertical, Cmd+Shift+D for horizontal), carried over the visual bell, inactive pane dimming, the lot. Then it opened my VS Code and Cursor settings files, found terminal.external.osxExec: "iTerm.app" in both, and changed them to "kitty.app". Finally it installed duti and registered Kitty as the default handler for shell file types using its bundle ID (net.kovidgoyal.kitty).

A human doing this manually - reading iTerm2 documentation, decoding a binary plist, translating color values between two different config formats, hunting for the right duti incantation - would spend an hour. Easily. I know because I’ve done similar migrations before.

Config migration is exactly the kind of task AI handles better than humans: tedious, well-documented, error-prone, with no room for creativity. AI doesn’t get bored halfway through and start approximating. It does all ten steps the same way.

The irony wasn’t lost on me. I was using AI to build a terminal optimised for AI. The ouroboros of developer tooling.

Thanks for reading Olko - Tech/Engineering! Subscribe for free to receive new posts and support my work.

What an AI-Optimised Terminal Looks Like in 2026

The full config is here. What follows is the philosophy behind it.

The emulator: Kitty

Kitty is GPU-accelerated. This matters more than it sounds. When Claude Code streams a long response, or a BullMQ worker dumps a thousand lines of agent output, the renderer doesn’t choke. CPU usage stays flat. The fans stay quiet. That was the first thing I noticed after switching.

The second thing: Kitty has a remote control protocol. With allow_remote_control yes and a unix socket, other processes can send commands to Kitty - open a tab, split a pane, send keystrokes - without any GUI automation. This is the terminal as an API, not just a UI. AI tools can drive it programmatically.

The third thing, which turns out to matter most: the config is a plain text file. ~/.config/kitty/kitty.conf. One file. Human-readable. Version-controllable. And - this is the key - fully legible to AI tools. iTerm2’s configuration lives in a binary plist that’s opaque to everything except iTerm2 itself. Kitty’s config is just text with clear semantics. That’s what made the migration possible in the first place.

The AI layer

On top of Kitty, I run RTK - a personal tool that acts as a transparent proxy between Claude Code and the shell, compressing command output before it hits the token counter. It saves 60-90% on tokens for routine dev operations, wired in via Claude Code hooks so it’s invisible in normal use.

Claude Code runs with shell_integration enabled, which gives it awareness of the terminal state - working directory, running processes, exit codes. Cursor and VS Code both point to Kitty as their external terminal. The BullMQ agent workers that run my AI simulation backend stream their output through Kitty’s GPU renderer.

The most useful consequence of a plain text config: the terminal is now AI-maintained. Need a new keybinding? I describe it to Claude Code. Want to port a color scheme from another tool? It reads the source format and writes the Kitty equivalent. The config is a living document that gets better over time without me manually editing it.

The shell layer

My zsh setup - Powerlevel10k, fzf, Atuin for shell history, zsh-autosuggestions - moved from iTerm2 to Kitty unchanged. This layer is terminal-agnostic. It doesn’t care what’s rendering it.

That’s the point of the whole stack: Kitty renders, zsh orchestrates, AI tools inhabit. Each layer is independently replaceable.

The Tool That Disappears

The best infrastructure is invisible. You don’t think about your database when you’re writing a query; you think about the data. You don’t think about your terminal when you’re working; you think about the code.

iTerm2 was invisible for ten years. Then the work changed, and it became visible - in the lag, the memory pressure, the AppleScript dead ends. Kitty is invisible again.

That said: Kitty is not a drop-in replacement for everyone. There’s no GUI preferences pane - you edit a text file. The macOS integration is less polished; some things that just work in iTerm2 require config tweaks in Kitty. If you rely heavily on tmux integration, iTerm2’s implementation is more mature. And if you’re not running AI agents, worker processes, or heavy streaming output in your terminal, you probably won’t feel the difference. iTerm2 is still excellent software.

But if your terminal has become an AI workspace - and for a growing number of engineers it has, or soon will - it’s worth thinking about whether your emulator was built for that world.

Your terminal is the last tool that should slow down your AI.

Share Olko - Tech/Engineering

Your Prompts Don't Have Tests. That's a Problem.

Ed — Fri, 27 Feb 2026 11:34:28 +0000

Nobody knows when a prompt got worse.

I mean that literally. A teammate tweaks a system prompt on Tuesday - removes a constraint, adds a sentence. By Thursday the AI assistant is giving shorter, vaguer answers. Support tickets go up. On Monday someone finally traces it back to that one edit.

No diff. No test. No review. The prompt lived in a Google Doc, and someone changed it the way you’d change a wiki page.

I keep hearing the same story from different teams, and it always plays out the same way.

Thanks for reading Olko - Tech/Engineering! Subscribe for free to receive new posts and support my work.

Prompts drift. Nobody notices.

Once LLM systems move from prototype to production, a specific failure mode kicks in. Prompts drift - not dramatically, but incrementally. Someone adjusts wording. Someone switches the model. Someone adds a paragraph of context. Each change seems fine on its own. Nobody measures the cumulative effect.

[

](https://substackcdn.com/image/fetch/$s_!8tSM!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe7117c66-13dc-417d-b866-198a394bab9b_1456x816.png)

This is the LLM equivalent of configuration drift. The prompt that worked last month might not work today, and without measurement, you won’t know until a user complains or your API bill spikes with no explanation.

The deeper issue: prompt quality is treated as vibes. An engineer writes a prompt, eyeballs the output, decides it’s “good enough,” and ships. No baseline. No regression check. No way to compare version A against version B.

For any other production component, we’d call that reckless.

How this started

I kept writing the same prompt structure by hand - persona, task decomposition, constraints, output format - and I kept doing it badly. Inconsistently. Forgetting sections. So I built a CLI that takes a rough intent and produces a structured prompt. No LLM call - the engine is rule-based and deterministic. It classifies your intent into one of eleven task types and applies the right structure.

Here’s what that looks like:

$ promptctl create "review auth middleware for security issues"
# Generates a structured prompt with:
# - Security engineer persona
# - OWASP-aligned review checklist
# - Specific output format (findings table, severity, remediation)

$ promptctl cost --compare
# Shows token cost across 10 models before you spend anything
# Claude Sonnet 4.5: ~$0.018
# GPT-5: ~$0.022
# DeepSeek V3: ~$0.003
# Llama 4 Maverick: ~$0.001

Useful, but the real problem showed up when I looked at cost patterns.

Unstructured prompts waste money in ways that aren’t obvious. The model rambles because nothing constrains it. You send follow-ups because the first response missed the point. Each retry costs tokens. Structured prompts consistently cost 55-71% less across ten models - not because the model gets cheaper, but because you need fewer calls and get tighter output.

What surprised me: showing engineers the cost before they hit send changed their behavior more than any other feature. People make different choices when the price tag is visible.

Treating prompts like code

The interesting shift happened when I started thinking about prompts the way I think about any other artifact that changes over time.

Here’s the pipeline:

[

](https://substackcdn.com/image/fetch/$s_!0NDK!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffe0eac20-5aaf-4dc6-aa13-8c7eeb97d9b1_5589x780.png)

Versioning. A prompt template is a YAML file with variables and metadata. When you change the body, you bump the version. Old versions stay. You can diff them, you can roll back. We do this with code. Almost nobody does it with prompts.

Quality scoring. The --score flag evaluates prompt structure on a 0-100 scale: does the prompt match the intent, does it have clear sections and constraints, are there common mistakes like duplicate instructions or vague asks. It’s not a replacement for judgment - but it catches obvious failures before you burn tokens.

Baseline comparison. This is where it starts to feel like actual engineering. Concrete example:

You have a code review template. Version 1 scores 82. You tweak the constraints and persona - that’s version 2. Before shipping v2, you evaluate it against the v1 baseline:

$ promptctl score --template=code-review --baseline=v1

Template: code-review
Current (v2): 78/100
Baseline (v1): 82/100
Delta: -4

⚠ Regression detected. v2 scores lower than baseline.
  - Structural quality dropped (missing constraint section)
  - Fidelity unchanged

$ promptctl record --template=code-review
# Saves v2 score to benchmark history
# Next comparison will use v2 as the new baseline

[

](https://substackcdn.com/image/fetch/$s_!oVfc!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff0d0bd8d-7f07-48c7-84e5-3fb3b4be1d11_1485x3433.png)

Wire that into CI and prompt changes get the same regression gate as code changes. A PR that degrades prompt quality fails the pipeline, same as a PR that breaks a test.

The tradeoffs are real. Versioning adds overhead - you have to decide what counts as a “new version” versus a minor edit. Deterministic evaluation at temperature=0 gives reproducible results but won’t tell you how the prompt behaves at higher temperatures or across different models. CI gating creates friction - engineers push back on quality gates for something they consider subjective. My counter: if this prompt runs in production and affects user-facing output, it should meet a minimum bar. And that bar should be measurable.

What I actually learned building this

Determinism beats cleverness. Early versions tried multi-model scoring with fancy aggregation. Output was non-reproducible. I ripped it out. Simple, deterministic evaluation on one model. Less impressive on a slide deck, actually useful for catching regressions.

The generation engine has to be deep. First version produced skeleton prompts - a heading for “Expert role,” a heading for “Constraints,” some placeholder text. Useless. Anyone could write that in thirty seconds. I rewrote the engine with domain knowledge: when someone says “build a virtual football manager,” it needs to surface match simulation mechanics, player attribute models, transfer market economics, engagement loops. Not “Subject: virtual football manager.” The prompt has to know what the user needs but hasn’t thought to ask for yet.

Pipe-anywhere was the right call. Everything goes to stdout. Pipe to Claude CLI, OpenAI CLI, clipboard, file. Cost tracking and scores go to stderr so they don’t pollute the output stream. Basic Unix philosophy, but a lot of developer tools get this wrong.

Feature creep kills CLI tools. I kept wanting to add dashboards, team management, analytics. None of it belonged in a CLI. The rule I settled on: if it doesn’t make the next prompt better or cheaper, it doesn’t ship.

Where this is going

The current version handles prompt creation, cost comparison, and quality scoring. The next step is persistent benchmarks - tracking how prompt versions perform over time, not just at the moment of creation.

But there’s a bigger shift happening that matters more than any feature.

Prompt engineering is becoming operational. Two years ago prompts were a novelty. Today they’re load-bearing. A chatbot resolves support tickets. An assistant generates code that ships to production. A summarizer produces reports that executives act on. The prompt is often the single biggest factor in output quality, and it’s the least tested part of the stack.

We’ve seen this pattern before. Early DevOps was deployment scripts in a shared folder. They worked - until they didn’t. The shift from “deployment scripts” to “deployment pipelines” wasn’t a tooling change. It was teams recognizing that something critical needed discipline, not ad-hoc management.

Prompt workflows are at that same point. The work happening now is useful but fragile. No regression protection. No quality history. No review process for the thing that most directly controls what your LLM produces.

That will change. Not because prompt engineering is exciting or trendy, but because the cost of not doing it keeps going up. Every production LLM system that ships without prompt quality tracking is accumulating risk the same way teams accumulated risk shipping without tests in 2008.

The question isn’t whether your prompts need regression protection. It’s how expensive the next silent failure has to be before you add it.

This is the first in a series on applying engineering discipline to LLM workflows. Next up: what happens when you actually run prompt regression testing on a real codebase.

[

](https://substackcdn.com/image/fetch/$s_!P-Yz!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe461f7fc-6c25-4db5-b8e6-5cc0ad9b1e14_1200x628.png)

promptctl is a free CLI. Install via brew

brew install --cask oleg-koval/tap/promptctl

or grab it from prompt-ctl.com.

I built a Cursor Skill that ships code

Ed — Sat, 24 Jan 2026 08:13:01 +0000

The Problem Every AI-Assisted Developer Knows

Last month I asked Cursor to build a dashboard. Simple request: “Build me an analytics dashboard with auth.”

[

](https://substackcdn.com/image/fetch/$s_!VJpD!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8c55afa8-3783-4ec3-af88-2bec92eb0360_4896x3264.jpeg)

What I got:

“What framework would you prefer?”
“Should I include dark mode?”
“What authentication provider?”
“Do you want SSO support?”

Four questions before a single line of code. By the time I finished the interrogation, I could have scaffolded the project myself.

This isn’t a Cursor problem. It’s an LLM problem. Models are trained to be helpful, and “helpful” often means clarifying requirements before committing to implementation. In a chatbot context, that’s reasonable. In a code editor where you’re paying per request and context window is precious, it’s friction.

I spent a week building a Cursor skill that flips this behavior. The result: describe what you want, get working code. No clarifying questions unless genuinely ambiguous.

What Cursor 2.4 Actually Enables

Cursor’s January 2025 release introduced two features that make this possible:

Subagents : Independent agents that handle discrete subtasks in parallel. Each gets its own context window, custom prompts, and tool access. The main agent delegates to specialists instead of trying to do everything in one thread.

Image generation : Generate mockups and assets directly from prompts. Save to your project’s assets folder. Useful for visualizing before building.

The subagent architecture is the key innovation. Instead of one overloaded context trying to be a UI expert, database architect, and test writer simultaneously, you get specialists that excel at their domain.

Thanks for reading Olko - Tech/Engineering! Subscribe for free to receive new posts and support my work.

The Skill Architecture

The skill lives in .cursor/rules/ and .cursor/agents/:

.cursor/
├── rules/
│ └── imagine-builder.mdc # Core behavior rules
└── agents/
    ├── ui-designer.md # Tailwind, shadcn, animations
    ├── schema-architect.md # Prisma, indexes, relations
    ├── api-builder.md # Routes, validation, auth
    └── test-writer.md # Vitest, RTL, Playwright

The main rule file (imagine-builder.mdc) establishes the core principle: build first, ask later.

---
description: AI Product Builder - Build functional products through conversation
globs: "**/*"
alwaysApply: true
---

# AI Product Builder

## Core Principles

1. **Build real products, not prototypes** - Every output should be deployable
2. **Infer intent aggressively** - Don't ask unnecessary clarifying questions
3. **Ship fast** - Prioritize working code over perfect code

The “infer intent aggressively” directive is doing the heavy lifting. It tells the model to make reasonable assumptions based on context rather than seeking explicit confirmation for every decision.

Why Subagents Matter

Consider a request like “Build a SaaS dashboard with org management and billing.”

Without subagents , the main agent tries to:

Design the database schema
Create API routes
Build React components
Wire up auth
Add Stripe integration

By step 3, the context window is polluted with schema definitions the model no longer needs to reference. Quality degrades.

With subagents , the main agent:

Plans the architecture
Delegates schema design to @schema-architect
Delegates API routes to @api-builder (in parallel)
Delegates UI components to @ui-designer (in parallel)
Aggregates results

Each specialist operates with clean context focused on its domain. The schema architect isn’t thinking about button hover states. The UI designer isn’t worrying about database indexes.

Here’s what the schema architect subagent looks like:

# Schema Architect Subagent

You are a specialized database architect focused on designing 
scalable, efficient data models.

## Schema Design Principles

### 1. Always Include Audit Fields
model BaseModel {
  id String @id @default(cuid())
  createdAt DateTime @default(now())
  updatedAt DateTime @updatedAt
}

### 2. Use Proper ID Strategies
- cuid() for distributed systems (default)
- uuid() for high-security requirements
- autoincrement() only for analytics tables

### 3. Soft Deletes for User Data
model User {
  deletedAt DateTime?
  isActive Boolean @default(true)
}

The subagent includes complete code patterns, not abstract principles. When it generates a schema, it follows these patterns automatically.

The Default Stack Decision

One of the hardest choices: what tech stack to default to when the user doesn’t specify?

I went with:

Layer Choice Rationale Framework Next.js 14 (App Router) Server components, great DX, wide adoption Language TypeScript (strict) Type safety catches bugs early Styling Tailwind + shadcn/ui Fast iteration, good defaults Database Prisma + PostgreSQL Type-safe queries, easy migrations Auth NextAuth.js Flexible, well-documented Validation Zod Runtime + compile-time safety

These aren’t universally “best” choices. They’re sensible defaults that work for most projects. The skill explicitly says: override these if you specify otherwise.

## Tech Stack Defaults

Unless user specifies otherwise:

- **Frontend** : React 18+ with TypeScript, Tailwind CSS, shadcn/ui
- **Backend** : Next.js App Router with Server Actions
- **Database** : Prisma with SQLite (dev) / PostgreSQL (prod)

The “(dev) / PostgreSQL (prod)” detail matters. SQLite for local development means zero setup. The schema is portable to Postgres when you deploy.

What the Output Actually Looks Like

Request: “Build an expense tracking app with receipt uploads”

Output (abbreviated):

// prisma/schema.prisma
model Expense {
  id String @id @default(cuid())
  amount Decimal @db.Decimal(10, 2)
  description String
  category Category @relation(fields: [categoryId], references: [id])
  categoryId String
  receipt String? // S3 URL
  userId String
  user User @relation(fields: [userId], references: [id])
  createdAt DateTime @default(now())

  @@index([userId, createdAt(sort: Desc)])
}

// app/api/expenses/route.ts
export async function POST(request: NextRequest) {
  const session = await auth();
  if (!session?.user) {
    return NextResponse.json(
      { success: false, error: { code: 'UNAUTHORIZED' }},
      { status: 401 }
    );
  }

  const formData = await request.formData();
  const result = createExpenseSchema.safeParse({
    amount: formData.get('amount'),
    description: formData.get('description'),
    categoryId: formData.get('categoryId'),
  });

  if (!result.success) {
    return NextResponse.json(
      { success: false, error: { code: 'VALIDATION_ERROR', details: result.error.flatten().fieldErrors }},
      { status: 400 }
    );
  }

  // Handle receipt upload...
  // Create expense...
}

Notice what’s not there: no comments explaining what @id does, no placeholder // TODO: implement auth, no skeleton code. It’s complete, runnable, follows the established patterns.

The Trade-offs

This approach works when:

You want a working prototype fast
The requirements are reasonably common (CRUD apps, dashboards, landing pages)
You’re comfortable reviewing and adjusting generated code
You trust the default stack choices

This approach fails when:

Requirements are genuinely ambiguous (”build something cool”)
You need an unusual stack (Remix, Hono, Drizzle)
The project has existing patterns the skill doesn’t know about
You want educational explanations alongside code

The skill doesn’t replace understanding. It replaces boilerplate. If you don’t understand the code it generates, you’ll struggle to debug it.

Making It Your Own

The skill is MIT licensed and designed to be forked. Common customizations:

Change the stack:

## Tech Stack Defaults
- **Frontend** : Vue 3 + Nuxt
- **Database** : Drizzle ORM + SQLite

Add domain-specific subagents:

# E-commerce Specialist Subagent

You are an expert in building e-commerce systems.

## Patterns
- Always use optimistic UI for cart operations
- Implement idempotency keys for payment endpoints
- Use database transactions for inventory decrements

Adjust the aggression level:

If you want some clarification, add conditions:

## Response Protocol

Ask for clarification ONLY when:
- Request involves external services not mentioned (Stripe, AWS, etc.)
- Security implications are unclear (public vs. private data)
- Multiple valid interpretations exist with different costs

The Broader Point

AI coding assistants are converging on a question: should the model be a collaborator (asks questions, explains trade-offs, teaches) or an executor (takes requirements, ships code)?

The answer is probably “both, depending on context.” But right now, most defaults lean heavily toward collaborator. That’s appropriate for learning, frustrating for shipping.

Cursor’s subagent architecture lets you build specialized executors that know your patterns, your stack, your preferences. The main agent collaborates at the architecture level; the subagents execute at the implementation level.

This is where AI coding tools are heading: not one general-purpose assistant, but orchestrated specialists that handle different phases of the development workflow.

Try It

The skill is on GitHub: cursor-product-builder

Clone it, copy .cursor/ to your project, and ask it to build something. Start with something concrete: “Build a todo app with categories and due dates” rather than “build me something.”

If you improve it, PRs welcome. Particularly interested in:

Alternative stack presets (Vue, Svelte, HTMX)
Domain-specific subagents (e-commerce, fintech, devtools)
Better test generation patterns

This is part of my series on distributed systems and developer tooling. Subscribe for more deep dives on the infrastructure that actually runs production systems.

Subscribe now

Question for readers : What’s your experience with AI code assistants? Too many questions? Not enough? Have you built custom rules or prompts that changed the behavior significantly? Drop a comment - curious what patterns others have found.

33 Million Accounts Exposed: What the Condé Nast Breach Teaches Engineering Leaders

Ed — Sun, 28 Dec 2025 10:09:34 +0000

Here’s what happened, what went wrong, and the concrete steps you should implement Monday morning.

The Breach in Brief

An attacker exploiting multiple vulnerabilities in Condé Nast’s systems exfiltrated data on 33 million user accounts across their publication portfolio - including WIRED, Vogue, The New Yorker, and others. The compromised data included email addresses, names, phone numbers, physical addresses, gender, and usernames.

The attacker initially posed as a security researcher seeking responsible disclosure. When Condé Nast failed to respond for weeks, 2.3 million WIRED records ended up leaked publicly and indexed by Have I Been Pwned.

As of this writing, Condé Nast has issued no public statement.

[

](https://images.unsplash.com/photo-1765410845769-9c931a7728b7?fm=jpg&q=60&w=3000&ixlib=rb-4.1.0&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D)

Thanks for reading Olko - Tech/Engineering! Subscribe for free to receive new posts and support my work.

Five Systemic Failures

1. No Vulnerability Disclosure Infrastructure

Condé Nast—a multi-billion dollar media conglomerate—had no security.txt file. No clear process for reporting vulnerabilities. The attacker spent days trying to find someone to contact.

This is inexcusable for any organization handling user data, let alone 33 million accounts.

2. Zero Response to Disclosure Attempts

Multiple contact attempts via email and through WIRED staff went unanswered for weeks. The security team only engaged after a third-party blogger intervened repeatedly.

This silence transformed a potential controlled disclosure into a public breach.

3. API Authorization Failures at Scale

The vulnerabilities reportedly allowed attackers to view any account’s information and change any account’s email and password. This pattern—IDOR (Insecure Direct Object Reference) combined with broken access controls—suggests fundamental failures in API security architecture.

When an attacker can enumerate 33 million records, you don’t have a vulnerability. You have an architectural deficiency.

4. No Rate Limiting or Anomaly Detection

Downloading 33 million user records takes time and generates traffic. Either no monitoring existed, or alerts were ignored. Both scenarios indicate operational blind spots.

5. Post-Breach Silence

Even after data appeared on breach forums and HIBP, Condé Nast issued no public acknowledgment. Users whose data was exposed learned about it from security bloggers, not the company entrusted with their information.

Prevention Checklist for Engineering Leaders

Disclosure Infrastructure (Implement This Week)

Deploy a security.txt file at /.well-known/security.txt with contact email, PGP key, and expected response timeframe
Establish a dedicated security@ alias routed to a monitored, triaged queue—not a black hole
Define SLAs: acknowledge within 24 hours, triage within 72 hours, remediation timeline within 7 days
Consider a vulnerability disclosure program (VDP) or bug bounty—even a modest one signals maturity

API Security Architecture (Q1 Priority)

Audit all endpoints for authorization checks: never rely on obscurity of IDs
Implement rate limiting per endpoint, per user, and per IP—with graduated responses
Enforce object-level authorization: every request must validate the authenticated user has permission to access the specific resource
Deploy anomaly detection on bulk data access patterns: 33 million sequential reads should trigger alerts within minutes

Incident Response Readiness

Document and drill an incident response playbook quarterly
Pre-draft breach notification templates for regulators and affected users—you won’t have time during a crisis
Establish a cross-functional incident team: engineering, legal, communications, and executive sponsor
Define escalation triggers and communication protocols before you need them

Monitoring and Detection

Log all authentication events, password changes, and bulk data access
Alert on mass enumeration patterns: sequential ID access, unusual query volumes, scraping signatures
Implement honeypot records in your database that trigger alerts when accessed
Conduct purple team exercises: have your own team attempt exfiltration and measure detection time

The Organizational Dimension

Technical controls matter, but this breach also exposed cultural failures.

When disclosure attempts go unanswered for weeks, it signals that security is someone else’s problem—or no one’s. Lead engineers must ensure that vulnerability reports reach people empowered to act, not bureaucratic dead ends.

When breaches happen (and they will), the first hour matters. Having legal and communications aligned in advance isn’t optional. The absence of any public statement from Condé Nast isn’t prudent caution—it’s reputational damage compounding daily.

What This Means for Your Organization

The Condé Nast breach wasn’t caused by zero-days or nation-state actors. It was caused by missing basics: no disclosure process, unmonitored APIs, and organizational silence.

If you’re a lead engineer or responsible person, ask yourself:

Can a security researcher contact us easily right now?
Would we know if someone was enumerating our user database?
Do we have a communication plan ready for breach disclosure?

If the answer to any of these is “no” or “I’m not sure,” you have work to do.

The attackers aren’t getting less sophisticated. But in this case, they didn’t need to be.

What’s your organization’s disclosure process? I’m curious how other engineering teams handle vulnerability reports—especially at scale. Drop a comment or reply.

Leave a comment

When Someone Dismisses Your Work. Why You Should Keep Building Anyway

Ed — Sun, 07 Dec 2025 11:32:18 +0000

Recently I came across a post that summarized a mindset many engineers know too well.

The author argued that:

the AI bubble is about to collapse
“real engineers” will return to low-level tools like Zig
modern frameworks and simple app domains (especially todo apps) are “just vibe coding”
most developers today are not serious enough
only hardcore, old-school engineering counts

No names needed. You’ve seen this type of sentiment before.

It is a mix of nostalgia, irritation at hype cycles, and a belief that anything simple is automatically trivial.

If you are building something today, this kind of message can hit unexpectedly hard.

Especially if your current project looks “simple from the outside”.

It can make you feel like your work is not substantial.

Like your domain is not respectable.

Like you are wasting time.

That reaction is human.

But it does not reflect reality.

Here is the reality.

Thanks for reading Olko - Tech/Engineering! Subscribe for free to receive new posts and support my work.

1. A simple domain does not make you a weak engineer

Todo projects look basic to people who never built one properly.

The moment you add:

authentication
sync logic
permissions
offline support
clean architecture
performance constraints
mobile/web parity
design decisions for real users

…the “simplicity” disappears.

Trello was “just a board”.

Linear was “just issues”.

Notion was “just an editor”.

Everything looks trivial until you build it well.

2. Hype or anti-hype does not define your value

Some people react to AI hype with frustration.

Some defend older ecosystems like Zig, Rust, C, Vim.

Some prefer JS, TS, Python.

Some love LLM-assisted workflows.

These are perspectives, not universal truths.

Your right to build is not dependent on:

someone else’s taste
someone else’s nostalgia
someone else’s frustration with the industry
someone else’s idea of “real engineering”

The only thing that defines the seriousness of your work is how you execute it.

3. Every project is a training ground for real skills

Even the simplest domain forces you to learn:

systems thinking
clean abstractions
readable code
data modelling
handling real-world edge cases
making trade-offs
designing for maintainability
iterating and shipping

That is engineering.

Not ideology. Not hype.

Practice.

4. Dismissive takes are noise. Your progress is signal.

People online can say whatever they want.

They do not see:

your architecture
your reasoning
your growth
your consistency
your ability to refine, fix, and finish

They comment.

You build.

Those are different worlds.

5. Keep building, even if someone belittles your path

Every engineer hits moments where a throwaway opinion cuts into motivation.

Where someone’s “hot take” makes your work feel small.

Where you suddenly doubt the value of what you are doing.

The answer is simple.

Keep going.

What you build, what you ship, what you learn -that is what counts.

Not commentary. Not condescension. Not hierarchy of “real” and “not real” engineering.

The industry moves forward because of people who show up and create.

And if you are one of them, your work matters.

The Reality Behind All the Noise

There is a recurring character in tech who loves announcing what counts as “real engineering”.

They deliver manifestos instead of products, write diagnoses instead of code, and somehow always know exactly what everyone else should be building.

It would almost be impressive if it wasn’t so predictable.

This type doesn’t talk about your work.

They talk about any work they’re not doing themselves.

Their formula is simple:

When they slow down, they preach.

When they stop shipping, they set standards.

When they feel irrelevant, they declare which domains “matter”.

It is the safest place to be — above the builders, below the responsibility.

That mindset doesn’t deserve outrage.

It deserves indifference.

Because the only thing that shuts down this performance is progress they are not part of.

Nothing is colder than quietly moving forward while others debate definitions of “real engineering”.

Let them keep writing their think-pieces.

Let them keep defining seriousness from the sidelines.

People who build don’t need permission.

And they don’t seek validation from those who stopped.

Thanks for reading Olko - Tech/Engineering! This post is public so feel free to share it.

___

Leave a comment

From Idea to Impact: Applying the "RADIO-AI" Pattern for Effortless AI Agent MVPs

Ed — Wed, 26 Nov 2025 12:27:10 +0000

Why Most AI Projects Stall - and How to Fix It

We all know the feeling: a flash of inspiration for the “next big thing” (a killer recommendation engine, an email assistant) - but too often, execution gets lost in the weeds.

Thanks for reading Olko - Tech/Engineering! Subscribe for free to receive new posts and support my work.

The real secret? Structure. The difference between endless tinkering and delivering something amazing is a system you can rely on, especially as you iterate.

Today, I’ll share the “RADIO-AI” pattern- a system design framework distilled for rapid, scalable, maintainable AI integrations. Inspired by modern social app architectures and proven engineering practice, it’s your shortcut to building MVPs that actually ship.

The RADIO-AI Pattern: Five Steps to Realising AI Goals

[

](https://images.unsplash.com/photo-1670234192944-5d5d06b762be?fm=jpg&q=60&w=3000&ixlib=rb-4.1.0&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D)

1. Requirements: Focus on What Matters

Define what your AI agent should do.

Don’t boil the ocean. Start with one pain-point: “Auto-tag my photos,” “Summarize my chats,” or “Offer smarter search results.”

Functional : What is the one user story?
Non-Functional : How fast should it respond? Does it need to be explainable?

2. Architecture: Strong Foundations, Fast Iterations

Lay out where AI fits in your flow:

Frontend: Does the agent spark in the UI, or work quietly behind-the-scenes?
Layering: Keep code modular (think: plug-and-play). One place for core logic; one for presentation.
Evolve without fear: Use feature flags to test and swap models painlessly.

3. Data Model: Speak the Agent’s Language

Good contracts = less tech debt, more speed.

Inputs/Outputs : Sketch what your agent receives/returns (draw a simple JSON or TypeScript interface).
Feedback Loop : Log every decision- future-you (or your users) can help refine the magic.

4. Interface: Make It Plug-and-Play

Expose clear, simple APIs:

REST, GraphQL, or event-driven - whatever matches your stack.
Power users can hook in, but even your UI “just works.”

5. Optimization: Iterate Like the Pros

Cache clever answers for speed.
Bake in user feedback for smarter AI.
Use toggles: Roll out changes to a few, observe, tweak, repeat.

🎯 How Simple Can It Be?

Imagine you want to add an AI helper to your photography site:

Requirement : “Auto-tag photos as ‘street’, ‘portrait’, or ‘landscape’.”
Architecture : A lightweight serverless API triggers every upload.
Data Model :

// Simple input/output contract
type PhotoInput = { url: string };
type TagsOutput = { tags: string[] };

Interface :
Optimisation :

Repeat for any feature. It’s that modular.

Kickstart Your MVP - Today

You don’t need a team of AI PhDs or a 100-page system doc.

You need intentional structure and the courage to start small (but right).

Next Actions:

Identify one task your audience would love automated.
Sketch the “RADIO-AI” steps on paper or Notion.
Build your first agent behind a feature flag - even a mock model works.
Share progress. Iterate. Involve your audience.

You’re not “late” to AI. You’re exactly on time.

Don’t overcomplicate - build, ship, and learn.

Let’s get started.

Have questions? Need more concrete code? Drop a comment; let’s turn ideas into energy.

Leave a comment

How Distributed Systems Really Talk: REST, gRPC, GraphQL, and tRPC in Practice

Ed — Sun, 23 Nov 2025 11:56:59 +0000

REST, GraphQL, gRPC, and tRPC are not just protocol choices. They are political choices between teams. Here is when each one works best, and when it will fight you.

When you scale from a handful of backend services to a network of teams shipping independently, you discover something quickly: APIs are politics. Distributed systems are rarely brought down by storage engines or consensus protocols. More often, teams trip over something far more mundane: how services talk to each other.

Every endpoint, response code, and contract revision is a small treaty between teams.

Over the past two decades, we’ve accumulated a surprising number of “languages” for machines to negotiate with. Each emerged for a reason, solved a pain, and introduced new ones. Below is a practical, engineer-first view of how these protocols evolved, where they shine, and where they are the wrong tool.

Thanks for reading Olko - Tech/Engineering! Subscribe for free to receive new posts and support my work.

Thanks for reading oleg’s Substack! Subscribe for free to receive new posts and support my work.

[

](https://substackcdn.com/image/fetch/$s_!I7Ab!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F52ce7a5c-dd31-4f50-b096-cf9889ea0caf_2400x1601.jpeg)

1. HTTP: The Common Language We Still Rely On

HTTP/1.1 remains the most widely adopted communication layer in distributed systems.

It works because it is:

simple
predictable
well-understood
universally supported

It gave us a shared vocabulary:

GET – retrieve
POST – create
PUT – replace
DELETE – remove

And a tone of voice through status codes:

200 : all good
404 : not here
500 : your server caught fire

Despite its age, HTTP continues to be the stable foundation on which everything else is built. You can replace frameworks. You cannot replace HTTP.

2. REST: When APIs Became Civilized

REST was not created to be “yet another API style.”

It was created to impose order.

Roy Fielding’s dissertation argued that networked systems should revolve around resources (nouns), not verbs:

GET /orders/123
POST /users
DELETE /sessions/42

REST gave teams:

statelessness
predictable semantics
cacheability
standardization

This model worked exceptionally well for public APIs and internal boundaries where stability mattered more than speed or flexibility.

The biggest complaint? Overfetching.

REST returns the full representation of a resource, even when the client wants two fields.

Where REST is a good fit

Example: Stripe / GitHub style public APIs.
Example: Internal “platform” APIs between teams.

Where REST is usually the wrong tool

Your mobile app makes 6–8 REST calls to render a single screen, overfetching large payloads. Latency and data usage are now a product problem.
Your frontend needs to assemble data from 4–5 services per view. You start building ad-hoc “backend for frontend” aggregation endpoints to paper over REST limitations. At this point, GraphQL is often a better abstraction.

3. SOAP, RMI, CORBA: The Empires That Collapsed

Before REST, enterprise engineering tried RPC over heavy, rigid contracts:

RMI
CORBA
SOAP

They assumed distributed systems should behave like local function calls.

It looked nice on whiteboards and failed in practice:

systems tightly coupled across languages and vendors
brittle XML contracts
versioning nightmares
huge payloads

These approaches taught the industry one important lesson:

When your service boundary looks like a shared class, you’re building a distributed monolith.

Where SOAP still makes sense

Example: Legacy payment, insurance, telecom backends.

Where SOAP is the worst option

Greenfield product, 2025, building a public API for a web and mobile client. Choosing SOAP here is self-sabotage. REST, GraphQL, or gRPC will all be strictly better in tooling, performance, and developer experience.

4. GraphQL: The Negotiator

Facebook developed GraphQL because frontend teams were tired of REST returning massive payloads.

GraphQL flipped the contract upside down:

The client chooses the shape of the data.
The server must comply.

Example:

query {
  shipment(id: 42) {
    origin
    destination
    fuelUsed
  }
}

Benefits:

precise data retrieval
strong types
single endpoint

Costs:

complex caching rules
backend resolvers easily turn into N+1 factories
schema ownership shifts toward frontend teams
more infra and operational overhead

GraphQL is great when UI teams outnumber backend teams and iteration speed matters more than backend control.

Where GraphQL is a good fit

Example: Shopify Storefront API / GitHub GraphQL API.
Example: Mobile apps under tight latency and bandwidth constraints.

Where GraphQL is the wrong tool

Internal service-to-service communication where:
Simple public APIs where you:

5. gRPC: The High-Performance Hotline

As systems grew, some teams needed:

speed
type safety
bi-directional streaming
efficiency over public accessibility

gRPC (Google’s modern RPC over HTTP/2) filled that gap.

Proto files define strict typed contracts:

service TradeService {
  rpc InitiateTrade(TradeRequest) returns (TradeResponse);
}

Strengths:

binary, compact
extremely fast
codegen for many languages
great for internal service-to-service calls

Tradeoffs:

harder to debug over the wire
not ideal for random third-party clients
browser support requires gRPC-Web and extra infra

Where gRPC is a good fit

Example: Internal trading / risk / pricing microservices.
Example: ML inference and data pipelines.

Where gRPC is the wrong tool

Public APIs aimed at:
Browser-only clients without gRPC-Web. You will end up building HTTP/JSON shims anyway.

6. tRPC: The TypeScript Shortcut

Teams building full-stack TypeScript apps (Next.js, Node) realized something:

“Why maintain two schemas—one for backend, one for frontend—if both sides already use TypeScript?”

tRPC removed the boundary entirely:

no schema files
no codegen
types flow automatically through the stack

It works exceptionally well when:

the entire system is TypeScript-based
you control both sides of the API
you want fast iteration over long-term stability

Where tRPC is a good fit

Example: B2B SaaS admin / dashboard app.
Example: Internal tools and ops consoles.

Where tRPC is the wrong tool

You need a public API for third-party customers.
You expect to migrate parts of the stack to Go, Rust, or Java.

7. API Gateways: Bureaucracy That Scales

Once an organization has enough services, APIs need governance.

This is where API gateways and service meshes come in.

Gateways handle:

authentication
rate limits
versioning
routing
monitoring

Service meshes handle:

mTLS
retries
service discovery
request shaping

This structure turns chaotic service communication into enforceable policy.

Real-world patterns

Example: Public-facing platform API.
Worst case: “Gateway as dumping ground.”

8. When to Use What: A Practical Decision Table

[

](https://substackcdn.com/image/fetch/$s_!NrEr!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83ed548f-a966-4b6a-a80f-6ed47cfffa7a_1834x508.png)

9. The Real Lesson

APIs are not technology choices.

They are alignment choices.

The real job is ensuring that:

teams agree on contracts
versioning is respected
breaking changes are controlled
services remain independently deployable
communication remains predictable

Distributed systems fail on semantics long before they fail on transport.

The better the protocol, the easier the negotiation.

But the negotiation is unavoidable.

10. Final Thought

The industry keeps inventing new ways for machines to communicate.

REST will not die.
GraphQL will not replace REST.
gRPC will not replace GraphQL.
tRPC will not replace anything outside TypeScript ecosystems.

Each solves a different piece of the same problem:

How do we get independent systems—and teams—to understand each other well enough to ship?

In that sense, HTTP really is the diplomacy layer of modern software. Everything else is negotiation technique.

Thanks for reading oleg’s Substack! Subscribe for free to receive new posts and support my work.

Thanks for reading Olko - Tech/Engineering! Subscribe for free to receive new posts and support my work.

Consistency Models: Why Your Database Lies to You (And When That’s Fine)

Ed — Wed, 05 Nov 2025 09:28:01 +0000

The Day Amazon’s Shopping Cart Became Haunted

In 2007, Amazon engineers discovered something disturbing: deleted items were coming back from the dead.

A customer would remove an item from their cart, continue shopping, then check out—only to find the “deleted” item had reappeared and been charged to their card. Support tickets flooded in. The bug wasn’t random—it happened during network partitions between Dynamo replicas.

Here’s what was happening: when a customer deleted an item, the write went to Replica A. Replica B, temporarily partitioned, never got the message. When the network healed, Replica B’s version—which still contained the item—merged with Replica A’s version. No clear “winner,” so the item resurrected.

Amazon’s response? They didn’t fix the bug. They redesigned the entire system to embrace it.

The result was Dynamo, the database that pioneered eventual consistency at scale, introduced vector clocks for conflict resolution, and changed how we think about distributed data.

This is where most engineers’ understanding stops: “CAP says you can’t have consistency and availability during partitions.” True. But that binary misses the entire spectrum of how consistent your system actually needs to be.

Thanks for reading Olko - Tech/Engineering! Subscribe for free to receive new posts and support my work.

The Consistency Spectrum Nobody Explains

CAP doesn’t say your data is either “perfectly consistent” or “complete chaos.” Between those extremes lies a gradient as wide as the difference between a bank transfer and a Twitter like.

The full spectrum:

Strong ←────────────────────────────────────────────────→ Weak
│ │ │ │ │
Linearizable Sequential Causal Session Eventual

High latency Low latency
Expensive Cheap
Simple reasoning Complex bugs

Every distributed database picks a point on this line. Your job as an engineer: know which point, understand the trade-offs, and not promise guarantees your system doesn’t provide.

The Four Models That Matter

1. Linearizability (Strongest)

Definition: Every operation appears to execute atomically at some point between its start and completion. All clients see operations in the same real-time order.

Think of it as: A single-threaded system, no matter how distributed it actually is.

Code behavior:

// Client A writes
await db.write(’x’, 5); // completes at time T1

// Client B reads immediately after
const val = await db.read(’x’); // guaranteed to see 5
console.log(val); // always prints 5, never 0 or stale value

Real systems: Google Spanner (using TrueTime atomic clocks), etcd, Zookeeper

Cost: High latency. Every write needs global coordination. Cross-region writes can take 100ms+.

When to use: Financial transactions, inventory management, anything where “approximately right” means “catastrophically wrong.”

2. Causal Consistency (The Pragmatic Middle)

Definition: If operation A causally affects operation B (e.g., B reads A’s write), all nodes see them in that order. Independent operations can appear in any order.

Think of it as: Preserving the story’s plot, but letting unrelated subplots unfold in parallel.

Code behavior:

// Thread 1: Post a tweet
await db.write(’tweet:123’, ‘Hello world’);
await db.write(’tweet:123:likes’, 0); // causally dependent

// Thread 2: Read elsewhere
const tweet = await db.read(’tweet:123’); // may be stale
const likes = await db.read(’tweet:123:likes’);
// Guarantee: if likes exists, tweet must also be visible
// No guarantee: how fresh either value is

Real systems: Azure Cosmos DB (default), Cassandra with careful config, some Redis setups

Cost: Medium latency. Requires tracking causal dependencies (vector clocks, version vectors) but no global locks.

When to use: Social networks, collaborative apps, anywhere causality matters but exact timing doesn’t.

3. Sequential Consistency

Definition: Operations from each client execute in program order, but there’s no guarantee of real-time ordering across clients.

Think of it as: Everyone agrees on A happening before B for one client, but Client 1’s timeline and Client 2’s timeline might interleave differently.

Code behavior:

// Client A
await db.write(’x’, 1);
await db.write(’x’, 2);

// Client B always sees 1 then 2 (or just 2, never 2 then 1)

// But Client C might see:
// Client A: x=1, x=2
// Client D: something else from D, x=1, x=2
// Different global orders are allowed

Real systems: Some distributed SQL databases, older MongoDB replica sets

Cost: Medium-low latency. Cheaper than linearizability but still requires some coordination.

When to use: Analytics systems, read-heavy workloads where per-user consistency matters but cross-user doesn’t.

4. Eventual Consistency (Weakest, Fastest)

Definition: If no new updates occur, eventually all replicas converge to the same value. Before then: anything goes.

Think of it as: “Trust me, it’ll make sense... eventually.”

Code behavior:

// Write to DynamoDB
await dynamoDB.putItem({
  TableName: ‘Users’,
  Item: { userId: ‘123’, name: ‘Alice’ }
});

// Read immediately from different replica
const user = await dynamoDB.getItem({
  TableName: ‘Users’,
  Key: { userId: ‘123’ },
  ConsistentRead: false // eventual consistency
});

console.log(user.Item.name);  
// Might print: undefined (write hasn’t propagated)
// Might print: “Bob” (old value still cached)
// Eventually prints: “Alice”

Real systems: DynamoDB (default), Cassandra at consistency level ONE, most CDNs

Cost: Lowest latency. Writes return immediately, reads hit local cache.

When to use: View counts, likes, analytics dashboards, CDN content—anywhere staleness is annoying but not damaging.

The Hybrid Models You’ll Actually Use

Real systems rarely pick one model globally. Instead, they offer tunable consistency per operation.

DynamoDB: Eventual by Default, Strong on Demand

// Fast but potentially stale
const staleUser = await dynamoDB.getItem({
  TableName: ‘Users’,
  Key: { userId: ‘123’ },
  ConsistentRead: false // default, eventual
});

// Slow but guaranteed fresh
const freshUser = await dynamoDB.getItem({
  TableName: ‘Users’,
  Key: { userId: ‘123’ },
  ConsistentRead: true // forces read from primary
});

AWS documentation: Read Consistency Options

Cassandra: Choose Per Query

// Write to majority of replicas (CP-leaning)
await cassandra.execute(
  ‘INSERT INTO users (id, name) VALUES (?, ?)’,
  [123, ‘Alice’],
  { consistency: cassandra.types.consistencies.quorum }
);

// Read from any replica (AP-leaning)
const result = await cassandra.execute(
  ‘SELECT name FROM users WHERE id = ?’,
  [123],
  { consistency: cassandra.types.consistencies.one }
);

Pattern: Strong writes for critical data, eventual reads for performance.

Session Consistency: The Mobile App Sweet Spot

Definition: Within a single user session, you see your own writes and causally-related operations. Across sessions: no guarantees.

Why it matters: Users expect their own actions to be reflected immediately. They don’t care if other users see stale data for a few seconds.

// Mobile app example
async function updateProfile(userId, newBio) {
  // Write with session token
  await db.write(
    `user:${userId}:bio`, 
    newBio,
    { sessionToken: user.session }
  );

  // Immediate read in same session - guaranteed to see new bio
  const profile = await db.read(
    `user:${userId}:bio`,
    { sessionToken: user.session }
  );

  return profile; // always returns newBio
}

// Different user reads same profile - might see old bio
// But that’s fine, they’ll see the update within seconds

Systems that provide this: Azure Cosmos DB (session consistency), MongoDB with read preference primary

The Decision Tree

Choose Linearizable when:

Money is involved (payments, account balances)
Inventory is limited (ticket sales, product stock)
Regulatory compliance requires audit trails
Pattern: Correctness > Speed, always

Choose Causal when:

Social interactions (posts, comments, reactions)
Collaborative editing (Google Docs, Figma)
Chat applications
Pattern: User expectations require logical ordering

Choose Session when:

User profiles and preferences
Shopping carts
Any single-user workflow
Pattern: “I see my changes immediately” matters, cross-user sync doesn’t

Choose Eventual when:

Analytics and metrics
Content recommendations
Search indices
View/like counts
Pattern: Speed matters, approximate is good enough

How Systems Fail (And Recover)

Twitter’s 2014 Timeline Ordering Bug

Problem: Tweets appeared out-of-order during replication lag

Root cause: Causal consistency violated—replies appeared before original tweets

Fix: Added explicit happens-before tracking for tweet threads

Reddit Vote Count Jumps

Problem: Upvote count changes dramatically on page refresh

Root cause: Eventual consistency—counting across replicas with stale reads

Fix: Added “updated X seconds ago” indicators, set user expectations

The Amazon Shopping Cart (2007)

Problem: Deleted items resurrected after partition

Root cause: Eventual consistency with no conflict resolution

Fix: Vector clocks + client-side merge logic in Dynamo

Key lesson: Weak consistency isn’t failure. Hiding it from users is.

Debugging Consistency Issues

Symptom: “My write disappeared”

Check if you’re reading from a different replica than you wrote to
Verify write acknowledgment (did it actually succeed?)
Look for partition healing—merges can “undo” writes

Symptom: “Data time-travels backward”

Reading from replicas with different lag
Check read-after-write guarantees in your client library
Consider session consistency or sticky routing

Symptom: “Conflicts I can’t explain”

Concurrent writes during partition
Check if system uses last-write-wins (LWW) or vector clocks
Look for application-level conflict resolution bugs

Practical Takeaways

1. Consistency is a spectrum, not a binary

Stop saying “my DB is consistent.” Specify: linearizable? causal? eventual?

2. Most apps need multiple consistency levels

Bank balance: strong. User bio: session. Feed ranking: eventual.

3. Latency and consistency are inversely related

Every 9 of consistency costs you a 0 in latency. Pick your battles.

4. Communicate your guarantees to users

“Updated 5 seconds ago” > silent staleness. Honesty scales.

5. Test with partitions, not just load

Use chaos engineering (Chaos Monkey, Gremlin) to simulate real failure modes.

Question for readers: Have you debugged a consistency bug in production? What was the “aha” moment when you realized it was a replication/consistency issue? Drop a comment—building a collection of war stories.

Leave a comment

What I Learned From 60 Days of Brutal Honesty in My Work Journal

Ed — Fri, 31 Oct 2025 08:37:47 +0000

Two months ago, I started keeping a work journal. Not just task lists and meeting notes - the sanitized version of my workday — but the unfiltered reality: how I felt, where I failed, and what scared me.

Here are four practices that emerged from this experiment in radical self-management.

Thanks for reading Olko - Tech/Engineering! Subscribe for free to receive new posts and support my work.

1. I Stopped Hiding From My Mistakes

After a significant fuckup that frustrated management, I could have written a polite note of apology and moved on. Instead, I wrote two words at the top of my journal: “I FUCKED UP.”

Then I built a system to prevent it from happening again:

A formal requirements validation process before writing code
A design review checkpoint to verify understanding
Clarifying questions to ask upfront, including: “What would you NOT build?”

But the most important step was writing a document titled “How You Can Help Me” for my manager. I outlined how leadership could provide more explicit constraints and challenge my assumptions.

I wasn’t just fixing my mistake—I was taking ownership of how clearly I understood future assignments. By defining how my manager could help me succeed, I was actively coaching my own leadership. This shift from reactive apology to proactive system-building changed how I approach every project.

2. I Started Tracking Energy Like a Performance Metric

Every day, I rate three things on a 1-to-10 scale: Energy, Focus, and Mood.

This felt silly at first. But after two months, the patterns became impossible to ignore.

During a brutal multi-week grind on a complex feature, my scores plummeted. Energy dropped to 3-4/10. Focus deteriorated. Mood stayed consistently poor for over two weeks. The moment the feature shipped, everything rebounded to 10/10.

This isn’t just feelings—it’s data. I can now see exactly which types of work drain me and which energize me. I can predict when I’m trending toward burnout weeks before it hits. I can make strategic decisions about when to push and when to recover.

Most professionals treat well-being as an afterthought. I now treat it as a critical resource to be monitored, managed, and strategically deployed.

3. I Stopped Going to Meetings and Started Managing Up

I used to show up to 1-on-1s with my manager and wait for direction. Now I prepare like I’m co-creating my role.

Before my last 1-on-1, I prepared three categories of questions:

To Understand Broader Context:

“From your perspective, what are the team’s biggest challenges right now?”

To Proactively Define Success:

“I want to balance delivery with deeper architectural understanding—what would success look like for me in the first 3 months?”

To Show Initiative on Team-Wide Issues:

“What’s your vision for developer experience improvements over the next quarter?”

This preparation changed everything. I’m no longer a passive employee waiting for instructions. I’m a strategic partner aligning my growth with the company’s most important objectives.

My manager once told me these conversations are the most valuable hour of their week. That’s not because I have all the answers—it’s because I show up prepared to help lead.

4. I Started Analyzing Team Dynamics Under Pressure

During a particularly difficult period for the team, I noticed how stress changed our communication patterns. People became more reactive, less constructive. Frustration leaked into conversations in ways that hurt rather than helped.

In my journal, I started analyzing what made communication break down:

Venting without specific problems to solve
Blame without actionable feedback
Emotion without structure for moving forward

Writing these observations wasn’t just venting. It was practice. By deconstructing what went wrong, I was training myself to communicate better under pressure. I was reinforcing the principles of psychological safety that high-performing teams need.

Now when I feel frustrated, I pause. I ask: “What specific behavior or outcome am I concerned about? What would improvement look like?” This mental discipline protects my team’s ability to perform when stakes are high.

What Changed

Peak performance isn’t about maintaining a perfect image. It’s about engaging in a consistent, private practice of radical self-awareness.

In my journal:

Failures become systems
Moods become data
Meetings become strategy sessions
Communication breakdowns become learning opportunities

The insight isn’t in the words I write. It’s in the discipline of writing them—of being brutally honest with myself about what’s working and what isn’t.

What’s the one thing you could start tracking tomorrow?

Thanks for reading oleg’s Substack! Subscribe for free to receive new posts and support my work

CAP Theorem: What Senior Engineers Actually Need to Know

Ed — Tue, 28 Oct 2025 16:17:34 +0000

Subscribe now

When GitHub Chose Consistency Over Your Pull Request

At 3:11 a.m. UTC on October 21, 2018, GitHub’s monitoring detected replication lag spiking from milliseconds to hours between U.S. data centers. Within minutes, the platform serving 30 million developers began showing stale data.

GitHub’s response? They deliberately froze all writes for five hours.

The banner every developer saw that morning:

“GitHub is experiencing degraded availability while we recover replication consistency.”

This wasn’t incompetence—it was the most expensive proof of CAP Theorem in production history.

When their MySQL cluster split across coasts, GitHub faced the choice every distributed system eventually faces: serve requests with potentially wrong data, or stop serving requests entirely. They chose the latter. Their post-mortem is worth reading.

If you’re designing anything beyond a single server—DynamoDB tables, Kafka clusters, replicated Postgres, microservices—you will make this exact choice. Understanding CAP isn’t optional. It’s the gravitational law of distributed computing.

The Theorem in 30 Seconds

CAP Theorem (Brewer 2000, formalized by Gilbert & Lynch 2002): In a distributed system experiencing a network partition, you can guarantee at most two of:

Consistency (C) – All nodes see identical data simultaneously
Availability (A) – Every request gets a non-error response
Partition Tolerance (P) – System functions despite network failures between nodes

Since network partitions are inevitable in distributed systems, you’re really choosing between C or A when failures occur.

Why This Matters: The Coffee Shop Test

Two branches of a coffee chain share a loyalty-points database. The network between them fails.

Scenario 1: Choose Availability (AP)

Both locations keep accepting purchases
Customer spends 500 points at Downtown, then immediately drives to Uptown and spends the same 500 points again
System reconciles later, discovers the conflict
Result: Lost revenue, but no angry customers

Scenario 2: Choose Consistency (CP)

Downtown can’t verify points balance with Uptown
Register locks until sync returns
Result: Guaranteed correctness, angry customers waiting

No architecture eliminates this trade-off. You only choose where to suffer.

Code: Seeing the Choice

AP System (Availability-First)

// DynamoDB-style eventual consistency
async function purchaseWithPoints(customerId, points) {
  try {
    // Write to local node immediately
    await localDB.write({ customerId, points, timestamp: Date.now() });

    // Replicate asynchronously (fire and forget)
    replicateAsync(customerId, points).catch(err => {
      logger.warn(’Replication delayed’, err);
    });

    return { 
      success: true, 
      consistency: ‘eventual’,
      message: ‘Purchase recorded, replicating...’
    };
  } catch (err) {
    // Still succeed even if remote nodes unreachable
    return { success: true, replicated: false };
  }
}

This code prioritizes uptime. Even if 2 of 3 replicas are down, customers transact. Conflicts get resolved through:

Last-write-wins (simple, potentially lossy)
Vector clocks (complex, deterministic)
Conflict-free replicated data types (CRDTs)

Real-world AP systems: DynamoDB, Cassandra, Riak

CP System (Consistency-First)

// MongoDB-style primary-secondary with write concerns
async function purchaseWithPoints(customerId, points) {
  const session = await startTransaction();

  try {
    // Write to primary
    await primaryDB.write({ customerId, points }, { session });

    // Block until majority of replicas confirm
    await waitForReplication({
      writeConcern: ‘majority’,
      timeout: 5000 // fail if can’t replicate in 5s
    });

    await session.commitTransaction();
    return { 
      success: true, 
      consistency: ‘strong’,
      message: ‘Purchase confirmed on all replicas’
    };
  } catch (err) {
    await session.abortTransaction();
    // Return error if replication fails
    return { 
      success: false, 
      error: ‘Network partition detected, refusing write’
    };
  }
}

This code prioritizes correctness. During partitions, requests fail rather than create divergent data. MongoDB’s evolution toward this model came after data loss incidents in early versions.

Real-world CP systems: Spanner, HBase, MongoDB with majority write concern

The Hybrid Reality: Tunable Consistency

Modern databases rarely pick a single point on the spectrum. Instead, they let you choose per operation.

DynamoDB example:

// Strong read (CP behavior)
const item = await dynamoDB.getItem({
  TableName: ‘Users’,
  Key: { userId: ‘123’ },
  ConsistentRead: true // Forces read from primary
});

// Eventual read (AP behavior)
const item = await dynamoDB.getItem({
  TableName: ‘Users’,
  Key: { userId: ‘123’ },
  ConsistentRead: false // May read from stale replica
});

Cassandra’s quorum tuning:

// Write to all 3 replicas, succeed when 2 confirm (CP-leaning)
await cassandra.execute(query, params, { 
  consistency: Consistency.QUORUM 
});

// Write to 1 replica, succeed immediately (AP-leaning)
await cassandra.execute(query, params, { 
  consistency: Consistency.ONE 
});

The Decision Tree: When to Pick What

Choose CP (Consistency > Availability) when:

Financial transactions (double-spending is catastrophic)
Inventory systems (overselling angers customers)
Config management (inconsistent configs break prod)
Medical records (wrong data kills people)

Pattern : When incorrect data causes more damage than downtime.

Choose AP (Availability > Consistency) when:

Social media feeds (stale likes don’t matter)
Analytics dashboards (approximate counts acceptable)
Shopping cart contents (user fixes conflicts themselves)
Messaging “read receipts” (eventual accuracy sufficient)

Pattern : When downtime frustrates users more than temporary staleness.

The gray area:

Most systems need both —strong consistency for writes, eventual for reads. This is where PACELC extends CAP: even when there’s no partition, you trade latency for consistency.

What About “CA” Systems?

The CA corner (Consistency + Availability, no Partition Tolerance) is theoretically impossible in truly distributed systems. Network partitions aren’t optional—they happen.

Single-node databases like standalone Postgres or Redis are “CA” but only because they’re not distributed. The moment you add a replica, you’re playing CAP.

The Real Lesson: Partitions Choose For You

Eric Brewer (the “CAP” in CAP Theorem) said it best in his 2000 PODC keynote: partitions aren’t theoretical. They’re Tuesday.

Cross-datacenter links fail
Rack switches crash
OS updates hang
Cloud providers have “availability zone degradation”

If you haven’t explicitly chosen CP or AP, the network will decide for you during the outage—usually in the worst possible way.

GitHub chose CP, went down for 5 hours, but preserved data integrity.

AWS DynamoDB chooses AP by default, stays up during partitions, accepts stale reads.

Neither is “better.” Both are intentional choices aligned with business requirements.

Takeaway Box

Use CP when:

Incorrect data causes financial loss, safety issues, or user-facing inconsistency that support can’t fix.

Use AP when:

Downtime loses revenue, user trust, or competitive edge faster than eventual consistency causes problems.

Use tunable consistency when:

Different operations have different requirements. Read-heavy analytics can be eventual; write-heavy checkout must be strong.

Test your choice:

Simulate partitions using chaos engineering (Netflix’s Chaos Kong, Gremlin.com). Discover your actual behavior before production does.

A collaborative collection of Interview questions

Ed — Thu, 05 Dec 2019 21:57:39 +0000

Ideally, the interview process is a conversation between equal parties. Not among superiors and inferior – it's hard to find good companies because it's even harder for them to find good employees. You, as Interviewee, should have a way to distinguish good from the bad - this list of questions will help you facilitate that process. So that's why this resource exists: https://counter-interview.dev

Feel free to add new questions!