DEV Community: Ciphemic academia

SQL Injection vs. XSS vs. CSRF: 1 Million Users in 20 Hours and a £400,000 Fine

Ciphemic academia — Tue, 13 Oct 2026 01:50:51 +0000

SQL Injection vs. XSS vs. CSRF: 1 Million Users in 20 Hours and a £400,000 Fine

On October 4, 2005, a MySpace user posted a profile containing a few lines of JavaScript that slipped past the site's filters. Anyone who viewed that profile ran the script without knowing it, and the script copied itself onto the viewer's own profile. Within about 20 hours, over one million users had run it, and MySpace temporarily shut the site down to fix the flaw. The worm was harmless, but it proved something alarming: a single input-handling mistake could turn a website's own visitors into its distribution network.

A decade later, in October 2015, the UK telecom TalkTalk lost the personal data of about 157,000 customers, including bank account numbers and sort codes for roughly 15,700 of them. The attacker used SQL injection against three vulnerable web pages sitting on a database TalkTalk had inherited in a 2009 acquisition and never fully inventoried. The regulator found the same flaw had been exploited successfully twice earlier that year, and fined the company a then-record £400,000.

And in 2008, Princeton researchers showed a third kind of attack hitting ING Direct, YouTube, MetaFilter, and The New York Times. An attacker's web page could make a logged-in visitor's own browser perform actions on those sites, such as opening an account at a bank, without the visitor ever clicking anything on the real site. That's cross-site request forgery, or CSRF.

These three attacks get lumped together as "web security stuff," but they fool three different victims. SQL injection fools the database. Cross-site scripting (XSS) fools the user's browser. CSRF fools the website itself. Once you see who's being fooled, the defenses stop feeling like a random checklist. This guide breaks down each attack, where the real numbers come from, and the one control that matters most for each.

If you're newer to security work, our Penetration Testing and Auth & Identity courses cover the fundamentals this comparison builds on.

This post originally appeared on the Ciphemic Academia blog.

The Short Version

SQL injection happens when user input gets treated as part of a database command. The attacker's text becomes code the database runs.
Cross-site scripting (XSS) happens when a site puts untrusted input into a page without encoding it. The attacker's script runs in other users' browsers, with the site's privileges.
CSRF happens when a site trusts any request that arrives with a valid session cookie. A malicious page makes the victim's browser send a request the victim never intended.

OWASP's Top 10:2025 reflects how differently these behave. It places Injection at A05 and notes that XSS shows up very often but with lower average impact (more than 30,000 CVEs), while SQL injection shows up less often but with higher impact (more than 14,000 CVEs). CSRF is mapped into Broken Access Control, OWASP's number one category, where 100% of tested applications had some form of the problem.

If you want one default: learn each attack by asking who is being fooled, then memorize one primary defense for each: parameterized queries for SQL injection, output encoding for XSS, and anti-CSRF tokens plus SameSite cookies for CSRF.

What All Three Are Actually Trying to Solve (For the Attacker)

Before the differences, the shared root cause:

Misplaced trust: each attack works because an application trusts something it shouldn't, whether that's user input, page content, or a browser's cookies
Ordinary features: the vulnerable code is usually a search box, a comment field, or a "transfer" button, not an exotic edge case
Well-known fixes: every one of these has had mature, documented defenses for years, which is why breaches that involve them draw regulator attention
Cheap to attempt: automated tools and copy-paste payloads let low-skill attackers try them at scale. The TalkTalk case involved a teenager using an off-the-shelf SQL mapping tool

The skill isn't memorizing payloads. It's recognizing where your application crosses a trust boundary and handling that crossing safely.

SQL Injection

What it actually is: an attack where input intended as data ends up interpreted as part of a SQL command. If your code builds queries by pasting user input into a string, an attacker can change what the query means.

A small example (vulnerable code, then the fix):

# Vulnerable: input becomes part of the SQL itself
query = f"SELECT * FROM users WHERE email = '{email}'"
cursor.execute(query)
# If email is:  ' OR '1'='1   the WHERE clause matches every row

# Safe: a parameterized query keeps input as data, never as SQL
cursor.execute("SELECT * FROM users WHERE email = %s", (email,))

Why it's dangerous:

Direct access to the data: a successful injection can read, change, or delete records, sometimes including data the page never meant to show
High impact when it lands: OWASP describes SQL injection as lower frequency but higher impact than XSS, which matches what TalkTalk experienced: 157,000 customers affected and a regulator's record fine
Easy to automate: attackers can point scanning tools at thousands of sites and see who's vulnerable

Why it keeps happening:

Legacy and inherited systems: TalkTalk's vulnerable pages sat on outdated, unpatched database software it hadn't known it owned
String-building habits: quick scripts and raw queries inside otherwise safe frameworks reintroduce the flaw
Ignored warnings: the regulator highlighted that the same vulnerability had been exploited twice earlier in 2015

How to defend:

Parameterized queries or prepared statements, everywhere, for every database call
Safe ORM APIs, and extra care wherever you drop to raw SQL
Least-privilege database accounts, so a compromised query can't reach everything
Patch and inventory your systems, including acquired ones
Input validation as a secondary layer, never as the primary defense

Who is fooled: the database, which can't tell where the developer's SQL ends and the attacker's begins.

Cross-Site Scripting (XSS)

What it actually is: an attack where untrusted input is included in a web page without proper encoding, so the browser treats it as script. The script runs with the site's privileges inside the victim's browser. It comes in three common forms: stored (saved on the server, such as a comment), reflected (bounced back from a request, such as a search parameter), and DOM-based (created by client-side code).

A small example (vulnerable template, then the fix):

<!-- Vulnerable: the comment is inserted into the page as raw HTML -->
<p>Latest comment: {{ comment|safe }}</p>
<!-- A comment containing <script>alert('XSS')</script> runs in every visitor's browser -->

<!-- Safe: output is HTML-encoded, so the text is displayed, not executed -->
<p>Latest comment: {{ comment }}</p>
<!-- Jinja2 and most modern frameworks auto-escape by default -->

Why it's dangerous:

It runs as your site: injected script can act as the logged-in user, read page content, and make requests on their behalf
It spreads: the Samy worm showed a stored XSS flaw can propagate itself, reaching over one million MySpace users in 20 hours
It's extremely common: more than 30,000 CVEs, which is why OWASP calls it high frequency, even if average impact per instance is lower than SQL injection

Why it keeps happening:

Filters that miss cases: MySpace did filter user content, and the Samy author found a way to sneak JavaScript past the filters
Developers opting out of safe defaults: using raw-HTML escape hatches such as |safe in templates or dangerouslySetInnerHTML in React
Many output contexts: HTML text, attributes, JavaScript, and URLs each need different encoding

How to defend:

Context-aware output encoding, which modern frameworks do automatically unless you disable it
A Content Security Policy, to limit what scripts the browser will run even if something slips through
HttpOnly cookies, so injected script can't read session cookies
A vetted sanitizer such as DOMPurify when you genuinely must allow user HTML
Avoid building HTML from strings in your own front-end code

Who is fooled: the browser, which trusts script that appears to come from the site you're visiting.

Cross-Site Request Forgery (CSRF)

What it actually is: an attack where a malicious page makes a logged-in user's browser send a request to a trusted site. Because browsers automatically attach cookies, the site sees a valid session and assumes the user meant to do it. The attacker never needs to see the response; they just need the action to happen.

A small example (the attacker's page, using placeholder domains):

<!-- On a page the victim is lured to: a hidden form that submits itself -->
<form action="https://bank.example/transfer" method="POST">
  <input type="hidden" name="to" value="attacker-account">
  <input type="hidden" name="amount" value="1000">
</form>
<script>document.forms[0].submit()</script>
<!-- If the victim is logged in at bank.example and the site has no CSRF defense,
     the request arrives with their cookies and looks legitimate -->

Why it's dangerous:

It borrows the victim's identity: the request comes from the real browser with real cookies, so the site can't distinguish it from a genuine action
No code runs on the target site: the Princeton researchers stressed that a site fully protected from XSS can still be open to CSRF if it takes no CSRF protections
Real consequences: in their 2008 findings, one attack could open accounts and move funds at a bank, another exposed users' email addresses, another let an attacker take over a MetaFilter account through its password-reset feature
HTTPS doesn't help: ING's use of SSL did nothing to prevent the attack

Why it keeps happening:

Developers don't think about it: the researchers noted CSRF has been widely overlooked because of a lack of awareness
State-changing actions on GET requests, or POST handlers with no verification of intent
Cookie-based sessions by default: browsers send cookies automatically, which is exactly what the attack relies on

How to defend:

Anti-CSRF tokens, a secret value tied to the user's session that the attacker's page cannot know, verified on every state-changing request
SameSite cookies, which tell the browser not to send cookies on most cross-site requests. Modern browsers default many cookies to a restrictive setting, but don't rely on that alone
Check the Origin or Referer header as an additional layer
Re-authenticate for sensitive actions such as changing a password or moving money
Use your framework's built-in protection rather than inventing your own

Classic CSRF mostly targets cookie-based sessions. APIs that authenticate with a token the client must attach explicitly, rather than a cookie the browser sends on its own, aren't open to it in the same way; our JWT vs. Sessions post covers that trade-off.

Who is fooled: the website, which trusts any request that arrives with a valid session cookie.

Side-by-Side Comparison

	SQL Injection	XSS	CSRF
Who is fooled	The database	The user's browser	The website
What the attacker abuses	Input treated as SQL code	Input treated as page script	A browser's automatic cookies
Where the attacker's content runs	In the database	In the victim's browser	In no one's; the victim's browser just sends a request
Typical impact	Data theft, tampering, deletion	Session hijacking, account actions, worms	Unwanted actions as the victim
OWASP 2025 view	Injection (A05): low frequency, high impact, 14k+ CVEs	Injection (A05): high frequency, lower impact, 30k+ CVEs	Broken Access Control (A01)
Primary defense	Parameterized queries	Context-aware output encoding	Anti-CSRF tokens and SameSite cookies
Backup defenses	Least privilege, patching, validation	Content Security Policy, HttpOnly cookies	Origin checks, re-authentication
Famous case	TalkTalk, 2015	Samy worm, 2005	Princeton's 2008 findings (ING Direct, others)

How the Three Fit Together

A quick way to hold all three in your head:

SQL injection:  attacker → [web app] → database runs the attacker's SQL
XSS:            attacker → [web app] → victim's browser runs the attacker's script
CSRF:           attacker's page → victim's browser → [web app] acts on the victim's behalf

They also combine. A single XSS flaw can defeat CSRF protections, because script running inside the site's own pages can read anti-CSRF tokens and send valid requests. That's why fixing only one of these isn't enough, and why a threat model should ask all three questions: what input reaches my database, what input reaches my pages, and which of my actions can be triggered by a request alone.

How This Fits a Career Path

Backend and full-stack engineers: parameterized queries and your framework's CSRF and XSS defaults are a baseline expectation, and missing them is a classic code review finding
Application security engineers: you'll find and fix these in design reviews, code reviews, and automated scanning
Penetration testers: these three are staples of web application testing, and our threat modeling vs. penetration testing post shows where testing fits in a security program
DevSecOps roles: you'll wire scanners and dependency checks into delivery pipelines; see our DevSecOps career guide
Interviews: "what's the difference between XSS and CSRF?" is one of the most common web security interview questions, and the "who is fooled" framing gives you a clear answer

How to Defend Against All Three Without Overthinking It

Never build SQL by joining strings with user input. Use parameterized queries every time, and restrict database account permissions.
Encode output by context and keep your framework's defaults on. Treat every raw-HTML escape hatch as a code review red flag, and add a Content Security Policy.
Protect every state-changing request with an anti-CSRF token and SameSite cookies, and never change state on a GET request.
Patch and inventory everything, including acquired and forgotten systems. TalkTalk's breach started on infrastructure it didn't fully know it had.
Test for all three regularly. Combine code review, automated scanning, and a penetration test, and only test systems you own or have written permission to test.
Practice legally. Intentionally vulnerable training apps such as OWASP Juice Shop and DVWA are designed for exactly this.

A note on honesty: OWASP's CVE counts measure how many vulnerabilities have been reported across software, not how likely your specific application is to be affected. TalkTalk's total cost is reported at different figures across sources, with estimates ranging from about £42 million to about £60 million, so treat the £400,000 fine, the only official figure, as the anchor. The Samy worm and the 2008 Princeton findings are historical, and the affected sites fixed those flaws long ago. They're useful because they illustrate each class of flaw clearly, not because those exact bugs still exist.

Common Mistakes When Learning Web Security

Trying to "sanitize" instead of parameterize. Stripping quotes or blocking keywords is fragile. Parameterized queries remove the problem at its root.
Assuming HTTPS stops these attacks. HTTPS protects data in transit. It doesn't stop a database from running injected SQL, a browser from running injected script, or a site from honoring a forged request.
Assuming XSS protection covers CSRF. The Princeton researchers made this exact point: they're different problems with different fixes.
Using raw-HTML escape hatches carelessly. |safe, innerHTML, and dangerouslySetInnerHTML bypass the encoding that protects you.
Changing state on GET requests. Links and image tags can trigger GET requests from any page, which makes forged requests trivial.
Leaving session cookies without HttpOnly and SameSite. Both flags are cheap and block whole classes of attack.
Ignoring inherited and legacy systems. A forgotten, unpatched database is exactly how a record fine started.
Testing without permission. Only probe systems you own or are explicitly authorized to test.

Frequently Asked Questions

Which of the three is the most dangerous?

By typical impact per instance, SQL injection, since it can expose or alter the database directly. By sheer frequency, XSS. CSRF's danger depends on what actions your site lets a logged-in user perform. In practice, all three need fixing.

Does HTTPS prevent SQL injection, XSS, or CSRF?

No. HTTPS encrypts traffic between browser and server. All three attacks work through ordinary, correctly encrypted requests.

Can XSS be used to bypass CSRF protection?

Yes. Script running inside your own pages can read anti-CSRF tokens and send valid requests, which is why XSS defenses are also CSRF defenses in practice.

Is CSRF still a problem now that browsers default to SameSite cookies?

It's much less common than it was, but not gone. Defaults differ across browsers and cookie types, and some designs still need explicit tokens. Treat SameSite as one layer, not the only one.

Do ORMs make SQL injection impossible?

They greatly reduce it by parameterizing queries for you, but raw-SQL escape hatches and unsafe string-building inside an ORM can reintroduce it.

How can I practice these safely and legally?

Use deliberately vulnerable training apps such as OWASP Juice Shop or DVWA on your own machine, never a real website without written permission.

How do interviewers evaluate this topic?

Usually by asking you to explain the difference and the primary defense for each. Answering with "who is fooled" and one concrete fix per attack shows real understanding instead of memorized definitions.

Know Who Is Being Fooled

SQL injection fools the database, XSS fools the browser, and CSRF fools the website. Learn the attack, the single most important defense, and where your own application crosses each trust boundary. Explore the Penetration Testing and Auth & Identity courses to practice finding and fixing all three in safe, hands-on projects.

Unit vs. Integration vs. End-to-End Testing: The $125 Million Bug Between Two Components

Ciphemic academia — Mon, 12 Oct 2026 01:22:59 +0000

Unit vs. Integration vs. End-to-End Testing: The $125 Million Bug Between Two Components

In September 1999, NASA's Mars Climate Orbiter reached Mars and fired its engine to enter orbit. It never called home. The investigation found that two pieces of software had each done their jobs. The spacecraft's flight software expected thruster impulse data in newton-seconds, as the interface specification required. The ground software that produced that data reported it in pound-force seconds instead, a factor of 4.45 off. Each side was internally consistent. The bug lived in the gap between them, and a check that the two sides used compatible units was never done. The result was a lost orbiter, commonly cited at $125 million.

That's the story of a defect that no test of one component alone is designed to catch, and it's the cleanest way to understand why software teams write three kinds of tests, not one. Unit tests check a piece in isolation. Integration tests check that pieces work together. End-to-end tests check that the whole system serves a real user journey. Google's testing team suggests a 70/20/10 split between them as a first guess, and a 2026 flaky-test roundup reports that Meta's end-to-end tests run at roughly 10% flakiness while unit tests on the same codebases stay under 1%. Each layer buys you something different, and each costs something different.

This guide breaks down what each type of test actually does, where each is blind, and how to decide the right mix without cargo-culting a ratio. If you're newer to pipelines, our CI/CD Pipelines roadmap shows where these tests run once you've written them.

This post originally appeared on the Ciphemic Academia blog.

The Short Version

Unit tests check one small piece of logic (a function or class) in isolation. They're fast, cheap, and pinpoint failures precisely.
Integration tests check that several pieces work together: your code with a real database, a queue, another service, or a third-party API. They catch problems at the boundaries.
End-to-end (E2E) tests drive the whole system the way a user would, through the UI or public API, in a production-like environment. They give the most realistic confidence at the highest cost.

If you want one default: write many unit tests, a moderate number of integration tests at your system's boundaries, and a small set of end-to-end tests for your most critical user journeys. Treat Google's 70/20/10 as a starting guess, not a law.

What All Three Are Actually Trying to Solve

Before the differences, the shared goal:

Catching regressions before users do, so a change in one place doesn't quietly break another
Giving you confidence to change code, because refactoring without tests is gambling
Fast feedback, since a failure found in seconds is cheaper than one found in production
Documenting intended behavior, because a good test shows how something is supposed to work
Supporting automated delivery, since pipelines are only as trustworthy as the tests gating them

The tension is always the same: the more realistic a test is, the slower, costlier, and flakier it tends to be. The three layers exist to balance that trade-off.

Unit Tests

What it actually is: a test of the smallest meaningful piece of logic, run in isolation, with anything slow or external (databases, networks, clocks) replaced by simple stand-ins. Unit tests answer one narrow question: given these inputs, does this function return the right output?

A small example (a unit conversion, the kind of function that sits at the heart of the Mars Climate Orbiter story):

def to_newton_seconds(value, unit):
    if unit == "lbf_s":
        return value * 4.44822
    if unit == "n_s":
        return value
    raise ValueError(f"unknown unit: {unit}")

def test_pound_force_seconds_convert_to_newton_seconds():
    assert round(to_newton_seconds(1.0, "lbf_s"), 2) == 4.45

def test_newton_seconds_pass_through_unchanged():
    assert to_newton_seconds(2.5, "n_s") == 2.5

Where it shines:

Speed: Google's testing team has said that even a tenth of a second is slow for a unit test, which is why thousands can run on every commit
Reliability: with nothing external to go wrong, unit tests rarely fail for reasons unrelated to the code
Precise failures: when one breaks, you know which function to look at
Cheap to write and maintain when code is designed to be testable

Where it struggles:

Blind to interaction bugs: the Mars Climate Orbiter's two components could each pass their own tests and still fail together
Mocks can lie: if you replace a dependency with a stand-in that behaves differently from the real thing, the test passes while reality fails
Hard on untestable code: tightly coupled code with hidden dependencies makes unit tests painful to write, and writing testable code is a skill in itself

Who this suits: every project, as the foundation. Business logic, calculations, validation rules, and data transformations belong here.

Integration Tests

What it actually is: a test that exercises several real components together across a boundary: your service with a real database, your code against a message queue, one service calling another, or your app against a third-party API's test environment. The goal is to verify that the pieces agree with each other, not just that each works alone.

A small example (an API endpoint plus a real test database):

def test_create_order_saves_to_database(client, db):
    response = client.post("/orders", json={"sku": "A1", "qty": 2})

    assert response.status_code == 201
    assert db.query("SELECT qty FROM orders WHERE sku = 'A1'") == [(2,)]

Where it shines:

Catches boundary mismatches: wrong units, wrong field names, wrong data types, and schema drift between components are exactly what this layer is for
Tests real behavior of your infrastructure: SQL queries, transactions, queue delivery, and serialization behave differently against the real thing than against a mock
A cheaper way to find interaction bugs than E2E: Google's own guidance asks why you'd write a large end-to-end test when a smaller, focused integration test finds the same bug
Medium speed: slower than unit tests, much faster than driving the whole product

Where it struggles:

More setup: databases, containers, and test data need to be provisioned and reset
Harder to localize failures: a failure could be in either component or in the glue between them
Slower feedback than unit tests, so teams tend to run them less often, such as per pull request rather than per keystroke
Fuzzy definition: teams use the term differently, so agree internally on what "integration" means for you

Who this suits: any system with databases, queues, external APIs, or multiple services, which is nearly every real backend. Microservice teams often add contract tests here to check that services agree on message formats.

End-to-End Tests

What it actually is: a test that drives the complete system from the user's side, through a real browser or the public API, against a production-like environment with all the real pieces running. It verifies that a whole user journey works from start to finish.

A small example (a browser test with Playwright):

import { test, expect } from '@playwright/test';

test('a new user can sign up and reach the dashboard', async ({ page }) => {
  await page.goto('https://staging.example.com/signup');
  await page.fill('#email', 'test.user@example.com');
  await page.fill('#password', 'a-long-test-password');
  await page.click('button[type=submit]');

  await expect(page.locator('h1')).toHaveText('Welcome');
});

Where it shines:

Most realistic confidence: if the signup journey passes, the front end, API, database, and configuration are all wired together correctly
Catches environment and wiring problems that lower-level tests can't see: missing configuration, a broken deploy, an incompatible release
Protects your critical business paths: sign up, log in, checkout, and payment are worth verifying as a user would experience them

Where it struggles:

Slow feedback: Google's guidance notes you have to wait for the whole product to be built, deployed, and then for the tests to run
Flakiness: end-to-end tests touch timing, networks, and shared environments. One 2026 roundup reports roughly 10% flakiness for Meta's end-to-end tests against under 1% for its unit tests, and Google reported in 2016 that around 16% of its tests showed some level of flakiness
Hard to debug: a failure tells you a journey broke, not where
Expensive to maintain: UI changes break tests that have nothing to do with a real bug

Who this suits: every product with real users, but in small numbers: a handful of critical journeys, not an exhaustive tour of every feature.

Side-by-Side Comparison

	Unit	Integration	End-to-End
Scope	One function or class	Several real components together	The whole system, as a user
Speed	Fastest	Medium	Slowest
Reliability	Highest	Medium	Lowest, most prone to flakiness
Pinpoints failures	Precisely	Roughly	Poorly
Best at catching	Logic errors	Boundary and contract mismatches	Wiring, config, and journey-level failures
Maintenance cost	Low	Medium	High
Runs	On every commit	Per pull request or merge	Before release or on a schedule
First-guess share of tests	About 70%	About 20%	About 10%
Example tools	pytest, JUnit, Jest	Testcontainers, Supertest, contract tests	Playwright, Cypress, Selenium

How the Layers Fit Together

The classic picture is a pyramid: lots of cheap, fast tests at the bottom, fewer slow, expensive ones at the top.

          /\
         /E2E\          few: slow, realistic, critical journeys
        /------\
       / Integr.\       some: medium, boundaries and contracts
      /----------\
     /    Unit    \     many: fast, precise, logic
    /--------------\

Take the Mars Climate Orbiter as a worked example of what each layer would ask. We can't know for certain what any one practice would have caught, but we can see the question each raises:

Unit tests ask: does each conversion function return the right number for the unit it's given? Both sides could answer yes.
An integration test asks: when the ground software hands this value to the navigation software, does the number mean the same thing on both sides? That's the question nobody checked.
An end-to-end check asks: does the full trajectory, built from the real pieces together, behave as predicted?

The failure lived at the interface, which is where integration testing earns its place.

How This Fits a Career Path

Backend and full-stack engineers: writing good unit and integration tests is a baseline expectation, and it's often how code reviewers judge your work
QA, SDET, and test automation roles: you'll own the end-to-end layer and the strategy for keeping it small and trustworthy
DevOps and platform engineers: you decide which tests gate which stage of the pipeline; see our CI/CD pipelines vs. manual deployment post for how that automation fits together
Interviews: "how would you test this?" is a common question, and strong answers name all three layers and justify the mix for the specific system

How to Choose Without Overthinking It

Start with unit tests for your logic. Calculations, validation, and transformations are cheap to cover and pay back constantly.
Add integration tests at every boundary. Anywhere your code meets a database, a queue, a third-party API, or another team's service, test the real interaction at least once.
Pick three to five critical journeys for end-to-end tests, such as signup, login, and checkout, and resist the urge to add more. Run them after deploys or on a schedule, not on every keystroke.
Treat flaky tests as bugs. Quarantine them, fix the root cause, and never train the team to shrug off red builds.
Adjust the ratio to your system. Many external integrations call for more integration tests. A UI-heavy product may justify more end-to-end coverage. Treat 70/20/10 as a starting point.

A note on honesty: the 70/20/10 split is described by Google as a first guess, and engineers disagree about how to measure the ratio at all. The flakiness figures come from industry roundups and vary a lot by organization, test type, and how "flaky" is defined, so treat them as evidence of a real pattern, not as numbers you'll see in your own pipeline. The Mars Climate Orbiter's $125 million figure is the commonly cited cost of the orbiter itself. Figures vary by source, and the broader Mars Surveyor '98 mission, including the lander, cost considerably more.

Common Mistakes When Learning to Test

The inverted pyramid. A suite that's mostly slow end-to-end tests feels thorough but gives slow, flaky feedback. Push checks down to the cheapest layer that can catch the bug.
Mocking everything. If every dependency is a stand-in, your unit tests can pass while the real integration is broken, which is the Mars Climate Orbiter pattern in miniature.
Calling a test a unit test when it isn't. If it needs a network, a database, or the file system, it's an integration test, and it probably runs slower than you think.
Chasing a coverage percentage. High coverage with weak assertions gives false comfort. Test behavior that matters.
Testing implementation details. Tests tied to internal structure break on every refactor and teach the team to distrust them.
Ignoring flaky tests. Once people assume a red build is "probably flaky," real failures slip through.
Skipping integration tests for database code. SQL, transactions, and migrations behave differently against a real database than against a mock.

Frequently Asked Questions

Which type of test should a beginner write first?

Unit tests. They're quick to write, give instant feedback, and teach you to structure code in testable pieces. Add integration tests as soon as your code talks to a database or another service.

Is 70/20/10 a rule I have to follow?

No. Google describes it as a first guess. Systems with many integrations need more integration tests, and UI-heavy products may need more end-to-end tests. The shape matters more than the exact numbers: many cheap tests, fewer expensive ones.

What's the actual difference between integration and end-to-end tests?

Integration tests check that a few components work together across a specific boundary, often without a UI. End-to-end tests drive the entire system as a user would, through a real interface, in a production-like environment. End-to-end tests are broader, slower, and flakier.

How many end-to-end tests do I need?

Fewer than you think. Cover your most important user journeys, such as signup, login, and checkout, and add more only when a real production failure justifies it.

What is a flaky test, and what should I do about it?

A flaky test sometimes passes and sometimes fails without any code change, usually from timing, shared state, or environment issues. Quarantine it, find the root cause, and fix or delete it. Leaving flaky tests in the main suite teaches people to ignore failures.

How do interviewers evaluate this topic?

They usually give you a feature and ask how you'd test it. A strong answer names unit tests for the logic, integration tests for the database and external calls, and a small number of end-to-end tests for the key journey, and explains why you wouldn't test everything at the top layer.

Build a Test Suite You Can Trust

The Mars Climate Orbiter wasn't lost because anyone skipped testing their own piece. It was lost in the space between two pieces that each worked, which is the exact space integration tests exist to check. Start with fast unit tests, test every boundary for real, and keep your end-to-end tests few and meaningful. Our free CI/CD Pipelines roadmap walks you through running these tests automatically on every change, and you can explore the courses for the backend and DevOps foundation underneath it.

Threat Modeling vs. Penetration Testing: The $80 Million Lesson From Capital One

Ciphemic academia — Sat, 10 Oct 2026 01:36:30 +0000

Threat Modeling vs. Penetration Testing: The $80 Million Lesson From Capital One

In 2019, an attacker got into Capital One's cloud environment through a misconfigured web application firewall. From there, a single server-side request forgery (SSRF) trick let the attacker ask the server for its own cloud credentials. Those credentials belonged to an over-permissioned role, which could read hundreds of storage buckets. About 106 million people in the US and Canada were affected. The intrusion went unnoticed for roughly four months, and the bank was later hit with an $80 million penalty from the US Office of the Comptroller of the Currency and agreed to a class-action settlement of about $190 million. The regulator's reasoning was telling: it pointed to the bank's failure to set up effective risk assessment before moving major operations to the public cloud.

Closer to home, IBM's 2025 Cost of a Data Breach Report put the average breach in India at ₹220 million (₹22 crore), a 13% jump from ₹195 million the year before, with an average of 263 days to identify and contain a breach. Globally, the average was $4.44 million and 241 days.

Those numbers raise two different questions. First: did anyone ask, before launch, what could go wrong with this design? That's threat modeling. Second: did anyone attack the real, running system to see what actually breaks? That's penetration testing. Most teams hear these two terms used interchangeably, then run one, call it "security," and assume the other is covered. This guide explains what each practice actually does, where each one is blind, and how they fit together.

If you're newer to security work, our Threat Modeling and Penetration Testing courses cover the fundamentals this comparison builds on.

This post originally appeared on the Ciphemic Academia blog.

The Short Version

Threat modeling is a structured way to analyze a system's design and ask what can go wrong, before or while you build it. It produces a prioritized list of threats and the decisions to address them.
Penetration testing is an authorized, simulated attack against a real, running system to find and prove exploitable weaknesses. It produces evidence and a report of findings to fix.

If you want one default: they aren't alternatives, and the order matters. Threat model during design, build, then pen test the result. If you can only do one right now, do the lightweight threat model first, because it's cheaper and it makes the pen test far more focused.

What Both Are Actually Trying to Solve

Before the differences, the shared goal:

Finding security problems before attackers do, while fixing them is still your choice and not an incident response
Prioritizing by real risk, not by a flat list of every theoretical weakness
Producing something a team can act on, which means concrete findings, owners, and fixes
Building security thinking into how the team works, so the same mistake doesn't ship twice

The difference is when each one looks and what it can see.

Threat Modeling

What it actually is: a structured conversation, usually around a diagram of your system, that asks four simple things: what are we building, what can go wrong, what will we do about it, and did we do a good enough job. Teams map the system with a data flow diagram (components, data stores, and the trust boundaries between them), then walk each piece looking for threats. A common checklist for that walk is STRIDE: spoofing, tampering, repudiation, information disclosure, denial of service, and elevation of privilege.

A small example (one threat from a model of a feature that fetches a user-supplied URL):

Component:   Web app that fetches a URL supplied by the user
Threat:      Information disclosure / elevation of privilege
Scenario:    Attacker submits an internal address (the cloud metadata endpoint)
             so the server fetches its own temporary credentials
Key question: What can those credentials reach?
Mitigations: Block internal and link-local addresses, require IMDSv2,
             scope the server's cloud role to the minimum it needs

Where it shines:

Early and cheap: it runs on a whiteboard before code exists, when changing the design is cheapest. Fixing a design problem after release generally takes more work than fixing it on paper, though how much more varies widely
Finds design flaws, not just bugs: over-permissioned roles, missing trust boundaries, and risky data flows don't show up as a vulnerability in a scanner
Builds shared attacker thinking: developers, architects, and ops learn to ask "how would I abuse this?" as a habit
Focuses everything downstream: a good threat model tells your pen testers and reviewers where to spend their time

Where it struggles:

It's only as good as the team's knowledge and imagination: if nobody in the room knows about SSRF against metadata services, it won't make the list
It's analysis, not proof: a threat on paper may or may not be exploitable in the real system
It goes stale: a model made at project kickoff stops matching reality after a few releases unless someone updates it
It needs time and buy-in: skipping it is easy because nothing visibly breaks when you do

Who this suits: anyone designing a new system or feature, teams migrating to the cloud (the exact situation the regulator flagged in the Capital One case), and any team that wants security decisions made before they're expensive.

Penetration Testing

What it actually is: a simulated attack, performed with written authorization, against a live or staging system to find weaknesses and prove they can be exploited. Testers follow a process: agree the scope and rules of engagement, gather information, look for vulnerabilities, try to exploit them, see how far access extends, and write up what they found with severity ratings and fix guidance. Engagements are often described by how much the tester knows up front: black box (nothing), grey box (some access or documentation), or white box (full knowledge, including code). Common reference guides include the Penetration Testing Execution Standard, the OWASP Web Security Testing Guide, and NIST SP 800-115.

A small example (one finding from a report):

Finding:   Server-side request forgery in /fetch-url
Severity:  High
Evidence:  Request to the cloud metadata address returned the instance role name
Impact:    Attacker can retrieve temporary cloud credentials
Fix:       Allow-list outbound destinations, enforce IMDSv2,
           reduce the role's permissions

Where it shines:

It proves exploitability: a working exploit settles arguments about whether something is a real risk
It finds implementation and configuration problems: misconfigured firewalls, weak settings, and forgotten endpoints that no design document mentions
It produces evidence that gets budget and attention: a demonstrated path to customer data persuades leadership in a way a theoretical threat often doesn't. Many frameworks that cover card payments or similar regulated data also expect regular penetration testing, so check the requirements that apply to you
It reflects what attackers actually do: the Verizon 2025 Data Breach Investigations Report found vulnerability exploitation was the initial access vector in 20% of breaches, up 34% from the prior year's report, across 12,195 confirmed breaches

Where it struggles:

It's a snapshot: the system you tested on Tuesday changes the next time someone deploys
It only covers what's in scope: anything outside the agreed boundary stays untested, including a forgotten server or a third-party integration
It happens late: by the time you pen test a finished system, a design-level problem is more disruptive to fix than the same problem caught at the whiteboard
Quality varies: a rushed test, or one that's really just an automated scan, can give false comfort

Who this suits: teams with a running system to validate, anyone with compliance or customer requirements for independent testing, and anyone who needs evidence rather than opinions.

Side-by-Side Comparison

	Threat Modeling	Penetration Testing
Core question	What could go wrong with this design?	What can an attacker actually break right now?
When it happens	Design and build, then revisited	After build, on a running system
Input	Architecture diagrams, data flows, team knowledge	A live or staging system, an agreed scope
Output	Ranked threats and design decisions	Proven findings with severity and fixes
Evidence type	Reasoned analysis	Demonstrated exploitation
Best at finding	Design flaws, missing controls	Misconfigurations, implementation bugs
Blind spot	Doesn't prove exploitability	Only sees what's in scope, at one point in time
Typical cost profile	Time from your own team	Specialist time, scoped per engagement
Common frameworks	STRIDE, data flow diagrams, the four-question approach	PTES, OWASP WSTG, NIST SP 800-115

How They Fit Together

The two practices feed each other:

Design → Threat model → Build → Pen test → Fix → Update threat model → Repeat

The threat model tells testers where the interesting risks are. The pen test confirms which threats are real and often surfaces threats the team never imagined, which then go back into the model. Run on their own, each leaves a gap: threat modeling without testing stays theoretical, and testing without a model spends time wandering.

Take the Capital One chain as a worked example of the questions each practice asks. We can't know whether either would have prevented that breach, but we can see the kind of question each one raises:

Threat modeling asks: this service fetches requests on behalf of users and runs with a cloud role. What can that role reach? What happens if someone points the server at its own metadata endpoint? Why can one role read so many storage buckets?
Penetration testing asks: can I make this firewall forward a request to the metadata service, and what happens if I use the credentials that come back?

Different questions, both relevant to the same chain.

How This Fits a Career Path

Application security or security engineer: threat modeling is a core, everyday skill, since it's how you influence design before code is written
Penetration tester or red team roles: this is its own career track, built on hands-on offensive skills and clear report writing
DevSecOps roles: you'll fold both into the delivery process, including lightweight threat modeling for new features and automated testing in pipelines; see our DevSecOps career guide for how that fits together
Developers and architects: even if security isn't your title, being able to run a one-hour threat model on your own design is a standout skill
Interviews: security-focused interviews often ask you to threat model a design on the spot, so practice walking through trust boundaries and abuse cases out loud

How to Choose Without Overthinking It

Designing something new or changing the architecture? Start with a threat model, even a one-hour whiteboard session.
Have a running system and need proof or compliance evidence? Commission a penetration test with a clearly written scope.
Small team or startup? Do a lightweight threat model per major feature, add automated scanning, and book an independent pen test before a major launch or when a customer requires it.
Treat a clean pen test as a snapshot, not a certificate. It means those testers, with that scope, didn't find a way in during that window.
Close the loop. After every pen test, update the threat model with what you learned, so the next design starts smarter.

A note on honesty: the figures above are averages and large-sample statistics, not predictions for your organization. IBM's cost numbers come from breached organizations in its study, and the Verizon figures come from its dataset of confirmed breaches. The Capital One chain is well documented, but whether a specific practice would have stopped it is a counterfactual nobody can prove.

Common Mistakes When Learning Security Testing

Calling a vulnerability scan a penetration test. A scan lists known issues automatically. A pen test has a human chaining weaknesses together and proving impact.
Threat modeling once and never again. A model that doesn't change when the system changes is a historical document.
Testing without permission. Only test systems you own or have explicit written authorization to test. Unauthorized testing can be illegal, even with good intentions.
Fixing only the symptom. If a pen test finds SSRF, fix the root causes too: outbound controls, metadata protections, and excessive permissions.
Skipping the scope conversation. Vague scope leads to either missed risk or accidental damage. Write down what's in, what's out, and when testing is allowed.
Ignoring third parties. IBM's India data listed third-party and supply chain compromise among the top causes of breaches, at 17%, and your own tests won't cover vendors unless you scope them in.

Frequently Asked Questions

Do I need both threat modeling and penetration testing?

For anything that matters, yes. They answer different questions at different times. If resources are tight, start with threat modeling because it's cheaper and sharpens everything after it.

Is a vulnerability scan the same as a penetration test?

No. A scan is automated and finds known issues. A penetration test uses human judgment to chain weaknesses, test logic flaws, and demonstrate real impact.

How often should we pen test?

It depends on how fast the system changes and on any requirements that apply to you. Common triggers are an annual test, a major release, or a significant architecture change. Check the specific standards you're subject to.

Can a beginner do threat modeling?

Yes. Draw the system, mark where data crosses trust boundaries, and ask what could go wrong at each point using STRIDE as a checklist. You'll improve with practice and with every real incident you study.

Which is better for starting a security career?

Both are valuable. Threat modeling fits application security and engineering paths, while penetration testing is a more specialized offensive track. Learning both makes you better at either.

What is STRIDE?

A mnemonic for six threat categories: spoofing, tampering, repudiation, information disclosure, denial of service, and elevation of privilege. It helps teams avoid forgetting whole classes of threats when reviewing a design.

Build Both Skills, in the Right Order

The Capital One story isn't a case for one practice over the other. It's a reminder that real breaches chain together design decisions and implementation mistakes, and that you need both kinds of scrutiny to see them. Explore the Threat Modeling and Penetration Testing courses to learn how to ask the design questions first, then test the answers against the real system.

GitHub Actions vs. Jenkins vs. CircleCI: How FreeAgent Cut Its Test Infrastructure Spend by 90%

Ciphemic academia — Fri, 09 Oct 2026 01:25:21 +0000

GitHub Actions vs. Jenkins vs. CircleCI: How FreeAgent Cut Its Test Infrastructure Spend by 90%

FreeAgent, the accounting software company, moved its CI/CD workflows off Jenkins and onto GitHub Actions, and reported a 90% reduction in its monthly infrastructure spend for running tests. Nylas made the same Jenkins-to-Actions move and cut its deployment time by 30%, then started seeing whole weeks with zero build failures. Neither team changed what they were building. They changed the machine that builds it, and the difference showed up on the invoice and in the engineers' week.

Then there's the other side of choosing a CI platform, the one nobody puts in a benchmark. In January 2023, CircleCI told every customer to rotate every secret stored on the platform, after attackers got access to customer secrets. Open-source project conda-forge, which had stopped using CircleCI but never removed its tokens, ended up with an outage of about five days while it rotated credentials across all its CI services. A CI tool isn't just a build runner. It's a place where your most sensitive keys live, which makes this choice a security decision as much as a speed decision.

This guide compares GitHub Actions, Jenkins, and CircleCI honestly: what each one actually is, where the real numbers come from, what each one costs you beyond the invoice, and how to pick one without cargo-culting whatever your last employer used. If CI/CD concepts are still new, start with our CI/CD Pipelines roadmap, then come back here.

This post originally appeared on the Ciphemic Academia blog.

The Short Version

GitHub Actions is CI/CD built into GitHub itself, configured in YAML files inside your repository, with a huge marketplace of reusable actions.
Jenkins is a self-hosted, open-source automation server with a massive plugin ecosystem and total control, plus all the maintenance that comes with running it yourself.
CircleCI is a dedicated, managed CI/CD service known for fast pipelines and strong configuration tooling, separate from wherever your code lives.

If you want one default: if your code is on GitHub, learn GitHub Actions first. It has the lowest setup cost, and the concepts (triggers, jobs, steps, runners, secrets) carry straight over to the other two. Reach for Jenkins when you need full control, self-hosting, or you're inheriting it. Reach for CircleCI when you want a dedicated managed CI service and accept that your secrets live with a third party.

What All Three Are Actually Trying to Solve

Before the differences, the shared job, since this is what you're really learning:

Running your tests automatically on every push or pull request, so broken code gets caught before it merges
Building your application the same way every time, removing "works on my machine" from the conversation
Deploying through a repeatable process instead of a person remembering the right commands in the right order
Handling secrets such as API keys and cloud credentials without hardcoding them in the repository
Giving fast, visible feedback so a developer knows within minutes whether their change is safe

If these ideas are new, our post on CI/CD pipelines vs. manual deployment covers why this automation exists at all. Learn these ideas in any one tool and most of the skill transfers to the others.

GitHub Actions

What it actually is: a CI/CD system built directly into GitHub. You describe a workflow in a YAML file stored in .github/workflows/, and GitHub runs it on its own hosted machines (called runners) whenever an event you choose happens, such as a push or a pull request. You can also run jobs on your own self-hosted runners.

A small example (test a Node.js project on every push):

name: CI
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - run: npm test

Where it shines:

Zero extra infrastructure: if your code already lives on GitHub, there's nothing to install, host, or sign up for separately
Readable configuration: FreeAgent's engineers specifically cited that YAML workflows were easier for their developers to learn than Jenkins pipelines written in Groovy
Huge reusable ecosystem: the Actions marketplace covers most common tasks without writing scripts from scratch, and GitHub reports its platform runs roughly 71 million jobs a day
Tight GitHub integration: pull request checks, status badges, code scanning, and deployments all connect natively
Free for public repositories: open-source projects get CI at no cost, which is why so many of them use it

Where it struggles:

Tied to GitHub: your pipelines only make sense if your code lives there, which makes switching platforms later a real migration
Third-party actions are code you run with your secrets: every action you pull from the marketplace executes inside your workflow, so pinning versions and reviewing what you use matters
Pricing has been in flux: GitHub cut hosted-runner prices by up to 39% in January 2026, and announced then postponed a per-minute fee for self-hosted runners, so check the current pricing page before budgeting

Who this suits: teams and learners with code on GitHub, open-source projects, and anyone who wants the shortest path from zero to a working pipeline.

Jenkins

What it actually is: an open-source automation server you install and run yourself, on your own servers or cloud machines. Pipelines are typically written as a Jenkinsfile in Groovy, and Jenkins extends through plugins for nearly everything: source control systems, build tools, cloud providers, notifications.

A small example (the same Node.js pipeline, as a Jenkinsfile):

pipeline {
  agent any
  stages {
    stage('Install') {
      steps { sh 'npm ci' }
    }
    stage('Test') {
      steps { sh 'npm test' }
    }
  }
}

Where it shines:

Total control: you decide where it runs, how it scales, what network it can reach, and who can access it, which matters in regulated or air-gapped environments
Works with almost anything: it isn't tied to any one code host, and its plugin ecosystem is among the largest in the CI world
No per-minute platform fee: the software is free, so your cost is the machines and the people who run them
Deep customization: complex, unusual build flows that hosted services struggle with are often possible in Jenkins

Where it struggles:

You own the maintenance: FreeAgent's write-up describes manually spotting problem build agents from logs and removing them by hand, and keeping a wiki of manual changes in case they needed to roll back
Plugins are a double-edged sword: every plugin is something to update, secure, and keep compatible with the others
Groovy pipelines have a steeper learning curve than plain YAML for many developers
Infrastructure cost is real: FreeAgent's 90% drop in test infrastructure spend after leaving Jenkins is a reminder that always-on servers and build agents cost money, even when the software is free

Who this suits: organizations needing self-hosting or full control, teams with existing Jenkins investment and the people to run it, and anyone working in environments where SaaS CI isn't allowed.

CircleCI

What it actually is: a managed CI/CD platform that connects to your code host (such as GitHub or Bitbucket) and runs your pipelines on its infrastructure. You configure it in a YAML file at .circleci/config.yml, and reusable packages of configuration called orbs let you share setup across projects.

A small example (the same pipeline, as a CircleCI config):

version: 2.1

jobs:
  test:
    docker:
      - image: cimg/node:20.11
    steps:
      - checkout
      - run: npm ci
      - run: npm test

workflows:
  main:
    jobs:
      - test

Where it shines:

Built for pipeline speed: features like caching, parallelism, and test splitting are core to the product, not afterthoughts
Dedicated CI focus: it does CI/CD and little else, with tooling like SSH access into failed builds for debugging
Reusable configuration: orbs let teams share tested pipeline building blocks instead of copy-pasting YAML
Works across code hosts: useful when your repositories aren't all in one place

Where it struggles:

Your secrets live with a third party: the January 2023 incident showed what that means in practice, with customers told to rotate every secret, including cloud credentials and tokens stored on the platform
Another vendor and another bill: unlike Actions, it's a separate service to sign up for, secure, and budget
Smaller ecosystem than Actions: fewer ready-made integrations than GitHub's marketplace for GitHub-centric teams
Lingering trust questions after the incident: CircleCI published a detailed incident report and says customers can safely build on the platform, but each team has to decide for itself how much it weighs that history

Who this suits: teams that want a dedicated managed CI service, care about pipeline speed features, and have a clear policy for how secrets are stored and rotated.

Side-by-Side Comparison

	GitHub Actions	Jenkins	CircleCI
Hosting	Managed by GitHub (self-hosted runners optional)	Self-hosted by you	Managed by CircleCI
Config format	YAML in `.github/workflows/`	`Jenkinsfile` (Groovy)	YAML in `.circleci/config.yml`
Setup effort	Lowest, if code is on GitHub	Highest, you install and run it	Low to moderate
Ecosystem	Actions marketplace	Very large plugin ecosystem	Orbs
Maintenance burden	Low	High	Low
Cost model	Free for public repos, usage-based for private	Free software, you pay for servers and people	Paid plans beyond a free tier
Secrets live	With GitHub	On your own infrastructure	With CircleCI
Best for	GitHub-based teams, open source	Control, on-prem, regulated environments	Teams wanting dedicated managed CI

How This Fits a Career Path

DevOps or platform engineer: you'll likely meet all three, so learn the shared concepts first; our CI/CD Pipelines roadmap is built around those fundamentals
Backend or full-stack engineer: GitHub Actions is usually the first one you'll touch, since it sits right next to your code
Engineer at a large or regulated company: expect Jenkins, either as legacy you maintain or as a deliberate self-hosting choice
DevSecOps roles: how secrets are stored, scoped, and rotated across CI tools is central, and the CircleCI incident is a standard case study; see our DevSecOps career guide for how this fits into security work
System design interviews: be ready to explain how you'd design a pipeline and justify the platform, including where secrets live and what happens if the CI provider is compromised

How to Choose Without Overthinking It

Start with where your code lives. On GitHub? Actions is the path of least resistance. Spread across hosts, or on-premises? Jenkins or CircleCI become more attractive.
Count the people, not just the dollars. Jenkins is free software, but someone has to patch it, scale its agents, and babysit its plugins. FreeAgent's savings came from removing exactly that overhead.
Decide where secrets are allowed to live. If your policy says credentials must never sit on a third-party SaaS, that rules out hosted CI for sensitive deployments and pushes you to self-hosted runners or Jenkins.
Use short-lived credentials where you can. Whatever platform you pick, prefer short-lived, narrowly scoped tokens over long-lived keys pasted into a settings page, so a platform incident has a smaller blast radius.
Don't wait for the perfect choice. The core ideas transfer, so building one real pipeline in any tool beats comparing all three for a month.

A note on honesty: FreeAgent's 90% and Nylas's 30% are real results from real teams, but they come from specific setups, with specific starting points (FreeAgent's earlier Jenkins setup had its own spot-instance and agent-management overhead). They show what's possible, not what you'll get. GitHub Actions pricing is also actively changing, so check current numbers on GitHub's own pricing page before you budget around anything in this post, including the figures above.

Common Mistakes When Learning CI/CD Tools

Learning one tool's syntax without learning the concepts. Triggers, jobs, caching, secrets, and artifacts exist in all three. Learn those, and switching tools takes days instead of weeks.
Leaving secrets behind in tools you've stopped using. conda-forge had deprecated CircleCI but never removed its tokens, which is a big part of why the 2023 incident hurt. When you retire a CI service, delete the secrets stored in it.
Using unpinned third-party actions or plugins. Anything you pull in runs with access to your pipeline. Pin versions and review what you depend on.
Putting long-lived cloud keys in CI settings. Short-lived credentials limit the damage when something goes wrong.
Choosing Jenkins "because it's free" without budgeting for maintenance. Free software with a part-time operator is rarely free.
Treating the CI platform choice as permanent. Both FreeAgent and Nylas migrated, and the pipeline logic you write is more portable than it feels.

Frequently Asked Questions

Should a beginner learn GitHub Actions, Jenkins, or CircleCI first?

GitHub Actions, for most beginners. If your projects are on GitHub, you can have a working pipeline in minutes with nothing to install, and the concepts transfer directly to the others.

Is Jenkins still worth learning?

Yes, especially for enterprise, on-premises, and regulated environments, where it remains common. You may well inherit a Jenkins setup in a real job even if you'd never choose it for a new project.

Did the CircleCI security incident make it unsafe to use?

CircleCI published a detailed incident report, revoked tokens, and states that customers can build safely on the platform. Whether that's enough is a risk decision for your team. The lasting lesson applies to every hosted CI tool: minimize what secrets live there, scope them tightly, and rotate them.

Is GitHub Actions really free?

It's free for public repositories, and private repositories get a monthly allowance of free minutes before usage-based billing applies. GitHub changed its pricing in 2026 and has postponed a planned fee for self-hosted runners, so check the current pricing page for exact numbers.

Can I use more than one CI tool?

Yes, and many organizations do, often because of acquisitions or gradual migrations. FreeAgent's own migration was a deliberate move from one to another, not an overnight swap.

How do interviewers evaluate this topic?

Rarely by asking which tool you prefer. They tend to ask you to design a pipeline: what runs on pull requests, what gates a deploy, where secrets live, and how you'd roll back. Explaining those decisions clearly matters more than any specific tool's syntax.

Pick One and Build a Real Pipeline

The honest lesson from FreeAgent's 90% and the 2023 CircleCI incident is the same: the CI platform you choose affects your costs, your engineers' time, and your security posture, not just your build speed. Pick one, build a real pipeline for a real project, and make sure you know exactly where your secrets live. Our free CI/CD Pipelines roadmap walks you through building one end to end, and you can explore the courses for the cloud and DevOps foundation underneath it.

Kafka vs. RabbitMQ vs. SQS: The Message Queue That Started at 400,000 Messages a Second

Ciphemic academia — Wed, 07 Oct 2026 01:48:41 +0000

Kafka vs. RabbitMQ vs. SQS: The Message Queue That Started at 400,000 Messages a Second

Kafka exists because RabbitMQ couldn't keep up. LinkedIn's engineers, back when the company was building the activity feed and metrics pipeline behind its own platform, found their existing message broker setup topping out two orders of magnitude below what they actually needed, and built Kafka specifically to publish more than 400,000 messages a second. Over a decade later, the gap is still real and still measured: a 2024 Confluent benchmark clocked Kafka sustaining roughly 605,000 messages per second on a three-broker cluster, while a widely cited 2026 comparison puts classic RabbitMQ queues around 40,000 to 50,000 messages per second per node, with RabbitMQ's newer Streams feature closing much of that gap on tuned clusters.

None of that makes RabbitMQ a worse technology, it makes it a different one, solving a different problem than raw throughput. And neither answers the third option most backend teams actually reach for first: Amazon SQS, a fully managed queue that trades control for zero operational overhead. This guide compares all three honestly, what each one actually optimizes for, where these real numbers come from, and how to decide before you've built a message queue architecture around the wrong one.

If messaging and distributed systems fundamentals are still new ground, our System Design course covers what this comparison builds on.

This post originally appeared on the Ciphemic Academia blog.

The Short Version

Kafka is a distributed, append-only log built for extremely high throughput and replayable event streams, at the cost of real operational complexity.
RabbitMQ is a traditional message broker built for flexible routing and reliable per-message delivery, simpler to run at smaller scale, with lower raw throughput in its classic queue mode.
SQS is AWS's fully managed queue service, built to remove operational overhead entirely, at the cost of some control and features the other two offer natively.

If you want one default: start with SQS if you're on AWS and want to avoid running infrastructure. Reach for RabbitMQ when you need flexible routing and don't yet have Kafka-scale throughput needs. Reach for Kafka specifically once you have a measured, real need for high-throughput event streaming or replay, the way LinkedIn did.

What All Three Are Actually Trying to Solve

Before the differences, the shared job, since this is what any message queue exists to do:

Decoupling producers from consumers: a service that sends a message shouldn't need to know or care which service, or how many, eventually processes it
Absorbing load spikes: a queue lets a slow consumer catch up without the producer blocking or failing
Guaranteeing delivery: messages shouldn't vanish if a consumer crashes mid-processing
Enabling asynchronous work: background jobs, notifications, and cross-service communication all depend on not forcing everything to happen synchronously

The real difference is architecture: a log you can replay, a broker that routes intelligently, or a managed queue you don't have to think about at all, and that architectural choice is exactly what LinkedIn's 400,000-messages-a-second number and RabbitMQ's per-node ceiling both trace back to.

Kafka

What it actually is: a distributed, append-only log, split into partitions, where producers write messages and consumers read them at their own pace by tracking an offset, a position in the log. Unlike a traditional queue, messages aren't removed once consumed; they stay for a configured retention period, which means multiple independent consumers can replay the same stream from different points.

A small example (producing and consuming, conceptually):

producer.send("orders-topic", orderEvent)

# consumer tracks its own offset in the partition
consumer.subscribe("orders-topic")
for message in consumer.poll():
    process(message)
    consumer.commit_offset()

Where it shines:

Extremely high throughput: this is the entire reason Kafka exists, LinkedIn's original 400,000-messages-a-second figure and Confluent's more recent 605,000-messages-per-second benchmark both measure exactly this strength
Replayable event log: since messages persist for a retention window instead of disappearing on consumption, multiple services (billing, analytics, fraud detection) can independently read the same event stream without copying data between them
Strong ordering guarantees per partition: messages within a single partition are strictly ordered, which matters for event sourcing and audit-style use cases
Built for streaming, not just queuing: Kafka's ecosystem (Kafka Streams, Kafka Connect) extends naturally into real-time stream processing, not just message passing

Where it struggles:

Real operational complexity: running Kafka well requires genuine expertise, partition management, broker tuning, and historically JVM-specific operational knowledge, a cost one team's own migration write-up specifically flagged as a reason they chose RabbitMQ Streams instead
Higher latency under light load: Kafka trades per-message latency for batch throughput; published comparisons consistently show RabbitMQ winning on raw per-message latency at lower volumes, precisely because Kafka's batching exists to serve throughput, not speed per message
Operational floor is higher: a single RabbitMQ node is a legitimate production deployment for a small service; Kafka's minimum viable setup for real resilience typically starts at multiple brokers

Who this suits: systems with a genuine, measured need for very high throughput or event replay, exactly the problem LinkedIn built Kafka to solve, and increasingly common at companies running real-time analytics, activity feeds, or event-sourced architectures.

RabbitMQ

What it actually is: a traditional message broker built on the AMQP protocol, where producers publish messages to exchanges, which route them to queues based on configurable rules, and consumers receive and acknowledge each message individually. Once a message is acknowledged, it's removed from the queue, a fundamentally different model from Kafka's persistent log.

A small example (publishing with routing, conceptually):

channel.exchange_declare(exchange='orders', exchange_type='topic')
channel.basic_publish(exchange='orders', routing_key='order.created', body=order_json)

# a specific queue only receives messages matching its binding
channel.queue_bind(exchange='orders', queue='billing_queue', routing_key='order.created')

Where it shines:

Flexible, powerful routing: AMQP's exchange types (direct, topic, fanout) let you route messages to exactly the right consumers based on real routing logic, something Kafka's simpler topic-and-partition model doesn't natively offer
Lower operational floor: a single, well-configured RabbitMQ node is a genuinely reasonable production setup for smaller services, with a built-in management UI that lowers the barrier to running it confidently
Strong per-message delivery guarantees: RabbitMQ tracks and acknowledges every individual message, which suits workloads where guaranteed, precise delivery matters more than raw volume
Closing the throughput gap where it matters: RabbitMQ's newer Streams feature, a Kafka-like append-only log mode, reaches into the million-messages-per-second range on tuned clusters, narrowing a gap that used to be far more one-sided

Where it struggles:

Lower throughput in classic queue mode: the widely cited 40,000-to-50,000-messages-per-second-per-node figure for classic queues is real, and it's precisely the ceiling that pushed companies like the AppDirect engineering team to replace RabbitMQ with Kafka once their data volume outgrew it
No native long-term replay in classic mode: once a message is consumed and acknowledged in a standard queue, it's gone, so replaying historical events requires the newer Streams feature specifically, not RabbitMQ's original model
Scaling requires more deliberate work at very high volume: RabbitMQ can scale, but matching Kafka's highest-throughput numbers requires specific tuning and the Streams feature, not the default configuration

Who this suits: systems that need flexible, rule-based routing, moderate-to-high throughput workloads, and teams who want a lower operational floor than Kafka while keeping more control than a fully managed service offers.

SQS

What it actually is: Amazon's fully managed queue service. There's no broker to run, no cluster to tune, no servers to patch, you create a queue through the AWS console or API, and AWS handles the infrastructure entirely. SQS offers two queue types: Standard (near-unlimited throughput, at-least-once delivery, no strict ordering) and FIFO (strict ordering and exactly-once processing, with a lower throughput ceiling).

A small example (sending and receiving, conceptually):

sqs.send_message(QueueUrl=queue_url, MessageBody=json.dumps(order))

messages = sqs.receive_message(QueueUrl=queue_url, MaxNumberOfMessages=10)
for msg in messages['Messages']:
    process(msg)
    sqs.delete_message(QueueUrl=queue_url, ReceiptHandle=msg['ReceiptHandle'])

Where it shines:

Zero operational overhead: no servers, no cluster management, no capacity planning, which removes an entire category of engineering work that both Kafka and RabbitMQ require
Scales automatically: Standard queues handle very high throughput without any manual tuning or provisioning on your part, AWS absorbs that scaling work entirely
Tight AWS ecosystem integration: native integration with Lambda, SNS, and the rest of AWS's services means less custom plumbing for teams already building on AWS
Pay-per-use pricing: you pay for requests made, not for idle broker capacity sitting unused, which can be genuinely cheaper for bursty or moderate-volume workloads

Where it struggles:

Less architectural flexibility: no custom routing logic like RabbitMQ's exchanges, and no replayable log like Kafka's, messages are simply consumed and gone, closer to RabbitMQ's model than Kafka's
FIFO queues trade throughput for ordering: SQS FIFO queues cap out at a few thousand messages per second without batching, a real ceiling compared to Kafka's partition-based throughput
Less control over delivery semantics: you're working within AWS's model, which is simpler but genuinely less tunable than self-managing Kafka or RabbitMQ's finer-grained configuration options
Vendor lock-in: SQS is AWS-specific, so a multi-cloud or cloud-agnostic architecture needs a different answer

Who this suits: teams already on AWS who want to avoid running message infrastructure entirely, moderate-throughput workloads without complex routing needs, and smaller teams where operational simplicity outweighs fine-grained control.

Side-by-Side Comparison

	Kafka	RabbitMQ	SQS
Model	Distributed, persistent log	Traditional broker with routing	Fully managed queue
Peak throughput (published figures)	~605K msgs/sec (3-broker benchmark)	~40–50K msgs/sec/node (classic queues)	Near-unlimited (Standard), lower for FIFO
Message replay	Yes, native and core to the design	No (classic queues); yes via Streams	No
Routing flexibility	Basic (topics, partitions)	Strong (exchanges, routing keys)	Basic
Operational overhead	High, real infrastructure to run	Moderate, lower floor than Kafka	None, fully managed
Ordering guarantees	Per-partition	Per-queue, configurable	FIFO queues only, with throughput trade-off
Best for	High-throughput event streaming, replay	Flexible routing, moderate-scale reliability	Zero-ops simplicity on AWS

How This Fits a Career Path

Backend Engineer (general): understanding when and why a system needs a message queue at all, before picking a specific one, is the more foundational skill most interviews actually probe for
Platform or infrastructure engineer: Kafka expertise is close to a standard expectation for high-scale, event-driven architectures; our CI/CD Pipelines course covers the surrounding infrastructure skills this role assumes
Cloud-focused or AWS-heavy roles: SQS fluency, including when to reach for it instead of self-managed infrastructure, is directly relevant and frequently tested
System design interviews: be ready to justify a queue choice using the same reasoning LinkedIn, AppDirect, and the RabbitMQ Streams teams above actually used, a specific, measured throughput or routing requirement, not a default preference

How to Choose Without Overthinking It

Start with your actual throughput requirement, measured, not guessed. If you're nowhere near tens of thousands of messages per second, the gap between these three matters far less than other factors like operational cost and team familiarity.
If you're on AWS and want to avoid running infrastructure, start with SQS. It removes an entire category of operational work, and for most moderate-throughput backend workloads, it's genuinely sufficient.
Reach for RabbitMQ when routing logic matters. If your system needs messages intelligently routed to different consumers based on content or type, RabbitMQ's exchange model does real work SQS and Kafka don't offer as naturally.
Reach for Kafka only once you have Kafka-scale problems. LinkedIn built it because RabbitMQ topped out two orders of magnitude below their actual need. If you're not near that kind of volume, or you don't need event replay, you're very likely taking on real operational complexity for a problem you don't have yet.

A note on honesty: the specific throughput numbers above come from different benchmarks, run on different hardware, at different points in time, LinkedIn's original figure is now over a decade old, and RabbitMQ's Streams feature has closed much of the historical gap since. Treat these as evidence that a real difference exists, not as numbers that will reproduce exactly on your own infrastructure. Benchmark your actual workload before committing.

Common Mistakes When Learning Messaging Systems

Reaching for Kafka by default because it's the most talked-about option. Its operational complexity is real and needs to be paying for a genuine throughput or replay requirement, not just reputation.
Underestimating SQS because it's "just AWS." For a large share of real backend workloads, SQS's zero-operational-overhead model is a genuinely better fit than self-managing a broker.
Ignoring RabbitMQ's routing strength. Teams focused purely on throughput numbers sometimes miss that RabbitMQ solves a routing problem the other two don't address as naturally.
Assuming message delivery guarantees are the same across all three. At-least-once, exactly-once, and ordering guarantees differ meaningfully between Kafka, RabbitMQ, and SQS's Standard versus FIFO queues; read the actual guarantees, don't assume.
Migrating before measuring. AppDirect's own migration write-up describes a careful, gradual, dual-pipeline rollout, not a sudden cutover, precisely because getting this decision wrong under production load is expensive to undo.

Frequently Asked Questions

Should a beginner learn Kafka, RabbitMQ, or SQS first?

SQS, if you're working within AWS, since it requires the least infrastructure knowledge to get started and still teaches core queueing concepts. RabbitMQ is a strong second step for understanding routing-based messaging, with Kafka's log-based model as a distinct third concept once the fundamentals are solid.

Is Kafka always faster than RabbitMQ?

In raw sustained throughput, consistently yes, which is the entire reason Kafka was built. But RabbitMQ has historically posted lower per-message latency at smaller scale, and its newer Streams feature narrows the throughput gap considerably on tuned clusters.

Can SQS replace Kafka for event streaming?

Not directly. SQS doesn't offer Kafka's persistent, replayable log model; once a message is processed, it's gone. For genuine event-sourcing or multi-consumer replay needs, Kafka (or RabbitMQ Streams) solves a problem SQS isn't designed for.

Why did LinkedIn build Kafka instead of just scaling RabbitMQ?

Because the throughput gap they measured was described as two orders of magnitude, not a tuning problem solvable within their existing broker's architecture. That's a genuinely different scale of need than most teams will ever hit, which is worth remembering before assuming you need the same solution.

Does RabbitMQ Streams make Kafka unnecessary?

Not entirely. Streams narrows the throughput gap significantly, but Kafka's broader ecosystem (Kafka Connect, Kafka Streams, its maturity at extreme scale) still gives it real advantages for teams already investing deeply in event-driven architecture.

How do interviewers evaluate this topic?

Usually by asking you to design a system and justify your queue choice for its specific requirements, throughput, ordering, routing, operational cost, rather than testing for a fixed favorite. Being able to explain why LinkedIn needed Kafka, and why most systems don't need Kafka's exact scale, signals real understanding.

Measure Your Actual Throughput Before You Choose

The real lesson from LinkedIn's 400,000-messages-a-second origin story isn't "always pick Kafka," it's that the decision followed a specific, measured bottleneck their existing broker genuinely couldn't handle. Explore the System Design course to build the skills that let you make that same evidence-based call on your own architecture.

Passwords vs. Passkeys vs. MFA: How CVS Health Cut Account Takeover Fraud by 98%

Ciphemic academia — Wed, 07 Oct 2026 01:28:18 +0000

Passwords vs. Passkeys vs. MFA: How CVS Health Cut Account Takeover Fraud by 98%

CVS Health rolled out passkeys across its platform and measured a 98% reduction in account takeover fraud. Around the same time, Microsoft ran a large-scale measurement study across Azure Active Directory accounts and found that simply turning on multi-factor authentication reduced the risk of account compromise by 99.22%, even when the user's password had already leaked. Two different companies, two different authentication strategies, both converging on the same conclusion: whatever you're doing with passwords alone, it isn't enough, and the fix isn't a stronger password policy.

These numbers point at a genuine shift happening in how backend and security engineers think about authentication. Passwords aren't disappearing overnight, but understanding exactly what a password alone can't protect against, and what passkeys and MFA each add on top of it, is quickly becoming a baseline expectation for anyone building or securing a login flow. This guide breaks down what each approach actually does, where these real numbers come from, and how they fit together, because the honest answer here isn't "pick one."

If auth fundamentals like hashing and tokens are still new ground, our Auth & Identity course covers what this comparison builds on.

This post originally appeared on the Ciphemic Academia blog.

The Short Version

Passwords are a shared secret the user memorizes and the server verifies, typically against a hashed value. Simple, universal, and the weakest link in nearly every account takeover.
Passkeys are a cryptographic key pair, generated and stored on the user's device, that replace the password entirely using the FIDO2/WebAuthn standard, phishing-resistant by design.
MFA (multi-factor authentication) adds a second proof of identity on top of whatever the first factor is, typically a code, a push notification, or a hardware key, specifically to catch the cases where the first factor alone fails.

If you want one default: don't treat this as a single choice. Passkeys are the strongest long-term direction for the first factor itself. MFA remains essential wherever passwords are still in use, and ideally even alongside passkeys for your most sensitive accounts.

What All Three Are Actually Trying to Solve

Before the differences, the shared goal, since it's easy to lose in a "passwords vs. passkeys" framing:

Proving identity: every approach exists to answer "is this really the account owner?" before granting access
Resisting theft: a credential that can be stolen, guessed, or phished is a real liability, regardless of how it's implemented
Staying usable: an authentication method nobody can actually use correctly gets worked around, which is its own security failure
Scaling to real-world failure modes: lost devices, forgotten credentials, and account recovery all need real answers, not just a clean happy path

The actual decision isn't which one is "most secure" in the abstract. It's understanding exactly which attack each one blocks, and which ones it doesn't, which is exactly what both CVS Health's and Microsoft's numbers measured directly.

Passwords

What it actually is: a shared secret, chosen by the user, verified by the server against a stored hash. The user types it in, the server hashes the input and compares it to the stored hash, and access is granted on a match. No device, hardware key, or secondary factor is required by the method itself.

A small example (what the server stores, simplified):

user: alex@example.com
password_hash: $2b$12$KIXxPfn...  (bcrypt hash, never the plain password)

Where it shines:

Universal support: every system, every platform, and every user already understands how passwords work, with zero new infrastructure required
Simple to implement at a basic level: a working password system is the lowest-effort way to add authentication to a new application
No special hardware or device dependency: a password can be typed anywhere, on any device, without relying on a specific phone or key being present

Where it struggles:

Phishable by design: a password can be typed into a fake login page just as easily as a real one, which is exactly the attack vector passkeys were built to eliminate
Reused across services: when one service leaks a password database, the same password often works on other services, turning a single breach into many
Guessable or crackable: weak or common passwords remain a major cause of account compromise, despite decades of password-strength guidance
The clearest, most direct evidence against passwords alone: Microsoft's own Azure AD study found that accounts without MFA were compromised at a dramatically higher rate than those with it, even when the underlying password had already leaked, which is specifically a statement about the password's weakness as a standalone defense

Who this suits: nowhere, as a standalone, complete authentication strategy for anything that matters. Passwords remain a practical fallback or first factor, but not a sufficient defense on their own.

Passkeys

What it actually is: a cryptographic credential, built on the FIDO2/WebAuthn standard, where a device generates a public-private key pair during registration. The private key never leaves the user's device (or a synced keychain), and the server only ever stores the public key. Logging in means the device cryptographically proves it holds the private key, typically unlocked by a fingerprint, face scan, or device PIN, with nothing transmitted that a phishing site could steal and reuse.

A small example (the conceptual flow):

1. Registration: device generates a key pair, sends only the public key to the server
2. Login: server sends a challenge
3. Device signs the challenge with the private key (unlocked via Face ID/fingerprint)
4. Server verifies the signature using the stored public key, no secret ever crosses the network

Where it shines:

Phishing-resistant by design: since there's no shared secret being typed in, a fake login page has nothing to steal, which is the core reason CVS Health's rollout drove such a large drop in account takeover fraud
Nothing to remember or reuse: there's no password to forget, reuse across services, or leak in a breach, since the private key never leaves the device
Real, measured adoption gains: Amazon reported 175 million passkeys created after rolling them out platform-wide, alongside a 30% improvement in sign-in success rates, a genuine usability win, not just a security one
Reduces support burden: the State of Michigan's MiLogin system reported 1,300 fewer support calls after switching users to passkeys, since there's no password to reset

Where it struggles:

Device and ecosystem dependency: losing access to the device or account holding your passkeys can complicate recovery, and synced passkeys require trusting the cloud provider syncing them
Enterprise rollout is still genuinely slow: account recovery flows and legacy systems that assume passwords remain real blockers to full adoption, even where the technology itself is ready
Not universally supported yet: while adoption is accelerating quickly, not every service or platform supports passkeys, so most users and systems still need a password-based fallback for now

Who this suits: any new login system being designed today, and a high-value upgrade path for existing systems protecting sensitive accounts, exactly the kind of account takeover problem CVS Health's rollout was built to solve.

MFA (Multi-Factor Authentication)

What it actually is: a second (or third) proof of identity required alongside the first factor, typically something you have (a phone, a hardware key) or something you are (biometrics), layered on top of something you know (a password) or something you hold (a passkey). Common implementations include authenticator app codes (TOTP), push notifications, SMS codes, or physical security keys.

A small example (a typical login sequence):

1. User enters password (first factor)
2. Server accepts the password and prompts for a second factor
3. User approves a push notification or enters a 6-digit code (second factor)
4. Access granted only after both factors succeed

Where it shines:

Dramatic, well-measured risk reduction: Microsoft's Azure AD study found MFA reduced account compromise risk by 99.22% overall, and more than 99.99% of MFA-enabled accounts stayed secure over the study period
Catches exactly what passwords alone miss: even when a password has already leaked, MFA reduced compromise risk by 98.56% in the same study, directly addressing the password's core weakness
Flexible across authentication methods: MFA can layer on top of either passwords or passkeys, making it a genuinely complementary strategy rather than a competing one
Widely adopted and increasingly expected: workforce MFA adoption reached roughly 70% by early 2025, according to identity platform telemetry, up from 64% the year before, and continues climbing

Where it struggles:

Not all MFA is equally strong: the same research found authenticator apps outperforming SMS-based codes specifically, since SMS remains vulnerable to SIM-swapping and interception in ways app-based or hardware-key factors aren't
Real usability friction: a second step at every login is genuine friction, and "MFA fatigue," users approving push notifications reflexively without checking them, is a documented, real attack vector
Doesn't eliminate the weak first factor: MFA on top of a weak, reused password is far better than a password alone, but it's still not as strong as removing the phishable first factor entirely, which is exactly what passkeys do

Who this suits: essentially every account that matters, as a baseline layered on top of whatever first factor you're using, password or passkey alike.

Side-by-Side Comparison

	Passwords	Passkeys	MFA
What it is	A memorized shared secret	A device-held cryptographic key pair	A required second proof of identity
Phishing resistance	None	Strong, by design	Depends on the method (app/hardware key strong, SMS weaker)
Measured real-world impact	N/A, the baseline being improved on	98% reduction in account takeover fraud (CVS Health)	99.22% reduction in compromise risk (Microsoft)
User experience	Familiar, but prone to reuse and forgetting	Fast, biometric-based, nothing to remember	Adds a step, but increasingly expected
Adoption maturity	Universal	Growing quickly, not yet everywhere	Mainstream, roughly 70% workforce adoption
Best role today	Fallback only, not a standalone defense	Primary credential for new and high-value systems	Required layer on top of either of the above

How This Fits a Career Path

Backend Engineer (general): implementing secure password handling (proper hashing, never storing plaintext) and understanding why it's insufficient alone is a baseline expectation in nearly every real codebase
Security-focused or Auth-focused roles: passkey implementation (WebAuthn) and MFA design are increasingly central, not optional extras; our Cryptography course covers the underlying mechanisms that make passkeys phishing-resistant
Full-stack or product engineer: you'll likely be the one implementing the actual login flow, so understanding the real usability trade-offs (Amazon's 30% sign-in improvement, MiLogin's support-call reduction) matters directly for product decisions
System design interviews: be ready to design a login flow that layers these appropriately, not just name-drop all three, since interviewers are specifically testing whether you understand what each layer actually defends against

How to Choose Without Overthinking It

Never ship passwords as a standalone defense for anything sensitive. The evidence here isn't subtle, Microsoft's own measurement found MFA cutting compromise risk by over 99%, specifically because passwords alone fail so often.
Add passkeys as the primary option wherever you can. For new systems, offer passkeys first and relegate passwords to a fallback. For existing systems, a phased passkey rollout targeting your highest-risk accounts first mirrors how real companies like CVS Health approached it.
Layer MFA regardless of your first factor. Even with passkeys, MFA remains valuable for especially sensitive actions (admin access, financial transactions), and it's essential wherever a password-based fallback still exists.
Prefer app-based or hardware-key MFA over SMS when you can. The research is specific here: phishing-resistant methods meaningfully outperform SMS, which remains vulnerable to interception and SIM-swapping.

A note on honesty: CVS Health's 98% and Microsoft's 99.22% are both real, measured numbers, but they come from different studies, different systems, and different threat models. They aren't directly comparable as a head-to-head "passkeys vs. MFA" scorecard. What they both demonstrate, independently, is the same underlying point: moving beyond a password alone produces a dramatic, measurable security improvement, however you implement it.

Common Mistakes When Learning Authentication

Treating MFA as a checkbox instead of a layered defense. Not all MFA is equal; SMS-based codes are meaningfully weaker than app-based or hardware-key methods, and the choice matters.
Assuming passkeys solve account recovery for free. Losing device access remains a real, unsolved friction point in passkey systems; a thoughtful recovery flow is still required engineering work.
Rolling out passkeys without a password fallback for unsupported platforms. Adoption is accelerating, but it isn't universal yet, so most real systems still need to support both during the transition.
Underestimating MFA fatigue as a real attack vector. Push-notification approval fatigue is a documented way attackers bypass MFA; it needs active design consideration, not just assuming the second factor is inherently safe.
Storing passwords insecurely. Even as a fallback, proper hashing (bcrypt, Argon2) remains non-negotiable; this is a different failure mode than the ones passkeys and MFA address, and it's still entirely your responsibility to get right.

Frequently Asked Questions

Should a beginner learn password-based auth before passkeys?

Yes. Understanding how password hashing and verification actually work is foundational, and it's also what most existing production systems you'll work on still use, alongside an increasing passkey option.

Do passkeys replace the need for MFA?

Not entirely. Passkeys are inherently strong as a first factor, but MFA remains a valuable additional layer for especially sensitive actions, even on top of a passkey-based login.

Is SMS-based MFA still worth implementing?

It's meaningfully better than no MFA at all, the research shows even SMS blocking the bulk of automated credential attacks, but app-based or hardware-key MFA is a stronger choice where you can offer it.

Why did CVS Health see such a large reduction with passkeys specifically?

Because passkeys eliminate the phishable shared secret entirely; a fake login page simply has nothing to capture, which directly targets the attack vector responsible for a large share of real-world account takeover.

Can I implement passkeys without replacing my entire authentication system?

Yes, and most real rollouts work this way, offering passkeys as an additional login option alongside existing passwords, then gradually encouraging adoption rather than forcing an all-at-once cutover.

How do interviewers evaluate this topic in interviews?

Usually by asking you to design a real login flow and justify your choices, rather than testing whether you can name all three. Explaining specifically what each layer defends against, and why passwords alone fail, is what signals real understanding.

Build Authentication That Reflects the Real Numbers

The lesson from CVS Health's 98% and Microsoft's 99.22% isn't "pick the single best method," it's that moving beyond a password alone, through passkeys, MFA, or both, produces a measurable, serious security improvement. Explore the Auth & Identity course to build login flows that reflect what the evidence actually shows.

Python vs. Go for Backend Services: 65% Lower Cloud Costs, One Rewrite

Ciphemic academia — Mon, 05 Oct 2026 03:05:51 +0000

Python vs. Go for Backend Services: 65% Lower Cloud Costs, One Rewrite

In late 2023, an engineering team at Vodafone rewrote a suite of Python AWS Lambda microservices in Go. They didn't touch the business logic, just the language. Functions that took 800 milliseconds in Python came back in 120 milliseconds in Go, and the migration cut their cloud costs by 65%. A few years earlier, Khan Academy ran a much bigger version of the same bet: a multi-year project to rewrite its entire Python 2 monolith as Go services, eventually shipping more than 500,000 lines of Go to production. One specific case they documented, a class of 1,000 students that took 28 seconds to load in Python, loaded in 4 seconds in Go.

Neither team rewrote their backend because Go is trendy. They rewrote because a specific, measured bottleneck, cost in Vodafone's case, raw load time in Khan Academy's, made the switch worth real engineering effort. This guide compares Python and Go honestly for backend work: what each language actually optimizes for, where these real numbers come from, and how to decide before you've built years of Python you'll later have to migrate off, or worse, rewritten in Go when you never needed to.

If backend fundamentals are still new ground, our Backend Engineer roadmap covers what this comparison builds on.

This post originally appeared on the Ciphemic Academia blog.

The Short Version

Python is a dynamically typed, interpreted language with an enormous ecosystem, famous for fast development speed and readability, at a real runtime performance cost.
Go is a statically typed, compiled language built by Google specifically for simple, fast, concurrent backend services, trading some of Python's expressiveness for speed and predictability.

If you want one default: start with Python if you're optimizing for development speed, a smaller team, or heavy use of Python-specific ecosystems (data science, ML). Reach for Go specifically once you have a measured performance or cost bottleneck, the way Vodafone and Khan Academy both did, not before.

What Both Are Actually Trying to Solve

Before the differences, the shared job, since this is what any backend language exists to do:

Handling requests reliably: both need to receive a request, do some work, and return a response, correctly, under real load
Managing concurrency: both need a story for handling many requests at once without one slow request blocking everything else
Integrating with the rest of your stack: databases, queues, caches, third-party APIs, both need solid, well-supported ways to talk to all of it
Staying maintainable as a team grows: code that's easy to read, test, and change safely matters regardless of which language you chose

The real difference is what each language assumes is more valuable: developer speed and flexibility, or raw execution speed and predictability, and that trade-off is exactly what shows up in Vodafone's and Khan Academy's numbers.

Python

What it actually is: a dynamically typed, interpreted language where you don't declare variable types, the interpreter figures them out at runtime, and you don't compile your code before running it. For backend work, frameworks like Django, Flask, and FastAPI handle the web-serving layer on top of the language itself.

A small example (a simple endpoint, FastAPI):

@app.get("/orders/{order_id}")
async def get_order(order_id: int):
    order = await db.fetch_order(order_id)
    return {"id": order.id, "total": order.total}

Where it shines:

Fast development speed: Python's concise, readable syntax and lack of a compile step mean you can write and test code quickly, which is exactly why most teams, including Vodafone's original stack, start there
Enormous ecosystem: mature libraries for nearly everything, especially data science, machine learning, and scripting, make Python a natural default when a backend touches those domains
Gentle learning curve: Python is widely considered one of the easier languages to learn and read, which helps onboarding and lowers the bar for new contributors
Great for prototyping and iteration: when a product's requirements are still shifting, Python's flexibility lets you change direction without fighting the language

Where it struggles:

Real runtime performance ceiling: Python's interpreted execution and the Global Interpreter Lock (GIL) genuinely limit raw speed and true multi-threaded concurrency, which is precisely the bottleneck Vodafone's 800ms functions were hitting
Higher resource cost at scale: slower execution translates directly into needing more compute to handle the same load, which is where Vodafone's 65% cost reduction came from
Dynamic typing can hide bugs until runtime: without discipline (and tools like type hints and mypy), type-related bugs that a compiler would catch in Go can slip through to production
Concurrency is less natural: Python's asyncio and threading models work, but they're less of a first-class, built-in feature than Go's goroutines

Who this suits: teams prioritizing development speed and iteration, products with heavy data science or ML components, and most backends until a specific, measured bottleneck says otherwise.

Go

What it actually is: a statically typed, compiled language created at Google specifically to make writing simple, fast, highly concurrent network services easier. Code compiles to a single binary, concurrency is a built-in language feature (goroutines and channels), and the language deliberately has a small feature set to keep code consistent and readable across large teams.

A small example (the same endpoint, Go):

func getOrder(w http.ResponseWriter, r *http.Request) {
    id := chi.URLParam(r, "orderID")
    order, err := db.FetchOrder(id)
    if err != nil {
        http.Error(w, err.Error(), 500)
        return
    }
    json.NewEncoder(w).Encode(order)
}

Where it shines:

Real performance gains: compiled, statically typed execution is exactly why Vodafone's functions dropped from 800ms to 120ms, and why Khan Academy's 28-second case became 4 seconds, both real, measured, production numbers
Built-in concurrency: goroutines make writing concurrent code dramatically simpler than in most languages, which matters directly for backend services juggling many simultaneous requests
Lower resource usage: faster execution and a smaller memory footprint translate into genuine infrastructure cost savings at scale, the direct driver of Vodafone's 65% figure
Catches errors earlier: static typing and compilation surface a real class of bugs before code ever reaches production, something Khan Academy's engineers specifically noted as a benefit during their migration

Where it struggles:

More verbose than Python: Khan Academy's own engineers, after porting 500,000 lines, noted plainly that Go is more verbose in general than Python, certain things that are concise in Python take more code in Go
Smaller ecosystem for some domains: data science and ML tooling, in particular, remain far more mature in Python; Go is not generally the language teams reach for there
Steeper initial learning curve for dynamic-language developers: engineers coming from Python need to adjust to explicit typing, error handling as return values instead of exceptions, and a more rigid structure
Migration itself is real work: both Vodafone's and Khan Academy's rewrites were deliberate, resourced engineering projects, not quick swaps, Khan Academy's took roughly 20 months

Who this suits: backend services with a genuine, measured performance or cost bottleneck, systems with heavy concurrency needs, and teams building for scale where infrastructure cost is a real, ongoing line item.

Side-by-Side Comparison

	Python	Go
Typing	Dynamic	Static
Execution	Interpreted	Compiled to a single binary
Concurrency model	`asyncio`/threading, limited by the GIL	Goroutines, built into the language
Measured real-world case	800ms per function (Vodafone, before)	120ms per function (Vodafone, after)
Development speed	Faster to write and iterate	More verbose, more upfront structure
Resource cost at scale	Higher	Lower, direct driver of Vodafone's 65% savings
Ecosystem strength	Data science, ML, scripting	Networked services, CLI tools, infrastructure
Error detection	Mostly at runtime	Many errors caught at compile time

How This Fits a Career Path

Backend Engineer (general): Python remains an extremely strong starting point and stays relevant for a huge share of real jobs, especially where ecosystem breadth matters more than raw throughput
Platform, infrastructure, or high-scale backend engineer: Go is close to a standard expectation here, and understanding when and why teams like Vodafone and Khan Academy migrated is exactly the reasoning interviewers probe for
Data-heavy or ML-adjacent backend roles: Python's ecosystem advantage is large enough here that it typically remains the default regardless of raw language performance
System design interviews: be ready to justify a language choice the way Vodafone and Khan Academy did, a specific, measured problem, not a language preference stated as fact

How to Choose Without Overthinking It

Start with Python unless you already know you need Go. Development speed and ecosystem breadth win for most new backend projects, especially early on when requirements are still shifting.
Look for an actual, measured bottleneck before migrating. Vodafone didn't start with Go, they measured real cost and latency problems in Python first. Rewriting a working service because a language sounds faster, without your own numbers, is how teams waste real engineering time.
Weigh your team's concurrency needs honestly. If your service is CPU-bound and needs genuine, heavy concurrency, Go's goroutines solve a problem Python's GIL makes genuinely harder.
Consider a partial migration, not an all-or-nothing rewrite. Vodafone kept Python and Go running side by side using weighted routing during their transition, and Khan Academy's 20-month migration moved services incrementally behind a GraphQL gateway rather than cutting over all at once.

A note on honesty: Vodafone's 65% and Khan Academy's 28-to-4-second numbers are real, but they're specific to their own workloads, infrastructure, and code patterns. Khan Academy's own engineers were explicit that they weren't even prioritizing performance work during their port, the gains came largely from Go's baseline execution model, not from heavy optimization. Measure your own service before assuming these exact numbers will repeat for you.

Common Mistakes When Learning Backend Development

Choosing Go before you have a measured problem. Go's verbosity and stricter structure are a real cost; that cost should be paying for a genuine, current performance or scale need, not a hypothetical future one.
Assuming Python is "too slow" for everything. The vast majority of backend services never hit the kind of bottleneck Vodafone and Khan Academy measured; Python handles enormous scale successfully across the industry.
Rewriting everything at once. Both real migrations in this post were incremental and ran old and new code side by side; an all-at-once rewrite carries far more risk than either team accepted.
Ignoring ecosystem fit. A backend doing heavy data science or ML work loses real Python ecosystem advantages by moving to Go, regardless of raw execution speed.
Treating the language choice as permanent. Khan Academy's own migration shows that even a mature, successful Python codebase can be rebuilt gradually when the numbers genuinely justify it.

Frequently Asked Questions

Should a beginner learn Python or Go first?

Python, for nearly all beginners. Its readability and gentler learning curve make foundational programming concepts easier to absorb, and its ecosystem breadth makes it useful immediately across many domains, not just backend work.

Is Go always faster than Python?

In the vast majority of raw execution benchmarks, yes, which is exactly why Vodafone's and Khan Academy's numbers moved the direction they did. But "faster" only matters if your actual bottleneck is execution speed; many real backend bottlenecks are elsewhere (database queries, network calls), where the language matters far less.

Do I need to learn Go if I only do backend work in Python?

Not urgently, but it's a genuinely valuable second language for backend engineers, specifically for the performance and concurrency scenarios in this post. Many strong backend engineers are comfortable in both and choose deliberately per project.

Why did Khan Academy's migration take 20 months?

Because it was a full rewrite of a large, mature monolith into services, done incrementally and safely behind a GraphQL gateway, with old and new code running side by side throughout. Real migrations of this scale are genuine, resourced engineering projects, not quick rewrites.

Can Python and Go coexist in the same backend?

Yes, and Vodafone's own migration is a direct example, they ran Python and Go Lambda functions side by side using weighted routing while the migration was in progress, rather than cutting over all at once.

How do interviewers evaluate this topic?

Usually by asking you to reason about a specific scenario, team size, performance needs, ecosystem fit, rather than testing for a fixed language preference. Being able to explain why Vodafone or Khan Academy made their specific choice, and what evidence justified it, signals more real understanding than reciting that "Go is faster."

Measure Before You Rewrite

The real lesson from Vodafone's 65% savings and Khan Academy's 500,000-line migration isn't "always choose Go," it's that both teams measured a specific, real problem before committing serious engineering time to a rewrite. Explore the Backend Engineer roadmap to build the skills that let you make that same evidence-based call on your own projects.

React Native vs. Flutter vs. Native: Why Cure.fit Rebuilt Its Navigation From Scratch

Ciphemic academia — Sun, 04 Oct 2026 06:30:44 +0000

React Native vs. Flutter vs. Native: Why Cure.fit Rebuilt Its Navigation From Scratch

Cure.fit, the Indian fitness platform, had built its app on React Native since day one. Then they hit a wall that kept showing up in the same place: animations. The bottom navigation bar, the kind of UI every user touches constantly, never felt right, and the team traced the problem specifically to React Native's architecture. Their fix wasn't a patch, it was rebuilding the entire bottom navigation layer in Flutter, piece by piece, until the whole experience ran noticeably smoother. A separate, independent benchmark that rewrote the same simple app three times, once native, once in Flutter, once in React Native, measured Flutter's memory footprint at 23MB against React Native's 27MB, with the native version at 14MB, a real, measurable gap between all three, not just a feeling.

These two data points capture the actual decision you're making when you pick a mobile stack: not "which one is popular," but how much of that 14MB-to-27MB gap your specific app can afford to give up in exchange for faster, cheaper development. This guide compares all three honestly, what each one actually does under the hood, where the real numbers come from, and how to decide before you've built a year of app on an architecture you'll later have to rebuild, the way Cure.fit did.

If mobile fundamentals are still new ground, our Mobile Engineering roadmap covers what this comparison builds on.

This post originally appeared on the Ciphemic Academia blog.

The Short Version

Native development (Swift/Kotlin, separately for iOS and Android) gives you the best possible performance and full platform access, at the cost of building and maintaining two separate codebases.
React Native renders using actual native platform widgets, controlled from JavaScript through a "bridge," letting web developers reuse familiar skills, with real architectural overhead exactly where Cure.fit hit its wall.
Flutter draws its own UI from scratch using its own rendering engine, skipping native widgets entirely, which gives it more consistent performance across platforms, at the cost of a less familiar language (Dart) for most teams.

If you want one default: start with React Native if your team already knows JavaScript/React, start with Flutter if you're optimizing for consistent performance and don't mind learning Dart, and go native only once you have a specific, measured reason the other two can't satisfy.

What All Three Are Actually Trying to Solve

Before the differences, the shared problem, since this is what every mobile approach is answering:

Rendering a UI people can touch and interact with smoothly: all three ultimately have to draw pixels and respond to touch with low enough latency that it feels instant
Accessing device capabilities: camera, GPS, push notifications, sensors, every approach needs a path to the platform's native APIs
Shipping to both iOS and Android: even "native" development means solving this problem twice, just with two separate codebases instead of one
Keeping development sustainable over time: all three need to remain maintainable as a team and a codebase grow, not just work well on day one

The real difference is architectural: how each approach gets from your code to pixels on the screen, and that architecture is exactly where Cure.fit's animation problems and the thoughtbot benchmark's memory numbers both originate.

Native Development

What it actually is: building genuinely separate apps for iOS (typically Swift) and Android (typically Kotlin), each using the platform's own UI framework and talking directly to the operating system with no intermediate layer at all.

A small example (a button, Swift):

let button = UIButton(type: .system)
button.setTitle("Checkout", for: .normal)
button.addTarget(self, action: #selector(checkout), for: .touchUpInside)

Where it shines:

Best possible performance: no bridge, no abstraction layer, direct access to the platform's own rendering pipeline, which is exactly why the thoughtbot benchmark found native using the least memory (14MB) of the three
Full, immediate platform access: new OS features and APIs are available to native code first, often before cross-platform frameworks catch up
Platform-idiomatic UI by default: native components automatically match each platform's look, feel, and accessibility behavior without extra work
No framework-layer bugs to debug around: you're working directly with Apple's and Google's own tools and documentation, with nothing else in between

Where it struggles:

Two full codebases: essentially building and maintaining two separate applications, in two separate languages, which is real, ongoing engineering cost
Slower feature development: every feature gets built twice, once per platform, which is a genuine velocity cost most teams feel immediately
Requires two skill sets on the team: hiring and retaining both strong iOS and strong Android engineers is a real organizational cost smaller teams often can't absorb

Who this suits: apps where performance is genuinely critical (heavy animation, gaming, AR/VR), teams with the resources to maintain two codebases well, and situations where deep platform-specific integration matters more than development speed.

React Native

What it actually is: a framework where you write UI logic in JavaScript and React, and React Native translates your component tree into calls that render using the platform's actual native widgets, through a communication layer historically called "the bridge." Your JavaScript doesn't draw the UI directly; it instructs the native side what to draw.

A small example (the same button, React Native):

<TouchableOpacity onPress={checkout}>
  <Text>Checkout</Text>
</TouchableOpacity>

Where it shines:

Massive talent pool: any team with React or JavaScript experience can contribute meaningfully with a relatively short ramp-up
Genuinely native UI components: because it renders actual platform widgets rather than drawing its own, UI elements inherit native look-and-feel and accessibility behavior by default
Huge ecosystem: a large library ecosystem and strong community support cover most common app needs without custom native code
Hot reload and fast iteration: changes reflect almost instantly during development, which speeds up the build-test loop considerably

Where it struggles:

The bridge is a real architectural cost: every interaction between JavaScript and native code crosses that bridge, and Cure.fit's own engineering write-up specifically traced their animation and bottom-bar problems to this layer, not to a specific bug they could just patch
Performance ceiling for heavy UI work: complex animations and gesture-heavy interactions are exactly where the bridge's overhead becomes visible to users, as Cure.fit experienced directly
Version and dependency churn: the React Native ecosystem has historically had more breaking changes and upgrade friction than some teams expect going in

Who this suits: teams with strong existing JavaScript/React skills, apps without extremely heavy custom animation requirements, and products that benefit from a large, mature plugin ecosystem.

Flutter

What it actually is: a UI toolkit from Google where you write in Dart, and Flutter renders every pixel itself using its own graphics engine (Skia, with newer Flutter versions moving toward Impeller), completely bypassing native platform widgets. Nothing is translated to native UI components; Flutter draws its own UI from scratch on both platforms.

A small example (the same button, Flutter):

ElevatedButton(
  onPressed: checkout,
  child: Text('Checkout'),
)

Where it shines:

Consistent performance across platforms: since Flutter draws everything itself rather than bridging to native widgets, iOS and Android apps behave far more consistently, which is exactly why Cure.fit chose Flutter specifically for its animation-heavy navigation layer
Closer to native-level performance: the thoughtbot benchmark found Flutter's resource usage (23MB) meaningfully closer to native (14MB) than React Native's (27MB) in that specific test
Single codebase, true pixel-level control: since Flutter isn't constrained by native widget behavior, teams get precise, consistent control over complex custom UI and animations
Strong tooling and hot reload: Flutter's development experience, including fast iteration, is widely regarded as a genuine strength

Where it struggles:

Dart is a real learning curve: unlike React Native's JavaScript, Dart is a less widely known language, which is real onboarding friction for most existing web or mobile teams
UI doesn't use native widgets: because Flutter draws everything itself, platform-specific look-and-feel updates (new OS-level UI changes) don't arrive automatically the way they sometimes do with React Native's native-widget approach
Smaller long-term talent pool than JavaScript: while growing quickly, the pool of experienced Flutter/Dart developers remains smaller than the React/JavaScript pool

Who this suits: teams optimizing specifically for consistent, high-quality UI and animation across platforms, projects without an existing deep investment in JavaScript, and exactly the kind of animation-heavy component Cure.fit chose to rebuild.

Side-by-Side Comparison

	Native	React Native	Flutter
Language	Swift (iOS), Kotlin (Android)	JavaScript	Dart
Rendering approach	Platform's own UI framework	Native widgets via a bridge	Flutter's own engine, no native widgets
Codebases	Two, separate	One, shared	One, shared
Measured memory use (benchmark)	14MB	27MB	23MB
Performance ceiling	Highest	Lower, bridge overhead	Closer to native than React Native
Talent pool	Two specialized skill sets	Largest (JS/React)	Smaller, growing
Best for	Performance-critical, heavy native integration	Teams with strong React skills, standard app UI	Consistent cross-platform UI, animation-heavy screens

How This Fits a Career Path

Mobile Engineer (general): understanding all three, not just one, is increasingly the expectation, since real companies (Cure.fit included) mix approaches rather than picking one dogmatically
Frontend-to-mobile transition: React Native is the natural bridge for an existing React developer; our React & Frontend roadmap covers the web-side fundamentals that transfer directly
Cross-platform specialist: Flutter is worth deliberate investment if you're aiming at roles optimizing for consistent UI and animation quality across platforms
Platform or performance-focused roles: native development remains the standard where the thoughtbot-style performance gap genuinely matters to the product (gaming, AR, heavy real-time graphics)

How to Choose Without Overthinking It

Start with your team's existing skills. Strong JavaScript/React team? React Native removes a real learning-curve cost. No strong investment either way? Flutter's more consistent cross-platform behavior is worth the Dart learning curve for many teams.
Identify your actual animation and performance requirements early. Cure.fit didn't start with Flutter, they migrated specifically once a real, measured UI problem appeared. If your app is animation-light, React Native's bridge overhead may never become a real issue.
Don't default to native "to be safe." Two codebases is a real, ongoing cost; reserve native for situations with a specific, articulable performance or platform-access requirement the other two can't meet.
Consider a hybrid approach for an existing app. Cure.fit's "Add-to-App" style migration, replacing just one problematic screen or flow with Flutter inside an existing React Native app, is a real, lower-risk pattern worth knowing about rather than an all-or-nothing rewrite.

A note on honesty: the thoughtbot benchmark's 14MB/23MB/27MB numbers came from one simple timer app on specific devices; your app's real numbers will depend heavily on what you're actually building. Cure.fit's decision was driven by a specific, felt problem in one screen, not a blanket verdict against React Native. Measure your own app's pain points before committing to a framework based on someone else's numbers, including these ones.

Common Mistakes When Learning Mobile Development

Picking a framework before identifying your actual UI complexity. A simple CRUD-style app rarely needs to worry about the bridge-overhead problems that drove Cure.fit's migration.
Assuming "cross-platform" means identical performance to native. The real benchmark gap (14MB vs. 23MB vs. 27MB) is small for most apps, but it's not zero, and heavy animation is exactly where it shows up.
Underestimating Dart's learning curve for Flutter. It's a real, if often overstated, cost; budget genuine ramp-up time for a team new to it.
Rewriting an entire app instead of migrating the problem screen. Cure.fit's incremental, screen-by-screen approach is a lower-risk pattern than an all-at-once rewrite most teams should consider first.
Treating framework choice as permanent and un-revisitable. Real companies, Cure.fit included, have mixed frameworks within one app when a specific part of the product demanded it.

Frequently Asked Questions

Should a beginner learn React Native, Flutter, or native development first?

If you already know JavaScript and React, React Native gives you the fastest path to a working app. If you're starting fresh with no strong language preference, Flutter's consistent cross-platform behavior and strong tooling make it a genuinely good first choice too.

Is React Native being replaced by Flutter?

No. Both remain widely used in production, often by the same companies for different parts of their app, as Cure.fit's own hybrid approach shows. React Native's architecture has also continued to evolve specifically to reduce the bridge overhead that caused problems like Cure.fit's.

Does Flutter always perform better than React Native?

Not universally, but it was meaningfully closer to native in the specific thoughtbot benchmark cited above, and Cure.fit's real production experience with animation specifically favored Flutter. Your own app's results will depend on what you're actually building.

When does the performance difference between these frameworks actually matter?

Specifically in animation-heavy, gesture-intensive, or otherwise performance-critical UI, exactly the kind of bottom-navigation problem that pushed Cure.fit to migrate. A typical CRUD or content-display app is far less likely to hit this ceiling with either framework.

Can I mix native, React Native, and Flutter in one app?

Yes, and Cure.fit's real migration is a direct example, they embedded Flutter views inside an existing React Native app's navigation rather than rewriting everything at once.

How do interviewers evaluate this topic?

Usually by asking you to reason about trade-offs for a specific scenario, team skills, performance needs, timeline, rather than testing for a fixed framework preference. Being able to explain why a real team like Cure.fit chose to migrate, and what specifically drove that decision, is exactly the kind of reasoning that signals real experience.

Build the Judgment, Not Just the Framework Skill

The real lesson from Cure.fit's migration and the thoughtbot benchmark isn't "Flutter beats React Native," it's that both teams measured a specific, real problem before making a change. Explore the Mobile Engineering roadmap to build the skills that let you make that same evidence-based call on your own projects.

Logs vs. Metrics vs. Traces: How PepsiCo Cut Incident Resolution Time by 30%

Ciphemic academia — Sun, 04 Oct 2026 06:28:55 +0000

Logs vs. Metrics vs. Traces: How PepsiCo Cut Incident Resolution Time by 30%

PepsiCo was running over 55 separate observability tools across its digital infrastructure, logs in one place, metrics in another, traces scattered across a third, with no single view connecting them. When something broke, engineers spent real time just stitching the story together before they could even start fixing it. After consolidating metrics, logs, and traces into one unified platform, PepsiCo cut its mean time to resolution by 30%, reduced its tool count from 55 to under 20, and saved up to 25% in annual hardware costs along the way.

That number didn't come from a smarter alerting rule or a faster server. It came from treating logs, metrics, and traces as three parts of one system instead of three separate problems. Most engineers learn each of these signals in isolation, if they learn traces at all, and never build a clear mental model of what each one is actually for. This guide breaks down what each signal tells you, where each one falls short on its own, and how real teams wire them together into something that actually shortens an incident.

If observability concepts are still new ground, our Observability course covers the fundamentals this comparison builds on.

This post originally appeared on the Ciphemic Academia blog.

The Short Version

Logs are detailed, timestamped records of discrete events, the richest source of context, but the hardest to search at scale without structure.
Metrics are numeric measurements aggregated over time (CPU usage, request count, error rate), cheap to store and great for spotting that something is wrong, fast.
Traces follow a single request as it travels through a distributed system, showing exactly where time was spent across every service it touched.

If you want one default: you need all three, not one. They answer different questions, and real observability platforms, including the one behind PepsiCo's 30% improvement, exist specifically to connect them, not to replace one with another.

What All Three Are Actually Trying to Solve

Before the differences, the shared goal, since it's easy to lose sight of when people argue "logs vs. metrics" as if it's a single choice:

Detecting that something is wrong: all three can surface an anomaly, though some do it faster than others
Diagnosing why it's wrong: all three contribute context, but at very different levels of detail and cost
Reconstructing what happened: after an incident, you need to explain the timeline, and each signal captures a different slice of that timeline
Doing this at scale without drowning the team: raw data from any of the three, unfiltered, becomes noise instead of signal past a certain volume

The real skill isn't picking a favorite. It's knowing which signal answers which question, and building a system where they connect.

Logs

What it actually is: a timestamped, detailed record of a discrete event, a request came in, an error was thrown, a job finished, usually written as structured or semi-structured text by your application code directly.

A small example:

2026-01-15T09:42:11Z ERROR order-service: Failed to charge card for order=482, customer=c91, reason=insufficient_funds

Where it shines:

Maximum detail: logs capture exactly what happened, in plain language, including context a metric or trace alone would never show
Flexible and ad hoc: you can log anything, at any point in your code, without planning a schema in advance
Essential for root-cause investigation: once you know roughly where a problem is, logs usually hold the specific detail that explains why

Where it struggles:

Expensive at scale: storing and indexing every log line from a high-traffic system gets costly fast, which is exactly the kind of overhead PepsiCo's consolidation was fighting
Slow to search without structure: unstructured, free-text logs are genuinely hard to query efficiently across millions of lines
No inherent sense of "normal": a log line doesn't tell you whether an error rate is unusual, just that one error happened

Who this suits: deep, specific investigation once you already have a rough idea of where to look, and the layer most teams reach for last, not first, during an incident.

Metrics

What it actually is: numeric measurements, aggregated over time, that summarize system behavior: requests per second, average latency, error rate, CPU usage, memory consumption. Metrics are typically sampled or counted at regular intervals and stored efficiently as time-series data.

A small example:

http_requests_total{service="order-service", status="500"} 142
http_request_duration_seconds{service="order-service", quantile="0.99"} 1.8

Where it shines:

Cheap to store and query at scale: a single number per time interval is far lighter than a full log line or trace, which is part of why metrics are the natural backbone of dashboards and alerts
Great for detecting anomalies fast: a sudden spike in error rate or latency is immediately visible on a metric dashboard, often before a single customer complains
Natural fit for alerting: thresholds on metrics (error rate above 5%, latency above 2 seconds) are the standard way teams get paged in the first place

Where it struggles:

No individual context: a metric tells you error rate spiked to 12%, but nothing about which specific requests failed or why
Aggregation can hide real problems: an average latency metric can look fine while a specific subset of requests is badly broken, a well-known blind spot in metrics-only monitoring
Doesn't explain causation on its own: metrics are excellent at telling you that something changed, and far weaker at telling you why without other signals alongside them

Who this suits: the first layer of detection in almost every incident, dashboards, alerting, and getting a fast, high-level read on system health.

Traces

What it actually is: a record of a single request's journey through a distributed system, broken into "spans," each representing the time spent in one service or operation. A trace shows you the full path, including exactly where time was spent, across every microservice the request touched.

A small example (a simplified trace):

Trace: GET /checkout (total: 842ms)
 ├─ auth-service        (12ms)
 ├─ order-service        (430ms)
 │    └─ payment-service (390ms)  ← most of the time is here
 └─ notification-service (25ms)

Where it shines:

Pinpoints exactly where time went: in a system with a dozen microservices, a trace tells you precisely which one is the bottleneck, something logs and metrics alone genuinely struggle to show
Reveals the shape of distributed systems: traces expose real dependency chains and latency sources that can be invisible from any single service's perspective
Essential for microservice debugging: as the Irdeto case study on reducing MTTR put it, going "from logs to traces" was specifically what let their team stop guessing and start seeing the actual request path

Where it struggles:

More setup required: tracing needs instrumentation across every service a request touches, which is real upfront engineering work compared to logs or metrics
Sampling trade-offs: at high volume, many systems sample only a fraction of traces to control cost, risking missing the exact trace that mattered
Less useful in isolation: a trace tells you where time was spent, but you'll often still want a log from that specific span to understand exactly why

Who this suits: microservice and distributed-system debugging specifically, and the layer that turns "something in our system is slow" into "this exact service call is slow, and here's why."

Side-by-Side Comparison

	Logs	Metrics	Traces
What it captures	Discrete events, full detail	Aggregated numeric measurements	A request's path across services
Storage cost at scale	High	Low	Moderate, often sampled
Best for	Deep root-cause detail	Fast anomaly detection, alerting	Pinpointing where time was spent
Setup effort	Low, just write log lines	Moderate, needs instrumentation	High, needs distributed instrumentation
Answers	"What exactly happened?"	"Is something wrong, and since when?"	"Where, specifically, did it go wrong?"
Weakness alone	Hard to search at scale	No individual request context	Limited without logs for final detail

How This Fits a Career Path

Backend Engineer (general): structured logging is close to a baseline expectation, and understanding what a metric dashboard is actually telling you is assumed in nearly every on-call rotation
SRE or platform engineer: this is close to the center of the job; our Observability course covers building the instrumentation and dashboards that make incidents like PepsiCo's resolvable in minutes instead of hours
Microservices or distributed-systems engineer: tracing becomes non-negotiable once a request crosses more than two or three services, since logs and metrics alone genuinely can't show you the path
System design interviews: be ready to explain how you'd instrument a new service with all three signals, and why one alone wouldn't be enough, a common way interviewers probe real operational experience

How to Choose Without Overthinking It

Don't choose. Layer them. Metrics tell you something's wrong fast. Traces show you where. Logs tell you exactly why. Removing any one layer leaves a real gap, which is precisely what PepsiCo's fragmented 55-tool setup was suffering from before consolidation.
Start with metrics and alerting if you have nothing yet. They're the cheapest to implement and give you the fastest path to knowing something broke.
Add tracing once you have more than a couple of services talking to each other. A single-service app rarely needs it; a system with five or more collaborating services genuinely does.
Invest in structured logging early, not as an afterthought. Structured logs (consistent fields, not free text) are dramatically easier to search and correlate with traces and metrics later.

A note on honesty: the specific improvement numbers above, PepsiCo's 30%, Irdeto's reported MTTR drop, redBus's 50% MTTR reduction, all came from consolidating and correlating signals, not from adopting any single one of the three. The lesson isn't "traces are the best signal." It's that disconnected tools, even good ones, cost real time during an incident.

Common Mistakes When Learning Observability

Treating this as "pick one." Logs, metrics, and traces answer genuinely different questions; a team that only has logs, or only has metrics, is missing real diagnostic capability, not just a nice-to-have.
Logging everything without structure. Free-text logs at scale become expensive and hard to search; structured logging pays off the moment you need to correlate a log with a trace.
Adding tracing too late. Retrofitting distributed tracing onto an already-complex microservices system is real work; instrumenting from early on is far cheaper than doing it under pressure during an incident.
Alerting on raw metrics without context. A threshold alert with no linked trace or log tells you something's wrong but forces engineers to start the investigation from zero every time.
Ignoring tool sprawl. PepsiCo's starting point, 55 separate tools, is a cautionary example: more observability tools doesn't automatically mean better observability if they don't talk to each other.

Frequently Asked Questions

Should a beginner learn logs, metrics, or traces first?

Logs, since nearly every application already produces them and they require the least new infrastructure to start with. Move to metrics and alerting next, then traces once you're working with more than a couple of interacting services.

Do I need all three for a small application?

Not urgently. A small, single-service application can get a long way with good logging and a handful of key metrics. Tracing earns its place specifically once requests start crossing multiple services.

What's the difference between monitoring and observability?

Monitoring typically refers to watching known metrics and alerting on them. Observability is the broader ability to ask new questions about your system's behavior after the fact, which is exactly why logs, metrics, and traces working together, rather than in isolation, define the term.

Why did PepsiCo's MTTR improve by consolidating tools rather than adding new ones?

Because the bottleneck wasn't a lack of data, it was disconnected data. Engineers were manually correlating signals across 55 separate tools; a unified platform let them move from symptom to root cause in one workflow instead of many.

Is distributed tracing only useful for microservices?

It's most valuable there, but even a handful of services calling each other (an API calling a database and a third-party service) can benefit from tracing to see where latency actually originates.

How do interviewers evaluate this topic?

Less about definitions, more about whether you can describe a real incident-response workflow: which signal you'd check first, how you'd narrow down from "something's wrong" to a specific root cause, and why no single signal would have been enough alone.

Build Observability That Actually Shortens Incidents

The real lesson from PepsiCo's 30% improvement isn't which signal to prioritize, it's that logs, metrics, and traces only pay off fully when they're connected. Explore the Observability course to learn how to instrument, correlate, and actually use all three signals together, not just collect them.

PostgreSQL vs. MongoDB: A 10x Speedup, a 4-Hour Outage, and the Real Question Behind "SQL vs. NoSQL"

Ciphemic academia — Thu, 01 Oct 2026 03:27:18 +0000

PostgreSQL vs. MongoDB: A 10x Speedup, a 4-Hour Outage, and the Real Question Behind "SQL vs. NoSQL"

The team behind Shippable, a CI/CD platform running over 50 microservices, rebooted their MongoDB server one day as routine maintenance. It took four hours to come back online. The culprit traced back to index rebuilding, a process that locked the entire database while millions of documents sat untouched by their application. They migrated to PostgreSQL shortly after and never looked back. Years later, the team behind Agenta ran their own migration in the opposite direction of "obvious," moving off MongoDB and onto Postgres, and measured up to 10x faster performance in some of their real workloads.

Neither story means one database is simply better. They mean the same thing every real "SQL vs. NoSQL" decision eventually teaches: the right choice depends entirely on your actual data shape and access patterns, and the wrong choice shows up later as an outage or a performance cliff, not as an error message on day one. This guide compares PostgreSQL and MongoDB honestly, what each one actually optimizes for, where the real numbers come from, and how to decide before your schema assumptions get expensive to undo.

If database fundamentals like indexes, normalization, and query planning are still new to you, our Data Engineer roadmap builds that foundation before this comparison will fully click.

This post originally appeared on the Ciphemic Academia blog.

The Short Version

PostgreSQL is a relational database: structured tables, enforced schemas, and powerful SQL for querying across related data, with full ACID transaction guarantees.
MongoDB is a document database: flexible, schema-less JSON-like documents, built for fast iteration and data shapes that don't fit neatly into rows and columns.

If you want one default for a new application: start with PostgreSQL. It handles a wider range of use cases well, including many that look "document-shaped," since modern Postgres has strong native JSON support. MongoDB earns its place in specific situations this guide covers below.

What Both Are Actually Trying to Solve

Before the differences, the shared job, since it's easy to lose in the SQL-vs-NoSQL framing:

Storing data durably: both need to persist your data reliably and get it back when asked
Querying efficiently: both use indexes and query planners to avoid scanning every record for every lookup
Scaling with your application: both offer ways to handle more data and more traffic as you grow, just through different mechanisms
Maintaining data integrity: both have answers for "what happens if two things try to update the same data at once," though the strength of that answer differs significantly

The actual decision isn't "relational vs. modern." It's a trade-off between enforced structure and flexible structure, and each comes with real, measurable consequences.

PostgreSQL

What it actually is: a relational database management system where data lives in tables with defined columns and types, related tables are linked through foreign keys, and you query across them using SQL. Every row in a table follows the same schema, enforced by the database itself, not by application code remembering to check.

A small example:

CREATE TABLE orders (
  id SERIAL PRIMARY KEY,
  customer_id INT REFERENCES customers(id),
  total NUMERIC(10,2),
  created_at TIMESTAMP DEFAULT now()
);

SELECT orders.id, customers.name, orders.total
FROM orders
JOIN customers ON orders.customer_id = customers.id
WHERE orders.total > 100;

That JOIN is doing real work: combining data from two related tables in a single, efficient query, something a document database handles very differently.

Where it shines:

Strong data integrity: foreign keys, constraints, and full ACID transactions mean the database itself prevents a huge class of data corruption bugs, rather than relying on application code to catch them
Powerful querying across relationships: joins let you ask complex questions spanning multiple related tables in one query, which gets genuinely painful to replicate in a document model
Mature and battle-tested: PostgreSQL has decades of production use, extensive tooling, and a published benchmark found it ran tens of percent to three times faster than MongoDB on equivalent JSON workloads in one FOSDEM PGDay study
Native JSON support: modern PostgreSQL's JSONB column type lets you store flexible, document-like data inside a relational database when you genuinely need that flexibility for a specific field, without giving up SQL and transactions for everything else

Where it struggles:

Schema changes require more care: adding or changing a column on a huge table can require real planning, migrations, and sometimes downtime windows, compared to MongoDB's inherently flexible documents
Horizontal scaling is less automatic: scaling PostgreSQL across many servers (sharding) is possible but historically requires more deliberate setup than MongoDB's built-in sharding
Rigid structure can slow early iteration: when your data shape is still genuinely unsettled in an early prototype, defining and migrating schemas repeatedly can feel like friction you don't yet need

Who this suits: applications with clearly related data (orders, customers, inventory), anything where data correctness and consistency genuinely matter, and the sensible default for most new backend projects.

MongoDB

What it actually is: a document database where data is stored as flexible, JSON-like documents (BSON, specifically) inside collections, with no required schema enforced by the database. Different documents in the same collection can have different fields entirely, and nested, complex structures live naturally inside a single document instead of being split across tables.

A small example:

db.orders.insertOne({
  customerId: "c482",
  items: [
    { sku: "A1", qty: 2, price: 24.99 },
    { sku: "B7", qty: 1, price: 79.99 }
  ],
  total: 129.97,
  shippingAddress: { city: "Bhubaneswar", zip: "751007" }
});

db.orders.find({ total: { $gt: 100 } });

Notice the entire order, including nested items and a nested address, lives in one document. There's no join required to read it back as a single unit.

Where it shines:

Flexible schema: different documents can have different shapes, which genuinely speeds up early development when your data model is still evolving
Natural fit for document-shaped data: data that's naturally nested and self-contained (a user profile with embedded preferences, a product catalog entry with variable attributes) maps directly onto MongoDB's model without the joins a relational schema would require
Built-in horizontal scaling: MongoDB's sharding is a first-class, well-documented feature, designed from early on for distributing data across many servers
Fast for single-document reads and writes: when your access pattern is "fetch this whole record by ID," retrieving one self-contained document is often simpler and faster than assembling it from several joined tables

Where it struggles:

Weaker guarantees without careful design: MongoDB does support transactions, but relational integrity like foreign key constraints isn't enforced the way it is in PostgreSQL by default, so invalid references can silently creep in if the application doesn't catch them
Operational issues can be more disruptive: the Shippable team's four-hour outage traced directly to how MongoDB handled index rebuilding under load, a real, documented failure mode at scale
Querying across relationships is harder: modeling genuinely relational data (customers, their orders, those orders' line items, as separate connected entities) in MongoDB often means duplicating data across documents or writing more complex application-side logic to join it yourself
Schema-less can become schema-chaos: without discipline, a flexible schema can drift into a collection full of inconsistent documents that are genuinely hard to query and reason about later

Who this suits: applications with naturally document-shaped data, early-stage products where the data model is still actively evolving, and systems that need to scale horizontally across many servers from the start.

Side-by-Side Comparison

	PostgreSQL	MongoDB
Data model	Tables, rows, columns	Flexible JSON-like documents
Schema	Enforced by the database	Flexible, enforced by convention or app code
Relationships	Native, via joins and foreign keys	Modeled via embedding or manual references
Transactions	Full ACID, mature and well-tested	Supported, less central to the model historically
Horizontal scaling	Possible, more manual setup	Built-in, first-class sharding
Query language	SQL	MongoDB Query Language (JSON-like syntax)
Best for	Related, structured data, correctness-critical systems	Document-shaped data, rapidly evolving schemas
Documented failure mode	Migration and schema-change overhead at scale	Index rebuilds and operational issues under load

How This Fits a Career Path

Backend Engineer (general): SQL and relational thinking are close to a universal requirement, and understanding PostgreSQL well transfers to nearly every other relational database you'll encounter
Data Engineer: both come up constantly, but relational modeling, normalization, and SQL are foundational skills this role assumes; see our Data Engineer roadmap for how this fits into broader pipeline work
Full-stack or product-focused roles: MongoDB's document model often maps naturally onto JavaScript-heavy stacks, which is part of why it's historically popular in Node.js-based projects
System design interviews: be ready to justify a database choice using the same reasoning Shippable and Agenta used, a specific access pattern and a specific, measured problem, not a format preference

How to Choose Without Overthinking It

Start with your data's actual shape. Clearly related entities (customers, orders, products)? PostgreSQL's relational model fits naturally. Self-contained, nested, variably-structured records? MongoDB's document model fits naturally.
Consider how much you need enforced correctness. If invalid or inconsistent data would cause real business problems (financial records, inventory counts), PostgreSQL's constraints and transactions do real protective work for you.
Don't choose MongoDB "for flexibility" without a genuine need for it. Flexible schemas are a real benefit for a genuinely evolving data model, not a reason to skip schema design thinking altogether.
Remember PostgreSQL's JSONB support before assuming you need a separate document database. A lot of "we need MongoDB for flexible data" problems can be solved with a JSONB column inside an otherwise relational Postgres schema, keeping transactions and joins for everything else.

A note on honesty: the FOSDEM benchmark above found Postgres running tens of percent to three times faster than MongoDB in one specific JSON workload test, and Agenta measured up to 10x gains going the opposite direction. Benchmarks like these depend heavily on the specific workload, schema design, and configuration being tested. Treat any single number, including the ones in this post, as a reason to measure your own system, not as a universal verdict.

Common Mistakes When Learning Databases

Picking MongoDB because the data "feels like JSON." Plenty of JSON-shaped data is still genuinely relational underneath, and PostgreSQL's JSONB support often covers the flexibility you actually need.
Skipping schema design because NoSQL "doesn't need one." MongoDB not enforcing a schema doesn't mean your application doesn't have one; it just means the database won't catch you when you violate it.
Underestimating operational differences. Shippable's four-hour outage wasn't a MongoDB bug; it was a real operational characteristic of how indexes rebuild under load. Learn your database's actual maintenance behavior before you're depending on it at scale.
Assuming NoSQL automatically means "scales better." Both databases scale well with the right design; MongoDB's sharding is more built-in, but PostgreSQL handles substantial scale for the vast majority of real applications without ever needing to shard.
Treating this as a permanent, un-revisitable choice. Both Shippable's and Agenta's stories are migrations, real companies that changed direction once their actual usage patterns revealed a genuine mismatch.

Frequently Asked Questions

Should a beginner learn SQL or MongoDB first?

SQL, and specifically PostgreSQL. Relational thinking, joins, and normalization are foundational skills that transfer to nearly every other database you'll encounter, including understanding when a document database would actually help.

Is MongoDB being phased out in favor of SQL databases?

No. Both remain widely used in production, often by the same companies for different parts of their system. The Shippable and Agenta stories are real migrations, not evidence that one database is disappearing.

Can PostgreSQL handle large-scale, high-traffic applications?

Yes, extensively. Many very large, high-traffic systems run on PostgreSQL. Horizontal scaling requires more deliberate setup than MongoDB's built-in sharding, but the vast majority of applications reach significant scale on PostgreSQL alone.

When does MongoDB's flexible schema actually pay off?

When your data is genuinely variable in structure, different records honestly have different shapes, not just different values and when your application's data model is still actively evolving during early development, rather than pretending its shape is settled.

Does using MongoDB mean I don't need to think about data structure?

No. Skipping schema design just moves the responsibility for consistency into your application code, where it's genuinely easier to get wrong, not somewhere it disappears.

How do interviewers evaluate this topic in system design interviews?

They're typically testing whether you can reason about a specific access pattern and justify a choice for it, not whether you have a fixed database preference. Being able to explain a trade-off, the way Shippable's and Agenta's real migrations illustrate, matters more than reciting definitions.

Measure Your Own Access Patterns

The real lesson from Shippable's outage and Agenta's speedup isn't "always pick one," it's that both teams made their decision based on their actual, measured access patterns, not a general reputation. Explore the Data Engineer roadmap to build the relational and document-modeling skills that let you make that same kind of evidence-based call on your own projects.

JSON vs. Protocol Buffers vs. Avro: 60% Less Latency, 80% Smaller Payloads, One Serialization Decision

Ciphemic academia — Thu, 01 Oct 2026 03:01:37 +0000

JSON vs. Protocol Buffers vs. Avro: 60% Less Latency, 80% Smaller Payloads, One Serialization Decision

LinkedIn's engineering team moved their internal microservice traffic off JSON and onto Protocol Buffers, and measured a 60% drop in service latency, without touching their underlying REST framework at all, just swapping the format carrying the data. Atlassian ran the same kind of migration on part of Jira's backend and cut that cluster's data size by 80% and its CPU usage by 75%, enough to shrink the cluster itself by more than half. Neither team changed what their services did. They changed how the bytes on the wire were shaped, and that single decision showed up directly on their infrastructure bill.

This is what a serialization format actually is: the rules for turning your in-memory data into bytes that can cross a network or sit on disk, and back again. JSON, Protocol Buffers (protobuf), and Avro all do this, but they make genuinely different trade-offs, and picking the wrong one for your situation is exactly how a team ends up re-doing this kind of migration two years in. This guide breaks down what each format actually does under the hood, where the real numbers come from, and how to decide before you've shipped a year of JSON APIs you'll later have to migrate off.

If you're newer to how data moves through a system at all, our Binary & Encoding course covers the fundamentals this comparison builds on.

This post originally appeared on the Ciphemic Academia blog.

The Short Version

JSON is a human-readable, text-based format with no required schema. Universally supported, easy to debug, and the default most APIs still start with.
Protocol Buffers is a binary, schema-first format from Google. Smaller and faster than JSON, with generated, type-safe code in whatever language you need.
Avro is a binary, schema-first format from the Apache Hadoop ecosystem, built around strong schema evolution and tight integration with big-data pipelines like Kafka.

If you want one default: start with JSON. Move to Protobuf when you're optimizing internal service-to-service traffic for size and speed. Move to Avro specifically when you're in a data pipeline or event-streaming context where schema evolution across many producers and consumers is the real problem.

What All Three Are Actually Doing

Before the differences, the shared job, since this is what every serialization format exists to do:

Encoding: turning an in-memory object (a struct, a dict, a record) into a sequence of bytes
Decoding: turning those bytes back into a usable object on the other end
Defining structure: some way of saying "this data has a name field that's a string and an age field that's a number," whether that's enforced strictly or left loose
Handling change over time: every real system eventually adds a field, removes one, or renames one, and each format has a different answer for what happens to old data and old readers when that happens

The size and speed numbers above come directly from how differently each format approaches that last point, structure, and that first point, encoding.

JSON

What it actually is: a text-based format where data is represented as nested objects, arrays, strings, numbers, and booleans, written in a syntax most developers already know from JavaScript. There's no required schema, any JSON parser can read any valid JSON, and every field name is spelled out, in full, every single time it appears.

A small example:

{
  "id": 482,
  "customer_name": "Alex Chen",
  "total": 129.99,
  "items": [{"sku": "A1", "qty": 2}]
}

Notice "customer_name" and "sku" and "qty" are written out as text, every time, in every single record. That repetition is exactly what LinkedIn's and Atlassian's teams were measuring when they counted bytes.

Where it shines:

Universal support: every language, every tool, every browser can read and write JSON with zero setup
Human-readable: you can open a JSON payload in a text editor and understand it immediately, which makes debugging dramatically faster
No schema required: you can start sending data immediately without agreeing on a formal structure in advance
Massive tooling ecosystem: logging, API testing tools, documentation generators, and nearly every web framework assume JSON by default

Where it struggles:

Verbose by design: repeating every field name in every record is exactly where JSON's size disadvantage comes from, and it compounds at scale
No enforced structure: without a schema, nothing stops a producer from silently sending "total": "129.99" as a string instead of a number, and that mismatch surfaces as a runtime bug, not a build-time error
Slower to parse: text parsing is inherently more work for a CPU than reading dense binary fields, which is where real latency differences like LinkedIn's come from
No native schema evolution strategy: adding or changing fields is informal, and nothing enforces backward or forward compatibility between old and new versions of your data

Who this suits: public APIs, anything where human-readability and zero-setup compatibility matter more than raw size or speed, and the correct default for most new projects.

Protocol Buffers

What it actually is: a binary, schema-first format where you define your data's structure in a .proto file, then generate code in your target language from it. Fields are identified by small numeric tags instead of full names, so the wire format only sends 1: 482 instead of "id": 482, which is most of where the size savings come from.

A small example (a .proto schema):

message Order {
  int32 id = 1;
  string customer_name = 2;
  double total = 3;
  repeated Item items = 4;
}

The numbers (= 1, = 2, = 3) are the actual field tags sent on the wire, not the field names. That's the core trick: the schema lives in the .proto file, shared and version-controlled, not repeated in every message.

Where it shines:

Smaller payloads: encoding Avro and Protobuf against compressed JSON in one published benchmark across varied real-world data showed Protobuf cutting an additional 67% to 80% off the already-compressed JSON size, which is the same category of saving behind Atlassian's 80% figure
Faster serialization: binary encoding with numeric tags is cheaper for a CPU to produce and parse than text parsing, directly contributing to the latency gains LinkedIn measured
Strong, enforced contracts: the .proto file generates type-safe client and server code, catching structural mismatches at compile time instead of in production
Built-in schema evolution rules: adding a new field with a new tag number is safe by design, old readers simply ignore fields they don't recognize

Where it struggles:

Not human-readable: you can't open a Protobuf payload in a text editor and make sense of it; you need the schema and the right tooling
Requires a build step: every service needs the generated code regenerated whenever the .proto file changes, which is real workflow overhead compared to JSON's zero-setup nature
Smaller talent pool: fewer developers have hands-on Protobuf experience than JSON, which is a real onboarding cost for a team
Awkward for ad-hoc, human-facing debugging: inspecting a raw Protobuf message during an incident takes more steps than just reading JSON in a browser

Who this suits: internal service-to-service communication, performance-sensitive APIs, gRPC-based microservices, and any system where LinkedIn's or Atlassian's exact problem, too much JSON overhead at real scale, has already shown up as a real bottleneck.

Avro

What it actually is: a binary, schema-first format from the Apache Hadoop ecosystem, where the schema is defined in JSON (yes, Avro's own schema definition is JSON) and typically travels alongside the data itself, either embedded in the file or stored centrally in a schema registry. This makes Avro especially well-suited to systems like Kafka, where many independent producers and consumers need to agree on evolving data shapes over time.

A small example (an Avro schema, defined in JSON):

{
  "type": "record",
  "name": "Order",
  "fields": [
    {"name": "id", "type": "int"},
    {"name": "customer_name", "type": "string"},
    {"name": "total", "type": "double"}
  ]
}

Unlike Protobuf, there are no manually-assigned numeric tags. Avro instead relies on the schema itself, often paired with a schema registry, to resolve differences between what a writer encoded and what a reader expects.

Where it shines:

Strong schema evolution model: Avro's reader/writer schema resolution is specifically designed for the case where producers and consumers are updated independently and at different times, which is extremely common in streaming pipelines
Compact size: since the schema isn't repeated in every record, Avro files are genuinely small; one published benchmark measured Avro at 100% size reduction against uncompressed JSON in its best cases and still meaningfully ahead of compressed JSON in realistic ones
Deep ecosystem fit: Avro is the default serialization format in much of the Hadoop and Kafka world, with first-class support in tools like Kafka Connect and Confluent's Schema Registry
Splittable for big-data processing: Avro files work naturally with distributed processing frameworks that need to split large files into chunks

Where it struggles:

Schema must be available to decode: unlike Protobuf's self-describing numeric tags, you genuinely cannot read Avro data without its schema, which makes a schema registry close to mandatory in production
Less common outside data engineering: Avro is comparatively rare in typical web backend or mobile API contexts, which narrows where this specific skill transfers
Real-world performance can disappoint without the right library: one published .NET benchmark found a specific Avro library performing worse than plain JSON serialization, a reminder that library quality, not just format choice, affects your real numbers
Smaller general tooling ecosystem outside its niche: compared to Protobuf's broad gRPC ecosystem, Avro's tooling is concentrated specifically around data pipelines

Who this suits: Kafka-based event streaming, Hadoop-ecosystem data pipelines, and any system where many independent teams produce and consume evolving data over a long time horizon.

Side-by-Side Comparison

	JSON	Protocol Buffers	Avro
Format type	Text	Binary	Binary
Schema	None required	Required, in `.proto` files	Required, defined in JSON
Human-readable	Yes	No	No
Typical size vs. JSON	Baseline	Significantly smaller	Significantly smaller
Schema evolution	Informal, unenforced	Strong, via numeric tags	Strong, via reader/writer resolution
Needs schema to decode	No	No (tags are self-describing)	Yes, close to mandatory
Best ecosystem fit	Web APIs, public APIs	gRPC, internal microservices	Kafka, Hadoop, data pipelines
Setup overhead	None	Build step, code generation	Schema registry, typically

How This Fits a Career Path

Backend Engineer (general): JSON fundamentals are non-negotiable, since nearly every web API still uses it, and understanding its real costs is what makes the case for anything else legible
Microservices or platform engineer: Protobuf becomes directly relevant once internal API performance is a real, measured problem, not a hypothetical one; our API Design course covers where this decision fits alongside REST and gRPC more broadly
Data Engineer: Avro is close to a baseline expectation in Kafka and Hadoop-adjacent roles, and understanding schema registries and reader/writer resolution is a real, frequently-tested skill
System design interviews: be ready to justify a serialization choice with the same kind of reasoning LinkedIn and Atlassian used, a specific, measured problem, not a format preference

How to Choose Without Overthinking It

Start with who's reading your data. Public clients, browsers, or third parties who need to debug payloads easily? JSON. Internal services you control end to end? Protobuf or Avro become worth considering.
Look for an actual, measured bottleneck before migrating. LinkedIn and Atlassian didn't start with Protobuf, they measured a real cost in JSON first. Don't rewrite a working API because a format sounds more advanced.
If you're in a streaming or big-data pipeline, lean Avro. Its schema evolution model is specifically built for many independent producers and consumers changing over time, which is exactly the Kafka and Hadoop situation.
If you're optimizing internal RPC calls, lean Protobuf. Its tooling, gRPC integration, and broader adoption outside data engineering make it the more generally useful binary format to learn first.

A note on honesty: the specific percentages above came from real, published migrations and benchmarks, but your numbers will depend heavily on your actual data shape, field count, and library choice, the .NET Avro benchmark above found a specific library that underperformed plain JSON. Measure your own system before committing to a number you read in someone else's blog post, including this one.

Common Mistakes When Learning Serialization

Reaching for a binary format before you have a measured problem. Protobuf and Avro both add real setup cost; that cost needs to be paying for something concrete.
Treating "schema-first" as optional busywork. Skipping proper schema evolution planning is exactly how teams end up with the kind of incompatible-message incidents these formats exist to prevent.
Assuming all binary formats perform the same. As the .NET benchmark above shows, library quality within a format matters as much as the format itself, benchmark your actual stack, not someone else's blog post.
Ignoring human-debuggability costs. Moving everything to Protobuf or Avro without keeping any human-readable path (even just for internal debugging) makes incident response measurably harder.
Never revisiting the decision. A format chosen for a small early-stage service can become a real bottleneck at scale, as both LinkedIn's and Atlassian's stories show, revisit the decision when the data actually tells you to.

Frequently Asked Questions

Should a beginner learn JSON, Protobuf, or Avro first?

JSON. It underlies most web development you'll touch early on, requires no extra tooling, and understanding its real limitations is exactly what makes Protobuf and Avro make sense later.

Is Protobuf always faster than JSON?

In the vast majority of measured cases, yes, both in size and in CPU cost to encode and decode, which is exactly what drove LinkedIn's 60% latency improvement. But the margin depends on your specific data shape and libraries, so measuring your own case is still worth doing before committing.

Why would I choose Avro over Protobuf if they're both binary and schema-first?

Avro's reader/writer schema resolution is specifically built for situations where many independent producers and consumers evolve their data shape over time without perfect coordination, which is the normal state of a large Kafka deployment. Protobuf's numeric-tag approach to evolution is simpler but assumes slightly tighter coordination through shared .proto files.

Do I need a schema registry if I use Avro?

In production, close to always yes. Since Avro data generally cannot be decoded without its schema, a central schema registry (like Confluent's) is the standard way teams keep producers and consumers in sync.

Can I mix formats in one system?

Yes, and real systems often do, JSON facing external clients, Protobuf for internal gRPC services, Avro inside a Kafka pipeline. These aren't mutually exclusive choices for an entire company.

How much engineering effort does a JSON-to-Protobuf migration actually take?

Atlassian's and LinkedIn's write-ups both describe incremental, no-downtime migrations run over real engineering time, not a weekend project, typically involving dual-format support during the transition. Budget for genuine migration work, not a quick swap.

Measure Before You Migrate

The real lesson from LinkedIn's and Atlassian's numbers isn't "always use Protobuf," it's that they measured a specific, real cost before changing anything. Explore the Binary & Encoding course to understand what's actually happening on the wire in your own systems, so your next format decision is based on your own numbers, not someone else's blog post.

JWT vs. Sessions: Which Authentication Should You Use?

Ciphemic academia — Wed, 30 Sep 2026 01:48:06 +0000

JWT vs. Sessions: Which Authentication Should You Use?

Every backend project eventually needs to answer: how do we know this request is really from a logged-in user? Two answers dominate real-world systems: session-based authentication, where the server keeps track of who's logged in, and JWT (JSON Web Token) authentication, where the server hands the client a signed token that proves who they are. Tutorials often pick one without explaining why, and beginners frequently pick JWTs simply because they sound more modern. This guide compares both honestly, so you can choose based on your actual system, not on which one trended on Twitter.

If you haven't covered the basics of hashing, tokens, and identity yet, our Auth & Identity course builds that foundation before you need to make this call on a real project.

This post originally appeared on the Ciphemic Academia blog.

The Short Version

Sessions store login state on the server, usually in a database or memory store, and give the browser a small session ID cookie to reference it.
JWTs store login state inside a signed token itself, which the client holds and sends with every request, so the server can verify it without looking anything up.

If you want one default recommendation for a typical web app with a browser frontend and a single backend: use sessions. They're simpler to reason about, easier to revoke, and match how browsers already work. JWTs earn their place in specific situations this guide covers below.

What Both Are Actually Trying to Solve

Before the differences, the shared goal, since it's easy to lose sight of in the JWT-vs-sessions debate:

Identity across requests: HTTP itself is stateless, so both approaches exist to answer "who is this?" on every single request
Tamper resistance: neither approach is useful if a user can just edit their own identity and claim to be someone else
Expiration and logout: both need a real answer for what happens when a login should stop being valid
Secure transport: both are only as safe as the connection carrying them, which is exactly why HTTPS is non-negotiable for either approach

Get these fundamentals right, and the specific mechanism you choose matters less than doing it correctly.

Session-Based Authentication

What it actually is: when a user logs in, the server creates a session record, typically in a database or an in-memory store like Redis, containing the user's ID and any relevant data. The server then sends the browser a session ID in a cookie. On every subsequent request, the browser automatically sends that cookie back, and the server looks up the session to know who's making the request.

A small example (the flow, simplified):

1. User logs in with email/password
2. Server creates a session record: { sessionId: "abc123", userId: 42 }
3. Server sets a cookie: Set-Cookie: sessionId=abc123; HttpOnly; Secure
4. Browser sends that cookie automatically on every future request
5. Server looks up "abc123" to find userId 42

Where it shines:

Instant revocation: deleting the session record on the server immediately logs the user out everywhere, with no waiting
Smaller attack surface for stolen tokens: a session cookie set with HttpOnly can't be read by JavaScript, which blocks a whole category of token-theft attacks
Simple mental model: the server is the single source of truth for who's logged in, and there's nothing to decode or verify beyond a database lookup
Well-suited to traditional web apps: if your frontend and backend are closely coupled and served from the same domain, sessions and cookies work with the browser's behavior, not against it

Where it struggles:

Requires server-side storage: every request needs a lookup, which means a shared session store (like Redis) is necessary once you have more than one backend server
Harder across domains and native apps: cookies don't travel cleanly to a separate API domain or a mobile app the way they do to a same-site browser app, often requiring extra configuration
Scaling considerations: a session store adds an extra piece of infrastructure to run and keep available

Who this suits: traditional web applications, server-rendered apps, and any system where the frontend and backend are tightly coupled and mostly serving browsers.

JWT (Token-Based) Authentication

What it actually is: when a user logs in, the server creates a JWT, a signed, self-contained piece of data (typically containing the user's ID and an expiration time), and hands it to the client. The client stores this token and sends it with every request, usually in an Authorization header. The server verifies the token's signature to confirm it's legitimate, without needing to look anything up in a database.

A small example (the flow, simplified):

1. User logs in with email/password
2. Server creates a signed JWT: { userId: 42, exp: 1735689600 }
3. Client stores the token and sends it on every request:
   Authorization: Bearer eyJhbGciOi...
4. Server verifies the signature and expiration, no database lookup needed

Where it shines:

Stateless verification: the server doesn't need a shared session store, since the token itself carries everything needed to verify it, which simplifies scaling across multiple servers
Works well across domains and platforms: a mobile app, a single-page app on a different domain, and a third-party API can all use the same token-based approach cleanly
Good fit for microservices: any service that has the shared signing key can verify a token independently, without calling a central session service

Where it struggles:

Revocation is genuinely hard: once a JWT is issued, it's valid until it expires, and there's no built-in way to invalidate it early without adding extra infrastructure (a token blocklist), which undercuts the "stateless" benefit
Token theft is more dangerous: if a JWT is stolen, usually via cross-site scripting, since it's often stored in a way JavaScript can access, it works until it expires with nothing on the server actively checking whether it should
Larger payload: a JWT is bigger than a simple session ID cookie, since it carries actual data, which slightly increases every request's size
Easy to misuse: it's common to see JWTs stored insecurely, given overly long expiration times, or used without proper signature verification, which negates their security benefits entirely

Who this suits: APIs consumed by mobile apps or third-party clients, microservices architectures where multiple independent services need to verify identity, and systems genuinely spanning multiple domains.

Side-by-Side Comparison

	Sessions	JWT
Where state lives	Server (database or memory store)	Client (inside the token itself)
Revocation	Instant, delete the record	Hard, requires extra infrastructure
Scaling across servers	Needs a shared session store	Naturally stateless
Cross-domain and mobile use	Requires extra configuration	Works naturally
Payload size	Small (just an ID)	Larger (carries actual data)
Common vulnerability	Session fixation, CSRF if misconfigured	XSS-driven token theft, long-lived stolen tokens
Best for	Traditional web apps, browser-based apps	APIs, mobile clients, microservices

How This Fits a Career Path

Backend Engineer (general): understand sessions deeply first, since they map directly onto how HTTP and browsers actually work, and most entry-level roles involve maintaining systems that already use one or the other
API or platform engineer: JWTs become directly relevant once you're building APIs consumed by multiple clients or services, so understanding their revocation trade-offs matters for real production systems
Security-focused roles: know both deeply enough to explain their specific failure modes, since interviewers in security-adjacent roles frequently probe this exact trade-off; see our Penetration Testing course for how these vulnerabilities get exploited in practice
Full-stack roles: you'll likely touch both, sessions for a traditional app, JWTs when your app talks to a separate API or third-party service

How to Choose Without Overthinking It

Start with your architecture. One frontend, one backend, same domain? Sessions are simpler and safer by default. Multiple clients, multiple services, or a public API? JWTs fit more naturally.
Consider how much revocation matters to you. If instantly killing a compromised login matters a lot (banking, admin panels), sessions' instant revocation is a real advantage worth the extra infrastructure.
Don't choose JWTs "because they're stateless" without a stateless need. If you don't actually have multiple independent services or servers needing to verify tokens, you're taking on JWT's revocation problems for a scaling benefit you're not using yet.
Consider hybrid approaches. Some real systems use short-lived JWTs alongside a server-side refresh token that can be revoked, combining benefits of both. This adds complexity, so only reach for it once you understand why.

A note on honesty: JWTs aren't "more secure" or "more modern" by default, they solve a different problem than sessions do. Plenty of production systems, including large, security-conscious ones, use sessions successfully. The right choice depends on your architecture, not on which one is newer.

Common Mistakes When Learning Authentication

Storing JWTs in localStorage without understanding the risk. This is a common but genuinely risky pattern, since it exposes the token to any successful XSS attack. HttpOnly cookies are safer for either approach when possible.
Giving JWTs excessively long expiration times. A long-lived, unrevokable token is a much bigger liability than a short-lived one paired with a refresh mechanism.
Forgetting CSRF protection with cookie-based sessions. Cookies sent automatically by the browser need CSRF defenses; this is a real, well-known vulnerability class, not a hypothetical one.
Rolling your own token signing or session logic. Both approaches have mature, well-tested libraries. Writing this from scratch is a common source of real vulnerabilities for learners and production systems alike.
Treating authentication as a checkbox instead of an ongoing responsibility. Token expiration policies, revocation, and secure storage all need periodic review, not just a correct initial implementation.

Frequently Asked Questions

Should a beginner learn sessions or JWTs first?

Sessions. They map directly onto how HTTP and cookies actually work, which builds the right mental model before adding a signed-token abstraction on top.

Are JWTs insecure?

Not inherently, but they're commonly misused, long expiration times, insecure storage, and skipped signature verification are the real sources of JWT vulnerabilities, not the token format itself.

Can I revoke a JWT before it expires?

Not natively. Doing so requires extra infrastructure, like a server-side blocklist of revoked tokens, which reintroduces some of the server-side state JWTs were meant to avoid.

Do most real companies use sessions or JWTs?

Both, often in the same company for different purposes, sessions for a main web app, JWTs for a public API or mobile app. Neither has fully replaced the other in real production use.

What's a refresh token, and how does it relate to this?

A refresh token is a longer-lived, securely stored credential used to issue new short-lived JWTs without requiring the user to log in again. It's a common pattern for getting JWT convenience with better control over long-term access.

Which one comes up more in interviews?

Both, but interviewers are typically more interested in whether you understand the trade-offs (revocation, statelessness, storage risk) than in which one you'd default to. Being able to justify a choice for a given scenario matters more than a fixed preference.

Build Authentication That Actually Holds Up

Understanding sessions and JWTs well means being able to explain exactly what breaks if either is implemented carelessly, not just how each one works in theory. Explore the Auth & Identity course to build real authentication flows and see these trade-offs firsthand.