What Replacing Calendly Taught Me About Trusting Open Source

cal.com, Calendly, zcal... booking SaaS isn't short on options, and most of them are genuinely decent. Free tiers cover the basics for a lot of freelancers. The catch: you're the product (nothing's really free), and your customer data lives somewhere you don't fully control and can't fully audit.

A dysfunction I ran into on another SaaS tool was the trigger. Trusting a third-party service by default, just because it's widely used and billed monthly, doesn't always hold up. That episode was enough to make me reconsider every external service this site was relying on for functionality that's actually simple to self-host — and the booking widget, running on Calendly, was one of them.

Nothing wrong with Calendly specifically. It worked fine. But structural friction had been building regardless: a recurring subscription for something as simple as displaying open slots and recording a choice, a hard dependency on a third party for a component with nothing exceptional about it technically, and customization capped by whatever the vendor exposes in settings — no way to go further if a need falls outside that box. On top of that, an integration constraint that mattered more than any of the above: the site runs on Astro, generating lightweight static pages by design, specifically to avoid the weight of third-party scripts and dependencies — the exact opposite of what embedding a SaaS widget implies.

So: could a self-hosted alternative match the experience, without the monthly bill and without handing a core commercial function (people booking a call with me) to an external vendor? This is the write-up of that search, the codebase audit that came out of it, and the production rollout.

The landscape

Four self-hosted candidates stood out as genuinely comparable — not just UI skins sitting on top of someone else's API, not just internal-scheduling tools with the public-facing UX as an afterthought.

CloudMeet — Svelte + TypeScript, deployed on Cloudflare Pages/Workers/D1, free-tier friendly. MIT licensed. Clean booking UX. Single maintainer, ~490 stars, 37 commits — young, not enough track record to trust blind.

Cal.diy — the community fork of cal.com's booking engine, spun up after cal.com closed-sourced their core product in April 2026, citing security risk from AI-assisted code scanning against their public repo. MIT licensed, maintained by former cal.com interns. Full scheduling engine, app-store integrations, Stripe/PayPal payment support carried over. Most feature-complete on paper. Also the youngest as an independent community project — their own docs still discourage production use without caveats.

booking-calendar — React + TypeScript + Bun + SQLite, single-admin design, native bidirectional CalDAV sync instead of a Google/Outlook lock-in. Clean architecture (repository/service/entity separation via TypeORM). Lightweight, portable by design. Booking UI is a scrollable list of time slots — functional, but nowhere near the polish of a commercial scheduler.

Easy!Appointments — PHP/CodeIgniter + MySQL, ten years of active development, 3000+ stars, a paid tier that's existed for years. Most battle-tested of the four, and the one with the closest booking UX to Calendly itself: monthly calendar view, multi-step wizard.

What I was actually optimizing for: framework-agnostic integration (no PHP running on the public site itself), booking UX quality, native GDPR consent handling, and enough production maturity to put in front of real prospects rather than an experimental project.

Narrowing down

CloudMeet and Cal.diy dropped out early, for the same underlying reason: not enough track record for production use, not a specific flaw found. CloudMeet's single-maintainer status and short commit history made it too much of a bet. Cal.diy's own documentation still hedges on production readiness three months post-launch. Whether either becomes a serious contender or stays a one-shot project is a question for later.

Worth being honest about what that is: a decision made on reputation and project age, the exact shortcut this piece argues against later on. Auditing four codebases in the same depth as the two finalists below wasn't a realistic use of time, so CloudMeet and Cal.diy got a lighter pass — young-project heuristics instead of a source read. That's a real gap in the method, not just a caveat to mention in passing.

That left booking-calendar and Easy!Appointments. And this is where the UX gap tipped it: booking-calendar's public booking page is a plain scrollable list of Monday, February 23, 2026 at 08:00 AM rows — accurate, but visually miles from what Calendly conditioned people to expect. No month view, no staged flow, just text to scroll through.

I briefly considered franken-stacking the two — CloudMeet's frontend on top of booking-calendar's CalDAV backend, or something along those lines. Not viable in practice: different runtimes (Cloudflare Workers/D1 vs Bun/SQLite), no shared API contract, no shared data model. Bolting two incompatible stacks together usually creates more work than picking one and adapting it. Went with separation of concerns instead: pick the tool with the better public-facing UX, and if native CalDAV sync becomes a real need later, that's a second, independent tool — not a merge.

Easy!Appointments won on UX. Onto the part that actually mattered for a production decision.

The audit: a TOCTOU race condition, in both remaining candidates

Before committing to either, I wanted to check one specific thing: what happens if two visitors click the same slot at nearly the same instant? Not a theoretical concern — it's the kind of bug that never shows up in solo development and blows up the day a booking link gets shared a bit wider (a newsletter blast, a LinkedIn post, a batch of new slots opening at a fixed time).

This is a classic TOCTOU (time-of-check to time-of-use) race condition: the code checks that a slot is free, then writes the booking — and nothing stops a concurrent request from doing the exact same check in between, before either write lands. Without a lock spanning both the check and the write, two requests can both conclude "free" before either commits.

booking-calendar

TypeORM-based, repository pattern, otherwise clean separation of concerns. The overlap check:

async hasOverlapInSlot(
    slotId: number,
    startAt: string,
    endAt: string,
    manager?: EntityManager,
): Promise<boolean> {
    const count = await this.repo(manager)
        .createQueryBuilder("a")
        .where("a.slot_id = :slotId", { slotId })
        .andWhere("a.canceled_at IS NULL")
        .andWhere("a.status != 'rejected'")
        .andWhere("NOT (a.end_at <= :startAt OR a.start_at >= :endAt)", {
            startAt,
            endAt,
        })
        .getCount();

    return count > 0;
}

Called inside a transaction (AppDataSource.transaction), but no .setLock("pessimistic_write"), no exclusion constraint at the schema level. Under SQLite, this never surfaces: the engine serializes writers, one at a time, by design. It's a free safety net courtesy of the storage engine — not a guarantee the application code actually enforces. The project's own architecture explicitly anticipates a migration path to Postgres or MySQL via TypeORM's driver abstraction (type: "sqlite" → type: "postgres", straightforward on paper). That's exactly the migration that removes the net: under READ COMMITTED isolation, two concurrent transactions can each read "no overlap" before either one's INSERT commits.

Confirmed by reading the CalDAV sync layer too — a second, independent race, this one against the external calendar rather than the local database:

private async getCachedBusyIntervals(startAt: string, endAt: string): BusyInterval[] | null {
    const cache = CalDAVService.busyIntervalCache;
    if (!cache || cache.expires_at <= Date.now()) {
        return null;
    }
    if (startAt < cache.start_at || endAt > cache.end_at) {
        return null;
    }
    return cache.intervals.filter(
        (interval) => !(interval.end_at <= startAt || interval.start_at >= endAt),
    );
}

A process-wide static cache with a TTL, invalidated only after a successful write. Two bookings arriving seconds apart, inside the cache window, can both read the same "free" snapshot before either has finished writing to the external calendar — the exact same TOCTOU shape, just with the external CalDAV server as the source of truth instead of the local DB.

Easy!Appointments

Ten years in production, 3000+ GitHub stars, a paid tier that's existed for years. The working hypothesis going in: more real-world traffic means more chances this exact class of bug already got hit and fixed. That hypothesis doesn't survive reading the code.

The public booking flow:

// Check appointment availability before registering it to the database.
$appointment['id_users_provider'] = $this->check_datetime_availability();

if (!$appointment['id_users_provider']) {
    throw new RuntimeException(lang('requested_hour_is_unavailable'));
}

// ... customer lookup/creation, GDPR consent records, Jitsi link generation ...

$appointment_id = $this->appointments_model->save($appointment);

Several unrelated operations sit between the check and the write — the race window here is wider than booking-calendar's, where check and insert at least shared a transaction. And the insert itself:

protected function insert(array $appointment): int
{
    $appointment['book_datetime'] = date('Y-m-d H:i:s');
    $appointment['create_datetime'] = date('Y-m-d H:i:s');
    $appointment['update_datetime'] = date('Y-m-d H:i:s');
    $appointment['hash'] = random_string('alnum', 12);

    if (!$this->db->insert('appointments', $appointment)) {
        throw new RuntimeException('Could not insert appointment.');
    }

    return $this->db->insert_id();
}

No transaction, no lock, no unique constraint. The docblock on check_datetime_availability() reads:

"It is possible that two or more customers select the same appointment date and time concurrently. The app won't allow this to happen."

Documented intent, not what the code actually does.

The interesting part

Appointments_model contains a method that does the correct overlap check, with a properly built query:

public function has_provider_conflict(
    int $provider_id,
    string $start_datetime,
    string $end_datetime,
    ?int $exclude_appointment_id = null,
): bool {
    $this->db->select('id')->from('appointments')->where('id_users_provider', $provider_id);

    if ($exclude_appointment_id) {
        $this->db->where('id !=', $exclude_appointment_id);
    }

    // Overlap: (existing_start < new_end) AND (existing_end > new_start)
    return $this->db
        ->group_start()
        ->where('start_datetime <', $end_datetime)
        ->where('end_datetime >', $start_datetime)
        ->group_end()
        ->get()
        ->num_rows() > 0;
}

It's never called anywhere in the public booking flow. The correct primitive exists in the codebase; it's just not wired in where it would matter.

Takeaway

Neither project is "more robust" than the other on this specific point — both share the same design gap, independent of relative maturity. Reputation, age, an established commercial tier: reasonable statistical priors, not proof. The only way to know whether an open-source project actually guards against this class of bug is to read the code doing the work, not the comment claiming it does.

For what it's worth, the fix on the Easy!Appointments side is close to a drop-in — the correct primitive (has_provider_conflict) already exists, it's a matter of wiring it in with a lock around check+write rather than designing a new one:

$lock_name = "provider_{$provider_id}_booking";

if (!$this->db->query("SELECT GET_LOCK(?, 10)", [$lock_name])->row()->{"GET_LOCK(?, 10)"}) {
    throw new RuntimeException('Could not acquire booking lock.');
}

try {
    if ($this->appointments_model->has_provider_conflict(
            $appointment['id_users_provider'],
            $appointment['start_datetime'],
            $appointment['end_datetime']
        )) {
        throw new RuntimeException(lang('requested_hour_is_unavailable'));
    }
    $appointment_id = $this->appointments_model->save($appointment);
} finally {
    $this->db->query("SELECT RELEASE_LOCK(?)", [$lock_name]);
}

GET_LOCK rather than a schema-level exclusion constraint because MySQL has no equivalent to Postgres' EXCLUDE USING gist — no declarative way to say "no two overlapping ranges for this provider" at the database level. The lock key is scoped per-provider (provider_{id}_booking) rather than global, so two visitors booking with two different providers at the same instant don't block each other for no reason. RELEASE_LOCK sits in a finally so a failed write doesn't leave the lock held until timeout.

One dependency this fix quietly assumes: non-persistent database connections. GET_LOCK is scoped to the MySQL session, not the PHP request — it lives and dies with the connection. That holds cleanly on a standard non-persistent connection (the CodeIgniter default), where each request gets its own connection and the lock disappears cleanly when it ends, even on an uncaught fatal. It stops holding if pconnect is enabled and connections get reused across unrelated requests from a pool: RELEASE_LOCK in the finally might release a lock a different request just acquired on the same recycled connection, or a lock could outlive the request that took it. Worth stating explicitly in the patch rather than assuming, since it's not something has_provider_conflict() or the surrounding code makes obvious either way.

This didn't end up shipping in my deployment — the LOCK/VERIFY/WRITE/UNLOCK skeleton above is close to production-ready, but I'd want load-test coverage on the ANY_PROVIDER branch (where the code searches for any available provider — the lock needs to span that search too, or two "any provider" requests can still land on the same provider/slot in parallel) before calling it done. Worth a PR upstream at some point.

Production rollout: a simple embed, an unexpected block

The integration itself was straightforward: Easy!Appointments runs on its own subdomain, the contact page just drops an <iframe> pointing at it. No PHP touches the public Astro site at all — that separation was the whole point.

First test after deploying: the iframe wouldn't render. Console error:

Refused to display 'https://cal.example.com/' in a frame because it set
'X-Frame-Options' to 'sameorigin'.

First guess, wrong

Given the server setup (a hosting panel with a reverse-proxy layer in front of the site), the obvious first suspect was that layer, not the app itself. Tried unsetting the header at the .htaccess level:

<IfModule mod_headers.c>
    Header always unset X-Frame-Options
    Header always set Content-Security-Policy "frame-ancestors 'self' https://backstage.click"
</IfModule>

No effect. Header still showed up on curl -I.

The actual cause

Grepping the Easy!Appointments source turned it up directly:

./application/hooks/security_headers.php:    header('X-Frame-Options: SAMEORIGIN');
./application/config/routes.php:header('X-Frame-Options: SAMEORIGIN');

The app sets the header itself, in PHP, at two separate points in its own bootstrap — a hardcoded anti-clickjacking default, sensible for the admin panel, applied indiscriminately to every route including the public booking page that's meant to be embedded elsewhere. Header unset in .htaccess runs at the web-server response-table level, ahead of the PHP process; a header() call executed later by the script itself simply overrides it. The .htaccess fix couldn't have worked against this, structurally, regardless of server (Apache, Nginx, OpenLiteSpeed) — PHP has the last word on its own headers as long as output hasn't started.

The working fix patches both call sites, scoped to the booking controller only so the admin panel keeps its default protection:

$CI =& get_instance();
if (get_class($CI) === 'Booking') {
    header("Content-Security-Policy: frame-ancestors 'self' https://backstage.click");
} else {
    header('X-Frame-Options: SAMEORIGIN');
}

Using the instantiated controller class rather than parsing $_SERVER['REQUEST_URI'] — this install doesn't have clean URLs enabled (index.php shows up in the booking URL), so a naive strpos($uri, '/booking') === 0 check would silently fail to match.

One thing worth flagging for anyone patching the same two files: they're core application code, not a plugin layer. A future git pull or a Docker image rebuild will silently overwrite this fix. Worth keeping it as a versioned patch file (git diff before/after) to reapply after upgrades, rather than losing it the next time the container gets rebuilt.

Result

Customer data (name, email, meeting reason) staying on infrastructure I control instead of a third party's, and the site's initial page weight untouched — no third-party script added for this one feature. Cost wasn't really the driver here; free tiers cover the basics for a lot of freelancers, mine included. What I was buying back wasn't a subscription fee, it was the part where a vendor decides what "the basics" are, and where my prospects' contact details end up.

Worth noting: which of the four tools actually fits depends entirely on what a given business needs, not on which one "won" here. Easy!Appointments, for instance, is also a solid self-hosted alternative to Bookly for anyone running WordPress and looking to drop a booking plugin subscription — different starting point, same underlying question.

But the tool swap itself isn't really the point. This is one instance of a pattern I keep coming back to: default to self-hosted where it's reasonable, and don't let "widely used" or "ten years old" or "has a paid tier" stand in for actually checking. Reputation is a prior, not a verdict. The Easy!Appointments audit is the clearest example in this piece — a decade of production traffic, a commercial offering, thousands of stars, and a race condition sitting in the exact code path that mattered most, with the correct fix already written elsewhere in the codebase and simply never called. Maturity didn't catch it. Reading the code did.

Here's the part worth sitting with, though: the fix never shipped on my own deployment either. It's sketched out above, unfinished, blocked on load-testing I haven't done. Which means the instance running this site's booking page right now is, as far as I know, still exposed to the exact TOCTOU window this whole piece is about. Self-hosting bought me visibility into that gap — Calendly would have hidden it behind a vendor's SLA and I'd have had no way to know either way. It didn't buy me the fix. That's a separate piece of work, and skipping it doesn't get excused by having found the bug in the first place. Self-hosting gets you control. Security still has to be built, on your own time, by someone — and until that patch actually lands, that someone is a task on my list, not a claim in this article.

Neither was this the fastest of the four options, or the most obvious. It took a real comparison between projects, a debugging detour once in production, and time spent reading source instead of trusting a README. That's the actual cost of running your own infrastructure instead of renting someone else's.

Comments

[Nazar Boyko]

On the MySQL side you might get away without GET_LOCK for the common case. Appointment slots come off a fixed grid rather than arbitrary ranges, so a unique index on (id_users_provider, start_datetime) makes the second insert fail on its own and you turn the duplicate key error into the same "unavailable" message. Doesn't help with the ANY_PROVIDER search, but there's no lock lifetime to reason about either.

[Pascal CESCATO]

That's a good point. For a single provider, a UNIQUE constraint is indeed a much simpler solution and lets the database enforce consistency naturally.

The tricky part was the ANY_PROVIDER workflow, where the application first has to decide which provider gets the slot before inserting the appointment. That's where the race condition appeared and why I ended up looking at explicit locking.

[Mudassir Khan]

the cal.diy detail is the most interesting part — a fork that exists because the upstream project's threat model changed (AI assisted scanning against a public repo), not because the code went bad. that's a new class of open source trust failure: the license stays open but the maintainer's incentives stop aligning with community use.

finding the TOCTOU race in both finalists after all the UX deliberation is the actual argument of the post. the "trust open source" conclusion earns its weight because you did the work that most devs skip.

curious whether Easy!Appointments has acknowledged the race condition or if the fix ended up being your own patch?

[Pascal CESCATO]

Thanks! That was really the point I wanted to make: open source isn't about assuming the code is bug-free, it's about having the opportunity to verify the assumptions yourself.

As for Easy!Appointments, I haven't submitted a patch. For my own use case, the probability of hitting that race condition is extremely low—I'm expecting around a hundred appointments per month, spread over time—so the practical risk is close to zero.

If I were running a high-volume public booking service, I'd definitely revisit that decision. But for my deployment, understanding the limitation was more important than eliminating a corner case that is unlikely to occur in practice.

[Shweta Mishra]

Interesting perspective. I think one of the biggest advantages of open source isn't just avoiding vendor lock-in—it's understanding the system well enough to adapt it as your requirements evolve. That said, replacing a mature SaaS also means taking ownership of reliability, security, and long-term maintenance. Curious to know what the biggest unexpected challenge was after the migration.

[Pascal CESCATO]

The biggest surprise wasn't the migration itself—it was discovering that the real challenge wasn't replacing the SaaS, but validating the assumptions behind the open-source alternative.

I expected configuration and feature gaps. I didn't expect to spend so much time auditing concurrency logic and finding a race condition that had apparently gone unnoticed for years.

In the end, the migration became much more of a code review exercise than a deployment exercise.

[Emmanuel Salomon]

Try cal.rs

[Pascal CESCATO]

Interesting… I'll try it.

[Tomasz]

By migrating customer data to your own infrastructure, you assume full responsibility for its security and compliance with the GDPR. How do you handle backups, monitoring, and a potential DDoS attack on your instance?

[Pascal CESCATO]

That's a fair point. Self-hosting doesn't remove responsibility, it moves it.

The trade-off is that I now control the infrastructure, the update cycle, and the data flow instead of delegating everything to a SaaS provider with limited visibility.

For this kind of application, I would treat it like any other business system: automated backups with restore tests, monitoring, restricted access, security updates, and a layer in front of the instance for common attacks.

The interesting question is not whether self-hosting is risk-free (it isn't), but whether the added responsibility is worth the additional control and transparency.

[Talha Ramzan]

The "documented intent, not what the code actually does" line on Easy!Appointments is the part that really lands. A docblock confidently stating "the app won't allow this to happen" sitting right next to code that does nothing to enforce it is exactly the kind of gap that survives ten years of production traffic, because nobody goes looking for a bug the comments already told them doesn't exist.

The fact that the correct primitive (has_provider_conflict) already existed in the codebase but was never wired into the actual booking flow is almost more unsettling than if the logic had just been missing entirely, it means someone built the right fix at some point, and it still didn't make it into the path that mattered.

Respect for the closing honesty too. Most writeups like this end on "and here's the fix, problem solved," but stating plainly that your own deployment is still exposed to the exact race window you just spent the whole article documenting is a rare thing to admit publicly. Curious whether you're leaning toward the per-provider GET_LOCK approach as the eventual fix, or reconsidering the non-persistent-connection assumption first, since that seems like the part most likely to bite you quietly in a shared hosting environment.

[Pascal CESCATO]

The has_provider_conflict point bothered me for exactly the same reason. Finding missing logic is one thing; finding logic that exists but is disconnected from the path where the decision actually happens is much harder to spot.

Regarding the fix, I'm still leaning toward solving the concurrency at the database boundary rather than relying on application-level assumptions. The per-provider GET_LOCK() approach is attractive because it matches the actual resource being protected, but you're right that the connection lifetime assumption deserves more investigation, especially outside controlled environments.

The interesting lesson for me is that the race was not caused by a missing feature. It was caused by a gap between the model the code claimed to implement and the execution path that actually ran.

[xulingfeng]

Solid piece. Makes me wonder though — in a world where AI writes more code every day, are we quietly losing the ability to actually read it? That TOCTOU gap didn't surface because the project was mature. It surfaced because you sat down and read the source. That's the kind of thing that stops mattering once everyone stops looking.

[Tom Jones]

The TOCTOU find is the interesting part, and I want to put a number against your question, because we measured the thing people reach for once they stop reading.

The usual fallback is "have a second model check it." We tested what that is worth on our own serving path: two different models, serve the cheap answer when they agree. On code, with executable tests as ground truth, the gate still lets through 1.7 to 3.5 percent wrong. On faithfulness judgement, with nothing to execute, agreement approves a wrong answer 27.5 percent of the time, 95 percent interval 16.1 to 42.8. So agreement is a decent cost lever and it is not an audit.

A race condition is close to the worst case for it. Two checkers only substitute for a reader if their errors are independent, and a concurrency bug is invisible in a single-threaded read of the source, so two readers with the same habit miss it together. Shared structure in the input is a correlation source, which is why agreement tends to look strongest exactly where it deserves the least trust.

What would have caught this one mechanically is not a better reader, it is a test that executes: fire two concurrent requests at the same slot and assert that exactly one wins. That keeps working after everyone stops looking, which is the part your question is really about. The reading found it once. The test finds it every time.

One trap on writing that test, straight out of the post above: SQLite serialises writers, so the race cannot reproduce there. The test passes vacuously and the bug ships anyway on Postgres or MySQL. It has to run on the engine you actually deploy on. We learned that expensively this month, on a change that was green in staging and took our cheap model tier off in production, because staging could not exercise the failing path at all.

[xulingfeng]

Years in QA here — "run it on the engine you actually deploy on" hit hard. Lost count of how many times staging was all green and production went up in flames a few hours later. Appreciate you sharing those numbers. More testing is always better in theory, but time and cost have a say too. Finding that balance — or a decent enough one — is kind of an art.

[Pascal CESCATO]

That's exactly why I included the TOCTOU example. Finding the bug was valuable, but turning it into a regression test is what makes the discovery useful over time. Otherwise, it's just tribal knowledge waiting to be forgotten.

[Tom Jones]

Both of those land, and they meet at a filter that is free to apply, so it is worth naming.

On the balance question: it is judgement in general, and there is one case where it is not, and it is the case in this post. A SQLite test for a race that only exists under concurrent writers is not a weaker test. It has no power at all, because the engine serialises the writers and removes the failure mode from the universe. The test passes by being unable to fail. So before the time and cost question there is a cheaper yes or no: can the environment I am testing in physically produce this failure? If the answer is no, running it is worse than not running it, because you have replaced a known unknown with a green check.

That is also the sharp edge of the tribal-knowledge point. A regression test only converts the discovery into something durable if it can still fail. Ours could not, and we did not notice for weeks. Four of our routing safety suites were passing while a helper we mock had gained a fourth return value and our fake had not, so the suites were exercising an object that no longer resembled the real one. Same disease as the SQLite case arrived at from the opposite direction: the knowledge had been written down, the test was green, and the protection was gone.

So the version of your rule I would carry is that a regression test needs a second act after it is written. Break the thing it guards and prove the test goes red. We do that now, and it caught a suite where the asserted string appeared at two call sites, so breaking one of them was invisible and only breaking both turned it red. Without that step you have converted tribal knowledge into a green check, which is the more expensive of the two, because a green check stops anyone from looking.

Everything after that filter really is judgement and cost, and I do not think there is a formula. But the filter is free, and it tends to delete a couple of tests people feel guilty about skipping and promote one or two they were not thinking about.

[Pascal CESCATO]

That's exactly where my skepticism comes from.

I don't distrust tests themselves, I distrust the confidence people sometimes attach to them. A green test suite can mean "the system is protected", but it can also mean "we successfully tested the assumptions we made when writing the tests".

The production environment remains the ultimate judge because it introduces the combinations nobody thought about: real data, real users, real timing, real failures.

For me, the value of a regression test is not that it exists, but that it represents a failure that actually happened and that we proved it can fail when the condition comes back. Otherwise, it is just another artifact that creates confidence without necessarily creating safety.

[Tom Jones]

"It represents a failure that actually happened and we proved it can fail when the condition comes back" is the definition I would put in front of most testing guides, and the second clause is the one people drop.

Proving it can fail is mechanical, which is what makes the omission unnecessary. Break the code the test covers on purpose, re-run, and require red. Two minutes, and it converts a belief about the test into an observation.

We had a test named, almost word for word, that a failed operation leaves the original untouched. It exercised three operations, each with a bad identifier. Replacing the defensive copy with a plain assignment on a successful path passed the entire suite. Every case in the immutability test was invalid input, so a rejected call returned at the guard clause before touching anything, and the assertion held on code with no copying in it at all. It was a test of guard clauses wearing an immutability name.

The harm was on the untested half, and it is the half that matters. Nobody is hurt by a rejected call leaving state alone. The damage comes from a call that succeeds and quietly edits the caller's copy, because the caller is usually holding that value as the previous state, which is what an undo restores and a retry re-sends.

So your skepticism has a cheap discharge. A green suite that has never been watched going red is a claim about the author's imagination.

[Pascal CESCATO]

That’s a very good way of putting it — especially the distinction between a test existing and a test actually demonstrating that it can detect the failure it claims to cover.

Your “test of guard clauses wearing an immutability name” example is almost a perfect illustration of the problem I was getting at. The test suite was green, the test name was reassuring, and yet the critical execution path was never exercised.

And I particularly like the “watch it go red” criterion. It turns something that is usually treated as an article of faith — “this test would catch the regression” — into an observable fact.

So yes: I think you’ve found the cheap discharge for the skepticism I was describing. Don’t just ask whether the test passes. Deliberately break the behaviour it is supposed to protect and make sure the test objects.

That’s a much stronger definition of regression testing than simply “we added a test for the bug.”

[Tom Jones]

Glad it landed. One thing worth adding, because it is where that criterion bit me afterwards: a surviving mutant has three causes, and only the first is the one anyone reaches for.

The obvious reading is that the test is weak. The second is that the behaviour is over-determined, so removing one producer changes nothing observable and the mutant is genuinely harmless. The third cost me an afternoon. The mutated code is never reached. You cannot change the behaviour of a branch that has no behaviour, so absence of effect is what the mutant and the original both produce, and no assertion over outcomes can separate them.

Mine was a key binding. The case compared against " " while the runtime actually produced the string "space", so that branch had never matched once in the entire life of the file. I had already told my founder it was a regression I had introduced. It was a dead binding the refactor had faithfully preserved, and the mutant survived because there was nothing there to kill.

So the rung I would put underneath yours: before concluding the test is weak, prove the path runs at all. A counter, or a fatal inside the arm, costs one line. And a dead path is a better finding than a weak test, because it is invisible to every other instrument you own, whereas a weak test at least has a chance of failing one day.

The sharpest instances are all literals in a condition. A key name, an enum string, a header, a flag value. The compiler cannot object, review reads them as obviously right, and behavioural tests are structurally incapable of seeing them.

[Pascal CESCATO]

Yes — that third case is an important correction to the rule.

A surviving mutant doesn't necessarily mean “the test failed to protect the behaviour”. Sometimes it means “there was no behaviour to protect in the first place”.

And your key-binding example makes the distinction particularly nasty: the code is syntactically valid, looks perfectly reasonable in review, survives the test suite, and even survives mutation — precisely because the condition is never true. The test isn't weak; it is being asked to observe something that never happens.

That suggests a useful ordering before even judging the quality of an assertion:

1. Is the path reachable?
2. If it is, does the test actually observe the relevant behaviour?
3. If it does, does deliberately breaking that behaviour make the test fail?

Only then does “the test is weak” become the right diagnosis.

And I agree that the dead-path finding can actually be more valuable. A weak test is a latent problem; a dead branch is evidence that part of the system's supposed behaviour exists only in the source code's narrative.

The literal-value cases are especially insidious for exactly that reason. There is no type error, no compiler complaint, nothing visually suspicious in review — just two perfectly valid strings that will never meet at runtime.

That also makes me wonder whether mutation testing is ultimately less a test-quality technique than a way of challenging the story the code tells about itself. The surviving mutant is sometimes the first indication that the story and the runtime have diverged.

[Tom Jones]

Your closing line is the one I would keep, and I can give it a sharper form from the same incident, because the story the code was telling had already been repeated out loud by a human before the mutant corrected it.

The sequence went like this. I consolidated some scattered key handlers, added a guard so only Enter jumped back to the shell, wrote a test, and it passed on the first run. Then I told our founder I had introduced a regression and was fixing it. The mutant that deletes the guard survived. My instinct matched yours in reverse: strengthen the assertion. What settled it was measuring the runtime, where the key event for a space bar stringifies to "space" while the branch compared against a literal " ". That branch had never matched in the entire life of the file. Space had never once done the thing the code said it did, and nobody had introduced a regression at all.

So the correction ran past the code. The account I had already given a person was wrong in the same direction the source was wrong, and I would have spent the afternoon repairing a fault that never existed. Your point with a second layer: the source tells a story, people repeat it, and the mutant is the only participant in that conversation with direct access to the runtime.

One caution on your ordering, because we found a fourth rung beneath it. Reachable, observed, and breaking it still produces a pass, with the test perfectly healthy. Two guards in our terminal code look identical. One is load-bearing. The other duplicates a refusal further down the stack, so removing it changes nothing a test could see, and the code is correct as written. Your three questions all answer yes there, and the honest verdict is redundancy, which says nothing about the test at all.

Which leaves your final sentence carrying more weight than the mutation-testing literature usually gives it. A surviving mutant marks a divergence between the story and the runtime, and that story has at least three authors: the source, the suite, and whoever last described the system to another person. Only one of the three can stay wrong quietly for a year.

[Pascal CESCATO]

That fourth rung is important, because it prevents us from turning mutation testing into another binary oracle.

A surviving mutant can mean a weak test, unreachable code, or simply a redundant piece of correct code. So the mutant itself isn't the diagnosis — it is the anomaly that forces us to investigate the relationship between the code, the test, and the runtime.

And I think your last formulation is stronger than my original one:

the source, the suite, and the person who last explained the system are three authors of the same story.

The dangerous part is that they can reinforce each other. The source says “this is what happens”, the test appears to confirm it, and the human explanation gives everyone a reason to stop looking. You can therefore have a perfectly coherent story which is completely disconnected from reality.

The mutant breaks that coherence without needing to know which part of the story is wrong.

That may actually be the most interesting thing about mutation testing: not that it proves a test is good, but that it creates a controlled contradiction between the story and the runtime — and forces you to find out which one is lying.

And in your space-key example, the uncomfortable answer was: all three narratives were wrong in exactly the same way.

That is a much more dangerous failure mode than a simply weak test.

[Tom Jones]

They can reinforce each other is the part I keep coming back to, and I got a small demonstration of it today that adds a fourth author to your three.

I wrote a guard with a documented negative control, which is a fixture that must make it fail. It failed correctly on the day I wrote it. Then I narrowed what the guard treats as a defect, for reasons I still think were right, and the fixture I had written described the old definition. So the control now passed against the new code, which means the guard was quietly reporting that it could not fail at all.

Every author in your set agreed at that moment. The source said the guard detects a defect. The test said the guard works. My own commit message said what the guard now detects. All three were coherent, and the coherence was the problem, because the control had stopped being about anything.

None of the three broke it. A separate check runs each guard's declared control and requires it to go red. It never opens the report, the commit message, or the guard's own account of itself. It makes the thing demonstrate a failure, and it named mine within one run of the change.

So the fourth author I would add is the artefact that only ever answers by doing. Your mutant qualifies, and so does a control that must produce a red. Neither of them can be told the story, which is what makes them the only participants who can disagree with it. The uncomfortable part of the space-key example was that all three narratives were wrong the same way, and a mutant fixes that through ignorance, never through intelligence. It never heard any of them.

One thing your framing sharpened for me. The dangerous moment arrives later than a wrong story. It arrives when the story changes and the artefacts that were supposed to contradict it get updated in the same breath, by the same person, for the same reason. My control did not decay on its own. I retired the definition it was testing and left it looking healthy, which is the same failure as citing a cross-check you edited in the commit it is supposed to check.

[Pascal CESCATO]

Yes. I think that last distinction gets us somewhere deeper than mutation testing itself.

The real danger isn't that the story is wrong. Software can survive a wrong story for quite a long time.

It's that the mechanism intended to contradict the story is allowed to evolve with the story.

At that point you don't have verification anymore. You have two artefacts agreeing with each other.

Your control example makes that painfully clear: the fixture didn't become obsolete by itself. The definition changed, and the same change effectively edited the evidence that was supposed to challenge it. From inside the system, everything remained coherent.

That's why I really like your fourth author. It isn't necessarily “another test”; it's an artefact whose validity is defined operationally rather than narratively:

“Show me the failure.”

The mutant says: remove this behaviour and see whether reality changes.

The negative control says: give me the declared defect and see whether I actually fail.

Neither needs to know what the code is supposed to mean.

And that may be the strongest lesson here: independent evidence doesn't mean another description of the same behaviour. It means something whose answer cannot be made consistent merely by changing the description.

Which makes your final analogy particularly uncomfortable. Editing a cross-check in the same commit that changes the thing being checked isn't really maintaining the check. It's maintaining the appearance of the check.

I suspect that's why these tiny “make it go red” mechanisms are disproportionately valuable: they introduce a little piece of evidence that the narrative cannot edit for itself.

And this is probably where our perspectives diverge.

I completely agree with the mechanism you're describing. If we claim that a test protects a behaviour, making it demonstrate that protection removes one layer of self-deception.

My skepticism starts one level further up, though: the things we can test are necessarily the things we have already thought about.

Unit and regression tests are very good at protecting known behaviours and known failure modes. They are much less good at discovering the failures we didn't know to specify — unexpected interactions, assumptions about real data or the environment, or simply users doing something that never appeared in the model.

Those are often the failures that eventually show up in production, if they show up at all.

So for me, “make the test go red” is evidence that one particular piece of our mental model has a functioning tripwire. It isn't evidence that we've covered the important failure modes.

The production environment is the one test suite whose authors we don't control — and whose test cases we don't get to write in advance.

[Prahlad Yeri]

Forget 'Infrastructure I control instead of third party's'. The world is now moving towards offline-first and local-first approaches, architect the web app in such a way that the most critical data won't leave your computer's shore in the first place, except for syncs and backups.

The presently prevalent cloud server (or client server) paradigm is a vestige of an era when most browsers used to be 'thin clients' and lacked capabilities of compute and storage. But this is no longer a case today and better approaches are possible.

[Pascal CESCATO]

I think that's a different discussion.

My goal wasn't to design a new scheduling application or rethink web architecture from scratch. It was to replace a proprietary SaaS with an existing open-source solution that I could deploy today.

A local-first scheduling application would certainly be an interesting project, but it would require building a completely different product, which is outside the scope of this article.