DEV Community: Mads Quist

Beyond Vibe-Coding

Mads Quist — Thu, 28 May 2026 10:59:00 +0000

Are You Paying for Reliability or Hype in the Age of Vibe-Coding and SaaS Subscriptions?

In 2026, many engineering teams are coming to terms with an odd new reality: a growing percentage of their codebase wasn’t written by their own team… or even a human, for that matter.

Rather than being pair-programmed, reviewed, tested, or even fully understood, it was AI-generated. Confidently, instantly, and sometimes a little chaotically, by something that simply “felt” like the code would work. In other words, it operates off vibes, not knowledge.

Welcome to the era of vibe-coding.

Big companies like Microsoft and Google are already using AI to write 30% of their code. And it doesn’t exactly get the code wrong; it runs well, passes all the necessary tests, and even looks good enough that no one questions it. Until it behaves unexpectedly, that is.

AI is making code shipping even easier than it ever was, but that doesn’t come for free: Day 2 operations are slowly becoming a nightmare, because when no one remembers writing the code, no one remembers how to fix it.

It’s time like these where the conversation shifts from productivity to practicality, and more importantly, reliability. And that leads us to the question: is your incident management setup built to survive the new world you’re operating in?

The Hidden Cost of Vibe-Coding for Day 2 Operations

I know what you’re thinking: yes, you absolutely can, and in some cases maybe you should, generate a lot of working code quickly and efficiently with AI. It’s a fast, confident solution that’s great when you’re prototyping, but much less great when you’re trying to understand why a service is suddenly swallowing 40% more memory on Thursdays.

In fact, AI-generated code produces 1.7x more issues than human-written code. And relying fully on it is where many teams seem to go wrong. They end up creating things too fast with very little specification and run into a mountain of problems before the new code is even out of the box. But the truth is very simple:

_AI has made writing code easier. But it hasn’t made owning that code any easier.
_
Instead, it’s introduced an onslaught of new problems for engineering managers and VPs, like:

More edge-case failures
More “no one touched this, why did it break?” moments
More debugging sessions
More operational load on teams who didn’t write, or understand, the original logic.

Code is the foundation of how online services work. If the codebase becomes a black box, your incident management becomes the only reliable source of truth.

Why Your Incident Management Can’t Live on the Same Servers as Your App

Enter: the build-vs-buy conversation. Plenty of teams have built their own alerting systems over the years and it’s easy to understand why. Engineers are good at building things and alerting seems simple enough.

But the slightly uncomfortable reality is that if your alerting lives on the same infrastructure as your application, it’ll fail the same way your app fails. One single issue can cascade across multiple services, so when your platform goes down, your homegrown alerting goes with it. Identifying the root cause of the problem is like looking for a needle in a haystack, as any issues faced during development might not be reproducible in the production environment.

Imagine using the foundation of your house to build a new roof. That’s what happens when your alerting and infrastructure co-exist in the same place. External, independent alerting isn’t a luxury, but a safety measure. If your cloud provider hiccups, your alerts hiccup with it. And if your AI-generated deployment throws a tantrum, your DIY alerting gets pulled into the meltdown.

Separating the two means only having to deal with one catastrophe rather than two. It guarantees that when your platform is collapsing, your incident management (and engineering team) isn’t joining it.

The New Reality of Build vs Buy in 2026

Engineering teams are perfectly capable of building their own alerting systems. They can even build databases, authentication systems, and CI/CD pipelines from scratch… but do they even want to? Are homegrown systems as reliable as plug-and-play?

That’s where the build-vs-buy conversation gets more interesting. It’s more than just a cost comparison; it’s a question of control, scalability and integration. Building gives you bespoke tools that do exactly what you need them to do, and are operationally efficient. Buying, on the other hand, is standardized and easy; there’s no maintenance costs, as those sit with the provider, and security is almost guaranteed.

Plus, DIY alerting comes with a few unavoidable truths:

It shares the same failure modes as your platform
It depends on the same infrastructure
It inherits the same outages
It expands your operational surface area
It becomes one more thing to maintain when maintenance is already hard enough.

Layer on the unpredictability of vibe-coded systems with incidents that are still less predictable and harder to solve, and the last thing you want to deal with is an alerting system that disappears the moment you need it most.

That’s where All Quiet steps in as the adult in the room.

Why All Quiet is Built for the World Vibe-Coding Created

All Quiet isn’t trying to be a fancy, flashy tool. It does what it says on the tin: keeps alert management quiet. It’s not trying to reinvent incident response with buzzwords or magic tricks; it’s built around one very simple and boring, yet extremely important, principle:

_Your alerting should stay independent when everything else goes offline. _

Nice and simple. And that principle shapes everything about how All Quiet works:

It runs independently from your infrastructure, so your alerts don’t vanish just because your servers took a last-minute holiday.
It has different failure modes, meaning your app can crash, your cluster can melt, your cloud can create a hurricane, and All Quiet doesn’t bat an eyelid.
It doesn’t rely on your servers, cloud provider or deployment pipeline, so your alerting isn’t married to the same weak dependencies as your product.
It doesn’t care if your AI-generated code crashed your entire platform, because it won’t be affected by the fallout.
It’s designed to be the one stable thing in a house that’s falling down, because it doesn’t panic while everything does.

AI-generated code means people don’t fully understand the very code they’re working with, so reliability becomes the only differentiator. You can have all the fancy dashboards and features you want, but they’re worth nothing if an incident creeps up and your code is hidden away in a treasure chest no one has the keys to.

All Quiet is built to be reliable in every situation. When things are calm, it’s calm. When things are less calm, it’s calm. And where your entire system is swirling into a vortex of chaos and fire, it’s still calm. When vibe-coding has taken over the world, the last thing you want is an alerting system that runs solely on vibes too.

SaaS Cost Optimization in the Vibe-Coding Age

There are thousands of SaaS tools out there, and hundreds of alerting tools. But there’s a new kind of bloat on the rise: tools that look great on paper but only exist because your AI-generated code keeps spitting out surprises.

Engineering leaders are asking themselves smarter questions: What’s actually mission-critical? What’s just hype? What’s duplicating functionality? What’s costing us more than it’s saving us?

But most importantly:

_What’s the cost of downtime if our alerting fails with everything else? _

The most expensive tool in your stack is the one that doesn’t work when you need it most. All Quiet isn’t about adding another subscription to your balance sheet, but removing uncertainty and wasted time from your operational risk.

The Future of Reliability Over Hype

AI is like a self-cleaning oven. It’s getting smarter and better, which means vibe-coding will keep getting weirder. But engineering teams will keep shipping faster than ever.

Despite that, the fundamentals won’t change: you still need to know when something breaks and you need to respond quickly.

And you’ll always need an alerting system that doesn’t disappear the moment your platform does. Your team shouldn't be asking which tool has the most features, but which tool will still be standing when everything else fails.

And the only answer is the one that doesn’t run on vibes too. It’s the one that does what it says it’ll do, and it does it independently of everything else.

Which is exactly what All Quiet does. Talk to us today to find out how.

SaaS is dead, long live SaaS

Mads Quist — Tue, 26 May 2026 09:59:00 +0000

While vibe-coding does favor build over buy for some products, it won't replace mission-critical tools in the foreseeable future

Here’s my take on what gets killed, what survives, and the build-vs-buy rule that actually works:

I believe that AI does not kill SaaS. It eliminates the “thin wrapper”: software that offers little more than a polished UI for simple automations. Today, any developer can stitch these tools together in an afternoon.

Still, SaaS is not disappearing; it is evolving. While products that sell mere convenience will face extinction, products that sell reliable outcomes for high-stakes problems will thrive. This guide covers what survives the shift toward vibe-coding and how to decide whether to build or buy.

The New Baseline: When "Good Enough" Becomes Free
Vibe-coding is great: It allows us to ship functional software at unprecedented speeds. Large Language Models (LLMs) now draft code, connect APIs, and generate UI scaffolding automatically.

We can see that this shift raises the bar for SaaS vendors. When basic functionality is cheap, the value of a paid tool must come from reliability, compliance, and long-term maintenance. High prices do not just signal quality. They create a financial incentive for teams to build their own "80% solution." Just look at the stock price of tools like PagerDuty or Hubspot or other SaaS giants that have seen their stock prices drop significantly.

SaaS Categories Facing Extinction

That said, I think vulnerable SaaS categories share three traits: a narrowly defined job-to-be-done, low failure costs, and easily verifiable outputs. Good examples are:

Workflow Wrappers: Basic CRUD (Create, Read, Update, Delete) tools for internal approvals or team dashboards.

Low-Stakes Content: Tools for text summarization, formatting, or template generation.

Basic Analytics: Simple data visualization layers that sit on top of a data warehouse.

So, how do SaaS players stay relevant? They need to be outstanding in the categories that are not vulnerable to vibe-coding.

The Survivors: Infrastructure You Trust

Based on the criteria mentioned above, my strong opinion is that SaaS products that sustain are those where the cost of being wrong is high. These are not features: they are mission-critical infrastructure.

1. Systems of Record and Compliance

When audit logs, access controls, and data retention are legal requirements, a vibe-coded script is not an acceptable answer to an auditor. Specialized SaaS platforms provide the security posture and regulatory guarantees that custom-built scripts lack.

2. High-Availability Operational Tools

If a system failure must wake an engineer at 2 a.m., the tool must be battle-tested. Incident management is the primary example. Reliability is the core feature.

In this category, vibe-coding errors are not cosmetic. They are existential. Failing to page the right person during a critical outage results in lost revenue and broken trust. All Quiet and other incident response systems offer reliability that a prototype cannot guarantee. And this guarantees are mission-critical.

There’s also an architectural reality: your incident management tool must keep working when you are down. If you self-host it on the same infrastructure, identity provider, or network dependencies as your primary systems, you will lose alerting precisely when you need it most. The whole point is having a system that is on high alert when you aren't to let you know when something is broken. Running incident management as a truly independent system (even when self-hosted) adds meaningful overhead, which is one more reason teams often prefer buying a dedicated service built to stay up during your worst day.

3. Complex Integration Ecosystems

Last but not least: integration ecosystems. Yes, AI can reduce some glue code and speed up building custom connectors. But the real pain is rarely the first implementation — it’s the ongoing reliability work: API changes, new auth flows, rate limits, subtle edge cases, and “it worked yesterday” incidents when a vendor ships an update. In practice, you usually can’t just delete integrations without deleting requirements. When a tool sits in the middle of many other tools, you’re effectively signing up to maintain that surface area for years. SaaS vendors sell the promise that these connections keep working without your team babysitting them.

The Build-vs-Buy Decision Matrix

Factor Build (Vibe-Coding) Buy (SaaS)
Risk of Failure Low (Embarrassing) High (Expensive/Existential)
Ownership Internal Team Vendor (SLA-backed)
Verification Obvious and Simple Subtle and Critical
Integrations 1 or 2 Static APIs Many Evolving Vendors

Final Rule of Thumb

Build if vibe-coding achieves your outcome with acceptable risk for your business. Buy if the operational burden or maintenance will quietly exhaust your team because you need the product to be battle-tested.

Modern SaaS wins when it removes long-term risk, not when it adds more features. You’re paying for reliability, security, compliance, and support, ensuring that the tool still works when your team is tired, busy, or in the middle of an outage.

AI makes building prototypes cheap. What doesn’t get cheap is owning the consequences: keeping it up, keeping it secure, keeping integrations working, and maintaining it for years. If you’re not willing to own that long-term responsibility, buy and focus your resources on building your core product.

Migrating from Opsgenie to All Quiet: A Full Terraform-First Guide

Mads Quist — Tue, 12 May 2026 15:33:44 +0000

Originally published on 12 May 2026 on the All Quiet Tech Blog.

If your Opsgenie config already lives in Terraform, you can migrate methodically instead of clicking two consoles side by side. This guide translates users, teams, integrations, on-call schedules, escalations, and routing into All Quiet - complete with example HCL, migration checklist, and tips for running both tools in parallel before you switch.

With the recent changes in the Atlassian ecosystem, many SRE and DevOps teams are finding themselves at a crossroads: adapt to the increasing complexity of Jira Service Management (JSM) or move to a leaner, more focused incident management platform.

At All Quiet, we believe incident management should stay close to the code. That's why our platform is built to be managed via Terraform from day one. In this guide, we'll walk through a complete technical migration from Opsgenie Terraform resources to All Quiet, resource by resource, with real HCL on both sides.

If you are still weighing vendors before you change tooling, start with our overview of All Quiet as an Opsgenie alternative, then use this article for the Terraform resource mapping and cutover checklist.

Why a Terraform-First Migration?

If you're already managing Opsgenie via Terraform, you have an advantage: your entire on-call configuration is already codified. Rather than clicking through two UIs in parallel, you can translate your .tf files directly from one provider to the other, terraform plan the result, and cut over with confidence.

The Strategy: "Logic-First" Migration

In Opsgenie, configuration is fragmented across six or more resource types: opsgenie_user, opsgenie_team, opsgenie_api_integration, opsgenie_schedule, opsgenie_schedule_rotation, and opsgenie_escalation. All Quiet centralizes this logic into fewer, more cohesive resources, most notably allquiet_team_escalations, which unifies schedules, rotations, and escalation policies into a single resource that can't get out of sync.

Here's the full resource mapping at a glance:

Opsgenie Resource	All Quiet Resource	Notes
`opsgenie_user`	`allquiet_user`	Standalone identity
`opsgenie_team`	`allquiet_team` + `allquiet_team_membership`	One membership resource per user–team pair
`opsgenie_api_integration`	`allquiet_integration`	Team-owned, strongly typed
`opsgenie_schedule`	`allquiet_team_escalations`	Merged into unified resource
`opsgenie_schedule_rotation`	`allquiet_team_escalations`	Rotations live inside escalation tiers
`opsgenie_escalation`	`allquiet_team_escalations`	Rules become escalation tiers
(routing within integration)	`allquiet_routing`	Explicit routing resource

1. Setting Up the Providers

First, initialize your environment. You'll need your All Quiet API Key, which you can generate in Organization Settings > API Keys (requires Owner or Administrator role). See our Terraform setup docs for the full walkthrough.

terraform {
  required_providers {
    allquiet = {
      source  = "AllQuietApp/allquiet"
      version = ">= 3.0.0"
    }
  }
}

provider "allquiet" {
  api_key    = var.allquiet_api_key
  api_region = "eu" # or "us" — must match your organization's data region
}

variable "allquiet_api_key" {
  type      = string
  sensitive = true
}

Tip: We recommend creating your organization with a shared admin account (e.g., admin@company.com) rather than a personal email. This way, every "real" on-call user can be provisioned via Terraform, and you won't have a chicken-and-egg problem with the account that created the org.

2. Teams, Users, and Memberships

In Opsgenie, users are standalone resources with roles, and team membership is defined inline within the team. This creates tight coupling, changing a user's team membership means editing the team resource.

All Quiet separates these concerns into three distinct resources: the team, the user identity, and the membership link between them. Each membership is its own resource (allquiet_team_membership), one resource per user–team pair. This allows for cleaner state management: adding or removing a single member doesn't trigger a plan change on the team or on any other member.

The Opsgenie Way:

resource "opsgenie_user" "sre_lead" {
  username  = "alex@company.com"
  full_name = "Alex Rivera"
  role      = "user"
  timezone  = "Europe/Berlin"
}

resource "opsgenie_team" "devops" {
  name        = "DevOps"
  description = "Core DevOps and SRE team"

  member {
    id   = opsgenie_user.sre_lead.id
    role = "admin"
  }

  member {
    id   = opsgenie_user.backend_eng.id
    role = "user"
  }
}

The All Quiet Way:

# 1. Define the team
resource "allquiet_team" "devops" {
  display_name = "DevOps"
  time_zone_id = "Europe/Berlin"
}

# 2. Define user identities
resource "allquiet_user" "sre_lead" {
  email        = "alex@company.com"
  display_name = "Alex Rivera"
}

resource "allquiet_user" "backend_eng" {
  email        = "jordan@company.com"
  display_name = "Jordan Lee"
}

# 3. Link each user to the team via a dedicated membership resource (one per user–team pair)
resource "allquiet_team_membership" "devops_sre_lead" {
  team_id = allquiet_team.devops.id
  user_id = allquiet_user.sre_lead.id
  role    = "Administrator" # "Administrator" or "Member"
}

resource "allquiet_team_membership" "devops_backend_eng" {
  team_id = allquiet_team.devops.id
  user_id = allquiet_user.backend_eng.id
  role    = "Member"
}

What changed:

Opsgenie's admin / user team roles map to All Quiet's Administrator / Member.
Each membership is its own resource (allquiet_team_membership), so adding or removing a single team member is a targeted change, no cascading diffs on the team or other members.
Users provisioned via Terraform receive an invite to set their password. If you use SSO (OIDC, Google, or Microsoft), configure that first, see the SSO docs.

3. Integrations: Team-Owned and Strongly Typed

Opsgenie requires separate resources for API integrations and their subsequent notification or routing actions. The integration itself is often a loose endpoint that you wire to teams via responders blocks.

All Quiet treats integrations as team-owned endpoints with strongly typed integration types, so there's no guesswork about payload format.

The Opsgenie Way:

resource "opsgenie_api_integration" "grafana" {
  name              = "Grafana-Alerts"
  type              = "Grafana"
  owner_team_id     = opsgenie_team.devops.id
  enabled           = true
  allow_write_access = true

  responders {
    type = "team"
    id   = opsgenie_team.devops.id
  }
}

The All Quiet Way:

resource "allquiet_integration" "grafana" {
  team_id      = allquiet_team.devops.id
  display_name = "Grafana Production"
  type         = "Grafana"
}

That's it, no responders block, no allow_write_access flag. The integration belongs to the team, and the team's escalation policy handles notification logic.

Common type mappings:

Opsgenie type	All Quiet type
Datadog	Datadog
Prometheus	Prometheus
Grafana	Grafana
CloudWatch	AmazonCloudWatch
API / Webhook	Webhook

The full list of supported integration types is available at: https://allquiet.app/api/public/v1/inbound-integration/types

Important: Treat the integration type and team assignment as fixed once created, changing them may require destroying and re-creating the resource, which generates a new webhook URL. Plan these carefully. After terraform apply, the new webhook URL will be available. For each supported inbound integration type you can download a default Terraform snippet for payload mapping: use https://allquiet.app/api/integrations/terraform/default/<Type>.tf where <Type> is the exact integration type identifier (see the inbound integration types list above). For example, Datadog is https://allquiet.app/api/integrations/terraform/default/Datadog.tf; Grafana, Webhook, AmazonCloudWatch, and every other supported type follow the same URL pattern with their own type name.

If you need to customize how payloads map to incidents (e.g., extracting severity from a specific JSON field), use the allquiet_integration_mapping resource. If you don't define one, All Quiet uses sensible defaults for each integration type. The mapping supports JSONPath, XPath, regex, and static values and every incident maps to three key attributes: Status (Open/Resolved), Severity (Minor/Warning/Critical), and an optional Title.

Migration tip: You can run both Opsgenie and All Quiet integrations in parallel during the transition period. Point your monitoring tools at both webhook URLs until you're confident in the All Quiet setup.

4. On-Call Schedules and Escalations: The Unified Resource

This is the most significant architectural difference between the two providers, and the biggest win in your Terraform-first migration.

In Opsgenie, on-call configuration is spread across three separate resources that reference each other by ID. In All Quiet, it's all one resource: allquiet_team_escalations. This resource follows a clear hierarchy: Escalation Tiers → Schedules → Rotations, which mirrors how on-call actually works: you have layers of people to notify (tiers), each layer has time-based coverage windows (schedules), and people rotate through those windows (rotations).

The Opsgenie Way (3 resources, fragile cross-references):

resource "opsgenie_schedule" "devops_oncall" {
  name          = "DevOps On-Call"
  timezone      = "Europe/Berlin"
  enabled       = true
  owner_team_id = opsgenie_team.devops.id
}

resource "opsgenie_schedule_rotation" "devops_weekly" {
  schedule_id = opsgenie_schedule.devops_oncall.id
  name        = "Weekly Rotation"
  type        = "weekly"
  length      = 1
  start_date  = "2024-01-01T09:00:00Z"

  participant {
    type = "user"
    id   = opsgenie_user.sre_lead.id
  }

  participant {
    type = "user"
    id   = opsgenie_user.backend_eng.id
  }
}

resource "opsgenie_escalation" "devops_escalation" {
  name          = "DevOps Escalation"
  owner_team_id = opsgenie_team.devops.id

  rules {
    condition   = "if-not-acked"
    notify_type = "default"
    delay       = 5

    recipient {
      type = "schedule"
      id   = opsgenie_schedule.devops_oncall.id
    }
  }

  rules {
    condition   = "if-not-acked"
    notify_type = "default"
    delay       = 15

    recipient {
      type = "user"
      id   = opsgenie_user.manager.id
    }
  }

  repeat {
    wait_interval          = 30
    count                  = 3
    reset_recipient_states = true
  }
}

That's 3 resources, 50+ lines, with IDs threaded between them. Delete the schedule without updating the escalation and you get a dangling reference.

The All Quiet Way (1 resource, self-contained):

resource "allquiet_team_escalations" "devops_oncall" {
  team_id = allquiet_team.devops.id

  escalation_tiers {
    # TIER 1: On-call rotation — alert the person on duty
    repeats               = 3
    repeats_after_minutes = 5

    schedules {
      display_name = "DevOps Weekly Rotation"

      rotation_settings {
        rotation_mode      = "auto"
        auto_rotation_size = 1         # One person on-call at a time
        repeats            = "weekly"
      }

      rotations {
        members {
          team_membership_id = allquiet_team_membership.devops_sre_lead.id
        }
        members {
          team_membership_id = allquiet_team_membership.devops_backend_eng.id
        }
      }
    }
  }

  escalation_tiers {
    # TIER 2: If Tier 1 exhausts its repeats, escalate to the manager
    repeats = 1

    schedules {
      display_name = "Manager Escalation"

      rotations {
        members {
          team_membership_id = allquiet_team_membership.devops_manager.id
        }
      }
    }
  }
}

Everything that was spread across opsgenie_schedule, opsgenie_schedule_rotation, and opsgenie_escalation is now a single allquiet_team_escalations resource. Schedules and rotations live inside escalation tiers, so there's no way for them to become orphaned. Every person, whether they're in a rotating schedule or a single-person escalation target, is referenced via their team_membership_id, which keeps the dependency graph clean.

Key mapping:

Opsgenie concept	All Quiet equivalent
Escalation rule with delay	`escalation_tiers` with `repeats_after_minutes`
`opsgenie_schedule` (time windows)	`schedules` block within a tier, define on-call times (e.g., weekdays 08:00–18:00)
`opsgenie_schedule_rotation`	`rotation_settings` within a `schedules` block, auto or explicit mode
Rotation participants	`rotations` → `members` → `team_membership_id` (always via membership)
Multiple schedules for follow-the-sun	Multiple `schedules` blocks in the same tier, each covering different hours/days
`repeat` block on escalation	`repeats` on the relevant tier
Recipient: schedule	Tier with a `schedules` block containing rotations
Recipient: user	Tier with one schedule, one rotation, one member

Advanced patterns: Round-robin alerting distributes incidents evenly when multiple people are on-call simultaneously. On-call overrides, both personal and team-level, can be managed via the allquiet_on_call_override Terraform resource without touching the escalation config. See the escalation docs for the full set of options.

Note on complexity: Opsgenie's delay/repeat model and All Quiet's tier-level repeats / repeats_after_minutes / auto_escalation_after_minutes don't map one-to-one in every case. Simple escalations translate cleanly, but complex multi-rule Opsgenie policies may need case-by-case tuning. We recommend testing each escalation path with a synthetic incident before cutting over.

Note on time restrictions: Opsgenie's time_restriction blocks on rotations (time-of-day and weekday-and-time-of-day) map to All Quiet's schedule on-call times. In All Quiet, each schedule defines its active hours and days directly (e.g., "Monday–Friday, 09:00–17:00"), which is more intuitive than Opsgenie's separate restriction blocks. Review these during migration.

5. Routing: The Incident Traffic Controller

In Opsgenie, routing logic is often buried inside the integration itself (via responders blocks) or handled by notification policies. In All Quiet, routing is an explicit, first-class resource with a powerful rules engine. Each rule has three components: Conditions (when to trigger), Actions (what to do), and Channels (how to notify).

resource "allquiet_routing" "prod_alerts" {
  team_id      = allquiet_team.devops.id
  display_name = "Production Alert Routing"

  rules = [
    {
      # Mute Slack for test environment alerts, only send email
      conditions = {
        statuses   = ["Open"]
        attributes = [{ name = "Environment", operator = "=", value = "Test" }]
      }
      actions = {
        change_severity = "Minor"
      }
      channels = {
        notification_channels = ["Email"]
      }
    },
    {
      # For escalated critical incidents, also trigger the PagerDuty outbound webhook
      conditions = {
        statuses   = ["Open"]
        severities = ["Critical"]
        intents    = ["Escalated"]
      }
      channels = {
        outbound_integrations = [allquiet_outbound_integration.pagerduty_webhook.id]
      }
    }
  ]
}

Valid notification_channels values are "Email", "Push", "SMS", and "VoiceCall".

Routing conditions can filter on severity, status, specific integrations, incident intents (created, escalated, resolved), custom payload attributes, and even time-of-day restrictions. Actions include discarding incidents, changing severity, assigning to other teams (within an Organization), adding interactions, and delaying execution. This replaces Opsgenie's scattered notification policies with a single, auditable, version-controlled resource.

Pro tip: For multi-team setups, you can create a "root team" that owns your integrations and uses routing rules to fan out incidents to the appropriate team based on payload attributes. See the routing docs for detailed examples.

Migration Checklist

Here's the step-by-step order we recommend. The key dependency is that allquiet_team_membership requires both the team and user to exist, and allquiet_team_escalations references membership IDs, so teams, users, and memberships must all be in place before you build escalation tiers.

Set up the All Quiet Organization and generate your API key.
Create teams (allquiet_team) and provision users (allquiet_user). These two have no dependency on each other and can be created in any order or in parallel.
Link users to teams (allquiet_team_membership), one resource per user–team pair. This requires both the team and user to exist.
Create integrations (allquiet_integration) for each monitoring source. Note the new webhook URLs.
Customize payload mappings (allquiet_integration_mapping) if the defaults don't fit your payload structure.
Configure notification preferences (allquiet_user_incident_notification_settings), this controls how each user gets alerted (push, SMS, voice call, email) and with what delay.
Build escalation policies (allquiet_team_escalations) by merging your Opsgenie schedules, rotations, and escalation rules into unified tiers. Rotations reference team_membership_id, so memberships must exist first.
Set up routing rules (allquiet_routing) for any advanced alert routing.
Set up outbound integrations (allquiet_outbound_integration) for Slack, Microsoft Teams, or webhook notifications.
Run both systems in parallel, point your monitoring tools at both Opsgenie and All Quiet webhook URLs for a burn-in period. Trigger test incidents to verify the full notification chain.
Cut over, update webhook URLs to point only to All Quiet, then terraform destroy the Opsgenie resources.

Why Switch to All Quiet?

Bootstrapped and independent. We aren't beholden to Private Equity, Venture Capital firms or enterprise conglomerates. We build for SREs, not for quarterly earnings.
Infrastructure as Code, natively. Our Terraform provider isn't an afterthought, it's built to be the primary way you manage your on-call setup. Resources provisioned via Terraform are locked in the web app to prevent drift.
Cost efficiency. Stop paying the "Atlassian Tax." All Quiet provides the same high-availability alerting at a fraction of the cost, with a transparent pricing model.
Less fragmentation, less drift. Opsgenie spreads on-call logic across separate schedule, rotation, and escalation resources that reference each other by ID. All Quiet collapses that into a single allquiet_team_escalations resource, fewer cross-references means fewer ways for your Terraform state to diverge from reality.

Ready to simplify your stack? Check out our Terraform Provider documentation and start your migration today.

AWS Elastic IP failover with Keepalived: how we keep self-managed loadbalancers redundant

Mads Quist — Mon, 11 May 2026 16:28:24 +0000

Originally published on 10 May 2026 on the All Quiet Tech Blog.

At All Quiet we build incident management: alerting, on-call rotations, escalation, status pages, and integrations with the monitoring stacks teams already run. A meaningful slice of my job is keeping the boring edges boring, especially ingress, when something breaks.

In parts of our stack we run loadbalancers ourselves on EC2 instead of putting every path behind an AWS-managed balancer. We do that in part to avoid leaning too hard on higher-level AWS abstractions for those tiers: we still rely on EC2 for reliable virtual machines, and we keep the design close to portable building blocks so we could run the same pattern in another data center or provider without a ground-up redesign. Once we made that choice, we still had a plain high availability (HA) problem for the active-passive pair: keep the public edge redundant.

For those tiers we use a small pattern: a stable Elastic IP (EIP), the address we publish in DNS and the stand-in for a floating IP on a traditional network; Keepalived running Virtual Router Redundancy Protocol (VRRP) between peers; and the EC2 API, mainly AssignPrivateIpAddresses and AssociateAddress, to move that EIP when mastership changes. We wire this with Ansible and the AWS Cloud Development Kit (CDK) in our infrastructure repo.

The problem in AWS terms

I grew up with patterns where a “floating IP” moves at layer 2 (L2) with gratuitous Address Resolution Protocol (ARP). Amazon Virtual Private Cloud (VPC) doesn’t work like your favorite rack fabric: public routing for Elastic IPs is enforced by AWS’s control plane, tied to a specific Elastic Network Interface (ENI) and private address on an instance.

So we split responsibilities deliberately:

Between our servers, we use Keepalived / VRRP, almost always unicast, to decide which node is primary.
Against AWS, we run a script on notify_master that calls the command-line interface (CLI) or API so the EIP actually attaches to the winner.

If we did only VRRP virtual-address tricks without AssociateAddress, we would not fix customer-visible public routing for that EIP. If we did only API moves without Keepalived, we’d lack a clean distributed agreement story on the pair. We need both layers.

Architecture at a glance

                    Elastic IP (stable in DNS)
                              │
                              ▼
              ┌───────────────────────────────┐
              │  EC2: EIP associated here     │
              │  (AssociateAddress, etc.)     │
              └───────────────────────────────┘
                              │
                  Our LB tier (e.g. HAProxy / nginx)
                              │
                           backends

On both nodes we run Keepalived with unicast peers, priorities, and a vrrp_script that reflects whether our LB process is actually alive (systemctl, curl to localhost, or whatever probe matches reality). When a node becomes MASTER, notify_master runs our failover shell script: ensure a secondary private IP, then associate the allocation.

Implementation sketch

Kernel: we often enable ip_forward / ip_nonlocal_bind where our HAProxy or nginx layout needs it. We validate per role, not globally.

Security groups: protocol 112 (Virtual Router Redundancy Protocol, VRRP) allowed between peers, not the open internet.

Keepalived: notify_master logs to a rotated file; credentials via an Identity and Access Management (IAM) instance profile where we can.

Instance identity: in production we fetch instance metadata using Instance Metadata Service version 2 (IMDSv2).

Example Keepalived skeleton (placeholders only, not a drop-in):

global_defs {
    enable_script_security
    script_user root
    vrrp_startup_delay 1
}

vrrp_script check_service {
    script "/usr/bin/systemctl is-active --quiet nginx"
    interval 2
    weight 2
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    unicast_src_ip 10.0.0.10
    unicast_peer {
        10.0.0.11
    }
    virtual_router_id 51
    priority 200
    advert_int 1
    track_script {
        check_service
    }
    notify_master "/etc/keepalived/aws-failover.sh >> /var/log/keepalived/aws-failover.log"
}

Example failover script shape (replace IDs and IPs; use IMDSv2 in production):

#!/usr/bin/env bash
set -euo pipefail

ALLOCATION_ID="eipalloc-REPLACE_ME"
PRIVATE_IP_SECONDARY="10.0.0.50"
INTERFACE="eth0"

INSTANCE_ID="$(curl -sf http://169.254.169.254/latest/meta-data/instance-id)"

ip addr add "${PRIVATE_IP_SECONDARY}/32" dev "${INTERFACE}" || true

NI_ID="$(aws ec2 describe-instances \
  --instance-ids "${INSTANCE_ID}" \
  --query 'Reservations[0].Instances[0].NetworkInterfaces[0].NetworkInterfaceId' \
  --output text)"

aws ec2 assign-private-ip-addresses \
  --network-interface-id "${NI_ID}" \
  --private-ip-addresses "${PRIVATE_IP_SECONDARY}"

aws ec2 associate-address \
  --allocation-id "${ALLOCATION_ID}" \
  --instance-id "${INSTANCE_ID}" \
  --private-ip-address "${PRIVATE_IP_SECONDARY}"

Tradeoffs of managing the edge ourselves

When we self-manage load balancer tiers instead of defaulting to AWS-managed front doors, we still need to evaluate the usual architectures: application or network load balancers, DNS failover, Kubernetes ingress, or the Elastic IP + Keepalived pattern this post describes.

Application Load Balancer (ALB)

Pros / cons (for anyone choosing ALB):

Upside: AWS-managed HA, health checks, Transport Layer Security (TLS) with AWS Certificate Manager (ACM), AWS Web Application Firewall (WAF), and a clear scaling story for HTTP.

Downside: cost at scale, less hands-on control over every packet and knob than raw EC2.

At All Quiet: we rely on managed load balancing for paths where we want AWS to own HA end-to-end, including customer-facing HTTP. We treat ALB-class tooling as the default when we do not want to operate the edge ourselves.

Network Load Balancer (NLB)

Pros / cons (for anyone choosing NLB):

Upside: TCP/UDP transparency, static IPs per Availability Zone (AZ), low listener overhead compared to full layer 7 (L7).

Downside: fewer HTTP-specific features than ALB; still another billable and operated AWS component.

At All Quiet: when we need AWS-managed HA but not full layer 7 (L7) termination at the edge, NLB-style fits better than ALB; we don’t replace every self-managed tier with NLB, but it’s on the same “managed edge” side of the spectrum as ALB.

DNS failover (Route 53)

Pros / cons (for anyone using DNS failover):

Upside: no instance-side EIP choreography; health-checked routing policies let AWS steer names.

Downside: DNS time to live (TTL) and caching stretch failover and failback; client stacks behave inconsistently; not a drop-in substitute for “one stable IP, instant swing.”

At All Quiet: DNS steering can complement other designs; we don’t rely on it alone when our mental model is exactly one Elastic IP jumping between two known EC2 nodes. That is what Keepalived plus the API covers.

Kubernetes / gateways (e.g. Amazon EKS)

Pros / cons (for anyone on Kubernetes):

Upside: HA via Services, Ingress / Gateway API, and cloud LB integration, which gives different primitives than a bare-metal pair.

Downside: cluster operational tax; not every workload belongs there.

At All Quiet: this article describes a pair pattern centered on virtual machines (VMs) because we still run meaningful tiers that way; where we use Amazon Elastic Kubernetes Service (EKS) or similar, ingress HA follows Kubernetes, not Keepalived on two fixed hosts.

Elastic IP + Keepalived + EC2 API (this post)

Pros / cons (for anyone building like this):

Upside: one stable public address in DNS; relatively few moving AWS objects; full control over timers and failover scripts.

Downside: you own IAM, idempotent scripts, logging, monitoring, and ambiguous states deserve runbooks and checks such as DescribeAddresses. Compared with a managed load balancer, cutover is not instantaneous on abrupt failure: traffic follows wherever the EIP is still associated until VRRP agrees on a new master, your health logic runs, and notify_master finishes calling AWS. The gap depends on advert_int, vrrp_script intervals, preempt settings, and API behavior. Those knobs trade sensitivity against stability; sub‑millisecond failover is not what this pattern promises.

At All Quiet: this is what we actually implemented for specific self-managed loadbalancer tiers: Ansible-deployed Keepalived, scripts on notify_master, AWS Cloud Development Kit (CDK) and infrastructure as code (IaC) for the EIP and IAM. That is the same stack this article walks through at a pattern level.

How we pick among them

Internally we ask: does this path’s service level objective (SLO) and budget justify a managed LB? Do we need layer 7 (L7) features only ALB gives us? Does scripted EIP failover fit this path’s resilience expectations (see the EIP downside above)? If not, we promote the tier (ALB/NLB or another design); we don’t stretch EIP+Keepalived past where it fits.

Takeaways

We run self-managed LBs in some slices of our infra; we needed explicit public HA there, and EIP + Keepalived + EC2 API is our compact answer.
VRRP decides who leads; AssociateAddress decides where the EIP points.
Managed ALB/NLB remain strong defaults when we want AWS to own HA at that layer.

Closing

We touch incident paths every day; when ingress misbehaves, people notice fast. If you operate similar edges, use this framing to decide when the pattern fits and when to promote that tier to managed load balancing instead.

Running our test environment fully on Nanos

Mads Quist — Thu, 18 Sep 2025 16:19:15 +0000

We've probably all had this dream of waking up in the morning and finding ourselves with 10,000 new users on our platform. (At least as a Co-Founder & CTO, that's the dream) 😁

So we asked ourselves: can we run a dedicated environment of our platform entirely on nanos?

Spoiler: it worked, and it taught us a lot about where our real inefficiencies were hiding.

Here are some of these learnings:

The Reality of Scaling:

When your startup does grow and you start scaling your infrastructure, what really happens? Often it turns out that:

Important indexes are missing, leading to CPU spikes.
Query results are loaded into memory without paging, causing out-of-memory errors.
Other inefficiencies appear that raw cloud scaling alone cannot solve.

The result is often not a sleek, hyper-scalable system but an extremely expensive piece of cloud infrastructure.

The best part about scaling: Downsizing

What's great about the cloud is not just that it scales up. It also lets you scale down.

Recently, we ran a load test we called nano testing. The idea was simple: take our infrastructure and run it on AWS EC2 nano instances. That is as small as it gets on AWS.

A nano instance has 512MB RAM and 2 virtual CPUs. That reminds me of my gaming PC from the year 2000. For comparison, your smartphone today probably has 10–20 times more RAM. We then dumped in about four times the amount of test data we normally process.

Can It Run on Nanos?

The fun begins when you try to make your platform run decently on nanos. It does not have to be blazing fast. The key question is: does it run at all?

If some API queries take a few seconds, or some asynchronous jobs and emails are processed with a bit of lag, that is perfectly fine. If your system can handle this, it means you have built something resilient.

Running on nanos lets you pinpoint weaknesses in your platform very clearly. Fixes are often low-hanging fruit:

Add batch processing here.
Add an index there.
Add throttling in another place.

Our Approach at All Quiet

At All Quiet, we now run a dedicated testing environment entirely on nanos. This allows us to observe scaling issues early and fix them before they turn into real problems.

Why We Built a MongoDB-Message Queue and Reinvented the Wheel

Mads Quist — Thu, 04 Jul 2024 04:33:19 +0000

Hey👋

I'm Mads Quist, founder of All Quiet . We've implemented a home-grown message queue based on MongoDB and I'm here to talk about:

Why we re-invented the wheel
How we re-invented the wheel

1. Why we re-invented the wheel

Why do we need message queuing?

All Quiet is a modern incident management platform, similar to PagerDuty. Our platform requires features like:

Sending a double-opt-in email asynchronously after a user registers
Sending a reminder email 24 hours after registration
Sending push notifications with Firebase Cloud Messaging (FCM), which can fail due to network or load problems. As push notifications are crucial to our app, we need to retry sending them if there's an issue.
Accepting emails from outside our integration and processing them into incidents. This process can fail, so we wanted to decouple it and process each email payload on a queue.

Our tech stack

To understand our specific requirements, it's important to get some insights into our tech stack:

We run a monolithic web application based on .NET Core 7. The .NET Core application runs in a Docker container.
We run multiple containers in parallel.
An HAProxy instance distributes HTTP requests equally to each container, ensuring a highly available setup.
We use MongoDB as our underlying database, replicated across availability zones.
All of the above components are hosted by AWS on generic EC2 VMs.

Why we re-invented the wheel

We desired a simple queuing mechanism that could run in multiple processes simultaneously while guaranteeing that each message was processed only once.
We didn't need a pub/sub pattern.
We didn't aim for a complex distributed system based on CQRS / event sourcing because, you know, the first rule of distributed systems is to not distribute.
We wanted to keep things as simple as possible, following the philosophy of choosing "boring technology".

Ultimately, it's about minimizing the number of moving parts in your infrastructure. We aim to build fantastic features for our excellent customers, and it's imperative to maintain our services reliably. Managing a single database system to achieve more than five nines of uptime is challenging enough. So why burden yourself with managing an additional HA RabbitMQ cluster?

Why not just use AWS SQS?

Yeah… cloud solutions like AWS SQS, Google Cloud Tasks, or Azure Queue Storage are fantastic! However, they would have resulted in vendor lock-in. We simply aspire to be independent and cost-effective while still providing a scalable service to our clients.

2. How we re-invented the wheel

What is a message queue?

A message queue is a system that stores messages. Producers of messages store these in the queue, which are later dequeued by consumers for processing. This is incredibly beneficial for decoupling components, especially when processing messages is a resource-intensive task.

What characteristics should our queue show?

Utilizing MongoDB as our data storage
Guaranteeing that each message is consumed only once
Allowing multiple consumers to process messages simultaneously
Ensuring that if message processing fails, retries are possible
Enabling scheduling of message consumption for the future
Not needing guaranteed ordering
Ensuring high availability
Ensuring messages and their states are durable and can withstand restarts or extended downtimes

MongoDB has significantly evolved over the years and can meet the criteria listed above.

Implementation

In the sections that follow, I'll guide you through the MongoDB-specific implementation of our message queue. While you'll need a client library suitable for your preferred programming language, such as NodeJS, Go, or C# in the case of All Quiet, the concepts I'll share are platform agnostic.

Queues

Each queue you want to utilize is represented as a dedicated collection in your MongoDB database.
Message Model

Here's an example of a processed message:

{
    "_id" : NumberLong(638269014234217933),
    "Statuses" : [
        {
            "Status" : "Processed",
            "Timestamp" : ISODate("2023-08-06T06:50:23.753+0000"),
            "NextReevaluation" : null
        },
        {
            "Status" : "Processing",
            "Timestamp" : ISODate("2023-08-06T06:50:23.572+0000"),
            "NextReevaluation" : null
        },
        {
            "Status" : "Enqueued",
            "Timestamp" : ISODate("2023-08-06T06:50:23.421+0000"),
            "NextReevaluation" : null
        }
    ],
    "Payload" : {
        "YourData" : "abc123"
    }
}

Let’s look at each property of the message.

_id

The _id field is the canonical unique identifier property of MongoDB. Here, it contains a NumberLong, not an ObjectId . We need NumberLong instead of ObjectId because:

While ObjectId values should increase over time, they are not necessarily monotonic. This is because they:

Only contain one second of temporal resolution, so ObjectId values created within the same second do not have a guaranteed ordering, and are generated by clients, which may have differing system clocks.

In our C# implementation, we generate an Id with millisecond precision and guaranteed ordering based on insertion time. Although we don't require strict processing order in a multi-consumer environment (similar to RabbitMQ), it's essential to maintain FIFO order when operating with just one consumer. Achieving this with ObjectId is not feasible. If this isn't crucial for you, you can still use ObjectId.

Statuses

The Statuses property consists of an array containing the message processing history. At index 0, you'll find the current status, which is crucial for indexing.

The status object itself contains three properties:

Status: Can be "Enqueued", "Processing", "Processed", or "Failed".
Timestamp: This captures the current timestamp.
NextReevaluation: Records when the next evaluation should occur, which is essential for both retries and future scheduled executions.

Payload

This property contains the specific payload of your message.

Enqueuing a message

Adding a message is a straightforward insert operation into the collection with the status set to "Enqueued".

For immediate processing, set NextReevaluation to null.
For future processing, set NextReevaluation to a timestamp in the future, when you want your message to be processed.

db.yourQueueCollection.insert({
    "_id" : NumberLong(638269014234217933),
    "Statuses" : [
        {
            "Status" : "Enqueued",
            "Timestamp" : ISODate("2023-08-06T06:50:23.421+0000"),
            "NextReevaluation" : null
        }
    ],
    "Payload" : {
        "YourData" : "abc123"
    }
});

Dequeuing a message

Dequeuing is slightly more complex but still relatively straightforward. It heavily relies on the concurrent atomic read and update capabilities of MongoDB.

This essential feature of MongoDB ensures:

Each message is processed only once.
Multiple consumers can safely process messages simultaneously.

db.yourQueueCollection.findAndModify({
   "query": {
      "$and": [
         {
            "Statuses.0.Status": "Enqueued"
         },
         {
            "Statuses.0.NextReevaluation": null
         }
      ]
   },
   "update": {
      "$push": {
         "Statuses": {
            "$each": [
               {
                  "Status": "Processing",
                  "Timestamp": ISODate("2023-08-06T06:50:23.800+0000"),
                  "NextReevaluation": null
               }
            ],
            "$position": 0
         }
      }
   }
});

So we are reading one message that is in state “Enqueued” and at the same time modify it by setting the status “Processing” at position 0. Since this operation is atomic it will guarantee that the message will not be picked up by another consumer.

Marking a message as processed

Once the processing of the message is complete, it's a simple matter of updating the message status to "Processed" using the message’s id.

db.yourQueueCollection.findAndModify({
   "query": {
     "_id": NumberLong(638269014234217933)
   },
   "update": {
      "$push": {
         "Statuses": {
            "$each": [
               {
                  "Status": "Processed",
                  "Timestamp": ISODate("2023-08-06T06:50:24.100+0000"),
                  "NextReevaluation": null
               }
            ],
            "$position": 0
         }
      }
   }
});

Marking a message as failed

If processing fails, we need to mark the message accordingly. Often, you might want to retry processing the message. This can be achieved by re-enqueuing the message. In many scenarios, it makes sense to reprocess the message after a specific delay, such as 10 seconds, depending on the nature of the processing failure.

db.yourQueueCollection.findAndModify({
   "query": {
     "_id": NumberLong(638269014234217933)
   },
   "update": {
      "$push": {
         "Statuses": {
            "$each": [
               {
                  "Status": "Failed",
                  "Timestamp": ISODate("2023-08-06T06:50:24.100+0000"),
                  "NextReevaluation": ISODate("2023-08-06T07:00:24.100+0000")
               }
            ],
            "$position": 0
         }
      }
   }
});

The dequeuing loop

We've established how we can easily enqueue and dequeue items from our "queue," which is, in fact, simply a MongoDB collection. We can even "schedule" messages for the future by leveraging the NextReevaluation field.

What's missing is how we will dequeue regularly. Consumers need to execute the findAndModify command in some kind of loop. A straightforward approach would be to create an endless loop in which we dequeue and process a message. This method is straightforward and effective. However, it will exert considerable pressure on the database and the network.

An alternative would be to introduce a delay, e.g., 100ms, between loop iterations. This will significantly reduce the load but will also decrease the speed of dequeuing.

The solution to the problem is what MongoDB refers to as a change stream.

MongoDB Change Streams

What are change streams? I can’t explain it better than the guys at MongoDB:

Change streams allow applications to access real-time data changes […]. Applications can use change streams to subscribe to all data changes on a single collection […] and immediately react to them.

Great! What we can do is listen to newly created documents in our queue collection, which effectively means listening to newly enqueued messages

This is dead simple:

const changeStream = db.yourQueueCollection.watch();
changeStream.on('insert', changeEvent => {
  // Dequeue the message
  db.yourQueueCollection.findAndModify({
    "query": changeEvent.documentKey._id,
    "update": {
      "$push": {
         "Statuses": {
            "$each": [
               {
                  "Status": "Processing",
                  "Timestamp": ISODate("2023-08-06T06:50:24.100+0000"),
                  "NextReevaluation": null
               }
            ],
            "$position": 0
         }
      }
   }
});

Scheduled and Orphaned Messages

The change stream approach, however, does not work for both scheduled and orphaned messages because there is obviously no change that we can listen to.

Scheduled messages simply sit in the collection with the status "Enqueued" and a "NextReevaluation" field set to the future.
Orphaned messages are those that were in the "Processing" status when their consumer process died. They remain in the collection with the status "Processing" but no consumer will ever change their status to "Processed" or "Failed".

For these use cases, we need to revert to our simple loop. However, we can use a rather generous delay between iterations.

Wrapping it up

"Traditional" databases, like MySQL, PostgreSQL, or MongoDB (which I also view as traditional), are incredibly powerful today. If used correctly (ensure your indexes are optimized!), they are swift, scale impressively, and are cost-effective on traditional hosting platforms.

Many use cases can be addressed using just a database and your preferred programming language. It's not always necessary to have the "right tool for the right job," meaning maintaining a diverse set of tools like Redis, Elasticsearch, RabbitMQ, etc. Often, the maintenance overhead isn't worth it.

While the solution proposed might not match the performance of, for instance, RabbitMQ, it's usually sufficient and can scale to a point that would mark significant success for your startup.

Software engineering is about navigating trade-offs. Choose yours wisely.