DEV Community: Alex Gutscher

Building Multi-Region Consensus Checks with Cloudflare Durable Objects

Alex Gutscher — Mon, 24 Aug 2026 17:22:58 +0000

Every uptime monitor I've used has woken me up for an outage that never happened.

The mechanism is always the same, and it isn't a bug. A monitor checks your endpoint from one location. Somewhere between that prober and your server, a packet drops, or a route flaps, or a TLS handshake takes 400ms longer than the timeout allows. From the prober's point of view, that is indistinguishable from your service being down.

This is worth stating precisely, because it explains why the usual fixes don't work: a single observer cannot separate a fault in the thing being observed from a fault in the path to it. It's not a tuning problem. It's an epistemic limit of having one witness.

So we built the check layer around having several, and requiring them to agree. This post is how that works, including the four cases that took far longer to get right than the happy path.

Why the usual fixes don't fix it
Before the interesting part, the two things everyone tries first.

Retries. Check again after N seconds; only alert if it fails twice. This genuinely helps with single dropped packets, and you should turn it on whatever tool you use. But it doesn't help at all with anything that persists for more than a few seconds — a route flap lasting 90 seconds fails every retry from that one vantage point, and you get paged for a network condition that never touched your users.

Multiple probes in one region. Better, and it's what several vendors mean by "multi-node." But probes in the same datacenter share upstream transit, share a BGP view, and often share an ASN. Their failures are correlated, and correlated witnesses don't give you independent confirmation — they give you one opinion with extra steps.

What you actually need is witnesses that fail independently. That means genuine geographic and network separation, which historically meant operating servers in seven countries. That's why consensus verification only ever existed in enterprise tooling: not because the idea was hard, but because the infrastructure bill was.

Why Durable Objects change the economics
Cloudflare Durable Objects have one property that matters enormously here: you can pin an instance to a geographic region with locationHint, and it stays there, holding state, with no servers to operate.

const REGIONS = [
'wnam', // western north america
'enam', // eastern north america
'weur', // western europe
'eeur', // eastern europe
'apac', // asia-pacific
'apac-ne', // japan / korea
'apac-se', // southeast asia
] as const;

type Region = typeof REGIONS[number];

function probeFor(env: Env, region: Region): DurableObjectStub {
const id = env.PROBE.idFromName(probe:${region});
return env.PROBE.get(id, { locationHint: region });
}

That's the whole trick. Seven pinned instances, each egressing from a different part of Cloudflare's network, each capable of independently checking an endpoint and reporting what it saw. What used to be a procurement exercise is now a scheduling problem.

The cost collapse is the point. It means an architecture that was previously enterprise-only can run on a free tier, which is the reason we could build this as a small team at all.

The check cycle
Each cycle, a coordinator fans out to all regions in parallel and collects results. The important design decision is in the result type:

type ProbeResult =
| { region: Region; status: 'ok'; ms: number }
| { region: Region; status: 'fail'; errorClass: ErrorClass; ms: number }
| { region: Region; status: 'inconclusive'; reason: 'timeout' | 'flapping' | 'unreachable' };
Note that there are three states, not two.

Almost every monitoring system I've looked at models a check as binary — up or down. That collapse is where false positives are born. A region that didn't respond in time has not told you the target is down; it has told you nothing. Those are completely different pieces of information and they must not be added together.

The fan-out itself is unremarkable:

async function runCycle(env: Env, monitor: Monitor): Promise {
const settled = await Promise.allSettled(
REGIONS.map(async (region) => {
const stub = probeFor(env, region);
const res = await stub.fetch('https://probe/check', {
method: 'POST',
body: JSON.stringify({ url: monitor.url, timeoutMs: monitor.timeoutMs }),
});
return (await res.json()) as ProbeResult;
})
);

return settled.map((s, i) =>
s.status === 'fulfilled'
? s.value
: { region: REGIONS[i], status: 'inconclusive' as const, reason: 'unreachable' as const }
);
}
If we can't reach our own probe, that's inconclusive. Not a failure of the customer's endpoint. It sounds obvious written down; it is very easy to get wrong when you're mapping exceptions to booleans at 2am.

The local re-check
Before a region reports a failure at all, it re-checks immediately:

async function check(url: string, timeoutMs: number): Promise {
const first = await attempt(url, timeoutMs);
if (first.status === 'ok') return first;

// A failure is a hypothesis, not a conclusion. Test it again now,
// rather than waiting for the next cycle.
await sleep(250);
const second = await attempt(url, timeoutMs);

if (second.status === 'ok') {
// Transient. Record it for latency analytics, report healthy.
recordTransient(url, first);
return second;
}
return second;
}
This is cheap and it kills a surprising amount of noise — single dropped packets and one-off TLS timeouts die here, silently, without ever entering the consensus layer. Do this even if you never build quorum.

The quorum decision
Here's the core. Given seven results, decide what happened:

type Verdict =
| { kind: 'healthy' }
| { kind: 'regional_degradation'; failing: Region[]; dominant: ErrorClass }
| { kind: 'global_outage'; failing: Region[]; dominant: ErrorClass }
| { kind: 'insufficient_witnesses'; counted: number; required: number };

function decide(results: ProbeResult[], quorum = 4): Verdict {
// Inconclusive results are excluded from the denominator entirely.
const counted = results.filter((r) => r.status !== 'inconclusive');
const failing = counted.filter((r) => r.status === 'fail');

// If too few regions gave us a real answer, we cannot make a call.
// Saying "I don't know" is a valid and important output.
if (counted.length < quorum) {
return { kind: 'insufficient_witnesses', counted: counted.length, required: quorum };
}

if (failing.length >= quorum) {
return {
kind: 'global_outage',
failing: failing.map((f) => f.region),
dominant: dominantErrorClass(failing),
};
}

if (failing.length > 0) {
return {
kind: 'regional_degradation',
failing: failing.map((f) => f.region),
dominant: dominantErrorClass(failing),
};
}

return { kind: 'healthy' };
}
Two things in there are worth more than the rest of the function.

Inconclusive results are excluded from the denominator, not counted as failures. If a region times out and you count it as a failure, one congested route can push you over quorum and page someone for nothing. If you count it as a success, one congested route can suppress a real outage. Both are unacceptable, and the only correct answer is to remove it from the vote.

insufficient_witnesses is a real verdict. If five of seven regions come back inconclusive, you do not have enough information to say anything about the target. Most systems would either alert or stay silent; both are wrong, because both assert something you don't know. We surface it as its own state and it goes to our own operational alerting, not the customer's.

And the classification matters as much as the threshold. Three regions failing while four succeed is not an outage — it's almost always a CDN edge or a geo-routing rule, affecting some users and not others. That deserves a different notification, at a different urgency, than "your service is gone." Flattening both into the word DOWN is how you train people to ignore the word DOWN.

The four things that took longest
The happy path above took a couple of weeks. These took months.

Flapping probes A probe with a bad network path reports failures that aren't real. If one of your own regions is unhealthy, it poisons every decision it participates in. So probes are continuously assessed and ejected when they misbehave:

const FLAP_WINDOW_MS = 2 * 60 * 60 * 1000;
const FLAP_THRESHOLD = 3;

function isFlapping(transitions: number[], now: number): boolean {
const cutoff = now - FLAP_WINDOW_MS;
return transitions.filter((t) => t >= cutoff).length >= FLAP_THRESHOLD;
}
Three or more state transitions in two hours and the probe is marked flapping and removed from quorum until it stabilises. Its results become inconclusive rather than being discarded silently — the distinction matters when you're debugging your own fleet later.

We publish probe health, including flapping status, on a public page. That was uncomfortable to commit to and has been unambiguously worth it.

Correlated failure of the witnesses This is the subtle one, and if you build something like this, it's the bug you'll ship.

All seven regions run on Cloudflare. They are geographically independent. They are not independent of Cloudflare. If Cloudflare has a bad day, all seven probes can fail simultaneously — and a naive quorum implementation reads seven-of-seven unanimous failure as maximum confidence and pages every customer at once with a completely false alarm.

Your consensus system, at its moment of greatest apparent certainty, is maximally wrong.

The fix is a witness that doesn't share the failure mode: an out-of-band sentinel on a completely different ASN, on a cheap VPS, deliberately not on the same network as anything else we run.

function reconcile(edge: Verdict, sentinel: ProbeResult | null): Verdict {
// Unanimous edge failure + a healthy independent sentinel means
// the problem is far more likely to be us than the customer.
if (
edge.kind === 'global_outage' &&
edge.failing.length === REGIONS.length &&
sentinel?.status === 'ok'
) {
return { kind: 'insufficient_witnesses', counted: 1, required: 4 };
}
return edge;
}
When every edge probe disagrees with the one witness on a different network, suspect the observer, not the target. This is the single highest-value component of the whole design, and it runs on a VPS costing a few euros a month.

Distinguishing a blocked probe from a dead one If a probe stops reporting, you need to know whether it crashed or whether something is preventing it from reaching the target. Those imply completely different responses.

So every probe reports liveness on a channel entirely separate from its measurement path. A probe that is alive but measuring nothing is a routing or firewall problem. A probe that is silent on both is a crashed probe. Same symptom in the results table, different root cause, and you cannot tell them apart without the second channel.

Being un-allowlistable Worth mentioning because it's a real operational consequence of this architecture. Because probes run as Durable Objects egressing through Cloudflare's shared edge IP pool, there is no stable IP list customers can allowlist. We can't give them one, and any list we published would be wrong within a week.

Instead customers match on a header:

Cloudflare WAF custom rule

(http.request.headers["cf-worker"][0] eq "steadystack.dev")

Action: skip WAF / rate limiting

If you build synthetic checks on edge compute, plan for this early. It surprised us, and "just allowlist our IPs" is the first thing a security-conscious customer asks for.

What it costs
The honest accounting: seven regions checking every cycle is 7× the check volume of a single-probe monitor, per monitor, forever.

On classic infrastructure that would be prohibitive at the low end of the market. On edge compute, a check is a few milliseconds of CPU and one outbound request, and the per-check cost is small enough that we ship multi-region confirmation on our free tier rather than gating it behind a plan. That was a deliberate decision — the whole thesis is that consensus shouldn't be a premium feature, because a monitor you can't trust isn't a cheaper monitor, it's a broken one.

What I'd do differently
Model the three-state result on day one. We started with a boolean and retrofitted inconclusive later. That refactor touched everything, and every subtle bug we shipped in the first months traced back to somewhere the two-state assumption was still hiding.

Build the out-of-band sentinel before the seventh region. We added regions for coverage before we added a witness for independence. Coverage is worth less than independence, and it took a scare to notice.

Store the error class from the beginning. Knowing that four regions failed is useful. Knowing they all failed at TCP connect rather than with 5xx is the difference between "routing" and "your origin," and it's the first question anyone asks during an incident.

Honest limits
We check from North America, Europe, and Asia-Pacific. We do not have probes in Africa or the Middle East. If your users are concentrated there, a global outage will still page you, but a problem affecting only those users may not surface. That's on our locations page rather than buried here, because a padded coverage map is worse than a small one.

And consensus is not free of tradeoffs. Requiring four regions to agree means a genuine outage affecting only three regions won't page you as an outage — it'll be classified as regional degradation. We think that's correct. If you disagree for your workload, that threshold should be yours to set.

I build SteadyStack, which is where all of this runs. The free tier is 50 monitors with multi-region quorum included, if you want to look at the mechanism rather than read about it.

If you're building something similar, I'd genuinely like to compare notes on the correlated-witness problem — it's the part I've found least written about, and I suspect a lot of consensus systems have it without knowing.

I built a testimonial wall tool and the hardest part wasn't the product — it was the embed widget

Alex Gutscher — Tue, 21 Apr 2026 23:56:00 +0000

I launched KudosWall this week — a tool that lets you collect text and video testimonials via a shareable link and embed them on any site with one script tag. I want to talk about the technical decision that took the longest and caused the most pain: how to actually deliver the widget to a third-party page.

The problem with embedding on sites you don't control

When someone pastes your script tag into their WordPress site, their Webflow page, their Framer template, or their raw HTML file — you have zero control over what CSS is already on that page. A .card class on their site will clobber yours. Their * { box-sizing: content-box } will break your layout. Their p { margin: 2rem } will mess up your text spacing.

There are three ways to handle this:

Option 1: CSS namespacing

Prefix all your classes with something unique like .kw-card and hope nothing collides. Works until it doesn't. Some WordPress themes use aggressive selectors like div div div { margin: 0 } that blow straight through your namespacing.

Option 2: Shadow DOM

The widget's DOM lives inside a shadow root — a hard boundary host CSS cannot cross. Great for isolation, but forces you to bundle your entire UI library (React/Tailwind) into a single client-side JS file, which bloats the payload and hurts the initial "Time to First Testimonial."

Option 3: The "Modern" Iframe (what I went with)

I chose a server-rendered iframe approach. It provides total CSS isolation by default, but I solved the two classic iframe dealbreakers: height management and SEO.

By using a tiny loader script and a ResizeObserver inside the widget, the iframe communicates with the host page via postMessage to auto-resize instantly as content loads or carousels move. For SEO, I inject JSON-LD Structured Data directly into the embed page. This means even though the UI is in an iframe, Google still crawls the testimonials and awards the host page those coveted "Review Snippet" stars in search results.

The async script problem

Here's something that bit me early. The standard way to read a script tag's own attributes is document.currentScript:

// This breaks with async
const id = document.currentScript.getAttribute('data-id');

document.currentScript is often null inside an async script because the script executes after the DOM has already parsed. Since performance is non-negotiable — a synchronous script blocks page rendering — I built a robust fallback:

// Robust detection for async/defer
let targetScript = document.currentScript;

if (!targetScript) {
  const scripts = document.getElementsByTagName('script');
  for (let i = scripts.length - 1; i >= 0; i--) {
    const s = scripts[i];
    if (s.getAttribute('data-id') && s.src.indexOf('widget.js') !== -1) {
      targetScript = s;
      break;
    }
  }
}

This ensures the widget always finds its configuration ID regardless of how the host site handles script loading.

Keeping it small

The widget.js loader is under 3KB. Its only job is to detect the data-id, inject the iframe, and listen for resize events.

The heavy lifting — rendering the UI, handling dark/light mode, and animations — is done server-side in Next.js. This keeps the client-side JS footprint on the customer's site near zero while allowing me to use the full power of React and Tailwind CSS for the widget itself.

The data layer

The widget content is rendered via Next.js Server Components. When the iframe hits /embed/:id, we fetch the approved testimonials directly from the database:

// Fetching via Drizzle ORM in a Server Component
const testimonialsList = await db.query.testimonial.findMany({
  where: and(
    inArray(testimonial.projectId, projectIds),
    eq(testimonial.status, 'approved')
  ),
  orderBy: desc(testimonial.createdAt),
  limit: settings.maxItems || 20,
});

Because it's server-rendered, the first paint is nearly instantaneous. No loading spinner inside the widget while waiting for a client-side fetch.

The rest of the stack

Frontend: Next.js 16 (App Router), Tailwind CSS v4
Database: PostgreSQL (Neon) with Drizzle ORM
Auth: Better-Auth (handles multi-tenant session isolation)
Payments: Stripe Billing + Webhooks
Email: Resend
Storage: Cloudflare R2 (for high-bandwidth video testimonials)
Deployment: Vercel

Drizzle's relational queries handle multi-tenant data isolation. Each workspace's data is strictly separated at the query level.

What's live

KudosWall is live at kudoswall.org. Free tier: 5 testimonials, one active widget, no time limit. Pro: $29/mo for unlimited testimonials, all layouts, and removal of the "Powered by" branding.

The embed snippet is as simple as it gets:

<script
  src="https://kudoswall.org/widget.js"
  data-id="cfg_abc123"
  async
></script>

That's the whole integration. Works on WordPress, Webflow, Framer, and Shopify without any CSS collisions or layout shifts.

Happy to answer questions about the postMessage resize logic, the Drizzle + Neon setup, or why I moved away from Supabase to the Drizzle/Better-Auth stack.

Production OpenClaw in One Day: A Copy-Paste Checklist

Alex Gutscher — Wed, 25 Feb 2026 23:02:09 +0000

The distance between a "cool agent demo" and a "production AI fleet" is paved with edge cases, token leaks, and security vulnerabilities. Most teams spend months building the plumbing before they can trust their agents. We think that's a waste of time.

The "Day One" Production Checklist
[1] Telemetry & Visibility
☐ Centralized logging for Chain-of-Thought (don't rely on local stdout).
☐ Real-time token cost attribution per agent/task.
☐ Sub-10ms latency for tool-call interception.
[2] Security & Handshakes
☐ Zero-knowledge secret management (agents should never store API keys).
☐ Dynamic E2E encryption for agent-to-gateway telemetry.
☐ Tool-call signature verification to prevent command hijacking.
[3] Governance & Guardrails
☐ Runtime cost caps per agent session (the "$5 kill-switch").
☐ Domain/URL allowlists for agents with web access.
☐ PII scrubbing policies for logs and reasoning traces.

Conclusion: Tick Every Box Automatically
You can build all of this yourself. It will take you roughly 400 engineering hours and a dedicated infrastructure team. Or, you can connect your OpenClaw agents to ClawTrace and tick every single one of these boxes in about 15 minutes.

ClawTrace is the "Production Mode" button for OpenClaw. We provide the gateway, the dashboard, security, and governance out of the box, so you can focus on building the agents that drive your business.

📸 Meet env-snapper: Snapshot & Track Your .env Files with Ease

Alex Gutscher — Tue, 22 Apr 2025 20:56:07 +0000

Have you ever spent way too long debugging a broken app, only to realize your .env file silently changed?
Yeah… me too.

That’s why I built env-snapper — a simple CLI tool to help developers take snapshots of their .env files and compare them over time.

🧠 Why I built it
Environment variables are essential for modern apps, but they're also fragile.

One typo, one missing variable, or an accidental overwrite — and suddenly your app doesn’t work, or worse, it's working weirdly.

If you’re working across different machines, teams, or environments, tracking what changed (and when) in your .env can be a headache.

So I thought: “Why not just snapshot it and compare later?”

⚙️ What env-snapper does
Here’s what it brings to the table:

✅ Snapshot your current .env file

✅ Automatically timestamp and version each snapshot

✅ Diff between snapshots to spot changes

✅ CLI interface (easy to use via npx)

✅ No dependencies on external services — local and fast

🛠️ How to use it

npm install -g env-snap

## Initialize env-snap in your project
npx env-snap init

# Manually snapshot current .env
npx env-snap snapshot

# List all snapshots
npx env-snap list

# Revert to a previous snapshot
npx env-snap revert <snapshot-id>

# Show diff between snapshots or with current .env
npx env-snap diff <snapshot-id>            # Compare with previous snapshot
npx env-snap diff <snapshot-id> --current  # Compare with current .env

# Watch .env for changes and snapshot automatically
npx env-snap watch

# Add a description to a snapshot (after creation)
npx env-snap desc <snapshot-id> "Your description here"

# Example: create a snapshot with a description
npx env-snap snapshot --desc "Added Sentry config and removed Stripe keys"

# Prune old snapshots, keeping only the latest 5 (default)
npx env-snap prune

# Prune and keep only the latest N snapshots (e.g., 10)
npx env-snap prune 10

# Preview changes before restoring a snapshot
npx env-snap preview <snapshot-id>

You'll see a clean, side-by-side diff of what changed between your last two snapshots.

💡 Use cases
🧪 Testing config changes between deployments

👯 Collaborating across teams with different environments

🛠️ Debugging issues caused by subtle .env tweaks

🧳 Moving between machines without losing setup details

📦 Get it now
You can check out the package here:
👉 https://www.npmjs.com/package/env-snapper

The code is open source and I’d love your feedback, feature requests, or PRs!

💬 Let’s connect!
Got questions, ideas, or bugs? Drop a comment below or reach out on X/Twitter [@HustleStacked].
Let’s keep our environments sane. 😄

🚀 Introducing MonitorFlow: Your Open-Source SaaS for Real-Time Monitoring & Analytics 📊

Alex Gutscher — Tue, 12 Nov 2024 21:13:14 +0000

Hey Dev.to community! 👋 I’m thrilled to share MonitorFlow, an open-source SaaS platform built to simplify real-time monitoring and analytics for developers, data engineers, and DevOps teams.

Why I Built MonitorFlow
After struggling with expensive, overcomplicated monitoring tools that didn’t fit my needs, I decided to build MonitorFlow to be the tool I (and others) actually wanted. It’s built for teams who need to track, visualize, and respond to metrics in real time, without the bloat or hidden fees.

Key Features:
📊 Real-Time Monitoring: Keep track of server health, app performance, and other key metrics in real time.
🎛️ Custom Dashboards: Build your dashboards to focus on what matters to you.
🔔 Alerts & Notifications: Get notified instantly when something important happens.
🔄 Extensible and Open-Source: As an open-source project, MonitorFlow is designed to be customizable and community-driven.
How You Can Get Involved
MonitorFlow is still in the early stages, and I’m looking for contributors to help make it even better! Here’s how you can help:

Check it out on GitHub: Clone the repo, try it out, and let me know what you think.
Contribute to the Code: We have “good first issues” if you’re new to open source or looking for a place to start.
Suggest Features: Let’s make this tool better together! Have a feature in mind? Let me know, or open an issue.
Why Contribute?
Learn & Grow: Work on an open-source SaaS project, and build your portfolio with real-world contributions.
Collaborate with the Community: Work alongside other developers and data enthusiasts to improve the project.
Help Shape the Future: Your ideas and contributions will directly impact the direction of MonitorFlow.
🔗 [Check out MonitorFlow on GitHub]

Looking forward to hearing your thoughts, and if you want to collaborate, let’s chat! 🙌

Feel free to personalize the post and add any extra details that might help explain the project better to the Dev.to community.