DEV Community: binadit

From 4.2s to 380ms: debugging latency in a high availability infrastructure setup

binadit — Fri, 11 Sep 2026 07:05:45 +0000

Five silent bottlenecks that turned a 400ms API into a 4.2s crawl

No deploys. No schema changes. No new integrations. Just a scheduling and resource-planning SaaS with 40k users watching its p95 response time climb from 400ms to 4.2 seconds over six months. Nobody could point to a cause, because there wasn't one big cause, there were five small ones stacked on top of each other.

This is the writeup of how we found and fixed them, without a rearchitecture.

The symptoms before the numbers existed

Support tickets called the dashboard "laggy" long before anyone had hard metrics. By the time it got measured:

p95 latency: 400ms to 4.2s over ~6 months
Trial-to-paid conversion down 11% in the same window
Sales getting asked about performance during renewals

The team had already tried the usual moves: bigger instances, a read replica, scheduled restarts. None of it held. When scaling vertically twice does nothing, that's a strong signal the problem is architectural, not capacity.

Step one: measure before touching anything

We spent the first week purely on instrumentation, tracing requests across the API gateway, app servers, database, and cache layer under normal load. No fixes yet. Changing the system while you're trying to measure it just adds noise.

Five issues surfaced. None of them alone explained 4.2 seconds, but together they did.

The five issues

1. N+1 queries in a hot path

A dashboard endpoint fetched a customer's project list, then hit the DB once per project for status. 40 projects meant 41 round trips for one page load.

2. Connection pool exhaustion

8 app servers x 20 connections each = 160 possible connections, against a Postgres max_connections of 100. At peak, requests queued for a connection slot before a query even ran.

3. Cache invalidation nuking whole namespaces

Redis had a 30s TTL on project status, reasonable on paper. But a background job flushed entire cache namespaces on any write, including unrelated ones. Actual hit ratio: 34%.

4. Cross-AZ chatter

App tier and Redis cluster weren't pinned to the same AZ. ~40% of Redis calls crossed zones, adding 2 to 4ms each. At volume, that's hundreds of milliseconds per request.

5. A synchronous third-party call in the request path

A usage-tracking webhook to an external analytics provider ran synchronously inside the request cycle. On a bad day that provider took 800ms to 1.5s, and every request waited on it.

How we prioritized fixes

Impact vs. risk, not ease of implementation:

Sync third-party call, high impact, low risk
N+1 queries, high impact, low risk
Connection pool exhaustion, high impact, medium risk
Cache invalidation, medium impact, medium risk
Cross-AZ placement, medium impact, low risk

We explicitly skipped touching instance sizes again. Two rounds of vertical scaling with zero improvement was itself the data point: the bottleneck was I/O and architecture, not compute.

We also shipped fixes one at a time, measuring after each. Bundling everything into one release makes it impossible to know what actually helped, or what quietly broke something else.

The fixes

Move the analytics call off the request path

// before: synchronous call blocking the response
await analyticsClient.track(event); // 800ms-1.5s on slow days
return response;

// after: fire-and-forget via queue
await queue.push('analytics.track', event); // ~2ms
return response;

Used the existing Redis infra as the queue backend, no new dependency. A worker process handles delivery with retries and a dead-letter queue. This alone cut 800ms to 1.5s off the median request on affected endpoints, and when the analytics provider had two outages during the engagement, users never noticed.

Kill the N+1

-- before
SELECT * FROM projects WHERE account_id = $1;
-- then per project:
SELECT * FROM project_status WHERE project_id = $1;

-- after
SELECT p.*, s.status, s.updated_at
FROM projects p
JOIN project_status s ON s.project_id = p.id
WHERE p.account_id = $1;

41 round trips became 1. Query time for that endpoint went from ~620ms to ~45ms average.

Retune the connection pool

Dropped per-server pool size from 20 to 12 (8 x 12 = 96, under the 100 limit) and added PgBouncer in transaction pooling mode:

[databases]
app_db = host=127.0.0.1 port=5432 dbname=app_production

[pgbouncer]
pool_mode = transaction
max_client_conn = 500
default_pool_size = 25
reserve_pool_size = 5
reserve_pool_timeout = 3

Connection wait time went from ~180ms average at peak to under 5ms.

Fix cache invalidation

Switched from namespace-wide flushes on any write to key-level invalidation tied to the specific project changed. Kept the 30s TTL as a safety net, but it stopped being the primary mechanism.

Cache hit ratio: 34% to 91% within a week. Small diff, outsized effect on database load.

Takeaway

None of these fixes were exotic. This is what accumulated technical debt looks like in a system that grew for a few years without a dedicated performance pass. If your latency crept up gradually with no single obvious cause, look for a stack of small architectural issues before you reach for bigger hardware.

Full writeup with more detail: Debugging latency in a high availability infrastructure setup

Originally published on binadit.com

How Docker networking broke checkout under load: a container ecommerce infrastructure case study

binadit — Wed, 09 Sep 2026 07:02:54 +0000

Checkout was timing out at 900 req/s, and it had nothing to do with CPU

A marketplace client came to us with a scary but familiar symptom: checkout p95 latency jumping from 280ms to over 2.1 seconds during traffic spikes. Their instinct was to throw more containers at it. That made things worse. Here's what was actually going on, and how we fixed it without touching a line of application code.

Link to the full writeup: Docker networking broke checkout under load

The setup

PHP monolith, containerized about a year prior, running on a single host via Docker Compose: app containers, Redis, a worker queue, behind a managed load balancer. 40k DAU, peak ~900 req/s. It ran fine for six months. Then flash sales started producing "the site is freezing" tickets.

The team's first move was scaling app containers, assuming CPU/memory pressure. Query times on Postgres stayed stable at 8-14ms the whole time, so the database wasn't the culprit either. The bottleneck was hiding in the network layer between containers, a place default tooling barely gives you visibility into.

What the audit found

Three compounding issues, none fatal alone:

1. Default bridge network overhead. Every inter-container hop (app to Redis, app to Postgres, app to search) was going through userland proxying on the default bridge. We measured ~1.8ms added latency per hop. At 900 req/s with 3-4 internal calls per request, that adds up fast.

2. Conntrack table exhaustion. nf_conntrack_max was still at the kernel default of 65,536. Short-lived Redis/Postgres connections churned through the table during spikes, filled it, and the kernel silently started dropping packets. Nobody noticed because syslog wasn't shipped anywhere useful.

3. DNS resolution overhead. Docker's embedded DNS (127.0.0.11) was resolving service names on every new connection instead of the app caching results. Under load, with connection churn, lookups started queuing behind each other.

None of these show up with 10 test users in staging. All three show up hard with 900 real ones under concurrent load.

What we didn't do

We ruled out "just move to Kubernetes." It wouldn't have fixed anything here; K8s has its own version of the same problems (CNI choice, kube-proxy mode, CoreDNS caching). Swapping orchestrators without fixing the root cause just relocates it.

We also ruled out rewriting the app to reduce internal calls. The call pattern was normal. The network layer needed to handle it efficiently, not the other way around.

The fix, in four layers

Kernel tuning for conntrack:

net.netfilter.nf_conntrack_max = 262144
net.netfilter.nf_conntrack_tcp_timeout_established = 600
net.ipv4.tcp_tw_reuse = 1
net.core.somaxconn = 4096

Applied via /etc/sysctl.d/99-docker-network.conf and sysctl --system. We also started monitoring conntrack table utilization going forward, since a full table fails silently.

Moved internal traffic off the default bridge:

docker network create \
  --driver bridge \
  --opt com.docker.network.bridge.enable_icc=true \
  --opt com.docker.network.driver.mtu=9000 \
  --subnet 172.28.0.0/16 \
  internal-services

App containers, Redis, and the search sidecar moved onto this network, keeping east-west traffic off Docker's default NAT path. The public-facing load balancer stayed on a separate network; no change to external attack surface.

Local DNS caching with dnsmasq as a sidecar:

# dnsmasq.conf
no-resolv
server=127.0.0.11
cache-size=1000
local-ttl=10
neg-ttl=5

A 10-second TTL absorbed connection churn during spikes without causing stale resolution issues when containers got replaced on deploy. We specifically tested that a container restart got picked up within one TTL window.

PgBouncer for connection pooling:

[databases]
marketplace = host=postgres-primary port=5432 dbname=marketplace

[pgbouncer]
pool_mode = transaction
max_client_conn = 2000
default_pool_size = 50

This was sequenced last on purpose. Fewer short-lived connections means fewer conntrack entries and fewer DNS lookups, so it made everything upstream easier once the network layer was already sound.

Takeaway

If your containerized app slows down only under load and CPU/memory graphs look fine, stop scaling containers and go look at conntrack, your bridge driver, and DNS resolution behavior. That's usually where it's actually hiding.

Originally published on binadit.com

Solving the real ROI question behind moving off US hyperscalers

binadit — Wed, 02 Sep 2026 07:23:04 +0000

Why your hyperscaler exit math keeps failing finance review

You run the numbers on leaving AWS. Compute is cheaper elsewhere. Storage is cheaper elsewhere. Then finance asks about egress fees, migration hours, and cutover risk, and the whole business case falls apart. Sound familiar? This isn't a bad migration plan, it's a broken cost model.

Most engineers comparing hyperscaler costs against alternatives only look at list prices for compute and storage. That's maybe 40% of the real picture. Here's the rest of it, and a framework to fix your ROI math.

Why the comparison breaks down

Hyperscalers don't really compete on compute price. They compete on making it painful to leave. That's baked into the architecture, not just the contract.

Egress fees are the real anchor. AWS charges roughly $0.09/GB after the first GB out. For a platform pushing 50TB/month (video, API responses, backups, CDN pulls), that's about $4,500/month just to move data, before touching anything else. This fee isn't about bandwidth cost, bandwidth is cheap. It's a moat.

Managed services lock you in harder than compute does. If you're on RDS, DynamoDB, SQS, and Lambda, you're not renting VMs, you're depending on proprietary APIs. Migrating off means rewriting your data access layer, not just moving instances. This is the line item most spreadsheets underestimate by 3-5x.

Reserved instance pricing hides a bad assumption. Finance often compares a competitor's list price to a hyperscaler's 3-year reserved rate and calls it a win for the hyperscaler. But that rate assumes flat usage for 36 months. Most SaaS workloads have seasonal spikes and growth curves. A 3-year commitment is a liability wearing a discount's clothes.

Support cost is invisible until you actually need it. A named TAM tier starts around $15,000/month, and you're still in a ticket queue for anything real. Compare that to a partner where a senior engineer just picks up.

The fix: five cost categories, not one

1. Baseline 12 months of actual spend, not one month

aws ce get-cost-and-usage \
  --time-period Start=2024-11-01,End=2025-11-01 \
  --granularity MONTHLY \
  --metrics 'UnblendedCost' \
  --group-by Type=DIMENSION,Key=SERVICE

Break it into compute, storage, egress, managed services, support, and RI amortization. Egress alone is usually 8-15% of total spend for content-heavy platforms, and it almost never makes the first draft of a migration business case.

2. Model the actual migration cost

This means:

Engineering hours to replace managed-service dependencies (DynamoDB to Postgres, Lambda to containers, SQS to self-hosted queues)
One-time data transfer cost to move the dataset out
Parallel-run cost during validation (typically 4-8 weeks)
A downtime contingency line, even with a zero-downtime plan

For a mid-sized SaaS app with a 2TB database and 15 microservices, expect 200-450 engineering hours. At 80 euros/hour loaded cost, that's 16,000-36,000 euros, upfront. Put it in the model.

3. Recalculate egress under the new architecture

# AWS us-east-1 to internet: $0.09/GB after first 1GB free
# EU provider with peering to major IXPs: $0.01-0.02/GB typical

# 50TB/month egress:
# AWS: 50,000GB * $0.09 = $4,500/month
# EU provider: 50,000GB * $0.015 = $750/month
# Annual difference: $45,000

Colocate with your CDN's regional PoPs and egress typically drops 40-70%.

4. Swap proprietary APIs for open standards

This is the highest-leverage move in the whole exit. It's what makes the savings durable instead of a one-time discount you slowly erode by re-adopting vendor tooling:

DynamoDB → PostgreSQL or self-managed MongoDB
Lambda → containers on Kubernetes or Nomad
SQS/SNS → self-hosted RabbitMQ or Kafka
CloudWatch → Prometheus + Grafana

5. Cut over with zero downtime

Replicate continuously, run both environments in parallel, lower DNS TTL ahead of time, keep the old environment warm for at least one billing cycle.

# 48-72 hours before cutover
example.com.  300  IN  A  203.0.113.10

# Monitor both origins during cutover
curl -s -o /dev/null -w '%{http_code} %{time_total}s\n' https://old-origin.example.com/health
curl -s -o /dev/null -w '%{http_code} %{time_total}s\n' https://new-origin.example.com/health

For database-backed apps, run logical replication for days beforehand, not a single export/import window.

How to know it actually worked

Track these for at least two full billing cycles:

Total spend vs. your 12-month baseline, normalized for traffic growth
Egress as % of total spend (expect 8-15% → 2-4% with good peering)
p95/p99 latency on your top 10 endpoints, before and after
Error rate and uptime, 30 days before vs. 30 days after
Engineering hours on infra ops per month

One sanity check that matters: if spend drops 30% but p99 latency degrades 25%, you haven't won anything. You've just shifted cost into a metric that'll eventually cost you conversions.

Full framework and migration mechanics in the original piece: Solving the real ROI question behind moving off US hyperscalers

Originally published on binadit.com

How to move ecommerce infrastructure from a single VPS to HA without rewriting the application

binadit — Wed, 26 Aug 2026 07:22:59 +0000

Your VPS will fail at the worst possible time

Here's an uncomfortable truth: that single VPS running your store isn't a matter of if it fails, it's when. A kernel update reboot, a Black Friday memory spike, a disk that fills up overnight. None of these are exotic failure modes, they're Tuesday.

The good news: fixing this is an infrastructure problem, not a rewrite. If you're running PHP, Node, or Python against a relational database, you can go from one box to a highly available setup without touching your application code. This applies whether you're on WooCommerce, Magento, or something custom.

Before you touch anything

Check these boxes first:

Your app can run statelessly across multiple servers (or can be made to, sessions and uploads are the usual culprits)
You have root, not just FTP
You've scheduled a maintenance window for DNS and DB cutover
You have a tested, recent backup
You know your traffic pattern: peak RPS, DB connections, payload size

This walkthrough assumes a LAMP/LEMP-ish stack: Nginx or Apache, PHP-FPM or Node, MySQL or Postgres, Redis. Swap tooling as needed for other stacks, the pattern holds.

Step 1: Get state off local disk

This is the real blocker to scaling horizontally. Local disk state has to go before you add a second server.

Sessions → move to Redis:

; php.ini
session.save_handler = redis
session.save_path = "tcp://10.0.0.5:6379"

Uploaded media → object storage (S3-compatible) or shared NFS. For WooCommerce, WP Offload Media handles this out of the box. For custom apps, swap filesystem writes for an SDK call:

$s3->putObject([
    'Bucket' => 'store-uploads',
    'Key'    => $filename,
    'Body'   => fopen($tmpPath, 'r'),
]);

Cache → Redis or Memcached instead of local disk/OPcache-only.

Step 2: Put a load balancer in front, even with one backend

Spin up a small VM or a managed LB now, before you have a second server. Point DNS at the LB's IP immediately. This decouples DNS from your server count forever, no more DNS changes for future scaling.

upstream app_servers {
    server 10.0.0.10:80 max_fails=3 fail_timeout=30s;
    server 10.0.0.11:80 max_fails=3 fail_timeout=30s backup;
}

server {
    listen 443 ssl;
    server_name shop.example.com;
    location / {
        proxy_pass http://app_servers;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

Step 3: Clone the app server

Once sessions and media are externalized, your app server is basically stateless. Build a second node from the same provisioning script, Ansible, Docker, or a shell script, doesn't matter, just make it reproducible.

# minimal provisioning sanity check
php -v
nginx -v
composer --version
cat /etc/php/8.2/fpm/pool.d/www.conf | grep pm.max_children

Add it to the upstream block, drop the backup flag, confirm both nodes handle real traffic before moving to the database layer.

Step 4: Replicate the database

Highest-risk step. Set up primary-replica replication or move to a managed cluster.

# primary my.cnf
server-id = 1
log_bin = /var/log/mysql/mysql-bin.log
binlog_do_db = shop_production

# replica my.cnf
server-id = 2
relay-log = /var/log/mysql/mysql-relay-bin.log

CHANGE MASTER TO
  MASTER_HOST='10.0.0.5',
  MASTER_USER='replicator',
  MASTER_PASSWORD='***',
  MASTER_LOG_FILE='mysql-bin.000003',
  MASTER_LOG_POS=154;
START SLAVE;

Check lag before cutover:

SHOW SLAVE STATUS\G
-- Seconds_Behind_Master should be 0

Once stable, cut over writes using a virtual IP or ProxySQL, not a manual config edit during an incident.

Step 5: Real health checks, not port pings

upstream app_servers {
    server 10.0.0.10:80 max_fails=2 fail_timeout=10s;
    server 10.0.0.11:80 max_fails=2 fail_timeout=10s;
}

# haproxy / nginx plus
option httpchk GET /health
http-check expect status 200

Your /health endpoint needs to actually check dependencies:

try {
    $pdo = new PDO($dsn, $user, $pass);
    $redis = new Redis();
    $redis->connect('10.0.0.5', 6379);
    http_response_code(200);
    echo 'ok';
} catch (Exception $e) {
    http_response_code(503);
    echo 'unhealthy';
}

Prove it actually works

Don't trust the config, test it:

Kill a node during low traffic, confirm the LB routes around it inside your fail_timeout window and response times stay flat
Watch Seconds_Behind_Master during peak checkout load, not just idle
Load test both nodes: ab -n 5000 -c 50 https://shop.example.com/ and diff the access logs
Simulate a DB failover in staging, measure reconnect time (should be seconds, not minutes)

Baseline to aim for: zero customer-visible errors when one app node dies, database failover under 30 seconds with retry logic in place.

Mistakes that will bite you

Sticky sessions instead of externalized sessions: works fine until a node dies and half your logged-in users get booted
Duplicate cron jobs: order processing and cache warming running on both nodes doubles the work, pin them to one node or use a job queue
Shallow health checks: pinging port 80 says nothing about whether the DB connection is alive
Replication as a backup strategy: it protects against hardware failure, not against a bad DELETE that replicates instantly
Never testing failover: the first failover shouldn't happen during a real outage

Full walkthrough with more context here: How to move ecommerce infrastructure from a single VPS to HA without rewriting the application

Originally published on binadit.com

Measuring what FISA 702 reauthorization actually changes for EU SaaS infrastructure

binadit — Fri, 21 Aug 2026 08:24:39 +0000

Your "EU region" toggle probably isn't doing what you think

Here's an uncomfortable fact: setting your AWS region to eu-central-1 does nothing to shield you from FISA 702. None of it. If your provider is a US company, that data can still be compelled, regardless of which data center it physically sits in. We ran the numbers to find out what it actually takes to fix this, and whether the fix costs you performance.

FISA 702 got reauthorized in April 2024, running through 2026. It'll come up again, and every time it does, compliance teams ask engineering the same question: can we safely keep running on US-owned cloud infra? The answer isn't legal, it's architectural.

The setup

We benchmarked the same Laravel API workload (PostgreSQL 16, Redis 7, object storage) across three configs:

Config A: AWS eu-central-1, standard hyperscaler, US-headquartered
Config B: EU-owned provider (OVHcloud), data residency guarantees
Config C: Dedicated private cloud, Rotterdam data center, no shared hyperscaler control plane

Same hardware baseline everywhere: 8 vCPU / 32GB app nodes, NVMe DB nodes at 4 vCPU / 16GB, PgBouncer in front of Postgres, Nginx 1.25 as reverse proxy.

Load generated with k6, from a Frankfurt runner to kill long-haul network noise as a variable:

// k6 load profile
export const options = {
  stages: [
    { duration: '10m', target: 2000 }, // ramp 50 -> 2000 VUs
    { duration: '20m', target: 2000 }, // sustained peak
  ],
};
// mix: 70% GET, 30% POST/PUT

We measured two separate things: raw performance, and legal exposure surface (who can be compelled to hand over your data, under what instrument, with or without notifying you).

Result 1: performance is basically a wash

Config	p50	p95	p99	Max sustained req/s
A: AWS eu-central-1	42ms	118ms	210ms	3,150
B: OVHcloud (EU)	47ms	134ms	245ms	2,890
C: Private cloud (Rotterdam)	39ms	102ms	178ms	3,020

The private cloud setup wins on tail latency, but only by 15-20%. If you're picking your infra provider based on speed alone, don't bother, the difference is inside normal variance for most apps. This is not a performance decision.

Result 2: the legal exposure gap is real and it's structural

Config	Subject to FISA 702	Compellable without notifying you	Legal instrument
A: AWS eu-central-1	Yes	Yes, gag orders are standard	FISA 702 + CLOUD Act
B: OVHcloud	No direct exposure, but sub-processor risk	Depends on your stack	GDPR only, if fully EU-owned
C: Private cloud	No	No	GDPR + Dutch law only

Here's the part that should worry you: 60% of the "EU cloud" test deployments we checked were still routing through US-owned CDNs, DNS, or email providers. You think you're compliant because your DB lives in an EU region, but your DNS resolver, your CDN, and your transactional email are all silently reintroducing FISA 702 exposure.

The region setting is not the control point. The corporate parent of every vendor in your chain is.

What it actually costs to fix

Component	US-owned default	EU-owned alternative	Effort
DNS	Route 53	deSEC / EU-hosted BIND	Low, 2-4 hrs
CDN	CloudFront	Bunny CDN (EU entity)	Medium, 1-2 days
Email	SES / SendGrid	Mailjet (FR entity)	Medium, 1 day + DNS propagation
Object storage	S3	OVHcloud Object Storage / self-hosted MinIO	High, 3-5 days
Monitoring/APM	Datadog	Self-hosted Grafana + Prometheus	High, 1-2 weeks

Notice the pattern: cheap fixes first, expensive fixes require real migration planning.

The pragmatic order of operations

Swap DNS and CDN first. Cheap, fast, immediately reduces exposure. A few hours to a couple days.
Move transactional email next. Slightly more friction from DNS propagation, still low-risk.
Treat object storage as a proper migration, not a cutover. This one involves real data transfer volume. Plan it with a testing window, not a weekend deploy.
Self-host monitoring last, or accept the risk consciously if it's low priority for your threat model. This is the highest-effort item and often the lowest payoff per hour spent.

Caveats worth knowing

We tested from a single European location. If you've got US or APAC traffic, a fully EU-only stack will add real latency you need to benchmark separately.
Legal classification here is based on published transparency reports and EDPB guidance as of early 2025, not a review of your specific contracts. SCCs, the EU-US Data Privacy Framework, and things like AWS's "Digital Sovereignty Pledge" all interact with this picture in ways a pure infra test can't capture.

Bottom line

FISA 702 reauthorization doesn't change the underlying math, it's been in force since 2008. What's changed is that RFPs from regulated industries now explicitly ask vendors to document their exposure. If you haven't audited your sub-processor chain, that's the actual work here, not swapping your primary cloud region.

Full methodology and extended results: Measuring what FISA 702 reauthorization actually changes for EU SaaS infrastructure

Originally published on binadit.com

How DNS resolution works under the hood: a step by step guide for high availability infrastructure

binadit — Wed, 19 Aug 2026 07:10:37 +0000

DNS resolution: the seven hops nobody thinks about until failover breaks

You push a config change, wait for propagation, and traffic still hits the dead server. Sound familiar? Most DNS incidents come down to not knowing what actually happens between a browser typing a domain and a TCP handshake starting. Let's fix that.

This is a practical walkthrough of the resolution chain, plus the config patterns that turn DNS into an active part of your high availability setup instead of a silent single point of failure.

Assumptions

You control a domain through Cloudflare, Route 53, or similar, with dashboard or API access
You have a Linux box with dig, nslookup, and tcpdump/ngrep available
You know the basics of IP, TCP, and UDP
You're comfortable running commands against a real or disposable test domain

We'll use example.com everywhere below. Swap it for your own.

The resolution chain, hop by hop

1. Browser cache

Before anything hits the network, Chrome checks its own DNS cache:

chrome://net-internals/#dns

If there's a live entry, resolution stops right there. This is why a DNS change can look "stuck" in your own browser even after the TTL has expired everywhere else.

2. OS stub resolver

Next stop is the OS-level resolver, usually systemd-resolved or nscd on Linux:

resolvectl status
resolvectl statistics

It checks /etc/hosts, then its own cache, before forwarding anything upstream.

3. Recursive resolver

No local hit means the query goes to a recursive resolver: your ISP's, or a public one like 1.1.1.1 or 8.8.8.8. This is where +trace becomes your best debugging friend:

dig @1.1.1.1 example.com +trace

4. Root servers

The recursive resolver asks a root server who's authoritative for .com. Roots don't know about example.com, only about the TLD:

com. 172800 IN NS a.gtld-servers.net.
com. 172800 IN NS b.gtld-servers.net.

5. TLD servers

The TLD server hands back your domain's authoritative nameservers, the ones set at your registrar:

example.com. 172800 IN NS ns1.yourdnsprovider.com.
example.com. 172800 IN NS ns2.yourdnsprovider.com.

6. Authoritative nameserver

Finally, the actual record comes back:

example.com. 300 IN A 203.0.113.42

This is the step that matters for HA: which IP gets returned, how fast it can change, and what happens when that IP goes dark.

Configuring DNS for actual failover

Understanding the chain is step one. Making it work for you is step two.

Drop your TTLs on failover-critical records. A 3600s TTL means an hour-long tail of stale traffic during failover. Use 60-300s instead:

example.com. 60 IN A 203.0.113.42

You'll pay for it in query volume against your authoritative nameservers. Worth it.

Attach health checks. Route 53 and Cloudflare can pull an unhealthy origin out of the response set automatically:

aws route53 change-resource-record-sets \
  --hosted-zone-id Z1PA6795UKMFR9 \
  --change-batch '{
    "Changes": [{
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "example.com",
        "Type": "A",
        "SetIdentifier": "primary",
        "Failover": "PRIMARY",
        "TTL": 60,
        "ResourceRecords": [{"Value": "203.0.113.42"}],
        "HealthCheckId": "abcd1234-healthcheck-id"
      }
    }]
  }'

This is the same mechanism behind a clean zero-downtime migration: DNS shifts traffic instead of forcing a hard cutover.

Return multiple A records. Clients can fall back to a second IP:

example.com. 300 IN A 203.0.113.42
example.com. 300 IN A 203.0.113.43

Fine for stateless services behind a load balancer, but not a real substitute for health-checked failover; clients cache order and don't always retry smartly.

Verifying it actually works

Don't trust the dashboard. Check every layer.

dig example.com +noall +answer
dig example.com @8.8.8.8 +noall +answer
dig example.com @1.1.1.1 +noall +answer

Lower-than-configured TTLs from a resolver mean the change is propagating. Original TTLs mean you're still hitting a cached answer.

Check resolution latency directly:

dig example.com | grep "Query time"

Consistently above 100-150ms? Look at resolver placement or a provider with more edge presence. This happens before the TCP handshake even starts, so it's pure overhead on TTFB.

Simulate failover instead of waiting for an incident to test it for you:

iptables -A INPUT -s <health-check-ip> -j DROP
watch -n 5 'dig example.com +short'

Confirm the response flips to your secondary within the configured TTL, then remove the rule.

Add DNS resolution time to your monitoring as its own metric, separate from full page load. A spike there with stable backend times points straight at a resolver or nameserver problem, not your app.

Pitfalls that keep coming back

Default TTLs. 3600 or 86400 seconds quietly kills any failover plan built on top.
Single nameserver provider. One outage there takes your domain down regardless of server health.
DNSSEC misconfiguration. Broken signing chains fail silently for validating resolvers while looking fine in tools that skip validation.
Testing only from your machine. Your local cache hides what real users see. Always check against multiple public resolvers.
Deep CNAME chains. Each extra hop adds latency and another point of failure.

Understand the seven hops, and DNS stops being a mystery box that occasionally ruins your day.

Full original writeup: How DNS resolution works under the hood

Originally published on binadit.com

TTFB vs full page load: choosing the right metric for your ecommerce infrastructure

binadit — Fri, 14 Aug 2026 07:41:20 +0000

Stop chasing TTFB: it's not the metric that's tanking your conversions

Your TTFB dashboard is green. Your conversion rate is red. If that combination sounds familiar, you're not alone, and you're not chasing the wrong fix by accident, you're chasing it because every tool you use puts TTFB front and center.

Here's the problem: TTFB and full page load metrics answer completely different questions, and treating them as interchangeable wastes engineering time on ecommerce sites where every millisecond on checkout pages has a dollar value attached.

What TTFB actually tells you

TTFB is the clock between the request leaving the browser and the first byte coming back. That window includes:

DNS resolution (if uncached)
TCP/TLS handshake
Server-side work: routing, DB queries, template rendering, cache lookups
Network transit for that first byte

On a warm HTTP/2 or HTTP/3 connection, server processing time dominates this number. Which makes TTFB genuinely useful, but only for one job: telling you when your backend is under load.

TTFB baseline: 180ms
TTFB during flash sale: 900ms

That jump isn't noise. It's your app server, DB, or cache layer running out of headroom before a single byte of HTML goes out. On WooCommerce stacks especially, a TTFB spike during a traffic surge is usually the first hard evidence that your PHP-FPM pool or database is maxed out, well before cart abandonment shows up in analytics.

Where TTFB earns its keep

Catching server-side regressions. A slow query, an N+1 bug, a cold cache: TTFB flags these before users notice anything visually.
Load testing and capacity planning. Watch TTFB climb under simulated load and you know exactly where your app tier breaks.
Splitting backend vs frontend blame. Fast TTFB + slow-feeling page = stop looking at the server, start looking at JS and render-blocking assets.

Where it lies to you

A 100ms TTFB tells you nothing about what happens after that byte lands. The page can still take 6 seconds to become usable because of:

Render-blocking CSS/JS
Images without dimensions causing layout shift
A tag manager script hogging the main thread
Late-loading fonts (hello, flash of invisible text)

And here's the trap most teams miss: if a response is served from a CDN edge cache, TTFB measures the edge, not your origin. Your dashboard can look perfectly healthy while your actual application is quietly degrading, right up until a cache miss exposes it in production.

What full page load metrics tell you instead

LCP, Speed Index, fully-loaded time: these track what the user actually sees and feels. That's why Google leans on LCP and CLS for Core Web Vitals; TTFB alone was a bad predictor of both perceived speed and search ranking.

Strengths:

Correlates with real user experience, not just server response
Surfaces conversion killers directly (a bloated hero image, a blocking checkout script)
Feeds into Core Web Vitals and SEO scoring

Weaknesses:

Noisy. A regressed LCP could be caused by the backend, a new marketing pixel, an unoptimized CMS upload, or a hydration delay in your frontend framework
Lagging indicator. You need waterfall analysis and resource timing to find the actual cause after LCP tanks in prod
Out of infra's hands. You can run flawless high-availability infrastructure and still ship a slow LCP because a third-party payment widget is garbage

The decision framework

Stop asking "is TTFB good." Ask this instead:

if (TTFB_high && LCP_high) {
  fixBackendFirst(); // nothing downstream matters until this is solved
}

if (TTFB_low && LCP_high) {
  profileBrowserWaterfall(); // frontend/third-party problem
}

if (TTFB_high && LCP_low) {
  // rare, but check if LCP measurement is misleading you
}

if (servingMostlyFromCDN) {
  trackOriginResponseTimeSeparately(); // edge TTFB will lie to you
}

Practical ownership split:

Situation	Metric to watch	Who owns it
Debugging capacity/backend regression	TTFB	Backend/infra
Debugging conversion drop or CWV score drop	LCP, Speed Index	Frontend + infra
Prepping for a sale or launch	Both, together	Whole team

For traffic events specifically: TTFB tells you when the servers start straining, LCP tells you whether shoppers are actually feeling it. Alert on both, but don't confuse one for the other, and don't let a green TTFB dashboard talk you out of investigating a real conversion problem.

Full writeup with the complete comparison table and more edge cases: TTFB vs full page load: choosing the right metric for your ecommerce infrastructure

Originally published on binadit.com

Measuring how EuroStack changes the procurement conversation for infrastructure management services

binadit — Wed, 12 Aug 2026 07:12:58 +0000

We ran the numbers on 41 procurement deals to see if EuroStack is real or just RFP theater

If you've touched infrastructure procurement in the last year, you've probably seen "EU sovereignty" or "EuroStack" show up somewhere in the RFP. The question every engineering lead asks: is this actual signal that changes vendor selection, or is it a compliance checkbox that nobody enforces?

We had visibility into 41 procurement processes between January 2023 and September 2025, so instead of guessing, we split the data before and after EuroStack-style criteria showed up explicitly in tender docs and measured what actually shifted.

The setup

We defined "EuroStack criteria" narrowly. It had to include:

EU legal jurisdiction over the operating entity
Data processing location inside the EU/EEA
Disclosure of subcontractor data flows, including support and monitoring tooling

"GDPR compliant" with no operational detail didn't count. That's boilerplate, not a filter.

Sample: 41 contracts, €40k to €2.1M ACV, across SaaS (14), public sector (11), fintech/payments (9), and e-commerce/logistics (7). 19 processes ran before explicit sovereignty language appeared, 22 ran after.

We were not benchmarking server performance here. This is purely procurement mechanics: shortlist size, timelines, contract clauses, final price vs. quote.

What actually moved

Metric                              Before    After
Avg. vendors shortlisted            5.2       3.6
% non-EU hyperscaler-reseller-only  47%       12%
Avg. RFP-to-signature time          11.4wk    7.9wk
% with subcontractor disclosure     21%       86%
% with real data residency SLA      32%       79%
Avg. final value vs. initial quote  +14%      +4%

The timeline tail matters more than the average:

Percentile   Before (wk)   After (wk)
p50          9.5           6.0
p95          24.0          15.5
p99          31.0          19.0

Post-signature scope disputes also dropped hard: 42% of "before" contracts had a change request or renegotiation in the first 6 months, usually over data location or support access. In the "after" group, that fell to 18%.

Why this happens

The shortlist shrinking from 5.2 to 3.6 vendors isn't a loss, it's a filter working correctly. Reseller setups that white-label a US hyperscaler and can't answer "where does support tooling send logs" get cut early instead of surviving three rounds of vague reassurance.

That filtering effect is also why timelines compressed. When the RFP already specifies data residency and subcontractor disclosure, vendors don't need three rounds of clarifying questions. We saw this in our own bid pipeline: RFP-to-proposal time dropped from 9 days to 4 when the tender pre-answered the questions we'd otherwise have to ask ourselves.

The contract value delta is the part people underestimate. Vague sovereignty language at RFP stage doesn't disappear, it just resurfaces post-signature as a change order once someone discovers a gap. Explicit criteria up front push that negotiation into the cheap phase instead of the expensive one.

One more pattern worth flagging: buyers who specified sovereignty criteria were also far more likely to require named engineer contacts instead of ticket-based support (61% vs. 23%). Teams that care about knowing where their data lives also tend to care about knowing who picks up the phone.

Caveats before you cite this

Selection bias: we bid on EU-sovereign contracts, so our "before" sample skews toward processes where we were still invited. Fully non-sovereignty procurement is likely faster and larger in scope than what we captured.
Small sample: 41 deals over 2.5 years shows a trend, not statistical proof. One outlier public-sector tender inflated the before-p99 from 24 to 31 weeks.
No official standard exists yet: "EuroStack criteria" is our own classification. There's no certification scheme to point to as of late 2025.
No cost-per-compute comparison: this says nothing about whether EU infra is cheaper or pricier per server-hour than hyperscaler equivalents. Different question, different analysis.

Takeaway for anyone writing or responding to RFPs

If you're a buyer: specific sovereignty requirements (jurisdiction, data location, subcontractor disclosure) act as a fast, cheap filter that eliminates non-answers early and shrinks your timeline. If you're a vendor: if you can't answer where support tooling sends logs, you're going to keep losing time to vendors who can.

Full methodology and data breakdown in the original article.

Originally published on binadit.com

10 practices that keep WooCommerce fast at scale

binadit — Fri, 07 Aug 2026 07:11:37 +0000

Your WooCommerce store is fine until it isn't

Everything runs smoothly until a flash sale hits, or your ERP sync job starts locking tables during checkout hours. If you're running WooCommerce past a few hundred SKUs or a few thousand orders a month, you've probably already hit one of these walls. Good news: none of this requires a replatform. It's config, query discipline, and infrastructure decisions you can roll out incrementally.

Here are 10 practices that actually move the needle, ranked roughly by how fast you'll feel the impact.

1. Get sessions and cart data out of MySQL

By default, WooCommerce dumps cart and session data into wp_options and wp_woocommerce_sessions. Under concurrent load, that's write contention on tables that are already getting hammered by product and pricing reads.

Move it to Redis. Sub-millisecond lookups instead of a MySQL round trip per cart update.

// wp-config.php or a custom session handler plugin
define('WP_REDIS_HOST', '127.0.0.1');
define('WP_REDIS_PORT', 6379);
define('WP_REDIS_TIMEOUT', 1);
define('WP_REDIS_DATABASE', 2);

2. Don't conflate object cache with page cache

Object caching (Redis/Memcached) caches individual query results; product lookups, term queries, user meta. Page caching serves fully rendered HTML. They need different invalidation logic entirely.

Cart and category pages aren't fully cacheable at the page level since they're price-sensitive and user-specific, but they still benefit massively from object caching underneath. Keep them separated and you get both speed and correctness.

3. Never let full-page cache touch cart, checkout, or account pages

A cache plugin that doesn't understand WooCommerce's dynamic fragments will happily serve one customer's cart to another. That's not a performance win, that's a bug report waiting to happen.

# Nginx: never cache dynamic WooCommerce endpoints
location ~* ^/(cart|checkout|my-account) {
    proxy_no_cache 1;
    proxy_cache_bypass 1;
}

Everything else (product pages, categories, homepage) can be cached aggressively as long as cart fragments load via AJAX post-render.

4. Stop refreshing cart fragments on pages where nothing changed

The default mini-cart AJAX call (wc-ajax=get_refreshed_fragments) is what makes point 3 possible, but it's commonly left running on every page load, blog posts included. Throttle it or disable it on pages where cart state can't change. Noticeable PHP-FPM load reduction on content-heavy stores.

5. Audit your indexes, not just WooCommerce's defaults

Default indexes on wp_postmeta and wp_wc_order_stats are fine for standard queries. Custom filters, ERP syncs, and reporting plugins introduce meta queries that often aren't indexed at all.

-- Check for missing composite indexes
EXPLAIN SELECT post_id FROM wp_postmeta
WHERE meta_key = '_stock_status' AND meta_value = 'instock';

-- If it's a full table scan:
ALTER TABLE wp_postmeta ADD INDEX idx_meta_key_value (meta_key, meta_value(20));

Do this quarterly. Plugin updates change query patterns, and indexes don't fix themselves.

6. Queue everything that isn't the checkout itself

Order confirmation emails, ERP syncs, webhooks: none of that should run synchronously inside the checkout request. If a downstream system is slow, your customer is the one waiting.

// Defer post-order work instead of running it inline
add_action('woocommerce_order_status_completed', function($order_id) {
    as_schedule_single_action(time(), 'sync_order_to_erp', ['order_id' => $order_id]);
});

Action Scheduler ships with WooCommerce already; use it, or push to a Redis-backed queue. Keep the request path short.

7. CDN your images, not just your HTML

Product catalogs with hundreds of high-res images are usually the biggest bandwidth drain on the whole stack.

location ~* .(jpg|jpeg|png|webp|avif)$ {
    expires 30d;
    add_header Cache-Control "public, immutable";
}

Serve WebP/AVIF where supported. This has a direct effect on mobile checkout conversion, arguably more than most backend tuning.

8. Autoscale PHP-FPM based on concurrency, not vibes

A fixed pm.max_children tuned for average traffic will choke during a flash sale and sit idle the rest of the time.

pm = dynamic
pm.max_children = 50
pm.start_servers = 10
pm.min_spare_servers = 5
pm.max_spare_servers = 20
pm.max_requests = 500

pm.max_requests is underrated: it forces worker recycling and prevents slow memory leaks in plugin code from degrading performance over a day.

9. Send reporting and sync jobs to a read replica

Analytics dashboards and ERP syncs run expensive aggregate queries. Running them against your primary competes directly with checkout for connections and locks. Point them at a replica instead and isolate the load completely.

10. Load test the checkout flow, not the homepage

Homepage numbers look great because it's the most cacheable page you have. Checkout is the opposite: least cacheable, most DB-intensive, and the one that actually determines revenue. Simulate the real flow: add to cart, apply coupon, enter payment, submit.

Rollout order that won't blow up your team

Weeks 1-2: object cache + session offload (1-2). Zero code risk, immediate latency win.
Weeks 3-4: cache exclusions + CDN config (3, 7). Low risk, big TTFB improvement for anonymous traffic.
Month 2: query/index audit + queue migration (5, 6). Touches business logic, test accordingly.
Month 2-3: PHP-FPM tuning + read replica (8, 9). Do these once you have real traffic data to tune against.

Read the full original writeup here: woocommerce-fast-at-scale-managed-infrastructure-for-saas

Originally published on binadit.com

Setting up sovereign cloud reference architectures: three patterns that work

binadit — Wed, 05 Aug 2026 07:11:52 +0000

Three sovereign cloud patterns that actually hold up under audit

Here's the problem: picking "EU" from a cloud console dropdown is not sovereignty. It's a checkbox. Real data residency means your compute, your backups, your third-party services, and your failover path all stay inside a legal boundary, and you can prove it with a packet capture, not just a policy doc.

We've deployed three reference patterns repeatedly for SaaS platforms, agencies, and e-commerce clients dealing with GDPR, DORA, or public-sector requirements. Here's how each one works, with the configs we actually run.

Before you start

You'll need:

A stateless app tier (no session affinity dependencies)
Terraform/OpenTofu plus Ansible or equivalent
A clear map of which data is regulated and which isn't
Access to at least one EU-based provider, plus a public cloud account if you're going hybrid
Comfort with HAProxy/Nginx and PostgreSQL/MySQL replication

If you haven't decided between open-source infra and managed public cloud yet, sort that out first. Everything below assumes it's settled.

Pattern 1: Single-region EU

The baseline. Good for most SaaS teams that need provable EU residency without multi-region complexity.

Everything (compute, storage, backups, logging, error tracking) stays in one EU region. Spread compute across at least two AZs, never a single rack.

Load balancer tier:

frontend web_front
    bind *:443 ssl crt /etc/haproxy/certs/app.pem
    mode http
    default_backend app_servers

backend app_servers
    balance roundrobin
    option httpchk GET /health
    server app1 10.0.1.11:8080 check
    server app2 10.0.1.12:8080 check
    server app3 10.0.1.13:8080 check

PostgreSQL synchronous replication for zero data loss on failover:

# postgresql.conf on primary
synchronous_standby_names = 'standby1'
wal_level = replica
max_wal_senders = 5

# recovery config on standby
primary_conninfo = 'host=10.0.1.21 port=5432 user=replicator'

The part everyone forgets: route email, error tracking, and analytics through EU-based providers too. A perfectly compliant database means nothing if your error tracker ships stack traces to a US endpoint.

Back up daily, encrypt everything, keep it in-region. 30 days daily / 12 months monthly is a reasonable default for compliance workloads.

Pattern 2: Active-passive multi-region

For disaster recovery across two EU regions (say, Amsterdam and Frankfurt) without paying for a full duplicate stack.

Size the passive region at 30-50% capacity, enough to run degraded, not enough to bleed money idling.

Async replication to the secondary:

# On secondary region standby
primary_conninfo = 'host= port=5432 user=replicator sslmode=require'
restore_command = 'cp /archive/%f %p'

DNS-based failover:

failover_policy:
  primary: eu-west (Amsterdam)
  secondary: eu-central (Frankfurt)
  health_check_interval: 10s
  failover_threshold: 3 consecutive failures

Automate the promotion, don't SSH in under pressure:

pg_ctl promote -D /var/lib/postgresql/data
# update connection strings via config management, never manual edits

And actually test failover quarterly. This is the step everyone skips, and it's the only one that proves the whole thing works.

Pattern 3: Hybrid private-public

Best for regulated workloads (payments, health data, government contracts) that also need elastic burst capacity for traffic spikes.

Keep the system of record (database, PII, payment processing) on private EU infrastructure. Push only stateless, non-sensitive work (static assets, image processing, caching) to public cloud.

Enforce the boundary at the network layer, not just in app code:

# Restrict outbound DB traffic to private subnet only
iptables -A OUTPUT -p tcp --dport 5432 -d 10.0.0.0/8 -j ACCEPT
iptables -A OUTPUT -p tcp --dport 5432 -j DROP

Route burst traffic through a CDN that actually keeps EU traffic on EU edge nodes (not all of them do, check before you commit). Keep a data flow diagram ready; procurement and auditors will ask for it before signing anything.

Verify it, don't just deploy it

Residency: traceroute and packet capture a real transaction, confirm no hop leaves the EU
Failover time: target under 60 seconds with a 10-second health check interval; over 5 minutes means your thresholds are too conservative
Replication lag: SELECT now() - pg_last_xact_replay_timestamp(); on the standby, sustained lag over 5s means trouble on unplanned failover
Boundary enforcement: try connecting from the public segment to the private DB port, confirm the firewall (not app logic) rejects it
Real downtime: run a drill and measure customer-facing downtime, not just the infra switch; DNS caching often adds 2-5 minutes you didn't account for

Pitfalls worth repeating

Selecting "EU" in a console is not proof of anything, verify subprocessors and backup locations contractually and technically
Third-party services are the most commonly skipped audit item
Under-sizing the passive region means it won't actually hold real traffic when it matters

Full details and more context in the original article.

Originally published on binadit.com

How zero-downtime database migrations work

binadit — Fri, 31 Jul 2026 07:28:52 +0000

Zero-downtime database migrations: a practical breakdown

You ship a schema change on a Friday afternoon. Thirty seconds later, error rates spike across every pod running the old deploy. Nothing was technically wrong with your SQL; the problem was timing, not syntax. This is the scenario zero-downtime migration practices exist to prevent.

The real problem isn't the migration

Zero-downtime doesn't mean the migration completes instantly. It means your app keeps serving correct reads and writes while the schema shifts from old to new. Most migration failures trace back to one of three things:

Lock contention on the table you're altering
Application code assuming a schema state that no longer exists
A cutover step that briefly knocks the primary offline

The fix starts with a mental shift: treat migrations as a sequence of small, reversible steps, not one big-bang deployment.

What's happening under the hood

Your app servers carry a schema assumption baked into ORM models and prepared statements. The database holds the actual schema. During a migration, you're moving both in sync, and there has to be no moment where they disagree in a way that breaks a query.

Schema changes generally fall into three buckets:

Additive (new column, table, index): usually safe, but can still lock depending on the engine and default value behavior
Destructive (dropping a column, renaming a table): dangerous if any live code still references the old structure
Transformative (changing a column type, splitting a table): requires both old and new structures to coexist temporarily

The pattern: expand-contract

Also called parallel change. Four phases:

Expand: add the new element alongside the old one. Nothing removed yet.
Migrate: backfill data into the new structure, dual-write from the app, or read-with-fallback.
Verify: confirm consistency under real production traffic, not staging.
Contract: once every app instance is confirmed on the new structure, drop the old one.

This works because at every step, both the old and new app deploy can run against the schema without erroring. That matters because deploys are rolling, not instant; you might have two code versions hitting the same DB for 30-90 seconds. If your migration doesn't tolerate that overlap, you'll see error spikes that look random but are completely deterministic.

Locking details worth knowing

In PostgreSQL (11+), adding a column with a constant default is metadata-only; milliseconds regardless of table size. Adding a column with a non-constant default, or a NOT NULL constraint without a default, can force a full table rewrite under an ACCESS EXCLUSIVE lock, blocking everything.

MySQL's InnoDB has a similar split: some ALTER TABLE operations use INSTANT or INPLACE algorithms with no locking; others fall back to COPY, rebuilding the whole table.

Walking through a real example

Say you're renaming status to order_status on a 40 million row orders table.

What breaks:

ALTER TABLE orders RENAME COLUMN status TO order_status;

Fast in Postgres, metadata-only. But any app server still on the previous deploy referencing status starts throwing "column does not exist" the instant this commits. Across 12 pods mid-rollout, that's a guaranteed multi-minute error spike.

Expand-contract version:

-- Step 1: expand
ALTER TABLE orders ADD COLUMN order_status varchar(20);

-- Step 2: backfill in batches, avoid long-running transactions
UPDATE orders SET order_status = status
WHERE id BETWEEN 1 AND 50000 AND order_status IS NULL;
-- repeat in batches of 50k

-- Step 3: dual-write in app code so new rows populate both columns

-- Step 4: deploy code that reads order_status, falls back to status,
-- for one full release cycle

-- Step 5: after zero fallback reads in logs for 7 days
ALTER TABLE orders DROP COLUMN status;

Batch size matters at this scale. One giant UPDATE across 40M rows generates enough WAL/binlog volume to cause replication lag, which cascades into stale reads on any replica-dependent service. Batches of 10k-50k with short pauses keep lag under a second and avoid blocking autovacuum.

For bigger transformative changes (splitting a users table into users and user_profiles at 200M rows), expect this rough timeline:

Phase	Duration	Risk
Expand	Minutes	Low
Backfill	4-10 hours	Low, if throttled
Dual-write period	1-2 weeks	Medium, needs monitoring
Read cutover + verify	3-5 days	Medium
Contract	Minutes	Low, if verification passed

More elapsed time, dramatically smaller blast radius. That's the trade.

The trade-offs, honestly

Dual-write complexity: two write paths double your bug surface. A missed dual-write is the most common cause of silent data drift.
Longer total timeline: a 10-minute blocking migration can become a multi-week incremental one. For low-traffic internal tools, this often isn't worth it.
Replica lag: even throttled backfills add write volume that can measurably increase lag.
Rollback complexity: a single ALTER TABLE rolls back trivially. A multi-phase migration needs a rollback plan per phase.

Tooling notes

Online schema change tools like gh-ost or pt-online-schema-change for MySQL automate much of this by building a shadow table, replaying changes via triggers/binlog, and swapping atomically at the end. They save manual effort but add operational overhead: monitoring the copy, watching for lag, handling mid-copy failures on huge tables.

For PostgreSQL, CREATE INDEX CONCURRENTLY and instant column additions have reduced the need for third-party tools in many cases. Large rewrites (changing a column type) still often need manual expand-contract or extension support.

Read the full breakdown here: Zero-downtime database migrations

Originally published on binadit.com

Choosing between the open-source sovereign stack and managed public cloud

binadit — Wed, 29 Jul 2026 07:20:33 +0000

Sovereign stack or managed cloud: the real math behind the decision

Your cloud bill just crossed six figures a month and someone on the platform team asked, "could we just... run this ourselves?" If you've had that conversation, you already know the answer isn't obvious. Let's break down when self-hosting Proxmox, Ceph, OpenStack, and Kubernetes actually beats renting from a hyperscaler, and when it doesn't.

Who actually hits this decision

Three groups keep running into this:

SaaS teams where cloud spend stopped scaling sanely
Regulated companies (fintech, healthcare, gov contractors) that need provable EU data residency
Platform teams burned by a surprise pricing change or an outage nobody on their side could fix

This isn't really "open source vs cloud." It's "who operates your infrastructure": you, or a partner/hyperscaler doing it for you.

The sovereign stack, layer by layer

Kubernetes        <- container scheduling
   |
OpenStack         <- API layer: compute, Neutron networking, Keystone identity
   |
Ceph              <- distributed block/object/file storage
   |
Proxmox VE (KVM/LXC) <- virtualization layer

Proxmox VE: your VMware replacement. KVM + LXC, web UI, clustering, built-in backup.
Ceph: distributed storage with no single point of failure. Lose a node, keep serving traffic, if your CRUSH map and replication factors are sane.
OpenStack: turns a rack of Proxmox/KVM hosts into something you provision like EC2.
Kubernetes: usually sits on top, scheduling containers on the VMs OpenStack provisions.

Real result: a fintech client cut cloud spend 65% moving core services to Proxmox + Ceph, keeping only CDN/edge on public cloud. Gov and defense-adjacent contractors like this stack because OpenStack's API compatibility means AWS-style automation, but on hardware sitting in an EU data center with zero CLOUD Act exposure.

What you're actually signing up for

Pros:

No per-core licensing tax (Proxmox support subscription, Ceph/OpenStack are Apache-licensed)
Full control over physical location, network path, access
Cost flattens hard past ~15-20 sustained VMs
No rate limits, no vendor deprecation surprises

Cons:

Ceph will punish you if you don't understand placement groups, replication, failure domains. A bad CRUSH map turns one dead disk into cluster-wide latency.
OpenStack ships every 6 months. Skip releases and upgrades get harder, not easier.
Engineers who can debug a flapping Ceph OSD or trace a Neutron issue through OVS are rare and expensive.
Getting to production-ready is a multi-month project, not a weekend.

The managed cloud alternative

This means AWS/Azure/GCP, EU alternatives like Scaleway/OVHcloud, or a managed infrastructure partner running the stack (sovereign or otherwise) on your behalf.

Pros:

Compute/storage/managed DBs available in minutes
Managed Kubernetes, managed DBs, managed LBs remove whole failure categories from your plate
Autoscaling handles traffic spikes without manual capacity planning
Multi-continent presence without building your own PoPs

Cons:

Egress fees, IOPS charges, managed service premiums scale linearly or worse. We covered the real numbers in cheap hosting vs managed cloud infrastructure.
"EU region" doesn't mean sovereignty. A US-HQ'd provider's EU region still carries CLOUD Act exposure. More on that in our EU region toggle piece.
Vendor controls the roadmap: pricing, deprecations, API changes happen on their schedule.
You're boxed into whatever abstractions the provider gives you.

Quick decision framework

if dedicated_infra_engineers < 3:
    stay_on_managed_cloud() or hire_a_partner_to_run_sovereign_stack()

elif vm_count < 15:
    managed_cloud_or_simple_vps()  # capex won't pay back fast enough

else:
    run_the_numbers_on_sovereign_stack()  # now it's worth modeling

The honest takeaway: fewer than 3 dedicated infra engineers or under ~15 VMs means self-hosting Ceph/OpenStack is a risk you're probably underpricing. Above that threshold, the math starts favoring ownership, especially if data sovereignty is a hard requirement, not a nice-to-have.

Full breakdown with the complete comparison table and framework: read the original article.

Originally published on binadit.com