DEV Community: hello world_leo

19 OOM kills in 9 days: diagnosing a shared-hosting WordPress before the rebuild

hello world_leo — Sun, 06 Sep 2026 06:28:09 +0000

Nineteen OOM kills in nine days. Ten WordPress apps on a 32GB shared box. One of them a client site that took a CPU spike on 14 July during a paid campaign burst, and pushed the whole tenant into the wall.

This post is the diagnosis before the rebuild. What we actually found when we stopped guessing. Two layers of Cloudflare, one page cache plugin, one preloader being silently challenged, and a language subpath that was cold every time it mattered.

I'm writing it partly for anyone who runs multi-tenant WordPress on Cloudways or similar, and partly as a reminder for future-me. There's a checklist at the end. Steal it.

The client is anonymized throughout. Every number is real.

The stack

Traffic hits two Cloudflare layers before it reaches origin. Both are Cloudflare, but they're different zones on different accounts, and they own different things.

[ visitor ]
    ↓
[ Upstream Cloudflare zone (managed by a third party) ]  ← DNS, SSL, HTML edge cache
    ↓
[ Cloudflare Enterprise add-on sold by Cloudways ]  ← WAF, bot, rate limit, AI crawler block
    ↓
[ Cloudways origin: nginx + PHP-FPM ]
    ↓
[ WordPress + WPML + Elementor + FlyingPress ]

Two Cloudflares isn't a mistake. The domain has been on Cloudflare via an upstream party since before the site moved to Cloudways. When Cloudways later offered a Cloudflare Enterprise add-on for its security stack, we kept both. We manage the Cloudways side. We don't own the upstream zone, which shapes what we can and can't do without a request going out. The trap is that both layers can cache HTML, and both can serve security challenges. If nobody writes down which layer does what, they fight.

Our ownership split ended up like this:

Layer	Owns
Upstream Cloudflare (third party)	DNS, SSL, HTML edge cache, purge lifecycle
Cloudways CF Enterprise add-on	WAF, bot management, rate limiting, AI crawler blocking, ScrapeShield, Browser Integrity Check
FlyingPress	Origin page cache, Cloudflare integration pointed at the upstream zone, purge rules
Cloudways "Edge Page Caching" (feature)	OFF. Turning it on would create a third HTML cache.

That last row matters more than it looks. Cloudways sells "Edge Page Caching" as a nice-sounding thing you can toggle in their panel. Enable it and you have three parties trying to cache HTML for the same URL: the upstream CF zone, the Cloudways CF zone, and Cloudways' own edge. Purges from the plugin will hit one of them and leave the other two stale. You'll debug ghosts for a week.

Keep it off. FlyingPress owns the HTML cache. Full stop.

What broke on 14 July

The client ran a paid campaign. Traffic pushed harder than usual, mostly at one language subpath because the campaign was regionally targeted. CPU on the shared Cloudways box climbed and stayed climbed. PHP-FPM saturated. Response times went from 400ms to multi-second. Origin bandwidth was normal, which ruled out a DDoS or a bot flood.

I spent the first hour on the wrong hypotheses.

Not the problem:

Bot traffic. The Cloudways CF WAF logs were clean. Nothing unusual.
Disk I/O. iostat showed the array yawning.
MySQL. Slow query log was quiet.
A plugin gone rogue. wp plugin list matched the last known-good state.

The real problem was compound, and every part of it made the next part worse:

The targeted subpath's FlyingPress cache was cold. Not empty, but not warm at the pages the campaign was sending traffic to. Every ad click hit a URL that wasn't in cache.
The upstream CF zone was not caching HTML. No Cache Rule, no Page Rule. So the edge added no protection. Every request went to origin.
The FlyingPress preloader was silently failing. It crawls the sitemap and warms the origin cache. The Cloudways CF layer was serving it a Browser Integrity Check challenge. The preloader isn't a browser, so it never solved the challenge, so the pages never warmed.
Third-party oEmbed calls per page. Several of the affected pages embed content from a large external social network. WordPress calls that oEmbed endpoint on render, synchronously, and blocks PHP until it answers. Under load, the provider was slow. PHP workers held.
WPML + Elementor render cost. A cold page rendering through WPML routing and Elementor's builder is 100-200ms of PHP on a good day. Add oEmbed round-trips. Multiply by every worker being stuck.

Cold cache × silent preloader × blocking third-party call × heavy render stack × shared tenancy. Any one of those alone is a shrug. All five together is a CPU wall.

Why two Cloudflare layers, again

Because they own different concerns, and you don't want the security vendor also owning the cache purge lifecycle.

The upstream CF zone handles DNS. It came with an HTML cache nobody was using until the incident. FlyingPress has a native integration that manages that cache: it holds the Zone ID and a scoped API token, and it purges the right URLs when content updates. Since the zone isn't ours, both values had to be requested from the party that owns it.

The Cloudways Enterprise add-on gives us WAF rules, bot heuristics, rate limiting on abusive paths, and AI crawler blocking. That last one is worth its own post. The volume of scraper traffic from LLM crawlers has been non-trivial. We don't ask that layer to cache HTML. We ask it to reject the things we don't want to serve.

The mistake would be enabling HTML caching on both. Or worse, enabling Cloudways' third "Edge Page Cache" toggle and having three caches with three independent lifetimes.

The caching path that works

Once we understood the seams, the fix was mechanical. FlyingPress has a Cloudflare integration built in. Give it the right zone and the right token and it does the work.

Token scope. Don't over-scope this. If the zone belongs to a third party (as in our case), you're requesting the token from them. Ask for exactly this and nothing broader:

Zone → Cache Purge → Purge
Zone → Page Rules → Edit
Zone → Zone Settings → Edit
Zone → Cache Rules → Edit

That's it. No account-level permissions. No workers. No DNS edit. If someone in support tries to hand you a global API key, hand it back and ask for a scoped token.

Setup order.

Paste Zone ID and API token into FlyingPress → CDN → Cloudflare.
Click Test Connection. Get green.
Enable Cloudflare Page Caching.
Save.
Purge all cache once. This is the only time you do a full purge on purpose. Everything after this is targeted.
Verify.

Verify with curl, twice.

curl -sI https://client.example/some-page/ \
  | grep -i cf-cache-status

First request should return cf-cache-status: MISS. Second request, within a minute, should return cf-cache-status: HIT. If the second request also returns MISS, the edge isn't caching and you need to check Cache Rules on the zone.

Edit-and-purge test. Open a test page in the WordPress editor. Change one word. Update. Then:

curl -sI https://client.example/test-page/ \
  | grep -i cf-cache-status

Expect MISS (FlyingPress purged the URL on update), then HIT on the next call. If you get HIT immediately after an edit, the purge isn't wired. Check the token scope.

Editorial rules for the content team. These sound small. They keep the hit rate up.

After editing a page: Clear current page, never Clear all cache.
Don't purge multiple times in one editing session. One purge per finished edit.
Don't cache logged-in users. FlyingPress has this setting. It should be off.

The "Clear all cache" button is a footgun. One editor with a habit of clicking it after every save can flatten a 700-page cache twice a day, and the preloader has to rebuild it every time, and you're back to hitting origin for everything until it's warm again.

What the fixes killed

Once the caching path was clean and the preloader was allowed through the Cloudways CF layer, the symptoms stopped stacking.

The preloader now runs at sitemap scale. 775 pages cached, matching the site's public URL count.
The targeted subpath is warm before campaigns launch.
No more silent challenge on the preloader. We added a targeted allow rule for the preloader's origin IP on the Cloudways CF layer.
CPU under normal traffic returned to baseline. No campaign burst has hit the same wall since.

The oEmbed problem didn't go away. It got buried under caching. Pages that used to call the third-party endpoint on every render now call it on the render that populates the cache, and once. That's fine at current traffic. If we scale 10x it's back on the list.

The ceiling nobody was tuning past

Here's the part that reframed the whole project.

Even with the caching path fixed, the box is still ten WordPress apps on 32GB of RAM. WordPress with Elementor and WPML has a fat baseline. 100 to 200MB of resident memory per PHP worker is normal. Each app's FPM pool holds workers, and each pool competes with nine other apps' pools for the same RAM.

Nineteen OOM kills in nine days. Roughly two per day. That's not a tuning problem. That's a tenancy problem.

I ran another week of measurements to confirm. PHP-FPM pm.max_children was already conservative on each app. opcache.memory_consumption was fine. There was no single-app memory leak. What was happening was traffic peaks on any one of the ten apps could push the shared box past its budget, and the kernel would kill whichever process it liked least.

At that point, tuning has diminishing returns. The next lever is one of:

Upgrade the Cloudways plan. Bigger shared box, same ten apps. Buys headroom. Cheapest option. Reversible in an hour.
Consolidate. Move some apps off. Reduces contention. Reversible, but touches other clients' sites, which means coordination cost.
Move the biggest app to dedicated. OVH Advance-3 is on the shortlist. Reversible if we keep the Cloudways plan alive during transition for a week.

We're in the middle of that decision as I write this. Client hasn't picked yet.

Rules I set for myself before any migration

Written down so future-me doesn't get clever.

Reversibility test. Whatever I change, I need a path back to the previous state inside 24 hours. If I can't answer "how do I undo this by tomorrow", the plan isn't finished.

One variable at a time. If we move to dedicated, we don't also convert to Multisite in the same window. Two-variable moves are undebuggable when they break. Move first, convert later, or the other way around, never both.

Dual-run during migration. Old and new live in parallel for at least a week. DNS TTL down to 300 before the cutover so we can bounce back. No burning the old server until the new one has survived a real traffic day.

No infra work during editorial hours. The content team is in a different timezone from mine. If I break something they're editing, they can't reach me for hours. Migration windows are theirs, off-hours, negotiated.

What we're deferring on purpose

Every one of these has been raised at some point. Every one gets a "not yet" and a reason.

Headless WordPress rewrite. Six-month project. The current fixes bought us the runway to plan it properly instead of doing it under an incident.
WordPress Multisite conversion. High risk. Doesn't fix the tenancy problem, just moves it. Not in the migration window.
Laravel or Filament rewrite. Premature. Editors know WordPress. Retraining the content team is a bigger cost than the hosting bill.
Cancelling the Cloudways CF add-on. Still doing useful WAF and bot work. Don't remove security infrastructure during a period of instability.

Boring answer. Correct answer.

Checklist for future-me (and you, if it helps)

Save this. Run through it after every infra change, and before every campaign launch.

[ ] FlyingPress → Cloudflare integration shows Zone ID + scoped API token
[ ] Cloudways Edge Page Caching is OFF
[ ] curl cf-cache-status returns MISS then HIT on an anonymous URL
[ ] Edit a test page, save, verify purge (MISS on next curl, then HIT)
[ ] FlyingPress preloader completes; cached page count ≈ sitemap URL count
[ ] Preloader is not being challenged by the Cloudways CF layer
    (check firewall events for the preloader's source IP)
[ ] Logged-in user caching is OFF in FlyingPress
[ ] Editorial team briefed: "Clear current page", never "Clear all"
[ ] 48h monitoring after any change:
    - CPU peaks
    - PHP-FPM saturation (max_children hit count)
    - Origin request rate (should drop after caching is live)
    - p95 response time
[ ] If OOM count > 5 in 7 days → stop tuning, escalate to hardware decision
[ ] Before any migration:
    - Reversibility path in writing
    - One variable at a time
    - Dual-run for at least a week
    - Migration window agreed with editorial team

The thing I keep learning

Caching plugins get treated as install-and-forget. Set up FlyingPress, tick the boxes, move on. The interesting problems are never inside the plugin. They're at the seams. Which layer caches. Which layer purges. Which layer challenges. Whether the preloader looks like a browser to the security layer. Whether the token has the right scope.

The 14 July spike wasn't a FlyingPress bug. It wasn't a Cloudflare bug either. It was a seams bug. Two Cloudflares, one plugin, and nobody's integration was pointed at the right zone. The fix was ten minutes of configuration once we knew what to configure. Getting to "what to configure" took two weeks.

Boring headers. Expensive lessons. Write down which layer owns what before you need to know.

Verify That Your GitHub Actions Deployment Actually Landed on the Server

hello world_leo — Thu, 28 May 2026 04:22:01 +0000

I trusted GitHub Actions' green checkmark for months before I noticed it was lying to me. Not lying, exactly. It was reporting on whether curl returned 200. Whether the actual code on the server had changed was a separate question, and one nothing in the pipeline was answering.

Our deployment webhook, in outline:

GitHub Actions → HTTPS POST → deploy.php on server → runs deploy.sh

Why webhooks, not SSH

Hypernode (our host) IP-whitelists SSH access. GitHub Actions runs on rotating Azure IPs that never survive the whitelist. Port 443 has no such restriction, so a webhook it is. This is a common pattern on managed hosting.

The workflow:

- name: Trigger deploy on server
  run: |
      response=$(curl -s -w "\n%{http_code}" \
        -X POST https://www.yoursite.nl/deploy.php \
        -H "X-Deploy-Token: ${{ secrets.DEPLOY_WEBHOOK_TOKEN }}" \
        --max-time 300)

      code=$(echo "$response" | tail -n 1)
      if [ "$code" != "200" ]; then
          echo "❌ Deployment failed (HTTP $code)"
          exit 1
      fi
      echo "✅ Deployment successful"

And deploy.php runs deploy.sh synchronously:

exec("bash {$script} 2>&1", $output, $exit);
echo implode("\n", $output) . "\n";
http_response_code($exit === 0 ? 200 : 500);

exec() blocks until the script finishes. So GitHub gets an accurate exit code, right? Mostly. There's one case where the pipeline reports success while nothing on the server changed.

The silent lock skip

deploy.sh uses a lock file to prevent concurrent deploys:

LOCK_FILE="/tmp/mysite-deploy.lock"

if [ -f "$LOCK_FILE" ]; then
    log "ERROR: Deployment already in progress. Aborting."
    exit 1
fi

touch "$LOCK_FILE"
trap "rm -f $LOCK_FILE" EXIT

Normal case: a second push arrives while the first deploy is still running. The second deploy.sh sees the lock, exits 1, deploy.php returns 500, GitHub marks the workflow failed. Correct behavior.

Broken case: the first deploy dies mid-way. Network blip on git fetch, OOM kill, curl times out on the composer install, whatever. The trap doesn't fire cleanly, the lock file stays. Every subsequent deploy silently exits 1 with the "already in progress" message. Depending on how your workflow parses the response, that failure may or may not surface loudly.

Red flag: a lock file left behind by a crashed process is the deployment equivalent of a hung phone line. Sounds like success until you check who's actually there. If your deploy pipeline uses a lock, it needs a way to distinguish "genuinely running" from "abandoned lock from last time."

Beyond the lock issue, the pipeline had a deeper hole. Even when a deploy succeeded, we had no proof of what commit actually landed. Verifying meant SSHing in and checking file timestamps, or grepping for a string we knew existed only in the latest commit. Not scalable.

Fix part 1: return 423 when locked

Detect the lock in deploy.php before invoking deploy.sh and return HTTP 423 Locked. It's the correct status code for "this resource is currently locked", and it lets the workflow tell the difference between a real deploy failure and a stale lock.

$lockFile = '/tmp/mysite-deploy.lock';
if (file_exists($lockFile)) {
    http_response_code(423);
    echo "LOCKED: Deployment already in progress. Try again shortly.\n";
    exit;
}

exec("bash {$script} 2>&1", $output, $exit);
echo implode("\n", $output) . "\n";
http_response_code($exit === 0 ? 200 : 500);

Handle 423 explicitly in the workflow so the CI log names the actual failure:

if [ "$code" = "423" ]; then
    echo "⚠️  Deploy locked, previous deploy still running or lock is stale"
    exit 1
fi
if [ "$code" != "200" ]; then
    echo "❌ Deployment failed (HTTP $code)"
    exit 1
fi

You could add lock-age detection so a lock file older than X minutes gets treated as stale and removed automatically. I chose to keep this manual, on the theory that a stale lock indicates something worth investigating before the next automated deploy tramples over whatever broke.

Fix part 2: write a version file

At the end of deploy.sh, after every step has completed, capture the deployed commit SHA and write it to a public file:

# Right after git reset --hard
DEPLOYED_COMMIT=$(git rev-parse HEAD)
log "Code updated (commit: $DEPLOYED_COMMIT)"

# ... composer install, yarn build, artisan migrate, etc.

# At the very end, after every other step succeeded
echo "$DEPLOYED_COMMIT $(date -u +%Y-%m-%dT%H:%M:%SZ)" > public/version.txt
log "Version file written: $DEPLOYED_COMMIT"

Order matters. Write this after git clean (which would nuke a version.txt that isn't gitignored) and after every other build step (so a partial deploy doesn't advertise a SHA that isn't fully live). Add public/version.txt to .gitignore so it never gets committed.

Fix part 3: verify from the CI job

After the webhook returns, poll version.txt from GitHub Actions and confirm the SHA matches what we just pushed.

- name: Verify deployment landed
  run: |
      EXPECTED="${{ github.sha }}"
      echo "Expected SHA: $EXPECTED"

      for i in $(seq 1 20); do
          sleep 15
          DEPLOYED=$(curl -sf "https://www.yoursite.nl/version.txt" 2>/dev/null | awk '{print $1}')
          echo "Attempt $i/20: server=${DEPLOYED:-none}"
          if [ "$DEPLOYED" = "$EXPECTED" ]; then
              echo "✅ Deploy confirmed on server: $DEPLOYED"
              exit 0
          fi
      done

      echo "❌ Deploy verification FAILED after 5 minutes"
      echo "Expected: $EXPECTED"
      echo "Got:      ${DEPLOYED:-no response from server}"
      exit 1

Polls every 15 seconds for up to 5 minutes. Since deploy.php blocks synchronously on exec(), by the time this step runs the deploy has already finished and the first poll almost always matches. The retry loop only matters if there's a CDN cache between GitHub Actions and the origin.

What it looks like now

The GitHub Actions log after a successful deploy:

Run EXPECTED="6f250fe1a2b9bb11ea826325a8a486b25279dfb1"
Waiting for deploy to complete on server...
Expected SHA: 6f250fe1a2b9bb11ea826325a8a486b25279dfb1
Attempt 1/20: server=6f250fe1a2b9bb11ea826325a8a486b25279dfb1
✅ Deploy confirmed on server: 6f250fe1a2b9bb11ea826325a8a486b25279dfb1

Confirmed on the first attempt. And the same file is readable from anywhere without credentials:

$ curl -s https://www.yoursite.nl/version.txt
6f250fe1a2b9bb11ea826325a8a486b25279dfb1 2026-05-28T03:41:30Z

Lesson learned: the deploy pipeline should prove that it succeeded, not assume that a 200 means it did. curl returning 200 says the webhook responded. It doesn't say the code changed. Two lines of version.txt plus a poll loop is the difference between hoping and knowing.

Why not just `git log` on the server

You could SSH in and run git log -1. That requires SSH access (blocked in our case), a separate monitoring job, or a human at a keyboard. The version.txt approach works over plain HTTPS from anywhere, from any browser, with no credentials. Same information, easier to consume, and the CI checks it automatically as part of the deploy job.

You could SSH in and run git log -1 — but that requires SSH access from CI (blocked for us), a separate monitoring job, or manual checks. The version.txt approach works over plain HTTPS with no
credentials, from anywhere, including your browser.

It also separates concerns: GitHub Actions verifies the outcome, not the process. Even if the internals of deploy.sh change, the verification contract stays the same "does the server report the right SHA?"

Summary

┌──────────────────────────────────────────────────┬───────────────────────────────────────┐
│                     Problem                      │                  Fix                  │
├──────────────────────────────────────────────────┼───────────────────────────────────────┤
│ No signal when deploy is locked                  │ deploy.php returns HTTP 423           │
├──────────────────────────────────────────────────┼───────────────────────────────────────┤
│ No proof of what commit landed                   │ deploy.sh writes public/version.txt   │
├──────────────────────────────────────────────────┼───────────────────────────────────────┤
│ GitHub shows success without server confirmation │ GitHub Actions polls and verifies SHA │
└──────────────────────────────────────────────────┴───────────────────────────────────────┘

Three small changes. Zero new dependencies. Works with any stack that can serve a static file over HTTP.

Building a Multi-Tenant Firebase Application: Environment Setup and Role-Based Access Control

hello world_leo — Sat, 14 Feb 2026 14:27:01 +0000

I built a multi-tenant PWA on Firebase this year: three environments, role-based access across organizations, offline support. Firebase makes the front-end wiring easy. What took real thought was splitting environments so a staging accident could never touch prod, and getting RBAC to work offline without an extra Firestore read on every request.

Three Firebase projects, not one

Firebase's free tier caps you at one database per project. Trying to run dev, staging, and prod on a single project is the fastest way to have your staging misconfigure and start writing to production collections.

I use three completely separate Firebase projects: my-app-dev, my-app-staging, my-app-production. Each has its own auth users, Firestore, storage bucket, and hosting deployment. Isolated security rules, isolated usage quotas, isolated service account permissions.

Environment	Purpose	Deployment
Development	Local testing with emulator	Manual
Staging	Pre-production testing	GitHub Actions on `staging` push
Production	Live users	GitHub Actions on `main` push

Environment config: explicit, not clever

The layout:

app/
├── .env                    # Development (uses emulator)
├── .env.staging            # Staging deployment
├── .env.production         # Production deployment
└── src/
    └── config/
        └── firebase.ts     # Firebase initialization

Development .env:

REACT_APP_FIREBASE_PROJECT_ID=my-app-dev
REACT_APP_USE_EMULATOR=true
REACT_APP_FIREBASE_API_KEY=your-dev-api-key
REACT_APP_FIREBASE_AUTH_DOMAIN=my-app-dev.firebaseapp.com

Production .env.production:

REACT_APP_FIREBASE_PROJECT_ID=my-app-production
REACT_APP_USE_EMULATOR=false
REACT_APP_FIREBASE_API_KEY=your-prod-api-key
REACT_APP_FIREBASE_AUTH_DOMAIN=my-app-production.firebaseapp.com

My worst near-miss was letting the build script guess which environment file to load. Something upstream flipped the environment detection, staging built with production credentials pointed at the production Firestore, and I didn't catch it for hours.

Red flag: never let the build script guess the environment. Copy the file explicitly in the deploy step. If the command doesn't say the environment name in plain text, you have a foot-gun waiting to fire.

Right:

cp .env.production .env && pnpm build

Custom claims for multi-tenant RBAC

The classic pattern is storing roles in Firestore and reading them on every request. It works, but every protected page then loads two documents: the data you actually want, and the role check that guards it. It also breaks the moment the network drops.

Firebase Custom Claims live inside the JWT ID token. The client caches the token. Firestore security rules read the claims directly with no extra document lookup. And they work offline because the token is already on the device.

Claims structure for a user who admins one organization and views another:

interface CustomClaims {
  isAdmin?: boolean;                       // Platform super-admin
  organizations?: Record<OrgId, Role>;     // Org-specific roles
}

{
  "isAdmin": false,
  "organizations": {
    "org_abc123": "admin",
    "org_xyz789": "viewer"
  }
}

There is no Firebase Console UI for custom claims. You set them via the Admin SDK. I keep a small script in firebase-admin-scripts/:

// scripts/set-user-claims.js
const admin = require('firebase-admin');

admin.initializeApp({
  credential: admin.credential.cert('./service-account.json')
});

async function setUserClaims(email, organizations) {
  try {
    const user = await admin.auth().getUserByEmail(email);

    await admin.auth().setCustomUserClaims(user.uid, {
      isAdmin: false,
      organizations: organizations
    });

    console.log(`Claims updated for ${email}`);
  } catch (error) {
    console.error('Error:', error.message);
  }
}

setUserClaims('user@example.com', {
  'org_abc123': 'admin'
});

Lesson learned: custom claims cap at 1000 bytes per user. Enough for organization IDs and roles. For fine-grained per-resource permissions, keep those in Firestore and use claims only for the org membership set. Overload the claim and Firebase silently rejects future writes.

Security rules that use the claims

The rule below reads the org role straight from the token and applies different access per role, no extra Firestore read.

rules_version = '2';
service cloud.firestore {
  match /databases/{database}/documents {

    function hasOrgAccess(orgId, allowedRoles) {
      return request.auth != null &&
             request.auth.token.organizations[orgId] in allowedRoles;
    }

    match /organizations/{orgId} {
      allow read: if hasOrgAccess(orgId, ['admin', 'editor', 'viewer']);
      allow write: if hasOrgAccess(orgId, ['admin']);
    }

    match /organizations/{orgId}/documents/{docId} {
      allow read: if hasOrgAccess(orgId, ['admin', 'editor', 'viewer']);
      allow create, update: if hasOrgAccess(orgId, ['admin', 'editor']);
      allow delete: if hasOrgAccess(orgId, ['admin']);
    }
  }
}

Test rules before deploying. Firebase Console → Firestore → Rules → Rules Playground simulates a user with specific claims against a specific path. I also keep unit tests using @firebase/rules-unit-testing:

describe('Organization access', () => {
  it('allows an org member to read documents', async () => {
    const db = getFirestore(testEnv, {
      uid: 'user123',
      token: { organizations: { org_abc: 'viewer' } }
    });

    const doc = db.collection('organizations').doc('org_abc');
    await assertSucceeds(doc.get());
  });

  it('denies non-member access', async () => {
    const db = getFirestore(testEnv, { uid: 'user456' });

    const doc = db.collection('organizations').doc('org_abc');
    await assertFails(doc.get());
  });
});

Skip these tests and you'll ship a rule that permits more than you think. There is no runtime warning, just quiet exposure.

Deployment pipeline

Push to staging, GitHub Actions builds and pushes to the staging Firebase project:

# .github/workflows/deploy-staging.yml
name: Deploy to Staging

on:
  push:
    branches: [ staging ]

jobs:
  deploy:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v3

      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '18'

      - name: Install pnpm
        run: npm install -g pnpm

      - name: Install dependencies
        run: pnpm install

      - name: Build for staging
        run: |
          cp .env.staging .env
          pnpm build

      - name: Deploy to Firebase Hosting
        uses: FirebaseExtended/action-hosting-deploy@v0
        with:
          repoToken: '${{ secrets.GITHUB_TOKEN }}'
          firebaseServiceAccount: '${{ secrets.FIREBASE_SERVICE_ACCOUNT_STAGING }}'
          projectId: my-app-staging
          channelId: live

Store the Firebase service account JSON as a GitHub secret. Never commit it. The account has full write access to the Firestore database it belongs to.

Backup strategy for production

Three layers, each covering a different failure mode.

Daily backups with 7-day retention catch accidents you notice within the week: dropped collection, bad migration you spot on Monday.

gcloud firestore backups schedules create \
  --database='(default)' \
  --recurrence=daily \
  --retention=7d

Weekly backups with 8-week retention catch slow-burn corruption you don't notice for a month: a bad migration that only affects new writes.

gcloud firestore backups schedules create \
  --database='(default)' \
  --recurrence=weekly \
  --retention=8w \
  --day-of-week=SUN

Monthly exports to Cloud Storage with 365-day retention are the everything-is-on-fire recovery: full portable dump, retrievable a year out.

gsutil mb -l us-central1 gs://my-app-backups

gcloud scheduler jobs create http firestore-monthly-export \
  --location=us-central1 \
  --schedule="0 2 1 * *" \
  --uri="https://firestore.googleapis.com/v1/projects/my-app-production/databases/(default):exportDocuments" \
  --http-method=POST \
  --oauth-service-account-email=backup-sa@my-app-production.iam.gserviceaccount.com \
  --headers="Content-Type=application/json" \
  --message-body='{"outputUriPrefix":"gs://my-app-backups/monthly"}'

Native daily and weekly backups are free. You pay only when you restore. Cloud Storage exports run about $0.02-0.05/GB/month. For most apps under a few GB, all three layers combined stay under $5 a month.

Firebase Emulator: not optional

firebase emulators:start --only auth,firestore,storage

Skip this and you'll burn through your Firebase read quota on trivial dev work, or worse, write test data into production because you forgot to swap credentials. The emulator gives you zero API quota consumption, instant reset by restarting the process, offline development, and no chance of a stray test write hitting production.

What actually matters

The Firebase parts people underestimate are the ones that survive an incident, not the ones that make the app feel snappy.

Separate projects per environment eliminate the entire class of "staging leaked into prod" bugs. Custom claims give you RBAC that works offline and doesn't cost a Firestore read per page load, but remember the 1000-byte cap. Three-tier backups cover three failure modes for the price of a coffee per month. And the Firebase Emulator is the difference between working locally without thinking and getting surprised by a bill.

References I keep bookmarked:

Questions about Firebase infrastructure or DevOps? Connect with me on LinkedIn or visit my portfolio.

Emergency Server Recovery: A 4-Hour Race Against Time

hello world_leo — Tue, 18 Nov 2025 15:57:06 +0000

## The 3AM Wake-Up Call
You know that feeling when your phone buzzes at an ungodly hour and your stomach drops? That was me, staring at a frantic message: "Site is showing weird content. Help!"

I grabbed my laptop. The WordPress site was serving pharmaceutical spam to visitors. Classic compromise. The clock started ticking.

## Hour 1: Damage Assessment (03:00 - 04:00)
First rule of server emergencies: don't panic, but move fast.
What I Did First
SSH into the server, check if I still had access. Good news: credentials still worked. Bad news: everything else.

Check recent file modifications

find /var/www/html -type f -mtime -7 -ls | head -20

Tons of suspicious PHP files scattered everywhere. The wp-content/uploads folder was full of backdoors. Someone had gotten in through an outdated plugin, probably.
Quick Isolation
Pulled the site offline with a maintenance page. Better to show "down for maintenance" than spam pills to your visitors.

Quick nginx block

location / {

return 503;

}


Took a snapshot of everything before touching anything. You need evidence, and you might need to rollback if things go sideways.

## Hour 2: The Cleanup (04:00 - 05:00)
Finding the Entry Point
Checked Apache/Nginx logs for unusual POST requests:

grep -i "POST" /var/log/nginx/access.log | grep -E ".(php|asp|jsp)" | tail -100


Found it. An old contact form plugin with a known vulnerability. They uploaded a shell through a file upload field that wasn't properly validated.
Nuclear Option with Surgical Precision
Here's the thing about compromised WordPress sites: you can't trust anything. But you also can't just delete everything because you need the data.

My approach:

1. Backed up the database (even though it might be compromised)
2. Downloaded all uploaded media files
3. Saved the wp-config.php (to get database credentials)
4. Nuked everything else

Backup first

mysqldump -u dbuser -p dbname > backup_$(date +%Y%m%d_%H%M%S).sql

Fresh WordPress install

wget https://wordpress.org/latest.tar.gz

tar -xzf latest.tar.gz


Database Surgery
The database had malicious entries in these tables:

`wp_options` (autoload hooks)
`wp_posts` (spam content injected)
`wp_users` (unknown admin accounts)

Cleaned them manually. Yes, manually. Running automated scripts on a compromised database is asking for trouble.

-- Remove suspicious admin users

SELECT * FROM wp_users WHERE user_login NOT IN ('known_admin_1', 'known_admin_2');

DELETE FROM wp_users WHERE ID = [suspicious_id];

-- Check for injected JavaScript in posts

SELECT ID, post_title FROM wp_posts

WHERE post_content LIKE '%<script%'

OR post_content LIKE '%iframe%';


## Hour 3: Hardening & Recovery (05:00 - 06:00)

The Rebuild
Fresh WordPress core, clean database, restored media files. Now comes the part most people skip: actually securing the thing.
File Permissions That Make Sense

Directories: 755

find /var/www/html -type d -exec chmod 755 {} \;

Files: 644

find /var/www/html -type f -exec chmod 644 {} \;

wp-config.php: 440

chmod 440 /var/www/html/wp-config.php

chown www-data:www-data /var/www/html/wp-config.php
Disable File Editing
Added to wp-config.php:

define('DISALLOW_FILE_EDIT', true);

define('DISALLOW_FILE_MODS', true);


No more editing themes from the admin panel. If you need to update something, do it through SFTP like a proper developer.
Web Application Firewall
Configured ModSecurity with OWASP rules. Basic stuff:

apt-get install libapache2-mod-security2

cp /etc/modsecurity/modsecurity.conf-recommended /etc/modsecurity/modsecurity.conf

Changed SecRuleEngine DetectionOnly to SecRuleEngine On.

The Forgotten Hero: Fail2Ban
Set up Fail2Ban to block brute force attempts:

/etc/fail2ban/jail.local

[wordpress]

enabled = true

filter = wordpress

logpath = /var/log/auth.log

maxretry = 3

bantime = 3600


## Hour 4: Automation & Insurance (06:00 - 07:00)

Automated Backups That Actually Work
Wrote a bash script for daily backups to remote storage:

!/bin/bash

TIMESTAMP=$(date +%Y%m%d_%H%M%S)

BACKUP_DIR="/backups"

Database

mysqldump -u user -ppassword database > $BACKUP_DIR/db_$TIMESTAMP.sql

Files

tar -czf $BACKUP_DIR/files_$TIMESTAMP.tar.gz /var/www/html

Send to remote (S3, or whatever)

aws s3 cp $BACKUP_DIR/ s3://your-bucket/backups/ --recursive

Keep only last 7 days locally

find $BACKUP_DIR -type f -mtime +7 -delete


Added it to crontab: `0 2 * * * /home/scripts/backup.sh`

Monitoring Setup
Installed basic monitoring so next time we catch things early:

Uptime monitoring

curl -X POST https://cronitor.io/api/monitors \

-H "Content-Type: application/json" \

-d '{"name": "client-site", "url": "https://example.com"}'

File integrity monitoring

apt-get install aide

aide --init
SSL & Security Headers
Fresh SSL certificate with Let's Encrypt:

certbot --nginx -d example.com -d www.example.com

Added security headers to nginx:

add_header X-Frame-Options "SAMEORIGIN" always;

add_header X-Content-Type-Options "nosniff" always;

add_header X-XSS-Protection "1; mode=block" always;

add_header Referrer-Policy "strict-origin-when-cross-origin" always;




## The Aftermath
By 7AM, site was back online. Clean, secured, monitored.

What the client saw: Their site back up, faster than before, with new security measures.

What they didn't see: The 4 hours of SSH sessions, database queries, and three cups of coffee.

## Lessons From The Trenches

### What Worked

- Having a clear mental checklist for compromises
- Not trusting anything on a compromised system
- Taking backups before any action (even if you think you don't need them)
- Hardening during recovery, not after

### What I'd Do Different
- Should have set up monitoring earlier (obviously)
- Could have automated the cleanup scripts better
- Next time: keep a USB drive with common tools ready

### Prevention Is Cheaper Than Cure
After this incident, I set up these things for all client sites:

- Weekly automated backups (tested restores monthly)
- Security plugins with proper configuration
- Update automation for core/plugins/themes
- File integrity monitoring
- Login attempt limiting

### The Technical Stack Behind This Recovery
- OS: Ubuntu 20.04 LTS
- Web Server: Nginx 1.18
- Database: MySQL 8.0
- Backup Storage: AWS S3
- Monitoring: Uptime Robot + custom bash scripts
- Security: ModSecurity, Fail2Ban, Cloudflare WAF

## Final Thoughts
Emergency recoveries are stressful. Your hands shake a bit when you're running rm -rf on a production server at 5AM. But this is what separates someone who just "knows WordPress" from someone who actually understands infrastructure.

The client was happy. The site survived. And I learned (again) that keeping systems updated and monitored is way easier than 4-hour emergency sessions.

Now I keep this checklist printed and stuck to my monitor. Because there will be a next time. There's always a next time.

------

Time taken: 4 hours
Coffee consumed: 3 cups
Client panic level: Reduced from 10/10 to 2/10
Would I do it again: Absolutely. But let's try to avoid it, yeah?

5 WordPress Performance Issues I Fix Every Week (And How You Can Too)

hello world_leo — Fri, 26 Sep 2025 13:41:00 +0000

I've audited a lot of WordPress sites for clients who complained their pages felt slow. The same five things come up almost every time. None of them are exotic. All of them get worse quietly. None of them fix themselves.

Why it matters

Every second on page load costs conversions. Industry estimates put the hit at roughly 7% per second on e-commerce; even with generous rounding, a slow site is money left on the table. For a $100K/month store, a single second is $7K/month. For a service business, it's leads that never call.

I've seen 8-second WordPress home pages. All of them fixable, but only once you know where to look.

1. Database bloat

WordPress databases accumulate junk. Spam comments, post revisions (default: unlimited), expired transients, orphaned metadata from plugins that got uninstalled but left their tables behind. I recently audited a 2-year-old site with a 400MB database. Actual content was 50MB.

Start with a size check by table:

SELECT
    table_name AS 'Table',
    ROUND(((data_length + index_length) / 1024 / 1024), 2) AS 'Size (MB)'
FROM information_schema.TABLES
WHERE table_schema = 'your_database_name'
ORDER BY (data_length + index_length) DESC;

Sort by size, look at what's inflating. Common suspects: wp_options with autoloaded transient data, wp_postmeta from a form plugin that never cleans up submissions, wp_comments with 40,000 spam rows.

Cleanup tools like WP-Optimize handle the basics. For wp_options autoload issues, do it manually with SQL and measure query time before and after. One client's site dropped from 6.2s to 3.8s just from database cleanup and setting WP_POST_REVISIONS to a sane number in wp-config.php.

2. Unrestrained images

The most common thing I see is a 5MB hero image loading on mobile. A photography portfolio I inherited had 47 images totaling 180MB on the homepage. On 4G, that's a coffee break to load.

Three fixes, in order of leverage.

WebP conversion. WordPress supports the format if you register the MIME type:

// functions.php
function add_webp_support($mimes) {
    $mimes['webp'] = 'image/webp';
    return $mimes;
}
add_filter('upload_mimes', 'add_webp_support');

Then batch-convert the existing library (any WebP plugin will do it with JPEG fallback for older browsers).

Responsive sizes. WordPress generates them automatically but only if the theme uses wp_get_attachment_image() or the block editor's image block. Sites hand-coded three years ago often bypass this and serve one giant asset to every screen.

CDN with image transforms. Cloudflare's free tier resizes and reformats on the fly if you enable Polish and Mirage. Zero code changes, immediate weight reduction.

An e-commerce client I worked with went from 8.2MB homepage weight to 1.4MB. Load time improved by 65%.

3. Plugin overload

I audited a small business site running 47 active plugins. One was a full e-commerce suite, active because someone had briefly considered selling something two years ago and never turned it off. The contact form was a separate plugin. So was the "recent posts" widget. And the newsletter. And the redirects. And the schema markup.

The audit is dumb but works: list every plugin, deactivate them one at a time, measure page load between each. The ones with big TTFB improvements are the culprits.

Common offenders:

Page builders (Elementor, Divi) shipping 2MB+ of CSS and JS to render a landing page a static HTML file would render in 50KB
Social sharing plugins loading 20+ networks nobody uses
Two SEO plugins because someone installed Yoast, then someone else installed Rank Math, and nobody disabled the other
Backup plugins scheduled during peak hours

Removing 12 plugins from that 47-plugin site dropped Time to First Byte from 2.1s to 0.8s.

4. No caching, or worse, wrong caching

About 40% of the sites I audit have no caching at all. Another 30% have caching configured in a way that actively hurts them, most commonly a WordPress caching plugin fighting a server-level cache, both trying to store the same responses with different invalidation rules. Every deploy purges one but not the other. Every editorial change bypasses one but is stuck in the other.

The stack that works for most WordPress sites:

Server level: Nginx FastCGI cache (or Apache mod_cache). Serves static HTML for cached responses without touching PHP.
Application level: Redis object cache for query results. Cuts database round-trips on complex pages.
Browser level: proper Cache-Control headers on static assets.
CDN: Cloudflare or similar in front of everything for edge caching.

Minimum Nginx FastCGI cache config:

location ~ \.php$ {
    fastcgi_cache_valid 200 60m;
    fastcgi_cache_valid 404 10m;
    fastcgi_cache_bypass $skip_cache;
    fastcgi_no_cache $skip_cache;
    # ... other fastcgi settings
}

$skip_cache should trip for logged-in users, POST requests, and any admin path (wp-admin, wp-login.php, xmlrpc.php). Get this wrong and different users see each other's carts.

A high-traffic news site went from 1.2s server response time to 180ms after this configuration landed.

5. Hosting mismatch

Running a WordPress site with real traffic on $5/month shared hosting is optimizing the wrong side of the equation. You'll spend weeks tuning the site while the underlying constraint is a shared kernel with 400 other tenants competing for CPU.

Red flags on a hosting audit:

Shared hosting for sites with 10,000+ monthly visitors
No SSD storage (yes, this still happens)
PHP version older than 8.1
No option for server-level caching
Memory limit at 128MB or lower

Rough sizing guide by monthly traffic:

Visitors/month	Right tier	Cost range
0-5K	Quality shared hosting	$10-20/mo
5K-50K	Properly configured VPS	$20-50/mo
50K+	Managed WordPress or custom VPS	$50+/mo

Moving one client from $8/month shared to a $25/month VPS took their site from 4.5s to 1.8s. Same code, same content, different machine.

Where to start

If you're only going to fix one thing today, do the database cleanup. An afternoon of work, immediate visible impact, no product tradeoffs. If you're fixing two, right-size the hosting. Those two together cover most of the wins on a neglected WordPress site.

The other three (images, plugins, caching) touch product decisions. Which images belong on the homepage. Which plugins the team actually depends on. Which cache invalidation strategy fits the editorial workflow. Those conversations belong with the site owners, not the DevOps engineer running the audit.

Plugin optimization: 15-40% speed improvement
Proper caching: 30-70% speed improvement
Hosting upgrade: 25-60% speed improvement

Combined effect: Most sites see 40-70% overall improvement when all issues are addressed properly.

Common Mistakes to Avoid

Over-optimization: Don't chase perfect scores, chase good user experience
Plugin addiction: More caching plugins doesn't mean better performance
Ignoring mobile: 60%+ of traffic is mobile, optimize for it first
One-time fixes: Performance optimization needs ongoing maintenance
Cheap shortcuts: Free solutions often cost more in the long run

Your Next Steps

If your WordPress site is slow, start with these priorities:

Run a performance audit - Get baseline numbers first
Clean your database - Often the biggest quick win
Optimize images - Especially if you have a visual site
Audit plugins - Remove what you don't actually need
Set up proper caching - This can transform your site

Performance optimization isn't just about technical tweaks - it's about creating better user experiences that convert visitors into customers.