Muhammad Ali Mughal

Posted on Nov 10 Edited on Nov 10

Config & Feature Flags That Quietly Diverged Across Environments

#laravel #architecture #devops #webdev

We shipped a new invoicing flow on a Tuesday. It had passed three weeks of QA on staging, with automated tests and a client sign-off. Within an hour of release, production invoices for one tenant group were rounding tax differently from the PDFs they had approved. The code was identical. The difference was a feature flag: on staging, tax-engine-v2 had been switched on for everyone months earlier by a developer testing something else, and nobody turned it off. On production it was still off for most tenants. QA had been testing a code path that production customers had never run, and the old path had a rounding bug that only appeared with the new invoice layout.

I wrote this with help from AI tools and checked every config sample, command and code snippet myself. I am a Principal Software Engineer in Karachi with 9+ years across PHP/Laravel, Vue, Node and .NET microservices, and at Loop Origin we build and maintain Laravel SaaS platforms where staging and production must behave the same way to mean anything. This is Part 4 of the series Silent Failures in Production SaaS. Part 1 covered cron jobs that never ran, Part 2 covered webhooks that stopped arriving and Part 3 covered email that never landed. Config drift is the quietest of them all: nothing fails, the environments just stop being comparable.

📚 Series Roadmap — Silent Failures in Production SaaS

Part 1: Cron Jobs That Never Ran: Heartbeats & Dead-Man Switches for SaaS

Part 2: Webhooks That Stopped Arriving: Gap Detection & Reconciliation

Part 3: Email That Never Landed: SPF/DKIM/DMARC Drift & Bounce Pipelines

Part 4 (This Post): Config & Feature Flags That Quietly Diverged Across Environments

Part 5: Backups That Never Restored: Automated Restore Drills

Part 6: Synthetic Checks & SLOs for the Paths Nobody Clicks

Related earlier posts: Zero-Downtime Database Migrations in Laravel & Node.js SaaS · Delayed Jobs, Schedulers & Clock Skew Gotchas · Production Observability on a Budget

How do you detect config and feature flag drift between staging and production?

Direct answer: Define which config keys and flags must match across environments, publish a redacted fingerprint of their effective values from each environment, and diff them automatically on every deploy and on a schedule. Check environment variable keys against a template in CI. Give every feature flag an owner and expiry date, and alert on flags whose state differs without a recorded reason.

Why drift does not look like a failure

Config drift rarely breaks anything on the day it happens. Someone raises a queue timeout on production during an incident and forgets to update the infrastructure repo. Someone adds a variable to staging's .env by hand. A flag is flipped for a demo. Each change is reasonable in the moment. Weeks later, the cost shows up in three ways:

Tests stop predicting production. Staging passes because it runs different code or different limits.
Incidents take longer. The first hour goes on "but it works on staging" instead of the actual cause.
Rollbacks do not roll back. Reverting code does not revert a hand-edited value on one server.

Here are the drift types I see most often in Laravel and Node SaaS stacks:

Drift type	Example	Symptom	Detection
Missing env key	`STRIPE_WEBHOOK_SECRET` set on staging only	Feature works in QA, silently disabled in prod	Key parity check against `.env.example` in CI
Different value	`retry_after` 90 in prod, 600 on staging	Duplicate long-running jobs only in prod	Config fingerprint diff
Stale config cache	`config:cache` not re-run after env change	New value ignored until next deploy	Fingerprint after deploy vs expected
`env()` outside config files	`env('FEATURE_X')` in a controller	Returns `null` in prod once config is cached	Static grep in CI
Flag state divergence	Flag on for all in staging, 5% in prod	QA tests a path customers do not run	Flag state diff with owners
Per-server divergence	One worker host has an old `.env`	Intermittent behaviour by host	Fingerprint per host, not per environment

The Laravel trap that bites everyone once

Laravel's docs are explicit about this one. After you run php artisan config:cache, the .env file is not loaded, and calls to env() outside of config files return null. Locally nobody caches config, so the code works. In production, it quietly takes the default branch:

// Wrong: works locally, returns null in production after config:cache
if (env('INVOICE_V2_ENABLED')) { /* ... */ }

// Right: read env only inside config files
// config/invoicing.php
return [
    'v2_enabled' => (bool) env('INVOICE_V2_ENABLED', false),
];

// app code
if (config('invoicing.v2_enabled')) { /* ... */ }

A one-line CI guard catches regressions:

# Fail the build if env() called outside config/
if grep -rnE '\benv\(' app/ routes/ resources/views/ --include='*.php'; then
  echo "env() used outside config/. Use config() instead." && exit 1
fi

Key parity: every environment declares the same variables

Values should differ between environments. The set of keys usually should not. Treat .env.example as the contract and check it in CI and on each server at deploy:

#!/usr/bin/env bash
# env-parity.sh: compare keys in .env.example with the live environment file
keys() { grep -E '^[A-Z0-9_]+=' "$1" | cut -d= -f1 | sort -u; }

missing=$(comm -23 <(keys .env.example) <(keys "${1:-.env}"))
extra=$(comm -13 <(keys .env.example) <(keys "${1:-.env}"))

[ -n "$missing" ] && echo "Missing keys:" && echo "$missing"
[ -n "$extra" ]   && echo "Undeclared keys:" && echo "$extra"
[ -z "$missing$extra" ] && echo "env parity OK"
[ -z "$missing" ]

Undeclared keys are not always errors, but each one is something a new environment will not have.

A redacted config fingerprint per host

The most useful control I have added to client platforms is a small command that prints the effective values of the keys we care about, with secrets reduced to "set" or "missing". Because it reads config(), it reflects the cached config the app actually uses, not the .env file on disk:

// app/Console/Commands/ConfigFingerprint.php
namespace App\Console\Commands;

use Illuminate\Console\Command;
use Laravel\Pennant\Feature;

class ConfigFingerprint extends Command
{
    protected $signature = 'ops:config-fingerprint';
    protected $description = 'Print redacted effective config and flag state as JSON';

    private array $tracked = [
        'app.env', 'app.debug', 'cache.default', 'queue.default',
        'queue.connections.redis.retry_after', 'session.secure',
        'session.lifetime', 'mail.default', 'services.stripe.webhook_secret',
        'invoicing.v2_enabled',
    ];
    private array $secret = ['services.stripe.webhook_secret'];
    private array $flags = ['tax-engine-v2', 'new-checkout'];

    public function handle(): int
    {
        $out = ['host' => gethostname(), 'release' => config('app.release')];
        foreach ($this->tracked as $key) {
            $v = config($key);
            $out['config'][$key] = in_array($key, $this->secret, true)
                ? ($v ? 'set' : 'missing') : $v;
        }
        foreach ($this->flags as $flag) {
            $out['flags'][$flag] = Feature::for(null)->active($flag); // global scope
        }
        $this->line(json_encode($out, JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES));
        return self::SUCCESS;
    }
}

Run it on every host after deploy, store the JSON as a build artefact, and diff:

ssh app@staging-web-1 'cd /var/www/app && php artisan ops:config-fingerprint' > staging.json
ssh app@prod-web-1    'cd /var/www/app && php artisan ops:config-fingerprint' > prod.json
diff <(jq -S 'del(.host, .release)' staging.json) <(jq -S 'del(.host, .release)' prod.json)

Some keys are supposed to differ, such as app.env. Keep an allowlist of expected differences in the repo, and alert on anything else. Comparing hosts within one environment matters just as much: a worker server with an old .env produces the "only sometimes" bugs that are hardest to trace.

Feature flags need an owner and an expiry

Laravel Pennant, LaunchDarkly, Unleash or a homegrown table all share the same failure mode: flags outlive their purpose and their state stops being deliberate. I keep a flag registry in the repo:

# flags.yaml
- name: tax-engine-v2
  owner: billing-team
  created: 2026-09-14
  expires: 2026-11-30
  intended:
    staging: { rollout: 100 }
    production: { rollout: 10 }
  reason: "Gradual rollout; staging at 100% for QA of v2 only"

A scheduled job compares the registry's intended state with the fingerprint output and flags three things: state that differs from intended, flags past expires, and flags present in code but missing from the registry. With Pennant's database driver, remember that resolved values are stored per scope, so changing the definition does not change already-stored values until you purge them with php artisan pennant:purge.

Most importantly, QA should test the state production will have after release. If staging is at 100% and production will be at 10%, test both paths.

When we take on maintenance of an existing platform as part of our Laravel development work, a fingerprint and flag audit is one of the first things we run, because it explains a surprising number of "random" bugs. For multi-service systems we build through custom web application development, the same fingerprint idea applies per service, with one combined diff on the deploy dashboard.

Diagnostic checklist

[ ] .env.example lists every key, and CI checks key parity for each environment.
[ ] No env() calls outside config/; CI enforces it.
[ ] config:cache runs on every deploy after env changes, never by hand only.
[ ] A redacted config fingerprint is produced per host after each deploy.
[ ] Staging vs production fingerprints are diffed with an allowlist of expected differences.
[ ] Hosts within one environment are compared to each other.
[ ] Every flag has an owner, intended state per environment and an expiry date.
[ ] Flags in code but not in the registry fail CI.
[ ] QA tests the flag state production will have after release.
[ ] Hotfix config changes made during incidents are back-ported to the infra repo within a day.

FAQ

Should staging and production have identical config?

No. Credentials, hostnames and scale differ. What should match is the set of keys and the behaviour-changing values such as timeouts, limits and flags, unless a difference is documented.

Is infrastructure as code enough to prevent drift?

It reduces it a lot, but hand edits during incidents, manual flag changes and stale caches still happen. You need detection as well as declaration.

How often should the fingerprint diff run?

On every deploy, plus a scheduled run, for example hourly or daily, to catch changes made outside the pipeline.

Can I expose the fingerprint over HTTP?

Only on an internal or authenticated endpoint, and only with secrets redacted. Config values reveal a lot about your infrastructure.

What about Node services?

The same pattern works: load config once into a typed object at startup, print a redacted JSON fingerprint, and diff it the same way. Libraries such as zod or envalid help validate keys at boot.

How do I clean up old feature flags?

Use the expiry date in the registry. When a flag expires, either make the winning path permanent and delete the flag, or renew it with a new reason.

Next in this series, Part 5 covers backups that never restored and how to automate restore drills.

DEV Community