State-Based Testing for Feature Flags

#testing

Why state-based testing matters
Building a comprehensive on/off test matrix
Automating state verification in CI/CD pipelines
Common pitfalls that silently break toggles
Sign-off criteria and documentation for safe toggles
Practical Application: Runbook, checklists, and scripts

There’s a recognizable pattern: teams adopt feature toggles to move fast, then tests and ownership lag. Symptoms show up as flaky CI runs, production incidents after long-idle flips, or rollbacks that don’t actually revert state. The noise is familiar: missing fallback tests, tests that assume a single flag state, and documentation gaps that turn a simple toggle into an emergency maintenance item.

Why state-based testing matters

A toggle is only as safe as both of its states. Treat on and off as separate products that must each be proven stable. When either path is unverified, flipping the switch becomes a risky operational change rather than a low-risk configuration update.

A toggle that breaks the off path is a latent outage: the code behind the flag has diverged or relies on resources not present when the flag is off.
Rollback validation requires proving that toggling from on → off causes no side effects (data corruption, queue misrouting, orphaned background jobs). Demonstrate idempotency for the flip operation.
Testing only the on path creates brittle rollouts; testing only the off path leaves regressions hidden until a rollout. Both need deterministic, automated coverage.

Important: Every feature flag that reaches production must have a defined owner, a lifecycle (TTL or removal plan), and an automated way to test both on and off states.

Building a comprehensive on/off test matrix

Design a test matrix that covers the surface area without attempting impossible exhaustive combinatorics.

Start with this minimal matrix for a single-flag feature:

Flag state	What you verify	Test type	Evidence
`off`	Legacy behavior preserved; no UI entry points appear	Unit / E2E / Snapshot	Passing tests, UI snapshot, logs
`on`	New behavior present; correctness and performance validated	Integration / E2E / Perf	Metrics, traces, smoke-test logs
toggle `on`→`off`	No persisted side-effects; rollbacks revert behavior	E2E / Integration	DB snapshots, audit logs
toggle `off`→`on`	Feature activates without latency spikes	Gradual rollout / Canary	SLO metrics, error budget impact

For multiple flags, avoid exponential explosion by using risk-based selection and combinatorial techniques (pairwise / all-pairs). Pairwise testing provides strong defect detection while greatly reducing test count; it covers every pair of flag settings which, empirically, finds the majority of interaction bugs. Use pairwise generators or tools when you have many boolean flags.

Practical examples:

For a migration flag like new-search-algorithm, unit-test both implementations in isolation, run integration tests with each on/off state pointed at the respective backend, and snapshot the UI differences.
For permission toggles, validate both UI visibility and backend permission checks to avoid UI-only gating that leaves server APIs open.

Automating state verification in CI/CD pipelines

Automation is where state-based testing pays off in velocity and reliability. Make flag-state verification part of your CI matrix with these patterns.

Flag seeding for test runs
- Use a file-based flags fixture (flags.json) or a local dev-server to provide deterministic flag values to test environments. This eliminates flaky dependency on remote flag evaluation during CI. LaunchDarkly documents dev-server and flag files as common approaches for predictable test runs.
API or CLI-driven pre-test setup
- In a job step, set the exact flag values via your feature-management CLI or REST API, then run the test suite. Example (LaunchDarkly REST pattern):

# set a boolean flag for a context (user/environment)
curl -X PUT "https://app.launchdarkly.com/api/v2/users/<projectKey>/<envKey>/<userKey>/flags/<flagKey>" \
  -H "Authorization: <apiToken>" \
  -H "Content-Type: application/json" \
  -d '{"setting": true}'

Evidence: API endpoints exist to programmatically set a single-context flag value and are suitable for CI preconditioning.

Ephemeral dev-server approach (recommended for integration/E2E)

# simplified GitHub Actions excerpt
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Start LD dev-server
        run: ldcli dev-server start --project default --source staging --context '{"kind":"user","key":"ci-test"}' --override '{"my-flag": true}' &
      - name: Run tests
        run: pytest -q

Launching a local flag server synchronizes values into the test runtime and prevents race conditions with shared test environments.

Automated rollback validation
- Add CI jobs that exercise both on and off flows within the same pipeline: set on, run smoke and verification tests, set off, run the same smoke tests and verify no data regressions or lingering side effects.
Gate pipelines on evidence
- Require artifact evidence (screenshots, trace IDs, metrics snapshot) as part of successful pipeline runs for on and off tests before allowing a rollout step.

Common pitfalls that silently break toggles

Identify the failure modes I’ve seen in production and the precise checks that catch them.

Pitfall: UI-only guards, open server APIs.
- Symptom: UI hides feature but API endpoints still accept requests.
- Check: Add contract tests that call the backend with the flag set off and assert server-side enforcement exists.
Pitfall: Fallback value behavior differs from off.
- Symptom: SDK fallback or offline mode yields unexpected variation.
- Check: Include tests for the SDK fallback configuration and mock offline behavior to verify the fallback equals the intended off semantics.
Pitfall: Long-lived flags rot (bitrot + stale code paths).
- Symptom: A flag flipped months later triggers production errors.
- Check: Enforce TTL/cleanup metadata and run scheduled compatibility tests for older flags. Martin Fowler and engineering leaders emphasize lifecycle discipline for toggles.
Pitfall: Test suites only run in one flag state.
- Symptom: CI passes, but flip fails in production.
- Check: Make on and off runs standard pipeline stages; add a flag-matrix job that runs a reduced test set for each relevant state.
Pitfall: Hidden interactions between flags during rollouts.
- Symptom: Two flags enabled together create an unexpected path.
- Check: Use pairwise test generation to ensure critical two-way interactions are validated.
Pitfall: Security/SDK vulnerabilities in flag libraries.
- Symptom: Outdated flag SDK exposes sensitive information or control surfaces.
- Check: Include dependency scanning and security reviews of flag-related packages; treat SDK upgrades as part of flag hygiene. Evidence of real vulnerabilities exists in flag SDKs and should be part of threat modeling.

Sign-off criteria and documentation for safe toggles

Create a checklist that gatekeeps production toggles. Each item is binary: Pass/Fail — require artifacts.

Criterion	What to verify	Required artifact
Owner and TTL	A named owner and removal date or lifecycle step exists	Issue/Confluence entry with owner, TTL
On/off automated tests	CI job(s) covering `on`, `off`, and flip verification exist and passed	CI logs and test reports
Rollback validation	`on`→`off` preserves data integrity and system stability	DB snapshots, audit IDs, smoke test artifacts
Observability	Metrics and traces instrument feature-specific events	Dashboard link, example traces
Targeting verification	Targeting rules resolve predictably for test contexts	Targeting test results / export
Security review	Flag SDKs and APIs validated by SAST/DAST	Security scan report
Cleanup plan	Flag removal PR template created/queued after 100% rollout	Cleanup PR link / calendar reminder

A short sign-off statement to attach to release work: “Feature <<flag-key>> is covered by automated tests for both states, has an assigned owner and TTL, observability is in place, and a rollback path has been exercised in CI.” Store this statement and the evidence links in the feature’s issue tracker entry.

Practical Application: Runbook, checklists, and scripts

Use this runbook as a one-page operational protocol to validate a toggle during a rollout.

Pre-rollout (local/CI)
- Create or update flags.json for the test run or start the dev-server with overrides.
- Run: unit, integration, and a lightweight E2E smoke test in off and on states.
Canary rollout
- Target 1% of users via targeting rules. Monitor error rate, latency, and business metrics for 30 minutes.
Full rollout validation
- After canary confirms stability, increase percentiles in steps (1% → 10% → 50% → 100%) with automated test gates at each step.
Rollback simulation
- In a non-production environment perform on → off and validate DB/object state and side effects.
Clean-up
- Create a remove-flag PR and schedule TTL-based removal once the flag has been at 100% for the retention period.

Checklist (paste into PR template):

[ ] Owner assigned and TTL specified.
[ ] on and off tests added to CI; green.
[ ] Rollback test executed and evidence attached.
[ ] Observability: traces/metrics dashboards updated.
[ ] Security scan passed for flag SDK & code changes.
[ ] Cleanup PR created (or scheduled).

Example automated flip-and-test script (simplified):

#!/usr/bin/env bash
# flip-flag-and-test.sh
FLAG_KEY="$1"
PROJECT="myproj"
ENV="staging"
API_TOKEN="${LD_API_TOKEN}"

# enable for test user
curl -s -X PUT "https://app.launchdarkly.com/api/v2/users/${PROJECT}/${ENV}/ci-user/flags/${FLAG_KEY}" \
  -H "Authorization: ${API_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"setting": true}'

# run quick smoke tests
pytest tests/smoke/test_flag_flow.py::test_feature_on

# disable and re-run
curl -s -X PUT "https://app.launchdarkly.com/api/v2/users/${PROJECT}/${ENV}/ci-user/flags/${FLAG_KEY}" \
  -H "Authorization: ${API_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"setting": false}'

pytest tests/smoke/test_flag_flow.py::test_feature_off

This pattern seeds deterministic state for a test context, runs verification, flips the state, and re-runs the verification. Store the script in your repo and reference it in the CI job for quick validation.

Sources:
FeatureFlag (Martin Fowler) - Taxonomy of flag types, caution about long-lived release flags and advice on lifecycle/cleanup.
Testing code that uses feature flags (LaunchDarkly Docs) - Practical guidance on unit/mocking, file-based flags, dev-server, and testing in production.
5 tips for getting started with feature flags (Atlassian) - Governance, ownership, and cleanup practices used at scale.
Testing with feature flags (GitLab Docs) - E2E testing patterns and selector strategies to keep tests stable across flag states.
Update flag settings for context (LaunchDarkly API) - Example REST endpoints and request format for programmatically setting flag values for contexts.
All-pairs testing / Pairwise testing (Wikipedia) - Rationale and example techniques for covering interactions without exhaustive combinatorics.
Snyk vulnerability: flags package (SNYK-JS-FLAGS-10182221) - Example of security risks in flag SDKs and the need for dependency hygiene.