Tombstone v1.3-v1.4: Resilience Was the Easy Layer

I removed four || true statements from a GitHub Actions workflow on July 9th and watched CI go red in four different ways within the same run. Not one failure. Four — a Python pytest install that had never actually finished, a ruff lint violation nobody had looked at, a Ruby require path that resolved to nothing, and a Java Gradle wrapper that didn't exist in the repo. All four had been "passing" for who knows how long, because the test steps were configured to succeed no matter what came back.

That's the theme of this release window. v1.2 was about making the running system survive failure — retries, circuit breakers, idempotency keys, DLQs. This one is about making the layers above the running system — the Helm chart, the SDKs, the GitOps pipeline, the CI config — tell the truth about their own state. Resilience isn't a feature you ship, it's a property you discover you're missing.

The Helm chart was only deploying two of five services

Tombstone has five application services — flag-api, gateway, evaluator, intelligence, marketplace — plus the operator. Until v1.3.0, the Helm chart (infra/helm/flagmind) only had Deployment templates for two of them. This wasn't a secret; it was written down in COMPATIBILITY.md under a section literally titled "Known Gap." Run helm install in a fresh cluster and you got flag-api and gateway, nothing else. Anyone deploying evaluator, intelligence, or marketplace was hand-rolling manifests or copy-pasting the two existing templates and hoping the env vars lined up.

v1.3.0 closes that gap: deployment-evaluator.yaml, deployment-intelligence.yaml, and deployment-marketplace.yaml now ship in the chart. evaluator gets an optional HPA behind evaluator.autoscaling.*. intelligence exposes IS_PRIMARY_REGION straight from values.yaml, which matters for the multi-region setup where secondary regions run in read-only relay mode.

Here's the footgun worth flagging, because it isn't obvious until it bites you: every one of those templates uses tombstone.selectorLabels in spec.selector.matchLabels, not the separate tombstone.labels helper, which includes a version label. It's tempting to use the one "labels" helper everywhere for consistency. Don't. Kubernetes Deployment selectors are immutable once the object exists. If your selector helper includes a label that changes on every release — like a chart version — the second helm upgrade you ever run will fail outright, because the new selector no longer matches the old one. tombstone.selectorLabels is a narrower, stable subset — name and component, nothing that changes across releases — specifically so matchLabels never drifts. One line in a template, and it's the difference between a chart that upgrades cleanly forever and one that works exactly once.

SDK parity is a correctness bug, not a feature request

The TypeScript SDK (@tombstone/core) has had the full 5-step evaluation pipeline since v2.0 — preliminary checks, prerequisites, individual targeting, rule matching, fallthrough — with the complete operator set and semver comparisons. The Python SDK didn't. It had basic targeting but was missing large chunks of the operator surface and all of the prerequisite evaluation logic. That's not a nice-to-have gap: a flag with a semver_gte rule or a prerequisite on another flag could evaluate to true in a Node service and silently fall through to the default in a Python service, evaluating the same flag against the same user. Two SDKs, two answers, same input — the kind of bug that looks like a targeting mistake in the dashboard when it's actually a parity gap in the client.

v1.3.0 closes it. packages/sdks/flagmind-python/tombstone/matching.py now implements the full operator set — eq/neq/in/nin/contains/startsWith/endsWith, the four numeric comparisons, and all five semver operators (semver_gt/gte/lt/lte/eq) — plus date_before/date_after. The semver comparison is hand-rolled: a _padded_version() helper left-pads each numeric segment to 5 characters and appends a ~ sentinel for 3-part releases, so 1.0.0-beta sorts below 1.0.0 using pure string comparison. It's the same GrowthBook paddedVersionString() pattern the TypeScript SDK already used, with zero new runtime dependencies — no semver package, no extra install footprint.

Prerequisite evaluation was the other half. evaluation.py now threads an evaluation_cache: dict[str, bool] through the recursive prerequisite check, so a flag with three prerequisites sharing a common ancestor doesn't re-evaluate that ancestor three times. Circular chains are rejected via a _seen_keys tracking set rather than recursing forever. The SDK also distinguishes two failure modes with dedicated exception types: InconclusiveMatchError means a targeting condition couldn't be evaluated locally — missing attribute, type mismatch — and the caller should move on to the next rule. RequiresServerEvaluation means the evaluation genuinely needs data the local cache doesn't have, and the client falls back to a REST round-trip instead of silently returning a wrong default. Conflating the two used to mean callers couldn't tell "skip this rule" apart from "call the server."

Making the API impossible to not find

The previous article's two worst bugs — the Slack kill switch sending environment as a query param when the handler read it from the JSON body, and the four-eyes approval routes that were built, reviewed, and merged but never registered in flag-api/cmd/main.go — share a root cause that has nothing to do with the code itself. Nobody had an easy way to look at what flag-api actually exposed versus what the proto files said it should expose. The OpenAPI spec existed; nobody was looking at it, because there was nowhere convenient to look.

v1.3.0 adds a Redoc explorer at GET /api/v1/docs, embedded via go-redoc rather than pulled from a CDN — it reads the existing grpc-gateway OpenAPI spec at /api/v1/openapi.json, so there's no second source of truth to keep in sync. The implementation detail that made this take longer than expected: the plan referenced a chi adapter for go-redoc that doesn't exist in any published version — the library only ships gin/fiber/echo adapters. The fix was to call goredoc.Redoc{SpecPath: specURL}.Body() directly for pre-rendered HTML and wrap it in a plain http.HandlerFunc. Small feature, but it's the direct answer to how those two bugs happened in the first place: the API surface wasn't something anyone could casually glance at.

GitOps: ordering discipline, then a second controller on purpose

v1.4.1 brought Flux CD v2.3+ into gitops/, structured as three layered Kustomizations — infrastructure, apps, flags — each declaring dependsOn against the one before it. The infrastructure layer also sets healthChecks against the tombstone-operator HelmRelease, which matters more than it sounds: dependsOn alone only blocks applying a Kustomization until the upstream has been applied — it does not wait for the upstream to actually be healthy. Without healthChecks, Flux considers the operator's CRDs "done" the moment the HelmRelease manifest hits the API server, before the CRDs are actually established, and the apps layer — which deploys FeatureFlag CRs via the flagmind chart — could start reconciling against CRDs that technically exist but aren't ready yet. Pairing dependsOn with healthChecks on the upstream is what actually guarantees ordering — this is exactly the kind of GitOps bug that works fine in every test until the day the operator pod is slow to come up.

The more interesting decision is v1.4.2's addition of Argo CD v2.11 — not as a Flux replacement, as a second controller with a deliberately split job. Flux keeps infrastructure: the operator's CRDs, image update automation across the tombstone container images. Argo CD takes over the flagmind chart plus the FeatureFlag and FlagPolicy custom resources themselves. The reason to split: FeatureFlag CRs have a rolloutPct field that the intelligence service's ML rollout recommendations mutate live in the cluster, independent of Git. A naive GitOps setup would see that mutation as drift and revert it on the next reconcile — the platform's own ML-driven rollout logic fought and undone by its own deployment tooling every sync interval.

The fix lives in gitops/clusters/production/argocd/apps.yaml: an ignoreDifferences block excluding /spec/environments/production/rolloutPct and /spec/environments/staging/rolloutPct from diff detection, paired with RespectIgnoreDifferences=true in syncOptions. Both are required — ignoreDifferences alone only suppresses the OutOfSync display; without the sync-options flag, Argo CD still overwrites the field back to the Git value on every sync. Miss either half and the ML rollout percentage gets silently stomped on a fixed interval, a nasty class of bug to chase down because nothing in the intelligence service's logs would look wrong.

Argo CD also needed Lua health checks for Tombstone's own CRDs — FeatureFlag (Pending to Progressing, Synced to Healthy, Error to Degraded) and FlagPolicy (Compliant to Healthy, Violation to Degraded) — because its generic health rollup doesn't understand custom CRD status fields out of the box. Without them, the root Application would just show every FeatureFlag as permanently "Unknown" rather than reflecting real reconciliation state.

Blast radius, but as a deployment gate now

This is the part I like best, because it closes a loop back to the v1.0 launch: blast-radius scoring — the LOW/MEDIUM/HIGH/BLOCKED classification the evaluator computes for flag changes — now gates Kubernetes deployments, not just flag rollouts. v1.4.1 adds an Argo Rollouts AnalysisTemplate named tombstone-blast-radius that polls GET /api/v1/blast-radius?flag_key=<key> on the evaluator service during a canary step. gitops/apps/production/tombstone/rollout-analysis.yaml wires this into flag-api's own Rollout: step to 20% traffic, run the analysis template, promote to 100% only if the result is LOW or MEDIUM, abort immediately on HIGH or BLOCKED (failureLimit: 1, so the first bad reading stops it, no averaging across a window). The same scoring engine that decides whether a flag change is safe to ship now also decides whether a code deployment is safe to ship — the same signal doing double duty at two layers of the stack.

Argo CD Notifications routes sync-failed events into the existing marketplace Slack endpoint (marketplace.tombstone.svc:8086/api/v1/marketplace/slack/actions) rather than standing up a second webhook — one less integration surface to maintain.

Removing the safety net

Back to where I started. The || true removal in .github/workflows/ci.yml (commit 9c9bff9) touched test steps across the Python intelligence service, the TypeScript SDK build, the Python SDK, the Ruby SDK, and the Java SDK — every one configured to report green regardless of outcome. The very next commit (031d041) is titled, accurately, "fix pre-existing test failures surfaced by removing || true," and it fixes four of them:

• The Python SDK's pytest step installed mmh3 and the package itself but never installed pytest — the runner didn't exist in the environment, so python -m pytest had presumably been failing at the shell level the whole time.
• ruff check . --ignore F401 in the intelligence service flagged unused local variables (now_ts, model) a real lint gate should have caught the moment they were written.
• packages/sdks/flagmind-ruby/lib/tombstone.rb didn't exist at all — spec files require "tombstone" but the gem's actual entry point is flagmind.rb, a leftover from the gem's rename. I added a one-line alias file.
• The Java SDK's step ran ./gradlew test, but no gradlew wrapper is committed to the repo — the step was failing to even start.

Fixing the Java one turned into its own small saga across four more commits, because each fix uncovered the next problem: swapping to gradle/actions/setup-gradle failed because the pinned SHA wasn't resolvable; falling back to apt-get install gradle got Gradle installed, but the Ubuntu-runner package is old enough that it chokes on useJUnitPlatform(); installing Gradle 8.7 directly from services.gradle.org fixed that but exposed that build.gradle declared sourceCompatibility = JavaVersion.VERSION_21, an enum literal the older Gradle parser doesn't accept, which needed to become the plain string '21'. Even after all that, the Java tests still fail, but for a real reason: the source files declare package io.tombstone.* while living under io/flagmind/ directories, the same rename leftover that broke the Ruby require. That's now continue-on-error: true with a comment pointing at v1.5.0, not a silent || true — the failure is visible in the CI UI and tracked, instead of invisible and untracked.

Four bugs weren't introduced in this release. They'd been there for a while, hiding under a shell operator that made "it ran" indistinguishable from "it passed." The same commit mirrors a fix v1.2.1 made earlier, adding pytest-asyncio back for a similar reason — a test suite that can fail silently isn't testing the thing it claims to test, it's testing that the CI runner can execute a command.

Supply chain and the honest caveat

Alongside the CI honesty pass, all remaining GitHub Actions workflow files got pinned to immutable commit SHAs instead of mutable tags like @v4 — nine files in one commit (bfb167f), closing out the supply-chain hardening that earlier workflows (flux-bootstrap.yml) had already established as the convention. Two other release blockers landed in the same commit: an ImageUpdateAutomation resource still on the wrong beta API version, and a structural fix to the production rollouts kustomization that was silently missing a resource reference.

Everything above — the layered Flux Kustomizations, the Argo CD split, the blast-radius AnalysisTemplate — is validated against a local k3d test cluster: every kustomize build passes, Flux bootstraps cleanly, Argo CD installs and reconciles. What it is not yet validated against is the actual production target, Oracle Cloud Kubernetes, blocked on an Oracle Cloud account signup that hasn't happened yet. The operator Helm chart is already published to ghcr.io/sairam0424/charts/tombstone-operator at v0.1.0, ready for the day the cluster exists. Better to say that plainly than let "GitOps shipped" imply more than k3d has actually proven.

The throughline across v1.2 through v1.4 is the same lesson at three different altitudes. v1.2 was runtime resilience — the system surviving its own dependencies failing. v1.3 was correctness at the API and SDK boundary — two clients agreeing on what a flag evaluates to. v1.4 is deployment and CI honesty — the pipeline that ships the system telling the truth about its own state, in the right order, without silently eating failures. None of these layers were broken in an obvious way. They were all quietly returning something other than the truth, and the only way to find that out was to stop letting them.