Feature flags are supposed to be the easy part. You call getFlag('my-flag'), you get back a boolean, you branch. That's true right up until the code that needs the flag is running somewhere your normal "ask the container for a service" pattern doesn't reach, and then the flag becomes a genuinely hard distributed-systems problem, not a two-line if.
That's what happened when I needed to gate a set of workflow-step changes behind a single kill switch, in a pipeline where the steps execute inside Inngest, which on our infrastructure means each step is its own fresh Lambda invocation. The steps run inside what our codebase's ADR calls "the restricted container", a dependency injection container that, by design, only has memory in it. Nothing else. No CMA client, no request-scoped services, and critically, no straightforward path to "just ask for the current feature flag state."
This post is two related stories: why that restriction exists (a genuinely subtle AsyncLocalStorage bug, documented in an ADR at the company I work for), and how I threaded a feature flag through it without violating the restriction.
Why workflow steps can't just "ask the container"
Our dependency-injection container is scoped per-request via Node's AsyncLocalStorage (ALS); a request comes in over HTTP, middleware calls containerStore.run() to bind a container for that request's lifetime, and any code downstream calls getContainer() to pull services out of it. That works great for a normal request/response cycle.
Inngest breaks the assumption underneath it. The /inngest endpoint deliberately sits outside the normal HTTP middleware chain; it authenticates via Inngest's own signing key, not our usual auth; so the middleware that binds the container never runs for it. And on our infrastructure, each Inngest step is its own HTTP POST back to /inngest, meaning each step is a fresh Lambda invocation with its own empty ALS context. Once agents started resolving memory through the container (getMemory(), which reads from containerStore), every workflow step that called agent.generate() inside an Inngest step handler started throwing "memory not registered in container", the container simply didn't exist yet in that execution.
The first fix looked reasonable on paper: register a custom Inngest middleware, and in its transformInput hook (which the docs and API surface suggest runs in the step's execution path) call containerStore.enterWith() to bind a minimal container. Single registration point, every step benefits, no per-step boilerplate. It shipped. It broke in production.
The root cause is worth understanding even if you never touch Inngest, because it's a general trap with AsyncLocalStorage and promises: enterWith() only affects the async context it's called from and that context's descendants; not the context of whoever awaited the promise that led there. transformInput runs inside a nested Promise microtask. Calling enterWith() there binds the container for that microtask and its children. But the code that actually invokes the step callback resumes in a different async context (the one where the original promise continuation was registered) and that context never saw the enterWith() call. So getContainer() threw for every step callback, despite the middleware appearing to run correctly.
The actual fix, documented in the team's ADR, is to stop relying on enterWith() and use containerStore.run() instead, at a point high enough in the call stack that Inngest's entire execution engine (the step dispatcher, the user function, every step callback) inherits it as a descendant context:
Wrap the entire
/inngestHTTP handler incontainerStore.run()at the Hono route level. This creates a properly-scoped ALS context before Inngest's execution engine starts, so_start()and all its async descendants (user function, step callbacks,getMemory(), etc.) correctly inherit the container.
The distinction between run() and enterWith() is the whole bug: run() establishes a scope for a callback and everything that callback spawns; enterWith() mutates the current execution's context going forward, which only helps descendants of the call site, not ancestors or siblings. If you're threading anything through AsyncLocalStorage across a promise boundary you don't fully control (a library's internal scheduling, a queue, a retry engine), reach for run() first and treat enterWith() as the exception that needs a very specific justification.
The consequence that matters for this post: even after the fix, the Inngest container is a partial container. Only memory lives in it. Everything else (HTTP-scoped services like the CMA client) is deliberately absent, and workflow steps must not assume it's there. Which means: if you need a feature flag inside a workflow step, getContainer() is not the way, full stop. It was never going to have flags in it, and stuffing more services into a partial container built for one specific purpose (working around a request-scoping gap) is exactly the kind of scope-creep that makes a narrow fix into a maintenance trap.
The actual path a flag takes to reach a step
So if not the container, what? The answer turned out to already exist for a different field, and I just had to follow the same wire.
Feature flags are evaluated once, at the HTTP layer, in feature-flags-middleware.ts:
const featureFlagsMiddlewareHandler = async (c: Context, next: Next): Promise<void | Response> => {
const requestContext = c.get('requestContext')
const spaceId = requestContext.get('spaceId')
const userId = requestContext.get('userId')
const userEmail = requestContext.get('userEmail')
const evaluationContext: ContentfulEvaluationContext = {
targetingKey: spaceId,
kind: 'multi',
organization: { key: spaceId },
space: { key: spaceId },
...(userId ? { user: { key: userId, ...(userEmail ? { email: userEmail } : {}) } } : {}),
}
let flags: FeatureFlags
try {
await featureFlagsHandler.isReady()
const result = await featureFlagsHandler.evaluateFlagsArray(evaluationContext, FLAGS)
flags = result.flags as FeatureFlags
} catch (error) {
logger.error('Flag evaluation failed, defaulting all flags to false', { error })
flags = Object.fromEntries(FLAGS.map((f) => [f.key, f.defaultValue])) as FeatureFlags
}
requestContext.set('featureFlags', flags)
await next()
}
Two things here matter beyond the obvious "evaluate the flags." First, a failed flag-evaluation call defaults every flag to false rather than propagating the error; a flag service outage degrades to "everything's off," never to "request fails" or, worse, "flags silently evaluate to some unpredictable stale state." For a kill switch, fail-closed is exactly the behavior you want: if you can't reach the flag service, you should not accidentally turn a new code path on.
Second, and this is the part that matters for the Inngest problem, the resolved flags get written onto requestContext, not the DI container. requestContext (Mastra's request context, not our internal ALS container) is a different mechanism, and it's the one detail that makes this whole thing work: it's a plain, JSON-serializable object field on CustomMastraContext. Inngest's own connector (@mastra/inngest) propagates RequestContext by JSON-serializing it into the event payload it hands to each step. It has no AsyncLocalStorage mechanism and can't carry a class instance or a live service across a step boundary; that's precisely why memory needed the whole ADR-024 saga. But a plain object survives JSON serialization for free.
So when the workflow is started:
workflow.createRun().startAsync({ inputData, requestContext })
requestContext, including featureFlags, rides along in the Inngest event payload to every step, and each step's execute({ requestContext }) callback gets it back out, no container, no ALS, no custom middleware required. The flag isn't retrieved by the step; it's handed to the step, because it was captured before the request ever crossed into Inngest's execution model.
Reading it back is a five-line helper:
export function isGoogleDocsAgentImprovementsEnabled(requestContext: RequestContext<CustomMastraContext>): boolean {
const featureFlags = requestContext.get('featureFlags') as FeatureFlags | undefined
return featureFlags?.[GOOGLE_DOCS_AGENT_IMPROVEMENTS_FLAG] ?? false
}
The ?? false matters for the same reason the middleware defaults to false on evaluation failure: absence of the flag should never be interpreted as "on."
The migration shape: legacy untouched, new path additive
With the read path solved, the actual migration across three workflow steps (table classification, a preview agent, and a graph-review step) followed one consistent shape, which is worth naming because it's the shape you want for any real kill switch, not just this one:
const useExecutor = isGoogleDocsAgentImprovementsEnabled(requestContext)
if (tables.length === 0 && !useExecutor) {
// Legacy path only: with the flag off, short-circuit before calling the agent at all,
// exactly as before this migration.
return { contentTables: [], excludedTableIds: [], classifications: [], normalizedDocument, contentTypes }
}
// ...flag on: same LLM call, now wrapped in runLlmStep, with an abstain branch that
// covers the zero-tables case explicitly instead of short-circuiting before it.
The requirement, stated by the person who owns this flag, was explicit: this needed to be a real flag-off/flag-on branch, not just "the retry logic changed underneath the same code path." That's a meaningfully different bar than most feature flags get held to. A lot of flags in practice gate a config value or a threshold; the code path barely changes shape. A genuine kill switch needs the old code to still be reachable, untouched, byte-for-byte, when the flag is off, so that turning the flag off is a real rollback and not a "well, mostly the same thing minus this one adjustment."
That constraint shows up as a small but telling type-safety wrinkle at the call site. The new path always supplies a degradeTo, so runLlmStep never actually returns FAILED in practice; but the return type is still the full four-state union, and TypeScript can't know from the call site alone that FAILED is unreachable:
// FAILED is unreachable here because degradeTo is always provided above, so runLlmStep never
// returns FAILED -- but the LlmStepResult union's FAILED variant has no `value`, so this check
// is required for TypeScript to narrow `result` before the .value access below.
return result.outcome === 'FAILED' ? defaultToContentClassification(tables) : result.value
I could have used a non-null assertion or an as cast to silence this, but the branch it forces is genuinely cheap insurance: if a future edit ever removes degradeTo from this call site, the code still degrades gracefully to the same default instead of throwing on an undefined .value. The type system asking for a branch you believe is unreachable is sometimes just noise; but here it's noise that happens to double as a real fallback, so I kept it as a branch instead of asserting past it.
Each of the three migrated steps got its own version of "what does a reasonable degrade look like":
-
Classify tables - degrade to: every table classified as
content. Abstain on: zero tables (nothing to classify). -
Preview agent - no degrade at all. A failure throws
PREVIEW_AGENT_FAILED, matching the exact legacy failure behavior - this step's whole job is the LLM call, so there's no sensible fallback value to degrade to. - Review entry-block graph - degrade to: no corrections applied, proceed with the algorithmically-built graph. Abstain on: zero entries with a non-empty rich-text field.
None of those decisions are generic; they came from actually asking "what happens today, right now, in production, when this fails?" for each step individually, and making sure the flag-on path degrades to something no worse than that.
What I'd take from this beyond the specific bug
The Inngest/ALS bug is infrastructure-specific, but the shape of the mistake generalizes: a context-propagation mechanism that "usually works" is exactly the kind of thing that breaks silently under promise boundaries you don't control. enterWith() reads as correct in a synchronous mental model ("bind the context, now downstream code sees it") and only breaks once you trace exactly which async chain the call site's descendants are in versus which chain resumes the awaited promise. Test it against the actual execution engine you're trusting, not against your assumption of how it schedules.
The feature-flag piece generalizes even further, past Inngest entirely: when you need a value to survive a serialization boundary you don't control (a queue, a webhook payload, a distributed workflow engine), put it on the plain-data object that already crosses that boundary, not on the stateful container that never will. requestContext won this fight against the DI container not because it's architecturally superior in general, but because it happened to already be JSON-serializable and already wired into the exact payload Inngest carries. Once you notice that constraint, the design question stops being "how do I get the container to reach here" and becomes "what already reaches here, and can I ride it."
Top comments (0)