Invitation: Now, I am officially active on X (Twitter). For new DevOps ideas, you can join me on X (Twitter) as well. Click Here
Article Abstract:...
For further actions, you may consider blocking this person and/or reporting abuse
The framing of "stateless infrastructure supporting stateful intelligence" is the right mental model. You're not replacing one with the other — you're layering them.
The practical tension I see builders run into: context management starts simple (append to session history) and complexity explodes fast once you have multi-turn workflows, user-specific memory, and shared team context all in the same system. The "too little vs too much context" problem becomes a real engineering challenge, not just a tuning knob.
What's your take on where context responsibility lives in the stack? Session layer, application layer, or pushed down into a dedicated memory service? Curious how you'd architect this for a B2B SaaS where multiple users share context about the same account.
Great question, Max. In B2B SaaS, context responsibility is effectively the new "data layer" challenge. I tend to see this as a three-tier architecture:
For your B2B SaaS example, I'd architect it so that the Application Layer orchestrates the assembly. When User A interacts, the app pulls User A's current session + shared Account Context + relevant Account RAG.
The "too little vs too much" problem is solved by Semantic Gating: the Memory Service shouldn't just dump all account data, but use a ranking layer to provide the most relevant account-level context based on the current user's intent.
Essentially, context becomes a "federated query" problem rather than just a history append. Does that alignment match what you're seeing in your current builds?
That’s a very strong framing, and I think you’re describing the direction most mature systems are converging toward.
The three-tier separation you outlined makes a lot of sense in practice:
Session (ephemeral) → immediate intent and short-term continuity
Account (shared memory) → alignment, constraints, and cross-user consistency
Retrieval (long-term) → deeper knowledge and historical patterns
What I particularly like is your shift from “memory” to federated context assembly. That’s the real mental model change. Context isn’t a blob you pass around; it’s something you compose dynamically based on intent.
Overall, yes, this aligns very closely with what I’m seeing in current builds. The teams that treat context as a query + ranking + governance system (not just storage) are the ones scaling reliably.
I'm glad the 'federated context assembly' framing resonates, Jaideep. The mental shift from 'context as a state' to 'context as a dynamic query' is exactly what allows us to bypass the memory bloat of long-running sessions. In that model, the LLM stops being a 'state-holder' and becomes a 'state-composer'. It also naturally solves for multi-user consistency in B2B – you just update the 'Account' tier and every subsequent query across the team reflects that change immediately. It's essentially eventual consistency for AI memory.
That’s a strong way to frame it. Treating the LLM as a state composer instead of a state holder solves both scalability and consistency challenges.
And yes, the “eventual consistency for AI memory” idea fits perfectly; shared context updates propagate naturally without bloating sessions.
State composer vs state holder - that's the key conceptual shift. The eventual consistency angle is especially important in multi-agent systems where you can't afford synchronous context locks. In practice I've found that treating shared context as append-only event logs with async fan-out gives you the consistency without the bottleneck.
Your context engineering section hits on the core tension: context is no longer a nice-to-have layer, it's the API contract.
One pattern that addresses the scalability concern you raise: structured context injection via MCP (Model Context Protocol) servers. Instead of stuffing everything into a stateful backend, MCP servers provide on-demand context from external tools directly into the agent's context window. The agent requests what it needs, gets structured data back, and the backend stays stateless.
This preserves horizontal scaling while giving the AI exactly the context it needs for each request. It's essentially "lazy state" - state exists but only materializes when a specific query requires it.
The trade-off you mention about retrieval latency is real though. In practice, MCP server response time often becomes the bottleneck, not the LLM inference itself.
That’s a very sharp observation, and I think your framing of context as the API contract is exactly where things are heading.
The MCP pattern you described is a strong answer to the scalability problem. Treating context as on-demand, structured retrieval instead of preloaded state solves a lot of issues around memory bloat, synchronization, and horizontal scaling. “Lazy state” is a great way to describe it, state exists, but only materializes when the system explicitly asks for it.
It also introduces a cleaner separation of concerns:
The model handles reasoning
The MCP layer handles context retrieval
The backend remains stateless and scalable
That’s a much more sustainable architecture than trying to pack everything into a single persistent context layer.
In a way, we’re moving from optimizing prompts to optimizing context pipelines.
Exactly, Jaideep. The shift to context pipelines also forces us to rethink the 'Evaluator' role in LLM-native development. When context is dynamic and sourced via MCP, we need near real-time observability of what context was actually retrieved for a given reasoning step. It's no longer just about the output, but about the lineage of the state that led to it.
I'm actually exploring how to formalize these 'context contracts' to ensure that agents remain deterministic even as their retrieval sources scale. Have you seen any frameworks addressing this 'retrieval-lineage' problem specifically?
Checking out your new post on AI-Native products now - very timely! 🚀
That’s a great point. As context becomes dynamic, evaluating just the output isn’t enough; the lineage of retrieved context becomes critical.
I’m seeing early efforts in tracing and observability tools, but not a complete solution yet. What you’re describing around context contracts + lineage feels like the next important layer for making these systems reliable and debuggable.
Exactly. If context is a "federated assembly" rather than just retrieval, then every piece of assembled context needs a Context Contract—a guarantee of its constraints and freshness at the moment of assembly. Lineage then becomes the audit trail of these contracts. In B2B, this isn't just a debugging tool; it's a governance requirement. We're essentially moving towards "Context Observability" as a first-class citizen in the stack.
Exactly, Jaideep! You articulated the shift perfectly: 'moving from optimizing prompts to optimizing context pipelines.'
By offloading context retrieval to a dedicated MCP layer, we maintain the reasoning depth of the model without the overhead of massive, stale state. It's essentially 'Just-In-Time' context.
This architecture not only scales better but also aligns with the 'Sovereignty by Design' principle - we only pull in the data exactly when and where it's needed for a specific reasoning step. Glad you found the 'lazy state' framing useful! 🚀
Exactly - that meta-reflection layer is where it gets fascinating. Systems that observe their own pattern recognition start optimizing not just output, but the reasoning process itself. It is the difference between a tool and an agent that evolves its own heuristics.
That’s the inflection point.
When systems start optimizing their own reasoning process, they move from tools to adaptive agents that evolve heuristics over time.
AI-powered systems increasingly depend on context.