In the world of software engineering, we are witnessing a fundamental collision of two opposing paradigms. Classical programming is deterministic: based on Alan Turing’s theoretical model and the Von Neumann architecture, it operates on the principle that the same initial state plus the same program always equals the same final state. Conversely, Large Language Models (LLMs) are stochastic: they generate outputs by sampling from probability distributions, meaning the same input can—and often does—produce a different output every time.
The challenge for modern architects is not to eliminate this unpredictability, but to engineer around it. By using deterministic code as a "skeleton" or "container," we can bound the probabilistic intelligence of an LLM into a reliable, production-ready system.
1. The Core Tension: Determinism vs. Stochasticity
To build robust AI systems, we must first understand why these two worlds sit at odds:
- The Deterministic Gold Standard: Classical computation relies on pure functions—operations with no side effects that always return the same value for the same inputs. We reinforce this with static type systems (like Rust or TypeScript) that provide compile-time guarantees of correctness.
- The Stochastic Reality: LLMs operate via stochastic token sampling. Even when setting "Temperature = 0" for greedy decoding, distributed infrastructure and floating-point non-associativity can introduce micro-level variations that cascade into different tokens.
The goal of production AI is to ensure that while the LLM's "thinking" may be fluid, the system's behaviour is bounded.
2. The Three-Layer Architecture
The most effective production AI systems follow a layered model that wraps the "probabilistic brain" inside a "deterministic shell":
| Layer | Type | Responsibility | Examples |
|---|---|---|---|
| Deterministic Shell | Code / Logic | Routing, retries, and state transitions. | Temporal Workflows, FSMs |
| Probabilistic Core | LLM Inference | Extraction, generation, and interpretation. | LLM Activities, Embeddings |
| Validation Boundary | Hard Constraints | Checking outputs against formal rules. | Pydantic, SMT Solvers, JSON Schema |
3. Key Techniques for Bridging the Gap
Structured Outputs and Constrained Decoding
The "compiler goes blind" the moment text is passed to an LLM. To fix this, we use constrained decoding to restrict token selection at each step, ensuring the output adheres to a formal grammar or JSON Schema. Using libraries like Pydantic to bridge LLM responses to typed Python objects increases parsing success rates from ~60% to near 100%.
Deterministic Orchestration (Temporal)
A major hurdle in AI agents is ensuring reliability through infrastructure failures. Systems like Temporal separate the system into deterministic workflows and non-deterministic activities.
- Workflows are replayable; if a process crashes, the system reconstructs the exact state from an event log.
- Activities (like LLM calls) are retryable with exponential backoff, leading to a 99.99% workflow completion rate.
Finite State Machine (FSM) Guardrails
Rather than letting an agent "decide" its next move entirely through prompting, architects use FSMs to define allowed transitions. An agent in a "Planning" state might be physically prevented from calling an "Execute" tool until it transitions to the correct state. This makes certain failure modes structurally impossible.
Formal Verification and Model Checking
For high-stakes environments, we can use SMT Solvers (Satisfiability Modulo Theories) or model checking to verify that an LLM-generated plan satisfies logical constraints before execution. This provides a mathematical proof that the output is valid.
4. The "Blueprint First" Philosophy
The emerging best practice in AI architecture is the "Blueprint First, Model Second" approach. In this framework, the LLM never decides the high-level workflow path; instead, the code defines the blueprint, and the LLM is invoked only for bounded sub-tasks within that structure. Research shows this approach can yield a 10.1 percentage point improvement in complex user-tool scenarios over traditional agentic baselines.
Conclusion: The Flight Control Analogy
Think of a production AI system like a flight control system:
- The Autopilot (Deterministic FSM) handles the known rules of flight.
- The AI Co-pilot (LLM) interprets ambiguous radio calls and suggests routes.
- The Formal Verification Layer checks if the AI’s suggestions violate airspace rules; if they do, they are blocked deterministically, not negotiated.
Ultimately, we are not replacing software with AI; we are using deterministic software to contain AI. Classical code remains perfect for things with known rules—routing and validation—while LLMs fill the gaps that rules cannot reach, such as understanding intent and extracting meaning.
Top comments (1)
The "skeleton" framing maps well to what I found in production. In my pipeline I use the knowledge graph as the deterministic skeleton -- it determines which entities get retrieved and what relationships exist between them -- while the LLM only handles natural-language generation on top of that fixed structure. When outputs drift, the issue is almost never the generation step; it is almost always the retrieval boundary letting the wrong entities through, which is a deterministic problem with a deterministic fix.