LLM APIs as Infrastructure: Building Deterministic Systems Around Probabilistic AI

The nature of an AI API

Somebody built this thing before you ever touched it. A lab trained it on an enormous amount of text, aligned it, wrapped it behind an endpoint, and rented it to you. You don’t own the model. You inherit the interface: its capabilities, its limits, its context window, and its cost.

When you send a prompt, the model has two sources of information: what it learned during training and the context you provide. Training is fixed. Context is yours. From those two inputs, it produces the most probable continuation.

People hear “probable” and think “guessing.” But it isn’t guessing like a coin flip. It’s producing the most likely continuation given everything the model has learned and everything you just told it — a weighted, structured output shaped by patterns and distributions.

Most of the time, that’s useful.

Sometimes, it’s confidently wrong.

If it gets something wrong, that does not mean there is a bug waiting to be patched.

It’s the behavior you have to design around.

Beyond the API call

Traditional APIs train us to think in predictable systems: send a request, expect a response. Even complex systems have repeatable behavior. There may be state management, caching, retries, and rate limits, but your code usually knows what success and failure look like.

AI systems break that expectation at the model layer. Same prompt, same context, but different output. Not because something failed. Because that's how the component works. The request can succeed, the response can be 200 OK, the JSON can validate, and your logs can look clean, yet the answer can still be wrong. Hallucinations, dropped instructions, valid JSON with incorrect data. Nothing technically failed. The model simply produced the wrong output. That shifts the focus from whether the system responded to whether the response is right.

Traditional APIs usually fail in ways your code already knows how to handle. The request either resolves or rejects, and your error handling catches both.

try {
  const response = await fetchClaim(id);
  renderClaim(response);
} catch (error) {
  showError(error);
}

Traditional software encourages a simple mental model:

Request → Response → Done

AI systems need more checkpoints between the user's input and the final output — and a separate quality gate before anything ships at all.

Runtime (every request):
User Input → Prompt/Context → LLM → Structured Output → Schema Validation → Business Rules → UI

Pre-deployment (CI/CD):
Test Dataset → Full Pipeline → Eval → Pass/Fail Gate → Deploy

The API call is only one step. The model is only one part. It is not the whole app.

Where determinism matters

In a probabilistic system, determinism matters when the output becomes data, triggers an action, or changes the state of the application. If a user says, "I got rear-ended yesterday," the assistant can explain the interpretation in different ways. But the submitted incidentDate cannot be different every time. The system needs one resolved value, one validation path, and one record of what was submitted. The wording around the field can be flexible — the value that enters the system cannot.

Lock these down:

• Final form fields: incidentDate, injuries, accidentType
• Required fields: whether something is complete or missing
• Business rules: whether the form can be submitted
• Actions: sending the form, saving a record, approving a claim
• Audit/history: what value was used and why

Let these breathe:

• The assistant's wording
• The explanation shown to the user
• Suggestions for what to clarify
• Summaries or labels that do not trigger action

The boundary in practice

Every step between the model and your user is a layer you own and control. Here's what that looks like in a real integration.

In a form application, the model can help turn a user's plain language into structured data. A user might write:

"I got rear-ended yesterday. My side mirror broke, but nobody was hurt."

The application expects this shape:

interface IncidentExtraction {
  formData: {
    incidentDate: string;
    accidentType: string;
    damageDescription: string;
    injuries: "Yes" | "No" | "Unclear";
    notes: string;
  };
  feedback: string[];
  confirmation: string;
}

But the model does not receive a TypeScript interface. It receives instructions and a strict schema.

import { GoogleGenAI, Type } from "@google/genai";

const ai = new GoogleGenAI({});

const incidentSchema = {
  type: Type.OBJECT,
  properties: {
    formData: {
      type: Type.OBJECT,
      properties: {
        incidentDate: {
          type: Type.STRING,
          description: "Resolved date in YYYY-MM-DD format."
        },
        accidentType: {
          type: Type.STRING,
          description: "Short classification of the incident."
        },
        damageDescription: {
          type: Type.STRING,
          description: "Brief description of the damage."
        },
        injuries: {
          type: Type.STRING,
          enum: ["Yes", "No", "Unclear"],
          description: "Whether injuries were mentioned."
        },
        notes: {
          type: Type.STRING,
          description: "Any additional relevant details."
        }
      },
      required: ["incidentDate", "accidentType", "damageDescription", "injuries", "notes"]
    },
    feedback: {
      type: Type.ARRAY,
      items: { type: Type.STRING },
      description: "User-facing notes about assumptions or missing details."
    },
    confirmation: {
      type: Type.STRING,
      description: "Short message asking the user to review the filled form."
    }
  },
  required: ["formData", "feedback", "confirmation"]
};

const today = "2026-07-02";

const prompt = `
Today is ${today}.

Extract incident details from the user's description.

User description:
"I got rear-ended yesterday. My side mirror broke, but nobody was hurt."

Rules:
- Resolve relative dates using today's date.
- If a field is unclear, use "Unclear" or an empty string.
- Do not submit the form.
`;

const response = await ai.models.generateContent({
  model: "gemini-2.5-flash",
  contents: prompt,
  config: {
    responseMimeType: "application/json",
    responseSchema: incidentSchema,
    temperature: 0.1 // Lower temperature for more consistent extractions
  }
});

The schema shapes the output. It does not guarantee it. Everything after this line is your code enforcing your rules:

let result: IncidentExtraction;

try {
  result = JSON.parse(response.text);
} catch {
  // Unparseable output. Do not guess. Fall back.
  return askUserToFillManually(
    "I couldn't process that description. Please fill in the form directly."
  );
}

const { formData } = result;
const issues: string[] = [];

// The schema said "YYYY-MM-DD." Verify it anyway.
if (!isValidDate(formData.incidentDate)) {
  issues.push("Incident date could not be resolved.");
}

// Business rule the model knows nothing about.
if (formData.incidentDate > today) {
  issues.push("Incident date cannot be in the future.");
}

// Enums are enforced by the schema — trust, but verify.
if (!["Yes", "No", "Unclear"].includes(formData.injuries)) {
  issues.push("Injury status is unclear.");
}

if (issues.length > 0) {
  // Bad data never renders as truth. The user resolves it, not the model.
  return askUserToReview(formData, issues);
}

renderForm(formData, result.feedback, result.confirmation);

With the schema in place, the model can return a clean string matching your structure:

{
  "formData": {
    "incidentDate": "2026-07-01",
    "accidentType": "Rear-end collision",
    "damageDescription": "Broken side mirror",
    "injuries": "No",
    "notes": ""
  },
  "feedback": [
    "I interpreted \"yesterday\" as July 01, 2026.",
    "I marked injuries as No because the user said nobody was hurt."
  ],
  "confirmation": "I filled in the form based on your description. Please review before submitting."
}

Key practices for reliability:

• Use structured output with a strict schema.
• Always validate the output in your code — schemas are very helpful but not 100% guaranteed.
• Include a feedback array for transparency.
• Fall back gracefully when parsing or validation fails.

This is the boundary. The model interprets language. Your system owns execution and remains the source of truth.

Testing with Evals

The validation above protects each request. It says nothing about whether your pipeline is good overall — that's what evals are for.

With normal software, you test whether something works or fails. With AI, you also need to test whether the answer is good enough.

Evals let you run a test dataset through your pipeline, measure quality against a clear bar, and catch weak outputs before changes reach users.

const results = await Promise.all(
  testDataset.map(test => extractIncidentFromLLM(test.input))
);

const accuracy = results.filter((r, i) =>
  r.formData.injuries === testDataset[i].expectedInjuries &&
  isValidDate(r.formData.incidentDate)
).length / testDataset.length * 100;

if (accuracy < 95) {
  throw new Error(
    `Eval failed: accuracy dropped to ${accuracy.toFixed(1)}% (threshold: 95%)`
  );
}

Best practices: Include edge cases, ambiguous language, and adversarial inputs. Track schema compliance, hallucination rate, and feedback quality. Run evals in CI/CD.

In a real pipeline, you'd replace Promise.all with a concurrency-limited runner that handles retries and failed requests, so the eval tests the system without overwhelming the provider. A production eval keeps a separate score for each thing that can go wrong — like a report card instead of a GPA — so when the numbers drop, you know exactly what to fix.

Closing

The mistake is treating the model as though its job is to know the answer. Knowing was never the point.

Treat the LLM API as infrastructure. Like a database, a message queue, or an API gateway, it has capabilities, constraints, and mechanics.

Your job is not to make the model deterministic. It's to design the architecture around it so that probabilistic outputs become safe, predictable, and useful.

Comments

[Nazar Boyko]

One thing the CI eval gate can't see is the provider changing the model underneath you. Your pipeline runs when you push, but the endpoint can start behaving differently while your code and prompts stay frozen. Do you also run the eval set on a schedule against the production config? That seems like the only way to catch drift you didn't cause. The lock down versus let it breathe split is the most useful framing of this problem I've seen, most posts just say validate everything and stop there.

[Akilah Littlejohn]

Great question! And probably the follow-up article, honestly. Short answer: yes, in layers. CI evals protect against my changes before deployment. Scheduled evals + observability catch drift after deploy. I do rely on tools right now. Eval results feed into Datadog as metrics, monitors for alerts on trends, LangSmith for tracing, and Temporal for recovery (retries, fallbacks, human review). But not all variation is bad. The line I draw: let responses breathe where flexibility helps the user, lock down anything that becomes data, triggers actions, or affects system state. RAG grounds it; schemas shape it, the app decides what's accepted or escalated. CI evals gate my changes; scheduled evals catch theirs; orchestration recovers when something slips through anyway.

[Nazar Boyko]

Thanks for sharing!

[Akilah Littlejohn]

I wrote this as a guide that I wished I had when I first started working with LLM APIs.

[FCodeSolutions]

What LLM is better in function cost/quality to be apply in the laboratory Infrastructure like a lab environment to practice?

[Akilah Littlejohn]

I treat LLMs like JavaScript frameworks... it depends on the use case. For a lab environment, most companies already provide sandboxes, docs, and learning resources to help you practice. I personally like Google AI Studio and Gemini Enterprise Agent Platform because it has a generous free tier and good learning resources. I’d pick one solid model, learn the workflow, and build the infrastructure. Don’t marry the model; design the system.

[Raju Dandigam]

Framing the model endpoint as inherited infrastructure instead of "intelligence you control" is exactly the right mental model. A lot of reliability work gets cleaner once teams accept that the deterministic part lives around the model: tool boundaries, schemas, retries, evaluation gates, and what evidence you keep when the output is wrong. That’s also where trace quality starts to matter, because you can’t improve a probabilistic system if the surrounding application can’t explain which context, tool result, or branch produced the bad outcome. This is the same reason I like tools such as agent-inspect for local-first debugging of agent runs: they make the boundary between model behavior and application behavior much easier to reason about. Curious how you think about the minimum observability contract an LLM-backed feature should have before it ships.

[Mudassir Khan]

the lock down vs let breathe split is the most honest framing i have seen for this kind of problem. most posts say validate everything and stop, which skips the harder design question.

the gap we kept hitting: schema conformance does not guarantee semantic correctness. we had structured output validating perfectly, correct date format, enum values clean, but the model was resolving 'yesterday' relative to training instead of runtime context. schema said nothing. business rule caught it.

your validation layer is the actual contract — the schema just shapes the surface.

curious how you handle the case where structured output validates but the semantic interpretation is still wrong, like the model using Unclear as a hedge when the domain rule should have returned No?

[Michael Salinas]

Thank you for sharing such an excellent post. I really enjoyed reading it.

I’m a Python Full-Stack Engineer with over 10 years of experience designing and building scalable software solutions for clients across a variety of industries. Along the way, I’ve learned that successful projects depend not only on strong technical execution but also on creating real business value.

With my recent contract completed, I’m exploring new opportunities to collaborate with professionals who value innovation, practical problem-solving, and long-term partnerships. I enjoy discussing ideas that combine technical excellence with sound business strategy, creating outcomes that benefit everyone involved.

I believe every connection has the potential to become something meaningful. If you're interested in exchanging ideas, exploring opportunities, or simply connecting with someone who enjoys building impactful technology, I'd be happy to hear from you.

Wishing you success in your future endeavors, and I look forward to connecting.