I built an AI patient, then spent most of my time stopping it from behaving like AI

I’m a backend engineer, and my cofounder is a doctor training in emergency care.

Rounds began with something she kept returning to in our conversations. An exam gives you the relevant information. A patient gives you an opening complaint, and you decide what to ask, what to examine, which investigations matter, and when you know enough to commit.

We wanted to simulate that reasoning process. I assumed the conversational patient would be the easy part.

The first prototype felt impressive for about five minutes.

Then we questioned it more aggressively. Ask about the same symptom twice and part of the history might change. Request a troponin and the model could invent a perfectly plausible value. Phrase a leading question carefully enough and the patient might hand over the diagnosis.

It sounded like a patient, but it did not behave like one.

The model became the interface, not the database

That failure changed the architecture.

Each encounter now runs from a fixed clinical state containing the patient’s history, symptoms, examination findings, investigation results, accepted diagnoses and important alternative diagnoses.

The case is linked to a knowledge graph rooted in medical textbooks. Retrieval brings relevant textbook context into case creation and grading, but the facts of the active patient remain controlled by the simulator. The language model can decide how the patient says something. It cannot decide that the patient suddenly has a different potassium result.

This separation gives us two very different responsibilities:

1. The simulator owns clinical truth and session state.
2. The model translates between that state and messy human language.

That boundary turned out to matter more than the choice of model.

Accept ordinary language without generating clinical facts

A student should not have to know the simulator’s internal vocabulary.

If they type “order an FBC,” “check a CBC,” or “get a complete blood count,” those requests should resolve to the same investigation. We use semantic matching to identify the intended test, then retrieve the result authored for that patient.

The path is roughly:

student request
  -> semantic matching
  -> authored investigation
  -> session cache
  -> formatted result

The model helps interpret the request, but it never supplies the laboratory value. Results are cached for the session, so asking twice cannot produce two different answers.

The same principle applies to examinations. A respiratory examination can be requested in many ways, but the finding still comes from the case state.

Grading needed evidence, not just judgement

The second difficult problem was grading.

Clinical reasoning is expressed in language, so some model interpretation is useful. But a model should not be able to award credit for an action that never occurred.

Rounds therefore records the encounter as an evidence log. Every question, examination, investigation and treatment becomes an event tied to the session.

Deterministic rules handle things we can check reliably, such as accepted names for the diagnosis, important mimics and whether a required investigation was actually ordered. The model interprets less precise language. Before the scorecard is returned, any model-generated credit is checked against the recorded actions.

student action
  -> event log
  -> rules and model interpretation
  -> evidence check
  -> scorecard

This means the model can help understand what the student meant, but it cannot rewrite what the student did.

The useful AI lives at the boundary

The main lesson has been that the model is most valuable where language is ambiguous. The closer something gets to clinical truth, state or evidence, the more deterministic the system needs to become.

Prompting still matters. We use symptom constraints, result checks and prompt-extraction guards to reduce diagnosis leakage. But prompts are one layer of the system, not the safety model for the entire product.

Rounds is still early, and I am trying to find where these boundaries fail. There is one case open without an account:

https://roundsclinical.com/simulator?guest=1&utm_source=devto&utm_medium=article&utm_campaign=aug11_architecture

If you have built a constrained language-model product, how did you decide where probabilistic interpretation should stop and ordinary code should take over?