Building an Explainable ICD-10-PCS Recommender with Constrained LLM Traversal

During my internship at Conifer Health (Tenet Health), I got a task on a POC. I was working on a project where we were building an automated abstraction of fields from patient charts for Trauma Registry. For the registry, we also needed to enter ICD-10 Procedure and Diagnosis codes. Sometimes codes are noted in the chart itself. Sometimes there's nothing. We wanted to build a tool that abstracts the procedure codes from the chart and recommends them to abstractors.

As you see, we can't just send the patient chart to an LLM and ask it to give us the codes. LLM will give the codes, but this is where the biggest problem of the LLM comes in—THE HALLUCINATION. Hallucination happens in two ways: one, the LLM can hallucinate procedures performed on the patient, and two, it can hallucinate the codes. We also explored RAG to see how it would work. The problem is the way doctors note procedures in natural language, while the registry rules use strict medical terminology. This causes problems in retrieval and results in higher scores for wrong codes. If retrieval fails, then no matter how good the LLM is, the recommendations will still be bad.

Understanding ICD-10-PCS Code Structure

ICD-10-PCS codes are 7-character fixed length strings. Each position in this string has a specific meaning: section, body system, operation, body part, approach, device, qualifier.

Each position can take a value between A to Z and 0 to 9 except I and O.

The value at each position is not independent. The value at a position depends on values at its previous positions. So you can think of this like a mapping: first you select position 1 character, which opens a window for position 2 characters. Possible values depend on what was selected at position 1, and the same way possible values for position 3 depend on what was selected at positions 1 and 2.

Once positions 1, 2, and 3 are selected, this points to a table, and characters 4, 5, 6, and 7 must be picked as a row from this table.

As you can see, this is more like a tree structure. Once you understand this structure, the solution becomes clear.

If you see, at each position we have 34 possible characters which gives us 34^7 ~ 52.5 billion codes, but the actual valid codes in the CMS registry is 79k. Which tells us that not all combinations are valid combinations.

The Approach

Once you understand the tree structure, the solution becomes clear:

Level 1: Select Section (0 = Medical & Surgical)
  ↓
Level 2: Select Body System (D = Gastrointestinal)
  ↓
Level 3: Select Operation (T = Resection)
  ↓
Level 4: Select Row from Table 0DT
  ├─ Body Part: J (Appendix)
  ├─ Approach: 0 (Open)
  ├─ Device: Z (None)
  └─ Qualifier: Z (None)
  ↓
Result: 0DTJ0ZZ

Pruning the Search Space

At each level of the tree, we're making a decision that narrows down all future options:

• Position 1 has multiple possible values (the 17 sections). Think of each as the root of a separate tree.
• When you select a Position 1 value (say, Section 0), you've locked into that tree. All other trees are eliminated.
• Position 2 now only has valid values that exist under your selected Section 0 tree, not all 31 body systems, just the ones that apply to Medical & Surgical.
• As we traverse down each level, we're pruning the search space. Instead of exploring 79,000 possibilities, at each node we're deciding among just a handful of valid options (typically 5 to 15 characters).
• By the time we reach Level 4 (the table), we've narrowed down to one specific table with one specific set of rows to choose from.

Where the LLM Comes In

When traversing the tree, at each node we need to make a decision on which branch to take next. The LLM makes this decision.

The key is that the LLM doesn't see all 79,000 codes or even all possible options at each level. We give it only what it needs to decide.

At each level, we maintain two types of prompts:

1. Generic instructions :- How to evaluate and select the next child. These stay the same no matter which node you're at.
2. Child-specific context :- Based on the node you are at, you select the options available and their definitions with extra instructions.

For example, at Level 2 (Body System selection), if you're in Section 0 (Medical & Surgical), the LLM sees only the 31 body systems that belong to Section 0, not all possible body systems across all sections. It receives generic guidance on how to pick a body system, plus only the definitions it needs for those 31 options.

As a result of this setup, the LLM focuses on a small, constrained decision at each step. It can't hallucinate paths that don't exist because it literally cannot see them.

You can think of this as either a multi-agent system (a specialized agent at each level) or a single LLM traversing a constrained tree. Either way, the structure itself prevents invalid decisions.

The Implementation Pipeline

With this tree setup, the pipeline comes down to 3 stages: extract, traverse, validate.

Extract: Scan the chart once and pull out each procedure and its metadata. This way, we don't have to pass the entire patient chart repeatedly at each stage. During tree traversal, the LLM only sees the data it needs to make decisions.

Traverse: For each procedure, the LLM moves through the tree level by level, seeing only valid options at each node. After 4 decisions, it has picked a complete code.

Validate: At each stage, we only present valid options. If the model hallucinates and picks a character that isn't in the list, the tree won't progress further and we end up with a code shorter than 7 characters—immediate rejection. After row selection, we do a final complete code check against the set of 79k legal codes.

Special Cases

Bilateral Procedures

When a surgeon documents "bilateral chest tubes placed" in one sentence, that's actually two separate ICD-10-PCS codes because Left and Right are different body parts. The extraction layer flags bilateral procedures. When it does, we fork the procedure into two separate traversals—one for right side, one for left side. Each produces its own code.

Repeat Imaging

A trauma patient might get three CT scans of the head across 12 hours to monitor for bleeding. But NTDS rules say you only code the first scan, not the repeats. The pipeline tracks timestamps and filters out repeat imaging. Only the initial scan gets coded and sent to the registry.

Results

This system is a recommender for human abstractors. On our test set, we achieved 81.8% recall—we catch most procedures. Coders see the recommended code with verbatim evidence from the chart, verify it, and move on. No more manual chart hunting.

Comments

[SARAN]

"You my GOD SIRAJ"