LLMs can cite healthcare guidelines perfectly and still apply them wrong. AWS just released 38 open-source agent skills across 11 healthcare and life sciences domains that package decision frameworks as executable tools. The evaluation data shows a 70-86% win rate over baseline models on a 410-prompt test suite, exposing the gap between retrieval accuracy and reasoning correctness in regulated domains.
This pattern applies beyond healthcare. Financial compliance, legal reasoning, and underwriting logic all share the same problem: citation does not equal correct application. The plumbing here reveals how to bridge that gap with domain-specific tools that encode decision logic, not just reference documents.
The Citation vs. Application Gap
Standard RAG retrieval pulls the right guideline. The LLM reads it, summarizes it, and cites it correctly. Then it applies the logic incorrectly because healthcare decision frameworks involve multi-step reasoning, conditional branches, and context-dependent thresholds that retrieval alone cannot capture.
Example: A clinical guideline might say "use drug A for patients with condition X unless contraindication Y is present, in which case consider drug B but only if lab value Z is below threshold T." An LLM with RAG retrieval will cite the guideline. An LLM with a domain-specific skill will execute the conditional logic.
AWS packaged 38 of these decision frameworks as agent skills. Each skill is a callable tool that accepts structured inputs and returns a decision based on encoded domain logic. The LLM orchestrates the workflow, but the skill handles the deterministic reasoning.
Architecture: Skills as Executable Decision Logic
Agent skills sit between the LLM and the knowledge base. The LLM decides when to invoke a skill. The skill executes the framework logic and returns a structured result. The LLM uses that result to continue the conversation or trigger the next action.
Skill invocation flow:
- User query arrives (e.g., "What treatment protocol applies for this patient?")
- LLM analyzes query, identifies relevant domain (oncology, cardiology, etc.)
- LLM calls the appropriate skill with structured parameters (patient data, lab values, contraindications)
- Skill executes decision tree or scoring algorithm
- Skill returns structured result (recommended protocol, risk score, next steps)
- LLM formats the result into natural language response
The skill itself is deterministic code. It does not hallucinate. It does not misinterpret thresholds. It applies the framework exactly as encoded.
Skill implementation shape:
class ClinicalGuidelineSkill:
def __init__(self, guideline_version: str):
self.guideline = load_guideline(guideline_version)
self.decision_tree = parse_decision_tree(self.guideline)
def evaluate(self, patient_data: dict) -> dict:
# Deterministic traversal of decision tree
node = self.decision_tree.root
while not node.is_leaf():
condition = node.condition
if self._evaluate_condition(condition, patient_data):
node = node.left_child
else:
node = node.right_child
return {
"recommendation": node.recommendation,
"evidence_level": node.evidence_level,
"contraindications_checked": node.contraindications,
"guideline_version": self.guideline.version
}
def _evaluate_condition(self, condition: str, data: dict) -> bool:
# Parse and evaluate condition against patient data
# Handles thresholds, ranges, categorical values
pass
The LLM does not need to parse the guideline. It calls the skill and gets a structured answer.
Evaluation Methodology: 410 Prompts Across 11 Domains
AWS built a 410-prompt test suite covering 11 healthcare domains: oncology, cardiology, infectious disease, radiology, pharmacy, clinical trials, medical coding, prior authorization, care coordination, population health, and regulatory compliance.
Each prompt tests a specific reasoning scenario. The evaluation compares three configurations:
- Baseline LLM with no skills
- LLM with RAG retrieval only
- LLM with domain-specific skills
Human evaluators scored responses on correctness, completeness, and adherence to guidelines. The skill-enabled configuration won 70-86% of head-to-head comparisons against the baseline.
Evaluation architecture:
| Component | Purpose | Implementation |
|---|---|---|
| Prompt suite | 410 domain-specific scenarios | Curated by clinical experts, versioned with guideline updates |
| Baseline LLM | Control group | Claude 3.5 Sonnet or GPT-4 with no tools |
| RAG LLM | Retrieval-only comparison | Same LLM + vector search over guideline corpus |
| Skill LLM | Full agentic configuration | Same LLM + 38 callable skills |
| Human evaluators | Ground truth scoring | Clinicians and domain experts, blinded to configuration |
| Win rate metric | Primary outcome | Percentage of scenarios where skill LLM outperforms baseline |
The 70-86% win rate is not uniform across domains. Oncology and cardiology showed the highest improvement (82-86%) because decision trees are complex and thresholds are precise. Medical coding showed lower improvement (70-74%) because it is more lookup-heavy and less logic-heavy.
Versioning and Deployment: When Guidelines Change
Healthcare guidelines update quarterly or annually. A skill that encodes the 2025 version of a cardiology guideline becomes outdated when the 2026 version publishes. The deployment shape must handle versioning, rollback, and parallel execution of multiple guideline versions.
Versioning strategy:
- Each skill is versioned independently (e.g.,
oncology-nccn-v2026.1) - Skills declare their guideline version in metadata
- Orchestration layer routes queries to the correct skill version based on patient context or user preference
- Old versions remain callable for historical analysis or regulatory audit
Deployment shape:
- Skill repository: Git repo with one directory per domain
- CI/CD pipeline: Automated tests against known-good scenarios
- Skill registry: Central catalog of available skills with version metadata
- Orchestration layer: Agent framework (LangChain, LlamaIndex, or custom) that maps queries to skills
- Observability: Logs every skill invocation with input parameters, output, and guideline version
When a guideline updates, the skill developer updates the decision tree, increments the version, and deploys. The orchestration layer can run A/B tests between old and new versions before full rollout.
Boundaries: LLM Reasoning vs. Skill Logic
The LLM handles:
- Query understanding and intent classification
- Skill selection and parameter extraction
- Result interpretation and natural language generation
- Multi-turn conversation and clarification questions
The skill handles:
- Deterministic decision tree traversal
- Threshold evaluation and conditional logic
- Structured output generation
- Guideline version tracking
The boundary is critical. If the LLM tries to execute the decision logic itself, it will hallucinate thresholds or misinterpret conditions. If the skill tries to handle natural language understanding, it will fail on ambiguous queries.
Failure modes when boundaries blur:
- LLM applies guideline logic directly: Hallucinates thresholds, misses contraindications, invents steps
- Skill handles query parsing: Brittle to phrasing variations, requires exact parameter names
- LLM overrides skill output: Introduces bias or incorrect reasoning after skill returns correct answer
- Skill tries to explain reasoning: Generates generic text instead of citing specific guideline sections
The orchestration layer enforces the boundary. The LLM cannot bypass a skill. The skill cannot generate free-form text.
Observability: Logging Skill Invocations
Every skill call must be logged for audit, debugging, and compliance. Healthcare regulations require traceability from recommendation back to guideline version and input parameters.
Minimum logging schema:
{
"timestamp": "2026-09-16T14:32:01Z",
"session_id": "abc123",
"skill_name": "oncology-nccn",
"skill_version": "v2026.1",
"input_parameters": {
"patient_age": 62,
"tumor_stage": "IIIa",
"biomarker_status": "PDL1_positive",
"contraindications": ["renal_impairment"]
},
"output": {
"recommendation": "pembrolizumab_monotherapy",
"evidence_level": "1A",
"contraindications_checked": ["renal_impairment", "autoimmune_disease"],
"guideline_section": "NSCLC-3.2.1"
},
"execution_time_ms": 45,
"llm_model": "claude-3.5-sonnet",
"user_id": "clinician_456"
}
This log enables:
- Audit trail for regulatory compliance
- Debugging when recommendations seem incorrect
- Performance monitoring (execution time, error rate)
- A/B testing between skill versions
- Training data for improving skill logic
Security Boundaries: Skills Access Patient Data
Skills need access to patient data to execute decision logic. This creates a security boundary. The skill must not leak data. The LLM must not cache sensitive parameters. The orchestration layer must enforce access control.
Security controls:
- Skills run in isolated execution environments (Lambda functions, containers)
- Patient data is passed as ephemeral parameters, not stored in skill state
- LLM prompt history is scrubbed of PHI before logging
- Access control enforced at orchestration layer (RBAC, ABAC)
- Skills cannot make outbound network calls except to approved APIs
The open-source release of these skills means you can inspect the code. You can verify that a skill does not exfiltrate data. You can audit the decision logic. This is critical for regulated domains where black-box AI is unacceptable.
Transferable Patterns: Beyond Healthcare
The same architecture applies to:
- Financial compliance: Encode underwriting rules, KYC checks, and regulatory frameworks as skills
- Legal reasoning: Package case law analysis, contract review logic, and regulatory interpretation as tools
- Supply chain: Codify procurement policies, vendor selection criteria, and risk assessment frameworks
- HR and benefits: Automate eligibility determination, policy application, and compliance checks
Any domain where citation does not equal correct application benefits from this pattern. The skill encodes the decision logic. The LLM orchestrates the workflow.
Technical Verdict
Use domain-specific agent skills when:
- Decision frameworks involve multi-step conditional logic
- Thresholds and criteria are precise and change over time
- Regulatory compliance requires audit trails and version tracking
- Citation accuracy is high but application accuracy is low
- You need deterministic reasoning mixed with natural language interaction
Avoid this pattern when:
- The domain is simple lookup (RAG retrieval is sufficient)
- Decision logic changes too frequently to maintain skills (daily or hourly updates)
- You lack domain experts to encode and validate skill logic
- The LLM's reasoning is already correct without tools (measure first)
- Compliance does not require explainability or audit trails
The 38 open-source skills provide a reference implementation. You can fork them, adapt the decision trees to your domain, and deploy them in your orchestration framework. The evaluation methodology (410-prompt suite, human scoring, win rate metric) is reusable for testing your own skills.
The gap between citing guidelines and applying them correctly is measurable. Skills close that gap by moving deterministic logic out of the LLM and into executable tools. The orchestration layer enforces boundaries. Observability ensures traceability. Versioning handles guideline updates. This is the plumbing required to deploy agentic AI in regulated domains.
Top comments (0)