Getting structured output from LLMs is a notoriously tricky problem to solve given their non-deterministic nature. In our previous post, we explored using Pydantic and Instructor to ensure reliable JSON output from LLMs. Now, in Part 2, we dive deeper into Pydantic's validation capabilities. By setting up max_retries, we give our model multiple chances to produce accurate responses. If validations fail or data is improperly structured, Instructor handles retries until we get the desired result. This powerful feature is seamlessly managed with the Instructor patch for our OpenAI client.
Implementing Validation with Pydantic
Pydantic gives us a handy way to check if the responses we get are correct. It looks at the structure and types of data we expect and makes sure everything matches up. Plus, it goes beyond that by letting us add extra checks. So, if the LLM's response doesn't quite fit the bill, we can ask it to try again using maxRetries. This helps the model learn and get better at giving us the right information.
Let's go back to our example where we wanted to pull out important details from patient data. Here, we'll see how Pydantic, along with some smart validation tricks, can make sure we always get reliable and useful answers.
Implementing Detailed Validations
We can add specific checks to our response model. Since our output is organized like a structured form, adding checks is as easy as setting rules to follow.
Let's simplify this with an example about age validation. We want to ensure that the age entered is reasonable, so we'll make a rule that the age must be under 200. If it's not, we'll raise a ValueError error saying, "The age should be less than 200."
def validate_age(age: int) -> Optional[int]:
if age < 200:
return age
else:
raise ValueError("Age must be less than 200")
Now, let's include this rule in our class definition using Pydantic.
from pydantic import BaseModel, validator
from typing_extensions import Annotated, Literal
class PatientInfo(BaseModel):
sex: Literal['M', 'F']
age: Annotated[int, validator(validate_age)]
geographical_region: str
This ensures that the age given meets our standards after it's added to the model.
Additionally, the max_retries setting is crucial. It's similar to giving our model multiple attempts to provide the correct response. If our validations fail or our data isn't structured correctly, the instructor can send the error message back to the model several times until we receive the desired answer. Thankfully, managing this feature is simplified with the instructor patch for our OpenAI client.
Using LLM Validators:
Instructor offers a handy way to use Language Model Models (LLMs) to check if certain fields or logic are correct. Let's explain this with a simple example. We want to make sure that the locations mentioned are actual body parts.
from instructor import llm_validator
from pydantic import BeforeValidator
class Symptoms(BaseModel):
description: str
pain_type: str
locations: List[
str,
]
intensity: int
location_precision: int
pace: int
class MedicalHistory(BaseModel):
pathology: Annotated[
str,
BeforeValidator(
llm_validator(
"must be a valid disease name", openai_client=instructor_openai_client
)
),
]
symptoms: Symptoms
increase_with_exertion: bool
alleviate_with_rest
class PatientData(BaseModel):
patient_info: PatientInfo
medical_history: MedicalHistory
risk_factors: RiskFactors
differential_diagnosis: List[DifferentialDiagnosis]
This means we're using a smart tool to double-check if the disease names we're given are accurate. It's like having a helpful assistant who makes sure everything makes sense before we move forward.
Dynamic Validation Based on Runtime Data
Making Validation Flexible with Real-Time Data
Sometimes, making sure we grab all the right information from our data can be tricky, especially when the rules for what's right change depending on what we're looking at. This happens a lot when we're pulling out details from data used in programs, like APIs, where we need to be flexible to handle different kinds of requests.
Let's imagine a situation where we're pulling out details from a JSON file. We want to make sure we grab everything we need, and the rules for what we need might change depending on what's in the JSON. This was a problem I faced when I was trying to gather detailed information about medical diagnoses from an AI assistant.
Imagine we want to improve our diagnosis system. We don't just want the name of the diagnosis and how likely it is. We also want a clear explanation of what the diagnosis means and what treatments are possible.
from pydantic import create_model, Field, List
class Treatment(BaseModel):
treatment_name: str
treatment_procedure: str
comments: str
class Diagnosis(BaseModel):
diagnosis_name: str
diagnosis_description: str = Field(
...,
description="A professional but human way to describe the diagnosis and its effect."
)
probability: int
possible_treatment: List[Treatment]
description: str = Field(
...,
description="A Professional but sympathetic description of its treatment and its validity regarding other treatments according to you.",
)
def update_models_description(possible_diagnosis: List[DifferentialDiagnosis]):
# Create a new model with the updated description
diagnosis = ",".join([str((diag.disease_name, diag.probability)) for diag in possible_diagnosis])
print(diagnosis)
updated_model = create_model(
"DiagnosisAndTreatments",
disease=(
List[Diagnosis],
Field(
...,
description=f"Ensure that all the following possible information is present for all the following diagnoses: {diagnosis}"
),
),
)
return updated_model
completion = instructor_openai_client.chat.completions.create(
model="gpt-3.5-turbo",
messages=[
{
"role": "user",
"content": f"You are an AI doctor assistance. Please convert the following patient data to extrapolate the following data ${medical_info}",
}
],
max_retries=3,
response_model=update_models_description(medical_data.differential_diagnosis),
)
print(type(completion))
print(json.dumps(completion.model_dump(), indent=1))
<class '__main__.DiagnosisAndTreatments'>
{
"disease": [
{
"diagnosis_name": "Unstable angina",
"diagnosis_description": "Unstable angina is a condition where the blood flow to the heart is suddenly blocked, leading to chest pain and discomfort. This condition can be life-threatening and requires immediate medical attention.",
"probability": 26,
"possible_treatment": [
{
"treatment_name": "Coronary angiography",
"treatment_procedure": "This procedure uses contrast dye and X-rays to see inside the arteries of the heart. It can help identify blockages and other problems in the blood vessels around the heart.",
"comments": "Coronary angiography is a common and effective procedure to diagnose and treat heart conditions."
},
{
"treatment_name": "Medications",
"treatment_procedure": "Medications such as nitroglycerin and beta-blockers may be prescribed to manage symptoms and prevent future episodes of unstable angina.",
"comments": "Medications play a crucial role in controlling symptoms and improving the prognosis of unstable angina patients."
}
],
"description": "Unstable angina is a serious condition that requires prompt diagnosis and treatment to prevent heart damage and complications."
},
{
"diagnosis_name": "Stable angina",
...
},
]
}
In this setup, we're cleverly building a Pydantic model on the fly, depending on the data we're dealing with about possible diagnoses. This means our checks are customized to fit each diagnosis – making sure we cover all the bases.
Plus, by adding Pydantic checks to our model, we make sure we don't miss anything important for each diagnosis. Whether it's the name, description, likelihood, or treatment options, we're double-checking everything to make sure our data is complete and accurate.
Top comments (0)