That Pydantic Validation Error That Dropped Your Chunk

Your extraction pipeline fails silently. One chunk out of fifty gets dropped. You notice hours later when a user asks about data that should be there.

Here's what happened:

graphiti-service: Error ingesting chunk <doc>#chunk-22: 
1 validation error for ExtractedEntitiesFreeform

That's it. No field name. No type mismatch. Just "1 validation error." The chunk is gone. The doc is incomplete. Good luck figuring out which entity field broke.

Why This Happens

You're calling Claude Sonnet (or GPT-4o) to extract structured entities from a chunk of text. The prompt says "return valid JSON matching this schema." The LLM returns something that looks right — but isn't.

The common failure modes:

Field name drift:

# Your schema expects:
class ExtractedEntitiesFreeform(BaseModel):
    entities: list[Entity]
    relationships: list[Relationship]

# LLM returns:
{
    "entities": [...],
    "relations": [...]  # Wrong field name
}

Type mismatch:

# Schema says:
class Entity(BaseModel):
    name: str
    confidence: float

# LLM returns:
{
    "name": "Acme Corp",
    "confidence": "high"  # String, not float
}

Nested structure collapse:

# Expected:
{
    "entities": [
        {
            "name": "Acme",
            "properties": {"industry": "tech"}
        }
    ]
}

# Returned:
{
    "entities": [
        {
            "name": "Acme",
            "industry": "tech"  # Flattened
        }
    ]
}

The LLM is trying to be helpful. It's not. It's generating structurally valid JSON that doesn't match your Pydantic model. And Pydantic gives you a one-line error with no context about which field failed.

The Manual Fix

You have two options:

Option 1: Retry with better error messages

from pydantic import ValidationError

def extract_entities(text: str) -> ExtractedEntitiesFreeform | None:
    try:
        response = llm.call(
            model="claude-sonnet-4-20250514",
            messages=[{
                "role": "user",
                "content": f"Extract entities from:\n\n{text}"
            }],
            response_format={"type": "json_object"}
        )

        data = json.loads(response.content)
        return ExtractedEntitiesFreeform(**data)

    except ValidationError as e:
        # This still won't tell you which field
        logger.error(f"Validation failed: {e.errors()}")
        return None

    except json.JSONDecodeError as e:
        logger.error(f"Invalid JSON: {e}")
        return None

The e.errors() gives you a list of errors, but if the LLM returned a completely wrong structure (like nesting entities under a different key), you get a generic "field required" error.

Option 2: Validate before parsing

def validate_llm_response(raw: dict) -> ExtractedEntitiesFreeform | None:
    """Pre-validate and normalize common LLM quirks"""

    # Handle field name drift
    if "relations" in raw and "relationships" not in raw:
        raw["relationships"] = raw.pop("relations")

    # Handle nested structure issues
    if "entities" in raw:
        for entity in raw["entities"]:
            # Normalize confidence scores
            if isinstance(entity.get("confidence"), str):
                try:
                    entity["confidence"] = float(entity["confidence"].replace("%", "")) / 100
                except ValueError:
                    entity["confidence"] = 0.5  # Default

    try:
        return ExtractedEntitiesFreeform(**raw)
    except ValidationError as e:
        logger.error(f"LLM response structure: {json.dumps(raw, indent=2)}")
        logger.error(f"Validation errors: {e.errors()}")
        return None

This works. But you're writing defensive code for every schema change. Every new field is another normalization rule. It's brittle and it doesn't scale.

The Real Problem

You're debugging blind. You have:

1. The input text (chunk 22)
2. The LLM response (truncated error message)
3. The failed validation (no field context)

What you don't have is the exact LLM output that failed. You can't see what Claude actually returned. You can't replay the extraction with a fixed prompt. You're guessing.

Enter TracePilot

One line change:

from tracepilot import TracePilot

tp = TracePilot(api_key="tp_live_YOUR_KEY")

async def extract_entities(text: str):
    await tp.start_trace("entity-extraction")

    response = await tp.wrap_llm_call(
        llm_fn=lambda: client.messages.create(
            model="claude-sonnet-4-20250514",
            max_tokens=1024,
            messages=[{
                "role": "user",
                "content": f"Extract entities from:\n\n{text}"
            }]
        ),
        metadata={"chunk_id": "chunk-22", "doc_id": "doc-123"}
    )

    raw = json.loads(response.content[0].text)

    try:
        entities = ExtractedEntitiesFreeform(**raw)
        await tp.log_success(entities.dict())
        return entities
    except ValidationError as e:
        # TracePilot captures the full LLM output
        await tp.log_failure({
            "error": str(e),
            "raw_response": raw,  # Now you can see it
            "validation_errors": e.errors()
        })
        return None

Now when chunk 22 fails, you open your TracePilot dashboard. You see:

• The exact JSON Claude returned
• Which field failed validation
• The full error trace

Click Fork & Rerun. Edit the prompt to add "use 'relationships' not 'relations'". Replay. The fix takes 30 seconds. No redeployment.

The Hook

You've got 47 more chunks to process. Each one could fail with a different LLM quirk. You could write defensive code for every edge case. Or you could add one import and see exactly what the LLM returned when it broke.

Sound familiar?

Get your free API key — the first 10k traces are on us. Your debugging time is worth more than that.

---

Debugging AI agents shouldn't feel like reading The Matrix.
Join other engineers who are building reliable autonomous workflows in our community: TracePilot Discord

Comments

[Luis Cruzy]

I've encountered similar issues with Pydantic validation errors, and I think the approach of pre-validating and normalizing the LLM response is a good one. However, I'm concerned that the validate_llm_response function might become cumbersome to maintain as the schema changes. Have you considered using a more dynamic approach, such as using a library like jsonschema to validate the response against the expected schema? This could potentially simplify the validation process and make it more scalable. I'd love to hear your thoughts on this approach and whether you've explored it already.