How I Built a Reliable LLM Pipeline for Ad Creative Evaluation (with Strict Pydantic Contracts)

Art directors routinely spend 20–40 minutes per creative just checking brand guidelines, mandatory elements, and forbidden techniques.

I wanted to automate the first-pass review — without turning it into another unreliable ChatGPT wrapper.

So I built CreativeAudit: a production-oriented pipeline that evaluates ad creatives against a brief and returns a clear PASS / NEEDS_REVISION / FAIL verdict.

strelok25-dev / llm-creative-evaluator

Automated LLM pipeline for creative evaluation with Pydantic validation

CreativeAudit

Automated ad creative evaluation against a brief using a local LLM.

CreativeAudit takes a brief + creatives, builds a precise prompt, gets a structured evaluation from a language model, strictly validates the response, and returns a clear verdict (PASS / NEEDS_REVISION / FAIL), weighted score, and explanation.

This is not "just another AI chat". It is a production-oriented first-pass review pipeline: the machine catches routine and critical violations so humans only need to look at the borderline cases.

---

Interface

Campaign summary metrics, creative cards with verdict, detailed scores and model explanation. Smart-input mode converts free-form text into structured JSON with human review before evaluation starts.

---

The Problem

Manual first-pass creative review is slow, expensive and inconsistent:

• An art director spends 20–40 minutes per creative checking mandatory elements, forbidden techniques and tone of voice.
• A campaign with 10–30 creatives turns into hours of work that does not scale.
• …

View on GitHub

The Real Problem with "Just Ask the LLM"

A plain chat with an LLM has several fatal flaws for this use case:

• Responses are inconsistent in format
• The model frequently forgets rules or hallucinates structure
• You get free-form text instead of machine-readable output
• Client data goes to an external cloud

For any real workflow this is unacceptable.

Design Goals

I set a few hard requirements:

1. Strict output contracts — invalid responses must never become scores
2. Local inference — no client data leaves the machine
3. Prompts as code — versioned and editable independently of the app
4. Binary compliance for critical rules (a creative either violates the brand book or it doesn’t)
5. Human-in-the-loop on free-form input

Architecture Overview

Brief + Creatives → Jinja2 Prompt → Local LLM (Ollama)
                                      ↓
                               Pydantic Validation
                                      ↓
                          Score + Verdict + Feedback

Architecture diagram showing the flow from Brief and Creatives through Jinja2 Prompt and Ollama, followed by Pydantic Validation, resulting in Score, Verdict, and Feedback

Key components:

app/schemas.py — strict Pydantic v2 contracts

prompts/*.j2 — prompts treated as versioned code

app/main.py — orchestration (prompt assembly, LLM call, validation, scoring)

demo/streamlit_app.py — thin UI layer

The business logic is completely separated from the interface. You can call it from Streamlit, CLI, or any other service.

Key Engineering Decisions

1. Pydantic as a Hard Contract

The model is non-deterministic.
I treat the Pydantic schema as a hard boundary: if the response doesn’t match the schema, it is rejected as an error — it never becomes a fake score.

1. Binary Scale for Critical Compliance

For brand-book violations I use a binary 0 or 10 score.
There is no “slightly violated”. This removes a lot of model subjectivity on the highest-risk criterion.

1. Prompts as Code (Jinja2)

Prompts live in separate .j2 files.
This makes them versionable, reviewable, and easy to A/B test without touching application code.

1. Local Inference with Ollama

Everything runs locally.
Switching between qwen2.5:7b, qwen2.5:14b or llama3.2 is a single config change.

1. Smart Input Mode (Human-in-the-Loop)

Managers rarely provide clean JSON.
They paste chat fragments and rough descriptions.
So the pipeline first uses an LLM to extract structured data, shows the result to the user for correction, and only then runs the evaluation.

Scoring Model

Each creative receives three scores:

brand_alignment (1–10)

constraint_compliance (0 or 10)

message_clarity (1–10)

Final score is a weighted combination:

total = brand × 0.4 + compliance × 0.3 + clarity × 0.3

The verdict is derived from the total score and critical failures.

Testing Strategy

I wrote 38 unit tests.
The external LLM is fully mocked, so tests are deterministic and run in under 2 seconds.
They cover schema boundaries, malformed responses, connection failures, and scoring logic.

Results & Lessons

The biggest wins:

First-pass review time dropped from tens of minutes to seconds

Output became consistent and machine-readable

Critical brand violations are much harder to miss

The hardest part wasn’t the LLM call — it was designing the contracts and failure modes so the system stays reliable when the model behaves badly.

What’s Next

Multimodal support (evaluate the actual layouts, not only text)

Model benchmarks (accuracy vs speed)

REST API for integration into existing workflows

Evaluation history and campaign analytics

If you’re building production LLM pipelines, I’d love to hear how you handle structured output and hallucination control.

How do you currently deal with unreliable LLM responses in your projects?

Comments

[Любовь Авдеева]

If you’re building production LLM pipelines, I’d love to hear how you handle structured output and hallucination control.

How do you currently deal with unreliable LLM responses in your projects?