DEV Community: Любовь Авдеева

Parsing financial PDFs without chasing 100% accuracy

Любовь Авдеева — Tue, 01 Sep 2026 07:47:02 +0000

We needed a reliable way to pull structure out of financial PDFs: text, tables, page numbers, and bounding boxes. Not “roughly understand the document” — a checkable artifact we could index and query later.

This is not a tutorial on Docling. It’s a short write-up of four problems we actually hit, the options we considered, and why we chose the path we did.

The goal

Input: a PDF

Output: a structured object with text chunks, tables, multi-page table groups, and provenance (page_no, bbox) for every element. Plus a strict schema so nothing silently drifts downstream.

Stack for this stage: Docling for layout-aware parsing, Pydantic for the schema, tests from day one.

Problem 1. One form type “lost” its tables

We fed in a dense regulatory form. The parser found one table. The other three were flattened into text items — stock name, date, transaction code, quantity — each as a separate TextItem.

Options we considered:

Custom coordinate clustering

Rebuild rows and columns from X/Y positions ourselves.

Pros: we might recover all four tables.

Cons: several days of work that mostly reimplements what a layout parser already tries to do — for one awkward form type.

Ask an LLM to reconstruct the table

Pros: sometimes it understands context.

Cons: non-deterministic, slow, expensive. Bad fit for a base layer that must be repeatable.

Change the test document

Treat this form as a known edge case, switch to a report where tables have clear borders, and move on.

We took the third option. Not because those forms don’t matter, but because polishing one edge case for a week would block proving that the rest of the pipeline works on normal documents.

Takeaway: prove the pipeline on standard data first. Special forms can wait.

Problem 2. Bounding-box validation was a silent `pass`

In the schema, if left > right or top > bottom, the code just did pass. Classic “I’ll fix it later.” In production that means garbage coordinates flow downstream without anyone noticing.

Options:

Leave it — fast, unsafe
Raise ValueError and fail the whole document — too aggressive; one bad block kills the file
Normalize: swap the bounds when they’re inverted

We normalized. If a table is truly upside-down in a weird way, we might miss it. But we don’t drop the document, and we keep a clear invariant: left < right, top < bottom.

Takeaway: in a strict schema, never leave a silent pass. Either fix the data with an explicit rule or fail loudly.

Problem 3. Money normalization was wrong

Values like ($1,250) were supposed to become negative numbers. We got things like -$1250 — the dollar sign stayed. Tests that expected a clean number failed.

Root cause: we stripped parentheses first, then commas, and never touched the currency symbol. Order of operations mattered more than “what looks logical.”

Working approach:

Strip everything that isn’t a digit or a dot
Look at the original string to decide if parentheses mean “negative”
If cleaning leaves an empty string, don’t invent a number — keep the original or mark it

Takeaway: normalization is a contract. Define what counts as a number, what happens to junk, and in which order rules apply.

Problem 4. Table-grouping heuristic was too soft

We wanted to group multi-page tables: nearby pages, similar headers, same section. The first version used a soft check — “if a piece of one header appears inside the other.” In practice "Net" happily merged with "Net Income".

Options:

Soft substring match — many merges, including wrong ones
Strict header equality — fewer groups, each one explainable
Similarity score (e.g. Jaccard) with a threshold — flexible, but harder to tune and debug

For the first version we chose strict equality. Tables marked “continued” might not merge. That’s fine. We preferred a controlled miss over silent false merges.

Takeaway: for an MVP, a strict heuristic with known limits beats a clever one that fails quietly.

Bonus debate: “another model found 61 tables, we found 184”

Someone ran the same files through a different stack and got a smaller table count. The temptation was to add filters until our number looked “nicer.”

We didn’t. Different tools use different definitions of “table.” Our parser often treats every visual grid as a table. Some of that is noise; some is useful context. Trimming everything for a pretty metric means losing data and weakening auditability.

We documented the behavior instead: this is what the parser returns; these are the limits; the system has to live with that volume.

What we locked in

By the end of this stage we had:

Deterministic parse output with provenance (page, bbox, source ref)
A strict schema with no silent holes
A conservative table-grouping rule for the MVP
Tests for schema, money normalization, and grouping
An explicit list of known limits (collapsed multi-level headers, messy complex tables, occasional leftover currency symbols)

Principles we kept:

Don’t polish the parser forever for one form type
No silent pass in data contracts
In normalization, clean first, then apply meaning
Prefer a strict heuristic over a “smart” one that lies
Don’t chase someone else’s table count — define what you mean by a table and make it checkable

Next step is indexing and comparing chunking strategies (text-only vs hybrid vs table-aware). None of that matters if the PDF parse underneath is sand.

If you’ve parsed similar report-style PDFs: where did switching test data save you time, and where did you still have to write custom logic on top of a layout parser?

How I Built a Reliable LLM Pipeline for Ad Creative Evaluation (with Strict Pydantic Contracts)

Любовь Авдеева — Tue, 25 Aug 2026 14:02:13 +0000

Art directors routinely spend 20–40 minutes per creative just checking brand guidelines, mandatory elements, and forbidden techniques.

I wanted to automate the first-pass review — without turning it into another unreliable ChatGPT wrapper.

So I built CreativeAudit: a production-oriented pipeline that evaluates ad creatives against a brief and returns a clear PASS / NEEDS_REVISION / FAIL verdict.

strelok25-dev / llm-creative-evaluator

Automated LLM pipeline for creative evaluation with Pydantic validation

CreativeAudit

Automated ad creative evaluation against a brief using a local LLM.

CreativeAudit takes a brief + creatives, builds a precise prompt, gets a structured evaluation from a language model, strictly validates the response, and returns a clear verdict (PASS / NEEDS_REVISION / FAIL), weighted score, and explanation.

This is not "just another AI chat". It is a production-oriented first-pass review pipeline: the machine catches routine and critical violations so humans only need to look at the borderline cases.

Interface

Campaign summary metrics, creative cards with verdict, detailed scores and model explanation. Smart-input mode converts free-form text into structured JSON with human review before evaluation starts.

The Problem

Manual first-pass creative review is slow, expensive and inconsistent:

An art director spends 20–40 minutes per creative checking mandatory elements, forbidden techniques and tone of voice.
A campaign with 10–30 creatives turns into hours of work that does not scale.
…

View on GitHub

The Real Problem with "Just Ask the LLM"

A plain chat with an LLM has several fatal flaws for this use case:

Responses are inconsistent in format
The model frequently forgets rules or hallucinates structure
You get free-form text instead of machine-readable output
Client data goes to an external cloud

For any real workflow this is unacceptable.

Design Goals

I set a few hard requirements:

Strict output contracts — invalid responses must never become scores
Local inference — no client data leaves the machine
Prompts as code — versioned and editable independently of the app
Binary compliance for critical rules (a creative either violates the brand book or it doesn’t)
Human-in-the-loop on free-form input

Architecture Overview

Brief + Creatives → Jinja2 Prompt → Local LLM (Ollama)
                                      ↓
                               Pydantic Validation
                                      ↓
                          Score + Verdict + Feedback

Architecture diagram showing the flow from Brief and Creatives through Jinja2 Prompt and Ollama, followed by Pydantic Validation, resulting in Score, Verdict, and Feedback

Key components:

app/schemas.py — strict Pydantic v2 contracts

prompts/*.j2 — prompts treated as versioned code

app/main.py — orchestration (prompt assembly, LLM call, validation, scoring)

demo/streamlit_app.py — thin UI layer

The business logic is completely separated from the interface. You can call it from Streamlit, CLI, or any other service.

Key Engineering Decisions

Pydantic as a Hard Contract

The model is non-deterministic.
I treat the Pydantic schema as a hard boundary: if the response doesn’t match the schema, it is rejected as an error — it never becomes a fake score.

Binary Scale for Critical Compliance

For brand-book violations I use a binary 0 or 10 score.
There is no “slightly violated”. This removes a lot of model subjectivity on the highest-risk criterion.

Prompts as Code (Jinja2)

Prompts live in separate .j2 files.
This makes them versionable, reviewable, and easy to A/B test without touching application code.

Local Inference with Ollama

Everything runs locally.
Switching between qwen2.5:7b, qwen2.5:14b or llama3.2 is a single config change.

Smart Input Mode (Human-in-the-Loop)

Managers rarely provide clean JSON.
They paste chat fragments and rough descriptions.
So the pipeline first uses an LLM to extract structured data, shows the result to the user for correction, and only then runs the evaluation.

Scoring Model

Each creative receives three scores:

brand_alignment (1–10)

constraint_compliance (0 or 10)

message_clarity (1–10)

Final score is a weighted combination:

total = brand × 0.4 + compliance × 0.3 + clarity × 0.3

The verdict is derived from the total score and critical failures.

Testing Strategy

I wrote 38 unit tests.
The external LLM is fully mocked, so tests are deterministic and run in under 2 seconds.
They cover schema boundaries, malformed responses, connection failures, and scoring logic.

Results & Lessons

The biggest wins:

First-pass review time dropped from tens of minutes to seconds

Output became consistent and machine-readable

Critical brand violations are much harder to miss

The hardest part wasn’t the LLM call — it was designing the contracts and failure modes so the system stays reliable when the model behaves badly.

What’s Next

Multimodal support (evaluate the actual layouts, not only text)

Model benchmarks (accuracy vs speed)

REST API for integration into existing workflows

Evaluation history and campaign analytics

If you’re building production LLM pipelines, I’d love to hear how you handle structured output and hallucination control.

How do you currently deal with unreliable LLM responses in your projects?

AI Engineer. No fluff, just working products.

Любовь Авдеева — Mon, 24 Aug 2026 11:08:37 +0000