Document extraction APIs sit at a critical boundary in agent workflows. An agent that processes invoices, analyzes contracts, or automates form submissions needs structured JSON from unstructured PDFs. When the extraction fails or returns malformed data, the entire pipeline stalls. Midship (YC S24) launched with 121 HN points and a live demo showing PDF-to-JSON extraction, but the interesting plumbing question is not whether it works. It is how agents handle schema drift, validation failures, and confidence thresholds when documents do not match expectations.
The Tool Boundary Problem
Agents call tools. Tools expect typed inputs. PDFs are chaos: scanned images with OCR noise, layout variations between vendors, handwritten annotations, multi-column tables that break parsing logic. The extraction API becomes a translation layer, and someone has to own the contract.
Three handoff points where things break:
- Schema negotiation: Agent defines expected fields (invoice_number, line_items, total). Extraction returns what it finds. Mismatch causes downstream tool calls to fail.
- Validation layer: Who checks that extracted data meets constraints? If the API returns a date string but the agent's database tool expects ISO 8601, the error surfaces three steps later.
- Confidence thresholds: OCR on a blurry scan might return "Inv# 12O45" (letter O instead of zero). Does the agent retry, escalate to human review, or proceed with dirty data?
Most extraction APIs return JSON and call it done. The agent is left holding the bag when field names drift, values are malformed, or confidence scores are missing.
How Midship Positions the Plumbing
Midship combines OCR with LLMs to extract specific fields and tables. The LLM layer corrects OCR mistakes and normalizes field names (understanding that "Inv#" and "Invoice Number" are equivalent). This is useful, but the architecture question is where validation happens.
Likely flow:
- Agent uploads PDF via API
- OCR extracts raw text and layout
- LLM maps text to user-defined schema
- API returns JSON with extracted fields
- Agent validates and passes to next tool
The gap: what happens when step 3 produces low-confidence results or step 5 fails validation? Does the API expose confidence scores per field? Can the agent request a retry with a refined prompt? Is there a webhook for async processing when documents are large?
Schema Drift and Version Control
Document layouts change. A vendor updates their invoice template. A contract form adds a new section. The agent's schema is now stale.
Strategies for handling drift:
| Approach | Trade-off | When to Use |
|---|---|---|
| Strict schema validation | Fails fast, requires manual schema updates | High-value workflows where bad data is worse than downtime |
| Flexible extraction with confidence scores | Proceeds with partial data, logs low-confidence fields | Batch processing where human review happens later |
| Schema versioning with fallback | Tries new schema, falls back to old on failure | Gradual migration across document sources |
| LLM-driven schema inference | Adapts to new layouts automatically, unpredictable | Exploratory workflows, low compliance requirements |
Midship's LLM layer likely helps with minor variations (different field labels, reordered sections), but agents still need a way to detect when extraction quality degrades. Observability hooks matter: per-field confidence, extraction latency, retry counts.
Error Recovery in Multi-Step Pipelines
An agent processing 500 invoices hits a malformed PDF at document 237. What happens?
Recovery patterns:
- Fail the batch: Stop processing, alert operator. Simple but wasteful.
- Skip and log: Continue with remaining documents, flag failures for review. Requires idempotent downstream tools.
- Retry with fallback: Attempt extraction again with different OCR settings or prompt. Adds latency and cost.
- Human-in-the-loop escalation: Route low-confidence extractions to review queue. Breaks full automation but prevents bad data propagation.
The API needs to surface enough metadata for the agent to make this decision. A boolean success flag is not enough. The agent needs per-field confidence, OCR quality indicators, and extraction duration to implement smart retry logic.
Rate Limiting and Cost Control
Agents loop. An invoice processing agent might extract 10,000 documents per month. If each extraction costs $0.10 and the agent retries on failures, costs spiral.
Cost control mechanisms:
- Client-side rate limiting: Agent enforces max requests per minute. Prevents runaway loops but requires state management.
- API-side quotas: Hard limits per API key. Protects the service but causes agent failures when hit.
- Tiered pricing with burst allowances: Pay-as-you-go with monthly caps. Balances flexibility and predictability.
- Batch endpoints: Submit 100 documents, get async results. Reduces per-request overhead.
Midship charges volume-based pricing for the API. The question is whether the pricing model includes retry budgets or penalizes agents that implement quality checks.
Observability Hooks for Debugging
When extraction fails in a multi-step workflow, the agent needs to know why. Was it OCR quality? Schema mismatch? LLM hallucination?
Useful telemetry:
- Per-field extraction confidence (0-1 score)
- OCR quality metrics (contrast, resolution, text density)
- LLM prompt and response for each extraction
- Processing duration breakdown (OCR time, LLM time, validation time)
- Retry attempts and outcomes
Without this, debugging becomes guesswork. The agent logs "extraction failed" and the operator manually inspects the PDF, re-runs the extraction, and hopes for different results.
Code Example: Agent Validation Layer
Here is how an agent might validate extraction results before passing to downstream tools:
import requests
from pydantic import BaseModel, ValidationError
from typing import Optional
class InvoiceSchema(BaseModel):
invoice_number: str
date: str # ISO 8601
total: float
line_items: list[dict]
confidence: Optional[float] = None
def extract_and_validate(pdf_path: str, api_key: str):
# Call extraction API
with open(pdf_path, 'rb') as f:
response = requests.post(
'https://api.midship.ai/extract',
headers={'Authorization': f'Bearer {api_key}'},
files={'document': f},
json={'schema': InvoiceSchema.schema()}
)
result = response.json()
# Validate against expected schema
try:
invoice = InvoiceSchema(**result['data'])
except ValidationError as e:
# Log validation failure, decide on retry or escalation
if result.get('confidence', 1.0) < 0.7:
return retry_with_manual_review(pdf_path)
else:
raise e
# Check confidence threshold
if invoice.confidence and invoice.confidence < 0.85:
log_low_confidence(invoice)
# Proceed but flag for audit
return invoice
This pattern puts validation in the agent, not the API. The agent owns the contract and decides how to handle failures.
Technical Verdict
Use Midship-style extraction APIs when:
- You have high-volume document workflows where manual extraction is a bottleneck
- Document layouts are semi-structured (invoices, forms, contracts) with predictable fields
- You can implement validation and retry logic in your agent orchestration layer
- You need field-level extraction, not just full-text OCR
Avoid or supplement when:
- Documents are highly variable (handwritten notes, artistic layouts, multi-language)
- You need guaranteed accuracy for compliance (financial reporting, legal discovery)
- Your agent cannot handle partial failures gracefully
- Cost per extraction exceeds the value of automation (low-volume, high-touch workflows)
The real test is not whether the API extracts data. It is whether your agent can recover when extraction is ambiguous, validate results before acting, and surface failures without breaking the pipeline. Document extraction is a tool boundary problem, and the API is only half the solution. The other half is orchestration logic that handles the messy reality of PDFs in production.
Top comments (0)