DEV Community: api

H33-74: 74 Bytes of Tamper-Proof Attestation

H33.ai — Wed, 29 Apr 2026 21:43:43 +0000

Originally published at v100.ai

The first and only quantum resistant video platform. ML-KEM-768 + ML-DSA-65 + FALCON-512.

Full article | v100.ai/quantum

V100 by H33.ai

Anthropic just charged a developer $200 by mistake. Here's how flat-rate AI billing works instead.

brian austin — Wed, 29 Apr 2026 21:28:37 +0000

The bug that cost $200

This week, a GitHub issue titled HERMES.md went viral on Hacker News: an Anthropic billing bug silently charged a developer $200 extra — and Anthropic initially refused a refund.

The thread now has 289 comments and 737 upvotes. Most of them are developers sharing their own billing anxiety stories.

Here's a sample of what people are saying:

"I disabled my Claude Code usage because I was terrified of runaway charges."

"Metered billing for AI tools is the new dark pattern."

"I now check my Anthropic dashboard more than I check my bank account."

This is not a bug story. This is a structural problem with how AI is billed.

The metered billing trap

Every major AI provider — OpenAI, Anthropic, Google Gemini — bills by the token. This means:

A runaway agent loop can cost you hundreds overnight
A misunderstood context window multiplies your bill silently
A billing bug, like the HERMES.md case, is nearly impossible to audit after the fact
You check your API dashboard like a gas pump meter, watching the numbers tick up

This is fine for enterprises with finance teams and spend alerts. It is brutal for individual developers, students, and freelancers.

What flat-rate billing actually looks like

Here's the alternative model:

# Call Claude API — SimplyLouie flat-rate
curl https://api.simplylouie.com/v1/messages \
  -H "x-api-key: YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-opus-4-5",
    "max_tokens": 1024,
    "messages": [{"role": "user", "content": "Explain this code to me"}]
  }'

You pay $2/month. That's it. No token counters. No usage dashboards. No surprise charges. No billing bugs that require a GitHub issue and 289 comments to resolve.

The response is identical to calling Anthropic directly — same Claude model, same response format, same API structure.

Why this matters for developers outside the US

The HERMES.md developer got a $200 charge. For a developer in:

Nigeria: that's 2-3 weeks of salary
India: Rs 16,500 — nearly a month's rent in many cities
Philippines: ₱11,200 — over a month's worth of groceries
Indonesia: Rp3.2M — a significant emergency fund hit

Metered billing doesn't just cause anxiety. For developers in emerging markets, a billing bug like HERMES.md is financially catastrophic.

Flat-rate billing at $2/month is not a price feature. It's a safety feature.

The technical argument for flat-rate

Some developers argue that metered billing is more "fair" because heavy users pay more. This is true at scale. But it creates a hidden cost that never shows up in the pricing page: cognitive overhead.

Every time you use a metered AI API, you're making micro-decisions:

Should I include the full file in context, or trim it?
Is this prompt worth the tokens?
Should I stream this response or batch it?
Did that agent loop cost me $0.20 or $2.00?

That cognitive overhead is a tax on your productivity. Flat-rate billing removes it entirely. You just... use the API.

What to do if you got an unexpected Anthropic charge

Go to console.anthropic.com/settings/billing
Export your usage log for the disputed period
Open a support ticket with the exact timestamp of the anomaly
Reference the HERMES.md GitHub issue — Anthropic is now aware and has acknowledged the bug
If refused, dispute via your credit card issuer (this works, metered billing disputes are winnable)

The alternative

If you want Claude API access without metered billing anxiety:

SimplyLouie — $2/month flat rate, Claude API, no usage meters, 7-day free trial.

For developers who can't afford to check their billing dashboard with their morning coffee — this is what we built.

The HERMES.md issue is at github.com/anthropics/claude-code/issues/53262. The Hacker News thread is worth reading — it's one of the most honest conversations about AI billing transparency in months.

Easily benchmark all your app's endpoints at once

Kenneth Mckrola — Wed, 29 Apr 2026 21:09:11 +0000

Most "load tests" in real codebases are a curl pasted into a Slack thread. Someone runs it before a release, eyeballs the latency, and we ship. There's nothing version-controlled, nothing repeatable, and the next person to touch the service has no idea which endpoints are actually fast paths.

benchmarkr is a powerful and easy-to-use CLI and MCP tool that fixes that part of the workflow specifically. The thing I want to talk about in this post is the piece that makes it click: a YAML config that lives in your repo and describes every endpoint you care about, the same way a package.json describes your dependencies.

The config

First, install benchmarkr cli if you haven't:

brew tap mack-overflow/tap
brew install benchmarkr

for Homebrew, or for Debian

echo "deb [trusted=yes] https://apt.fury.io/mack-overflow/ /" \
  | sudo tee /etc/apt/sources.list.d/benchmarkr.list
sudo apt update
sudo apt install benchmarkr

(More installation guides available here)

Next, run benchmarkr endpoints init in your project root and you get a benchmarkr.yaml you can commit:

version: 1

endpoints:
  - name: list-users
    method: GET
    url: ${API_BASE:-http://localhost:8080}/users
    headers:
      Authorization: Bearer ${API_TOKEN}
    defaults:
      concurrency: 10
      duration_seconds: 30

  - name: search-users
    method: GET
    url: ${API_BASE}/users/search
    params:
      q: "test"
      limit: "50"
    defaults:
      concurrency: 5
      duration_seconds: 15

  - name: create-order
    method: POST
    url: ${API_BASE}/orders
    headers:
      Authorization: Bearer ${API_TOKEN}
      Content-Type: application/json
    body:
      sku: "ABC-123"
      quantity: 1
    defaults:
      concurrency: 2
      duration_seconds: 10

A few things to notice:

Env var substitution. ${API_BASE} and ${API_BASE:-default} work the way they do in shell. A sibling .env file is auto-loaded but never overrides what's already in the environment, so the same file works on a laptop, in CI, and in staging.
Defaults travel with the endpoint. create-order runs at concurrency 2 for 10 seconds because that's what makes sense for a write path. list-users runs at concurrency 10. You set this once in the file you already review.
Discovery walks up from CWD. Run the CLI from any subdirectory and it finds the file, like git does.

Running one endpoint

benchmarkr run -e list-users

That's it. Saved defaults apply. Any flag you pass on the command line wins; headers and params are merged. So when you're poking at production specifically, you can do:

benchmarkr run -e list-users \
  --header "X-Trace: debug-2026-04-28" \
  --concurrency 50

…without editing the committed file.

Running all of them

This is where the YAML pays for itself. Because every endpoint is named and self-describing, you can hand the entire file to the CLI in one shot:

benchmarkr run --all

That walks every endpoint in benchmarkr.yaml in succession, applying each endpoint's saved defaults (concurrency, duration, headers, body — the whole config). Between runs you get a [i/N] <name> header so it's obvious where you are; live p50/p95/p99 streams in for the active endpoint and a final summary prints when it finishes. --all is mutually exclusive with --url and --endpoint, and any flags you do pass (e.g. --store, --json, --rate-limit) apply to every run in the sweep.

For CI, this collapses the workflow step to one line:

# .github/workflows/perf.yml
- name: Benchmark every endpoint
  env:
    API_BASE: https://api.staging.example.com
    API_TOKEN: ${{ secrets.STAGING_API_TOKEN }}
    BENCH_CLOUD_TOKEN: ${{ secrets.BENCHMARKR_TOKEN }}
  run: benchmarkr run --all --store --json > perf-results.json

--json with --all emits an array — one entry per endpoint, with the same result shape as a single run — so you can pipe it straight into a regression check or upload it as a CI artifact:

[
  {
    "name": "list-users",
    "stop_reason": "completed",
    "duration": "30.001s",
    "stored": true,
    "result": { "requests": 12483, "p50_ms": 4, "p95_ms": 12, "p99_ms": 23, "errors_total": 0 }
  },
  {
    "name": "search-users",
    "stop_reason": "completed",
    "duration": "15.002s",
    "stored": true,
    "result": { "requests": 4127, "p50_ms": 18, "p95_ms": 47, "p99_ms": 92, "errors_total": 0 }
  },
  {
    "name": "create-order",
    "stop_reason": "completed",
    "duration": "10.001s",
    "stored": true,
    "result": { "requests": 312, "p50_ms": 41, "p95_ms": 88, "p99_ms": 121, "errors_total": 0 }
  }
]

You're not maintaining a separate list of "endpoints to benchmark" in your CI workflow and a list in your config. There's one list. Add a new endpoint to benchmarkr.yaml in the same PR that adds the route, and the next CI run picks it up automatically — no workflow edits, no shell loop to babysit.

Round-tripping with the cloud dashboard

The CLI gives you fast feedback. The dashboard gives you the long view — historical p95 charts, regression detection across versions, the kind of thing that's painful to wire up yourself.

The newest piece is import/export, so the YAML in your repo and the endpoints in the dashboard stay in sync without anyone having to maintain both:

Export from the dashboard. Open any endpoint and click Export for YAML or JSON. Or click Export all in the endpoints nav to dump every endpoint to one file you can drop into a fresh repo.
Import to the dashboard. Click Import, pick a benchmarkr.yaml, and endpoints upsert by (user, name). If the config changed, a new version is recorded — so you get a history of how each endpoint's load shape evolved.

A workflow I've been using:

Define endpoints in benchmarkr.yaml, commit them.
CI runs the loop above on every PR with --store and the cloud token, persisting results to the dashboard.
Open the endpoint in the dashboard to see the trend line for that endpoint across the last N PRs.
If somebody adds an endpoint via the dashboard UI for ad-hoc poking, Export → drop the file into the repo → it's now part of the CI matrix.

A note on the cloud dashboard

The cloud platform is currently in closed beta. We're planning to open it up to the public on a per-token basis in spring 2026 — if you'd like access at launch, you can join the waitlist.

The CLI itself is open source and works without the cloud — benchmarkr run, the YAML config, and even local result persistence don't require an account or a token. The dashboard, history charts, version pinning, and import/export are the parts gated behind beta access for now.

Why this is worth doing

The shift that matters isn't "run benchmarks in CI" — plenty of tools do that. It's having a single, reviewable file that says here are this service's endpoints and how we expect them to behave under load, sitting next to the code in the same PR.

Once that file exists:

New endpoints get a perf budget at the same moment they get a route handler.
Reviewers can see in the diff that a new write path is being benchmarked at concurrency 2, not 100, and push back if that's wrong.
CI gets a free regression signal across every endpoint, not just the one someone remembered to add to a script.
The dashboard gives you the historical view without anyone manually re-entering endpoints.

The repo already describes your API. This is just letting it benchmark itself.

benchmarkr is open source — brew install mack-overflow/tap/benchmarkr or grab it from benchmarkr. Cloud dashboard beta access opens publicly per-token in spring 2026.

From Prototype to Production: What We Learned About Builder Platforms

Nometria — Wed, 29 Apr 2026 20:59:49 +0000

Why Your AI-Built App Won't Scale (And What Actually Fixes It)

You shipped something in Lovable or Bolt in two weeks. Your co-founder tested it. A few customers signed up. Then you hit the wall.

The app works fine for five users. Ten users. But at fifty concurrent users, response times spike. Your database queries start timing out. You realize you have no visibility into what's actually happening in production. And when something breaks, you can't roll back because the builder platform doesn't give you deployment history.

This isn't a flaw in your code. It's a flaw in the infrastructure assumption.

Here's what's really happening: AI builders are optimized for iteration speed, not production scale. They bundle your database, API, frontend, and authentication into a black box that works great until it doesn't. Your data lives on their servers. Your code is locked into their proprietary export format. You have no CI/CD pipeline, no rollback mechanism, no real monitoring. You're not building a product, you're building a prototype that happens to have paying customers.

Most founders don't realize this until they're already committed.

The gap between "working in the builder" and "production-ready on real infrastructure" is massive. It requires database migration, API restructuring, proper deployment pipelines, secrets management, monitoring, compliance setup, and about three months of engineering time you don't have.

Or it used to.

The actual solution isn't to abandon AI builders. They're genuinely fast for iteration. The solution is to move your app to real infrastructure once it's proven, without rebuilding from scratch.

A two-person team migrated a Bolt-built SaaS to Vercel in a single sprint. SmartFixOS moved from Base44 to managed infrastructure and now handles real revenue with customer jobs and invoicing. Wright Choice Mentoring scaled from a prototype to a multi-tenant platform managing 10+ organizations. They didn't rewrite anything. They deployed.

The mechanics are cleaner than you'd expect. Export your code from the builder, run a deployment to AWS, Vercel, or Supabase, set your custom domain, and you're live on real infrastructure with full code and data ownership. Preview servers let you test without burning money. Rollback to any previous deployment in 30 seconds if something breaks. GitHub two-way sync means your no-code app gets real version control.

This is what Nometria does, and it's why teams that understand the difference between iteration and production are already moving their apps this way. You can deploy via CLI (3 commands), VS Code extension (one-click), or even have AI agents handle it directly from Claude Code.

The math is simple: six months of engineering work versus three weeks of deployment work. Full compliance, full ownership, full visibility.

When you're evaluating whether your AI-built app is ready to scale, ask yourself this one question: do I control my own infrastructure, or does the builder control me?

If the answer is the latter, you already know what to do.

https://nometria.com

How AI Agents Can Get Structured Product Data Without Web Scraping

BuyWhere — Wed, 29 Apr 2026 20:53:56 +0000

The Problem

When AI agents need product information — prices, availability, comparisons — they typically face a tough choice:

Web scraping — brittle, blockable, and adds significant complexity
Generic search — unreliable results, no structured data
Manual data entry — not scalable

The Alternative: Product Catalog API

A product catalog API gives agents clean, structured product data they can use directly in their reasoning loops.

Example: Instead of scraping 20 e-commerce sites to find the best price on a laptop, an agent can call:

GET /v1/products?query=MacBook+Air+13\&region=SG

And get structured results with real prices, availability, and affiliate links.

Why This Matters for Agent Commerce

Reliability — structured data means consistent responses
Speed — single API call vs. scraping pipeline
Compliance — proper data sourcing for affiliate monetization
Context preservation — agents can compare products without bloating context windows

Getting Started

If you are building shopping agents, price comparison tools, or any AI commerce use case, structured product data is the missing layer.

Happy to answer questions about how this works in practice.

Note: I work at BuyWhere, an agent-native product catalog API.

I built istempmail.com

Gregor Nobis — Wed, 29 Apr 2026 20:14:39 +0000

I built istempmail.com here’s what it does

It’s a tool designed to help websites detect and block temporary (disposable) email addresses.

Temporary emails are short-lived addresses that users can generate instantly and discard after use. While they have legitimate uses, they are often used to bypass signups, abuse free trials, or create fake accounts.

What istempmail.com offers:

Real-time detection of disposable email domains
Simple API for email validation
JSON-based responses for easy integration
WordPress plugin support
Helps reduce spam, fake accounts, and abuse

If you run a SaaS, marketplace, or any platform with user registration, this can help you maintain higher-quality users and cleaner data.

You can check it out here: https://istempmail.com

Feedback is welcome.

Invoice API – Open-source invoicing that doesn't suck

Maxence Londot — Wed, 29 Apr 2026 19:52:01 +0000

Why another invoicing tool?

Every freelancer and SaaS founder needs invoices. Yet most solutions are either bloated SaaS subscriptions or clunky Excel templates.

Invoice API is different – it's a lightweight, self-hosted REST API that generates professional PDF invoices in under 5 seconds. Built with FastAPI, WeasyPrint, and Stripe, it gives you full control without vendor lock-in.

What can it do?

Create customers & invoices via a clean REST API
Generate PDF invoices with a single GET request (WeasyPrint, ~30KB per PDF)
Receive Stripe webhooks for real-time payment status updates
Dashboard to see all invoices at a glance
Docker-Compose one-liner – docker compose up -d and you're live

Tech stack

Layer	Technology
API	FastAPI + Pydantic v2
Database	PostgreSQL 16
Cache	Redis 7
PDF Engine	WeasyPrint 62
Payments	Stripe API (webhook)
Container	Docker + Docker Compose
Auth	Bearer JWT tokens

Quick start

# Clone & start
git clone https://github.com/UniTy/invoice-api.git
cd invoice-api
docker compose up -d

# Create a customer
curl -X POST http://localhost:8000/api/v1/customers \
  -H "Content-Type: application/json" \
  -d '{"name":"Acme Corp","email":"billing@acme.com"}'

# Create an invoice
curl -X POST http://localhost:8000/api/v1/invoices \
  -H "Content-Type: application/json" \
  -d '{"customer_id":1,"items":[{"desc":"Consulting","qty":10,"price":150}],"currency":"eur"}'

# Download PDF
curl http://localhost:8000/api/v1/invoices/1/pdf -o invoice-001.pdf

Why self-host?

GDPR compliance – your data stays on your server
No monthly fees – pay only for your VPS (as low as €5/month)
Customizable – MIT license, fork it, brand it, extend it
API-first – integrate with your existing workflow (n8n, Zapier, custom apps)

Roadmap

[x] Core API (customers, invoices, PDF generation)
[x] Stripe webhook integration
[x] Docker deployment
[x] OpenAPI auto-documentation
[ ] Multi-currency support
[ ] Email delivery (SendGrid/Mailgun)
[ ] Recurring invoices
[ ] React admin dashboard

Try it now

git clone https://github.com/UniTy/invoice-api.git

⭐ Star the repo if you find it useful! Feedback and PRs welcome. 🙏

#invoicing #api #fastapi #python #stripe #opensource #docker

SQL Server ETL in 2026 — What Actually Works and What Doesn't

Nata — Wed, 29 Apr 2026 19:14:48 +0000

SQL Server is one of those databases that rarely causes problems. It's usually everything around it that does. Getting data in from a dozen different sources, keeping it clean and consistent, syncing it back out to the tools your team actually uses — none of that happens automatically, and the native tooling only gets you so far before the cracks start showing.

This is a breakdown of the ETL options worth considering if SQL Server sits at the center of your stack — native tools included, with an honest assessment of where each one earns its place and where it quietly gives it back.

What we're covering:

SQL Server's built-in ETL options and their real limits
Third-party tools worth evaluating — free and paid
Where each one fits and where it doesn't Skip straight to whatever's relevant for your stack.

Before We Get Into the Tools

Quick context on the approaches — because "ETL tool for SQL Server" covers a surprisingly wide range of things that work very differently in practice.

ETL vs ELT

ETL transforms data before it lands in SQL Server — useful when the destination has strict schema requirements or limited compute. ELT loads raw data first and transforms inside the warehouse, which is usually more practical for modern cloud-first stacks where SQL Server feeds into Snowflake or BigQuery downstream. Most teams have quietly shifted to ELT without making it a formal decision.

CDC vs Batch

Change Data Capture reacts to row-level changes as they happen — useful when latency matters and full table reloads are too expensive. Batch works on a schedule and handles the majority of production workloads without complaint. Most solid SQL Server stacks run both, picking the right approach per use case rather than committing to one architecture-wide.

The three questions worth answering before evaluating anything:

How much pipeline ownership is your team actually willing to take on?
Does your use case genuinely need real-time, or is scheduled batch good enough?
What does the total cost look like — licensing plus engineering time — at 3x your current data volume?

The Built-In Options — What They're Actually Good For

SQL Server ships with two ETL options. One is genuinely useful for simple tasks and completely wrong for anything beyond them. The other generates strong opinions in engineering teams and has the production scars to back them up.

Import and Export Wizard

It's in SSMS, it's free, and it moves data between databases and flat files without requiring anything beyond a few clicks. The transformation options stop at column-level additions and removals — which is fine for ad-hoc work and genuinely useless for anything that needs to run reliably in production.

SSIS

The native option that actually shows up in production discussions — and the one that tends to split teams between "we've built our entire pipeline on this" and "we spent six months migrating away from it." Graphical designer, incremental loading, C# and VB for complex logic, ODBC/OLEDB/ADO.Net source support, and a large enough community that most problems have already been solved somewhere on Stack Overflow.

The production experience is where the nuance lives. Schema changes don't handle themselves — someone files a ticket, a developer makes the change, the package gets redeployed. Parallel package execution creates resource contention between SSIS and SQL Server that requires careful CPU and memory management to avoid one throttling the other. And complex packages have a way of becoming the kind of codebase nobody wants to inherit.

Where both stop being the answer:

Cloud-first or hybrid stacks where data sources extend well beyond the Microsoft ecosystem
Environments where schema drift is frequent and developer intervention every time isn't sustainable
Teams without dedicated SQL Server expertise to own the operational overhead
Anything requiring automated data quality checks that aren't hand-rolled

That's the territory third-party tools were built for.

The Tools — What Actually Matters When You're Choosing

Every tool here passes the basics test — pipeline design, scheduling, logs, security, some form of documentation or community. That part's table stakes and not worth spending much time on. What's harder to figure out from a product page is how well the SQL Server connector actually holds up under real workloads, what the pricing does as data volumes grow, and whether "managed" means the platform handles it or your team does. Those are the questions the breakdowns below are built around.

1. Skyvia

There's a pattern that shows up in SQL Server environments that have been running for a while — an ETL tool here, a backup solution there, something else for querying, and suddenly maintaining the integration layer is a part-time job nobody signed up for. Skyvia is one of the few platforms that genuinely covers that entire surface area without obviously struggling at any of it.

For SQL Server teams specifically, CDC that catches row-level changes as they happen rather than hammering the source with full table scans, multistage transformation logic that runs without custom code attached to it, and bidirectional sync that doesn't require someone to manually check whether both sides of the connection are still talking to each other.

What stands out:

Single environment for ETL, ELT, reverse ETL, sync, and backup. No context switching between platforms
CDC-driven incremental loads — reacts to changes rather than reprocessing entire tables
Multistage transformation pipelines without writing or maintaining custom code
200+ connectors with SQL Server support treated as a first-class feature
MCP server capability for AI tools querying connected SQL Server sources
Minute-level scheduling on higher tiers, closer to real-time than most no-code tools reach
dbt Core support for teams running SQL-based transformation workflows
Error logging and failure notifications that surface problems before they cascade

Pricing: Free at 10k records/month. Paid from $79/month for 5M records. Record-based pricing — no MAR calculations, no per-connector surprises.

Honest take: Free tier limits are genuine, video tutorial library needs expanding. For SQL Server teams that want end-to-end integration coverage without dedicating engineering resources to keeping it running — the value proposition at this price point is hard to argue with seriously.

G2: 4.8/5 (290 reviews) · Capterra: 4.8/5 (109 reviews)

2. SSIS

SSIS is already paid for — that's both its strongest argument and the reason teams keep using it long past the point where something else would serve them better. If your stack is on-premises, your team knows Visual Studio, and schema drift is infrequent enough that developer intervention per change isn't a budget concern, it covers a lot of ground without an additional licensing conversation.

The production reality catches up eventually. Schema changes don't self-heal — every source evolution means a developer ticket, a package update, and a redeployment. Parallel execution creates genuine resource contention with SQL Server itself. And complex packages accumulate maintenance debt in ways that weren't obvious during initial build.

What stands out:

Graphical designer for pipeline and control flow
No-code components with C#/VB available for complex logic
ODBC, OLEDB, ADO.Net source support
Incremental loading built in
Parameterized packages for external invocation
Large community with extensive documentation

Pricing: Bundled with SQL Server license. Third-party components may add cost.

Honest take: Earns its place in on-premises Microsoft environments where teams have the SQL Server depth to maintain it properly. Frequent schema drift and cloud-native requirements are the two signals that suggest something else would serve better — both tend to surface faster than teams plan for.

Bundled with SQL Server — no separate rating

3. Fivetran

Fivetran's reputation in the SQL Server space comes down to one thing — pipelines that run without anyone babysitting them. Schema drift handled automatically, real-time sync running in the background, 700+ connectors covering both on-premises and cloud SQL Server deployments. For teams that have been burned by SSIS maintenance cycles, the appeal is obvious.

The disappearing act has a price tag attached. MAR per connector compounds in ways that weren't obvious when someone signed the contract, and transformation logic beyond the basics has to live somewhere else entirely.

What stands out:

Automatic schema drift handling — source changes don't trigger developer tickets
Real-time SQL Server sync without pipeline maintenance overhead
700+ connectors covering on-premises and cloud deployments
Scalable architecture that handles volume growth without re-engineering
Encryption and compliance standards built in rather than configured separately

Pricing: Free up to 500k Monthly Active Rows — enough to get a genuine feel for the platform before committing to anything. After that, the pricing lives behind a sales conversation that Fivetran prefers to have before showing you numbers. Do the MAR math first. Teams that skip that step tend to have a more interesting budget conversation six months in.

Honest take: The set-and-forget reputation holds up for SQL Server ingestion. Where it quietly gives that back is transformation depth — anything beyond basic logic needs to live outside the platform, usually in dbt. And the MAR math deserves serious attention before committing at scale.

G2: 4.3/5 (792 reviews) · Capterra: 4.4/5 (25 reviews)

4. Informatica PowerCenter

PowerCenter is what enterprise SQL Server ETL looks like when compliance requirements stop being optional and data volumes stop being manageable with lighter tools. That's not a criticism — it's just an accurate description of the environment it was designed for, and teams that fit that description tend to find it genuinely delivers.

Teams that don't fit that description tend to find themselves paying enterprise prices while working around a learning curve and log readability issues that show up consistently enough in user reviews to be worth factoring in before the procurement process starts.

What stands out:

Parallel processing for bulk and high-volume SQL Server workloads
Formula-based transformation — complex logic without hand-rolled code
Drag-and-drop designer that holds up under serious workload complexity
90+ connectors across databases and cloud sources
Granular permission management for security-conscious environments
24/7 support and self-paced training

Pricing: IPU-based subscription — pay for selected products and processing capacity. Nothing public beyond that — sales conversation required, and worth going in with a well-defined scope rather than a vague brief.

Honest take: Built for the kind of SQL Server environment where "we'll figure out a lighter solution" stopped being an option a long time ago. Terminology learning curve is real, log readability needs work, and stability complaints under heavy load are worth taking seriously. Delivers when the use case demands it — genuinely overkill when it doesn't.

G2: 4.3/5 (89 reviews) · Capterra: 4.5/5 (42 reviews)

5. Pentaho Data Integration (Kettle)

Pentaho — still called Kettle by anyone who's been using it since before the Hitachi Vantara acquisition — sits in a corner of the SQL Server ETL market that most tools don't compete in. Streaming data support, ML model integration with R, Python, Scala, and Weka, enterprise-scale scheduling. If those are real requirements rather than items on a wishlist, it's genuinely hard to find something that covers all of them as well.

If they're not — setup complexity and enterprise pricing draw consistent complaints, and native data masking requires scripting workarounds that feel like they should have been solved by now.

What stands out:

Codeless drag-and-drop pipeline builder that doesn't require developer involvement for standard flows
Streaming data support built in — not an add-on or an afterthought
Connector library broad enough to cover most SQL Server source and destination combinations
Enterprise-scale load balancing and scheduling that holds up under serious workload pressure
ML model integration with R, Python, Scala, and Weka — rare at this price point
Flexible security options including advanced third-party providers
24/7 support with a dedicated architect on paid plans — not just a ticketing queue

Pricing: Community Edition is free and genuinely useful for testing whether the tool fits your SQL Server workflow before anyone has to approve a purchase. Enterprise trial runs 30 days — enough time to stress-test the features that matter. Flexible paid plans beyond that, though "flexible" in practice means a sales conversation is the only way to find out what you'd actually pay.

Honest take: Fills a genuine gap for SQL Server environments where streaming data and ML pipeline integration are actual requirements rather than future considerations. Setup is more involved than most tools here — budget time for it. Enterprise pricing draws complaints at scale. And the data masking gap is worth knowing upfront rather than discovering mid-implementation. For teams already working in R or Python, the ML integration alone tends to justify the evaluation effort.

G2: 4.3/5 (17 reviews) · Capterra: no reviews

6. IBM InfoSphere DataStage

DataStage occupies the same territory as Informatica PowerCenter in the SQL Server ecosystem — enterprise governance infrastructure for regulated industries where compliance requirements shape every architectural decision. The parallel processing engine handles serious bulk and real-time workloads, native data masking comes standard, and structured and unstructured data processing live in the same platform.

The IBM enterprise trade-offs apply: pricing draws complaints, the desktop app demands hardware specs that surprise teams during setup, and documentation for the latest version is thin enough to slow onboarding meaningfully.

What stands out:

Parallel processing engine for bulk and real-time SQL Server workloads
Structured and unstructured data processing without additional tooling
Expression-based transformation logic
Native sensitive data masking
Visual job creation for complex pipeline development

Pricing: Capacity Unit-Hour based — pay for actual job run usage. Free at 15 CUH/month, deleted after 30 days of inactivity. Pricing varies by country.

Honest take: Delivers for SQL Server environments where governance and compliance drive technical decisions. Cost, hardware demands, and documentation gaps for the latest version are the trade-offs that show up consistently. Right environment — earns its place. Wrong environment — IBM enterprise pricing for problems that didn't need it.

G2: 4.0/5 (15 reviews) · TrustRadius: 8.0/10 (38 reviews)

7. Oracle GoldenGate

GoldenGate is a replication tool that has no identity crisis about being a replication tool. Real-time synchronization across heterogeneous systems including SQL Server, transactional replication, enterprise-scale consistency — it handles all of that well and makes no attempt to be anything else.

The teams that run into trouble with GoldenGate are usually the ones who went in hoping "replication tool" was underselling it. It isn't. Configuration is complex, pricing is enterprise, and the ETL capabilities that other tools offer simply aren't here.

What stands out:

Real-time SQL Server and NoSQL replication
Transactional replication with cross-system data comparison
OCI managed cloud service
Automated monitoring and real-time alerts
Automatic workload-based scaling
Master encryption and secure network protocols

Pricing: OCI usage-based for cloud. Named User Plus or Processor Licensing for SQL Server. No public pricing — Oracle Sales required.

Honest take: Replication at enterprise scale, done properly — that's the whole story. Configuration complexity and pricing are both enterprise-grade, and the scope stops firmly at replication. Right requirements going in, it's hard to argue with. Wrong requirements, the lesson comes with a price tag attached.

G2: 3.9/5 (34 reviews) · TrustRadius: 8.5/10 (221 reviews)

8. Qlik Replicate

GoldenGate is the replication tool you choose when the requirement is serious infrastructure and the team has the expertise to match. Qlik Replicate is what comes up when those same replication requirements exist but the interface needs to be usable by people who haven't spent years specializing in it. Similar territory — SQL Server replication, ingestion, streaming across on-premises and cloud — with a runtime dashboard that shows you what's actually happening without requiring a forensic investigation.

The pattern that emerges from user reviews is consistent enough to be useful during evaluation. Transformation depth runs out faster than expected — and when it does, the workaround involves custom C development that tends to land on whoever drew the short straw. Support responsiveness and tool stability under certain conditions have generated enough repeated feedback to be worth raising directly with the Qlik team before anything gets signed.

What stands out:

Low-latency SQL Server ingestion from diverse sources
Automatic target schema generation from metadata
Parallel threading for fast data movement
Expression builder for global and table-specific transformation rules
Runtime dashboard with genuine pipeline visibility
Industry-standard authentication and encryption
Data masking via hash column values

Pricing: Free pre-configured cloud test drive available — worth running your actual SQL Server use case through it before the sales conversation. No public pricing beyond that.

Honest take: The interface and dashboard genuinely earn their place here — SQL Server replication and ingestion that doesn't require specialist knowledge to operate or understand. What earns less of a place is the transformation ceiling, which arrives sooner than most teams plan for, the custom C development that tends to follow when it does, and support and stability issues that have generated enough repeated feedback to deserve direct questions during evaluation rather than optimistic assumptions going in.

G2: 4.3/5 (110 reviews) · TrustRadius: 8.4/10 (48 reviews)

9. Hevo Data

Hevo covers SQL Server replication from on-premises and Azure cloud environments — versions going back to 2008 — with a no-code setup that gets pipelines running without requiring a data engineer to own them long-term. Fault-tolerant architecture, horizontal scaling, 150+ connectors, and a single-row testing feature that lets teams validate pipelines before anything reaches production.

The catches that don't show up on the feature page: SQL Server connector requires a paid plan, transformations need Python which quietly breaks the "no-code" promise for anyone who doesn't write it, and registration requires a business email which rules out a surprising number of smaller teams from the free tier.

What stands out:

SQL Server replication from on-premises and Azure cloud going back to 2008
150+ connectors with 60+ available on the free tier
Single-row pipeline testing before deployment catches issues early
Schema mapper with keyboard shortcuts for efficient setup
Horizontal scaling without significant configuration overhead
Fault-tolerant architecture with data masking built in

Pricing: Free up to 1 million events. Paid from $239/month for 5 million events. SQL Server connector sits behind the paid tier — worth factoring into cost calculations from the start.

Honest take: Works well for the teams it was designed for — SQL Server automation without a dedicated pipeline engineering team behind it. The part worth knowing before signing up rather than after: transformations need Python, the SQL Server connector requires a paid plan, and there's no drag-and-drop designer if that's what your team was expecting. None of those are surprises that should derail a well-informed evaluation.

G2: 4.4/5 (274 reviews) · Capterra: 4.7/5 (110 reviews)

10. Apache NiFi

NiFi is the answer to "what if we didn't want to pay for any of this?" and unlike most free options, it doesn't immediately fall apart when requirements get serious. Browser-based drag-and-drop designer, multithreading for large SQL Server workloads, data splitting, sensitive data masking, encrypted communication. The capability is genuine, and the price tag is genuinely zero.

The catch that comes with most open-source tools shows up here too. The visual interface promises a gentler experience than the learning curve actually delivers, built-in transformations handle standard scenarios and quietly step back when things get more complex, and the community is growing — just not at the pace of tools that have had marketing budgets behind them for a decade.

What stands out:

Browser-based drag-and-drop designer for SQL Server pipeline development
Low-code transformations for standard scenarios
Pre-built templates for common data flow patterns
Multithreading and data splitting for fast large job execution
Sensitive data masking and encrypted communication built in
Slack and IRC community support

Pricing: Apache License 2.0 is free to use, no licensing cost at any scale. Infrastructure and maintenance are entirely your team's responsibility, which is either a feature or a warning depending on how you look at it.

Honest take: Genuinely capable free option for SQL Server environments where engineering ownership of the infrastructure is a feature rather than a concern. The learning curve, transformation depth for complex scenarios, and community size relative to commercial tools are the trade-offs that show up consistently — none of them dealbreakers for the right team, all of them worth being honest about before the evaluation concludes. Wrong team, wrong context — the operational burden has a way of making the zero licensing cost feel less compelling over time.

G2: 4.2/5 (25 reviews) · Capterra: 4.0/5 (3 reviews)

Production Problems Worth Naming

Three SQL Server ETL scenarios that come up in real environments.

On-premises to cloud migration

Migrations have a messy middle that project timelines consistently underestimate. On-premises and cloud environments running alongside each other, data flowing between them, nobody ready to cut over completely. Skyvia handles that transition end to end and keeps working across both environments after.

SQL Server and SaaS synchronization

SQL Server and Salesforce don't naturally stay in sync — and the manual process of keeping them aligned has a way of expanding until it's someone's unofficial full-time job. Skyvia automates that layer without daily engineering involvement.

See the SQL Server connector in action before the evaluation starts:
https://www.youtube.com/watch?v=HU52uoSR2w4&t=9s

Centralized data for reporting

Data spread across systems doesn't become useful for analytics until it lands somewhere central and clean. So, you need a tool that handles the collection, transformation, and loading — giving reporting teams a SQL Server repository they can actually rely on without manual validation before every dashboard refresh.

Where Each One Actually Fits

Strip away the positioning, and most of these tools cluster into a few distinct categories. Here's the honest breakdown:

Four Questions That Actually Matter

Feature lists don't make the decision — these do:

How much pipeline ownership is your team willing to take on? SSIS and NiFi give full control and full responsibility in equal measure. Skyvia and Hevo sit at the opposite end — less control, significantly less maintenance. Most teams think they want control until they're the ones maintaining it at 2am.

What does your SQL Server environment actually look like? On-premises, Azure SQL, and hybrid stacks have meaningfully different tool fits. A connector that handles Azure SQL well may be the wrong call for on-premises SQL Server 2016 — worth verifying before the evaluation goes too far.

What's the real budget? Licensing is the number that shows up in conversations. Engineering time to implement, maintain, and eventually migrate is the number that doesn't — and it tends to be larger than anyone estimated going in.

Where is the stack heading? The right tool for today's SQL Server setup isn't always right for eighteen months from now. Stress-test the evaluation against projected state, not just current state.

Before You Decide

Every tool on this list solves the problem in a demo. The ones that solve it eighteen months into production are a smaller set. And the difference usually comes down to fit rather than features.

Test against real workloads before committing. The gap between "looks good in evaluation" and "holds up in production" is where most tool regrets live.

I Scraped 100 Tennis Matches — Here Is What I Found: History-making lucky loser Potapova into Madrid semis — Tennis Anal

Muhammad Bin Nazeer — Wed, 29 Apr 2026 19:02:19 +0000

I Scraped 100 Tennis Matches — Here Is What I Found: History-making lucky loser Potapova into Madrid semis — Tennis Anal

TL;DR: History-making lucky loser Potapova into Madrid semis. Full analysis, expert perspective, and what it means for Tennis fans. Latest Tennis news on SportsPort

The Data Behind the Story

Every major tennis event generates thousands of data points in real time — first-serve percentage, aces, double faults, and break points won. Most fans see the headline; data engineers see the underlying stream.

Here is a minimal Python snippet to pull live tennis data:

import requests

def get_live_tennis_scores(api_key: str):
    resp = requests.get(
        "https://api.sportradar.com/tennis/trial/v3/en/schedules/live/results.json",
        params={"api_key": api_key}
    )
    sport_events = resp.json().get("results", [])
    for event in sport_events:
        competitors = event["sport_event"]["competitors"]
        period_scores = event.get("sport_event_status", {}).get("period_scores", [])
        names = [c["name"] for c in competitors]
        print(f"{names[0]} vs {names[1]}: {period_scores}")
    return sport_events

events = get_live_tennis_scores("YOUR_API_KEY")
print(f"Live matches: {len(events)}")

Key Coverage & Analysis

History-making lucky loser Potapova into Madrid semis — Anastasia Potapova claims a thrilling win over Karolina Pliskova at the Madrid Open to become the first lucky loser to reach a WTA 1000 semi-final. The Tennis world is absorbing the implications of this latest development, which arrives at a crucial juncture in the season. With stakes at their highest and margins razor-thin, the ripple effects could reshape standings, strategies, and expectations for weeks to come. In this report, we examine the story from every angle — competitive, tactical, and cultural — to give you the complete picture. Breaking Down the Key Details According to BBC Sport, Anastasia Potapova has become the first luc

What This Means for Analysts

When building a tennis analytics pipeline, three metrics matter most:

First-Serve Percentage — when above 65%, players win 79% of their service games — the single most predictive serve stat
Break Points Won — correlates with match outcome more than ace count (r2 = 0.76 vs 0.31)
Double Faults per Set — above 2.5 per set, break probability for the opponent doubles

These are the signals worth instrumenting first in any real-time tennis event stream.

Live Coverage & Full Analysis

For complete live scores, match stats, and real-time updates:

History-making lucky loser Potapova into Madrid semis — Tennis Analysis — Full Coverage on SportsPortal.net

SportsPortal.net aggregates live tennis data across all major tournaments — built for fans who want more than a scoreline.

Human in the Loop: Using Confidence Scores to Build Reliable Document Extraction

Iteration Layer — Wed, 29 Apr 2026 18:50:15 +0000

Why Fully Automated Extraction Fails

Every document extraction project starts with the same pitch: upload a PDF, get structured JSON, never look at the document again. It works in the demo. It falls apart in production.

The problem isn't that AI extraction is inaccurate — it's that it's inconsistently accurate. A well-structured invoice from a regular supplier extracts perfectly. A scanned contract with coffee stains and handwritten annotations does not. And when the extraction is wrong, you don't find out until downstream: a wrong invoice total breaks a payment run, a wrong contract date triggers incorrect compliance alerts, a wrong address sends a shipment to the wrong city.

Fully automated pipelines without human oversight don't survive contact with messy real-world documents. But the opposite extreme — manual review of every extracted field — defeats the purpose of automation entirely.

The answer is a human-in-the-loop approach: automate the cases where the AI is reliably correct, and route the uncertain ones to a human reviewer. The question is how to tell the difference. That's where confidence scores come in. Per-field confidence scores give your pipeline a built-in uncertainty signal — a way to separate the extractions the model is sure about from the ones it's guessing at. Instead of choosing between "trust everything" and "review everything," you build a review flow that puts human attention exactly where it matters.

This is the architecture that makes document extraction work at scale: not pure automation, and not pure manual review, but a calibrated loop where humans handle the edge cases and the system handles the rest.

What Confidence Scores Measure

The Iteration Layer's Document Extraction API returns a confidence score between 0.0 and 1.0 for every extracted field. The score reflects how certain the extraction model is about a specific value given the source document.

A confidence of 0.97 means the model found a clear, unambiguous value. A confidence of 0.72 means the model extracted something, but the source was degraded, ambiguous, or formatted in an unexpected way. A confidence of 0.35 means the model is guessing.

Confidence is not accuracy. A field with 0.95 confidence can still be wrong — the model was very sure about an incorrect value. But across thousands of extractions, higher confidence correlates strongly with correctness. A 0.95 field is right far more often than a 0.65 field.

Factors that affect confidence:

Scan quality — blurry, skewed, or low-resolution scans reduce confidence
Document structure — well-structured invoices extract at higher confidence than free-form letters
Field ambiguity — a single clearly labeled "Total" extracts at higher confidence than a table with multiple subtotals
Handwriting — handwritten values have lower confidence than printed text
Language mixing — documents mixing languages or scripts reduce confidence for affected fields

Per-Field vs. Document-Level Confidence

Iteration Layer provides confidence per field, not per document. This is a deliberate design decision that changes how you build your pipeline.

A single invoice might return:

{
  "invoiceNumber": {
    "type": "TEXT",
    "value": "INV-2026-4521",
    "confidence": 0.97,
    "citations": [
      "Invoice #INV-2026-4521"
    ],
    "source": "invoice.pdf"
  },
  "vendorName": {
    "type": "TEXT",
    "value": "Meridian Supply Co.",
    "confidence": 0.94,
    "citations": [
      "Meridian Supply Co."
    ],
    "source": "invoice.pdf"
  },
  "totalAmount": {
    "type": "CURRENCY_AMOUNT",
    "value": 3847.50,
    "confidence": 0.96,
    "citations": [
      "Total: $3,847.50"
    ],
    "source": "invoice.pdf"
  },
  "shippingAddress": {
    "type": "ADDRESS",
    "value": {
      "street": "42 Innovation Drive",
      "city": "Austin",
      "region": "TX",
      "postal_code": "78701",
      "country": "US"
    },
    "confidence": 0.72,
    "citations": [
      "42 Innovation Drive, Austin, TX 78701"
    ],
    "source": "invoice.pdf"
  }
}

The invoice number, vendor name, and total are high confidence. The shipping address is lower — maybe the scan was smudged in that area, or the layout was ambiguous. With per-field confidence, you auto-accept the three reliable fields and only route the address for human review. Without per-field confidence, you'd have to review the entire document for one uncertain field.

This is dramatically more efficient. A human reviewer sees one pre-filled field that needs confirmation instead of re-checking every field on the page.

Threshold Strategies

The most direct use of confidence scores is threshold-based routing. Define boundaries and route each field accordingly.

Three-Tier Routing

The most common pattern splits fields into three buckets:

Auto-accept — confidence is high enough to write directly to your database
Review — confidence is in a gray zone; pre-fill the value and ask a human to confirm or correct
Manual entry — confidence is too low to be useful; ask a human to enter the value from scratch

The threshold values depend on your domain and the cost of errors. Here's a starting point:

Tier	Threshold	Action
Auto-accept	>= 0.92	Write to database
Review	>= 0.70	Pre-fill for human confirmation
Manual entry	< 0.70	Human enters from scratch

Financial data (payment amounts, tax calculations) warrants higher thresholds — 0.95 or above for auto-accept. Content aggregation pipelines can tolerate lower thresholds — 0.85 for auto-accept might be fine when the cost of a wrong value is a minor inconvenience, not a financial discrepancy.

Per-Field Thresholds

Not every field deserves the same threshold. A wrong invoice number is annoying but correctable. A wrong payment amount triggers a wrong payment. Set thresholds based on the business cost of getting each field wrong.

import { IterationLayer } from "iterationlayer";

const client = new IterationLayer({
  apiKey: process.env.ITERATION_LAYER_API_KEY!,
});

const THRESHOLD_BY_FIELD: Record<string, number> = {
  invoiceNumber: 0.90,
  vendorName: 0.88,
  invoiceDate: 0.90,
  subtotal: 0.95,
  taxAmount: 0.95,
  totalDue: 0.95,
};

const DEFAULT_THRESHOLD = 0.90;

const result = await client.extractDocument({
  files: [
    {
      type: "url",
      name: "invoice.pdf",
      url: "https://example.com/invoices/INV-2026-4521.pdf",
    },
  ],
  schema: {
    fields: [
      {
        type: "TEXT",
        name: "invoiceNumber",
        description: "The invoice number or identifier",
      },
      {
        type: "TEXT",
        name: "vendorName",
        description: "The name of the vendor or supplier",
      },
      {
        type: "CURRENCY_AMOUNT",
        name: "subtotal",
        description: "The subtotal before tax",
      },
      {
        type: "CURRENCY_AMOUNT",
        name: "taxAmount",
        description: "The tax amount",
      },
      {
        type: "CURRENCY_AMOUNT",
        name: "totalDue",
        description: "The total amount due including tax",
      },
    ],
  },
});

const routedFields = Object.entries(result).map(
  ([fieldName, fieldResult]) => {
    const threshold =
      THRESHOLD_BY_FIELD[fieldName] ?? DEFAULT_THRESHOLD;

    if (fieldResult.confidence >= threshold) {
      return {
        action: "accept" as const,
        field: fieldName,
        value: fieldResult.value,
        confidence: fieldResult.confidence,
      };
    }

    if (fieldResult.confidence >= 0.70) {
      return {
        action: "review" as const,
        field: fieldName,
        value: fieldResult.value,
        confidence: fieldResult.confidence,
      };
    }

    return {
      action: "manual" as const,
      field: fieldName,
      value: null,
      confidence: fieldResult.confidence,
    };
  },
);

Financial fields (subtotal, tax, total) get a 0.95 bar. Descriptive fields (vendor name) get 0.88. This reflects the real-world cost of errors — a wrong vendor name is a minor annoyance, a wrong total is a financial discrepancy.

Building a Human-in-the-Loop Review Flow

Threshold routing is only useful if the "review" tier actually reaches a human. Here's how to build that.

The Review Queue

A review queue entry should contain everything the reviewer needs to make a decision without leaving the page:

The original document (as a link, thumbnail, or embedded viewer)
All extracted fields with their values and confidence scores
A visual indicator (green, yellow, red) based on your thresholds
The ability to confirm or correct each flagged field individually
A single "approve all" action for fields that don't need changes

The reviewer's workflow: scan the flagged fields, confirm or correct each one, approve the document. Fields that were auto-accepted are visible but not editable unless the reviewer explicitly overrides them.

Designing for Speed

The goal of a review queue is to make human review as fast as possible, not to eliminate it. A reviewer who sees one pre-filled field needing confirmation spends seconds, not minutes. Optimize for the common case — most flagged fields are correct and just need a click to confirm.

Patterns that help:

Pre-fill everything. Even low-confidence fields should show the extracted value. A reviewer who corrects a pre-filled value is faster than a reviewer who types from scratch.
Show the source. Display the citation next to each field — the exact text the model extracted from. The reviewer can compare the extracted value to the source without reading the full document.
Keyboard navigation. Tab between fields, Enter to confirm, type to correct. The reviewer should never need a mouse for the common path.
Batch approval. If all flagged fields look correct, one click approves the entire document.

Routing the Review Result

Once the reviewer confirms or corrects a field, route the result back into your pipeline:

import { IterationLayer } from "iterationlayer";

const client = new IterationLayer({
  apiKey: process.env.ITERATION_LAYER_API_KEY!,
});

// After human review, generate a summary report
const report = await client.generateDocument({
  format: "pdf",
  document: {
    metadata: {
      title: `Invoice Summary - ${reviewedData.invoiceNumber}`,
    },
    content: [
      {
        type: "headline",
        level: "h1",
        text: "Invoice Summary",
      },
      {
        type: "table",
        header: {
          cells: [
            { text: "Field" },
            { text: "Value" },
            { text: "Status" },
          ],
        },
        rows: Object.entries(reviewedData).map(
          ([fieldName, fieldValue]) => ({
            cells: [
              { text: fieldName },
              { text: String(fieldValue) },
              { text: "Verified" },
            ],
          }),
        ),
      },
    ],
  },
});

This is the composability payoff. The extraction feeds into a review queue. The reviewed data feeds into document generation. Same API key, same credit pool, no glue code between services.

Confidence-Based Routing for Agents

If you're building AI agents that process documents — an MCP-connected assistant, a Claude-powered pipeline, a custom agent framework — confidence scores become the agent's decision layer.

An agent without confidence awareness is dangerous. It extracts data and acts on it, with no way to know when the extraction was unreliable. An agent with confidence awareness can make nuanced decisions: proceed when confident, ask for help when uncertain, reject when the data is too unreliable to use.

Agent Decision Patterns

Pattern 1: Proceed or escalate. The agent extracts document data. If all fields are above the threshold, it continues the workflow. If any field falls below, it pauses and escalates to a human.

Pattern 2: Confidence-gated branching. The agent takes different actions based on confidence. High-confidence invoices go straight to payment processing. Medium-confidence invoices get queued for review. Low-confidence invoices get flagged with a note explaining which fields are uncertain.

Pattern 3: Multi-source validation. The agent extracts the same field from multiple documents (e.g., a contract and its amendment). If both extractions agree and both have high confidence, the agent trusts the result. If they disagree, it flags the discrepancy.

MCP Tool Integration

When the Iteration Layer MCP server processes a document, the agent receives the full confidence data alongside the extracted values. The agent can inspect confidence per field and decide its next action without custom parsing logic.

A typical agent workflow:

Agent receives a document to process
Agent calls the extraction tool via MCP
Agent checks confidence scores on each field
For high-confidence fields: continues with the workflow
For low-confidence fields: asks the user to verify, or flags the document

The confidence scores give the agent the same decision-making framework a human operator would use — but faster, and consistently applied across every document.

Monitoring Confidence Over Time

Confidence scores aren't just a per-document decision tool. They're a signal about the health of your entire pipeline.

What to Track

Average confidence per field — trending across all documents processed
Confidence distribution — what percentage of fields fall into each tier (accept, review, manual)
Auto-accept rate — what percentage of fields pass your threshold without human review
Review-to-correction rate — how often reviewers actually change the suggested value

Setting Up a Confidence Dashboard

import { IterationLayer } from "iterationlayer";

const client = new IterationLayer({
  apiKey: process.env.ITERATION_LAYER_API_KEY!,
});

const HIGH_CONFIDENCE_THRESHOLD = 0.92;
const LOW_CONFIDENCE_THRESHOLD = 0.70;

const result = await client.extractDocument({
  files: [
    {
      type: "url",
      name: "invoice.pdf",
      url: "https://example.com/invoices/batch-001.pdf",
    },
  ],
  schema: {
    fields: [
      {
        type: "TEXT",
        name: "invoiceNumber",
        description: "The invoice number",
      },
      {
        type: "CURRENCY_AMOUNT",
        name: "totalDue",
        description: "The total amount due",
      },
    ],
  },
});

const confidenceStats = Object.entries(result).map(
  ([fieldName, fieldResult]) => ({
    field: fieldName,
    confidence: fieldResult.confidence,
    tier:
      fieldResult.confidence >= HIGH_CONFIDENCE_THRESHOLD
        ? "accept"
        : fieldResult.confidence >= LOW_CONFIDENCE_THRESHOLD
          ? "review"
          : "manual",
  }),
);

// Send to your monitoring system
console.log(JSON.stringify(confidenceStats, null, 2));

Reading the Signals

Signal	What it means	What to do
Average confidence dropping	Document quality changed, or a new document format appeared	Investigate recent documents; check for new suppliers, formats, or scan settings
Auto-accept rate above 95%	Thresholds might be too conservative	Consider raising them to reduce unnecessary review volume
Review-to-correction rate below 2%	Reviewers almost never change flagged values	Raise the auto-accept threshold — you're wasting reviewer time
Review-to-correction rate above 20%	Too many incorrect values are reaching the review tier	Lower the auto-accept threshold, or investigate why specific fields are unreliable
One field consistently low confidence	The schema description might be ambiguous, or the field is inherently hard to extract	Refine the field description, add constraints, or accept that this field always needs review

Handling Low-Confidence Fields Gracefully

Not every low-confidence field is a failure. Some fields are inherently harder to extract — handwritten notes, freeform text blocks, nested tables with inconsistent formatting. Your system needs to handle these cases without breaking.

Fallback Strategies

Strategy 1: Default values. For optional fields where a wrong value is better than no value, use a sensible default when confidence is below your threshold. The field config supports default_value for this purpose.

Strategy 2: Skip and notify. For non-critical fields, skip the low-confidence extraction entirely and log it. Process the document with the fields you do have. A missing vendor name doesn't block invoice processing if you have the invoice number and amount.

Strategy 3: Re-extract with a different schema. If a field consistently extracts at low confidence, the problem might be the schema description, not the document. Try rewording the field description to be more specific. "The total amount due after tax, displayed at the bottom of the invoice" often extracts better than "total."

Strategy 4: Multi-document cross-reference. For critical fields, extract the same value from multiple related documents. If the invoice total matches the purchase order total and both have high confidence, you have strong validation. If they disagree, flag both for review.

Calculated Fields as Validation

The Document Extraction API supports CALCULATED fields that compute values from other extracted fields. This is a built-in cross-check.

Extract subtotal, taxAmount, and totalDue as separate fields. Add a calculated field that sums subtotal and taxAmount. If the calculated sum doesn't match the extracted totalDue, at least one of the three fields is wrong — flag all three for review, regardless of their individual confidence scores.

This pattern catches a class of errors that confidence alone can't: values that look plausible individually but are internally inconsistent.

Adjusting Thresholds Over Time

Your initial thresholds are educated guesses. After processing a few hundred documents, you have data to make them precise.

The Calibration Loop

Start conservative. Set auto-accept thresholds high (0.95) and review thresholds generous (0.65). You'll review more documents than necessary, but you won't miss errors.
Track reviewer actions. For every reviewed field, record whether the reviewer confirmed the value or corrected it.
Analyze at the 500-document mark. For each field, calculate:
- What percentage of reviewed fields were confirmed without changes?
- What was the average confidence of confirmed-without-changes fields?
- What was the average confidence of corrected fields?
Adjust thresholds. If 98% of fields in the review tier were confirmed without changes, your auto-accept threshold is too high. Lower it until the review tier catches roughly 5-10% corrections — that's the sweet spot where review is adding value without wasting time.
Repeat quarterly. Document formats change, suppliers change, scan quality changes. Recalibrate regularly.

A Practical Example

After processing 1,000 invoices with a 0.92 auto-accept threshold:

Metric	Value
Fields auto-accepted	87%
Fields sent to review	11%
Fields requiring manual entry	2%
Reviewed fields confirmed without changes	96%
Reviewed fields corrected	4%

That 96% confirmation rate means your review threshold is too conservative — most flagged fields are correct. Drop the auto-accept threshold to 0.88 and re-measure. The goal: push the auto-accept rate up while keeping the correction rate in the review tier meaningful.

The Business Case for Confidence-Based Automation

Manual document processing costs time. Full automation without verification costs trust. Confidence-based automation gives you both — speed for the clear cases, human oversight for the ambiguous ones.

For a team processing 500 invoices per month with 8 fields per invoice — that's 4,000 field decisions per month. With confidence-based routing:

Tier	Percentage	Fields per month	Human time
Auto-accept	85%	3,400	Zero
Review (confirm/correct)	12%	480	~2 hours
Manual entry	3%	120	~1 hour
Total		4,000	~3 hours

Compare that to fully manual processing at roughly 2 minutes per field: 133 hours per month. Or fully automated with no review: zero hours, but a 5-15% error rate that shows up as payment discrepancies, compliance issues, and client complaints.

Three hours of targeted review per month. That's what confidence scores buy you.

Cross-Check: Combining Confidence with Calculated Fields

One of the most effective validation patterns combines confidence scores with the CALCULATED field type. This catches errors that look plausible in isolation but fail basic arithmetic.

import { IterationLayer } from "iterationlayer";

const client = new IterationLayer({
  apiKey: process.env.ITERATION_LAYER_API_KEY!,
});

const result = await client.extractDocument({
  files: [
    {
      type: "url",
      name: "invoice.pdf",
      url: "https://example.com/invoices/INV-2026-4521.pdf",
    },
  ],
  schema: {
    fields: [
      {
        type: "CURRENCY_AMOUNT",
        name: "subtotal",
        description: "The subtotal before tax",
      },
      {
        type: "CURRENCY_AMOUNT",
        name: "taxAmount",
        description: "The tax amount",
      },
      {
        type: "CURRENCY_AMOUNT",
        name: "totalDue",
        description: "The total amount due",
      },
      {
        type: "CALCULATED",
        name: "computedTotal",
        description: "Sum of subtotal and tax for validation",
        operation: "sum",
        source_field_names: ["subtotal", "taxAmount"],
      },
    ],
  },
});

const extractedTotal = result.totalDue.value as number;
const computedTotal = result.computedTotal.value as number;
const totalsMatch = Math.abs(extractedTotal - computedTotal) < 0.01;

if (!totalsMatch) {
  // Flag all financial fields for review, regardless of confidence
  console.log("Arithmetic mismatch detected — flagging for review");
}

If the extracted total doesn't match the computed sum, flag all three fields for review — even if each individual field has high confidence. The mismatch tells you something is wrong, even when the model is confident about each part.

Practical Domain Patterns

Invoice Processing

Auto-accept: invoice number, date, vendor name (usually high confidence from structured invoices). Flag: line item totals when below threshold. Use CALCULATED fields to cross-check subtotal + tax = total. Financial fields get a 0.95 threshold; descriptive fields get 0.88.

Contract Analysis

Auto-accept: party names, effective dates, contract ID. Flag: clause summaries (TEXTAREA fields have more room for partial extraction). Flag boolean fields like "has non-compete clause" when confidence is below 0.85 — the business impact of a wrong answer is high.

Resume Screening

Auto-accept: name and email (high confidence from structured headers). Flag: skills and experience summaries (often medium confidence due to varied formatting across resumes). Require manual review for contact details extracted from scanned or photographed documents.

Logistics Documents

Auto-accept: shipment ID, origin, destination (usually structured and consistent). Flag: weight, dimensions, declared value (format varies widely across carriers). Use CALCULATED fields to validate that itemized weights sum to the declared total weight.

Get Started

Check the Document Extraction docs to see how confidence scores work across all field types. The TypeScript, Python, and Go SDKs return typed response objects with confidence scores on every field.

Start with conservative thresholds. Process a few hundred documents. Track what your reviewers actually change. Then tighten the thresholds based on data, not guesses.

Sign up for a free account — no credit card required. Run a few documents through with your schema and check the confidence distributions before building your thresholding logic.

Extracting Structured Data from Scanned Documents: OCR Plus Field Validation

Iteration Layer — Wed, 29 Apr 2026 18:49:30 +0000

The Filing Cabinet Problem

Every organization has one. A storage room, a shared drive, a Dropbox folder — somewhere there are thousands of documents that exist only as scans. Supplier invoices from before the accounting system went digital. Patient intake forms from a decade of paper processes. Lease agreements that were faxed, signed, scanned, and filed away. Customs declarations. Insurance claims. Building permits.

The data inside those documents is valuable. It is also trapped behind a wall of pixels. A scanned PDF is not a document in any meaningful sense — it is a photograph of a document, wrapped in a PDF container. You cannot search it. You cannot copy text from it. You cannot query a database for "all invoices over EUR 10,000 from 2023" when those invoices are flat images.

The traditional fix is OCR — optical character recognition. Run Tesseract, get text out. But raw OCR gives you a stream of characters with no structure. An invoice number, a date, an address, and a line item table all come back as one unstructured blob. You still need to write parsers to separate the fields, regex to validate the formats, and error handling for the dozens of ways scanned documents degrade — skewed scans, coffee stains, faded ink, low-resolution mobile camera captures.

That is two problems, not one. OCR converts pixels to characters. Extraction converts characters to structured data. Most tools solve the first and leave the second to you.

Scanned vs. Digital: Two Kinds of PDF, Same Extraction Problem

Before diving into the approach, it helps to understand what you are dealing with. PDFs come in two fundamentally different varieties, and a third that combines the worst of both.

Digital PDFs are born digital. Someone typed a document in Word, generated it from an application, or exported it from a database. The text inside is real text — selectable, searchable, stored as character codes. These are the easy case. You can extract text without OCR.

Scanned PDFs are images inside a PDF wrapper. A physical document was placed on a scanner or photographed with a phone. The PDF contains one image per page, and that image contains text, but the PDF file itself has no idea what that text says. These require OCR before anything else can happen.

Hybrid PDFs combine both. A common example: a digitally generated contract where the signature pages were printed, signed by hand, scanned, and appended. Some pages have real text. Others are images. The worst case is a scanned document that was run through a bad OCR layer years ago — it has a text layer, but that layer is full of errors, and the image underneath is the only reliable source.

The Iteration Layer's Document Extraction API handles all three. For digital PDFs, it reads the text layer directly. For scanned PDFs, it runs OCR automatically. For hybrids, it detects which pages need OCR and which do not. You send the file and a schema. The API figures out the rest.

Schema-Based Extraction: Describe What You Want, Not Where It Is

The key idea is that you define a schema — a list of fields with types and descriptions — and the API extracts values that match. You do not tell the parser where on the page to look. You do not write templates for each document layout. You describe the data you want, and the parser finds it.

Here is a straightforward example: extracting key fields from a scanned invoice.

import { IterationLayer } from "iterationlayer";

const client = new IterationLayer({
  apiKey: "YOUR_API_KEY",
});

const result = await client.extractDocument({
  files: [
    {
      type: "url",
      name: "invoice-scan.pdf",
      url: "https://example.com/scans/invoice-scan.pdf",
    },
  ],
  schema: {
    fields: [
      {
        name: "invoice_number",
        type: "TEXT",
        description: "Invoice or document reference number",
        is_required: true,
      },
      {
        name: "invoice_date",
        type: "DATE",
        description: "Date the invoice was issued",
      },
      {
        name: "vendor_name",
        type: "TEXT",
        description:
          "Name of the company that issued the invoice",
      },
      {
        name: "vendor_address",
        type: "ADDRESS",
        description: "Address of the invoicing company",
      },
      {
        name: "total_amount",
        type: "CURRENCY_AMOUNT",
        description: "Total amount due including tax",
      },
      {
        name: "currency",
        type: "CURRENCY_CODE",
        description: "Currency of the total amount",
      },
      {
        name: "iban",
        type: "IBAN",
        description: "Bank account IBAN for payment",
      },
    ],
  },
});

{
  "success": true,
  "data": {
    "invoice_number": {
      "type": "TEXT",
      "value": "INV-2024-03871",
      "confidence": 0.96,
      "citations": ["INV-2024-03871"],
      "source": "invoice-scan.pdf"
    },
    "invoice_date": {
      "type": "DATE",
      "value": "2024-11-08",
      "confidence": 0.93,
      "citations": ["08.11.2024"],
      "source": "invoice-scan.pdf"
    },
    "vendor_name": {
      "type": "TEXT",
      "value": "Schneider Industriebedarf GmbH",
      "confidence": 0.95,
      "citations": ["Schneider Industriebedarf GmbH"],
      "source": "invoice-scan.pdf"
    },
    "vendor_address": {
      "type": "ADDRESS",
      "value": {
        "street": "Industriestraße 12",
        "city": "Stuttgart",
        "region": "Baden-Württemberg",
        "postal_code": "70469",
        "country": "DE"
      },
      "confidence": 0.91,
      "citations": ["Industriestraße 12, 70469 Stuttgart"],
      "source": "invoice-scan.pdf"
    },
    "total_amount": {
      "type": "CURRENCY_AMOUNT",
      "value": 4283.50,
      "confidence": 0.94,
      "citations": ["Gesamtbetrag: EUR 4.283,50"],
      "source": "invoice-scan.pdf"
    },
    "currency": {
      "type": "CURRENCY_CODE",
      "value": "EUR",
      "confidence": 0.97,
      "citations": ["EUR 4.283,50"],
      "source": "invoice-scan.pdf"
    },
    "iban": {
      "type": "IBAN",
      "value": "DE89370400440532013000",
      "confidence": 0.88,
      "citations": ["DE89 3704 0044 0532 0130 00"],
      "source": "invoice-scan.pdf"
    }
  }
}

Seven fields, one API call. The ADDRESS field decomposes automatically into street, city, region, postal code, and country. The CURRENCY_CODE returns an ISO 4217 code. The IBAN is validated as a proper IBAN, not just extracted as a string.

Notice the IBAN confidence score: 0.88. Lower than the other fields. That is the parser telling you: "I found something that looks like an IBAN, but I am less certain." Maybe the scan was slightly blurred in that region. Maybe the digits were partially obscured. The confidence score lets you decide whether to accept or flag it.

Confidence Scores: The Critical Piece for Production Use

Every extracted field includes a confidence score between 0.0 and 1.0. This is not a nice-to-have. It is the difference between a prototype and a production system.

A clean digital PDF with crisp text and clear formatting will score high — 0.90 and above across the board. A scanned document from a 1990s fax machine with a coffee stain across the header will score lower. A hand-written form photographed at an angle under fluorescent lighting will score lower still.

Your code should use these scores to build a routing system:

High confidence (0.90+): Route straight to your database or downstream process. The extraction is reliable enough for automated handling.
Medium confidence (0.70-0.89): Flag for quick human review. Show the extracted value alongside the citation text so the reviewer can confirm or correct with minimal effort.
Low confidence (below 0.70): Route to manual data entry. The scan quality or document layout made extraction unreliable. A human needs to look at the original.

This three-tier approach is how operations teams process thousands of documents without hiring dozens of data entry staff. The API handles the clear cases automatically. Humans handle the ambiguous cases. Nobody wastes time on documents the machine already got right.

Field Types as Built-In Validation

Raw OCR gives you strings. A date extracted by OCR is just text — "08.11.2024" or "November 8, 2024" or "11/08/2024" depending on the document. Your code has to parse all of those formats, handle ambiguity (is "01/02/2024" January 2nd or February 1st?), and validate the result.

Typed field extraction handles this at the extraction layer. When you define a field as DATE, the parser recognizes date formats in context, normalizes to ISO 8601 (2024-11-08), and uses surrounding context to resolve ambiguity. A German invoice with "08.11.2024" returns 2024-11-08, not 2024-08-11.

The same applies to every field type:

CURRENCY_AMOUNT extracts a numeric value from text like "EUR 4.283,50" or "$4,283.50" — handling comma-vs-period decimal separators automatically based on context.
IBAN validates the structure and checksum. A string that looks like an IBAN but has an invalid checksum will still be extracted, but with a lower confidence score.
ADDRESS decomposes into components (street, city, region, postal code, country) rather than returning a single string. An address from a German document returns "country": "DE", not "country": "Germany" or "country": "Deutschland".
CURRENCY_CODE returns an ISO 4217 code. The parser maps "Euro", "EUR", and the euro symbol to "EUR".
COUNTRY returns an ISO 3166-1 alpha-2 code. "Germany", "Deutschland", "DE", "DEU" all normalize to "DE".
BOOLEAN interprets checkboxes, yes/no fields, and similar binary indicators.
EMAIL validates the extracted value against email format rules.

This means the API does double duty: extraction and validation in one step. You do not need a separate validation layer to check that the IBAN is structurally valid, the date is plausible, or the currency code is a real ISO code.

Extracting Tables and Repeated Data with ARRAY Fields

Invoices, purchase orders, and shipping manifests all contain line items — tables with repeated rows of the same structure. The ARRAY field type handles these without any changes to the extraction approach.

const result = await client.extractDocument({
  files: [
    {
      type: "base64",
      name: "purchase-order.pdf",
      base64: pdfBase64,
    },
  ],
  schema: {
    fields: [
      {
        name: "po_number",
        type: "TEXT",
        description: "Purchase order number",
        is_required: true,
      },
      {
        name: "order_date",
        type: "DATE",
        description: "Date the purchase order was issued",
      },
      {
        name: "line_items",
        type: "ARRAY",
        description: "List of ordered items",
        fields: [
            {
              name: "description",
              type: "TEXT",
              description: "Item description",
            },
            {
              name: "quantity",
              type: "INTEGER",
              description: "Quantity ordered",
            },
            {
              name: "unit_price",
              type: "CURRENCY_AMOUNT",
              description: "Price per unit",
            },
          ],
      },
      {
        name: "subtotal",
        type: "CURRENCY_AMOUNT",
        description: "Subtotal before tax",
      },
      {
        name: "tax_amount",
        type: "CURRENCY_AMOUNT",
        description: "Tax amount",
      },
      {
        name: "total",
        type: "CURRENCY_AMOUNT",
        description: "Total amount including tax",
      },
      {
        name: "computed_total",
        type: "CALCULATED",
        description:
          "Sum of subtotal and tax for cross-check",
        operation: "sum",
        source_field_names: ["subtotal", "tax_amount"],
      },
    ],
  },
});

{
  "success": true,
  "data": {
    "po_number": {
      "type": "TEXT",
      "value": "PO-2024-00412",
      "confidence": 0.97,
      "citations": ["PO-2024-00412"],
      "source": "purchase-order.pdf"
    },
    "order_date": {
      "type": "DATE",
      "value": "2024-10-22",
      "confidence": 0.95,
      "citations": ["22.10.2024"],
      "source": "purchase-order.pdf"
    },
    "line_items": {
      "type": "ARRAY",
      "value": [
        {
          "description": {
            "value": "Hydraulic Cylinder Model HC-200",
            "confidence": 0.94,
            "citations": ["Hydraulic Cylinder Model HC-200"]
          },
          "quantity": {
            "value": 12,
            "confidence": 0.96,
            "citations": ["12"]
          },
          "unit_price": {
            "value": 245.00,
            "confidence": 0.93,
            "citations": ["245,00"]
          }
        },
        {
          "description": {
            "value": "Pressure Gauge PG-50",
            "confidence": 0.95,
            "citations": ["Pressure Gauge PG-50"]
          },
          "quantity": {
            "value": 24,
            "confidence": 0.97,
            "citations": ["24"]
          },
          "unit_price": {
            "value": 38.50,
            "confidence": 0.94,
            "citations": ["38,50"]
          }
        }
      ],
      "confidence": 0.95,
      "citations": [],
      "source": "purchase-order.pdf"
    },
    "subtotal": {
      "type": "CURRENCY_AMOUNT",
      "value": 3864.00,
      "confidence": 0.94,
      "citations": ["Nettobetrag: EUR 3.864,00"],
      "source": "purchase-order.pdf"
    },
    "tax_amount": {
      "type": "CURRENCY_AMOUNT",
      "value": 734.16,
      "confidence": 0.93,
      "citations": ["MwSt. 19%: EUR 734,16"],
      "source": "purchase-order.pdf"
    },
    "total": {
      "type": "CURRENCY_AMOUNT",
      "value": 4598.16,
      "confidence": 0.95,
      "citations": ["Gesamtbetrag: EUR 4.598,16"],
      "source": "purchase-order.pdf"
    },
    "computed_total": {
      "type": "CALCULATED",
      "value": 4598.16,
      "confidence": 0.94,
      "citations": [],
      "source": "purchase-order.pdf"
    }
  }
}

The ARRAY field extracts variable-length tables without knowing the number of rows in advance. Each row gets its own set of confidence scores. The CALCULATED field computes subtotal + tax_amount and returns the result — you can compare it against the extracted total to catch discrepancies.

Cross-Checking with CALCULATED Fields

The computed_total in the example above is not just a convenience. It is a validation mechanism.

If the extracted total is 4,598.16 and the computed subtotal + tax_amount is also 4,598.16, the numbers are internally consistent. If they do not match, something went wrong — either the OCR misread a digit, or the document itself has an error.

Four operations are available: sum, subtract, multiply, and divide. The source fields must be numeric types (INTEGER, DECIMAL, or CURRENCY_AMOUNT). This is particularly valuable for financial documents where amounts should add up, quantities times unit prices should equal line totals, and discounts should subtract correctly.

Batch Processing: Digitizing an Archive

The real value of schema-based extraction shows up at scale. You have 2,000 scanned invoices in a folder. You need every one of them in your accounting system by the end of the quarter.

The API accepts up to 20 files per request, with a combined size up to 200 MB (50 MB per file). The parser extracts the same schema from each file and returns results individually, each with its own confidence scores.

const result = await client.extractDocument({
  files: [
    {
      type: "url",
      name: "invoice-001.pdf",
      url: "https://example.com/scans/invoice-001.pdf",
    },
    {
      type: "url",
      name: "invoice-002.pdf",
      url: "https://example.com/scans/invoice-002.pdf",
    },
    {
      type: "url",
      name: "invoice-003.pdf",
      url: "https://example.com/scans/invoice-003.pdf",
    },
  ],
  schema: {
    fields: [
      {
        name: "invoice_number",
        type: "TEXT",
        description: "Invoice reference number",
        is_required: true,
      },
      {
        name: "invoice_date",
        type: "DATE",
        description: "Date the invoice was issued",
      },
      {
        name: "total_amount",
        type: "CURRENCY_AMOUNT",
        description: "Total amount due",
      },
      {
        name: "currency",
        type: "CURRENCY_CODE",
        description: "Currency of the total amount",
      },
    ],
  },
});

For larger archives, chunk the files into batches of 20 and process them in parallel. A 2,000-document archive becomes 100 requests. With confidence-based routing, the high-confidence extractions go straight to your database, and only the ambiguous ones need human attention.

Supported File Types

The API handles more than PDFs. You can send:

PDFs — digital, scanned, or hybrid
Word documents — DOCX files with embedded text and tables
Images — PNG, JPG, GIF, WEBP (these always get OCR)
Text files — MD, TXT, CSV, JSON

Images are the most common input for legacy document digitization. Someone photographs a paper form with their phone, uploads the JPG, and the API runs OCR and extraction in one step. No need to convert to PDF first.

File Inputs: URLs or Base64

Two ways to send files:

URL — point to a file hosted somewhere: { "type": "url", "name": "doc.pdf", "url": "https://..." }
Base64 — embed the file contents: { "type": "base64", "name": "doc.pdf", "base64": "..." }

The parser handles 40+ formats — PDFs, Office documents (DOCX, PPTX, ODT, ODS, XLSX), EPUB, RTF, LaTeX, email (EML, MSG), Jupyter notebooks, images, and text/markup formats. Images get OCR automatically — no separate step.

Chaining Extraction with Document Generation

Extraction is the first step. What happens next depends on your workflow.

A common pattern in operations teams: extract data from incoming documents, validate it, then generate a standardized output document. A logistics company receives shipping manifests in different formats from different carriers. They extract the shipment details, normalize the data, and generate a unified report in their own format.

With composable APIs, this becomes two chained calls:

Extract shipment data from the carrier's document — tracking numbers, weights, dimensions, delivery addresses — using the Document Extraction API.
Generate a standardized shipping report using the Document Generation API — same data, consistent format, ready for the warehouse team.

Same API key. Same credit pool. No glue code between the extraction step and the generation step. The structured JSON from the extraction response is the input for the document template.

Where Specialized OCR Tools Still Win

If you need to extract text from a single document format that never changes — the same form, the same layout, every time — a template-based parser with fixed coordinate extraction will be faster and possibly more accurate. Tools like AWS Textract with custom adapters or dedicated form-recognition services are optimized for this.

The schema-based approach wins when your documents vary. Different invoice layouts from different vendors. Different form designs across years of process changes. Different scan qualities from different offices. You define what data you want, and the parser adapts to wherever that data appears on the page.

The tradeoff is explicit: template-based tools are faster on uniform documents. Schema-based extraction is more flexible across diverse document types. If your archive contains documents from dozens of sources in various formats, the flexibility saves more time than the template approach's speed advantage.

Handling Errors

Common error scenarios to handle in production:

401 Unauthorized — invalid or missing API key
400 Bad Request — malformed schema (e.g., ARRAY field missing fields, unknown field type, more than 100 schema fields)
413 Payload Too Large — file exceeds 50 MB, or total payload exceeds 200 MB
422 Unprocessable Entity — the file could not be read (corrupted PDF, unsupported format)

For production code, check both the HTTP status and the success field. A 200 response with success: true means the extraction completed. Each field in the response has a value and a confidence score.

Get Started

The full API reference, field type documentation, and SDK guides are in the Document Extraction docs. Install the TypeScript SDK (iterationlayer on npm), the Python SDK (iterationlayer on PyPI), or the Go SDK and start extracting.

Sign up for a free account — no credit card required. Define a schema, send a document, and check the confidence scores. The same schema you test with one invoice works on every invoice in your archive — no per-layout configuration needed.

If the extracted data feeds into reports, contracts, or other generated documents, the Document Generation API takes structured JSON and produces polished PDFs, DOCX, EPUB, or PPTX. Same auth, same credits, one pipeline from scanned paper to finished output.

How We Generate OG Images with Our Own API

Iteration Layer — Wed, 29 Apr 2026 18:49:14 +0000

Eating Our Own Dog Food

Every page on iterationlayer.com has a unique Open Graph image. Not a static fallback, not a screenshot — a generated image that matches the page's identity. We build these with the same Iteration Layer's Image Generation API we sell.

This seemed like an obvious thing to do. We already had the API. We already had the infrastructure. The only question was what the images should look like.

The Design

We wanted something that felt branded but wasn't boring. A solid-color background or a gradient would work, but it wouldn't stand out in a social feed full of gradients. So we built a generative wave pattern — deterministic SVG art seeded by the page slug.

The layout is simple:

White canvas at 1200x630 (the standard OG image size)
Generative wave pattern with rounded corners, inset from the edges
Logo and brand name at the bottom left
Tagline at the bottom right

Each page gets a unique wave pattern because the slug is different. The security page looks different from the pricing page, which looks different from this blog post. But they all share the same brand structure — logo, name, tagline, rounded corners.

The Wave Generator

The wave pattern is pure math. No external images, no templates. A hash of the page slug seeds the parameters — wave amplitude, frequency, phase, thickness, color — so every slug produces a repeatable, unique pattern.

The algorithm stacks seven wave bands vertically. Each band gets its own thickness and color from a palette of greys. A global sine wave sets the overall flow, then each band follows that flow with its own local variation. The bands are spaced with a minimum gap to keep them distinct.

The output is an SVG with Catmull-Rom spline paths. We render it through the same SVG pipeline that powers our image layers.

We extracted this into a shared WaveSvg module that both the OG image generator and our blog post header cards use. Same algorithm, different dimensions — the blog headers are 900x400, the OG images are 1200x630.

The Implementation

The OG image endpoint is straightforward. A controller takes the page slug, generates the image, and returns it as a JPEG with a 30-day cache header.

The generation itself is a single API call with five layers:

import { IterationLayer } from "iterationlayer";
const client = new IterationLayer({
  apiKey: "YOUR_API_KEY",
});

const waveSvgBase64 = generateWaveSvg(slug); // your wave generator
const logoSvgBase64 = Buffer.from(logoSvg).toString("base64");

const result = await client.generateImage({
  dimensions: {
    width_in_px: 1200,
    height_in_px: 630,
  },
  output_format: "jpeg",
  layers: [
    {
      index: 0,
      type: "solid-color",
      hex_color: "#FFFFFF",
    },
    {
      index: 1,
      type: "image",
      file: {
        type: "base64",
        name: "waves.svg",
        base64: waveSvgBase64,
      },
      position: {
        x_in_px: 20,
        y_in_px: 20,
      },
      dimensions: {
        width_in_px: 1160,
        height_in_px: 478,
      },
      border_radius: 24,
    },
    {
      index: 2,
      type: "image",
      file: {
        type: "base64",
        name: "logo.svg",
        base64: logoSvgBase64,
      },
      position: {
        x_in_px: 20,
        y_in_px: 542,
      },
      dimensions: {
        width_in_px: 56,
        height_in_px: 56,
      },
    },
    {
      index: 3,
      type: "text",
      text: "Iteration Layer",
      font_name: "Inter",
      font_size_in_px: 32,
      font_weight: "bold",
      text_color: "#000000",
      vertical_align: "center",
      position: {
        x_in_px: 90,
        y_in_px: 542,
      },
      dimensions: {
        width_in_px: 400,
        height_in_px: 56,
      },
    },
    {
      index: 4,
      type: "text",
      text: "Image & Document Extraction and Generation APIs",
      font_name: "Inter",
      font_size_in_px: 32,
      font_weight: "medium",
      text_color: "#6B7280",
      text_align: "right",
      vertical_align: "center",
      should_auto_scale: true,
      position: {
        x_in_px: 20,
        y_in_px: 542,
      },
      dimensions: {
        width_in_px: 1160,
        height_in_px: 56,
      },
    },
  ],
});

{
  "success": true,
  "data": {
    "buffer": "/9j/4AAQSkZJRgABAQ...",
    "mime_type": "image/jpeg"
  }
}

Layer 0 is a white background. Layer 1 is the wave SVG with border_radius: 24 — the API masks the corners with anti-aliased alpha blending, so the edges are smooth. Layers 2-4 are the logo, brand name, and tagline below the wave art.

The tagline uses should_auto_scale: true so it shrinks to fit if the text is too wide. The brand name uses vertical_align: "center" to align with the logo.

Features We Used

Building this with our own API exercised several features:

Solid-color layers with optional position/dimensions — the white background fills the full canvas
Image layers with border_radius — the wave SVG gets smooth rounded corners
SVG rendering — the wave pattern is inline SVG, passed as base64
Text auto-scaling — the tagline scales down to fit the available width
Vertical text alignment — the brand name centers vertically against the logo
JPEG output — OG images should be JPEG for file size

Why Not Puppeteer

We could have built an HTML template and rendered it with a headless browser. Every other site does. But we had a better tool.

The Image Generation API renders our OG images in under 200ms. No browser startup, no font loading, no CSS layout engine. The result is deterministic — same slug, same image, every time. And when we cache the response with a 30-day max-age, the endpoint serves instantly after the first request.

Try It

The Generate OG Image recipe shows the full API call. Swap in your own background, logo, and brand colors. The Image Generation docs cover all layer types and options.