Amazon Nova Pro matched a 4GB open-weight model running on a laptop on German PII redaction: 94% exact-value recall against 93%. Nova Micro, the cheapest model in the family, tied Amazon Comprehend on the same test at roughly a twentieth of the cost per document. And the model that lost hardest was the one fine-tuned for German.
Here is what we measured across six approaches, two languages, and four orders of magnitude of cost.
The blocker is not the model
You have data. It contains names, email addresses, phone numbers, IBANs, dates of birth, health codes, account numbers. You want a language model to summarize it, classify it, extract from it, or index it for search. The model is capable. The data is ready. The personally identifiable information in it is what stops you.
GDPR, HIPAA, and data processing agreements restrict where PII can transit, and approval for your cloud provider is not approval for every service inside it. Internal access controls make it worse rather than better: legal can see contract party details and finance cannot, but those boundaries live in your systems of record and dissolve the moment raw data enters a shared RAG index or a prompt template three teams call. An analyst asking for revenue from client X can get an answer derived from a contract they have no clearance to read.
Then there is the leak nobody plans for. Production data reaches development accounts constantly, through payloads copied while debugging and dumps used to build test fixtures. And when the compliant workflow takes three days and the non-compliant one takes three minutes, people take the three minutes: a support engineer pastes a complaint into a consumer chatbot, a recruiter runs a CV batch through a free tool. This is not a security failure. It's a workflow design failure.
A redaction layer separates the concerns. Process the data before it reaches any model, replace identities with typed placeholders, let the model work on structure and meaning.
Which scale are you at
Small: ad-hoc, individual use
A developer reviewing a contract. An analyst summarizing 20 support tickets.
Do not build anything. If your organization has Kiro, a Claude Enterprise licence, or Claude through Bedrock, the desktop app plus a PII redaction skill handles this today: the skill holds the tag taxonomy and the output-format constraint, the frontier model in your subscription does the work, and the data processing agreement you already signed covers it.
Cost is the wrong question at this volume. A subscription seat is a fixed monthly charge, and twenty contracts consume none of it in any meaningful sense. Pick the frontier model, because at twenty documents you want the best contextual judgment available rather than the cheapest tokens.
The skill is also the on-ramp. The instructions you write to make it work by hand are the same prompt artifact you move into a pipeline at the next scale, which is where the token arithmetic starts to matter.
Mid-size: team-level automation
A team redacting support tickets before summarization. Legal running contract review at volume. Data science cleaning a training set.
The answer splits on data residency.
If data can reach AWS, Amazon Nova Micro is the value pick: 92% on the curated set, 88% exact-value recall on German, sub-second latency, no infrastructure, about $3.90 per 100,000 short documents. Nova Pro buys the best cloud accuracy we measured (94% on German) at roughly $88 per 100,000, level with Comprehend and within a third of Haiku. Pay that when German financial identifiers are the point, and read the financial ID row below before deciding.
Amazon Comprehend remains a strong option with a different profile: 100ms rather than 400ms, 30+ entity types out of the box, no prompt to write or version, no model ID to pin, and DetectPiiEntities returns typed offsets rather than rewritten text. For a redaction step nobody has to tune, that is the one to reach for.
If data must stay on-premise or inside your own VPC, run mistral:7b with a Presidio injection pass. 4.1GB, no GPU, two to three seconds per document, 93% German recall and 83% English. On an m7g.xlarge (Graviton3, 16GB, about $116 a month) that covers roughly 50,000 documents a day.
Org-scale: platform-level
A gateway every internal service calls. A data platform enforcing redaction before documents land in a vector store. At that point you are not picking a model, you are building a routing layer with a fail-closed policy and an audit trail, which is a separate discussion. That is the follow-up piece.
What we tested
Six approaches, several models through them.
Presidio (Microsoft, open source) is a widely used rule-based PII detection library in enterprise Python stacks. It combines spaCy NER with regex patterns and checksum validators: Luhn for credit cards, country code and length for IBANs. These detections are deterministic. Same input, same output, every run.
It works well on structured PII in predictable formats: cards, emails, phone numbers in standard shapes, US SSNs. At 0.1 seconds per request with no warmup and no API call, it is the cheapest option at any volume.
It falls short wherever context carries the identification. "Her husband works at Deutsche Bank in Frankfurt" contains a location and an organization, and together they identify one person. Presidio tags Frankfurt and has no way to see the sentence. DATE_TIME precision is the other weak spot: it flags "quarterly" and "last year" as dates at 85% confidence.
So Presidio's role is the structured-PII safety net layered on a model pass, rather than a standalone answer. The combination beats either alone.
BERT NER (dslim/bert-base-NER): fine-tuned token classification, ~400MB, local via transformers. Good on English named entities, no LLM dependency, useful as an air-gapped fallback. For multilingual coverage, Ar86Bat/multilang-pii-ner covers German, Italian, and French.
Open-weight hybrid: an open-weight model doing the semantic pass, with a Presidio injection pass over the original text.
Two axes are easy to conflate here. The approach is the model plus the injection pass. Where it runs is an independent choice: Ollama on a laptop, a container on EC2 or Fargate, a Lambda function, or the same weights served by Bedrock. We benchmarked through Ollama because that is the cheapest way to iterate, and the recall figures carry across substrates because the weights are identical. What changes is latency, cold-start behaviour, and which line of your bill it lands on.
Amazon Comprehend: managed DetectPiiEntities. English and Spanish supported officially, German working in practice. The operational case is strong before you look at any score: nothing to host, nothing to warm, no prompt to maintain, latency that does not move under load, every call logged through AWS CloudTrail.
Claude Haiku 4.5 and Sonnet 4.6: the same access pattern as any other Bedrock-hosted model. Curated set only, not run at real-world scale.
Amazon Nova (Micro, Lite, Pro) via inference profiles in eu-central-1. All three passed the no-PII baseline with zero false positives.
Bedrock hosts plenty of other models in the same size and latency class, including the Llama, Mistral, Cohere, and Qwen families and the newer Nova generation. Any of them could displace Nova Micro on this task. We picked six approaches to cover the architectural spectrum rather than to rank a catalog, and adding one more model to the harness is an afternoon of work.
How we scored it
Two test sets. First: 15 hand-crafted cases covering structured PII (emails, credit cards, SSNs, IBANs), contextual PII (implied names, org references, @ handles), German text, code comments containing developer emails, and a clean-text baseline.
Second: 100 records per language sampled from ai4privacy/pii-masking-400k, a 406k-record annotated dataset across six languages.
Two scoring methods. Tag-match asks whether a placeholder appeared near the right location. Exact-value recall asks whether the actual PII string is gone from the output. Tag-match runs about 10 points optimistic. Exact-value is what an auditor checks: is "Klaus Bauer" still readable, is DE89370400440532013000 still in the file?
Coverage differs by test set. All approaches ran the curated set. The German real-world set ran Presidio, the mistral:7b hybrid, Comprehend, and all three Nova models. The English real-world set ran Presidio, the llama3.1:8b hybrid, and Comprehend. The Claude models were not run at real-world scale because of API cost, so there is no exact-value figure for them.
With ±3% run-to-run variance from model non-determinism, treat differences under 5 points as ties throughout. That rule is applied consistently below, including where it works against the more interesting story.
Curated benchmark (tag-match scoring, 15 cases)
| Approach | Score | Speed | Data stays local |
|---|---|---|---|
| Presidio (rules only) | 69% | 0.1s | Yes |
| BERT NER + presidio | 78% | 0.1s warmup | Yes |
| mistral:7b alone | 78% | 2-6s | Yes |
| Hybrid (mistral + presidio) | 83% | 2-6s | Yes |
| Claude Sonnet 4.6 | 83% | 5-8s | No |
| Amazon Comprehend | 85% | 0.1s | No |
| Claude Haiku 4.5 | 86% | 5-7s | No |
| Nova Lite | 87% | 0.4s | No |
| Nova Pro | 90% | 0.5s | No |
| Nova Micro | 92% | 0.4s | No |
Apply the 5-point rule and the table sorts into three bands. Presidio alone at 69% is the floor, and any model pass raises it substantially. Comprehend, both Claude models, the local hybrid, and Nova Lite occupy the middle at 83-87%, statistically indistinguishable from each other. Nova Micro and Nova Pro sit clear of that band at 92% and 90%, tied with each other.
That gives the useful result twice over: the smallest and cheapest Nova model performs at least as well as the largest, and the smaller Claude model performs at least as well as the larger. Redaction is constrained extraction and substitution. Find boundaries, classify, replace, preserve structure. Extra generative capability has nowhere useful to go.
Where the models differ is in what kind of PII they miss. Nova Micro and Nova Pro both tagged the email inside TODO(james.wilson@corp.io), a structural case that defeats Comprehend and most open-weight models. Nova Pro scored 5/5 on the dense multi-field record. But on the LinkedIn bio, where identity is implied rather than stated, Nova Micro managed 1/4 and Nova Pro 2/4, while Haiku scored 3/4. Sonnet scored 1/4 there, stripping "Senior Cloud Architect" as a person name and leaving BMW and Volkswagen standing.
So the AWS-native models are stronger on structure and Claude is stronger on implied context. Forms, tickets, and records: take the cheap option. Prose where identity has to be inferred: test both on your own text before committing.
On the clean baseline, all three Nova models, both Claude models, and Comprehend returned the no-PII input untouched. Presidio flagged "quarterly" as a date. Mistral invented a PERSON tag on the same clean input. A pipeline that mangles an earnings report gets switched off faster than one that misses an IBAN.
Real-world English: 100 cases, exact-value recall
| Type | Presidio | Hybrid (llama3.1:8b) | Comprehend |
|---|---|---|---|
| 61% | 87% | 92% | |
| DATE_TIME | 63% | 78% | 89% |
| LOCATION | 55% | 86% | 87% |
| PERSON | 56% | 82% | 86% |
| FINANCIAL_ID | 52% | 82% | 81% |
| PHONE | 53% | 85% | 83% |
| Overall | 55% | 83% | 86% |
Comprehend at 86% and the local hybrid at 83% are a tie under the 5-point rule. What separates them is not accuracy: Comprehend returns in 100ms against two to six seconds, with no model to host and no runtime to patch. In a request path, that decides the design on its own.
Presidio plateaus at 55%: strong on card numbers and emails, weak on names and locations in context.
Real-world German: 100 cases, exact-value recall
| Type | Presidio | Hybrid (llama3.1:8b) | Hybrid (mistral:7b) | Comprehend | Nova Micro | Nova Pro |
|---|---|---|---|---|---|---|
| 59% | 72% | 94% | 93% | 89% | 94% | |
| FINANCIAL_ID | 40% | 65% | 93% | 85% | 88% | 96% |
| LOCATION | 42% | 73% | 92% | 87% | 87% | 92% |
| PERSON | 45% | 73% | 93% | 86% | 88% | 95% |
| DATE_TIME | 53% | 87% | 90% | 90% | 90% | 99% |
| PHONE | 31% | 78% | 88% | 89% | 87% | 93% |
| Overall | 45% | 73% | 93% | 87% | 88% | 94% |
Two results clear the 5-point threshold here, and they are the two worth acting on.
Nova Pro at 94% and the local mistral:7b hybrid at 93% are tied, and both sit above Comprehend's 87%. A managed cloud model and a 4.1GB laptop model land in the same place. Which one you pick is a residency and latency question rather than an accuracy question.
The financial ID gap is the sharpest single number in the benchmark: Nova Pro at 96% against Comprehend at 85%, eleven points on the type that carries IBANs, tax IDs, and social insurance numbers. If your German workflows are financial, that row is the argument for paying Nova Pro rates.
Everything else on this table is a tie. Nova Micro at 88% and Comprehend at 87% are indistinguishable, which is the point: Nova Micro reaches Comprehend's accuracy at roughly a twentieth of the cost, rather than exceeding it. Nova Lite came in at 87%, level with both.
Read the Comprehend column with one thing in mind: Amazon Comprehend supports English and Spanish. German is not on its list. 87% on a language it does not advertise is a strong showing, and it is why Comprehend stays on the table for German traffic when 100ms and zero tuning matter more than the gap. Language coverage is a selection criterion in its own right, and for German the Nova models and the open weights currently have more of it.
mistral:7b is 4.1GB. No GPU. Zero egress.
Why the German-specialized model scored lower
We tested llama3-german-8b, fine-tuned explicitly for German. It scored below mistral:7b.
Fine-tuned for German conversation means the model learned to respond in German, which includes translating English input into German. Given "Please ship the order to Dr. Emily Watson, 742 Evergreen Terrace, Springfield, IL 62704," it rewrote the sentence in German instead of replacing the name and address with tags. That is the worst available output: structure changed, PII intact.
mistral:7b treats "return only the redacted text" as a format instruction. Conversation-tuned models treat it as an invitation. For constrained extraction, a general instruction-following model beats a domain specialist, because the specialization creates exactly the wrong generation bias.
What the benchmark does not cover
Precision. Every table here answers whether the PII value disappeared. None answer how much clean text got destroyed on the way. Our false positive evidence is anecdotal: Presidio's "quarterly", mistral's one invented tag, clean baselines from Nova, Claude, and Comprehend on 15 cases.
Nova Pro's 99% on DATE_TIME is the number to be suspicious of. A model tagging dates that aggressively will also tag things that only look like dates, and this benchmark has no way to see that.
Nova also has no English exact-value figure, so the cost saving on English traffic is unvalidated. That is the next run.
Cost comparison
Three different meters sit in this table, so the comparison only means something once you fix a document size. And one cost is easy to forget: the redaction instruction prompt rides along on every token-priced call. Ours is 102 tokens, so a 200-token document is a 302-token input. Comprehend never pays that, because there is no prompt to send.
Per 100,000 short documents (200-token body, 302-token input, 200-token output, about 800 characters):
| Approach | Bills by | Cost |
|---|---|---|
| Presidio (local) | nothing | $0, compute only |
| BERT NER (local) | nothing | $0, compute only |
| mistral:7b (local) | hour | $0 marginal, $116/mo EC2 fixed |
| Nova Micro | token | ~$3.90 |
| Nova Lite | token | ~$6.60 |
| Bedrock Mistral 7B | token | ~$8.50 |
| Lambda (10GB, 2s inference) | GB-second | ~$33 |
| Amazon Comprehend | 100 characters | ~$80 |
| Nova Pro | token | ~$88 |
| Claude Haiku 4.5 | token | ~$130 |
Two mistakes are easy to make here.
Pricing a redaction call off the input side alone understates it by four to five times, because output rates run three to five times input rates and a redaction response is roughly as long as its input. Output dominates every token-priced row.
Comprehend bills per 100 characters with a three-unit minimum, so its cost scales from a $0.0003 floor. That floor is where the widely quoted $30 per 100k comes from, and it applies only to strings under 300 characters. An 800-character ticket is eight units: $80 per 100k. Comparing that floor against a real token-priced call is how you end up thinking Comprehend is cheap.
Document length then moves the rows at different speeds:
| Approach | Short document | Contract length (2,000 tokens) |
|---|---|---|
| Nova Micro | ~$3.90 | ~$35 |
| Nova Lite | ~$6.60 | ~$61 |
| Bedrock Mistral 7B | ~$8.50 | ~$72 |
| Amazon Comprehend | ~$80 | ~$800 |
| Nova Pro | ~$88 | ~$808 |
| Claude Haiku 4.5 | ~$130 | ~$1,210 |
| EC2 m7g.xlarge | $116/mo flat | $116/mo flat |
The local option is the only row that does not move. An m7g.xlarge costs $116 a month whether it handles 10,000 documents or 500,000, so break-even against Comprehend sits near 145,000 short documents a month and drops to about 14,500 if your documents are contract-length. Long documents are the case for running your own inference, and the case gets stronger the longer they get.
What prompt caching does and does not fix
The instruction prompt is identical on every call, so caching it looks like the obvious optimization. Bedrock supports prompt caching for Nova Micro, Lite, Pro, and Premier and for several Claude models, with cache reads at roughly 10% of the normal input rate and no write premium on Amazon's own models.
It does nothing for a 102-token prompt. The prefix has to reach 1,024 tokens for Nova and 2,048 for Haiku before a checkpoint is created, and below that the cache point is ignored silently, with no error. Even if it applied, the saving would be around 8% of the call, because output dominates and the prompt is only a third of input.
What caching changes is which prompt you can afford to write. A 1,100-token prompt with few-shot examples costs roughly twice the short version at full price, and almost exactly the same once cached:
| 102-token prompt | 1,100 uncached | 1,100 cached | |
|---|---|---|---|
| Nova Micro | $3.86 | $7.35 | $3.89 |
| Nova Pro | $88.16 | $168 | $88.80 |
| Claude Haiku 4.5 | $130.20 | $230 | $131 |
That reframes the accuracy problem. The weakest results in this benchmark are the contextual cases, where Nova Micro scored 1/4 on an implied-identity bio. Few-shot examples of implied identity, German address and naming conventions, and an explicit entity taxonomy are the standard fix, and they are what pushes a prompt past 1,024 tokens. Caching means that fix costs nothing per call after the first.
Picking one
| Requirement | Approach |
|---|---|
| Ad-hoc, low volume | Frontier model in whatever subscription you have, via a skill |
| On-premise only, English | Hybrid (llama3.1:8b), 83% |
| On-premise only, German | Hybrid (mistral:7b), 93% |
| Air-gapped, no inference runtime | BERT NER (Ar86Bat/multilang-pii-ner) |
| Cloud, German, cost-sensitive | Nova Micro, 88%, ~$3.90/100k |
| Cloud, German, best accuracy | Nova Pro, 94%, ~$88/100k |
| Cloud, German financial identifiers | Nova Pro, 96% on FINANCIAL_ID |
| Latency-sensitive path, nothing to tune | Amazon Comprehend, 100ms |
| Documents over ~2,000 tokens at volume | Self-hosted, break-even near 14,500/month |
| GDPR EU residency | Container in EU region, or any Bedrock model in eu-central-1 |
Model size does not predict accuracy on this task. A 4GB local model and a managed AWS model tie on German at 93-94%. The cheapest Nova model scored highest on the curated set. The cheaper Claude variant matched the more expensive one. Redaction is extraction and substitution, and models optimized for generation bring capability the task cannot use.
Which means the model is the easy decision. Pick one from the table, then measure precision on your own corpus before it goes anywhere near a request path, because everything above is recall and a layer that catches 94% of PII while eating 4% of your product names will be switched off inside a week.
The harder part is the pipeline around it: the order the passes run in, what happens to HTML, what a refusal does to your throughput, and whether your placeholders are reversible. That is the next piece.
Read it next: Nothing throws when redaction fails
*Benchmarks run on a MacBook Pro M3. Dataset: ai4privacy/pii-masking-400k validation split, 100 cases per language. Real-world figures are exact-value recall. Curated benchmark (15 cases) uses tag-match scoring and runs roughly 10 points optimistic. LLM non-determinism introduces ±3% variance across runs; differences below 5 points are treated as ties. Bedrock Mistral 7B recall is extrapolated from Ollama (same weights, different endpoint). Claude models were run on the curated set only. Cost figures assume a 200-token body plus a 102-token instruction prompt (302 input tokens), 200 output tokens, about 800 characters, against published on-demand rates: Nova Micro $0.035/$0.14, Nova Lite $0.06/$0.24, Nova Pro $0.80/$3.20, Claude Haiku 4.5 $1.00/$5.00 per 1M tokens; Amazon Comprehend $0.0001 per 100-character unit with a three-unit minimum. Token-priced and character-priced options both scale with document length, so recompute for your own average before budgeting. Models tested: llama3.1:8b, llama3-german-8b, mistral:7b, llama3.2:3b via Ollama; dslim/bert-base-NER via transformers; Amazon Comprehend DetectPiiEntities in eu-central-1; Claude Haiku 4.5 and Sonnet 4.6 (accessed through Kiro CLI; the same models are available on Bedrock and the approach is unchanged); Amazon Nova Micro, Lite, and Pro via the Bedrock Converse API using eu-central-1 inference profiles.
Top comments (0)