Short answer: treat every scan as an auditable record, keep the original bytes immutable, and make language and metadata decisions reviewable before OCR output is promoted. The deciding constraint is whether an operator can explain what happened to a source image after a mixed-language batch fails.
I have been paged for missed jobs and duplicate deliveries in production. The same smell appears in document pipelines: a retry creates two thumbnails, cleanup removes the only copy an annotator needed, or a guessed locale silently changes the OCR model. A reviewable source gives the on-call engineer a stable starting point.
Keep it boring.
What should a scan record contain before OCR?
Store the original object under a content-derived identifier and write a separate inspection record. Include a cryptographic digest, byte length, observed media type, decodable dimensions, capture timestamp if supplied, declared language hints, and status history. Preserve a user language tag beside the detector result with confidence and detector version.
A media type header is a hint, not proof. File extensions are weaker still. Decode with a bounded parser, reject formats outside policy, and retain the reason. MDN's media-format guide maps browser-facing formats, while your service still needs an allowlist and limits for pixel count, decompressed size, and page count.
The inspection record points to immutable source bytes, never a temporary path. That makes a human review link deterministic across retries.
How can multilingual scans, metadata inspection, and reviewable images share one workflow?
Use a small state machine: received, inspected, awaiting-review, OCR-ready, quarantined, and archived. A worker advances a record only when the previous transition is idempotent. Queue messages carry the source identifier; consumers fetch the same bytes and compare the digest before doing work.
Here is a compact Go sketch for the inspection boundary. It returns data for a review UI instead of hiding decisions in a worker log.
package inspect
import (
"crypto/sha256"
"fmt"
)
type Source struct {
ID string
MediaType string
Bytes int64
DeclaredLang string
Digest [32]byte
}
func Inspect(id string, data []byte, mediaType, declared string) (Source, error) {
if len(data) == 0 { return Source{}, fmt.Errorf("empty source") }
return Source{ID: id, MediaType: mediaType, Bytes: int64(len(data)), DeclaredLang: declared, Digest: sha256.Sum256(data)}, nil
}
Language detection runs after media inspection and before model selection. For a page containing Arabic and French, record both labels if the detector supports a ranked list; otherwise mark the result uncertain and route it to review. A reviewer sees the original image, extracted metadata, and exact pause reason together.
The human gate belongs inside that workflow, and it should react to disagreement rather than one arbitrary confidence threshold. Signals include declared-versus-detected language mismatch, orientation metadata with unexpected dimensions, a digest mismatch on retry, and a sudden per-page byte-size change. A 422-style validation result can be persisted as a quarantine reason; it should not vanish into an exception counter. In a runbook, each signal gets an owner and a next action: mismatch means compare the two language fields, a digest change means stop downstream side effects, and a decoder surprise means preserve the bytes for a second parser. That mapping matters during a page because an alert without a safe action invites improvisation. The review queue should also show whether the item is blocked before or after OCR, so an operator never replays a side effect just to inspect an image.
I once assumed retries were harmless because OCR was read-only. That failed at the edges: thumbnail creation and review-ticket insertion were side effects, so duplicate delivery produced duplicate work. Put an idempotency key on each side effect, derived from source digest plus operation name, and expose it in the audit record.
Your mileage may vary on language confidence. A threshold that works for clean passports can be wrong for receipts under fluorescent light. Keep it configurable, sample quarantined items, and revisit it from evidence instead of tuning during an incident.
How do you verify and roll back an inspection pipeline?
Verification starts with fixtures: rotated pages, EXIF orientation, right-to-left scripts, mixed alphabets, truncated files, and byte-identical uploads with different filenames. Assert digest stability, no second review item on retry, and source-link resolution after a worker restart. Check quarantine rate, review age, duplicate side effects, and bytes retained per completed page.
Roll back by stopping promotion from inspected to OCR-ready while leaving intake and immutable storage enabled. Operators can continue reviewing records, and replay resumes from the last transition after a parser or detector replacement. Never repair history by editing it; append a correction event naming actor, reason, and detector version.
The catch is operational load. A human gate is not suitable when millions of low-risk pages arrive with no review staff; use sampling there and reserve full review for high-impact classes. Stick with a simpler single-language flow when language metadata has no downstream effect.
Top comments (0)