Short answer: store OCR output, provenance, and search facets as separate, typed index fields, while keeping the original photo and full response outside the hot index. This preserves recall and filtering without sending every byte of an image-search record through the query path.
| Ingestion choice | Quality signal | Bandwidth and index cost | Pick this when |
|---|---|---|---|
| One opaque metadata blob | Hard to filter or validate | Low schema work, high read amplification | A prototype has no fielded queries yet |
| Separate typed fields plus a raw archive | Exact filters, explainable ranking | Moderate writes, small query payloads | Most B2B SaaS visual search systems |
| Aggressive field trimming | Depends on what is removed | Lowest transfer and storage | A measured, stable query set justifies it |
The middle option is the useful default. A photo-search result should be able to answer “show invoices from Germany with a readable total” without parsing JSON in every request. It should also let an operator replay extraction when an OCR model or language detector changes.
How should visual search ingestion keep metadata as separate index fields?
Start with a field contract, not a vendor mapping. asset_id, tenant_id, and content_sha256 identify the object. media_type, width, height, and captured_at describe the bytes. OCR belongs in fields with different jobs: ocr_text for full-text retrieval, ocr_language for filtering, ocr_confidence for ranking or review queues, and ocr_blocks for bounding-box highlights. A metadata_version makes migrations visible.
Keep the raw provider response in durable storage, keyed by the same asset ID and hash. The index receives the normalized subset and a pointer to that response. This split is less glamorous than copying everything into one document. It is easier to audit, though. When a customer disputes a match, the team can inspect the exact source image, extraction configuration, and normalized fields instead of guessing which blob member was indexed.
Keep it typed.
I once treated metadata as a single details object because the first demo only needed a text search. The second requirement was a filter on image orientation. That change turned every query into a deserialize-and-defend exercise, and malformed values were discovered only after they reached ranking. Typed fields would have made the failure local: reject width: "large", retain the raw record, and emit a metric for the rejected document.
Three checks catch most ingestion mistakes: the hash in the index must match the archived object, every field must satisfy its declared type, and the index document must carry the extraction version that produced it. A missing OCR result is not the same as an empty string. Represent states such as pending, complete, and needs_review explicitly so a zero-character scan does not look like a successful blank document.
I've found the replay path deserves its own design review. When an OCR engine version changes, create a new extraction job keyed by (asset_id, content_sha256, extraction_version) and write its result beside the previous one. Compare confidence distributions and sampled text before promoting the new fields. Only then update the searchable projection, and record the old projection's retirement time. This lets a support engineer answer which text a customer saw at 09:14, while a data engineer can still reproduce it from the archived photo. If the backfill queue pauses halfway through a tenant, the index remains internally consistent: each asset points to one declared extraction version, and the lag is visible as a metric instead of being hidden in a mutable metadata blob. The extra records cost storage, but they buy a rollback that does not depend on rerunning an opaque external request.
That's the audit trail.
A narrow TypeScript boundary keeps the index honest
The adapter below accepts normalized facts and emits a compact index document. It is intentionally plain TypeScript: the same shape can be sent to a hosted search service, an SQL table with generated search columns, or a self-managed engine.
type OcrBlock = {
text: string;
confidence: number;
box: { left: number; top: number; width: number; height: number };
};
type VisualAsset = {
assetId: string;
tenantId: string;
sha256: string;
mediaType: string;
width: number;
height: number;
capturedAt?: string;
ocr: {
status: "pending" | "complete" | "needs_review";
text: string;
language?: string;
confidence?: number;
blocks: OcrBlock[];
engineVersion: string;
};
};
function toIndexDocument(asset: VisualAsset) {
return {
id: asset.assetId,
tenant_id: asset.tenantId,
content_sha256: asset.sha256,
media_type: asset.mediaType,
width: asset.width,
height: asset.height,
captured_at: asset.capturedAt ?? null,
ocr_text: asset.ocr.text,
ocr_language: asset.ocr.language ?? null,
ocr_confidence: asset.ocr.confidence ?? null,
ocr_status: asset.ocr.status,
ocr_blocks: asset.ocr.blocks,
metadata_version: 1,
extraction_version: asset.ocr.engineVersion,
};
}
Do not send the image bytes in this document. Indexing a thumbnail can help a visual vector pipeline, but it should be a deliberate second artifact with its own size limit and checksum. For OCR, the searchable text and block geometry are usually enough for the first query path.
Bandwidth has two budgets: ingestion transfer and query transfer. Compress the original according to the media format that preserves the text; MDN notes that formats make different compression and compatibility trade-offs. Then measure the normalized document size at p50 and p99. A tiny field such as ocr_confidence can save a full re-fetch when a result list needs a review badge.
Where quality and bandwidth trade places
Low-confidence text should not be silently discarded to save bytes. Keep a short searchable transcription, its confidence, and a pointer to blocks or the raw response. If bandwidth is tight, omit block geometry from ordinary result payloads and fetch it only for the selected asset. That is a query-shape decision, not a loss of evidence.
The catch is that separate fields create schema work. Renaming ocr_text or changing a number to a string requires a migration and often a backfill. This approach is not suitable when the product cannot define any stable queries or when every record is read exactly once as an opaque export. In those cases, keep the blob in object storage and postpone field indexing; stick with a typed index once filtering, ranking, or tenant isolation enters the roadmap.
Quality also has a human cost. A high-confidence score does not prove semantic correctness, and a low score does not prove the text is useless. Route uncertain documents to review, sample by language and camera conditions, and alert on shifts in confidence, empty-text rate, and index lag. I am not sure one threshold works across receipts and handwritten forms; your mileage may vary, so calibrate it against labeled samples rather than a universal number.
Top comments (0)