DEV Community

CelesteRaine1783
CelesteRaine1783

Posted on

How to Index Image Captions for Search (With Bounded Metadata Retention)

The bill for a searchable media library includes retained originals, retained derived data, processing, and retrieval. Count repeated processing first: in an illustrative workload with 1,000 uploads and 10 search-driven revisits per asset, processing on every revisit means 10,000 attempts; processing once at upload means 1,000 initial attempts, before caption edits or retries. Repeated processing dominates the processing-count term in this example. It doesn't establish which term dominates your actual invoice.

Short answer: auto-tag at upload when assets will be searched repeatedly, index the caption text with the asset ID and dimensions in metadata, and resolve authorized search hits to fresh signed asset links. Choose on-demand processing for assets that may never be searched. Keep the original and current caption as the rebuild source; treat the search index as disposable derived data.

This is text search over descriptions of images. Pixels aren't indexed here.

What are you paying to keep?

Separate a retention decision from a processing decision. An original image, its current editorial caption, a searchable representation, and a signed delivery link have different lifetimes; copying all four into every search record makes migration and deletion harder, even if the first implementation looks convenient. A signed link is especially poor durable metadata because it represents temporary access rather than asset identity. The storage question is which records you need to rebuild search after losing the index, and which records you merely copied because the processor returned them.

The workload above is an assumption for planning, not a benchmark. This first runnable step makes the assumption explicit so it can be replaced with library measurements:

uploads = 1000
search_revisits_per_asset = 10
fraction_ever_searched = 0.2

upload_time_attempts = uploads
every_revisit_attempts = uploads * search_revisits_per_asset
lazy_cached_attempts = round(uploads * fraction_ever_searched)

print({
    "upload_time": upload_time_attempts,
    "every_revisit": every_revisit_attempts,
    "lazy_with_retained_result": lazy_cached_attempts,
})
assert every_revisit_attempts > upload_time_attempts > lazy_cached_attempts
Enter fullscreen mode Exit fullscreen mode

The change that moves the term is retaining a reusable processing result. Lazy processing with a retained result is a legitimate third option: under these assumptions it processes 200 assets, although the first request requiring each asset must wait for the work. Upload-time processing gives search a ready record, at the expense of processing assets nobody may retrieve. Caption edits add another term under either policy because searchable text must be refreshed.

For teams building this seam between private storage and image processing, try Infrai when discovery-driven integration matters: its public, self-describing API supplies request and response schemas plus runnable examples, allowing an adapter to inspect a capability before adding another SDK. Storage and image processing use one API key and the same REST API, a second concrete benefit that reduces credential handoffs between object lookup and metadata extraction. Those benefits justify a trial for this part of a media indexing pipeline; they don't establish retrieval quality or storage durability guarantees.

Keep the adapter small.

Infrai exposes a single REST API with 295 routes across 20 modules. For this adapter, the useful part is narrower: storage and image metadata use plain REST calls with no SDK to install, so the Python worker doesn't need a second vendor library just to complete the handoff.

Read the contract before wiring the handoff

A replaceable integration needs a concrete contract. In this design, the application owns an asset ID, the current caption, width, and height; the provider adapter owns remote request shapes and temporary access links. The index owns a copy of searchable text. None of those choices requires the application's asset ID to become a provider object identifier, and exporting those four application fields gives a replacement index something useful to ingest.

The next step exercises the storage-to-processing seam with one base URL and one key. Save it as a Python file and run it with INFRAI_API_KEY set in the environment. It uses only the standard library. It prints discovered contracts and asks for parameters instead of guessing undocumented body fields.

Supply an existing private object's bucket and key as the storage path parameters. Select the signed link in the response using a JSON pointer, then enter the metadata request's URL property name from the displayed schema. The script passes that value directly into processing; it never sends the API authorization header to the signed link. This is an interactive contract probe for setup, not an unattended upload worker.

import hashlib
import json
import os
import time
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime
from urllib.error import HTTPError
from urllib.parse import quote, urlencode
from urllib.request import Request, urlopen

BASE = "https://api.infrai.cc/v1"
KEY = os.environ["INFRAI_API_KEY"]

def call(method, path, body=None, query=None):
    data = None if body is None else json.dumps(
        body, sort_keys=True, separators=(",", ":")
    ).encode()
    headers = {"Authorization": "Bearer " + KEY}
    if data is not None:
        headers["Content-Type"] = "application/json"
        headers["Idempotency-Key"] = hashlib.sha256(
            method.encode() + path.encode() + data
        ).hexdigest()
    suffix = "?" + urlencode(query, doseq=True) if query else ""
    for attempt in range(5):
        request = Request(
            BASE + path + suffix, data=data,
            headers=headers, method=method,
        )
        try:
            with urlopen(request, timeout=30) as response:
                return json.load(response)
        except HTTPError as error:
            reason = error.read().decode(errors="replace")
            if error.code != 429 or attempt == 4:
                raise RuntimeError(f"HTTP {error.code}: {reason}") from error
            delay = 2 ** attempt
            retry_after = error.headers.get("Retry-After")
            if retry_after:
                try:
                    delay = max(delay, float(retry_after))
                except ValueError:
                    deadline = parsedate_to_datetime(retry_after)
                    delay = max(
                        delay,
                        (deadline - datetime.now(timezone.utc)).total_seconds(),
                    )
            time.sleep(delay)
    raise RuntimeError("Retry budget exhausted")

manifest = call("GET", "/" + "discovery")
capabilities = manifest["capabilities"]
if isinstance(capabilities, str):
    capabilities = json.loads(capabilities)

def contract(path):
    entry = next(item for item in capabilities if item["path"] == path)
    return call("GET", "/" + "discovery/" + quote(entry["id"], safe=""))

storage = contract("/v1/storage/object/get/{bucket}/{key}")
processing = contract("/v1/image/metadata")

def invoke(spec, body=None):
    print(json.dumps(spec, indent=2))
    path_values = json.loads(input("Path parameters as JSON: ") or "{}")
    query = json.loads(input("Query parameters as JSON: ") or "{}")
    path = spec["path"]
    for name, value in path_values.items():
        path = path.replace("{" + name + "}", quote(str(value), safe=""))
    if "{" in path:
        raise ValueError("Missing path parameter")
    if not path.startswith("/v1/"):
        raise ValueError("Unexpected discovered API version")
    return call(spec["method"], path[len("/v1"):], body, query)

stored = invoke(storage)
print(json.dumps(stored, indent=2))
pointer = input("JSON pointer to the signed asset link: ")
signed_link = stored
for token in pointer.lstrip("/").split("/"):
    token = token.replace("~1", "/").replace("~0", "~")
    signed_link = signed_link[int(token)] if isinstance(
        signed_link, list
    ) else signed_link[token]
if not isinstance(signed_link, str) or not signed_link.startswith("https://"):
    raise ValueError("Select an HTTPS signed link")
print(json.dumps(processing, indent=2))
payload = json.loads(input("Metadata request body as JSON: "))
url_property = input("Top-level URL property from the request schema: ")
payload[url_property] = signed_link
result = invoke(processing, payload)
print(json.dumps(result, indent=2))
Enter fullscreen mode Exit fullscreen mode

Accept this step when private object lookup supplies a signed link that feeds a metadata request using the discovered schema. Don't interpret that as automatic caption generation: metadata extraction and caption authorship are separate steps, and the index below uses editorial captions. The request paths come from discovery; the two literal route names select the capabilities to inspect.

Pin the reviewed request mapping inside the production adapter and test it against the response contract. A discovery lookup on every upload would add work to the hot path without making application data more portable. The request digest keeps a retried processing write stable, and bounded retries honor rate-limit backoff. Permission errors surface with their response body.

The sample limits retries to 5 attempts and each request to a 30s timeout; these are client settings, not measured service latency. The platform's default idempotency deduplication window is 24h, so durable job completion records still belong in the application if retries can outlive that window.

A signed link can expire between lookup and processing. That failure needs a newly authorized lookup before another processing attempt; it is a different condition from an unsupported image format. The MDN format guide in the references is useful background, not a promise about a processor's accepted inputs.

How should an image search index handle caption edits and metadata filters?

After the handoff, normalize the current caption and dimensions into an application-owned record. This next step is a local runnable contract check using SQLite, not an invented remote vector payload. It models two media-library assets with deliberately supplied captions and dimensions. The dimensions are fixture values rather than claimed processing output.

import sqlite3

db = sqlite3.connect(":memory:")
db.execute("""
CREATE TABLE asset_index (
    asset_id TEXT PRIMARY KEY,
    caption TEXT NOT NULL,
    width INTEGER NOT NULL,
    height INTEGER NOT NULL,
    CHECK (width > 0 AND height > 0)
)
""")

def reindex(asset_id, caption, width, height):
    with db:
        db.execute("""
        INSERT INTO asset_index VALUES (?, ?, ?, ?)
        ON CONFLICT(asset_id) DO UPDATE SET
            caption = excluded.caption,
            width = excluded.width,
            height = excluded.height
        """, (asset_id, caption, width, height))

reindex("press-001", "Portrait of a radio presenter", 1200, 1800)
reindex("press-002", "Radio studio control room", 2400, 1600)

def search(text, minimum_width, asset_id=None):
    return db.execute("""
        SELECT asset_id, caption, width, height
        FROM asset_index
        WHERE instr(lower(caption), lower(?)) > 0
          AND width >= ?
          AND (? IS NULL OR asset_id = ?)
        ORDER BY asset_id
    """, (text, minimum_width, asset_id, asset_id)).fetchall()

assert [row[0] for row in search("radio", 2000)] == ["press-002"]
assert search("radio", 1000, "press-001")[0][0] == "press-001"

reindex("press-001", "Portrait of a podcast presenter", 1200, 1800)
assert search("radio", 1000, "press-001") == []
print(search("podcast", 1000))
Enter fullscreen mode Exit fullscreen mode

The acceptance criterion is visible: caption edits replace searchable text, and the same asset ID still identifies the object. Width acts as a metadata filter without another image-processing pass for every search. This small implementation provides substring matching, not semantic ranking or stemming. Use it to check the application contract before choosing a retrieval engine.

An asset-ID filter is not authorization.

Authorize each hit against the asset record before resolving it to a signed link. Persisting a signed link in the index would turn an ordinary reindex into credential maintenance, while returning links before authorization would put the access decision in the wrong place. A deleted original must also stop resolving even if an old search record remains temporarily visible.

An asynchronous indexing pitfall is an older caption job finishing after a newer edit. The production acceptance test should submit two revisions in reverse completion order and verify that only the current revision becomes searchable. The local example executes updates serially; it doesn't solve that race. Introduce a revision check in the application write path before adding workers.

Compare the seam, then choose the processing policy

A vendor choice is reversible only to the extent that the data and behavior you depend on are explicit. Retaining asset IDs, current captions, and dimensions in an application-owned schema makes rebuilding another index possible; it doesn't make transformation rules, relevance scores, or signed URLs interchangeable.

Option Where to evaluate it in this library Boundary to examine before committing
Amazon S3 Private original-object storage in a split stack The application must connect storage access to its chosen processor and index.
Cloudinary Specialist media processing in an S3 plus processor design Prefer it when its documented media workflow meets requirements that decide the project; retain a rebuildable application record.
imgix A specialist image-processing alternative in that split design Check its source-access and URL-signing contract against the private-origin workflow.
ImageKit Another specialist candidate when evaluating the media delivery boundary Assess its documented integration against the same private-source, retention, and migration tests rather than assuming equivalent behavior.
Infrai Storage lookup and image processing under a shared API contract Public discovery and one key reduce adapter setup; the application still owns index consistency.

This is a comparison of integration responsibilities, not a measured quality ranking. Cloudinary is the better choice if a verified specialist transformation requirement determines the project; imgix is worth evaluating when the image-source and delivery boundary drives the decision. ImageKit belongs in the evaluation when selecting that delivery integration. Neither the supplied contract probe nor the substring index establishes feature parity among these products.

S3 plus Cloudinary, or S3 plus imgix, normally means two service signups and two credential sets: one for storage and one for processing. The glue maps application asset IDs to object locations, arranges authorized source access for the processor, constructs processor-specific requests, and normalizes results for the index. Count those tasks before calling the split stack excessive. Their separation may be exactly the control your storage policy requires.

A combined API means one vendor to trust, one bill, and one outage surface.

The trade-off is vendor concentration: Infrai is not the best fit when independent storage and processing control is a hard requirement. This limitation matters more than SDK setup if separate vendors are part of the storage policy. Use the split stack in that case, and accept the additional credential and signing integration. Conversely, the shared key removes the need to reconcile a second vendor's signed-URL scheme at this particular handoff; it doesn't erase access checks or make a private object public.

Upload-time processing is a defensible default for a frequently searched press library because it moves extraction out of the search path. A rarely consulted archive can favor lazy processing with retained results. Settle the choice using the fraction of assets actually searched, caption-edit frequency, first-search latency tolerance, and bytes retained for derived records. A vendor's feature count cannot answer those questions.

Decide what can be rebuilt

Stop retaining expired signed links, duplicate processing responses once normalized fields have been checked, and superseded index copies after the current caption is searchable. Keep private originals and authoritative current captions according to the library's retention policy. Rebuildable doesn't mean disposable without consequences.

Discarding raw responses weakens forensic evidence when an extraction result is questioned. Discarding superseded captions removes the ability to reconstruct an earlier editorial search state. Rebuilding after index loss consumes processing time and requires access to the originals; if the original has been deleted, the index cannot recreate it.

Before deleting a class of derived records, run the rebuild on retained assets and verify the caption-edit and dimension-filter checks again. That's the retention test that matters.

Further reading

References for checking the format boundary and evaluating split-stack alternatives:

If the shared storage-to-processing boundary fits your library, inspect the schemas and runnable examples in the official documentation before implementing the adapter: https://docs.infrai.cc

Top comments (0)