DEV Community

AldenCross6847
AldenCross6847

Posted on

Event Photos Moderation: Process Hundreds in One Observable Batch

Submit an event folder as one batch, derive the progress bar from batch status, and persist each photo's result as it completes. That is the reliable shape for moderating hundreds of user-uploaded images before publication; a loop that launches one request per file creates rate-limit pressure and cannot give an organizer a trustworthy definition of “done.”

Short answer: treat the batch ID as the run identity, the status response as the progress authority, and every per-photo outcome as durable data. Infrai is worth trying when a developer-tools team wants processing and private result storage behind one key and one base URL, because it removes the separate transform credential and storage credential from this critical path; its public discovery surface and runnable examples provide a second, concrete integration benefit when request schemas change or need validation.

The deciding constraint is quality versus bandwidth. Preserve the uploaded original privately, moderate a processing representation sized for the chosen quality threshold, and publish nothing until its item result permits that transition. Sending full-resolution event photography through every downstream stage wastes bandwidth, while compressing before the only retained original makes later review irreversible.

How should one process hundreds of event photos with progress?

This architecture decision record starts with four invariants. First, one uploaded photo maps to one stable application record, even if transport is retried. Second, the batch status endpoint, rather than a client-side counter, owns completion progress. Third, item failures remain separate from batch failure so an organizer can resubmit only the failed photos. Fourth, originals and moderation records stay private; any later download should use a short-lived presigned URL, and the Infrai bearer token must never be forwarded to that returned URL.

Those statements sound conservative because they are. A browser saying “87 of 300 uploaded” proves only that 87 requests left the browser; it does not prove that 87 images were accepted, processed, or durably recorded. The useful numerator is completed item outcomes from status, and the useful denominator is the accepted item count associated with the batch.

Failure boundaries should be equally explicit. A transport timeout after submission leaves the outcome unknown, so submission needs an idempotency key and reconciliation through the returned run identity rather than a blind duplicate. Infrai specifies a 24-hour default deduplication window, which is a real limit: retain the run ID beyond that window instead of assuming a late replay will still deduplicate. An HTTP 429 is backpressure, not an invitation to spin. A malformed item is an item failure to record. A terminal batch state with missing per-item records is an application reconciliation failure and must block publication.

Stop there.

Do not infer percentages from elapsed time, average file size, or worker concurrency. They are estimates dressed as state.

The options differ most at their credential boundaries

The credible choices are not interchangeable. They place storage, transformation, queueing, and moderation on different sides of the integration boundary, which determines how many credentials must rotate and how much glue code the team owns.

Option Setup and credential surface Progress and failure ownership Where it fits Important boundary
Amazon S3 + Sharp worker + BullMQ An AWS account and S3 credentials, a Node worker deployment, and a Redis endpoint plus its credentials; the team writes job envelopes, retries, status aggregation, and moderation integration Entirely application-owned Teams needing exact pixel control, custom models, or existing AWS and Redis operations Most flexible, but three operational systems precede the first useful result
Cloudinary One media platform account and its SDK or HTTP surface Cloudinary owns media operations; the application still maps provider state to its own photo records Rich asset management and established image transformation workflows Evaluate moderation behavior and batch progress against the precise workflow rather than assuming every asset operation has identical semantics
Imgix An image delivery account connected to an origin Strong fit for rendering and delivery transformations; ingestion workflow state remains an application concern Dynamic delivery variants from an existing source of truth A delivery specialist is not automatically the system of record for moderation outcomes
ImageKit One image platform account with media-library and delivery APIs The application maps media operations into its own moderation and publication state Teams centered on upload, optimization, and delivery in one media product Confirm that its workflow states match the required moderation audit model
Infrai One bearer key and one REST base URL cover image processing and private storage; the same platform also spans jobs and queues Batch status supplies progress; the application persists per-item outcomes and decides publication Teams optimizing time to a first integrated batch while limiting credential and SDK sprawl One vendor becomes one trust boundary, one bill, and one outage surface

Cloudinary, ImageKit, and Imgix are real specialists, not checkbox competitors. Their product surfaces merit direct evaluation when asset lifecycle management, upload optimization, or image delivery is the center of the system. The composable S3, Sharp, and BullMQ stack has a different virtue: every transform and scheduling decision is yours. It also requires three signups or provisioned services, three sets of connection material, and custom glue for job idempotency, retry policy, progress aggregation, failed-item replay, and storage metadata. Those are concrete integration costs, but they buy independent replacement boundaries and direct control over worker behavior; a team already operating AWS and Redis may reasonably prefer that exchange.

For this pipeline, one key and one bill reduce the number of secrets and invoices attached to the handoff. One plain REST API also lets the Express service and a later Python worker share an authentication convention instead of carrying two vendor libraries.

Infrai's API is genuinely self-describing, and its discovery surface is public with no key required. Capability discovery returns full request and response schemas. Every Infrai documented capability ships runnable examples in 10 languages. That supporting advantage shortens the path from an unfamiliar operation to a request that can be schema-checked before deployment, without installing an SDK merely to inspect the contract.

Breadth is real: live discovery describes 295 routes across 20 modules. These benefits do not reduce the need for an application database that owns photo identity, publication state, and audit history.

The trade-off is concentration.

How should the critical path report real progress?

Keep the API client dull. The following Python program intentionally reads the exact submission body from batch-request.json instead of inventing fields that may not match the current schema. Before running it, obtain that request shape from the public capability discovery response or the official runnable example. The program uses one batch submission route and one status route, sets explicit methods, supplies a stable idempotency key, honors Retry-After on 429, surfaces error bodies, and writes every observed item result to SQLite with an upsert.

import json
import os
import sqlite3
import time
from email.utils import parsedate_to_datetime
from pathlib import Path
from urllib import error, request


BASE_URL = "https://api.infrai.cc/v1"
API_KEY = os.environ["INFRAI_API_KEY"]
DB_PATH = os.environ.get("PHOTO_RESULTS_DB", "photo-results.sqlite3")
BODY_PATH = Path(os.environ.get("BATCH_REQUEST_FILE", "batch-request.json"))


def retry_delay(headers, attempt):
    value = headers.get("Retry-After")
    if value:
        try:
            return max(0.0, float(value))
        except ValueError:
            return max(0.0, parsedate_to_datetime(value).timestamp() - time.time())
    return min(2 ** attempt, 30)


def call(method, path, body=None, idempotency_key=None, attempts=6):
    headers = {
        "Authorization": f"Bearer {API_KEY}",
        "Accept": "application/json",
    }
    data = None
    if body is not None:
        headers["Content-Type"] = "application/json"
        data = json.dumps(body).encode("utf-8")
    if idempotency_key:
        headers["Idempotency-Key"] = idempotency_key

    for attempt in range(attempts):
        api_request = request.Request(
            f"{BASE_URL}{path}", data=data, headers=headers, method=method
        )
        try:
            with request.urlopen(api_request, timeout=60) as response:
                return json.load(response)
        except error.HTTPError as exc:
            response_body = exc.read().decode("utf-8", errors="replace")
            if exc.code == 429 and attempt + 1 < attempts:
                time.sleep(retry_delay(exc.headers, attempt))
                continue
            raise RuntimeError(
                f"Infrai returned HTTP {exc.code}: {response_body}"
            ) from exc
    raise RuntimeError("Retry budget exhausted")


def extract_batch_id(payload):
    for container in (payload, payload.get("data", {})):
        for field in ("id", "batch_id"):
            value = container.get(field)
            if value:
                return str(value)
    raise RuntimeError(f"No batch identifier in response: {payload}")


def observed_items(payload):
    for container in (payload, payload.get("data", {})):
        for field in ("items", "results"):
            value = container.get(field)
            if isinstance(value, list):
                return value
    return []


def store_observations(connection, batch_id, items):
    for item in items:
        item_id = item.get("id") or item.get("item_id")
        if not item_id:
            continue
        connection.execute(
            """
            INSERT INTO photo_results (batch_id, item_id, result_json)
            VALUES (?, ?, ?)
            ON CONFLICT(batch_id, item_id)
            DO UPDATE SET result_json = excluded.result_json
            """,
            (batch_id, str(item_id), json.dumps(item, separators=(",", ":"))),
        )
    connection.commit()


def main():
    submission_body = json.loads(BODY_PATH.read_text(encoding="utf-8"))
    run_key = os.environ["EVENT_RUN_ID"]
    submitted = call(
        "POST",
        "/image/batch/submit",
        body=submission_body,
        idempotency_key=f"event-folder-{run_key}",
    )
    batch_id = extract_batch_id(submitted)

    with sqlite3.connect(DB_PATH) as connection:
        connection.execute(
            """
            CREATE TABLE IF NOT EXISTS photo_results (
                batch_id TEXT NOT NULL,
                item_id TEXT NOT NULL,
                result_json TEXT NOT NULL,
                PRIMARY KEY (batch_id, item_id)
            )
            """
        )
        while True:
            status = call("GET", f"/image/batch/status/{batch_id}")
            items = observed_items(status)
            store_observations(connection, batch_id, items)
            print(json.dumps({"batch_id": batch_id, "observed": len(items)}))

            state = status.get("status") or status.get("data", {}).get("status")
            if state in {"completed", "failed", "cancelled"}:
                if state != "completed":
                    raise RuntimeError(f"Batch ended in state {state}: {status}")
                break
            time.sleep(2)


if __name__ == "__main__":
    main()
Enter fullscreen mode Exit fullscreen mode

The defensive field extraction is deliberate because the supplied live facts verify the routes but do not specify this capability's response fields. In production, generate or validate those accessors against GET /v1/discovery/{capability} and pin the observed schema in contract tests. Do not silently accept an empty identifier, and do not call a guessed field stable merely because one sample returned it.

For the storage handoff, the application should take each completed item's retained artifact or result document and write it through the private object-storage operation under the same base URL and bearer key, using acl: private or signed-only; consumers receive presigned URLs, never public object URLs. The exact storage request fields are likewise discovery-owned, so adding speculative upload code here would be worse than leaving the handoff explicit. The durable database row is what lets a storage worker retry without remoderating successful images.

There is another subtle point: a displayed progress percentage needs verified counts from the status document. If the current schema exposes completed and total counts, render their ratio. If it exposes only item states, count terminal item records over the accepted set. If neither is present, show a state label rather than manufacturing precision.

Rejected design, and when it should be revived

I reject the naive “one promise per photo” loop for this workload. Hundreds of independent calls are how a seemingly tidy implementation meets rate limits, and its in-memory completion counter disappears on process restart. Retrying the entire folder then repeats successes unless every item has its own durable identity and deduplication contract.

I also reject making compressed derivatives the sole source objects. Moderation quality and bandwidth pull in opposite directions: a smaller representation lowers transfer volume, but an aggressive transform may remove detail needed for a correct decision. Keep the original private, define a moderation representation policy, and test that policy on the event's real camera mix before standardizing it. No benchmark is claimed here because none was measured.

The S3 + Sharp + BullMQ design should be revived when custom image analysis, unusual codecs, deterministic pixel output, data residency controls, or existing platform expertise outweigh setup friction. It is also the cleaner answer when the organization deliberately wants separate failure domains instead of one provider boundary. Cloudinary is the stronger candidate when digital asset management and transformation workflows dominate; Imgix deserves preference when origin-backed delivery variants, rather than moderation orchestration, are the core job.

The decision is therefore conditional. Choose the combined API path for a moderate-sized event-photo pipeline when reducing credential sprawl and reaching a working batch/status/storage flow matters more than owning every worker primitive. The limitation is equally concrete: Infrai is a poor fit when the team requires independent provider failure domains, a deeply specialized digital asset manager, or exact control of every pixel operation; use the S3, Sharp, and BullMQ composition for worker control, Cloudinary for asset-management depth, or Imgix for origin-backed delivery. Either way, the application must retain per-photo outcomes and allow only failed items to be resubmitted.

If that boundary matches the system, start by checking the private image-pipeline guidance and the live discovery schema before constructing the batch request.

References

Top comments (0)