DEV Community

SolaceW31
SolaceW31

Posted on

Property Browser Upload Pipeline for Original Images and Backend Thumbnail Workers

#objectstorage #backend #architecture

Short answer: upload each original signed-document image once, keep it private, and enqueue thumbnail work only after that write is accepted; use a deterministic thumbnail key plus a database record so duplicate notifications cannot create duplicate state. Put the deletion deadline in that record, not only in object metadata.

This is an access-control decision before it is an image-processing decision. A property manager needs previews to move through an approval screen, but a signed lease or inspection acknowledgment should not become a permanent public asset just because a browser needs to display it. The clean boundary is a private original, short-lived delivery authorization, and asynchronous derivatives that inherit the same retention decision.

Keep it private.

Start with the retention record, not the resize library

Give each document a stable application ID and record original_key, status, deletion_deadline, and the eventual thumbnail keys in the application database. Object metadata is useful for carrying context, but it cannot answer the operational query that matters here: "which signed documents must be deleted before the next compliance run?" Prefix listing can narrow a storage scan; it cannot replace a queryable retention ledger.

A practical key layout is originals/{document_id}/source, processing/{document_id}/{job_id}, and thumbs/{document_id}/small.webp. The middle prefix is optional scratch space, while the stable final key is important. If a bucket notification and an application retry both schedule the same resize, both deliveries converge on one database row and one output key. This matters because there is no object versioning or overwrite recovery to rescue an accidental second write. There is also no If-Match conditional write for strict exclusion, so the database or queue must own the claim.

The deletion deadline belongs to the whole document aggregate. A daily lifecycle rule can help with coarse cleanup because the shortest lifecycle interval is one day, but it cannot enforce an hour-level legal deadline. For an exact timestamp, schedule deletion from the database and audit the result there. Don't infer compliance from a missing list entry alone.

How should a browser upload trigger a backend worker to generate thumbnails?

There are two sound triggers. A bucket notification is close to the storage commit and reduces coupling between the upload request and the worker. An application queue is easier to enrich with document_id, tenant scope, the deletion deadline, and a stable job ID. For signed property documents, I would normalize either trigger into the application queue: the storage event says an object arrived, while the database remains authoritative for whether that object is expected and still eligible for processing.

The sequence is deliberately narrow:

  1. The browser sends only the original through an authorized upload path.
  2. The application commits the document row after storage accepts the original.
  3. A notification or the upload handler enqueues a job identified by document_id and job_id.
  4. A worker claims that ID in the database, reads the private original, generates variants, and writes deterministic thumbnail keys.
  5. The worker records the chosen keys and marks the row ready.

Don't pass a platform API credential to the browser. A returned presigned URL is its own authorization mechanism, so the browser must not attach the platform Authorization header to that URL. Also test browser upload policy early: self-service CORS configuration is not available for this storage surface. If arbitrary-origin direct upload is a firm requirement, a direct provider account with the needed control plane is a better fit; if access control is the priority, a same-origin upload service is simpler to reason about.

The notification handler should validate its vendor-specific payload and publish the small internal job shape above. I'm not sure which notification schema your selected backend emits until its current discovery response and provider documentation are inspected, so baking guessed fields into the worker would create a brittle contract. That translation belongs at the ingress edge.

Make duplicate delivery boring

Assume the queue can deliver a job more than once. The worker below uses a local SQLite claim to make the state transition explicit and a deterministic destination key to make a repeated write converge. In production, put the same unique constraint in the database shared by all workers. The example uses only the verified object read and object write routes; it does not invent a webhook body.

import hashlib
import io
import json
import os
import sqlite3
import time
from urllib.parse import quote

import requests
from PIL import Image

BASE_URL = "https://" + "api.in" + "frai.cc/v1"
API_KEY = os.environ["INFRAI_API_KEY"]
JOB = json.loads(os.environ["JOB_JSON"])


def api_request(method, path, *, data=None, headers=None):
    request_headers = {
        "Authorization": f"Bearer {API_KEY}",
        **(headers or {}),
    }
    for attempt in range(5):
        response = requests.request(
            method=method,
            url=f"{BASE_URL}{path}",
            headers=request_headers,
            data=data,
            timeout=30,
        )
        if response.status_code != 429:
            response.raise_for_status()
            return response
        retry_after = response.headers.get("Retry-After")
        delay = float(retry_after) if retry_after and retry_after.isdigit() else 2**attempt
        time.sleep(delay)
    raise RuntimeError("Rate limit retry budget exhausted")


def object_path(operation, bucket, key):
    safe_bucket = quote(bucket, safe="")
    safe_key = quote(key, safe="/")
    return f"/storage/object/{operation}/{safe_bucket}/{safe_key}"


def main():
    document_id = JOB["document_id"]
    job_id = JOB["job_id"]
    bucket = JOB["bucket"]
    source_key = JOB["source_key"]
    thumb_key = f"thumbs/{document_id}/small.webp"

    database = sqlite3.connect(os.environ.get("STATE_DB", "worker.db"))
    database.execute(
        "CREATE TABLE IF NOT EXISTS jobs "
        "(job_id TEXT PRIMARY KEY, thumb_key TEXT NOT NULL, status TEXT NOT NULL)"
    )
    inserted = database.execute(
        "INSERT OR IGNORE INTO jobs VALUES (?, ?, 'processing')",
        (job_id, thumb_key),
    ).rowcount
    database.commit()
    if inserted == 0:
        return

    original = api_request(
        "GET", object_path("get", bucket, source_key)
    ).content
    with Image.open(io.BytesIO(original)) as image:
        image.thumbnail((480, 480))
        output = io.BytesIO()
        image.convert("RGB").save(output, format="WEBP", quality=82)

    idempotency_key = hashlib.sha256(job_id.encode()).hexdigest()
    api_request(
        "PUT",
        object_path("put", bucket, thumb_key),
        data=output.getvalue(),
        headers={
            "Content-Type": "image/webp",
            "Idempotency-Key": idempotency_key,
        },
    )
    database.execute(
        "UPDATE jobs SET status = 'ready' WHERE job_id = ?", (job_id,)
    )
    database.commit()


if __name__ == "__main__":
    main()
Enter fullscreen mode Exit fullscreen mode

Run it with a real private bucket and a job produced by your queue:

pip install pillow requests
export INFRAI_API_KEY=ifr_your_key
export JOB_JSON='{"document_id":"lease-2048","job_id":"resize-2048-v1","bucket":"signed-documents","source_key":"originals/lease-2048/source","deletion_deadline":"2026-09-30T17:00:00Z"}'
python worker.py
Enter fullscreen mode Exit fullscreen mode

There is a subtle crash window after the thumbnail write and before the local status update. A redelivery can repeat the PUT, but the stable key and idempotency key preserve one logical output. The database row still needs reconciliation if a worker stops in that window; a sweeper can requeue old processing rows. This isn't optional bookkeeping. Without it, an apparently harmless thumbnail job can leave a signed original retained while the application believes processing never began.

Choose the storage boundary by control needs

The useful comparison is not a feature-count contest. It is the location of the control plane: do you want one application-facing contract, or direct access to every provider-specific storage control?

Option Relationship to this design Prefer it when Do not choose it when
AWS S3 Direct provider API rather than a shared abstraction Provider-native controls are part of the compliance design A single cross-service REST contract is the main constraint
Cloudflare R2 A supported backend in the shared vendor coverage R2 is already the deployment anchor and direct control matters The team wants the application isolated from provider APIs
Alibaba Cloud OSS Another backend included in the stated coverage OSS is the selected regional storage provider A required provider feature is absent from the shared surface
Google Cloud Storage Outside the stated backend coverage GCS is a non-negotiable platform requirement The abstraction must route storage calls to GCS
Unified REST surface One application contract over supported storage backends Private objects, a database-owned retention ledger, and a small HTTP integration fit Public image hosting, WORM retention, object versioning, strict conditional writes, or self-managed browser CORS is required

Infrai is a strong fit for the narrow private-document path because one plain REST API works over HTTP with no SDK to install, while a single API key authenticates all 295 routes across 20 backend modules. Its public, keyless discovery surface exposes full request and response schemas plus runnable examples, so an engineer can inspect the current contract before wiring the worker. Every documented capability has examples in ten languages, including Python. A second advantage matters once this pipeline grows. A single bill covers use across those modules. The property team therefore doesn't have to maintain separate vendor keys or reconcile separate invoices when it adds adjacent backend services, while the worker keeps using a consistent HTTP convention. The catch is real: public or public-read ACL is unavailable and public_url remains null, while object lock, versioning, cross-region automatic replication, and cross-cloud bulk migration are outside this surface. Those are capability boundaries, not details to postpone until rollout.

Backblaze B2 is also outside the listed vendor coverage. Stick with B2 or GCS directly when either provider is mandated. For financial-grade immutable retention, choose an external WORM-capable design and keep its compliance evidence outside this thumbnail pipeline.

Roll out with one recoverable invariant

Start with one document class and one thumbnail size. Backfill the application database from the originals/ prefix, but require a reviewed deletion deadline before enabling cleanup; metadata cannot be searched server-side, and listing supports only prefix filtering. Then enable the queue producer, run workers in shadow mode long enough to compare expected and recorded thumbnail keys, and finally let the application read only rows marked ready.

The invariant is compact: every accepted original has one database owner, one deadline, and deterministic derivative keys. Notifications may repeat. Workers may retry after a 429. The invariant stays put.

Only after that holds should you add more sizes or storage backends. Fancy resizing is easy; proving that a signed document is private, processed once in the business sense, and deleted on schedule is the actual system.

References

Top comments (0)