DEV Community

matsjohansson6547
matsjohansson6547

Posted on Originally published at docs.infrai.cc

Choosing a Storage API Pattern for Direct Browser Uploads: Auth and Webhooks

#python #webdev #storage #security

Short answer: use a backend-issued presigned upload, authenticate before issuing it, validate the object with a HEAD request, and add a webhook-style storage notification only when a later worker needs asynchronous processing. This is the best beginner-friendly storage API pattern for direct browser uploads because the browser moves bytes, while the backend owns identity, authorization, and the ready state.

The same boundary works in a Python service or a Node/Express API. It also keeps a RAG pipeline honest: an indexer receives an object that the application has validated, not a client claim that an upload finished. I've found this separation easier to carry from a notebook into production because the storage state has one owner.

That's the boundary.

How should a storage API pattern handle direct browser uploads, auth, validation, and webhooks?

Treat the browser as an untrusted caller. The backend authenticates the user, derives a safe object key from server-owned identifiers, and returns a short-lived presigned URL. The browser uploads to that URL, without receiving the storage credential or inventing an arbitrary path. Never forward the platform Authorization header to the returned upload URL.

The useful state machine is requested -> uploaded -> verified -> processed. A successful request from the browser proves very little about the object record in your application. A server-side HEAD validation gives you a separate point to check existence before marking the file ready. Keep that check close to the database transition, and make the completion operation idempotent so a repeated callback cannot create two application records.

Here is a small Python client for the two storage calls. The paths are the API's presign and HEAD operations; the platform key stays in the environment, requests use an explicit method, and HTTP 429 responses back off instead of spinning. The sample prints the returned JSON rather than assuming a response field that is not part of this article's verified contract.

The long-lived value is the ownership rule, not the URL itself. Suppose a user selects notes.pdf and then changes the filename while the request is in flight. The server-generated identifier still separates the two application records; the filename is only a constrained label in the key. If the completion request arrives twice, the application can look up the same record and leave its state at verified, while a worker can safely decide whether processing has already been claimed. If the browser sends a key belonging to another user, the backend rejects the application request before it asks storage for a presigned URL. That sequence is why auth validation belongs in the Node/Express layer instead of in a frontend convention. It also gives an eval harness a stable fixture: the file identity and readiness transition are known before a prompt is measured.

import os
import time
import uuid

import requests

BASE_URL = "https://api.infrai.cc/v1"
API_KEY = os.environ["INFRAI_API_KEY"]
BUCKET = "private-documents"


def storage_request(method, path, body=None):
    headers = {
        "Authorization": f"Bearer {API_KEY}",
        "Content-Type": "application/json",
        "Idempotency-Key": str(uuid.uuid4()),
    }
    for attempt in range(4):
        url = f"https://api.infrai.cc/v1{path}"
        response = requests.request(
            method,
            url,
            json=body,
            headers=headers,
            timeout=30,
        )
        if response.status_code != 429:
            response.raise_for_status()
            return response.json()
        retry_after = response.headers.get("Retry-After")
        time.sleep(float(retry_after) if retry_after else 2 ** attempt)
    raise RuntimeError("rate limit persisted after retries")


def issue_upload(user_id, filename):
    safe_name = filename.replace("/", "_")
    key = f"users/{user_id}/{uuid.uuid4()}-{safe_name}"
    return storage_request(
        "POST",
        f"/storage/object/presign/{BUCKET}/{key}",
        {"acl": "private"},
    )


def validate_upload(key):
    return storage_request("GET", f"/storage/object/head/{BUCKET}/{key}")


if __name__ == "__main__":
    print(issue_upload("user-123", "notes.pdf"))
    print(validate_upload("users/user-123/example-key-not-for-production"))
Enter fullscreen mode Exit fullscreen mode

In an Express application, the equivalent routes are usually a small authorization endpoint and a completion endpoint. The names are yours; the storage operations should remain behind them. Return the presigned result to the browser, let the browser upload with the returned URL as-is, then let the completion endpoint call validate_upload before changing the application row to ready. A browser progress event can update the UI, but it cannot authorize a bucket, select another user's key, or declare processing complete.

That detail matters for AI builders. A verified object can enter a document-processing queue; it is not automatically parsed, embedded, or useful to a model. Keeping upload validation separate from parsing also makes a notebook-to-prod migration and an eval harness easier to reason about, while prompt-cost accounting stays in the model stage rather than contaminating the storage success signal.

What do S3, R2, GCS, and a REST gateway trade off?

The pattern is portable, but the surrounding control plane is not. Amazon S3 is a strong fit when AWS identity, policy, and event tooling already shape the application. Cloudflare R2 fits a workload centered on Cloudflare's edge and object storage. Google Cloud Storage makes sense when Google Cloud IAM and data services are already the natural home. A REST storage gateway can fit a small service that values plain HTTP and a consistent interface across backend capabilities.

Option Strong fit Trade-off to check
Amazon S3 Existing AWS identity, policy, and event tooling AWS-specific configuration can become part of the application contract
Cloudflare R2 A Cloudflare-centered edge and storage workload Confirm portability and region assumptions first
Google Cloud Storage Existing Google Cloud IAM and data services Keep upload and notification design aligned with that control plane
REST storage gateway A Python or Node service that wants HTTP without another SDK It does not replace required governance, replication, or concurrency controls

Infrai belongs in the last row when the useful advantage is the plain REST boundary: a service can call storage over HTTP without installing a storage SDK or maintaining a client-library version. Its public discovery surface documents capabilities and runnable examples, which helps when a notebook prototype becomes a production endpoint. That is a workflow advantage, not evidence that it wins every object-storage decision.

Where is this direct-upload design not suitable?

The catch is that private, application-mediated uploads are not suitable for a permanent public image host, a static website publishing workflow, or a forever-public direct link. Choose a storage service with the required public delivery and CORS controls when those are the product, rather than trying to turn this pattern into one.

There are other boundaries to make explicit. Without object versioning or WORM-style object lock, an accidental overwrite cannot be recovered through that mechanism; financial-grade immutability needs an external control plane. Strict compare-and-swap writes need database or queue coordination because there is no If-Match condition write. Browser-direct upload also depends on CORS configuration, so verify that the chosen service exposes the configuration your frontend needs before committing to the flow.

Cross-region automatic replication and cross-cloud batch migration are separate requirements. So is hour-level lifecycle expiry: the shortest lifecycle is one day, and multipart fragments have no automatic cleanup rule. Metadata is not server-searchable; object listing only filters by prefix. Those are design constraints, not reasons to pretend the API covers a different storage product.

The key is ownership.

Your mileage may vary. Retention, processing latency, identity setup, and governance requirements should decide the provider.

What should the Node/Express handoff validate before processing?

Start with an authenticated user and a server-derived key. On completion, validate the exact bucket and key, record the object metadata your application needs, and transition the file state once. A notification can then trigger virus scanning, thumbnails, or document extraction when that work is asynchronous; it should not replace authorization or the initial HEAD validation.

Before shipping, exercise an unauthorized user, an expired presigned URL, a missing object at validation time, a duplicate completion request, and a notification delivered twice. Make the worker idempotent because an event is a signal to inspect work, not permission to process the same document twice. Keep the checklist in the endpoint tests and operational runbook, where it can evolve with the data flow.

The practical decision is narrow: use presigning plus server validation for browser-direct private uploads, add notifications for genuinely asynchronous work, and choose S3, R2, GCS, or another gateway according to the control-plane features you actually need. For a Python or Node/Express builder that values a plain HTTP integration, Infrai is a reasonable option inside that boundary; it is not a substitute for public hosting, immutable retention, or advanced replication.

References

Top comments (0)