DEV Community

SunspireValerius59
SunspireValerius59

Posted on

Python Browser Upload Webhooks for Storage Notifications After Private Report Uploads

Short answer: for an e-commerce site serving generated reports, let the browser upload directly to private object storage, then treat a signed storage event as the hand-off to a queue. Generate thumbnails and run a virus scan from that event, never from the browser's success callback. This keeps large-file throughput off the application tier and gives each report an auditable state.

The constraint is bandwidth. A 700 MB export that passes through an API worker consumes memory, sockets, and retry capacity before a customer even sees it. A direct upload moves those bytes from browser to storage; the application authorizes a narrowly scoped object key and waits for a notification containing that key.

What should happen after a browser upload reaches private storage?

Use a small state machine: issued, uploaded, quarantined, scanned, ready, or rejected. The upload URL is not proof that the object is safe or complete. A storage event is a delivery hint, so the worker must perform a HEAD or equivalent metadata read, verify the tenant and expected size, and make its transition idempotent.

The event handler should acknowledge quickly and enqueue work. Thumbnail generation belongs after the quarantine decision; otherwise a crafted image can reach a derivative bucket before inspection. Virus scanning should consume the original object from a private location and write a verdict to a separate record. Keep derivatives private too, then issue short-lived download URLs only after the report is ready.

That ordering matters for retries. Duplicate notifications are normal in event-driven systems, while a delayed notification is possible. Store an idempotency key such as (bucket, object_key, version_id) and make the transition conditional. If the same event arrives twice, the second worker should observe the existing state and exit without creating a second thumbnail or charging a second scan.

One rule saves many incidents: the browser never calls the scanner.

Keep it boring.

How do private bucket events support thumbnails, virus scans, and report delivery?

The notification payload should carry an object identifier, an opaque version or generation, event time, and enough tenant context to locate the authorization record. Do not trust a filename extension or a browser-provided MIME type. OWASP recommends allowlisting extensions, validating file signatures, limiting size, and storing uploads outside the web root; those checks still apply when the browser uploads directly.

I have fought delivery gaps where a queue was healthy but the consumer could not tell whether an object had been replaced. Version the object key or record the storage generation, and include that value in every job. A 413 from an upload gateway is a policy signal, not a scan result. A timeout is also not a rejection: leave the object quarantined and retry with a bounded deadline.

The long path looks like this:

  1. The authenticated API creates a random object key and a short-lived upload authorization for one customer and report.
  2. The browser streams bytes directly to a private bucket and reports only the storage response to the UI.
  3. The storage notification is authenticated, deduplicated, and placed on a durable queue.
  4. A worker checks metadata and policy, runs the scanner, and records clean, infected, or indeterminate.
  5. Only a clean object enters thumbnail generation and report indexing; rejected objects remain inaccessible and are retained according to policy.

Here is a framework-neutral verifier. The real signature scheme depends on the storage service, so the example shows the control flow rather than inventing a provider-specific header.

import hashlib
import hmac
import json


def verify_event(raw_body: bytes, signature: str, secret: bytes) -> dict:
    expected = hmac.new(secret, raw_body, hashlib.sha256).hexdigest()
    if not hmac.compare_digest(expected, signature):
        raise ValueError("invalid event signature")

    event = json.loads(raw_body)
    required = {"bucket", "object_key", "generation", "tenant_id"}
    if not required.issubset(event):
        raise ValueError("incomplete storage event")
    return event


def enqueue_once(event: dict, state_store, queue) -> None:
    key = (event["bucket"], event["object_key"], event["generation"])
    if state_store.claim(key, expected_state="uploaded"):
        queue.publish({"dedupe_key": key, "tenant_id": event["tenant_id"]})
Enter fullscreen mode Exit fullscreen mode

The claim operation must be atomic. If the scanner is unavailable, keep the state quarantined and retry; do not silently mark the report ready. I'm not sure one retention period fits every store, so make quarantine retention a documented policy rather than a hidden queue setting.

Where do throughput and safety collide?

Large-file throughput is more than network speed. It includes multipart sizing, browser retry behavior, storage limits, scan concurrency, and the time a customer waits for a signed download. Measure bytes per second separately from time-to-ready. A fast upload can still produce a slow product if the scan queue is starved by ten huge reports from one tenant. In one review, I found the apparent bottleneck was not storage at all: a single tenant's retries occupied every scanner slot, so completed uploads waited behind work that had already been attempted. The fix was a per-tenant semaphore plus a queue metric for age, not another upload endpoint.

Use per-tenant concurrency limits and backpressure. A worker should stream the object into the scanner instead of loading it into memory, cap decompression expansion, and reject archives whose expanded size exceeds policy. Thumbnail workers need similar limits because image dimensions can be tiny on disk but enormous when decoded.

The trade-off is visible: direct-to-storage reduces API bandwidth, but it adds event delivery, reconciliation, and lifecycle work. A small catalog with 2 MB images may prefer a synchronous application upload because the simpler failure model outweighs the saved hop. For multi-hundred-megabyte reports, the extra state machine is usually the more predictable choice. Your mileage may vary with browser mix and regional latency.

Which implementation boundaries keep the pipeline replaceable?

Keep four interfaces separate: upload authorization, storage notification, scanning, and derivative generation. Each interface should accept an object identity plus a tenant and policy version, and return a durable decision. This makes a scanner replaceable without changing browser code or report URLs.

At minimum, compare any candidate storage service on multipart upload support, private-by-default policy, event retry semantics, version identifiers, lifecycle rules, and audit exports. DigitalOcean Spaces documents S3-compatible object operations, but compatibility does not make notification delivery or malware scanning identical across providers; verify those contracts in a test bucket. The same caution applies to any managed object store.

Run a replay test before production: upload a known corpus, duplicate each notification, delay some events by several minutes, replace one object before processing, and interrupt workers during scanning. The pass criteria are concrete: no public object, no duplicate derivative, no cross-tenant key, and no report marked ready without a recorded clean verdict.

The catch is operational ownership. Teams that cannot staff reconciliation, scanner updates, and retention reviews should stick with a managed workflow that includes those controls, even if it costs an extra integration fee. Teams that need custom retention or regional processing should keep the interfaces generic and own the queue and audit trail.

References

Top comments (0)