Short answer: use a server-authorized, direct browser upload, but make your application database the authority for tenant ownership, retention, and deletion. Verify the exact object after the upload, then let a signed webhook start asynchronous processing. For an edtech product that stores course and product images and exports one tenant's data, this division is less complicated to reason about than treating object storage as both the file system and the policy engine.
The byte path and the policy path are different things. A browser can send a large image directly to object storage without consuming an Express worker, while Node or Express still decides who may create the upload, which tenant owns it, and when the image is eligible for an export. A successful browser request proves very little about the product state.
Deletion is where a superficially tidy upload design usually becomes difficult. A tenant can request deletion while an export job is queued, a webhook can arrive twice, and an object can exist after its application record has become ineligible for use. The design has to make those races boring.
Where tenant exports lose their data boundary
The common failures are coordination failures, not failed byte transfers. An export that lists storage by prefix can include an object whose application record belongs to another state or tenant. An export that trusts a cached list can include an image deleted after the list was produced. A worker that accepts an old upload event can create a thumbnail after a deletion request. Each failure comes from asking storage to answer an application question.
Build the export from application records. The query must select one tenant, require an eligible state such as active, and return the exact object keys to read. A prefix like tenant-17/ can help organize data, but it is not a permission check. The worker should re-check state before publishing an export, especially if the job runs for a long time.
The export itself needs a retention decision. A generated archive may contain images that were later deleted, so give exports their own owner, expiration time, and deletion event. Do not let a long-lived download URL quietly outlive the tenant's policy. The same rule applies to thumbnails and moderation copies: derived objects are part of the data inventory, not an implementation detail that can be forgotten because the original key was removed.
HTTP semantics give useful vocabulary for these transitions. A non-success response such as 403 is a failed operation and should remain observable as one; a retry loop must not turn it into an active upload merely because the client eventually stopped waiting. RFC 9110 is a good baseline for treating status codes, request methods, and response meaning as part of the contract rather than incidental transport details.
Here is a compact test matrix I would run before expanding the feature:
| Scenario | Required application result | Storage-side assumption to avoid |
|---|---|---|
| Browser uploads within the issued constraints | Record can move to verification | A client completion message is proof of ownership |
| Browser retries completion | Same state transition, no duplicate work | Every callback is a new upload |
| Upload targets another tenant's record | Authorization failure and audit event | A valid object key is valid for every caller |
| Deletion races with an export | New export excludes the record | Physical deletion timing controls visibility |
| Webhook is delivered twice | One idempotent processing result | Notification delivery is exactly once |
| Worker sees an old event | Re-read state before publishing output | Event order equals business order |
| Tenant cancellation starts | Image, derivatives, and exports enter a tracked workflow | One bucket delete completes the policy |
The hardest case is not exotic. It is a normal deletion arriving at an inconvenient time.
Which storage boundary matches the retention requirement?
The comparison should happen after the policy is clear. Otherwise teams select an API for upload convenience and discover months later that they cannot explain deletion, restore a mistaken removal, or prove which records fed an export.
| Boundary | Good fit | Cost or limitation to accept |
|---|---|---|
| Application plus one native object-store API | A team whose retention controls and operational tooling already match one provider | Provider-specific code and migration work remain in the application |
| Application plus a storage abstraction | A team that needs a stable internal contract across storage backends | The abstraction can hide important provider differences and must expose the controls that matter |
| Express proxy for every byte | A workflow that must inspect or transform content before storage | The application carries bandwidth, latency, and upload-capacity responsibility |
| Direct browser upload with server-issued capability | Large media where the application should enforce policy without carrying the bytes | Verification, expiry, cleanup, and browser CORS behavior need explicit testing |
There is no universal winner here. A direct upload is a transport choice, not a retention strategy. An abstraction is valuable only if it preserves the controls the policy requires; a thin interface that omits immutability, conditional writes, or deletion observability may create false portability. Your mileage may vary when a provider's consistency, lifecycle, or notification semantics differ from the assumptions in the interface, so test those semantics instead of promising that an adapter makes them identical.
The catch is that some retention regimes require controls beyond a simple delete call. Regulated records may need an immutable history, legal holds, separation of duties, or evidence that a deletion request was processed across derivatives and exports. A design that cannot supply those controls is not suitable for that workload, even if its upload path is elegant. Stick with a storage and governance combination that directly supports the requirement, and treat FedRAMP authorization as a separate compliance question rather than as a label that automatically proves an application's data handling is acceptable.
On the other hand, a private bucket, server-generated keys, database-backed state, and a small idempotent worker are often enough for ordinary product images. Do not buy operational complexity for a retention feature the product does not need. Define the boundary, test the races, and document what “deleted” means.
Start with a retention contract
Before designing an upload endpoint, define what “deleted” means. There are at least three different events: the application stops using an image, the storage object is removed, and replicas, caches, backups, or derived thumbnails expire. Those events may not happen at the same instant, and a retention policy that names only a number of days is not yet an operational policy.
For the image workflow, create an application record before issuing upload authority. It should contain a server-generated object key, the authenticated tenant ID, the product or course relationship, the expected media type and size, a retention class, and a state. The filename supplied by a browser is display data. It must not become a namespace or an authorization decision.
A modest state machine is enough:
reserved -> uploaded -> verified -> active -> deletion_requested -> deleted
The names are less important than the transitions. A record in deletion_requested must be excluded from new tenant exports immediately, even if the storage deletion worker has not run. An object that has landed in storage but has not passed verification must not be exposed as an active product image. This is the distinction between logical deletion and physical deletion, and it should be visible in the data model rather than hidden in a worker's timing.
Write down the clock as well. Does retention begin at upload, publication, the last course use, or tenant cancellation? What happens during a legal hold? Is there a grace period in which an administrator can restore a mistaken deletion request? I would rather see an explicit policy with an awkward exception than a lifecycle rule that silently deletes the wrong class of asset.
Three fields deserve unusual attention: tenant_id, state, and delete_at. Put an audit event beside each state change. When a customer asks why an image appeared in an export or why it disappeared, the answer should come from a queryable record, not from reconstructing the order of webhook deliveries. I don't want a support engineer comparing object timestamps, queue logs, and browser screenshots to answer a retention question; the application record should make the answer ordinary, even when the underlying storage system is eventually consistent in a way the product team did not anticipate.
Keep it boring.
How does an edtech storage API pattern protect direct browser uploads?
Authentication and authorization belong to the application boundary. The server validates the session, derives the tenant from trusted identity data, checks the caller's relationship to the course or catalog item, allocates an opaque key, and records the pending upload. Only then should it create a narrowly scoped capability for the browser to write that one object for a limited time. The API pattern is intentionally unexciting: direct browser uploads carry bytes, while the application retains the authority to name, verify, export, and delete them.
That capability is deliberately constrained. The browser should not choose a neighboring tenant's key, overwrite a stable asset by guessing its name, or decide that an internal bucket is public. Direct-to-storage means the application has left the byte path; it does not mean the application has left the authorization path. Proxy every upload through Express only when the application genuinely needs to inspect or transform the bytes in transit, accepts the bandwidth cost, and has capacity to operate that path.
Validation has two stages. Before the upload, check the declared content type, size limit, tenant relationship, and retention class. After the upload, inspect the exact key allocated by the server and compare observed metadata with the pending record. The client can report completion, but it cannot promote its own record to active. I have seen completion messages treated as durable state and then spent time explaining why a closed tab, a retry, and a real object were three different facts. Keep them different.
The following is intentionally a storage-neutral boundary. The interface names are application concepts, so changing the object-store client does not change the deletion rule.
from dataclasses import dataclass
from datetime import datetime
from typing import Protocol
class ObjectStore(Protocol):
def inspect(self, key: str) -> dict: ...
def delete(self, key: str) -> None: ...
@dataclass
class Image:
tenant_id: str
key: str
expected_size: int
expected_type: str
state: str
def verify_upload(store: ObjectStore, image: Image) -> None:
observed = store.inspect(image.key)
if observed["size"] != image.expected_size:
raise ValueError("object size differs from the upload contract")
if observed["content_type"] != image.expected_type:
raise ValueError("object type differs from the upload contract")
if image.state != "uploaded":
raise ValueError("image is not awaiting verification")
image.state = "verified"
def request_deletion(image: Image, requested_at: datetime) -> dict:
if image.state == "deleted":
return {"state": "deleted", "changed": False}
image.state = "deletion_requested"
return {
"tenant_id": image.tenant_id,
"key": image.key,
"requested_at": requested_at.isoformat(),
}
The code does not claim that a storage deletion has already happened. That distinction lets an export query exclude deletion_requested before the worker removes the bytes. The deletion worker should re-read the row, perform the storage operation for the exact key, and record the result idempotently. A repeated job must not resurrect an image or publish a derived thumbnail.
Callbacks and webhooks are useful after a state transition has a clear meaning. Use them to enqueue thumbnailing, moderation, or export indexing after verification. Authenticate the sender, persist an event identifier, and make the consumer safe to run twice. A webhook is a notification channel. It is not proof of tenant ownership, upload authorization, or completed deletion.
How can Node and Express roll out tenant deletion with an audit trail?
Start with one private namespace and one image class. Add the application record, server-issued upload capability, post-upload verification, and a tenant-scoped export query before adding notifications. Exercise abandoned uploads, expired capabilities, repeated completion requests, cross-tenant attempts, deletion during export, and duplicate webhook events. Record each transition with the tenant, object key, actor or worker, and timestamp.
Then add cleanup in stages. First stop deleted records from entering new work. Next remove the original object. Then remove or expire thumbnails, archives, and other derived objects. Finally verify that download links and caches follow the same policy. A daily reconciliation job can find records stuck in an intermediate state, but it should report and repair through the same state machine rather than silently deleting by prefix.
Keep metrics attached to the policy: age of pending uploads, time from deletion request to physical removal, number of export jobs blocked by a missing object, webhook redeliveries, and objects with no application record. These measurements expose the gap between logical and physical deletion. They are more useful than a single “upload success rate” metric.
The least complex design that meets the requirement is usually the best starting point: browser-to-object storage for the bytes, Node or Express for authorization and state, the database for tenant-scoped selection, and workers for deletion and derived work. Revisit the boundary only when a concrete retention, inspection, or compliance requirement makes it insufficient.
Top comments (0)