Short answer: put private object storage behind an application-owned authorization service, send large user-uploaded documents directly with multipart transfers, and issue short-lived signed URLs only after checking ownership and the deletion deadline. The least complex option is the one that passes that contract under the application's real file-size and concurrency distribution.
For an edtech application retaining enrollment agreements, consent forms, or accommodation records, the hard constraint is rarely raw capacity. It is moving a large file without routing every byte through the API, while preserving private access and proving that an explicit deletion deadline was enforced. Those requirements shape the design before a storage brand enters the discussion.
Keep it boring.
Let retry cost drive the storage decision
A throughput test should model files, not isolated requests. Use the expected distribution of file sizes, simultaneous uploads, client network quality, and retry frequency. Measure time to successful completion, abandoned uploads, bytes retransmitted, part-level retry counts, and application API saturation. A headline bandwidth number from a clean data center connection won't predict a teacher uploading a scanned agreement over unstable Wi-Fi. I'm not sure what your actual network distribution looks like; production telemetry from consented, non-sensitive transfer metrics is what resolves that uncertainty.
Retries decide this.
Multipart upload is useful because it bounds retry cost and permits controlled parallelism, but it also adds state: the application needs an upload session with an expiry, a list of completed parts or an opaque completion token, and cleanup for abandoned sessions. Limit concurrency on the client rather than launching every part at once, because excessive parallelism can worsen contention, drain mobile batteries, and make retries arrive in a burst. Your mileage may vary, which is why concurrency belongs in the load-test matrix rather than in a universal recommendation.
Treat integrity separately from transport success. A completed request proves that the storage service accepted bytes; the application still needs a documented integrity signal that its chosen implementation can provide and that its client can verify correctly. Avoid assuming that one response field always represents a whole-file hash across single-part, multipart, encrypted, and proxied transfers. The portable contract is simpler: define the checksum algorithm your system expects, persist its value with the document, and verify it at the appropriate boundary supported by both client and storage.
One failure deserves an explicit test: the final part reaches storage, but the client loses the response. The retry must not create a second logical document or reset the retention deadline. Use an application-generated upload ID as an idempotency key, keep the opaque object key stable for that session, and make completion safe to repeat. A 403 after a signed upload URL expires should trigger fresh authorization and a newly signed operation; it should never cause the client to make the bucket public.
How should a startup app handle private user document uploads with signed URLs?
Split the path into control plane and data plane. The application API authenticates the user, creates a document record, chooses an opaque object key, and authorizes an upload. The browser or mobile client sends bytes directly to object storage. After completion, the application verifies the resulting object metadata and changes the record from pending to available. Downloads follow the reverse pattern: authorize against the database first, then mint a short-lived signed URL for one object.
That split matters more as files grow. Proxying a multi-part document through the application ties up its connections, duplicates network traffic, and makes a routine client retry consume application capacity. Direct transfer keeps the API focused on identity, policy, and state transitions. It also gives the client a clean retry boundary: a failed part can be retried without repeating the whole file, provided the selected storage implementation and client support multipart upload.
A signed URL is a bearer credential — possession is enough to use it until it expires — so it must come after authorization, never replace authorization. Don't put a predictable user ID, email address, or course name in the object key. Generate an opaque key, store the relationship in the database, and return the URL only over an authenticated endpoint. Access logs should record the internal document ID and decision outcome, but not the signed query string. Logging that string quietly turns a short-lived credential into data copied across log processors, support tools, and alerts.
The three controls are straightforward: private-by-default objects, authorization before signing, and expiry bounded by the document's deletion deadline. If a record must be deleted at 14:00 UTC, a URL minted at 13:58 must not remain valid after 14:00. A convenient fixed URL lifetime can violate retention policy at exactly the point where the policy matters.
Make deletion a state machine, not a bucket setting
An explicit deletion deadline belongs in application data as delete_at, alongside status, object_key, owner_id, and a storage generation or checksum. A bucket lifecycle rule can provide defense in depth, but it cannot express every application transition: a student may replace a form, an administrator may place an authorized hold, or an incomplete multipart upload may never become a document record. The database is where those meanings live.
Use a small state machine: pending, available, deleting, and deleted. At the deadline, an idempotent worker claims the row, marks it deleting, requests object deletion, verifies the object is absent, and records deleted_at. If the worker stops between those steps, the next run can continue from durable state. Do not mark the row deleted before storage confirms the outcome; doing so loses the key needed to retry and creates an object that is difficult to discover.
There is a second race. A download request can pass authorization while the deletion worker starts. The signing operation should re-read the record, reject any state other than available, and cap URL expiry at delete_at. The deletion worker should revoke future issuance by changing state before it removes bytes. An already issued URL may remain usable until its capped expiry, so choose the maximum signing window as a policy decision and include that window in deletion semantics.
A compact Python policy function makes the boundary testable without binding the application to one provider:
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
@dataclass(frozen=True)
class Document:
object_key: str
owner_id: str
status: str
delete_at: datetime
def signing_expiry(
document: Document,
requester_id: str,
now: datetime,
maximum_ttl: timedelta = timedelta(minutes=5),
) -> datetime:
if now.tzinfo is None:
raise ValueError("now must be timezone-aware")
if document.owner_id != requester_id:
raise PermissionError("document access denied")
if document.status != "available":
raise PermissionError("document is not available")
expires_at = min(now + maximum_ttl, document.delete_at)
if expires_at <= now:
raise PermissionError("document retention deadline has passed")
return expires_at.astimezone(timezone.utc)
This function doesn't mint a URL. Good. Signing belongs behind a narrow storage adapter, while ownership and retention policy stay in ordinary application code. That boundary lets a startup test policy once and swap S3-compatible implementations without scattering provider calls through handlers and jobs.
Compare contracts before products
An S3-compatible label narrows client choices, but it doesn't answer the operational questions. Build a small conformance suite against every candidate and run it from the regions and client paths that matter. The suite should cover private creation, multipart initiation and completion, abort cleanup, signed upload and download expiry, metadata round trips, deletion confirmation, pagination, retry behavior, and the exact authentication mode your adapter uses.
Use a decision table with disqualifiers first:
| Constraint | Evidence to collect | Reject when |
|---|---|---|
| Large-file path | Completion time and retransmitted bytes across the real size distribution | Required throughput fails under expected concurrency |
| Private access | Tests for anonymous denial and expired signed URLs | An object can be fetched outside the authorized path |
| Retention | Deadline tests, deletion verification, and abandoned-upload cleanup | The team cannot demonstrate timely removal |
| Compatibility | Adapter conformance results for operations the app actually uses | A required semantic differs without a contained adapter |
| Operations | Export procedure, observability, support boundaries, and recovery drill | Exit or incident diagnosis depends on undocumented behavior |
| Cost model | Storage, request, retrieval, and network assumptions using measured traffic | A plausible traffic shift breaks the approved budget |
This is deliberately not a vendor scorecard. An S3-compatible API can narrow client choices, but a compatibility label does not remove the need to test the contract. Commercial terms, network paths, operational ownership, and semantic boundaries can change. Record the dated source for every commercial assumption, then rerun the model with measured request and download volumes.
The catch is operational ownership. Self-hosted object storage may suit a team that needs infrastructure control and already operates durable distributed systems; it is not suitable when the same small team would also own disks, replication, upgrades, monitoring, and recovery without enough staffing. A managed service reduces that burden, but it still requires exit testing, account security, quotas, and a support plan. Retain the current storage class when it passes the conformance and throughput tests and migration risk outweighs a marginal paper advantage.
Change only when a requirement fails.
Preserve download behavior without leaking policy
Download responses need deliberate browser behavior. Content-Disposition communicates whether content should render inline or download as an attachment and can provide a filename. MDN documents the header syntax and the filename and filename* parameters. For signed object downloads, configure response metadata through the signing mechanism supported by the chosen implementation, or use a narrowly scoped download gateway when policy requires headers the direct path cannot guarantee.
Treat the displayed filename as presentation data. Store a sanitized name separately from the opaque object key, reject control characters, and keep authorization independent of that name. Sensitive documents should generally download as attachments unless the product has reviewed inline rendering, content types, and browser exposure. A content-type mismatch is a security and compliance concern, not cosmetic polish.
Delivery logs should answer who requested access, which internal document was involved, what policy allowed or denied it, and when the credential expires. They should not retain the URL signature, document contents, or an unnecessary original filename. That distinction is familiar from OTP systems: observability needs event identity and outcome, while secrets and personal data stay out of the trail.
Roll out with a reversible adapter
Start with one storage adapter and one document cohort. Shadow the new policy decisions without issuing URLs, compare expected allow and deny outcomes, then move uploads before downloads so newly written objects have complete metadata. Backfill delete_at before enabling deletion workers. Run a dry report of due objects, sample it against source records, and only then enable small deletion batches with an audit trail.
The adapter should expose application verbs such as begin_upload, complete_upload, sign_download, delete, and exists; it should not leak a vendor client through the service layer. Keep the old read path available during migration until object counts, checksums, and retention metadata reconcile. Rollback then means changing routing for a cohort, not rewriting policy.
The final selection rule is compact: choose the simplest private object store that passes your measured large-file workload, signed-URL authorization tests, and deletion-deadline drill while leaving the application in control of identity and retention state. No provider compensates for a missing state machine.
Top comments (0)