HayesSterling2614

Posted on Nov 6

Customer Support Training Artifacts: Deletion-First Storage for Small Image Uploads

#storage #customersupport #sre #dataretention

The page fires because a deletion backlog has crossed its SLO threshold. The on-call sees 18,400 training artifacts waiting for deletion, a queue age of 46 minutes, and no corresponding increase in upload volume. The immediate action is to stop treating object storage as the incident: inspect the retention worker, its eligibility query, and the metric that should have warned us before the page.

TL;DR: For low-volume image uploads, the least complex design is usually private object storage behind a small metadata table, with an explicit delete_at timestamp, an idempotent deletion worker, and an alert on deletion age rather than on bucket size. The right choice is the one that makes retention observable and portable. A cheaper storage rate does not compensate for an unbounded compliance queue or an irreproducible training set.

This article uses customer-support training artifacts as the concrete case: annotated screenshots, avatar-like examples, and short image attachments used to teach agents how to classify requests. The artifact is useful for a limited period. It should then disappear on schedule, with an audit record that proves what happened.

Why did the retention page fire after the bucket looked healthy?

A low-volume bucket can look healthy for months. Its byte count stays small, upload latency is ordinary, and the application returns successful writes. None of those signals proves that deletion works.

The useful alert is downstream. Define an SLO such as: 99% of eligible artifacts are deleted within 15 minutes of delete_at, measured over a rolling window. The exact target is a policy decision; the important part is measuring elapsed eligibility time. A bucket-size alert is a capacity signal, not a retention signal.

Work backwards from the page:

The deletion-age histogram crossed the SLO threshold.
The worker's eligible-object count rose while successful deletes fell.
The metadata query identified artifacts by policy version and delete_at, rather than by a guessed object prefix.
The upload path wrote the policy version and deletion deadline at creation time.

That fourth step is where many small systems quietly fail. A timestamp calculated during cleanup cannot reconstruct which policy applied when an artifact was created. Store the decision next to the object key. A later policy change can affect new artifacts without silently rewriting old promises.

Here is the shape of the record I would make the source of truth. It is deliberately boring.

type TrainingArtifact struct {
\tID            string
\tObjectKey     string
\tPolicyVersion string
\tDeleteAt      time.Time
\tDeletedAt     *time.Time
\tDeleteError   string
}

The code requires time in a real implementation, but the design matters more than the struct: the application owns the retention decision, while object storage owns bytes and access control. That boundary gives the team somewhere to test policy without creating a second, hidden retention engine.

The threshold has a cost in both directions. Set it too low and a brief provider delay, database lock, or worker restart creates pages that do not need human attention. Set it too high and the team discovers a broken deletion path only after a material amount of data has missed its policy. Page on sustained deletion age and preserve a lower-severity metric for backlog size.

What should the storage contract guarantee?

The contract should answer five questions before a vendor or self-hosted implementation is selected: can an upload be private by default, can its key be reproduced from metadata, can an eligible object be deleted repeatedly without harm, can a reader receive a bounded access grant, and can the system prove the final state?

For support training artifacts, the object key should not contain a customer email, ticket subject, or other identifier that may leak through logs. Use an opaque artifact ID and keep searchable business metadata in the database. A private object plus a short-lived read authorization is easier to reason about than a public URL that must be remembered and revoked later.

Deletion must be idempotent. A worker can crash after the storage delete succeeds but before the database update commits. On retry, the worker should treat “already absent” as a completed storage operation and record the terminal state. It should not create a new artifact, extend delete_at, or bury the error in a generic retry counter.

type ObjectStore interface {
\tPut(ctx context.Context, key string, body io.Reader, contentType string) error
\tDelete(ctx context.Context, key string) error
\tExists(ctx context.Context, key string) (bool, error)
}

func deleteArtifact(ctx context.Context, store ObjectStore, a TrainingArtifact) error {
\terr := store.Delete(ctx, a.ObjectKey)
\tif err != nil {
\t\treturn err
\t}
\treturn markDeleted(ctx, a.ID)
}

A production adapter can map a missing-object response to success, but that mapping belongs at the adapter boundary. Keeping it out of policy code makes the same test suite useful for different storage backends. The upload path also needs bounded size and content-type checks, checksum validation where supported, and a quarantine decision for files that fail inspection. “Small upload” describes traffic volume, not permission to trust bytes.

Cross-origin browser uploads add another boundary. The browser sends a preflight request when the requested method or headers require it, and the server's CORS response controls whether the browser exposes the result to the calling origin. CORS is a browser access rule; it is not object authorization. Keep the bucket private and test allowed origins, methods, and headers as a deployed configuration.

What are the best object storage solutions for low-volume image uploads?

The buying decision is less about a nominal storage rate than about the operational surface around it. I use this comparison before looking at a price sheet.

Option	Useful boundary	Operational burden	Retention risk to examine
Managed object storage	Durable bytes, access policies, lifecycle primitives	Provider configuration, identity, and egress review	A lifecycle rule may not cover metadata, exceptions, or an audit trail
Self-hosted S3-compatible storage	Local control and a familiar object interface	Disks, replication, upgrades, recovery tests, and on-call ownership	The team must implement and verify durability and deletion behavior
Filesystem attached to the application	Very small deployment with one writer	Backups, capacity, permissions, and migration work	File deletion can diverge from database state and backup retention
Database large-object storage	Transactional relationship with metadata	Database growth, backup size, and restore performance	A row delete may not mean every backup copy is gone

For this scenario, the default is managed object storage with a database record and an application-owned deletion worker. That is a decision rule, not a product recommendation. The limitation is important: this default is a poor fit when data residency, offline operation, or an existing storage platform makes external object storage unacceptable. In those cases, self-hosting becomes defensible when the team already operates the disks, replication, restore drills, and security boundary; choosing it solely for low upload volume usually converts a small bill into a larger on-call obligation. A filesystem is reasonable for a temporary local development path, but it is a weak production contract once workers, replicas, or independent backups exist.

The comparison also needs a portability test. Keep object operations behind a narrow interface, use opaque keys, store provider-specific response details separately from policy fields, and run contract tests against the chosen adapter. The goal is not to pretend every backend behaves identically. The goal is to make the differences visible before a migration is urgent.

How do you prove deletion instead of merely requesting it?

A successful delete request is an event, not proof of the whole workflow. The audit record should include the artifact ID, policy version, delete_at, deletion attempt time, final outcome, and a stable reason such as retention_expired. Do not put the image itself or a secret-bearing authorization URL into ordinary logs.

The useful metrics are narrow:

eligible artifacts by age bucket;
deletion success, retry, and terminal-error counts;
oldest eligible artifact age;
storage reads and bytes by purpose, separated from uploads.

Those metrics support capacity planning too. A low upload count can coexist with expensive reads during training sessions, and a tiny current bucket can hide a retention defect that will grow every day. Estimate the upper bound from artifact count, maximum accepted bytes, retention duration, and read behavior; then verify it with observed distributions. Do not use an average upload size as the only capacity number.

The test matrix should include a worker crash before the delete call, after the delete call, and after the audit update; duplicate delivery of the same job; a clock discrepancy around delete_at; an object missing before the worker runs; a storage timeout; and a policy change that affects only future artifacts. Run the deletion path in a staging bucket whose access policy matches production. A green unit test that mocks away authorization and timeout behavior is not a retention test.

Backups need their own statement. “Deleted from the active bucket” does not automatically mean “removed from every backup, replica, cache, or exported training set.” A common first assumption is that a small active bucket makes this distinction harmless; the correction is that recovery copies can have a separate lifecycle and owner. The retention policy must say which copies are in scope, how long recovery media persist, and what evidence is available after deletion. If the business requirement cannot delete historical backups immediately, document that boundary instead of implying stronger erasure than the system provides.

A decision rule that survives a price change

Choose the simplest storage boundary that meets the artifact's retention, privacy, recovery, and access requirements. For a support-training workload, that normally means private objects, application metadata, explicit deadlines, an idempotent worker, and an alert based on deletion age. Compare providers or self-hosting only after those invariants are written down.

Then make the policy executable: reject artifacts without a policy version, reject records without delete_at, and fail the deployment when the deletion SLO has no dashboard or alert. The least expensive option on a calculator is not the least expensive system if an engineer must reconstruct what should have been deleted.

DEV Community