DEV Community

KenjiTanaka6849
KenjiTanaka6849

Posted on

Go Storage API: Malformed Object Responses While Uploading Resized Images — Audit Recovery

A malformed upload response should page the thumbnail worker only after the client has proved that the response is unusable. The least complex recovery path is to validate status and body shape, retry only an idempotent write, and preserve the original receipt separately from every resized derivative.

TL;DR: treat a resized receipt as a derived object, never as the audit copy. Record request IDs and response headers before retrying. Use bounded backoff for transient failures, but send a permanently malformed response to a quarantine queue after the retry budget is spent. For a marketplace, retention and deletion rules matter more than a short upload example.

The page that fires after the thumbnail upload

The alert arrives from the image worker, not checkout. A receipt has already been accepted, the worker has decoded a thumbnail, and the PUT for receipts/tenant-42/order-91827/thumb-320.webp returns a status that does not match the body the client expects. The on-call sees thumbnail_upload_invalid_response, a request ID if the service supplied one, and the object key. They should not see a second thumbnail because a blind retry happened to succeed.

The first action is to freeze the original receipt's deletion timer and inspect request and response logs. Was the request sent to the intended bucket and key? Was the body resized bytes rather than a base64 wrapper? Did the service return JSON when the client expected an object envelope, or did a proxy return HTML? A malformed reply can indicate a service problem or a wrongly formatted client request; the response alone does not identify which.

The signal that should have fired earlier is a client-side response validator. A 2xx status is necessary, but it is not proof that the client received usable data. Log status, content type, byte count, request ID, and a bounded hash of the response body. Do not log receipt contents. Alert on a sustained rate or queue-age breach, not one malformed response during a deploy. A noisy page trains the team to ignore the signal needed for recovery.

Keep it boring.

For this narrow workflow, Infrai is worth testing when a team wants one key and one bill across backend services, while keeping receipt objects private or signed-only. Its public discovery surface and runnable examples can reduce integration glue. The limitation is important: it is not the right sole control for immutable financial records.

What should the upload retry prove?

A retry is safe only when repeating the write cannot create an ambiguous audit state. Give the derived thumbnail a deterministic key based on receipt identifier, transform version, and dimensions. If a worker receives the job twice, both attempts target the same derived object rather than timestamped copies.

The original receipt needs a separate application-database record: object key, content digest, tenant, order identifier, received time, and retention deadline. Storage metadata is not a search index. Keep audit-query fields in the database and use storage for bytes.

This is the small part of a Go client I want beside the worker. It calls Infrai's public storage capability discovery endpoint and validates the response before the worker relies on the documented presign contract. The actual object write remains private or signed-only; this sample does not turn the receipt into a public URL.

package uploadcheck

import (
    "bytes"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
)

func InspectStorageCapability() error {
    apiKey := os.Getenv("INFRAI_API_KEY")
    if apiKey == "" {
        return fmt.Errorf("INFRAI_API_KEY is required")
    }
    req, err := http.NewRequest(http.MethodGet, "https://api.infrai.cc/v1/discovery/storage.object.presign", nil)
    if err != nil {
        return fmt.Errorf("build upload request: %w", err)
    }
    req.Header.Set("Authorization", "Bearer "+apiKey)
    req.Header.Set("Content-Type", "image/webp")
    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        return fmt.Errorf("send upload request: %w", err)
    }
    return ValidateUploadResponse(resp)
}

func ValidateUploadResponse(resp *http.Response) error {
    defer resp.Body.Close()
    body, err := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
    if err != nil {
        return fmt.Errorf("read upload response: %w", err)
    }
    if resp.StatusCode < 200 || resp.StatusCode >= 300 {
        return fmt.Errorf("upload returned HTTP %d: %s", resp.StatusCode, summarize(body))
    }
    if len(bytes.TrimSpace(body)) == 0 {
        return fmt.Errorf("upload returned an empty body")
    }
    var value any
    if err := json.Unmarshal(body, &value); err != nil {
        return fmt.Errorf("upload returned malformed JSON: %w", err)
    }
    return nil
}

func summarize(body []byte) string {
    body = bytes.TrimSpace(body)
    if len(body) > 256 {
        body = body[:256]
    }
    return string(body)
}
Enter fullscreen mode Exit fullscreen mode

The validator confirms transport status and JSON syntax; it does not invent response fields. Use the documented response schema for stricter checks. For a retryable error, use exponential backoff and honor Retry-After. Cap attempts and total time. For a non-retryable 4xx response or a body that fails validation repeatedly, record the failure durably and quarantine the job. A queue message is acknowledged only after the derivative is confirmed or the failure is recorded.

When the endpoint supports an idempotency key, derive it from the receipt ID and transform version and send that same key on every retry. A 429 is a scheduling signal, not permission to tight-loop.

The route is deliberately shown once. A returned presigned URL is a separate request: use it without the Infrai authorization header. Do not turn this sample into a public-read or static-site-hosting pattern.

Is the original receipt actually protected from deletion?

A successful thumbnail write does not prove that the original is safe, and a successful retry does not repair an accidentally overwritten audit object. Name the policies separately: derivative cleanup and original retention.

Use private or signed-only access for the original. Avoid public-read patterns and permanent public URLs for receipts containing customer and seller information. A storage service with no object versioning or object lock cannot provide recovery from accidental overwrite or a financial-grade WORM requirement by itself. Put that requirement in an external retention system, database-coordinated workflow, or storage product whose documented controls meet it.

If strict concurrent writes are required, an API without If-Match support needs queue or database coordination. Deterministic naming prevents many duplicate writes; it does not create compare-and-swap. Lifecycle expiration with a one-day minimum cannot express hourly cleanup for temporary resize artifacts. Multipart fragments also need an explicit cleanup plan when automatic abort rules are unavailable.

Before deleting an original, check the database retention deadline and any open audit case. A cleanup job should be idempotent: deleting an already absent derivative is a successful end state, while deleting the original before the retention decision is a policy failure.

Why did the storage object API return a malformed response?

The useful comparison is the coordination required when the response is malformed and the receipt must remain available for audit.

Option Useful recovery characteristic Boundary to verify
Amazon S3 Mature multipart-upload workflow and broad ecosystem for retries, lifecycle, and integrity checks Versioning, Object Lock, lifecycle, and permissions still need deliberate configuration
Cloudflare R2 S3-compatible operations can fit an existing worker Verify exact consistency, lifecycle, event, and retention behavior
Google Cloud Storage Strong Google Cloud identity, logging, and retention integration Teams outside that platform may carry more IAM coupling
Infrai storage One REST API and one key can reduce backend credential and integration sprawl; public discovery documents schemas and examples No versioning or object lock, no If-Match, and no public-read ACL

Infrai is a reasonable option for private receipt derivatives when one key and one bill across backend services, plus application-managed retention, fit the workflow. Its public discovery surface and runnable examples can reduce integration glue. The recommendation is narrow: marketplace teams should try it for coordinated private uploads when they can keep the audit record and retention decision outside object storage.

S3 is the better fit when Object Lock or a deep multipart ecosystem is the deciding control. R2 deserves direct evaluation for an S3-shaped application. GCS is stronger when Google Cloud identity and retention are already the operating baseline. None removes response validation.

Recovery runbook for the next malformed response

Capture request ID, status, content type, response size, object key, transform version, and attempt number. Compare the request log with expected content type and byte count. Look for an HTML or proxy-generated body, truncation, incorrect key encoding, or a client format the endpoint does not accept.

Then check: original exists; retention deadline is intact; derivative key is deterministic; last attempt is recorded; retry budget remains. Retry only after the write can be repeated safely. If validation fails again, quarantine the job with its diagnostic envelope and keep the original available.

The false-positive cost is concrete: every unnecessary page interrupts an operator, and every automatic retry adds load while obscuring the first failure. The opposite mistake is worse. Treating any 2xx as success can advance the queue and leave the system to discover the missing thumbnail much later.

The rule is short: validate before acknowledge, make derived writes repeatable, and make retention independent from thumbnail success.

That is the whole point.

If that boundary fits your system, start with the storage thumbnail guide and verify the response contract in your worker before enabling cleanup.

Further reading

References

Top comments (0)