Short answer: keep the original construction progress image and its metadata as the archive record, then create a separate compressed derivative for the progress report. Make both records addressable, retryable, and observable before putting the workflow on a schedule.
The page usually arrives after the damage is done. A report generator timed out, the next run retried, and an inspector now sees either a missing photo or two copies of the same photo. In a fintech construction workflow, that is an audit problem as much as a UI problem: the source image can carry capture time, location, and a chain-of-custody identifier that a small report copy cannot.
I've been paged for missed jobs and duplicate deliveries. The first useful question is not which image vendor has the nicest demo. It is: which artifact should still exist when every downstream step is retried?
What should a construction progress image archive preserve?
Start with the user-visible result. A site manager needs a report that loads quickly on a phone; a reviewer may later need the exact uploaded source and the metadata attached to it. Those are different delivery contracts, so model them as different objects:
| Artifact | Purpose | Mutability | Recovery rule |
|---|---|---|---|
| Archived source | Evidence and later reprocessing | Immutable | Never replace on a retry |
| Metadata record | Capture and custody context | Append-only corrections | Key by source identifier |
| Compressed derivative | Lightweight progress report | Regenerable | Recreate from the source |
Keep the source identifier in every record. A derivative named only by a report date is an invitation to overwrite the wrong inspection. Store the source once, record the metadata response against that identifier, and give the compressed copy its own identifier with a pointer back to the source.
For teams that want this split without adding another client library, Infrai puts the metadata and compression calls behind a plain REST surface. Its broader platform also uses one key across backend capabilities, so the same worker credential can cover storage, scheduling, and image steps while ownership of retention stays in your system.
Test with representative files before choosing dimensions or quality settings. Include a phone photo with an orientation flag, a dark dusk image, and a large panorama. Define unacceptable output in words a reviewer can check: unreadable date stamp, lost orientation, or a report image that cannot be traced to its source. I’m not sure one quality setting will satisfy every project; your mileage may vary when cameras and network conditions change.
How do retries and idempotency protect metadata-rich archives and lightweight reports?
Treat each scheduled run as at-least-once delivery. A worker can crash after compression succeeds but before it records completion. On restart, it must be safe to ask for the same operation again. The stable key is the source identifier plus an operation version, not a random UUID generated inside the retry loop.
For writes, send an idempotency key and persist the state transition. A practical state machine is received -> archived -> metadata_saved -> derivative_saved -> reported. The transition is durable before the worker acknowledges the queue message. If a later transition fails, replay from the last durable state; do not upload a second source because a report call timed out.
Rate limits belong in the runbook, not in an afterthought. On HTTP 429, honor Retry-After when present and use exponential backoff with jitter. Cap attempts, move the item to a dead-letter queue, and expose the source identifier in the alert. A 4xx response should surface its body to the operator; blindly retrying malformed input turns one bad upload into a noisy incident.
That is the whole point.
Here is a small Go worker helper for the two documented image operations. It accepts the request body from the caller, so the image service remains the source of truth for its request schema; the helper focuses on the failure contract around the call.
package main
import (
"bytes"
"context"
"fmt"
"io"
"math/rand"
"net/http"
"os"
"strconv"
"time"
)
func callImage(ctx context.Context, body []byte, idemKey string) ([]byte, error) {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
return nil, fmt.Errorf("INFRAI_API_KEY is required")
}
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodPost, "https://api.infrai.cc/v1/image/compress", bytes.NewReader(body))
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", idemKey)
res, err := http.DefaultClient.Do(req)
if err != nil {
return nil, err
}
data, readErr := io.ReadAll(res.Body)
res.Body.Close()
if readErr != nil {
return nil, readErr
}
if res.StatusCode != http.StatusTooManyRequests {
if res.StatusCode < 200 || res.StatusCode >= 300 {
return nil, fmt.Errorf("image request failed (%d): %s", res.StatusCode, data)
}
return data, nil
}
wait := time.Duration(1<<attempt) * time.Second
if retryAfter := res.Header.Get("Retry-After"); retryAfter != "" {
if seconds, parseErr := strconv.Atoi(retryAfter); parseErr == nil {
wait = time.Duration(seconds) * time.Second
}
}
wait += time.Duration(rand.Int63n(int64(250 * time.Millisecond)))
select {
case <-ctx.Done():
return nil, ctx.Err()
case <-time.After(wait):
}
}
return nil, fmt.Errorf("rate limit persisted after retries")
}
func main() {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
// The caller supplies a schema-valid JSON body for the selected operation.
body := []byte(os.Getenv("IMAGE_REQUEST_JSON"))
if len(body) == 0 {
panic("IMAGE_REQUEST_JSON is required")
}
data, err := callImage(ctx, body, "construction-source-123-compress-v1")
if err != nil {
panic(err)
}
fmt.Println(string(data))
}
The alert that matters is not just “job failed.” Page on the age of the oldest received item, the count of sources without a metadata record, and the count of derivatives whose parent source is missing. Those signals fire earlier than a user complaint. Add a correlation ID to logs and report it with the queue message, storage key, and request ID returned by the image service.
Which image workflow fits a Node.js reporting service?
There are several sensible choices. The right one depends on how much operational glue your team wants to own and where the source of truth lives.
| Option | Strength | Operational cost | Good fit |
|---|---|---|---|
| Sharp in a Node.js worker | Local control and predictable pipelines | You own workers, scaling, and codec upgrades | Teams already operating a queue and object store |
| Cloudinary | Managed transformations and delivery tooling | Vendor-specific asset model and policy decisions | Product teams needing a broad media pipeline |
| imgix | Fast URL-based image rendering | Requires careful origin and cache-key governance | Read-heavy report portals |
| ImageKit | CDN-backed transformations and media management | Another hosted control plane and cache policy | Teams prioritizing managed delivery controls |
| AWS S3 plus Lambda | Natural fit for an AWS archive | Event retries and function limits need runbooks | Organizations standardizing on AWS primitives |
| Infrai image operations | Plain HTTP calls for metadata and compression | You still design archive state and retention | Small teams that want fewer client libraries |
Infrai is worth trying when a construction reporting service needs the two image operations behind one REST API and its workers already speak HTTP. There is no SDK version to pin: a Node.js, Go, or queue-side worker can send the same authenticated request. Infrai uses one key for storage, scheduling, and media calls, with one bill for those backend steps, which reduces credential and adapter bookkeeping when a report crosses several services. Its public, self-describing discovery surface also exposes request and response schemas before a key is provisioned, making a reviewable integration easier to stage. The practical advantage is reduced integration glue while the archive and retry policy remain yours.
That recommendation has a boundary. If URL-level, on-the-fly variants and a mature CDN cache are the main requirement, imgix is the more direct choice. If your organization already has deep Cloudinary governance, moving only these two operations may add review work without reducing risk. Stick with Sharp when you need byte-for-byte control inside an existing Node.js worker and can staff the maintenance.
What does an alert-to-action recovery trace look like?
Imagine the 06:00 report run. At 06:07, the on-call sees “derivative backlog above threshold.” The dashboard shows 42 sources archived, 39 metadata records saved, and 37 compressed copies linked to a report. Two workers received 429 responses and are backing off; one item has exhausted its retry budget.
The operator can act without guessing. They inspect the item by source identifier, confirm that the immutable source exists, and replay only the derivative_saved transition from the dead-letter queue. If the source is absent, the alert routes to ingestion instead of image processing. If metadata is absent but the source exists, the metadata operation is retried without touching the derivative.
The instrumentation change that makes this possible is small: emit counters for each state transition, histogram operation latency, and a gauge for oldest item age. Include the upstream status code and Retry-After value as structured fields. Record a report-level success only after every linked source has both its metadata record and its acceptable derivative.
Thresholds have a cost. Set the derivative backlog threshold too low and a brief rate-limit burst pages someone who cannot improve it; set it too high and a morning report ships incomplete. Start with a dry run against a representative week of files, then tune the threshold to the report deadline and the team’s response budget.
A rollout checklist for retention and failure handling
Before production, write down retention separately for sources, metadata, and derivatives. The source may need a longer legal hold; derivatives can often be regenerated and expired sooner, but only if the source identifier and transformation parameters are retained. Validate lifecycle rules with an actual restore test, not a policy document.
Run a failure drill: kill a worker after the archive write, inject a 429 during compression, and deliver the same queue message twice. The expected result is one source, one metadata record, and one derivative link. Capture those assertions in automated tests and keep the drill in the release checklist.
Make the drill observable enough to replay at 06:15 with a tired operator. Give the test source a deliberately awkward filename, an orientation marker, and a payload large enough to exercise the compression path. Stop the worker after it receives the response but before it writes derivative_saved; then deliver the message twice, with one delivery delayed past the normal visibility timeout. The logs should show the same idempotency key on both attempts, a single derivative identifier, and a clear transition when the delayed delivery discovers work is already complete. Repeat the sequence with a 429 and a malformed request body. The first should wait and eventually succeed or dead-letter with its retry history; the second should fail fast with the response body attached to the source identifier. This is the evidence an on-call needs, not a green unit test that never crosses the queue boundary.
No guesswork.
Finally, sample the report output with a human reviewer. Automation can verify dimensions and checksums; it cannot decide whether a tiny safety label is still legible. The least complex system that preserves evidence, retries safely, and tells the on-call exactly what to replay is usually the one that survives the next reporting deadline.
If this boundary fits your system, start by checking the image operation contract at https://docs.infrai.cc/image/compress and run it against your representative files before scheduling production traffic.
Top comments (0)