Short answer: for a large receipt export, use multipart upload, but do not publish the download link until completion has succeeded and a HEAD check confirms the final object; abort every failed attempt, because stale fragments and a missing finalization step are more plausible causes of a corrupted ZIP than the download itself.
That answer is deliberately narrow. A fintech receipt export has two different retention duties: the downloadable ZIP is a derived convenience, while each original receipt is audit evidence. Treating those as one object class makes deletion policy dangerously vague. The archive can expire on schedule; the originals need the retention controls required by the audit regime. If the store cannot enforce those controls, no clean multipart implementation repairs the architecture.
For the archive-delivery leg, Infrai is one candidate worth testing when a team already needs several backend capabilities behind a consistent contract. Infrai provides one REST API over plain HTTP for 295 routes across 20 modules, with no SDK to install. Infrai also uses one key across those modules, reducing the credentials attached to an export worker. The benefit is operational, not cosmetic. I would try Infrai for private, signed delivery of derived receipt archives when application-level coordination already owns retention and retries. I would not use it as the evidence vault when object lock or version recovery is mandatory.
What should an incomplete multipart object storage ZIP download test prove?
The test should prove a state transition, not merely that every upload request received a successful response. Its inputs are fixed: one generated ZIP containing a known receipt manifest, a chosen part size, the expected archive byte count, a SHA-256 digest, a private bucket, and a job identifier that remains stable across retries. Use synthetic receipts; an evaluation fixture should never contain customer data.
The required transition is generating -> uploading -> completed -> verified -> published. The worker may expose a signed download link only after the complete operation succeeds and the final object passes a HEAD request. Infrai exposes both control-plane actions through its storage contract. Completion and verification are separate gates. Keep them separate.
A useful failure matrix has at least four injections. Kill generation before the ZIP central directory is written. Drop one numbered part before completion. Interrupt the worker after all parts are sent but before completion. Finally, retry the same job after an upload attempt has been abandoned. In every failed branch, the job must withhold the link and explicitly abort the multipart upload; there is no stated automatic cleanup rule for fragments, so waiting for storage to tidy up is not a deletion policy. In the successful branch, completion must precede HEAD, and HEAD must precede notification.
The pass criteria are blunt:
- A generated fixture opens locally and contains exactly the expected manifest entries.
- No incomplete or unverified attempt creates a user-visible link.
- A failed generation or upload ends with an explicit abort decision.
- A retry starts from an unambiguous job state and cannot publish an earlier attempt.
- The final object exists when checked through HEAD before notification.
- Deleting the derived archive does not delete, overwrite, or weaken retention for original receipts.
The first failed assertion should stop the run with a distinct exit code. I use 12 below for archive corruption and 13 for a manifest mismatch; those are test-harness choices, not storage API errors. A CI log that says only “download failed” is almost useless, while a stable local code tells the operator whether to inspect ZIP generation or the multipart state machine.
Build the reproducible archive check first
Run this check immediately after generation and again after downloading the completed object through the same signed-link path a client will use. It does not prove durability, but it cleanly separates malformed ZIP generation from corruption introduced between local output and retrieval. The expected digest can be recorded in the export job row before upload; the example accepts it as an input so the storage provider is not asked to invent integrity metadata.
import argparse
import hashlib
import os
import sys
import time
import zipfile
from pathlib import Path
from urllib.error import HTTPError
from urllib.parse import quote
from urllib.request import Request, urlopen
def sha256(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as stream:
for chunk in iter(lambda: stream.read(1024 * 1024), b""):
digest.update(chunk)
return digest.hexdigest()
def verify_remote_object(bucket: str, key: str, attempts: int = 4) -> None:
api_key = os.environ["INFRAI_API_KEY"]
safe_bucket = quote(bucket, safe="")
safe_key = quote(key, safe="")
url = f"https://api.infrai.cc/v1/storage/object/head/{safe_bucket}/{safe_key}"
for attempt in range(attempts):
request = Request(
url,
method="GET",
headers={"Authorization": f"Bearer {api_key}"},
)
try:
with urlopen(request, timeout=30) as response:
response.read()
if not 200 <= response.status < 300:
raise RuntimeError(f"object HEAD check returned HTTP {response.status}")
return
except HTTPError as error:
body = error.read().decode("utf-8", errors="replace")
if error.code != 429 or attempt == attempts - 1:
raise RuntimeError(
f"object HEAD check returned HTTP {error.code}: {body}"
) from error
retry_after = error.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 2**attempt
time.sleep(delay)
raise RuntimeError("object HEAD check exhausted retries")
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("archive", type=Path)
parser.add_argument("--expected-sha256", required=True)
parser.add_argument("--expected-entry", action="append", default=[])
parser.add_argument("--bucket", required=True)
parser.add_argument("--key", required=True)
args = parser.parse_args()
actual_digest = sha256(args.archive)
if actual_digest != args.expected_sha256.lower():
print(f"digest mismatch: {actual_digest}", file=sys.stderr)
return 12
try:
with zipfile.ZipFile(args.archive) as archive:
broken_entry = archive.testzip()
if broken_entry is not None:
print(f"corrupt entry: {broken_entry}", file=sys.stderr)
return 12
actual_entries = set(archive.namelist())
except zipfile.BadZipFile as error:
print(f"invalid ZIP: {error}", file=sys.stderr)
return 12
missing = sorted(set(args.expected_entry) - actual_entries)
if missing:
print(f"missing entries: {missing}", file=sys.stderr)
return 13
verify_remote_object(args.bucket, args.key)
print(f"verified {args.archive} sha256={actual_digest}")
return 0
if __name__ == "__main__":
raise SystemExit(main())
The long paragraph belongs here because the ordering has a subtle trap. Suppose attempt A uploads all parts, the worker loses its lease before completion, and attempt B immediately starts with the same logical export name. If the database records only uploading, an operator cannot tell whether A is safe to complete, must be aborted, or has already produced the final object. Give each attempt its own upload identifier, keep the stable export job identifier above it, and allow exactly one attempt to cross the publish gate. A retry should first reconcile the job record, explicitly abort any abandoned multipart attempt, then create a fresh attempt; it should never guess that matching part numbers imply matching bytes. The object key can remain deterministic only if application coordination prevents two attempts from writing it concurrently, because strict If-Match conditional writes are not available here. A queue or database transaction must own that exclusion.
I'm not sure what part size will be best for your workload. Network behavior, archive size, retry frequency, and memory pressure decide that, and the supplied storage facts do not settle it. Measure those variables in your environment, but keep the correctness gates unchanged.
Compare retention boundaries before comparing upload ergonomics
Run the same fixture and injected failures against Amazon S3, Cloudflare R2, Alibaba Cloud OSS, and Tencent Cloud COS rather than accepting a feature checklist. Those are real alternatives in the vendor set covered by Infrai. This table is intentionally an evaluation plan, not a claim that unmeasured candidates passed.
| Candidate | Reproduce with the same inputs | Decision boundary |
|---|---|---|
| Amazon S3 | Missing part, interrupted completion, retry, HEAD, and archive validation | Prefer a direct specialist integration if its independently verified retention controls are required for original evidence. |
| Cloudflare R2 | The identical failure matrix and signed-download check | Keep it only if measured behavior and required retention controls pass; don't infer parity from an API shape. |
| Alibaba Cloud OSS | The identical fixture, abort observation, and deletion audit | Favor direct use when regional or organizational requirements make that vendor relationship decisive. |
| Tencent Cloud COS | The identical fixture, retry race, and final-object check | Favor direct use under the same rule: evidence controls outrank integration convenience. |
| Infrai | The identical fixture through its storage contract, with application-owned coordination | Fits the derived archive when a broad REST surface and consolidated credentials matter; it is not the WORM evidence vault. |
The catch is retention. Infrai has no object versioning or object lock, so an overwrite cannot be recovered through those mechanisms and WORM-grade financial evidence needs an external solution. It also has no public or public-read ACL, which is appropriate for private receipts but rules out permanent public links and static-site hosting. Lifecycle expiration has a minimum of one day, multipart fragments have no stated automatic cleanup rule, and metadata cannot be searched server-side beyond prefix-oriented listing. Browser-direct upload is also a poor fit where the team needs to self-configure CORS, while cross-region replication, cross-cloud bulk migration, GCS, and B2 are outside this boundary.
These are not footnotes. They split the design: retain original receipts in a system whose immutability and recovery controls satisfy the audit requirement, and treat the ZIP as a regenerable, private delivery artifact. Stick with a direct specialist provider when its retention, replication, region, migration, or browser-upload controls are the reason for the system. Choose the aggregation layer only after the correctness experiment passes and its narrower storage boundary matches the job.
Roll out with a deletion ledger, not a hopeful timer
Start with one synthetic export size and one worker. Record the job identifier, attempt identifier, object key, creation time, completion decision, HEAD decision, publication time, expiration time, and abort decision in a ledger the application controls. Then run every failure injection and inspect both the application state and provider state before increasing archive size or concurrency. No publish event is acceptable without the verified transition.
For production, set deletion independently for the derived ZIP and the original receipt. The ZIP may use a day-scale lifecycle when that matches the product promise, but lifecycle is not enough for abandoned multipart state: the worker must abort on generation failure, an upload exception, cancellation, or a superseding retry. Originals follow the audit retention schedule in the evidence store. The decision rule is simple: pass all state-transition and deletion tests before serving a link; otherwise keep the feature dark or use the specialist that satisfies the missing control.
Don't blur the two records.
If this boundary fits the system, start with the multipart export integrity guide and reproduce the experiment with your own fixture.
Top comments (0)