Before you move a bucket to escape egress fees, measure two things: your byte-weighted cache hit ratio and your requests per unique object. A low hit ratio caused by unstable cache keys is a bug you can fix in an afternoon; a low hit ratio caused by genuinely one-off assets is a structural fact no CDN can fix, and that's the case where zero-egress storage earns the migration. Most teams never separate the two and end up paying a migration to hide a caching bug.
A commenter on an earlier post about storage egress put it well: audit how many unique assets you serve versus how many come from cache, before you touch anything.
Why doesn't my egress bill match my download volume?
Because two meters are running, and they bill different bytes.
- CDN data-out: every byte delivered to a browser, cached or not.
- Origin egress: only the bytes your CDN had to fetch from the bucket — your cache misses.
If you serve 5 TB to users at a 95% hit ratio, your bucket only emitted about 250 GB. That's the number a storage migration changes. The other 4.75 TB is billed by whoever runs your edge, and moving the bucket doesn't touch it.
This is why the AWS-native path confuses people: S3-to-CloudFront transfer is free, so putting CloudFront in front of S3 relabels the egress bill as CloudFront data-out rather than removing it. Cloudflare doesn't meter CDN bandwidth per GB, though as of mid-2026 its terms restrict serving large volumes of non-HTML content on non-enterprise plans.
Takeaway: a storage migration only moves the origin-egress meter, so measure cache misses — not total downloads — before estimating the savings.
How do I measure cache hit ratio without buying an analytics add-on?
Start with two curls. The first fills the cache, the second should hit it:
BASE=$(printf 'https://%s' "cdn.example.invalid") # your edge hostname
for i in 1 2; do
curl -sSI "$BASE/u/9f3ab2/avatar.webp" \
| grep -iE '^(cf-cache-status|x-cache|age|cache-control|vary):'
done
If the second response still says Miss from cloudfront or cf-cache-status: MISS, your cache key is unstable and no TTL tuning will help. An Age header that resets to 0 on every request is the same symptom with a friendlier face.
Two curls tell you about one object. For the real number, parse the logs. CloudFront access logs are tab-separated with a #Fields: header line, so map columns instead of hardcoding positions:
#!/usr/bin/env python3
"""Byte-weighted hit ratio and asset fan-out from CloudFront access logs."""
import gzip, sys
from collections import Counter
HIT = {"Hit", "RefreshHit", "OriginShieldHit"}
requests_by_obj, bytes_by_obj = Counter(), Counter()
hit_bytes = miss_bytes = 0
for path in sys.argv[1:]:
opener = gzip.open if path.endswith(".gz") else open
with opener(path, "rt") as fh:
fields = None
for line in fh:
if line.startswith("#Fields:"):
fields = line.split()[1:]
continue
if line.startswith("#") or not fields:
continue
row = dict(zip(fields, line.rstrip("\n").split("\t")))
size = int(row.get("sc-bytes") or 0)
requests_by_obj[row["cs-uri-stem"]] += 1
bytes_by_obj[row["cs-uri-stem"]] += size
if row.get("x-edge-result-type") in HIT:
hit_bytes += size
else:
miss_bytes += size
total_bytes = hit_bytes + miss_bytes
if not total_bytes:
sys.exit("no log rows parsed - check the file paths")
uniq = len(requests_by_obj)
top = bytes_by_obj.most_common(50)
print(f"byte-weighted hit ratio : {hit_bytes / total_bytes:.1%}")
print(f"origin egress (misses) : {miss_bytes / 1e9:.2f} GB")
print(f"unique objects : {uniq:,}")
print(f"requests per object : {sum(requests_by_obj.values()) / uniq:.1f}")
print(f"top 50 objects : {sum(b for _, b in top) / total_bytes:.1%} of bytes")
For Cloudflare Logpush (newline-delimited JSON), the same numbers stream out of jq plus awk:
zcat logs/*.log.gz \
| jq -r '[.CacheCacheStatus, .EdgeResponseBytes, .ClientRequestURI] | @tsv' \
| awk -F'\t' '
{ bytes += $2; if ($1 == "hit") hits += $2
if (!($3 in seen)) { seen[$3] = 1; uniq++ } }
END { printf "hit ratio : %.1f%%\nunique URIs: %d\nreq/URI : %.1f\n",
100 * hits / bytes, uniq, NR / uniq }'
Two caveats. sc-bytes is bytes sent to the viewer, so ranged and aborted downloads make it an approximation of origin fetch bytes — good enough to size a decision, not to reconcile an invoice. Counting only hit as a hit is deliberately conservative: revalidated still costs an origin round trip.
Takeaway: byte-weighted hit ratio is the only version of the metric that maps to money — a 99% request hit ratio means nothing if the 1% misses are your video files.
What makes a hit ratio quietly collapse?
Rarely short TTLs, which is what everyone checks first. It's almost always that the cache key isn't stable across requests for the same bytes:
-
Presigned URLs. An S3 presigned URL carries
X-Amz-SignatureandX-Amz-Datein the query string, and both change on every generation. If query parameters are in your cache key, every request is a unique object to the edge and your hit ratio is structurally zero. This is the big one. -
Cache-busting parameters added by an image component or analytics wrapper (
?v=<timestamp>,?_=<random>). - Forwarded cookies. Include a session cookie in the key and your hit ratio equals your per-user re-download rate.
-
Vary: User-AgentorVary: Acceptfrom an image-transform layer, fragmenting one object into many variants — as do multiple hostnames for the same asset.
Check your cache policy's query-string, header, and cookie inclusion lists, then re-run the two-curl test with a stripped URL to confirm which one it is.
Presigned URLs can't be made cacheable — you can only stop using them on the hot path. What works is a path-based token with a quantized window, so every request inside that window produces byte-identical URLs:
import hashlib, hmac, time
WINDOW = 3600 # seconds; also your worst-case revocation lag
def asset_path(key: str, secret: bytes) -> str:
slot = int(time.time()) // WINDOW
mac = hmac.new(secret, f"{key}:{slot}".encode(), hashlib.sha256)
return f"/a/{slot}/{mac.hexdigest()[:16]}/{key}"
Your edge (a worker, Lambda@Edge, or your own origin) recomputes the HMAC for the current and previous slot and rejects anything else. The drawback is real: revocation is coarse, so a link stays valid up to one full window after you cut access. Fine for avatars and exports; for sensitive objects, keep presigned URLs and accept that those bytes are always origin egress.
Takeaway: if assets are served through rotating signatures, your hit ratio isn't low — it's zero by construction, and that's a URL design problem, not a storage problem.
What do the two numbers actually tell me to do?
| Requests per unique object | Byte-weighted hit ratio | Diagnosis | Fix first |
|---|---|---|---|
| High (20+) | High (90%+) | Working as intended | Nothing — origin egress is already small |
| High (20+) | Low (under 60%) | Unstable cache key or too-short TTLs | Cacheability; a migration would hide a bug |
| Low (1–3) | Low, and it can't go higher | Genuine long tail: one-off generated files | Storage with no egress charge |
| Low overall, bytes concentrated in a few objects | Mixed | A handful of large files dominate | Long TTLs on those; leave the rest alone |
The third row is the one people miss. AI-generated assets — a rendered PDF, a one-off image, an export requested once and never again — have a fan-out near 1. Caching them is nearly pointless: you pay the origin fetch either way, and the object is evicted before a second request that never comes. If you want the managed version of this, Cloudflare R2 is the one that removes the egress meter entirely instead of discounting it, so long-tail traffic stops being a variable cost. Backblaze B2 gets most of the way there with free egress up to a multiple of what you store — and free to Cloudflare's network via the Bandwidth Alliance — until your read-to-store ratio drifts past that multiple.
Takeaway: fan-out near 1 is the only signal that reliably justifies migrating for egress, because it's the one thing better caching cannot improve.
How do I turn this into a before/after number?
Take the miss bytes the script printed — that's the monthly origin egress you actually pay for. Price that figure two ways: per GB above whatever allowance applies (S3, and B2 past its included multiple), or zero (R2, and B2 within the allowance). Then add operations, because with fan-out near 1 every delivered object is roughly one origin GET, and on R2 the Class A writes on the upload side are the pricier meter. Storage is usually the rounding error, which is why ranking providers by per-GB storage rate ranks them wrong.
Run the script against a month of logs before and after any cacheability fix. In my experience the fix lands first and the number moves enough that the migration case either becomes obvious or evaporates.
Takeaway: the migration decision is one subtraction — miss bytes times the egress rate, minus the operations you'd pay at the new provider.
FAQ
How do I check if my CDN is caching my S3 files?
Request the same URL twice with curl -sSI and read the x-cache (CloudFront) or cf-cache-status (Cloudflare) header. The first response will be a miss; if the second is also a miss, something in your cache key — query strings, cookies, or a Vary header — differs between the two requests.
Does putting a CDN in front of S3 eliminate egress fees?
No, it moves them. S3-to-CloudFront transfer is free, so the bucket's egress drops to near zero, but CloudFront then bills data-out to viewers. The saving comes from cache hits reducing origin fetches, not from the CDN being free.
What's a good cache hit ratio for user-uploaded assets?
For shared assets like avatars and thumbnails, aim for 90%+ byte-weighted. For one-off generated files with a fan-out near 1, even 30% may be the ceiling — there the ratio isn't the problem to solve, and egress pricing models are.
Bottom line
Run the log script before you run a migration. If requests per unique object is high and the hit ratio isn't, you have a cache-key bug — fix the presigned URLs or the stray query strings and the egress bill drops without changing providers. If fan-out is genuinely near 1, caching has nothing left to give, and zero-egress storage like R2 (or B2 within its free-egress multiple) is the honest answer. Either way, the two numbers cost you an afternoon and tell you which of those two projects you're actually signed up for.
Top comments (0)