Authored by Mike Neville-O'Neill
A company processing tens of TBs of Fastly logs daily through traditional logging vendors faces an impossible choice: pay enterprise prices for comprehensive logging, or fly blind with minimal retention.
That's why most teams default to a 7-day retention window. Not because they don't need historical data — because the economics don't work.
That was the exact situation for Contentstack, a leading global platform for digital experiences. They now keep a full year of logs, run unlimited queries, and report 50% cost savings versus their previous solution.
The Real Cost of Restricted Logging
Processing 10TB of Fastly logs daily can cost ~$30,000/month for ingestion alone on traditional platforms. To reduce costs, teams resort to:
- Abbreviating field names (
EdgeResponseStatus→ers) to save bytes - Sampling 1 in 100 requests and hoping issues appear in that sample
- Keeping only 7 days of data when seasonal patterns span months
- Dropping entire log categories to keep storage costs down
When ingestion and storage are expensive, comprehensive logging becomes a luxury instead of standard practice. But what if the economics actually worked?
What Changes When You Can Keep Everything
Bronto's architectural efficiency — 90% compression, optimized storage, purpose-built for logs — means you pay 50–90% less to ingest the same data. This changes everything.
Consider tracking API performance across enterprise customers:
-- Monitoring performance across multiple API customers
SELECT count(*), avg(resTime)
WHERE (apiKey = 'customer_key_1' OR apiKey = 'customer_key_2' OR apiKey = 'customer_key_3')
AND resTime > 500
AND cache != 'MISS, MISS'
GROUP BY shield, cache
-- Time range: 30 days
When ingestion is affordable, you can actually keep the 30 days of high-volume logs this investigation requires. No sampling, no abbreviations, no compromises.
30-Day Performance Archaeology
A customer report of intermittent slowdowns triggered a deep investigation:
SELECT max(resTime), avg(resTime), count(*)
WHERE apiKey = 'customer_12345'
GROUP BY shield, cache
LIMIT 100
-- Analyzing response times week-over-week for the past month
Analyzing max and average response times over 30 days revealed certain endpoints were gradually slowing down — a pattern invisible in a 7-day window but clear with month-long data.
The team spotted week-over-week degradation that would have caused a critical outage within weeks. With 7-day retention, they never would have seen it coming.
Multi-Month Error Forensics
When 404 errors started appearing randomly across the CDN, the team needed to know: new issue or recurring pattern?
SELECT count(*)
WHERE url ILIKE '%/api/v2/content/%'
AND response_status = '404'
GROUP BY cache, shield, pop
LIMIT 100
-- 90-day investigation of 404 errors for content API endpoints
Filtering 90 days of complete, unsampled logs revealed what looked like random errors was actually a cache invalidation race condition — one that only occurred under specific geographic and timing conditions. With 7-day windows and aggressive sampling, this would have remained an unsolved mystery.
The Intelligence Layer: From Logs to Insights
Long-term retention unlocks intelligence — but only if you can query it. Bronto's analytics engine lets you:
- Run SQL queries across months of data in seconds
- Set up change detection that automatically flags when cache hit rates drop
- Build filterable dashboards to click into any spike or anomaly
- Use natural language with the AI dashboard builder: "Show me API response times by customer and region for the last 90 days"
- Use BrontoScope to interrogate logs with context, correlating events and surfacing root causes that would take hours to piece together manually
What Becomes Possible
API customer journey mapping — track the entire API consumer lifecycle:
- Which endpoints do new users test first?
- How does usage scale over months?
- What usage patterns precede account downgrades?
Intelligent cache optimization — with a year of data you can finally answer:
- How do cache hit ratios vary by season, not just by day?
- Which content genuinely needs global distribution vs. regional?
- How do different shield configurations perform under various traffic patterns?
Security pattern recognition — long-term retention enables sophisticated threat detection:
- Bot networks that rotate through IP ranges over weeks
- Credential stuffing campaigns that evolve over months
- DDoS reconnaissance that occurs weeks before an attack
Before and After Bronto
| Before | After |
|---|---|
| Field names abbreviated to save bytes | Full field names: readable and searchable |
| Aggressive sampling: keeping 1% of logs | 100% coverage: every request, response, error |
| Short retention: 7 days max | 365-day retention: full seasonal patterns visible |
| Blind spots: log categories dropped to save costs | Complete visibility: CDN and compute logs |
Technical Foundation: Built for Fastly's Scale
Bronto integrates with Fastly through a built-in logging endpoint. Configure it once in your Fastly service settings and logs start streaming in minutes:
# Simple Fastly Integration
name: Bronto
format: JSON
url: https://logs.bronto.io/fastly
compression: zstd # 10:1 automatic compression
method: POST
header_name: X-Bronto-Token
header_value: ${BRONTO_TOKEN}
Why it scales:
- Elastic ingestion that grows with traffic
- 90% data compression
- Time-series optimization for CDN access patterns
- Intelligent tiering keeps recent data hot, historical data accessible
- Column-oriented storage for efficient aggregations
Bronto Express: From Ingestion to Insights in Minutes
Setting up this level of insight shouldn't require weeks of work. Bronto Express for Fastly uses AI to take raw log streams and dynamically build Fastly-specific queries, monitors, and dashboards personalized for your organization.
It includes pre-built dashboards for real-time performance, geographic traffic, API usage, and security monitoring; a saved searches library for common investigations; and intelligent monitors for performance degradation and error rate thresholds.
Setup is four steps:
- Create your Bronto instance (selecting the Fastly Express pack)
- Add Bronto as a logging endpoint in Fastly (5-minute configuration)
- Watch pre-built dashboards populate with your data
- Customize searches and alerts for your specific needs
No agents. No forwarders. No complex pipelines.
The Platform Effect: When Every Team Has Access
When CDN logs become affordable and accessible, every team benefits:
- Engineering — debugs issues with complete historical context
- Security — investigates threats across meaningful timeframes
- Product — understands real usage patterns, not samples
- Sales — identifies expansion opportunities from actual usage data
- Finance — accurately allocates costs to business units
- Support — resolves tickets with full forensic capability
This isn't just about saving money on logging (though 50–90% savings helps). It's about removing the artificial constraints that prevent teams from fully understanding their infrastructure.
Top comments (0)