Ever chased a bug for days, only to discover the "bug" was actually the platform working exactly as designed? That happened to me building a client reporting pipeline. The lesson stuck.
Here's what nobody tells you about pulling marketing data from multiple ad platforms: the hard part was never the dashboard. It was everything underneath it.
The Setup That Looked Simple on Paper
The brief sounded easy. Pull spend, clicks, and conversions from Google Ads and Meta. Store it. Display it in a chart. A junior dev could knock this out in a sprint, I figured.
Reality disagreed. Google Ads API enforces operation quotas per developer token, and those quotas scale differently depending on account tier. Meanwhile, Meta's Marketing API throttles based on a rolling usage score tied to the ad account itself, not your app.
Two platforms. Two completely different throttling philosophies. Neither documented in a way that made the actual limits obvious until you hit them in production.
Where Things Actually Broke
My first version polled every client account every hour. Fine for three clients. Then we onboarded client number twelve, and Meta started returning 429s intermittently. Not consistently — intermittently. That's the worst kind of bug.
I initially assumed it was a code issue. Retry logic, maybe a race condition in my job scheduler. I spent three weeks going down that path. Eventually, I found the real cause: cumulative API call volume across all client accounts was tripping Meta's app-level rate limit, not the individual account limit.
The fix wasn't more retries. It was a request queue with exponential backoff, plus a priority system so active dashboards refreshed before idle ones. Simple in hindsight. Expensive in dev hours.
The Real Architecture Behind Multi-Platform Reporting
If you're building this yourself, here's what a production-grade pipeline actually needs, based on what broke for me.
A Queue, Not a Cron Job
Don't just fire off API calls on a schedule and hope for the best. Use a proper job queue — something like BullMQ or Sidekiq — with retry policies and backoff built in from day one. This alone would've saved me those three weeks.
Token Refresh as a First-Class Concern
OAuth tokens expire silently. Build monitoring specifically for auth failures, separate from your general error logging. Otherwise, you'll find out a client's Google Ads token expired only when they ask why their dashboard looks empty.
A Normalization Layer Between Raw Data and Storage
Google Ads calls it "cost." Meta calls it "spend." GA4 buries similar metrics under different dimension names entirely. Build one internal schema and map every platform into it, rather than letting frontend code handle platform-specific field names.
javascript
// Simplified normalization example
function normalizeMetric(platform, rawData) {
const mapping = {
google_ads: { cost: rawData.cost_micros / 1e6, clicks: rawData.clicks },
meta: { cost: rawData.spend, clicks: rawData.clicks },
};
return mapping[platform] || rawData;
}
This function looks trivial. It isn't, once you're handling currency conversions, timezone mismatches, and attribution window differences across five platforms simultaneously.
Should You Actually Build This?
Here's the honest answer, from someone who's now maintained this kind of system for two years. If reporting infrastructure isn't your product's core differentiator, building it yourself is usually the wrong call.
I don't say that lightly. I like building things. However, every hour spent debugging Meta's rate limit quirks was an hour not spent on features that actually made our product better for users.
What Changed My Mind
Eventually, we moved most of our client reporting workflow onto a dedicated platform, RaiseReturn, instead of continuing to maintain our in-house pipeline. It already handled the multi-platform normalization, the token refresh monitoring, and the rate-limit-aware queuing I'd built by hand.
The API integrations existed already, tested against the platforms' quirks by a team that deals with exactly this problem full-time. That's not something you replicate cheaply in a side project or even a dedicated sprint.
Where Custom Code Still Makes Sense
To be clear, this isn't an argument against ever building integrations. If your reporting needs pull from truly proprietary internal systems — a custom CRM, an internal data warehouse — no off-the-shelf tool covers that. You'll build custom connectors regardless.
For standard ad platforms though, the problem is already solved. Well-tested, production-hardened, and maintained by people who get paged when Meta ships a breaking API change at 2 AM. You probably don't want to be that person.
Practical Takeaways for Your Own Stack
If you're currently building or maintaining a custom reporting pipeline, a few things worth checking right now.
First, verify your retry logic actually implements exponential backoff, not fixed-interval retries. Fixed intervals make rate limiting worse, not better, under sustained load.
Second, add dedicated alerting for OAuth token failures. Silent auth failures are the most common cause of "why is this client's data missing" tickets.
Third, honestly evaluate the maintenance cost against a dedicated automated reporting tool. Sometimes the math favors buying. Sometimes it doesn't. Either way, run the numbers before defaulting to "we'll just build it."
Final Thought
Client reporting infrastructure is a genuinely interesting engineering problem. It's also, for most teams, not the problem worth solving from scratch. Know the difference, and your future 2 AM self will thank you.
Top comments (0)