DEV Community: Pallab Mondal

Stop Building SEO Reports by Hand: Automating Client Reporting With APIs, SQL, and Scheduled Jobs

Pallab Mondal — Tue, 08 Sep 2026 23:23:16 +0000

If you manage SEO for more than a couple of websites, you eventually hit the same wall.

The SEO work itself isn't always the slow part.

The reporting is.

Every month, you open Google Search Console. Then Google Analytics. Maybe PageSpeed Insights. You export CSV files, adjust date ranges, copy numbers into a spreadsheet, update charts, write a summary, check that nothing is obviously wrong, and finally turn everything into a PDF.

Then you do it again for the next client.

And the next one.

At first, this feels manageable. After all, you only need an hour or two per report.

But multiply that by 10, 20, or 50 clients and the problem becomes obvious.

You're not really doing SEO reporting anymore.

You're maintaining a manual data pipeline.

The good news is that most of this process can be automated with fairly ordinary tools: APIs, Python, SQL, scheduled jobs, and a reporting template.

You don't need a massive data platform.

You need a reliable pipeline.

This article walks through how I'd approach building one.

The Real Problem Isn't the Report

A typical SEO report contains several different types of information:

Organic clicks
Impressions
Click-through rate
Average search position
Top queries
Top landing pages
Organic sessions
Conversions
Technical issues
Period-over-period changes
Recommendations

The difficult part isn't displaying those numbers.

The difficult part is getting the right data, consistently, for the right date range, then turning it into something a client can understand.

A manual process might look like this:

Google Search Console
 ↓
CSV export
 ↓
Spreadsheet
 ↓
Manual calculations
 ↓
Charts
 ↓
Written summary
 ↓
PDF
 ↓
Client

An automated process can look more like this:

                 ┌──────────────────┐
                 │ Google Search     │
                 │ Console API       │
                 └────────┬─────────┘
                          │
                          ▼
┌──────────────┐   ┌───────────────┐
│ Google       │──▶│ Data          │
│ Analytics API│   │ Collection    │
└──────────────┘   └───────┬───────┘
                           │
                           ▼
                    ┌─────────────┐
                    │ PostgreSQL  │
                    └──────┬──────┘
                           │
                           ▼
                    ┌─────────────┐
                    │ Calculations│
                    └──────┬──────┘
                           │
                           ▼
                    ┌─────────────┐
                    │ Report      │
                    │ Generator   │
                    └──────┬──────┘
                           │
                           ▼
                     PDF / HTML
                           │
                           ▼
                        Client

That distinction is important.

You're not automating a PDF.

You're automating the system that produces the PDF.

Decide What the Report Should Answer

Before touching an API, decide what questions your report should answer.

Clients rarely care about a number just because it exists.

They care about questions such as:

Did organic traffic grow?
Did search visibility improve?
Which pages generated the most traffic?
Which pages lost traffic?
Are rankings moving in the right direction?
Did organic visitors generate leads or sales?
What changed compared with last month?
What should we work on next?

A practical overview of what belongs in an SEO report is also worth keeping nearby when designing the template.

The answers determine what data you actually need.

For example, if your goal is to report on organic growth, Search Console data might be enough for search performance.

If you want to connect SEO to business results, you'll probably also need Google Analytics data.

If you're reporting on technical SEO, you'll need another data source entirely.

The first lesson is simple:

Don't collect data just because an API makes it available.

Collect data because it answers a useful question.

Start With Google Search Console

Google Search Console is usually the best place to start.

The Search Console API gives you access to performance information such as:

Clicks
Impressions
CTR
Average position
Queries
Pages
Countries
Devices
Search appearance

You can request a date range and optionally group the results by dimensions such as query or page.

Conceptually, a response might look like this:

{
  "rows": [
    {
      "keys": ["seo automation"],
      "clicks": 124,
      "impressions": 2400,
      "ctr": 0.0516,
      "position": 8.4
    },
    {
      "keys": ["automated seo reporting"],
      "clicks": 97,
      "impressions": 1800,
      "ctr": 0.0538,
      "position": 10.2
    }
  ]
}

You can then transform that response into your own internal format.

For example:

keyword = {
    "query": row["keys"][0],
    "clicks": row["clicks"],
    "impressions": row["impressions"],
    "ctr": row["ctr"],
    "position": row["position"]
}

Once the data is inside your application, you shouldn't care too much about where it originally came from.

That's an important architectural decision.

Your report generator shouldn't need to know whether a metric came from Search Console, Analytics, a CSV file, or a database.

Compare Periods Instead of Showing Snapshots

A report saying:

Organic clicks: 14,320

isn't particularly useful.

Is that good?

Was it 12,000 last month?

Was it 20,000?

Without context, the number means very little.

Instead, calculate changes.

For example:

def percent_change(current, previous):
    if previous == 0:
        return None

    return ((current - previous) / previous) * 100

Then:

current_clicks = 14320
previous_clicks = 12000

change = percent_change(current_clicks, previous_clicks)

print(f"{change:.1f}%")

The result is:

19.3%

Now your report can say:

Organic clicks increased 19.3% compared with the previous period.

That's much more useful than displaying two disconnected numbers.

You can use the same approach for:

Impressions
Sessions
Conversions
Revenue
Landing-page traffic
Keyword groups
Technical issues

The comparison itself often contains more information than the raw metric.

Then Bring in GA4

Search Console tells you what happened in Google Search.

Google Analytics can help answer what happened after someone arrived.

Depending on the reporting goal, you might collect:

Organic sessions
Engaged sessions
Users
Conversions
Revenue
Landing pages
Engagement metrics

A simple landing-page dataset might look like this:

[
  {
    "page": "/seo-guide",
    "sessions": 1820,
    "conversions": 42
  },
  {
    "page": "/technical-seo",
    "sessions": 1240,
    "conversions": 31
  },
  {
    "page": "/keyword-research",
    "sessions": 980,
    "conversions": 18
  }
]

You can combine this with Search Console information.

For example:

Search Console
    ↓
Which pages receive search visibility?

GA4
    ↓
Which pages receive organic visitors?

Conversions
    ↓
Which pages contribute to business outcomes?

That gives you a much stronger report.

Instead of:

This page ranks well.

You can potentially say:

This page is receiving significant organic visibility and is also contributing to conversions.

That's the difference between an SEO report and a collection of SEO metrics.

Authentication Is Where the Fun Begins

APIs are convenient once authentication works.

Getting authentication configured is often the annoying part.

For Google APIs, you'll typically need to deal with:

A Google Cloud project
API access
Credentials
OAuth or service-account authentication
Appropriate permissions
Property/account access

For a single internal project, you can keep this fairly simple.

For an agency platform, things become more interesting.

You may have:

Client A
  ├── Search Console property
  ├── GA4 property
  └── Credentials

Client B
  ├── Search Console property
  ├── GA4 property
  └── Credentials

Client C
  ├── Search Console property
  ├── GA4 property
  └── Credentials

Your database might contain something like:

CREATE TABLE clients (
    id SERIAL PRIMARY KEY,
    name TEXT NOT NULL,
    search_console_property TEXT,
    ga4_property_id TEXT
);

Credentials should not simply be stored as plaintext database fields.

Use environment variables, a secret manager, or encrypted credential storage appropriate for your deployment.

For example:

GOOGLE_APPLICATION_CREDENTIALS=/secrets/google-service-account.json

The exact authentication architecture will depend on whether you're building a personal tool, an agency platform, or a SaaS product.

Don't Generate the PDF Directly From the API

This is one of the architectural mistakes I'd avoid.

You don't want code that effectively does this:

API request
   ↓
PDF generation

It works initially.

Then the report becomes difficult to maintain.

Instead, create layers.

API
 ↓
Raw data
 ↓
Normalized data
 ↓
Metrics
 ↓
Report model
 ↓
Template
 ↓
PDF

For example, your normalized object could look like:

report_data = {
    "period": {
        "current": "2026-08-01/2026-08-31",
        "previous": "2026-07-01/2026-07-31"
    },
    "organic_search": {
        "clicks": 14320,
        "impressions": 248000,
        "ctr": 0.0577,
        "position": 9.8
    },
    "analytics": {
        "organic_sessions": 17640,
        "conversions": 412
    }
}

Now your PDF generator doesn't need to understand Google APIs.

It just receives structured data.

That makes testing much easier.

You can even generate a report from a fake dataset without making a single external API call.

Store Historical Data

If you're serious about automation, don't rely entirely on live API calls.

Store historical data.

A simple PostgreSQL table might look like:

CREATE TABLE seo_daily (
    id BIGSERIAL PRIMARY KEY,
    client_id INTEGER NOT NULL,
    date DATE NOT NULL,
    clicks INTEGER DEFAULT 0,
    impressions INTEGER DEFAULT 0,
    ctr NUMERIC,
    position NUMERIC,
    organic_sessions INTEGER DEFAULT 0,
    conversions INTEGER DEFAULT 0
);

Then your reporting system can query historical data:

SELECT
    date,
    SUM(clicks) AS clicks,
    SUM(impressions) AS impressions,
    SUM(organic_sessions) AS sessions,
    SUM(conversions) AS conversions
FROM seo_daily
WHERE client_id = 42
GROUP BY date
ORDER BY date;

This gives you several advantages.

Historical comparisons

You can compare:

Month over month
Quarter over quarter
Year over year
Custom periods

Faster reports

Your report generator doesn't have to request everything from external APIs every time.

Debugging

If an API returns strange data today, you can inspect exactly what was stored previously.

Reproducibility

You can regenerate an old report using the data that existed when the report was originally produced.

That's valuable when clients ask:

"Can you resend the report from six months ago?"

Keep Raw and Processed Data Separate

A useful pattern is to distinguish between raw data and processed data.

For example:

Raw API response
      ↓
Normalization
      ↓
Business calculations
      ↓
Report data

Raw data might look messy.

That's okay.

The normalized layer should be predictable.

For example:

{
    "date": "2026-08-31",
    "source": "search_console",
    "metric": "clicks",
    "value": 14320
}

Then your calculation layer can work with consistent structures.

This also means you can change your report design without changing your API integration.

That's exactly what you want.

Build the Report Around Decisions

A common reporting mistake is starting with charts.

Instead, start with decisions.

Imagine these two sections.

Version A

Organic clicks: 14,320
Impressions: 248,000
CTR: 5.77%
Average position: 9.8

Version B

Organic clicks increased 19% compared with the previous period.

The strongest growth came from informational pages,
while two commercial landing pages lost visibility.

The next priority is improving internal linking to the
declining commercial pages and refreshing their content.

Version B is much more useful.

The numbers can still be included.

But they support the explanation.

A good report should answer:

So what?

That's the question that turns analytics into strategy.

Add Simple Anomaly Detection

You don't need machine learning to make automated reporting smarter.

Start with rules.

For example:

def detect_change(current, previous, threshold=20):
    if previous == 0:
        return None

    change = ((current - previous) / previous) * 100

    if abs(change) >= threshold:
        return {
            "change": change,
            "significant": True
        }

    return {
        "change": change,
        "significant": False
    }

Then you can flag situations like:

Clicks ↓ 28%
Conversions ↓ 34%

or:

Impressions ↑ 41%
Clicks ↑ 3%

The second example is interesting.

Visibility increased substantially, but clicks barely moved.

That could justify investigating:

Search intent
SERP features
Titles
Meta descriptions
Ranking distribution
Query mix

The system doesn't need to diagnose everything automatically.

It just needs to surface things worth investigating.

Schedule the Whole Pipeline

Once your pipeline works manually, schedule it.

A simple architecture might be:

Every Monday at 06:00
        ↓
Fetch API data
        ↓
Validate response
        ↓
Store raw data
        ↓
Normalize data
        ↓
Calculate metrics
        ↓
Detect anomalies
        ↓
Generate report
        ↓
Upload PDF
        ↓
Send email

For a small deployment, cron may be enough.

Example:

0 6 * * 1 /usr/bin/python3 /app/run_reports.py

For larger systems, you might eventually use:

Celery
Cloud Scheduler
GitHub Actions
AWS EventBridge
Google Cloud Scheduler
Kubernetes CronJobs
A managed workflow system

Don't start with the most complicated scheduler you can find.

Start with the simplest reliable option.

Make Failures Boring

Automated systems fail.

APIs timeout.

Credentials expire.

Rate limits happen.

A property gets removed.

A database connection fails.

The goal isn't to eliminate every failure.

The goal is to make failures easy to understand.

For example:

import logging

logger = logging.getLogger(__name__)

try:
    data = fetch_search_console_data()
except Exception:
    logger.exception("Search Console request failed")

You should ideally track:

START
  ↓
FETCH
  ↓
VALIDATE
  ↓
STORE
  ↓
CALCULATE
  ↓
GENERATE
  ↓
SEND
  ↓
SUCCESS

If something breaks:

START
  ↓
FETCH
  ↓
ERROR

You immediately know where to look.

Good logging is one of those boring engineering practices that becomes incredibly useful once you have dozens of automated jobs running.

Add Retries Carefully

Temporary API failures shouldn't necessarily kill the entire report.

A simple retry mechanism can help:

import time

def retry(func, attempts=3, delay=2):
    for attempt in range(attempts):
        try:
            return func()
        except Exception:
            if attempt == attempts - 1:
                raise

            time.sleep(delay * (attempt + 1))

But don't blindly retry every error.

A retry makes sense for things like:

Temporary network errors
Timeouts
Some server-side failures

It doesn't fix:

Invalid credentials
Invalid property IDs
Permission problems
Malformed requests

Good retry logic should distinguish temporary problems from permanent ones.

Generate Different Report Formats From the Same Data

Once you have a clean report model, you can generate multiple outputs.

For example:

Normalized report data
        │
        ├── PDF
        ├── HTML
        ├── Email
        ├── Dashboard
        ├── JSON
        └── Spreadsheet

This is another reason not to couple your calculations directly to the PDF.

Your report model might be:

report = {
    "summary": {},
    "organic_search": {},
    "landing_pages": [],
    "conversions": {},
    "technical": {},
    "recommendations": []
}

The PDF template consumes it.

An email template consumes it.

A dashboard consumes it.

A future API endpoint consumes it.

One data model.

Multiple outputs.

A Practical Report Structure

If I were building an automated client report today, I'd probably start with something like this.

1. Executive Summary

Three to five important observations.

Not 20 metrics.

For example:

Organic traffic increased 18% this month.

Search visibility also improved, with impressions increasing 31%.

However, conversions grew only 4%, suggesting that traffic quality
should be investigated before scaling content production.

2. Organic Search Performance

Include:

Clicks
Impressions
CTR
Average position
Period comparison

A line chart can show the trend.

3. Top Landing Pages

Show:

Page	Clicks	Sessions	Conversions
/seo-guide	3,420	4,120	86
/technical-seo	2,870	3,180	72
/keyword-research	2,140	2,630	51

But don't stop there.

Highlight unusual changes.

For example:

/technical-seo

Traffic: +38%
Conversions: +52%

That's much more useful than simply ranking pages from highest to lowest.

Don't Automate the Human Explanation Too Aggressively

Automation can easily produce terrible writing.

For example:

Organic clicks increased by 19.3%, indicating a positive performance trend and suggesting that SEO visibility has improved during the reporting period.

Technically correct.

Painfully robotic.

A better approach is to use structured templates and leave room for human judgment.

For example:

if clicks_change > 15 and conversions_change > 10:
    summary = (
        "Organic search improved strongly this period, "
        "with both traffic and conversions moving higher."
    )

If traffic rises but conversions fall:

if clicks_change > 15 and conversions_change < -10:
    summary = (
        "Search traffic increased, but conversions declined. "
        "The next step is to investigate whether the additional "
        "traffic is reaching the right landing pages."
    )

These aren't meant to replace an SEO professional.

They're meant to handle the repetitive 80%.

The human should still be able to edit the final interpretation.

Where AI Actually Helps

AI can be useful here, but I'd give it a narrow job.

Don't send raw API responses to an AI model and ask:

"Write my SEO report."

That's an easy way to get invented conclusions.

Instead, calculate the facts yourself.

Then provide structured information.

For example:

{
  "organic_clicks_change": 19.3,
  "organic_sessions_change": 17.8,
  "conversions_change": 4.1,
  "top_growth_page": "/seo-guide",
  "declining_page": "/technical-seo",
  "technical_issues": 3
}

Then ask the model to interpret those facts.

You can constrain it further:

Use only the supplied metrics.

Do not invent causes.

Do not invent rankings.

Do not claim that a change was caused by an algorithm update
unless that information is explicitly provided.

Return:
1. Three observations
2. Two risks
3. Three recommended next actions

That's a much safer use of AI.

The application owns the facts.

The model helps turn structured facts into readable language.

The Architecture I'd Build

For a small SEO agency, I'd keep the first version relatively boring.

Something like:

                  ┌──────────────────┐
                  │ Search Console   │
                  └────────┬─────────┘
                           │
                           ▼
                  ┌──────────────────┐
                  │ Data Collector   │
                  └────────┬─────────┘
                           │
                  ┌────────┴─────────┐
                  │                  │
                  ▼                  ▼
          ┌──────────────┐   ┌──────────────┐
          │ Raw Storage  │   │ GA4 Collector│
          └──────┬───────┘   └──────┬───────┘
                 │                  │
                 └────────┬─────────┘
                          ▼
                  ┌──────────────┐
                  │ PostgreSQL   │
                  └──────┬───────┘
                         │
                         ▼
                  ┌──────────────┐
                  │ Calculations │
                  └──────┬───────┘
                         │
                         ▼
                  ┌──────────────┐
                  │ Report Model │
                  └──────┬───────┘
                         │
                ┌────────┴─────────┐
                ▼                  ▼
          ┌───────────┐      ┌───────────┐
          │ PDF       │      │ HTML      │
          └───────────┘      └───────────┘

The stack could be surprisingly simple:

Python
PostgreSQL
Google APIs
Jinja2
HTML/CSS
PDF renderer
Cron
Object storage

You don't need Kubernetes on day one.

You don't need a distributed event system.

You don't need 17 microservices.

You need a reliable pipeline.

What I Would Automate First

If you're starting from zero, I'd build it in this order.

Step 1: Automate data collection

Get Search Console data without manual exports.

Step 2: Store the data

Put it somewhere persistent.

Step 3: Add GA4

Connect search performance with website behavior.

Step 4: Calculate comparisons

Month over month is enough initially.

Step 5: Add anomaly detection

Flag large changes automatically.

Step 6: Create a report model

Separate data from presentation.

Step 7: Build one good template

Don't create five formats immediately.

Step 8: Schedule it

Make the pipeline run without manual intervention.

Step 9: Add AI carefully

Use it for interpretation, not fact generation.

This order matters.

If the underlying data isn't reliable, a beautifully designed AI-generated report doesn't solve anything.

It just produces unreliable information faster.

The Biggest Reporting Mistake

The easiest metrics to automate aren't necessarily the most valuable ones.

It's very easy to create a report containing:

Keywords: 4,281
Backlinks: 1,204
Domain Rating: 62
Pages Crawled: 8,291
Impressions: 248,000

It looks impressive.

But what did the business actually gain?

A better report might contain fewer numbers:

Organic traffic: +18%

Organic conversions: +14%

Revenue from organic traffic: +21%

Three commercial pages lost visibility.

Next priority:
refresh those pages and strengthen internal links.

That's a business conversation.

The purpose of SEO reporting isn't to prove that you've collected a lot of data.

It's to explain what happened and what should happen next.

A Useful Rule for Automated SEO Reporting

Here's the rule I'd keep on the wall:

Automate collection. Automate calculation. Automate repetition. Keep judgment human.

APIs are excellent at collecting information.

SQL is excellent at storing and aggregating it.

Python is excellent at transforming it.

Schedulers are excellent at repeating it.

Templates are excellent at formatting it.

But deciding what matters?

That's where human expertise still earns its keep.

From Spreadsheet Chore to Reporting System

The first time you automate an SEO report, the result might not look revolutionary.

Maybe you save 30 minutes.

That's fine.

Then you automate another client.

Then another.

Soon the same system is collecting data, calculating changes, generating charts, producing reports, and preparing summaries for dozens of properties.

That's when the economics change.

Instead of spending Monday morning copying metrics into spreadsheets, you can spend that time looking at the exceptions.

Which pages are growing?

Which pages are declining?

Where are conversions moving?

What changed?

What should we test next?

That's where SEO work becomes more interesting.

The goal isn't to eliminate reporting.

The goal is to eliminate the repetitive parts that don't require your attention.

Build the pipeline once.

Let the machine collect the numbers.

Then spend your time explaining what they mean.

I Spent 3 Weeks Debugging Rate Limits Before I Realized the Problem Wasn't My Code

Pallab Mondal — Thu, 23 Jul 2026 21:22:36 +0000

Ever chased a bug for days, only to discover the "bug" was actually the platform working exactly as designed? That happened to me building a client reporting pipeline. The lesson stuck.

Here's what nobody tells you about pulling marketing data from multiple ad platforms: the hard part was never the dashboard. It was everything underneath it.

The Setup That Looked Simple on Paper

The brief sounded easy. Pull spend, clicks, and conversions from Google Ads and Meta. Store it. Display it in a chart. A junior dev could knock this out in a sprint, I figured.

Reality disagreed. Google Ads API enforces operation quotas per developer token, and those quotas scale differently depending on account tier. Meanwhile, Meta's Marketing API throttles based on a rolling usage score tied to the ad account itself, not your app.

Two platforms. Two completely different throttling philosophies. Neither documented in a way that made the actual limits obvious until you hit them in production.

Where Things Actually Broke

My first version polled every client account every hour. Fine for three clients. Then we onboarded client number twelve, and Meta started returning 429s intermittently. Not consistently — intermittently. That's the worst kind of bug.

I initially assumed it was a code issue. Retry logic, maybe a race condition in my job scheduler. I spent three weeks going down that path. Eventually, I found the real cause: cumulative API call volume across all client accounts was tripping Meta's app-level rate limit, not the individual account limit.

The fix wasn't more retries. It was a request queue with exponential backoff, plus a priority system so active dashboards refreshed before idle ones. Simple in hindsight. Expensive in dev hours.

The Real Architecture Behind Multi-Platform Reporting

If you're building this yourself, here's what a production-grade pipeline actually needs, based on what broke for me.

A Queue, Not a Cron Job

Don't just fire off API calls on a schedule and hope for the best. Use a proper job queue — something like BullMQ or Sidekiq — with retry policies and backoff built in from day one. This alone would've saved me those three weeks.

Token Refresh as a First-Class Concern

OAuth tokens expire silently. Build monitoring specifically for auth failures, separate from your general error logging. Otherwise, you'll find out a client's Google Ads token expired only when they ask why their dashboard looks empty.

A Normalization Layer Between Raw Data and Storage

Google Ads calls it "cost." Meta calls it "spend." GA4 buries similar metrics under different dimension names entirely. Build one internal schema and map every platform into it, rather than letting frontend code handle platform-specific field names.

javascript // Simplified normalization example function normalizeMetric(platform, rawData) { const mapping = { google_ads: { cost: rawData.cost_micros / 1e6, clicks: rawData.clicks }, meta: { cost: rawData.spend, clicks: rawData.clicks }, }; return mapping[platform] || rawData; }

This function looks trivial. It isn't, once you're handling currency conversions, timezone mismatches, and attribution window differences across five platforms simultaneously.

Should You Actually Build This?

Here's the honest answer, from someone who's now maintained this kind of system for two years. If reporting infrastructure isn't your product's core differentiator, building it yourself is usually the wrong call.

I don't say that lightly. I like building things. However, every hour spent debugging Meta's rate limit quirks was an hour not spent on features that actually made our product better for users.

What Changed My Mind

Eventually, we moved most of our client reporting workflow onto a dedicated platform, RaiseReturn, instead of continuing to maintain our in-house pipeline. It already handled the multi-platform normalization, the token refresh monitoring, and the rate-limit-aware queuing I'd built by hand.

The API integrations existed already, tested against the platforms' quirks by a team that deals with exactly this problem full-time. That's not something you replicate cheaply in a side project or even a dedicated sprint.

Where Custom Code Still Makes Sense

To be clear, this isn't an argument against ever building integrations. If your reporting needs pull from truly proprietary internal systems — a custom CRM, an internal data warehouse — no off-the-shelf tool covers that. You'll build custom connectors regardless.

For standard ad platforms though, the problem is already solved. Well-tested, production-hardened, and maintained by people who get paged when Meta ships a breaking API change at 2 AM. You probably don't want to be that person.

Practical Takeaways for Your Own Stack

If you're currently building or maintaining a custom reporting pipeline, a few things worth checking right now.

First, verify your retry logic actually implements exponential backoff, not fixed-interval retries. Fixed intervals make rate limiting worse, not better, under sustained load.

Second, add dedicated alerting for OAuth token failures. Silent auth failures are the most common cause of "why is this client's data missing" tickets.

Third, honestly evaluate the maintenance cost against a dedicated automated reporting tool. Sometimes the math favors buying. Sometimes it doesn't. Either way, run the numbers before defaulting to "we'll just build it."

Final Thought

Client reporting infrastructure is a genuinely interesting engineering problem. It's also, for most teams, not the problem worth solving from scratch. Know the difference, and your future 2 AM self will thank you.

How We Built an Asynchronous Multi-API Pipeline for Marketing Automation

Pallab Mondal — Wed, 27 May 2026 20:33:23 +0000

When you are building a platform designed to consolidate data from Google Ads, GA4, Meta Ads, and Google Search Console, you quickly run into a massive bottleneck: API latency and rate limiting.

If an agency has 50 clients, and each client needs data pulled from 4 different ad networks, a chronological extraction loop will completely stall your application. Your server will time out, your database connection pools will exhaust, and your users will look at a spinning loading wheel forever.

While the core engine of our platform—RaiseReturn—remains a proprietary, private codebase, I wanted to share the exact architectural logic we used to solve the asynchronous fetching problem in Node.js.

The Bottleneck: Chronological vs. Parallel Processing

Most standard API wrappers encourage you to fetch data chronologically:

Wait for Google Ads to respond.
Take that data, then wait for Meta Ads to respond.
Wait for GA4 to respond.

This is a terrible approach for multi-tenant scalability. If one API experiences a 3-second network delay, your entire pipeline backs up. Instead, you need to execute these network requests concurrently, handle individual failures gracefully without crashing the whole sequence, and normalize the disparate payloads into a unified schema before writing to your database.

A Sanitized Example: The Asynchronous Batch Worker

Below is a lightweight, clean abstraction of how you can structure a data worker to fetch metrics from entirely different API endpoints concurrently using Promise.allSettled.

Using allSettled ensures that if the Meta Ads API randomly throws an authentication error or a 500 timeout, your script still successfully captures and processes the Google Ads and GA4 data for that client.

// A lightweight abstraction of a concurrent data extraction worker
const fetchClientMetrics = async (clientConfig) => {
  const providers = [
    { name: 'googleAds', fetchFn: callGoogleAdsAPI },
    { name: 'metaAds', fetchFn: callMetaAdsAPI },
    { name: 'ga4', fetchFn: callGA4API }
  ];

  // Execute all network requests concurrently
  const fetchPromises = providers.map(provider => 
    provider.fetchFn(clientConfig[provider.name])
      .then(data => ({ provider: provider.name, success: true, data }))
      .catch(error => ({ provider: provider.name, success: false, error: error.message }))
  );

  const results = await Promise.allSettled(fetchPromises);
  
  const payload = {
    clientId: clientConfig.id,
    timestamp: new Date(),
    metrics: {}
  };

  results.forEach((result) => {
    if (result.status === 'fulfilled') {
      const { provider, success, data, error } = result.value;
      if (success) {
        // Normalize your data pipeline here
        payload.metrics[provider] = data;
      } else {
        console.error(`Error pulling from ${provider}:`, error);
        payload.metrics[provider] = { error: 'Extraction failed' };
      }
    }
  });

  return payload;
};

Scaling Up to Production

In a real production environment, you cannot simply let hundreds of these tasks fire randomly inside an Express controller. Furthermore, you have to back this up with a robust message broker or worker layer:

Job Queuing: Push data sync tasks to a queue backend (like AWS SQS or BullMQ).
State Management: Track extraction states dynamically in a database like MongoDB or PostgreSQL to handle retries for failed API responses.
Payload Sanitization: Run the incoming arrays through a dedicated transformer layer to map differing currency codes, timezone variations, and metric naming conventions into a unified dashboard layout.

Building the Future of Reporting

Building out reliable, fault-tolerant infrastructure that constantly listens to fluctuating external APIs takes significant time and architectural planning. Therefore, that is the exact reason we built RaiseReturn—to give agencies enterprise-level reporting infrastructure right out of the box without forcing them to spend months writing custom ETL pipelines.

If you are currently building data pipelines or wrestling with marketing API schemas, let's talk architecture in the comments below!