How We Built an Asynchronous Multi-API Pipeline for Marketing Automation

`

When you are building a platform designed to consolidate data from Google Ads, GA4, Meta Ads, and Google Search Console, you quickly run into a massive bottleneck: API latency and rate limiting.

If an agency has 50 clients, and each client needs data pulled from 4 different ad networks, a chronological extraction loop will completely stall your application. Your server will time out, your database connection pools will exhaust, and your users will look at a spinning loading wheel forever.

While the core engine of our platform—RaiseReturn—remains a proprietary, private codebase, I wanted to share the exact architectural logic we used to solve the asynchronous fetching problem in Node.js.

The Bottleneck: Chronological vs. Parallel Processing

Most standard API wrappers encourage you to fetch data chronologically:

• Wait for Google Ads to respond.
• Take that data, then wait for Meta Ads to respond.
• Wait for GA4 to respond.

This is a terrible approach for multi-tenant scalability. If one API experiences a 3-second network delay, your entire pipeline backs up. Instead, you need to execute these network requests concurrently, handle individual failures gracefully without crashing the whole sequence, and normalize the disparate payloads into a unified schema before writing to your database.

A Sanitized Example: The Asynchronous Batch Worker

Below is a lightweight, clean abstraction of how you can structure a data worker to fetch metrics from entirely different API endpoints concurrently using Promise.allSettled.

Using allSettled ensures that if the Meta Ads API randomly throws an authentication error or a 500 timeout, your script still successfully captures and processes the Google Ads and GA4 data for that client.

// A lightweight abstraction of a concurrent data extraction worker
const fetchClientMetrics = async (clientConfig) => {
  const providers = [
    { name: 'googleAds', fetchFn: callGoogleAdsAPI },
    { name: 'metaAds', fetchFn: callMetaAdsAPI },
    { name: 'ga4', fetchFn: callGA4API }
  ];

  // Execute all network requests concurrently
  const fetchPromises = providers.map(provider => 
    provider.fetchFn(clientConfig[provider.name])
      .then(data => ({ provider: provider.name, success: true, data }))
      .catch(error => ({ provider: provider.name, success: false, error: error.message }))
  );

  const results = await Promise.allSettled(fetchPromises);
  
  const payload = {
    clientId: clientConfig.id,
    timestamp: new Date(),
    metrics: {}
  };

  results.forEach((result) => {
    if (result.status === 'fulfilled') {
      const { provider, success, data, error } = result.value;
      if (success) {
        // Normalize your data pipeline here
        payload.metrics[provider] = data;
      } else {
        console.error(`Error pulling from ${provider}:`, error);
        payload.metrics[provider] = { error: 'Extraction failed' };
      }
    }
  });

  return payload;
};

Scaling Up to Production

In a real production environment, you cannot simply let hundreds of these tasks fire randomly inside an Express controller. Furthermore, you have to back this up with a robust message broker or worker layer:

• Job Queuing: Push data sync tasks to a queue backend (like AWS SQS or BullMQ).
• State Management: Track extraction states dynamically in a database like MongoDB or PostgreSQL to handle retries for failed API responses.
• Payload Sanitization: Run the incoming arrays through a dedicated transformer layer to map differing currency codes, timezone variations, and metric naming conventions into a unified dashboard layout.

Building the Future of Reporting

Building out reliable, fault-tolerant infrastructure that constantly listens to fluctuating external APIs takes significant time and architectural planning. Therefore, that is the exact reason we built RaiseReturn—to give agencies enterprise-level reporting infrastructure right out of the box without forcing them to spend months writing custom ETL pipelines.

If you are currently building data pipelines or wrestling with marketing API schemas, let's talk architecture in the comments below!

`