A Data Pipeline Gets Expensive When Every Source Becomes Its Own Little System

A multi-source data project can look surprisingly simple at the beginning.

Connect one publisher. Add another. Write some source-specific logic. Keep going.

Then the system reaches ten or more sources, and the work starts changing. You are no longer maintaining a scraper. You are operating a collection of external dependencies that all happen to feed the same product.

That is where the cost starts growing.

Not because source number 13 is magically more expensive than source number 3, but because every source brings its own structure, failure patterns, access constraints, and maintenance work.

Every source behaves differently

One publisher might expose a clean WordPress REST API.
Another might place useful data inside server-rendered JSON state.
Another may expose schema.org metadata.
Another may require source-specific DOM parsing.

Some may also need different access handling because direct requests do not behave consistently.

The extraction layer has to understand those differences, but the rest of the product should not have to.

That is where a shared pipeline starts becoming much more useful than a collection of unrelated scripts.

Keep source-specific logic at the edge

There will always be logic that belongs to one publisher.

Trying to force every source through one generic scraper usually creates another kind of maintenance problem.

A cleaner structure looks more like this:

Source A adapter ─┐
Source B adapter ─┤
Source C adapter ─┤
Source D adapter ─┤
        ...        ├──> Shared normalization
Source M adapter ─┘

Each adapter deals with what is unique to that source.

The shared pipeline deals with everything that should remain consistent.

That separation matters because one publisher changing its markup should not force changes throughout the rest of the application.

Normalization becomes part of the product

Successful extraction does not automatically mean usable data.

Different publishers can represent the same information in very different ways.

Dates vary. Author fields vary. Categories and tags vary. Images vary. Canonical URLs vary. Article bodies vary.

If every downstream consumer has to understand those differences again, the pipeline has only moved the problem somewhere else.

A shared article model gives the rest of the product one structure to work with.

For example:

type NormalizedArticle = {
  headline: string;
  summary?: string;
  url: string;
  canonicalUrl?: string;
  publishedAt?: Date;
  authors: string[];
  categories: string[];
  tags: string[];
  body: string;
  sourceId: string;
};

The exact schema will differ from product to product, but the principle stays the same:

Source differences belong at ingestion boundaries, not throughout the entire application.

Once the information is normalized, another product should not need to know whether the original article came from an API, structured metadata, JSON state, or DOM parsing.

Then comes the operational work

This is where multi-source systems can become expensive.

A publisher changes its markup.

A scheduled run fails.

One source starts returning partial records.

Another responds successfully but produces unusable content.

If the system only tells you that the collection process started, that is not enough visibility.

You usually need to know things like:

• Which sources are healthy?
• When did each source last complete successfully?
• Which run failed?
• Did the system retry it?
• Is the scheduler still active?
• Has runtime configuration changed?
• Is one source producing unusual results?

Without that layer, teams often discover ingestion failures somewhere downstream.

A report looks incomplete. A product has missing articles. Someone opens the database, then the logs, then the scheduler, and eventually works backwards until the broken source is found.

That is expensive debugging for something the platform could have surfaced much earlier.

Source health changes the economics

Monitoring can look like extra engineering when a pipeline only has two or three sources.

At thirteen sources, the situation is different.

Manually checking every publisher regularly already becomes tedious. Finding failures only after a customer or downstream product notices missing data is worse.

A useful operator view can surface:

• source health
• recent runs
• failures
• retry behaviour
• scheduler state
• runtime settings

That changes the recovery path considerably.

Instead of:

Missing data appears downstream
        ↓
Check database
        ↓
Check scheduler
        ↓
Search logs
        ↓
Test individual sources
        ↓
Find the broken publisher

the workflow can become:

Source health shows a problem
        ↓
Open the failed run
        ↓
Inspect source-specific issue
        ↓
Fix or retry

The collection logic did not become simpler.

The operational path became clearer.

That difference saves more time as the number of integrations grows.

Scheduling needs ownership too

A multi-source pipeline is rarely just a set of functions that somebody runs manually.

Some sources need scheduled collection. Operators may also need to trigger runs manually, pause a source, inspect recent activity, or change runtime behaviour.

If every publisher ends up with its own scheduler configuration and its own assumptions, another maintenance layer appears.

Shared scheduler controls give the team one place to understand what should be running and when.

That sounds small until a source silently stops running and nobody is sure whether the failure came from extraction logic, scheduling, deployment, or configuration.

Collection is only half of the system

Once the data is collected and normalized, another product usually needs it.

An internal application might consume it.

A reporting workflow might consume it.

Another SaaS product might consume it.

Giving every consumer direct database access creates unnecessary coupling.

A controlled read-only API is easier to reason about.

That API can include:

• API keys
• usage limits
• request logging
• revocation
• health endpoints
• OpenAPI documentation

Now the responsibilities become clearer.

External publishers
        ↓
Source-specific ingestion
        ↓
Shared normalization
        ↓
Storage + run auditing
        ↓
Authenticated API
        ↓
Downstream products

The ingestion system owns collection and data quality.

The API owns delivery.

Downstream products consume a stable interface instead of depending directly on storage internals.

Think in terms of one platform with many adapters

There is a big operational difference between these two setups.

Setup 1

13 sources
13 scripts
13 schedules
13 sets of debugging assumptions
13 separate maintenance paths

Setup 2

13 source adapters
        ↓
1 normalization layer
        ↓
1 run model
        ↓
1 source-health view
        ↓
1 scheduler control surface
        ↓
1 authenticated API

Both setups may collect the same articles.

Only one is deliberately designed to stay manageable as the source count grows.

A system where we applied this structure

We recently built a real estate news data platform around 13 publisher integrations.

The sources used different collection patterns including WordPress REST, server-rendered state, schema.org metadata, source-specific DOM parsing, and other source-dependent approaches.

The platform brought those sources into one shared operating model with:

• source-specific ingestion
• shared normalization
• MongoDB persistence
• run auditing
• source-health visibility
• scheduler controls
• authenticated API access
• quota and usage tracking
• Docker deployment support

Operators could see source health, recent runs, failures, scheduler configuration, and runtime settings without needing to work directly with server code.

Downstream products could consume normalized records through a controlled API instead of depending directly on the database.

The public project breakdown is here:

Related work:

Real Estate News Data Pipeline & API Dashboard

One pipeline is easier to operate than thirteen little systems

Adding another source is usually easy to estimate if you only count the extraction work.

The longer-term cost sits in everything around it.

Keeping sources healthy. Keeping records consistent. Recovering from failed runs. Managing schedules. Giving downstream products stable access. Understanding what broke without spending an hour tracing the whole system backwards.

Once those things start mattering, the scraper is only one component.

The pipeline around it is what makes the system maintainable.