DEV Community

Samcorp
Samcorp

Posted on

Building a Reusable ERP Integration Layer

#erp #crm

Building a Reusable ERP Integration Layer
The first ERP integration is usually easy.

You have one ERP and one external system:

CRM ──────────→ ERP
Enter fullscreen mode Exit fullscreen mode

So you write an API client, map a few fields, add authentication, handle some errors, and ship it.

Then another integration arrives:

E-commerce ───→ ERP
Enter fullscreen mode Exit fullscreen mode

Then:

Warehouse ────→ ERP
Enter fullscreen mode Exit fullscreen mode

Then Finance needs data exported. A supplier portal needs purchase orders. The ERP gets upgraded.

Eventually the architecture looks like this:

CRM ────────────────→ ERP
E-commerce ─────────→ ERP
Warehouse ──────────→ ERP
Finance ────────────→ ERP
Supplier Portal ────→ ERP
Support ─────────────→ ERP
Enter fullscreen mode Exit fullscreen mode

Every integration now has its own authentication, field mappings, retry behavior, logging, and assumptions about ERP data.

That's when we stopped thinking about individual connections and started thinking about a reusable ERP integration layer.

The goal wasn't another abstraction for the sake of architecture.

It was to create one place where ERP-specific complexity could live.

The Problem With Point-to-Point Integrations

Imagine an ERP exposes customers like this:

{
  "customer_no": "C-10492",
  "cust_name": "Acme Manufacturing",
  "payment_code": "N30",
  "currency_code": "USD"
}
Enter fullscreen mode Exit fullscreen mode

Your CRM wants:

{
  "externalId": "C-10492",
  "accountName": "Acme Manufacturing",
  "paymentTerms": "NET_30",
  "currency": "USD"
}
Enter fullscreen mode Exit fullscreen mode

Writing that transformation once isn't difficult.

The problem starts when five applications independently implement it:

CRM ──────────────┐
E-commerce ───────┤
Warehouse ────────┼──→ ERP
Finance ──────────┤
Support ──────────┘
Enter fullscreen mode Exit fullscreen mode

Now every application understands ERP internals.

If:

N30

changes to:

NET30

you may need to modify several integrations.

That's coupling.

And ERP environments generate a lot of it.

Put the ERP Behind a Stable Contract

Instead of allowing every system to communicate directly with ERP-specific APIs, we introduced a boundary:

CRM ───────────────┐
E-commerce ────────┤
Warehouse ─────────┼──→ Integration Layer ───→ ERP
Finance ───────────┤
Supplier Portal ───┘
Enter fullscreen mode Exit fullscreen mode

The ERP integration layer became responsible for authentication, transformation, validation, retries, idempotency, logging, observability, error classification, rate limiting, and ERP-specific API behavior.

Applications no longer needed to understand everything about the ERP.

They needed to understand the integration contract.

This kind of architecture is also why broader ERP system integration and API integration planning matters. Once an ERP needs to exchange data with CRM, finance, HR, supply-chain, e-commerce, legacy, and third-party applications, integration architecture becomes a platform concern rather than a collection of isolated API calls.

That distinction became increasingly valuable as the number of connected systems grew.

Start With Canonical Business Objects

One mistake is exposing the ERP schema directly.

Suppose the ERP represents a customer as:

{
  "CUST_NO": "100842",
  "CUST_NM": "ACME INDUSTRIAL",
  "PYMT_TRM": "03",
  "CURR": "USD"
}
Enter fullscreen mode Exit fullscreen mode

We didn't want every consuming system to understand those conventions.

Instead, the integration layer exposed something closer to:

{
  "id": "100842",
  "name": "Acme Industrial",
  "paymentTerms": "NET_30",
  "currency": "USD"
}
Enter fullscreen mode Exit fullscreen mode

The canonical model represented the business concept rather than one application's database schema.

Conceptually:

ERP Customer
     ↓
Canonical Customer
     ↓
CRM Customer
Enter fullscreen mode Exit fullscreen mode

That extra transformation looks unnecessary when you have one integration.

It becomes useful when you have ten.

Don't Build One Giant Universal Model

Canonical models can also become a trap.

We initially wanted something like:

UniversalCustomer

that represented every possible customer field across every application.

Eventually it becomes:

UniversalCustomer
├── 190 fields
├── ERP-specific fields
├── CRM-specific fields
├── E-commerce fields
├── Legacy fields
└── Fields nobody understands anymore

That's not abstraction.

That's schema accumulation.

We had better results with smaller contracts aligned to business capabilities:

  • CustomerIdentity
  • CustomerCreditProfile
  • CustomerAddress
  • CustomerPricing
  • CustomerTaxProfile

Not every consumer needs every attribute.

Separate Transport From Business Mapping

A common integration implementation mixes everything together:

HTTP Request
   ↓
Authentication
   ↓
Parse JSON
   ↓
Business Mapping
   ↓
Validation
   ↓
ERP Request
   ↓
Retry
Enter fullscreen mode Exit fullscreen mode

inside one large service.

We separated those responsibilities.

Transport
    ↓
Contract Validation
    ↓
Business Transformation
    ↓
ERP Adapter
    ↓
ERP
Enter fullscreen mode Exit fullscreen mode

The transport layer answers:

  • How did the message arrive?
  • The transformation layer answers:
  • What does this data mean?
  • The ERP adapter answers:
  • How does this particular ERP expect to receive it?

That separation made testing much easier.

Put ERP-Specific Behavior Behind an Adapter

Suppose one ERP version requires:

POST /customers/create

while another requires:

POST /api/v2/business-partners

Consumers shouldn't care.

Our interface might conceptually look like:

interface CustomerRepository {
  create(customer: Customer): Promise<CustomerResult>;
  update(customer: Customer): Promise<CustomerResult>;
  findByExternalId(id: string): Promise<Customer | null>;
}
Enter fullscreen mode Exit fullscreen mode

The ERP adapter implements that contract.

Customer Service
       ↓
CustomerRepository
       ↓
ERP Adapter
       ↓
ERP API
Enter fullscreen mode Exit fullscreen mode

Now ERP-specific implementation details stay at the edge.

  • Idempotency Became Mandatory
  • Distributed systems retry.
  • Networks fail.
  • Workers crash.
  • Clients time out.

Suppose:

Create Purchase Order
        ↓
ERP creates PO
        ↓
Network timeout
        ↓
Caller assumes failure
        ↓
Retry
Enter fullscreen mode Exit fullscreen mode

Without idempotency:

PO-10001
PO-10002

may represent the same business request.

That's a serious problem.

We started requiring an idempotency key for operations that could create duplicate business transactions.

Idempotency-Key: order-78432-v1

The integration layer stores something conceptually like:

order-78432-v1
      ↓
ERP PO 10001
Enter fullscreen mode Exit fullscreen mode

If the same request arrives again:

Same key?
   ↓
YES
   ↓
Return previous result
Enter fullscreen mode Exit fullscreen mode

Retry safety isn't optional when you're creating financial or operational records.

Retry Only Errors That Might Recover

Our earliest retry logic was effectively:

Request failed?
↓
Retry

That's dangerous.

Consider:

HTTP 400
Invalid tax code

Retrying it ten times changes nothing.

But:

HTTP 503
ERP temporarily unavailable

may recover.

We classified errors:

TRANSIENT
├── Timeout
├── Connection failure
├── 429
└── 503
Enter fullscreen mode Exit fullscreen mode 
PERMANENT
├── Invalid customer
├── Invalid currency
├── Missing required field
└── Unsupported tax code
Enter fullscreen mode Exit fullscreen mode

Then:

Transient error
      ↓
Retry with backoff
Enter fullscreen mode Exit fullscreen mode

while:

Permanent error
      ↓
Exception queue
Enter fullscreen mode Exit fullscreen mode

This reduced both ERP load and pointless retry storms.

Exponential Backoff Needed Jitter

If 500 workers lose access to ERP simultaneously and all retry exactly five seconds later, you haven't recovered.

You've created another traffic spike.

Instead of:

5s
10s
20s
40s

we added jitter so retries were distributed across time.

Conceptually:

delay = backoff + random_jitter

The goal is to prevent synchronized clients from hammering an ERP that's trying to recover.

Validate Before the ERP Call

We originally relied too much on ERP validation.

Request:

{
  "customerId": "",
  "currency": "ABC",
  "quantity": -4
}
Enter fullscreen mode Exit fullscreen mode

Send it to ERP.

ERP rejects it.

That wastes an ERP request and often produces a worse error message.

Instead:

Incoming Request
      ↓
Schema Validation
      ↓
Business Validation
      ↓
Reference Validation
      ↓
ERP
Enter fullscreen mode Exit fullscreen mode

Examples include:

  • customerId required
  • quantity > 0
  • currency ∈ supported currencies
  • warehouse must exist
  • product must be active

The integration layer should reject obviously invalid data before expensive downstream work begins.

Don't Duplicate the Entire ERP Rule Engine

There's an opposite failure mode.

If the ERP has hundreds of validation rules, reproducing all of them in the integration layer creates two business-rule engines.

Eventually:

Integration says VALID
ERP says INVALID

or worse:

Integration says INVALID
ERP would accept it

We separated responsibilities:

Contract validation
→ Integration layer

Cross-system invariants
→ Integration layer

ERP-internal rules
→ ERP

The integration layer should protect the boundary.

It shouldn't become a second ERP.

Synchronous APIs Weren't Enough

Some operations need immediate answers:

  • Get Customer
  • Get Product Availability
  • Validate Address

Synchronous APIs are reasonable.

Other operations don't need the caller waiting:

  • Export 50,000 products
  • Import invoice batch
  • Synchronize inventory
  • Publish shipment updates

For those, we preferred:

Producer
   ↓
Queue / Event Bus
   ↓
Integration Worker
   ↓
ERP
Enter fullscreen mode Exit fullscreen mode

Now the producer doesn't remain coupled to ERP response time.

Queues Changed Failure Handling

Without a queue:

Application
    ↓
ERP unavailable
    ↓
Application fails

With durable messaging:

Application
    ↓
Queue
    ↓
ERP unavailable
    ↓
Retry later
Enter fullscreen mode Exit fullscreen mode

That doesn't magically solve every problem.

But it changes the failure boundary.

The upstream application may continue operating while ERP connectivity recovers, assuming the business workflow allows eventual consistency.

Eventual Consistency Needed to Be Explicit

Suppose inventory changes:

ERP
 ↓
Integration Layer
 ↓
E-commerce
Enter fullscreen mode Exit fullscreen mode

There may be a delay.

So we needed to define acceptable consistency windows.

For example:

Customer update
< 60 seconds

Product description
< 5 minutes

Inventory availability
< 10 seconds

Those aren't universal targets.

The point is to define the expectation.

"Near real time" isn't a useful architecture requirement.

A measurable latency target is.

Ordering Became Important

Suppose these events occur:

  1. Customer Created
  2. Customer Updated
  3. Customer Disabled

If distributed workers process them as:

3 → 1 → 2

the final state may be wrong.

For entities where order matters, we needed mechanisms such as partitioning by entity key, sequence numbers, version numbers, or stale-update rejection.

For example:

{
  "customerId": "C-1004",
  "version": 17
}
Enter fullscreen mode Exit fullscreen mode

If the target has already processed version 18:

version 17
     ↓
STALE
     ↓
Ignore / investigate
Enter fullscreen mode Exit fullscreen mode

Event ordering should be designed rather than assumed.

We Needed a Dead-Letter Strategy

Eventually, some messages will fail repeatedly.

Maybe:

  • Product doesn't exist
  • Customer mapping is missing
  • Tax code was retired
  • Payload is malformed

Retrying forever isn't reliability.

It's an infinite loop.

So after defined attempts:

Message
   ↓
Retry
   ↓
Retry
   ↓
Retry
   ↓
Dead-Letter Queue
Enter fullscreen mode Exit fullscreen mode

The important part wasn't creating the DLQ.

It was creating the operational process around it.

Every failed message needed enough information to answer:

  • What failed?
  • Why?
  • Which source produced it?
  • Which ERP operation failed?
  • Can we retry it safely?
  • Who owns resolution?

A dead-letter queue nobody monitors is just a database of forgotten failures.

Correlation IDs Saved Debugging Time

Imagine a sales order moves through:

E-commerce
    ↓
API Gateway
    ↓
Integration API
    ↓
Queue
    ↓
Worker
    ↓
ERP Adapter
    ↓
ERP
Enter fullscreen mode Exit fullscreen mode

Then someone reports:

Order 48392 didn't reach ERP.

Without correlation, you search several systems manually.

Instead, generate or propagate:

correlation_id = 7f91c...
Enter fullscreen mode Exit fullscreen mode

through the transaction.

Logs can now show:

[7f91c] Request accepted
[7f91c] Validation passed
[7f91c] Message published
[7f91c] Worker started
[7f91c] ERP request sent
[7f91c] ERP timeout
[7f91c] Retry scheduled
Enter fullscreen mode Exit fullscreen mode

That one design choice saved a lot of debugging time.

Logging Payloads Created a Security Problem

Detailed integration logs are useful.

But ERP payloads can contain customer information, employee information, addresses, financial data, banking details, pricing, and tax identifiers.

So:

console.log(JSON.stringify(request));
Enter fullscreen mode Exit fullscreen mode

isn't an observability strategy.

We moved toward structured operational metadata:

{
  "correlationId": "7f91c",
  "operation": "CreateCustomer",
  "externalId": "C-10482",
  "status": "FAILED",
  "errorType": "ERP_TIMEOUT"
}
Enter fullscreen mode Exit fullscreen mode

Sensitive fields should be masked or omitted according to the organization's security and compliance requirements.

Metrics Were More Useful Than More Logs

Logs tell you what happened to one transaction.

Metrics tell you whether the integration platform is healthy.

We tracked things such as:

  • Requests / minute
  • Success rate
  • Failure rate
  • Retry rate
  • ERP latency
  • Queue depth
  • Oldest queued message
  • Dead-letter count
  • Processing duration One metric became particularly valuable:

Oldest unprocessed message age

A queue depth of 10,000 might be expected during a large batch.

But:

Oldest message = 4 hours

is immediately concerning when the expected processing window is five minutes.

Rate Limiting Protected the ERP

External applications can scale faster than ERP systems.

Suppose:

E-commerce
10,000 requests/minute

while ERP safely handles:

500 requests/minute

Without protection:

High traffic
    ↓
ERP overload
    ↓
Timeouts
    ↓
Retries
    ↓
More traffic
    ↓
ERP outage
Enter fullscreen mode Exit fullscreen mode

The integration layer became the shock absorber:

Incoming traffic
      ↓
Rate Limit
      ↓
Queue
      ↓
Controlled concurrency
      ↓
ERP
Enter fullscreen mode Exit fullscreen mode

Sometimes slowing down work is the most reliable way to complete it.

Circuit Breakers Helped During ERP Outages

If ERP is clearly unavailable, repeatedly calling it wastes resources.

A circuit breaker gives us:

CLOSED
Requests flow normally

After enough failures:

OPEN
Fail fast / queue work

Later:

HALF OPEN
Test recovery

Then:

Healthy → CLOSED
Still failing → OPEN

This prevented a failing dependency from consuming every worker in the integration platform.

API Design Became Part of the Integration Architecture

Once several applications depended on the integration layer, its API stopped being an internal implementation detail.

It became a product boundary.

We needed consistent decisions around authentication, endpoint conventions, pagination, rate limits, error responses, compatibility, monitoring, and versioning.

This is where practices from custom API development and integration architecture become directly relevant. A reusable integration layer benefits from treating REST, SOAP, GraphQL or other interfaces as managed contracts, with rate limiting, versioning, testing, monitoring, and resilient error handling designed into the API lifecycle rather than added after production problems appear.

The implementation behind an endpoint could change.

The contract consumed by other systems needed much more discipline.

Version the Contracts, Not Every Internal Detail

Suppose:

{
  "customer": "C-100"
}
Enter fullscreen mode Exit fullscreen mode

becomes:

{
  "customerId": "C-100"
}
Enter fullscreen mode Exit fullscreen mode

That tiny rename can break every consumer.

For breaking changes, we needed something like:

/v1/orders
/v2/orders

or an equivalent schema-versioning mechanism.

Internally, we could refactor aggressively.

Externally, compatibility mattered.

Contract Tests Became Essential

Unit tests weren't enough.

The dangerous question was:

Does the adapter still satisfy the contract expected by the rest of the platform?

We created tests around canonical requests and expected ERP transformations.

Given
Canonical Customer

When
ERP Adapter transforms it

Then
Expected ERP payload is produced

And the reverse:

Given
ERP Customer response

When
Adapter maps it

Then
Expected canonical customer is returned

This became particularly useful during ERP upgrades.

ERP Upgrades Became Much Less Scary

Before the integration layer:

ERP Upgrade
CRM breaks
E-commerce breaks
Warehouse breaks
Finance breaks
Enter fullscreen mode Exit fullscreen mode

After introducing adapters:

ERP Upgrade
      ↓
ERP Adapter changes
      ↓
Canonical contracts remain stable
      ↓
Consumers mostly unchanged
Enter fullscreen mode Exit fullscreen mode

That's one of the strongest arguments for the architecture.

The ERP integration layer acts as a boundary between ERP-specific behavior and the rest of the application ecosystem.

Not every upgrade becomes free.

But the blast radius can become much smaller.

Batch APIs Needed Different Design

Creating one customer:

POST /customers

is easy.

Synchronizing 500,000 products isn't the same problem repeated 500,000 times.

For large datasets we considered:

  • Pagination
  • Batch size
  • Checkpointing
  • Parallelism
  • Rate limits
  • Partial failure
  • Resume behavior

A batch job needed to be restartable.

Instead of:

500,000 records
     ↓
Failure at 487,293
     ↓
Start again

we wanted:

Checkpoint: 487,000
     ↓
Resume
Enter fullscreen mode Exit fullscreen mode

Long-running ERP integrations need recovery semantics, not just loops.

Partial Success Needed an Explicit Model

Suppose a batch contains 1,000 invoices:

992 succeeded
8 failed

What does the API return?

Simply returning:

500 Internal Server Error

loses important information.

A better result might contain:

{
  "processed": 1000,
  "succeeded": 992,
  "failed": 8,
  "errors": [
    {
      "externalId": "INV-8472",
      "code": "INVALID_CUSTOMER"
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

Now the caller can distinguish complete failure from partial completion.

Reconciliation Was Still Necessary

Reliable messaging does not eliminate reconciliation.

Imagine:

Integration says:
10,000 orders sent

ERP says:
9,997 orders created

Where are the other three?

We built reconciliation around business identifiers:

source_order_id
erp_order_id
integration_status
erp_status
last_verified_at
Enter fullscreen mode Exit fullscreen mode

Then periodically checked:

Source
   ↕
Integration State
   ↕
ERP
Enter fullscreen mode Exit fullscreen mode

Distributed systems need a way to prove that both sides agree.

Secrets Didn't Belong in Configuration Files

An integration layer naturally accumulates credentials:

  • ERP credentials
  • API keys
  • OAuth secrets
  • Certificates
  • Database passwords

Those shouldn't live in source control or plain configuration files.

We separated:

Configuration

from:

Secrets

and used an appropriate secret-management mechanism.

That also made credential rotation easier.

A reusable integration platform should make the secure path the easy path.

Reusable Doesn't Mean Monolithic

Centralizing integration logic creates another risk:

Everything
    ↓
One enormous integration service
Enter fullscreen mode Exit fullscreen mode

Now every deployment affects every integration.

Instead, we preferred shared platform capabilities with independently deployable integration components:

             ┌─────────────────────┐
             │ Shared Capabilities │
             │                     │
             │ Auth                │
             │ Logging             │
             │ Retry               │
             │ Observability       │
             │ Contracts           │
             └─────────┬───────────┘
                       │
       ┌───────────────┼───────────────┐
       ▼               ▼               ▼
 Customer Adapter  Order Adapter  Inventory Adapter
       │               │               │
       └───────────────┼───────────────┘
                       ▼
                      ERP
Enter fullscreen mode Exit fullscreen mode

Reusable doesn't have to mean monolithic.

What We Standardized

Eventually, every integration started with the same baseline:

  • Authentication
  • Correlation IDs
  • Structured logging
  • Metrics
  • Tracing
  • Retry policy
  • Timeout policy
  • Idempotency
  • Error taxonomy
  • Dead-letter handling
  • Contract validation
  • Secret management
  • Health checks
  • Rate limiting

Developers shouldn't need to reinvent these for every ERP endpoint.

The reusable layer provided them by default.

That was where the architecture started paying dividends.

What We'd Build First Today

If starting again, we wouldn't begin by designing a massive enterprise integration framework.

We'd begin with one real integration and extract reusable patterns.

Integration #1
     ↓
Identify repeated concerns
     ↓
Extract shared capabilities
     ↓
Integration #2
     ↓
Validate abstractions
     ↓
Improve platform
     ↓
Integration #3
Enter fullscreen mode Exit fullscreen mode

That's safer than designing abstractions based on integrations that don't exist yet.

A reusable architecture should emerge from repeated requirements.

Not imagination.

A Practical ERP Integration Layer Checklist

Before calling an integration platform reusable, we'd verify:

  • ERP-specific APIs are isolated behind adapters.
  • Business contracts don't unnecessarily expose ERP internals.
  • Authentication is standardized.
  • Secrets are managed outside source code.
  • Requests have correlation IDs.
  • Logging is structured and sensitive data is protected.
  • Transient and permanent errors are distinguished.
  • Retry policies use bounded backoff and jitter.
  • Create operations support idempotency where required.
  • Asynchronous workflows have dead-letter handling.
  • Queue depth and message age are monitored.
  • Rate limiting protects ERP capacity.
  • Contracts have a compatibility/versioning strategy.
  • Contract tests cover adapters.
  • Batch processing can resume after failure.
  • Partial-success behavior is explicit.
  • Business records can be reconciled across systems.
  • ERP upgrades can primarily be absorbed at the adapter boundary.

The Biggest Lesson

At first, we thought an ERP integration layer was mostly about reusable API code.

It wasn't.

The valuable part was creating a boundary.

Without one:

Every Application
      ↓
Understands ERP
      ↓
Depends on ERP
      ↓
Breaks when ERP changes
Enter fullscreen mode Exit fullscreen mode

With one:


Applications
      ↓
Stable Business Contracts
      ↓
ERP Integration Layer
      ↓
ERP-Specific Complexity
      ↓
ERP
Enter fullscreen mode Exit fullscreen mode

That boundary gave us somewhere to centralize the difficult parts of integration: retries, identity mapping, validation, rate limits, idempotency, observability, error handling, security, and version compatibility.

The goal isn't to hide the ERP completely.

And it isn't to create a giant middleware platform before the first integration ships.

The goal is simpler:

Keep ERP-specific complexity from leaking into every application that needs ERP data.

  • That's what made the layer reusable.
  • Not the number of shared classes.
  • Not the number of endpoints.
  • Not the middleware product.

The real measure was this:

When the ERP changed, how many other systems had to change with it?

The smaller that answer became, the better our integration architecture was.

Top comments (0)