Why ERP integrations silently fail in production (and how I fixed it in Go)

Steffi — Mon, 27 Apr 2026 14:00:00 +0000

Most integration systems don’t break immediately. They fail silently over time by corrupting your data.

I learned this the hard way while building ERP integrations between retailers and suppliers. That retailer exchanged data with its suppliers: inventory updates, orders, shipping notices, and invoices. Each message came from different ERP systems with different formats and validation rules.

Now, as I’m preparing for a job interview in the field of ERP integration, I decided to approach this properly.

Integration Flow Design

No matter how different the systems were, every integration system I have ever seen has the similar pattern:

Reception of HTTP Request: The retailer receives an order via HTTPS or SFTP.
Decoding data: The payload is decoded and validated for syntactic correctness.
Validation: The data is validated from a business perspective.
Mapping: The external data is mapped to the internal model.
Response: A response is returned with a status of 201 Created.

The design mistake I made

The biggest mistake I made at the beginning was mixing external formats with internal business logic of the platform. This is the point where most integration systems start to become unmaintainable.

The transport model defines the structure of the incoming payload as defined by the supplier, ERP system, or external API. The external payload can change at any time.

The internal data model belongs to the retailer’s platform, not to the supplier. It should remain as stable as possible. The internal data model is optimized for business logic.

Therefore, I decided to separate this data.

How I fixed it in Go

We declare a struct, which represents the incoming external payload.

type IncomingOrderRequest struct {
 MessageID  string        `json:"message_id"`
 SupplierID string        `json:"supplier_id"`
 Order      IncomingOrder `json:"order"`
}

The second struct represents the order itself:

type IncomingOrder struct {
 OrderID     string              `json:"order_id"`
 OrderDate   string              `json:"order_date"`
 Currency    string              `json:"currency"`
 TotalAmount float64             `json:"total_amount"`
 Lines       []IncomingOrderLine `json:"lines"`
}

The third struct represents the order lines:

type IncomingOrderLine struct {
 SKU       string  `json:"sku"`
 Qty       int     `json:"qty"`
 UnitPrice float64 `json:"unit_price"`
}

These structs together are related to the external transport model. The next two structs are related to the internal data model:

type InternalOrder struct {
 MessageID   string
 OrderID     string
 SupplierID  string
 OrderDate   string
 Currency    string
 TotalAmount float64
 Lines       []InternalOrderLine
}
type InternalOrderLine struct {
 SKU       string
 Qty       int
 UnitPrice float64
}

The internal model does not depend on Json and on the ERP system.

Step 1: Reject bad requests early

With the following code snippet we make sure that our endpoint only accepts POST methods:

if r.Method != http.MethodPost {
    http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
    return
}

Any other HTTP methods are rejected. This ensures the system never processes invalid transport data.

Step 2: Decode, but don’t trust the data

You should never trust external input - even if it looks clean:

err := json.NewDecoder(r.Body).Decode(&req)
if err != nil {
    http.Error(w, "Invalid JSON payload", http.StatusBadRequest) // 400
    return
}

Step 3: Validate like your business depends on it

I am validating the data from a business point of view. This is where most production systems fail silently.

 if req.Order.OrderID == "" {
  return errors.New("missing order_id")
 }
 if req.Order.Currency == "" {
  return errors.New("missing currency")
 }
 if req.Order.TotalAmount <= 0 {
  return errors.New("total_amount must be > 0")
 }
 // Lines
 if len(req.Order.Lines) == 0 {
  return errors.New("order must contain at least one line")
 }

If there is an error, it sends an HTTP response (402 Unprocessable entity) to the client.

err = validateRequest(req)
if err != nil {
    http.Error(w, err.Error(), http.StatusUnprocessableEntity)
    return
}

Step 4: Map to something you control

I am translating the external payload to an internal domain model. This is the real architectural boundary that prevents my system from collapsing when external formats change.

func mapToInternal(req IncomingOrderRequest) InternalOrder {
    lines := make([]InternalOrderLine, 0, len(req.Order.Lines))

    for _, l := range req.Order.Lines {
        lines = append(lines, InternalOrderLine{
            SKU: l.SKU,
            Qty: l.Qty,
            UnitPrice: l.UnitPrice,
        })
    }
    return InternalOrder{
       MessageID: req.MessageID,
       OrderID: req.Order.OrderID,
       SupplierID: req.SupplierID,
       OrderDate: req.Order.OrderDate,
       Currency: req.Order.Currency,
       TotalAmount: req.Order.TotalAmount,
       Lines: lines,
       }
    }

Step 5: Send back response

My REST API should return an HTTP response code that the operation has succeeded.

w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusCreated)

Step 6: Create an order endpoint

We create an API endpoint at /orders, which handles the requests using the orderHandler function.

func main() {
    http.HandleFunc("/orders", orderHandler)
    fmt.Println("Server running on :8080")
    log.Fatal(http.ListenAndServe(":8080", nil))
}

Step 7: Test your service end-to-end

With the REST API up and running, we can now test and see if it works.

go run main.go

In a second Terminal session, we will use the curl request.

curl -X POST http://localhost:8080/orders \
     -H "Content-Type: application/json" \
     -d '{
            "message_id": "msg-1",
            "supplier_id": "supplier-1",
            "order": {
                        "order_id": "ord-1",
                        "order_date": "2026–02–01",
                        "currency": "EUR",
                        "total_amount": 100,
                        "lines": [
                          {
                            "sku": "SKU-1",
                            "qty": 1,
                            "unit_price": 100
                          }
                        ]
                      }
                    }'

What happens if you don’t do this

If you don’t separate transport and domain models, your system won’t fail immediately — it will fail the moment something external changes. Good integration is not about moving data. Integration systems don’t fail because of code. They fail because they don’t control change. Once you separate transport and domain models, your system becomes resilient by design.

You can find the implementation of the code discussed in this article on GitHub. Feel free to clone it and extend it for your own integration use cases.
Subscribe to my Substack newsletter to get future articles and engineering breakdowns.
If you want to go deeper, I created a premium version of this project on Gumroad. Especially useful if you’re a developer who want to build or understand real-world integration services faster or if you are preparing for backend or ERP integration interviews.

Thank you for taking the time to read my articles about building real-world integration services in Go. Happy Coding!

How to Find and Remove Anomalies with Local Outlier Factor (LOF)

Steffi — Wed, 25 Mar 2026 15:00:00 +0000

What Are Outliers?

Outliers are data points that differ significantly from the rest of the dataset.

Global Outlier: falls outside the normal range of a dataset
Local Outlier: are outliers that are within the normal range of a dataset but different from its neighbours

Machine learning algorithms don't work well when outliers are present. Outlier detection is important in many applications.

Why Local Outlier Factor (LOF)?

LOF is a density-based, unsupervised approach:

which identifies outliers relative to their local neighbourhood
LOF Score > 1 → Outlier
Fast and robust for clusters with varying densities

Implementation in Python

Import Libraries

from sklearn.neighbors import LocalOutlierFactor
import pandas as pd
import matplotlib.pyplot as plt

Load Dataset

data = pd.read_csv("fraud_lof_example.csv")

Define LOF Model

lof = LocalOutlierFactor(n_neighbors=20, contamination=0.1)
scores = lof.negative_outlier_factor_
clean_data = data[yhat != -1]

Visualise Outliers

outliers = data[yhat == -1]

plt.figure(figsize=(10,6))
plt.scatter(data['Amount'], data['Distance_to_Home_Town'], color='blue', label='Normal')
plt.scatter(outliers['Amount'], outliers['Distance_to_Home_Town'], color='red', label='Outliers')
plt.xlabel('Amount')
plt.ylabel('Distance to Home Town')
plt.title('Local Outlier Visualization')
plt.legend()
plt.show()

Conclusion

This was an introduction to local outlier detection and Local Outlier Factor. LOF is a simple yet effective method to detect and remove local outliers.

If you’d like to try LOF hands-on, I’ve put together a small practical bundle with:

A Jupyter Notebook (complete LOF implementation)
A Cheat Sheet (parameters and score interpretation)
A Sample Dataset (ready to use)

You can find it here:
Anomaly Detection Starter Kit

With the notebook, cheat sheet, and sample dataset, you can start cleaning your data for ML immediately.

Want more practical ML tutorials? Subscribe to my newsletter:
DataScience&AILearningJourney’s

For a deeper, more theory-focused explanation, you can check out the full version on Medium:
Medium Article

DEV Community: Steffi