How a Small Product Sync Automation Changed Onboarding at Scale

Suganth — Sun, 31 May 2026 15:35:34 +0000

How a Product Sync Automation Project Transformed Customer Onboarding

When people think about impactful engineering work, they often imagine distributed systems, high-scale infrastructure, or complex algorithms.

One of the most impactful projects I worked on wasn't any of those.

It was solving a seemingly simple problem:

Keeping product data in sync across multiple retail systems.

Years later, our CEO still remembers how much smoother customer onboarding became after this project.

The Context: What is Commerce Connect?

At Casa Retail AI, we have an internal platform called Commerce Connect (CC).

Commerce Connect acts as the central Product Information Management (PIM) system and serves as the source of truth for product information.

Under the hood, it is built on top of a customized version of the open-source e-commerce platform Spree Commerce, extended with multi-vendor and multi-tenant capabilities.

Its primary responsibility is simple:

Collect product information from multiple retail ecosystems and distribute it to every Casa product that needs it.

Once product data enters Commerce Connect, it is synchronized to multiple downstream systems.

Why Product Data Matters

Many applications inside Casa depend on product information.

Product Consumers

Once product data enters Commerce Connect, it is distributed to multiple systems across the Casa ecosystem.

Customer-Facing Applications

Several products rely on product information to provide context and improve customer experience:

Lead management applications use product information during customer interactions.
Ticket management systems link customer issues to specific products.
Digital receipts display product names, images, and related details.

Analytics & Reporting

Product data powers business dashboards and reports, helping retailers answer questions such as:

Which categories perform best?
Which products attract the most attention?
Which products generate the most complaints?

It is also used for filtering and segmentation across various dashboards.

External Integrations

Some tenants require product synchronization with external platforms, including:

B2B applications
E-commerce experiences
Digital catalog/flipbook solutions
Custom third-party integrations

Because multiple systems depend on the same product catalog, maintaining a single source of truth is critical.

This ensures every system across the ecosystem works with consistent and up-to-date product information.

The Original Process

Whenever a new tenant was onboarded, the first challenge was importing their entire product catalog.

Most retailers maintained product data in POS systems such as:

SAP
Wondersoft
APX
Other proprietary retail platforms

The onboarding process looked like this:

Customer Success (CS) teams collected product exports from the tenant.
The tenant provided CSV files containing product information.
CS teams manually mapped CSV columns to Commerce Connect fields.
The CSV was uploaded into Commerce Connect.
Each upload created a batch task.
Background workers processed the batch and created products inside Commerce Connect.
After products were created, additional synchronization jobs were triggered for:
- PostgreSQL
- ClickHouse
- B2B applications
- Other enabled integrations

Visually, the flow looked something like:

POS Export
    ↓
CSV File
    ↓
Manual Mapping
    ↓
Commerce Connect
    ↓
Batch Processing
    ↓
PG / ClickHouse / B2B Sync

At first glance, this seemed reasonable.

But as the business scaled, the cracks started to show.

The Problems We Started Seeing

1. Product Data Became a Bottleneck

A tenant could not fully use the platform until their product catalog was available.

This meant onboarding timelines were heavily dependent on receiving CSV files from the customer.

Sometimes the customer took days to provide them.

Sometimes the files were incomplete.

Sometimes they contained incorrect mappings.

Engineering wasn't the bottleneck.

Product data was.

2. Keeping Products Updated Was Painful

Product catalogs constantly change.

New products get added.

Existing products get updated.

Products become inactive.

All of these changes happen in the retailer's POS system.

Whenever a tenant changed something in their POS, Customer Success teams had to:

Request updated exports.
Receive new CSV files.
Upload them again.
Reprocess the entire catalog.

Keeping thousands of products synchronized through spreadsheets quickly became operationally expensive.

3. Every Tenant Required Manual Configuration

Different POS providers generated different CSV formats.

The Customer Success team had to repeatedly configure mappings for every upload.

The process was repetitive, error-prone, and difficult to scale.

The Key Observation

The product data already existed inside the POS systems.

Why were we asking humans to manually export and upload it?

Why not fetch it directly from the source?

The Solution: Pull Products Directly From POS Providers

SAP was the first provider that offered APIs for product retrieval.

Instead of waiting for CSV exports, we integrated directly with their APIs and automatically imported products into Commerce Connect.

The result was immediate.

No spreadsheets.

No manual exports.

No repeated uploads.

Just product synchronization directly from the source system.

After the success of the SAP integration, we expanded the same approach to other POS providers.

Designing for Scale

One requirement was clear:

We didn't want to rewrite the entire synchronization pipeline every time a new POS provider was added.

So we introduced a provider abstraction using a Factory Pattern.

Factory.build("sap_iplanet")

The factory dynamically resolves and instantiates the appropriate provider implementation.

Each provider only needs to answer a few questions:

How to authenticate?
How to fetch products?
How to transform data?

Everything else remains shared.

POS Provider
      ↓
Provider Adapter
      ↓
Commerce Connect
      ↓
Existing Sync Pipeline

This gave us a huge advantage.

Adding a new POS integration became as simple as creating a new provider class.

The provider automatically inherited:

Batch processing
Error handling
Task creation
Product synchronization
PostgreSQL sync
ClickHouse sync
B2B integrations

The integration surface became dramatically smaller.

Eliminating Repetitive Mapping Work

The old CSV process required Customer Success teams to repeatedly map columns.

For example:

CSV Column          → Casa Field

item_code           → SKU
item_name           → Product Name
dept_name           → Category

Every upload carried the risk of mapping mistakes.

With APIs, we could see the complete product schema exposed by the provider.

The mapping became a one-time configuration.

Provider Field      → Casa Field

item_code           → SKU
item_name           → Product Name
dept_name           → Category

After configuration, all future synchronizations happened automatically.

This significantly reduced operational effort.

More Control Over Synchronization

The CSV approach only supported full catalog uploads.

Whether one product changed or ten thousand products changed, the entire file was processed again.

With API-based synchronization, we gained much more control.

We could:

Pull products on a schedule.
Fetch only relevant updates.
Track synchronization status.
Improve error visibility.
Retry failed imports independently.

The synchronization process became far more predictable and manageable.

Solving the "I Can't Wait Until Tomorrow" Problem

While nightly synchronization worked for most tenants, some businesses needed updates immediately.

Waiting until the next scheduled pull was not acceptable.

To solve this, we introduced a product upsert API.

Now tenants had two options:

Pull Model

Commerce Connect fetches products from the provider daily.

Provider → Commerce Connect

Push Model

External systems push product updates directly into Commerce Connect whenever changes occur.

Provider → Product API → Commerce Connect

This hybrid model gave tenants the flexibility they needed while preserving a single source of truth.

Operational Reliability and Monitoring

Automation is only valuable if it is reliable.

Once product synchronization became dependent on external POS APIs, we had to account for situations outside our control:

API credentials changed
Access tokens expired
Provider endpoints became unavailable
Network failures occurred
Providers returned unexpected response formats
Product payload validation failures happened

To ensure synchronization issues never went unnoticed, we built proactive monitoring and alerting into the pipeline.

Whenever a synchronization task encountered unexpected failures, automated Discord alerts were triggered with relevant diagnostic information, including:

Tenant name
Provider name
Failure reason
API response details
Task identifiers

This allowed the Customer Success team to immediately investigate whether the issue was caused by invalid credentials, provider-side outages, configuration changes, or data quality problems.

Instead of discovering synchronization failures days later through customer complaints, the team could identify and resolve issues proactively.

The result was a much more reliable synchronization platform and significantly reduced operational risk as the number of integrated tenants continued to grow.

The Business Impact

From a technical perspective, this wasn't the most complex system I've built.

There were no distributed consensus algorithms.

No cutting-edge infrastructure.

No massive scale challenges.

But the impact was real.

Customer onboarding became significantly smoother.

Customer Success teams no longer spent hours chasing spreadsheets.

Product synchronization became reliable and automated.

New POS integrations became faster to implement.

Operational effort reduced dramatically.

Most importantly, the business could scale without product data becoming a bottleneck.

What I Learned

One lesson has stayed with me throughout my career:

The most valuable engineering work is not always the most technically sophisticated work.

Sometimes the highest-leverage solution is simply removing friction from a process people repeat every day.

At the time, this felt like a small automation project.

Today, many of the people using the platform don't even realize the problem it solved.

That's probably the best outcome an engineering solution can have.

When a process becomes so smooth that people forget it was ever difficult.

DEV Community: Suganth