DEV Community: Suganth

How We Saved Money and Prevented "Ghost" Messages by Respecting Telecom Quiet Hours

Suganth — Thu, 04 Jun 2026 13:30:11 +0000

Building an automated marketing platform comes with technical challenges.

But sometimes, the biggest challenges aren't technical. They are regulatory.

At Casa Retail AI, we built a robust journey module that targets specific customer cohorts and executes actions, like sending SMS or RCS messages, automatically.

But in India, you can't just text customers whenever you want.

Here is how we implemented a "Quiet Hours" deferral system to comply with telecom laws, save costs, and ensure zero dropped messages.

The Context: India's TRAI Regulations

India's Telecom Regulatory Authority of India (TRAI) enforces strict guidelines for commercial communications.

One of the most important rules is the "Quiet Hours" restriction.

To protect consumers from spam and disturbances, TRAI mandates that promotional SMS and RCS messages cannot be sent between 9:00 PM and 9:00 AM.

This is an important distinction. TRAI quiet hours apply only to promotional messages — re-engagement offers, discount campaigns, win-back journeys, and the like. Transactional messages, such as OTPs, order confirmations, and service alerts, are exempt and can be delivered 24/7.

Our journey module at Casa is purpose-built for marketing automations — targeting customer cohorts and triggering re-engagement actions. This puts every SMS and RCS event we fire squarely in the promotional category.

If one of these journeys triggers at 11:30 PM, the telecom provider will intercept and drop the message.

The Problem: Failed Messages and Hidden Costs

When a promotional journey triggers during quiet hours, the message gets blocked at the telecom provider level.

But this surfaced two serious problems for our tenants (retailers):

The Cost Problem: Some third-party message providers charge the tenant for the API request attempt, even when the message is rejected due to TRAI restrictions. The retailer was being billed for a message their customer never received.
The "Ghosting" Problem: The retailer intended to send a promotional offer — a re-engagement discount or a win-back campaign. Because the message failed silently at the provider level, the customer never received the offer, and that marketing touchpoint was permanently lost.

This is unique to promotional use cases. A transactional OTP or a shipping alert can wait for nothing — it needs to arrive immediately. But a promotional campaign triggered at 11 PM has a natural next-day intent: reach the customer when they are awake.

We needed a system that intercepted these promotional messages before they reached the provider, held them safely overnight, and sent them when it was legal to do so.

The Solution: The Deferral Queue

We introduced a strict time-window check right before message dispatch.

If a message attempts to fire outside of the 9 AM - 9 PM window, we halt the dispatch.

Instead of dropping the message or paying for a failed API call, we push the payload into a separate Kafka log to deferred-sms | deferred-rcs.

The next morning at exactly 9:00 AM, a scheduled worker sweeps these deferred queues and posts the messages back into the primary processing queue.

To give our tenants flexibility, we made this deferral behavior entirely configurable. Tenants can opt-in to deferral or choose to simply drop the messages if time-sensitivity is critical.

Finally, every deferred action is logged into ClickHouse. This provides our analytics engine full visibility into how many messages were deferred, allowing tenants to see exactly how their campaigns behave outside of business hours.

How it works:

Timezone Awareness: We strictly enforce the Asia/Kolkata time zone, meaning our platform can be hosted anywhere globally but will always respect Indian local time rules.
Redis Caching: Configuration reads are cached in Redis for one hour. Since journey engines process thousands of messages per second, hitting PostgreSQL for every message was out of the question.

The Business Impact

This small compliance feature transformed how we pitch our platform's reliability.

We can now promise our tenants two things:

No "Ghost" Messages: If a customer qualifies for a campaign at midnight, they will receive the message at 9 AM the next morning.
No Wasted Costs: Tenants no longer pay external API providers for messages that were guaranteed to fail at the telecom level.

The integration with ClickHouse means that tenants can visualize exactly how much money the deferral queue saved them, building trust in our platform's intelligence.

What I Learned

Regulatory constraints are not edge cases to route around — they are business rules to design for.

The natural instinct is to handle a compliance failure at the point of failure: catch the error, log it, move on. But that approach treats the regulation as an exception rather than a known, predictable constraint.

When we modeled TRAI quiet hours as a first-class concern — one with its own deferral pipeline, its own configuration surface, and its own audit trail in ClickHouse — the system became genuinely more reliable for our tenants.

Compliance and product quality are not in tension. Handled correctly, they reinforce each other.

How We Increased Campaign Speed by 10x: Overcoming Third-Party API Bottlenecks

Suganth — Wed, 03 Jun 2026 10:50:24 +0000

When you build a multi-tenant SaaS product, integrating with third-party providers is inevitable.

But what happens when their latency becomes your bottleneck?

Recently, we encountered a massive performance issue in our notification engine that was stalling campaigns for hours.

Here is how we solved it and increased our campaign dispatch speed by 10x.

The Context: Journey-Based WhatsApp Campaigns

At Casa Retail AI, we have a journey flow where retailers can target specific groups of customers.

For example:

Customers with a Lifetime Value (LTV) greater than ₹10,000.
Customers who haven’t made a purchase in the past year.

Once these cohorts are identified, the system sends them a targeted WhatsApp message to re-engage them.

Because we operate a multi-tenant platform, every tenant (retailer) has their own distinct WhatsApp service provider integrated with Casa.

The workflow seemed straightforward: identify the audience, construct the payloads, and push them to the provider.

The Problem: The API Latency Bottleneck

The issue wasn’t our database.

It wasn't our background job queue either.

The bottleneck was the WhatsApp message providers.

While sending messages, we discovered that some third-party providers took up to 2 seconds to process a single request.

If a tenant launched a campaign targeting a large cohort, processing these messages sequentially was agonizingly slow.

A single campaign could take approximately 5 hours to complete.

The Real Issue: The "Innocent Bystander" Problem

A 5-hour campaign doesn't just affect the tenant running it.

Because we use a shared background job system, those slow jobs were hogging the workers.

Other tenants' jobs were sitting in the queue, waiting for execution.

It wasn’t the other tenants' fault.

It wasn't even the active tenant's fault—they just had a slow provider.

We initially tried to mitigate this by routing payloads to a delayed-queue based on the average response time for that particular campaign.

While this prevented the main queue from grinding to a halt, it was merely treating the symptom, not the root cause.

We needed a way to dispatch these requests much faster.

The Solution: The Concurrent Dispatcher

If a single API call takes 2 seconds, calling it 1,000 times in sequence takes 2,000 seconds.

But if we can run them in parallel, we only pay the latency cost of the slowest concurrent batch.

To achieve this, we built a Concurrent Dispatcher in Scala to process these external API calls simultaneously.

The idea was simple:

Instead of sending one message at a time, group the tasks and fire them concurrently using a dedicated thread pool, awaiting their completion before proceeding.

How It Works:

Dedicated Thread Pool: We use a FixedThreadPool with a configurable concurrency limit (defaulting to 30). This isolates these slow, I/O-bound API tasks from the rest of the application's execution context.
Future Wrapping: Each task is wrapped in a Scala Future, allowing them to execute immediately on the dedicated thread pool.
Future.sequence: This powerful Scala standard library method transforms a Seq[Future[T]] into a Future[Seq[T]], effectively combining them.
Bounded Awaiting: We Await.result with a configurable timeout (defaulting to 5 minutes) to ensure a hung third-party API doesn't hold our workers hostage forever.

The Business Impact

The results were immediate and drastic.

By dispatching requests concurrently, a campaign that previously took 5 hours now finished in roughly a fraction of the time.

That's a massive 10x improvement in campaign speed.

Because campaigns finished faster:

Workers were freed up sooner.
Other tenants no longer experienced phantom delays in the job queue.
We no longer had to aggressively route tasks to a delayed queue just to survive high traffic.

What I Learned

When you integrate with external systems, you are bound by their worst-case latency.

If you process external I/O sequentially in a background worker, you are turning a fast, internal asynchronous system into a slow, synchronous one.

Sometimes, the best solution is simply refusing to wait in line.

By recognizing that our bottleneck was external I/O wait time, introducing a concurrent execution model allowed us to reclaim our system's throughput.

Ticket Desk: A Multi-Tenant CRM Helpdesk at the Heart of Retail

Suganth — Mon, 01 Jun 2026 10:27:40 +0000

When people picture a ticketing system, they usually imagine a basic support inbox — customer sends a message, agent replies, ticket closed.

What we built at Casa Retail AI is something very different.

Ticket Desk is not just a helpdesk. It is the customer issue resolution layer of a full-stack retail CRM and CDP platform, tightly integrated with products, stores, digital receipts, journeys, leads, and analytics. Every ticket is a thread connected to the customer's purchase history, their store, their product, and the SLAs a brand has committed to.

This is the story of how we built it — and what made it genuinely complex.

The Context: What is Casa Retail AI?

At Casa Retail AI, we build a multi-vendor, multi-tenant CRM and Customer Data Platform designed for the Indian retail ecosystem.

A single deployment of Casa serves multiple retail brands — each brand is a tenant. Each tenant may have dozens of stores, hundreds of agents, thousands of customers, and millions of interactions across channels.

Our platform handles marketing automation, loyalty, product catalogs, digital receipts, lead management, and customer support — all under one roof. Every module is interconnected.

Ticket Desk is the customer support module. But the word "support" undersells it.

Making It Multi-Tenant

The first and most foundational challenge was retrofitting multi-tenancy into a system that wasn't designed for it.

Every record — tickets, agents, groups, users, SLA policies, triggers — had to be scoped to a tenant. Every query, every API response, every background job needed to be tenant-aware.

Beyond isolation, each tenant needed their own:

Agent and group hierarchy
Role and permission configuration
SLA policies and business hours
Custom ticket fields
Channel configurations

And because Ticket Desk sits inside the Casa ecosystem, every tenant's data had to stay in sync with their stores, their product catalog, and their customer base — all of which live in other modules.

This tenant boundary is not just a database concern. It shapes every layer of the system: routing, authorization, background jobs, analytics pipelines, and API design.

Multi-Source Ticket Ingestion

Once multi-tenancy was in place, the next challenge was how tickets enter the system.

Customers don't raise issues through a single channel. A retail brand receives complaints and requests from:

Direct — agents creating tickets on behalf of walk-in customers
Email — customer emails routed into the system as tickets
WhatsApp — messages received via WhatsApp Business API, converted into ticket threads
Digital Receipt — a customer raising a ticket directly from the e-bill they received after purchase

Each channel has a different payload shape, different metadata, and different expectations for how the ticket should be created.

But once a ticket is created — regardless of source — it flows through the same core pipeline. Same SLA tracking, same agent assignment logic, same trigger-based workflows.

The channel is context. The ticket is the unit of work.

The Digital Receipt Flow

The digital receipt integration deserves its own mention because of how seamlessly it closes the loop between purchase and support.

When a customer makes a purchase at a retail store, they receive a digital receipt — delivered via WhatsApp, SMS, or email. That receipt is not just a bill. It is an entry point back into the platform.

Embedded in that receipt is a way for the customer to raise a ticket. They can report a damaged product, request a replacement, or flag a service issue — all without knowing they're interacting with a helpdesk system.

When they do, the ticket is created automatically, pre-populated with:

The store where the purchase happened
The product they bought
The invoice details

The agent receiving that ticket has full context before they say a word.

Product and Store Mapping

One of the biggest extensions we built on top of Zammad's core was the ability to link tickets to specific products and stores.

In a retail context, this matters enormously.

When a customer raises a complaint about a refrigerator they bought, the agent needs to know:

Which product (brand, model, SKU)
Which store it was purchased from
Which store is responsible for resolution

We built a product-linking widget inside the ticket view that connects directly to Commerce Connect — our internal Product Information Management system. Agents can search and attach the relevant product to a ticket in seconds.

Store-level grouping works alongside this. Tickets can be routed to the group responsible for a specific store, ensuring the right team handles the issue.

This transforms the ticket from a generic support record into a rich operational object tied to real retail data.

Journey-Driven Ticket Creation

Not all tickets are created by humans.

Some of our most valuable tickets are created automatically by our Journey module — a trigger-based automation engine that can initiate actions based on customer data and time conditions.

The clearest example comes from consumer electronics and appliance retailers like large electronics chains.

When a customer buys an appliance, their purchase is recorded. The journey engine monitors the elapsed time:

6 months after purchase → a service ticket is auto-created, prompting a routine check-in
1 year after purchase → a follow-up service ticket is created, aligned with warranty terms

The agent receives a fully formed ticket with all the context: who the customer is, what they bought, when they bought it, and what the service expectation is.

This is not reactive support. It is proactive, scheduled, warranty-aware customer service — driven by business rules configured once and executed at scale.

Ticket-to-Lead Conversion

Sometimes a support issue reveals a sales opportunity.

Consider a customer who calls in because the display on their phone is cracked. The agent checks with the service team and finds that the spare part is not available. The product cannot be repaired.

In most systems, that ticket would be closed as unresolved.

In Ticket Desk, the agent can convert that ticket into a lead — flagging the customer as a potential buyer for a replacement or upgrade. The lead flows directly into our Lead Management module, where a sales agent can follow up with the right product recommendation.

This is a business workflow that only makes sense inside a unified CRM platform. Ticket Desk is connected to the lead pipeline because they share the same customer record, the same product catalog, and the same organizational hierarchy.

Beyond Customer Support: Internal and Operational Tickets

Not every ticket in Ticket Desk originates from a customer complaint.

One of the less obvious — but genuinely useful — uses of the system is as an internal coordination and operations layer.

Journey Opportunity Suggestions

The Journey module identifies segments of customers who qualify for a targeted campaign — a loyalty offer, a win-back promotion, a seasonal push. But acting on those insights requires a human to sign off, configure, and launch.

Instead of that decision sitting in a dashboard nobody checks, Ticket Desk creates an internal ticket assigned to the relevant marketing or CRM team.

The ticket surfaces:

The customer segment identified
The opportunity type (reactivation, upsell, event-based)
The suggested journey to run
Context on why this segment was flagged

The team reviews it, approves or modifies the journey, and closes the ticket once executed. It keeps the insight actionable and the process auditable — every opportunity suggestion has a paper trail.

Operational Tickets

Beyond journey suggestions, Ticket Desk also handles general operational work that doesn't fit neatly into a customer support flow — store onboarding tasks, internal escalations, vendor coordination items, and process exceptions that need assignment and tracking.

Using the same ticket infrastructure for internal operations meant we didn't have to build or integrate a separate project management or task tracking tool. The same SLA rules, group routing, trigger workflows, and reporting pipeline that handles customer tickets handles operational ones too.

It is a small thing architecturally, but it significantly increased how much the platform was used day-to-day across teams.

Analytics: What the Tickets Tell Us

After a ticket is resolved, the work isn't done.

All ticket data — creation, updates, resolution times, linked products, linked stores, assigned groups, and SLA outcomes — is synchronized to ClickHouse. This is the same ClickHouse cluster that powers analytics across the rest of the Casa platform.

On top of this data, we built dashboards that let brand and operations teams answer questions like:

Which stores are generating the most complaints?
Which products have the highest issue frequency?
Which categories have the worst resolution times?
How are agents performing against SLA targets?
Are certain store clusters systematically underperforming?

This is not just a support dashboard. It is a retail operations intelligence tool.

The connection between tickets and products means that a pattern of complaints about a specific SKU shows up in the data before it becomes a brand reputation problem. Store managers can act on it. Buyers can flag it with the vendor. The loop closes.

What Made This Hard

Looking back, the technical challenges weren't in any single feature. They were in the integration surface.

Ticket Desk talks to nearly every other Casa module:

Commerce Connect for product data
Store Management for store and agent hierarchy
Digital Receipts for purchase context
Journey Engine for automated ticket creation
Lead Management for ticket-to-lead conversion
ClickHouse for analytics

Keeping all of these integrations consistent — as each module evolved independently — required careful API design, reliable event handling, and a shared understanding of the tenant boundary across teams.

The other challenge was the Zammad fork itself. Extending an open-source platform that wasn't designed for your requirements means you own every deviation. Every upstream change becomes a merge decision. Every extension has to live alongside the original code without breaking what already worked.

That discipline — knowing what to extend, what to replace, and what to leave alone — was as important as any technical decision we made.

What I Learned

Building Ticket Desk reinforced something I believe more firmly with every product I ship:

The value of a platform is not in its individual features. It is in how those features connect.

A digital receipt by itself is just a bill. A ticket by itself is just a support record. A product record by itself is just a data entry.

But when a customer raises a complaint from their receipt, the ticket arrives pre-loaded with their product and store context, gets assigned to the right team by a trigger, tracked against an SLA, and — if the product can't be repaired — flows into the lead pipeline for a sales follow-up…

That is not just engineering. That is a complete customer experience, built out of many small, well-connected systems.

That is what we built with Ticket Desk.

Ticket Desk is part of the Casa Retail AI platform — a multi-tenant CRM and CDP for the Indian retail ecosystem.

How We Stopped Tenants From Hogging the Job Queue - Tenant-Level Parallel Job Limiting in Sidekiq

Suganth — Mon, 01 Jun 2026 08:25:02 +0000

The Context: What is Commerce Connect?

At Casa Retail AI, we have an internal platform called Commerce Connect (CC).

Commerce Connect acts as the central Product Information Management (PIM) system — the single source of truth for product data across our entire retail ecosystem. Built on top of a customized version of the open-source e-commerce platform Spree Commerce, it is extended with multi-vendor and multi-tenant capabilities.

Its job is straightforward:
Collect product information from multiple retail ecosystems and distribute it to every Casa product that needs it — PostgreSQL for operational modules, ClickHouse for analytics, and external B2B integrations for partner systems.

All of this synchronization work happens in the background using Sidekiq.

The Problem: A Shared Queue With No Per-Tenant Guardrails

Sidekiq processes jobs concurrently using a fixed pool of worker threads. At any given time, there is a hard cap on how many jobs can run simultaneously across the entire system.

In practice, this meant:

A tenant uploading multiple CSV files in quick succession
A tenant triggering multiple sync operations to different external systems
Multiple tenants doing all of this at the same time

There was nothing stopping a single active tenant from claiming the majority of worker slots. If one tenant fired several heavy jobs back-to-back, every other tenant's jobs sat in the queue waiting — with no visibility into why.

This wasn't theoretical. As we onboarded more tenants, the pattern became real:

Smaller tenants experienced noticeable delays during high-activity periods from larger ones.
Large catalog syncs from one tenant could hold up all other background work.
There was no way to offer differentiated service — every tenant was treated identically, regardless of their plan or size.

We needed a way to restrict how many jobs a single tenant could run in parallel, at the queue level.

The Solution: Per-Tenant Job Slot Tracking

The idea was simple:

Each tenant gets a configurable maximum number of concurrent jobs per queue. If a job tries to run and the tenant is already at their limit, reschedule it for later and let the next tenant go.

We introduced a database record per tenant per queue, tracking two things:

max_job — how many parallel jobs this tenant is allowed
current_job — how many are currently running

Before any background job executes, it must acquire a slot. When it finishes — or fails — it releases the slot.

For most tenants, the default is max_job: 1. Single-job concurrency. Simple, predictable, fair.
For premium tenants with higher throughput requirements, we raise the limit — no code change required, just a configuration update.

The limit is also queue-aware. A tenant's slot count on the CSV import queue is tracked independently from the sync queue. One slow upload doesn't block a faster downstream sync for the same tenant.

The Flow

When a job is picked up by a Sidekiq worker, before any real business logic runs:

Look up the tenant's config record for this queue. Create it with defaults if it doesn't exist yet — new tenants need no manual setup.
Check if a slot is available (current_job < max_job).
If not — reschedule the job for 1 minute later and return. The work still gets done; it just waits its turn.
If yes — increment current_job and proceed with the actual job.
When the job finishes (or raises an exception), decrement current_job back.

The decrement in step 5 always happens — even on failures. This is the guarantee that prevents a crashed job from permanently consuming a slot and slowly starving the tenant out of all future work.

The Race Condition — and How We Handle It

This is where it gets interesting.

At first glance, "check if slots are available, then increment" sounds simple. But in a concurrent system with multiple Sidekiq threads running simultaneously, this is a classic read-modify-write race condition.

Imagine two jobs for the same tenant land in the queue at almost the same time. Both threads check current_job < max_job. Both see the count is within limits. Both decide to proceed. Both increment. Now the tenant is running more concurrent jobs than allowed — the limit is silently violated.

The naive fix — read the value, check it in application code, then write it back — doesn't hold up under concurrency. There's always a window between the read and the write where another thread can slip through.

The correct fix is to make the check and the increment a single atomic operation at the database level.

Instead of reading the value and then updating it separately, we issue a conditional SQL update that does both in one statement: "increment current_job by 1, but only if current_job is currently less than max_job." The database processes this as one indivisible operation. If two threads race to the same record simultaneously, only one of them will satisfy the condition — the other will see that zero rows were updated, and knows it lost the race.

No Redis locks. No application-level mutexes. No external coordination. The database's own row-level locking gives us the guarantee we need, for free.

The same pattern applies on release: the decrement is conditioned on current_job > 0, preventing the counter from going negative if release is somehow called more times than acquire in an unexpected error flow.

The Business Impact

This was a small module by any measure. But the effect on fairness across tenants was immediate.

No single tenant can monopolize the job queue, regardless of how aggressively they upload or sync.
Most tenants run comfortably on the default single-slot limit — simple, first-in-first-out behavior.
Premium tenants with higher throughput needs get their limit raised with a simple config update.
No new infrastructure. No Redis, no distributed lock service — the existing database was enough.

What I Learned

The most effective solutions are often the ones that fit the shape of the system you already have.

We had PostgreSQL. We had Sidekiq. We didn't need a new component.

One database table. One atomic update pattern. One shared module any worker can include.

That was enough to give every tenant a fair share of the job queue — regardless of how active their neighbors were.

Tags: #rails #sidekiq #ruby #backgroundjobs #softwareengineering

How a Small Product Sync Automation Changed Onboarding at Scale

Suganth — Sun, 31 May 2026 15:35:34 +0000

How a Product Sync Automation Project Transformed Customer Onboarding

When people think about impactful engineering work, they often imagine distributed systems, high-scale infrastructure, or complex algorithms.

One of the most impactful projects I worked on wasn't any of those.

It was solving a seemingly simple problem:

Keeping product data in sync across multiple retail systems.

Years later, our CEO still remembers how much smoother customer onboarding became after this project.

The Context: What is Commerce Connect?

At Casa Retail AI, we have an internal platform called Commerce Connect (CC).

Commerce Connect acts as the central Product Information Management (PIM) system and serves as the source of truth for product information.

Under the hood, it is built on top of a customized version of the open-source e-commerce platform Spree Commerce, extended with multi-vendor and multi-tenant capabilities.

Its primary responsibility is simple:

Collect product information from multiple retail ecosystems and distribute it to every Casa product that needs it.

Once product data enters Commerce Connect, it is synchronized to multiple downstream systems.

Why Product Data Matters

Many applications inside Casa depend on product information.

Product Consumers

Once product data enters Commerce Connect, it is distributed to multiple systems across the Casa ecosystem.

Customer-Facing Applications

Several products rely on product information to provide context and improve customer experience:

Lead management applications use product information during customer interactions.
Ticket management systems link customer issues to specific products.
Digital receipts display product names, images, and related details.

Analytics & Reporting

Product data powers business dashboards and reports, helping retailers answer questions such as:

Which categories perform best?
Which products attract the most attention?
Which products generate the most complaints?

It is also used for filtering and segmentation across various dashboards.

External Integrations

Some tenants require product synchronization with external platforms, including:

B2B applications
E-commerce experiences
Digital catalog/flipbook solutions
Custom third-party integrations

Because multiple systems depend on the same product catalog, maintaining a single source of truth is critical.

This ensures every system across the ecosystem works with consistent and up-to-date product information.

The Original Process

Whenever a new tenant was onboarded, the first challenge was importing their entire product catalog.

Most retailers maintained product data in POS systems such as:

SAP
Wondersoft
APX
Other proprietary retail platforms

The onboarding process looked like this:

Customer Success (CS) teams collected product exports from the tenant.
The tenant provided CSV files containing product information.
CS teams manually mapped CSV columns to Commerce Connect fields.
The CSV was uploaded into Commerce Connect.
Each upload created a batch task.
Background workers processed the batch and created products inside Commerce Connect.
After products were created, additional synchronization jobs were triggered for:
- PostgreSQL
- ClickHouse
- B2B applications
- Other enabled integrations

Visually, the flow looked something like:

POS Export
    ↓
CSV File
    ↓
Manual Mapping
    ↓
Commerce Connect
    ↓
Batch Processing
    ↓
PG / ClickHouse / B2B Sync

At first glance, this seemed reasonable.

But as the business scaled, the cracks started to show.

The Problems We Started Seeing

1. Product Data Became a Bottleneck

A tenant could not fully use the platform until their product catalog was available.

This meant onboarding timelines were heavily dependent on receiving CSV files from the customer.

Sometimes the customer took days to provide them.

Sometimes the files were incomplete.

Sometimes they contained incorrect mappings.

Engineering wasn't the bottleneck.

Product data was.

2. Keeping Products Updated Was Painful

Product catalogs constantly change.

New products get added.

Existing products get updated.

Products become inactive.

All of these changes happen in the retailer's POS system.

Whenever a tenant changed something in their POS, Customer Success teams had to:

Request updated exports.
Receive new CSV files.
Upload them again.
Reprocess the entire catalog.

Keeping thousands of products synchronized through spreadsheets quickly became operationally expensive.

3. Every Tenant Required Manual Configuration

Different POS providers generated different CSV formats.

The Customer Success team had to repeatedly configure mappings for every upload.

The process was repetitive, error-prone, and difficult to scale.

The Key Observation

The product data already existed inside the POS systems.

Why were we asking humans to manually export and upload it?

Why not fetch it directly from the source?

The Solution: Pull Products Directly From POS Providers

SAP was the first provider that offered APIs for product retrieval.

Instead of waiting for CSV exports, we integrated directly with their APIs and automatically imported products into Commerce Connect.

The result was immediate.

No spreadsheets.

No manual exports.

No repeated uploads.

Just product synchronization directly from the source system.

After the success of the SAP integration, we expanded the same approach to other POS providers.

Designing for Scale

One requirement was clear:

We didn't want to rewrite the entire synchronization pipeline every time a new POS provider was added.

So we introduced a provider abstraction using a Factory Pattern.

Factory.build("sap_iplanet")

The factory dynamically resolves and instantiates the appropriate provider implementation.

Each provider only needs to answer a few questions:

How to authenticate?
How to fetch products?
How to transform data?

Everything else remains shared.

POS Provider
      ↓
Provider Adapter
      ↓
Commerce Connect
      ↓
Existing Sync Pipeline

This gave us a huge advantage.

Adding a new POS integration became as simple as creating a new provider class.

The provider automatically inherited:

Batch processing
Error handling
Task creation
Product synchronization
PostgreSQL sync
ClickHouse sync
B2B integrations

The integration surface became dramatically smaller.

Eliminating Repetitive Mapping Work

The old CSV process required Customer Success teams to repeatedly map columns.

For example:

CSV Column          → Casa Field

item_code           → SKU
item_name           → Product Name
dept_name           → Category

Every upload carried the risk of mapping mistakes.

With APIs, we could see the complete product schema exposed by the provider.

The mapping became a one-time configuration.

Provider Field      → Casa Field

item_code           → SKU
item_name           → Product Name
dept_name           → Category

After configuration, all future synchronizations happened automatically.

This significantly reduced operational effort.

More Control Over Synchronization

The CSV approach only supported full catalog uploads.

Whether one product changed or ten thousand products changed, the entire file was processed again.

With API-based synchronization, we gained much more control.

We could:

Pull products on a schedule.
Fetch only relevant updates.
Track synchronization status.
Improve error visibility.
Retry failed imports independently.

The synchronization process became far more predictable and manageable.

Solving the "I Can't Wait Until Tomorrow" Problem

While nightly synchronization worked for most tenants, some businesses needed updates immediately.

Waiting until the next scheduled pull was not acceptable.

To solve this, we introduced a product upsert API.

Now tenants had two options:

Pull Model

Commerce Connect fetches products from the provider daily.

Provider → Commerce Connect

Push Model

External systems push product updates directly into Commerce Connect whenever changes occur.

Provider → Product API → Commerce Connect

This hybrid model gave tenants the flexibility they needed while preserving a single source of truth.

Operational Reliability and Monitoring

Automation is only valuable if it is reliable.

Once product synchronization became dependent on external POS APIs, we had to account for situations outside our control:

API credentials changed
Access tokens expired
Provider endpoints became unavailable
Network failures occurred
Providers returned unexpected response formats
Product payload validation failures happened

To ensure synchronization issues never went unnoticed, we built proactive monitoring and alerting into the pipeline.

Whenever a synchronization task encountered unexpected failures, automated Discord alerts were triggered with relevant diagnostic information, including:

Tenant name
Provider name
Failure reason
API response details
Task identifiers

This allowed the Customer Success team to immediately investigate whether the issue was caused by invalid credentials, provider-side outages, configuration changes, or data quality problems.

Instead of discovering synchronization failures days later through customer complaints, the team could identify and resolve issues proactively.

The result was a much more reliable synchronization platform and significantly reduced operational risk as the number of integrated tenants continued to grow.

The Business Impact

From a technical perspective, this wasn't the most complex system I've built.

There were no distributed consensus algorithms.

No cutting-edge infrastructure.

No massive scale challenges.

But the impact was real.

Customer onboarding became significantly smoother.

Customer Success teams no longer spent hours chasing spreadsheets.

Product synchronization became reliable and automated.

New POS integrations became faster to implement.

Operational effort reduced dramatically.

Most importantly, the business could scale without product data becoming a bottleneck.

What I Learned

One lesson has stayed with me throughout my career:

The most valuable engineering work is not always the most technically sophisticated work.

Sometimes the highest-leverage solution is simply removing friction from a process people repeat every day.

At the time, this felt like a small automation project.

Today, many of the people using the platform don't even realize the problem it solved.

That's probably the best outcome an engineering solution can have.

When a process becomes so smooth that people forget it was ever difficult.