DEV Community: Ruben Burdin

The Seven Engineering Problems That Make Real-Time Enterprise Sync Almost Impossible

Ruben Burdin — Mon, 06 Apr 2026 20:36:30 +0000

I spent 18 months trying to make two databases agree with each other. Not eventually. Not within 15 minutes. In real time, bidirectionally, without losing data.

The first version crashed after 10,000 records. The second version handled the volume but corrupted fields when both systems wrote to the same record within the same second. The third version solved conflicts but broke each time that Salesforce added a custom field. I threw out all three and started over.

That was 2022. I had a business degree, six months of self-taught programming, and a problem I could not stop thinking about: why is it so difficult to keep a CRM and a database synchronized in real time?

Three years and one Y Combinator batch later, Stacksync syncs millions of records across 200+ enterprise systems with sub-second latency. I want to explain why this problem is as hard as it is, because most engineering teams underestimate it until they're six months into a failing project.

1. Polling Is a Lie You Tell Yourself

The first approach every team tries is polling. You set up a cron job that checks Salesforce for changes every five minutes. Maybe every minute if you're ambitious. You pull the delta, write it to Postgres, and call it done.

It works on the demo. It falls apart in production.

Polling has a floor. You cannot poll faster than the API allows, and enterprise APIs are not built for high-frequency reads. Salesforce enforces a daily API call limit that depends on your license tier and the number of seats in your org. A mid-size company with 100 users gets roughly 100,000 API calls per day. That sounds like a lot until you realize that a single SOQL query checking for updated records across five objects consumes five calls. Run that query every minute and you've burned 7,200 calls by end of day on a single sync job. Add a second system, add more objects, add any complexity at all, and you hit the ceiling before lunch.

The deeper problem with polling is temporal. Between polls, the world changes. A sales rep updates an opportunity at 10:01:14. Your poll runs at 10:01:00 and again at 10:02:00. For 46 seconds, your database is wrong. If anything downstream reads that record during those 46 seconds, a customer portal shows stale pricing, an internal tool triggers the wrong workflow, a report goes out with yesterday's numbers.

Forty-six seconds sounds minor. Multiply it by every record, every object, every connected system, and you have a data infrastructure that is never fully consistent. You just hope the inconsistency window is small enough that nobody notices.

We noticed.

2. Change Data Capture Sounds Simple Until You Build It

The alternative to polling is event-driven architecture. Instead of asking "what changed?" on a timer, you listen for change events as they happen. Salesforce offers Change Data Capture. HubSpot has webhooks. NetSuite has SuiteScript triggers.

Each implementation is different. Each has its own delivery guarantees, event formats, retry logic, and failure modes.

Salesforce CDC publishes events to a streaming API channel with a 72-hour retention window. If your listener goes down for longer than that, events are gone. No replay. You need to detect the gap, fall back to a full or incremental poll to reconstruct the missing window, then resume streaming without duplicating records or missing the transition point. That recovery logic alone took our team weeks to get right, and it still needed refinement after edge cases surfaced in production.

HubSpot webhooks fire on property changes but batch them. You receive a payload containing multiple changes to multiple records, and the ordering within that batch is not guaranteed. If record A was updated before record B, you may receive B first. For independent records, ordering does not matter. For related records where a parent update must land before a child reference, out-of-order delivery corrupts your foreign key relationships.

NetSuite SuiteScript triggers fire synchronously inside the transaction. If your sync logic is too slow, the user's save operation in NetSuite hangs. You are directly on the critical path of another vendor's product. Time out, and the user sees an error they cannot diagnose.

Every connector demands its own CDC implementation, its own retry semantics, and its own failure recovery path. There is no universal standard. We build and maintain a separate ingestion pipeline for each of the 200+ systems Stacksync connects to.

3. Bidirectional Sync Is a Distributed Consensus Problem

One-way sync is a pipeline. Data flows from A to B. If B receives a record it already has, you overwrite it. If B receives a new record, you insert it. The logic fits in a page of pseudocode.

Two-way sync is a distributed systems problem. Both A and B can write to the same record at the same time. Neither system knows the other system is also making changes. There is no shared clock, no shared transaction log, no coordinator sitting between them enforcing order.

This is a variant of the same problem that databases solved decades ago with distributed consensus protocols like Paxos and Raft. The difference is that you do not control either endpoint. Salesforce is not going to implement your consensus protocol. Neither is Postgres. You are synchronizing two sovereign systems that have no awareness of each other.

The naive solution is "last write wins." Whichever timestamp is later takes precedence. This fails for three reasons.

First, clocks drift. Salesforce's server clock and your database clock are not perfectly synchronized. NTP reduces drift to milliseconds, but milliseconds matter when two writes happen within the same second.

Second, granularity varies. Salesforce timestamps record-level changes. If a rep updates the phone number at 10:01:00 and your system updates the email at 10:01:00, last-write-wins at the record level discards one of those changes. You need field-level conflict detection, which means tracking individual field timestamps across both systems and merging them independently. The data model for this alone is significant.

Third, intent matters. Some fields should always defer to the CRM because the sales team owns them. Other fields should always defer to the database because an automated pipeline owns them. A blanket conflict resolution strategy treats all fields the same, which is wrong for every real-world use case. You need per-field, per-object, per-direction conflict policies that the customer can configure without writing code.

We spent four months on our conflict resolution engine before a single customer used it. It is the hardest part of the system and the least visible.

4. Schema Is a Moving Target

Enterprise systems do not have fixed schemas. Salesforce admins add custom fields weekly. HubSpot users create custom properties for every new campaign. NetSuite consultants build custom record types for industry-specific workflows.

When a new field appears in the source system, your sync layer has three options: ignore it, fail, or adapt. Ignoring it means data loss. Failing means downtime. Adapting means your system must detect the schema change, determine its type and constraints, create the corresponding column in the target database, backfill historical values for that field, and resume syncing, all without interrupting the ongoing sync of every other field.

This is schema evolution, and it has to happen automatically because you cannot ask a customer to manually adjust their database schema every time a Salesforce admin adds a picklist field.

The edge cases are where it gets brutal. What happens when a field is renamed? Your sync layer mapped it by its API name, which in Salesforce is immutable, but in HubSpot it can change. What happens when a field type changes from text to number? Your Postgres column is a VARCHAR and the source is now sending integers. What happens when a field is deleted? Do you delete the column and lose historical data, or mark it deprecated and keep it?

Every schema change is a migration event that has to execute live, without locks, on a table that is actively receiving writes from both directions. We handle thousands of these per week across our customer base.

5. Rate Limits Are Adversarial by Design

Enterprise SaaS vendors set API rate limits to protect their infrastructure from misbehaving integrations. It's reasonable from their perspective and adversarial from yours.

Salesforce enforces both daily and concurrent request limits. HubSpot throttles at 100 requests per 10 seconds for OAuth apps. NetSuite's concurrency model limits you to a handful of simultaneous connections depending on the customer's license.

Your sync engine needs to be the most efficient possible consumer of these APIs. Every unnecessary call is a call your customer cannot use for something else. Every burst that triggers throttling creates a backlog that degrades your latency SLA.

We implemented adaptive rate management that monitors remaining quota in real time and adjusts throughput dynamically. When quota gets low, the engine shifts to larger batch sizes with fewer calls. When quota is abundant, it runs smaller, more frequent batches for lower latency. The system optimizes continuously for the tradeoff between freshness and quota consumption, and the optimal point shifts throughout the day as the customer's other integrations compete for the same pool.

This is invisible to the customer. They see sub-second sync. Underneath, the engine is playing a real-time resource allocation game against constraints it does not control.

6. Ordering Across Distributed Systems

When you sync a Salesforce Account and its child Contacts to Postgres, the Account must exist before the Contacts reference it. If the Contact arrives first and the Account does not yet exist in Postgres, the foreign key insert fails.

In a single-system database, this is solved by transactions. You wrap both inserts in a transaction and they either both succeed or both roll back. Across systems, there is no transaction boundary. Events arrive when they arrive. The Account creation event might be delayed by network latency, CDC channel ordering, or a retry cycle, while the Contact creation event arrives immediately.

You need a dependency graph that understands the relationships between objects, holds child records until their parents exist, and processes them in the correct topological order. This graph has to account for circular dependencies (Object A references Object B which references Object A), self-references (an Account with a parent Account), and polymorphic lookups (a field that can reference different object types depending on the record).

We process these dependency graphs in real time as events arrive, reordering them on the fly without buffering longer than necessary. Getting this wrong means either data loss (dropping events that cannot be ordered) or data corruption (inserting records with dangling references).

7. The Compounding Effect

Each of these problems is hard in isolation. Together, they multiply.

A schema change triggers a migration that temporarily increases API calls, which hits a rate limit, which creates a backlog, which delays events, which breaks ordering guarantees, which causes a child record to arrive before its parent, which triggers a dependency hold, which extends the latency, which means the conflict resolution window widens, which means more conflicts need to be resolved, which means more writes, which consumes more API quota.

One perturbation propagates through every layer. The system must absorb these cascades without data loss, without latency spikes visible to the customer, and without human intervention.

This is why most internal sync projects fail. The team builds something that works for the first use case, in the first month, with the first 10,000 records. Then the schema changes. Then the volume grows. Then a second system is added. Then an edge case creates a cascade, and the debugging obliges you to trace events across three systems, two CDC channels, a conflict resolution log, and a dependency graph.

Most teams give up and switch to batch ETL. They accept 15-minute delays because the alternative is building a distributed systems engine that very few engineering teams have the time, budget, or expertise to maintain.

What I Learned

Synchronizing enterprise systems in real time is a distributed systems problem wearing a SaaS costume. It looks like a simple data pipeline until you peel back the first layer and find consensus algorithms, schema evolution, adversarial rate limiting, and causal ordering all compressed into a single system that has to run 24/7 without losing a single record.

We have been building Stacksync for three years. The problem has not diminished. Every new connector, every new edge case, every new scale milestone reveals another layer of complexity that was invisible at the previous level.

If your team is considering building this in-house, my honest recommendation is to reflect carefully on the total cost. Not the cost of the first version, which will work. The cost of the 40th version, which is the one that actually survives production at scale.

The engineering is possible. I know because we did it. The question is whether that engineering is the best use of your team's time when their actual job is building your product, not maintaining the plumbing underneath it.

That distinction tends to clarify the decision quickly.

The Complete Guide to Two Way Sync: Definitions, Methods, and Use Cases

Ruben Burdin — Fri, 20 Feb 2026 22:04:16 +0000

Two-way sync is a method used to keep data consistent across two systems. It allows updates made in either system to appear in the other, so both always reflect the latest changes.

Key Takeaways

Definition: Two-way sync (bidirectional) maintains data consistency by reflecting changes made in either of two connected systems.
Comparison: Unlike one-way sync, which is unidirectional for backups or reporting, two-way sync enables active collaboration and operational workflows.
Primary benefits: Organizations typically see a 35–50% improvement in data accuracy and a 30–40% increase in team productivity.
Core use cases: Most common in CRM–ERP integrations, support ticket visibility, and HR onboarding or offboarding.
Implementation: Effective deployment requires precise field mapping, clear conflict resolution rules such as “latest update wins,” and continuous monitoring.

Two-way synchronization keeps data consistent across systems by allowing updates to flow in both directions. It differs from one-way sync in its bidirectional data flow and is commonly used in CRM-ERP integrations, support systems, and HR platforms. It also outlines key setup steps, common challenges, and practical considerations for implementation.

Two-way sync enables organizations to maintain data consistency across critical business systems, from CRM platforms to ERP solutions. The technical complexity is manageable when approached with the right framework and infrastructure.

What Is Two-Way Sync

Two-way sync (also called bidirectional sync or 2 way sync) automatically reflects data changes made in either of two connected systems in both systems. This ensures both systems always have the most up-to-date information.

Consider a practical scenario: when a sales representative updates a customer's phone number in Salesforce, that change automatically propagates to your PostgreSQL database. If someone later changes the address in Database B, that new address appears in Database A.

Unlike one-way sync, which pushes data from a source to a destination, two-way sync allows updates to flow in both directions.

Two-way synchronization delivers three critical capabilities that distinguish it from traditional data integration approaches:

Real-time updates: Changes in one system quickly appear in the other
Bidirectional flow: Data moves in both directions
Automated reconciliation: Systems resolve differences to stay in harmony

How Does Bidirectional Sync Differ From One-Way Sync

One-way synchronization operates as a unidirectional data pipeline, pushing updates from a source system to a destination without feedback. This approach works well for analytics and reporting workflows where the destination system serves as a read-only data consumer.

Two-way sync (also called bidirectional sync or 2 way synchronization) updates both systems whenever a change occurs in either one. This keeps both systems consistent by allowing data to flow in both directions.

One-way sync excels in analytics workflows, such as scheduled exports from Salesforce to Snowflake or BigQuery, where the data warehouse serves as the analytical endpoint rather than an operational system requiring updates.

Two-way sync becomes essential when operational systems like CRM and ERP platforms must maintain real-time data consistency to support critical business processes. If a customer's address is updated in either system, the change appears in both.

In one-way sync, the source system holds the authoritative version of the data. In two-way sync, both systems are considered sources of truth, which requires conflict resolution rules to handle situations where the same data is changed in both systems simultaneously.

Common Use Cases For Two-Way Synchronization

Organizations implementing two-way synchronization report 35-50% improvements in data accuracy and eliminate hours of manual reconciliation work across sales, support, and operations teams.

Logistics: Synchronize shipment tracking between databases and CRMs to provide real-time updates without manual entry.
Financial Services: Maintain compliance by ensuring client portfolio data remains consistent between trading platforms and CRM systems.
SaaS: Implement reverse ETL workflows to sync product usage analytics from data warehouses back to Salesforce.

Moving Forward With Two-Way Sync

Two-way sync delivers:

Improved Data Consistency: Automated updates reduce error rates and eliminate up to 40% of manual data entry.
Increased Productivity: Teams access real-time data without switching systems, boosting efficiency by 30-40%.
Better Decision-Making: Access to current, accurate data from across the entire organization.

Reliable two-way synchronization requires three technical foundations: precise data mapping, intelligent conflict resolution, and enterprise-grade security, all of which Stacksync delivers through its managed platform infrastructure. Each system involved must support consistent data formats and respond accurately to updates made in the other system.

Stacksync's mission centers on removing the technical barriers that traditionally make two-way synchronization complex and resource-intensive. By abstracting away API management, infrastructure provisioning, and conflict resolution logic, we enable organizations to implement enterprise-grade data sync in days rather than months, allowing technical teams to focus on innovation and business growth rather than integration maintenance.

Ready to see a real-time data integration platform in action? Book a demo with real engineers and discover how Stacksync brings together two-way sync, workflow automation, EDI, managed event queues, and built-in monitoring to keep your CRM, ERP, and databases aligned in real time without batch jobs or brittle integrations.