Three questions sit underneath every integration purchase, usually unasked in the sales call. What happens to our historical data? Why does data duplicate when systems sync? And how do we know the systems still agree a year from now? They deserve direct answers, because the fear behind them is legitimate: integration touches the one asset a business cannot re-buy.
Historical data: migrated, referenced, or left in place
When a new system joins the stack, historical data has three possible fates, and good projects choose deliberately per data type rather than defaulting.
Migrated data is moved into the new system, typically open transactions and active master data: unshipped orders, current customers, live products. Migration is a one-time project with validation before cutover.
Referenced data stays in the source system but remains reachable, typically closed transactions needed for service and reporting. Integration platforms make this practical: the new system queries what it needs without carrying a decade of history.
Archived data is retained for compliance and analytics outside the operational flow.
The mistake to avoid is migrating everything. Bulk-moving years of closed transactions imports every historical inconsistency into a clean system, extends the timeline, and serves no operating purpose that referencing would not. Integration, properly designed, is what makes selective migration safe: the connected stack can always reach back.
*Why data duplicates, and the design that prevents it *
Duplication during sync has a small set of root causes, all preventable. Retries without idempotency: integration systems re-deliver messages after failures by design, and if the receiving side cannot recognize a repeat, it creates a copy. The fix is deterministic record identity, keys derived from the data itself, so a retry produces the same record, not a sibling. Matching without rules: when two systems both hold customers, sync without explicit match logic, on what combination of fields two records are the same entity, manufactures duplicates at scale. And parallel entry during cutover: the window where staff still key into both systems is where duplicate clusters are born; shorten it deliberately.
A buyer can test for all three with one question: show me how a failed order sync is retried, and prove it cannot create a second order. A vendor with a real answer names idempotent processing and demonstrates the error queue.
Drift: the slow failure nobody demos
The third risk arrives after go-live. A mapping bug, a manual edit, a plugin update, and the systems begin to disagree quietly. The protection is reconciliation as a scheduled habit: automated jobs comparing record counts and checksums between systems, with diffs surfaced to a human. Mature integration deployments treat reconciliation the way finance treats month-end close, routine, boring and non-negotiable.
This is also where architecture pays rent. Hub-based designs with a canonical data model, where each system maps once to a shared representation, localize drift when it happens: one mapping to inspect, not a mesh of point-to-point translations. An explanation of canonical data models is at https://www.appseconnect.com/what-are-canonical-data-models/ and a comparison of the canonical and platform approaches at https://www.appseconnect.com/canonical-data-model-vs-ipaas
The questions to put in your next vendor call
Four, verbatim. What is your recommended treatment of our historical transactions, and why? How does your platform guarantee a retried message cannot duplicate a record? What are your default match rules for customers, and can we change them? And what does your reconciliation tooling show us, how often?
Integration vendors answer feature questions fluently. Data integrity questions reveal which of them run production systems and which run demos.
Frequently asked questions
Is historical data lost during system integration
No. Integration does not delete source data; the project decides per data type whether history is migrated, referenced in place, or archived. Loss risk concentrates in poorly validated one-time migrations, which is an argument for migrating less and referencing more.
Why does order data duplicate between synced systems
Almost always: retries without idempotent processing, or missing match rules. Both are solved problems in mature platforms, deterministic record keys and explicit matching logic, and both are worth verifying in a demo rather than assuming.
How often should connected systems be reconciled
Daily automated reconciliation on operational objects such as orders and inventory, with immediate alerting on diffs, is a reasonable production standard. Manual quarterly checks are how drift compounds unnoticed.
Top comments (0)