Direct answer: Once a catalog crosses roughly 50,000 SKUs, or the source system is a custom ERP rather than a standard cart, CSV imports and off-the-shelf migration apps stop being a data problem and start being a rate-limit and reconciliation problem. The fix isn't a bigger export file. It's treating the migration as a real ETL pipeline: extract with change data capture so the source stays live, stage and validate in S3, and load through Shopify's asynchronous Bulk Operations API instead of looping calls against the Admin API. This post walks through that architecture and where it breaks down at different scales.
Why the standard Shopify Plus migration path stops working at scale
Most Shopify Plus migrations start the same way: export CSVs from the old platform, clean them in a spreadsheet, and either run Shopify's native import or a migration app like Matrixify. That works fine for a few thousand SKUs with simple variant structures.
It stops working for two structural reasons, not just volume.
First, the Shopify Admin API is rate-limited by design, and the limit is enforced per shop regardless of which tool is calling it. Per Shopify's current API rate limit documentation, the GraphQL Admin API uses a calculated query-cost model: a Shopify Plus store restores 1,000 cost points per second against its bucket, versus 100 for a standard plan, and the REST Admin API caps Plus stores at 20 requests per second versus 2 for a standard plan. A migration app doesn't get a special exemption from this. If your product-plus-metafield-plus-variant graph requires four or five API calls per SKU, a six-figure catalog will take the better part of a week to load serially, and that's before retries.
Second, CSV round-trips lose structure. Metafields, metaobjects, B2B catalog assignments, and multi-location inventory don't map cleanly onto flat rows, so teams end up hand-patching data after import, which is exactly where SKUs go missing or price rules apply to the wrong customer group.
The practical implication: once you're migrating a catalog that's large, deeply structured, or sourced from a system with no clean export tool (a custom ERP, a legacy PIM, a headless commerce backend), the migration needs an actual extraction and transformation layer sitting in front of Shopify, not a spreadsheet.
What an AWS-backed migration pipeline actually looks like
The shape of this is closer to a data engineering pipeline than a website migration:
[Legacy DB / ERP / PIM]
|
v
AWS DMS (full load + CDC) ----> Amazon S3 (raw zone)
|
v
AWS Glue / Lambda (transform, dedupe, map schema)
|
v
Amazon S3 (validated zone) ----> reconciliation checks
|
v
JSONL build (products, variants, metafields, customers, orders)
|
v
Shopify staged upload ----> bulkOperationRunMutation
|
v
Shopify Plus store (staging, then production)
Each arrow in that diagram is a place teams lose data if it's skipped, which is really the argument for building it this way rather than scripting a one-off exporter.
Extraction (AWS DMS). Rather than taking the source database offline for a bulk export, AWS DMS performs a full load of existing records and then switches to change data capture (CDC), continuously replicating inserts, updates, and deletes from the source until you're ready to cut over: the source database stays operational throughout, and the target stays synchronized for as long as you need before you switch over. That matters specifically for commerce migrations because orders and inventory keep changing during the weeks a large migration takes to build and test, so you can't just freeze the source system.
Staging and transformation (S3, Glue, Lambda). Raw extracted data lands in an S3 "raw zone" untouched, then a transform layer (Glue, a serverless data integration service with no infrastructure to provision, for anything approaching bulk ETL, paired with Lambda for lighter per-record logic) reshapes source rows into Shopify's object model: variant option combinations, metafield namespaces, customer-to-company relationships for B2B. AWS's own reference pattern for this kind of pipeline explicitly separates validation, transformation, and partitioning into distinct orchestrated steps with automated retry and error handling. It's worth copying directly, because "transform" and "validate" being separate steps is what lets you catch bad data before it reaches Shopify rather than after.
Orchestration (Step Functions or EventBridge). A migration isn't one job, it's a dependency graph: customers before orders, products before variants, collections before the products that reference them. Step Functions state machines make that sequencing and its error handling visible and restartable at the step that failed, rather than needing to rerun an entire script.
Loading into Shopify: bulk operations, not a request loop
This is the step where AWS-side engineering meets Shopify-side engineering, and it's the one most agencies underbuild.
Shopify's Bulk Operations API is built for exactly this case: instead of one GraphQL mutation per record, you upload a single JSONL file where each line holds one mutation's variables, reserve the upload target with stagedUploadsCreate, then run bulkOperationRunMutation against the uploaded file, and Shopify executes every line asynchronously in the background. This entirely bypasses the per-request rate limit that would otherwise throttle a six-figure catalog load.
A minimal shape of the mutation that kicks off the import:
mutation {
bulkOperationRunMutation(
mutation: "mutation call($input: ProductInput!) { productCreate(input: $input) { product { id } userErrors { field message } } }"
stagedUploadPath: "tmp/4/bulk-op-inputs/product-import.jsonl"
) {
bulkOperation {
id
status
}
userErrors {
field
message
}
}
}
Two constraints worth designing around from day one: per Shopify's own API reference, only one bulk mutation operation can run at a time per shop (a bulk query can run alongside it, but not another bulk mutation), and result files are only available for seven days after the operation completes, so your validation step needs to run promptly rather than "whenever someone gets to it." Neither of these is a big deal if you planned for a sequential pipeline; both cause real problems if your Step Functions state machine assumed it could parallelize product and variant loads freely.
The validation layer that actually prevents launch-day surprises
The failure mode nobody plans for isn't "the migration failed." It's "the migration succeeded and the numbers are quietly wrong." A price rule applies to the wrong customer group. Three thousand SKUs land without their images because the CDN path changed casing. Order history imports but loses the tax line because the source system stored tax differently than Shopify's tax model expects.
This is where having the migration built on a real data engineering stack pays off, because reconciliation is a data problem, not a migration-app checkbox. A working validation pass typically checks:
- Row-count parity: does the object count in Shopify's bulk operation result match the row count in the source extract, per object type
- Referential integrity: does every variant resolve to a real product, every order to a real customer, every metafield to a defined definition
- Sampled field-level diffing: pull a random sample of records post-load and diff key fields (price, inventory quantity, SKU) against the source of truth, not just spot-check the ones that are easy to eyeball
- Business-rule checks specific to the store: B2B price list assignment, subscription contract continuity, multi-location inventory splits None of this is exotic. It's the same reconciliation discipline used in a data warehouse load, applied to a commerce migration instead. The teams that skip it are the ones that find the discrepancy from a customer support ticket three weeks after go-live.
Cutover: the part where architecture becomes a business decision
Because DMS is running CDC rather than a one-time dump, the technical cutover moment is genuinely small, often measured in minutes rather than hours, since the target has been kept in sync continuously and the source can remain operational until you choose the switchover moment. But "small technical window" and "safe business cutover" aren't the same thing.
A cutover plan that holds up under pressure usually includes:
- a defined freeze window for order-affecting writes on the source system, even if it's brief
- a final CDC catch-up and reconciliation pass immediately before DNS/storefront cutover
- a rollback path that doesn't require re-running the entire migration; usually this means keeping the old platform live and read-only for a defined window, not decommissioning it on day one
- a named owner for the go/no-go decision, separate from the engineer running the migration The architecture buys you a short technical window. It doesn't remove the need for a plan if something is wrong when that window opens.
When you actually need this versus when a migration app is enough
Building DMS, S3, Glue, and Step Functions into a migration is real engineering overhead. It's not the default answer for every store moving to Shopify Plus.
| Signal | Migration app / CSV is probably fine | AWS-backed pipeline is worth it |
|---|---|---|
| Catalog size | Under ~20,000 SKUs, simple variants | 50,000+ SKUs, or complex variant/metafield structure |
| Source system | Standard cart (WooCommerce, BigCommerce, Magento with clean export) | Custom ERP, in-house PIM, or a system with no reliable bulk export |
| Order/inventory activity during migration | Can tolerate a short freeze window | Needs to stay live and transacting throughout the build |
| Data model complexity | Flat product catalog | B2B pricing, multi-location inventory, subscriptions, metaobjects |
| Historical data | Products and current inventory only | Full order history, customer accounts, and audit trail required |
If you're in the left column, a well-run migration app project is faster and cheaper than standing up a DMS pipeline for its own sake. The pipeline earns its cost when the source system itself is the obstacle, not just the volume of rows.
What to test before you commit to an architecture
Before scoping a migration this way, it's worth running a small proof of concept rather than committing the full build:
- Extract a representative slice (a few hundred SKUs with your messiest variant structure, not your cleanest ones) and run it end to end through the pipeline
- Confirm the Bulk Operations import actually preserves the metafield and variant relationships you need. This is the step that most commonly reveals schema-mapping gaps
- Time a DMS CDC catch-up under realistic write volume, not idle volume, to get a real cutover-window estimate
- Decide your rollback trigger conditions in writing before go-live, not during an incident If that proof of concept surfaces more schema-mapping problems than rate-limit problems, that's a useful signal that the bottleneck was never really the API. It was the data model, and no amount of AWS infrastructure fixes a data model problem on its own. That's usually the point where it's worth bringing in a team that's done the Databricks-and-data-engineering side of this as often as the Shopify side, since Lucent Innovation's Shopify migration team works both ends of that pipeline rather than treating the AWS layer as an afterthought.
The short version
CSV imports and migration apps are the right tool until the source system or the catalog size makes them the bottleneck. Past that point, the pipeline that actually holds up looks like a data engineering job: CDC-based extraction so the source stays live, S3 as a staging and audit layer, Glue or Lambda for transformation, Step Functions for sequencing, and Shopify's Bulk Operations API, not a request loop, for the load itself. The architecture doesn't remove the need for reconciliation or a real cutover plan. It just gives you a pipeline where those things are possible to do properly instead of bolted on after the fact.
What's the largest catalog or messiest source system you've had to move onto Shopify Plus, and did the migration tooling hold up?
Top comments (0)