DEV Community

Magevanta
Magevanta

Posted on • Originally published at magevanta.com

Magento 2 Product Import & Export Optimization: Speed Up Large Catalog Operations

#magento #php #performance #ecommerce

If you've ever kicked off a large product import in Magento 2 and watched the progress bar crawl, you know the pain. A 50,000-product CSV that should take minutes ends up running for hours. Exports grind the database to a halt. Scheduled imports time out. It doesn't have to be this way.

In this guide we'll cover every meaningful lever you can pull to make Magento 2 imports and exports dramatically faster — from server-side tuning to code-level tricks.

Why Magento 2 Imports Are Slow

Before optimizing, it's worth understanding why imports are slow by default:

  • EAV overhead: Every product attribute write touches catalog_product_entity_* tables multiple times.
  • Reindex on import: By default, Magento triggers reindexing after each batch, not after the full import.
  • URL rewrite generation: Each imported product generates URL rewrites — one of the heaviest operations.
  • Image processing: Magento copies and resizes images synchronously during import.
  • Event observers: Third-party modules often hook into catalog_product_save_after and fire on every imported row.
  • Database locking: Large inserts cause table-level locks that block other queries.

Now let's fix all of that.

1. Use the Built-in Import UI Correctly

The Admin import UI (System > Data Transfer > Import) is fine for occasional imports but has limitations:

  • Upload limit: PHP's upload_max_filesize and post_max_size must be increased for large files.
  • Timeout: Apache/Nginx timeouts kill long-running imports. Set max_execution_time = 0 in php.ini for CLI, and configure Nginx timeouts for web uploads: 
fastcgi_read_timeout 600;
proxy_read_timeout 600;
Enter fullscreen mode Exit fullscreen mode
  • Behavior setting: Always use Add/Update unless you intend to delete. Replace is much slower because it deletes and re-inserts.

For anything above 10,000 products, use the CLI.

2. Run Imports via CLI

The CLI import avoids PHP-FPM timeouts entirely:

bin/magento import:run \
  --entity=catalog_product \
  --behavior=add_update \
  --validation-strategy=validation-stop-on-errors \
  --allowed-error-count=10 \
  --separator=',' \
  --multiple-value-separator=',' \
  --fields-enclosure='"' \
  /path/to/products.csv
Enter fullscreen mode Exit fullscreen mode

This runs in the CLI PHP context where you can set unlimited execution time.

3. Disable Reindexing During Import

This is the single biggest optimization. Switch all indexes to manual update mode before importing:

bin/magento indexer:set-mode schedule catalog_product_flat
bin/magento indexer:set-mode schedule catalog_product_price
bin/magento indexer:set-mode schedule catalog_search
bin/magento indexer:set-mode schedule catalogrule_product
Enter fullscreen mode Exit fullscreen mode

After the import completes, reindex everything at once and switch back:

bin/magento indexer:reindex
bin/magento indexer:set-mode realtime catalog_product_flat
# ... etc
Enter fullscreen mode Exit fullscreen mode

Reindexing once after a 50,000-product import is orders of magnitude faster than reindexing after every batch of 100.

4. Optimize MySQL for Bulk Imports

By default, MySQL is tuned for transactional workloads, not bulk inserts. Temporarily adjust these settings before a large import:

SET GLOBAL innodb_flush_log_at_trx_commit = 2;
SET GLOBAL sync_binlog = 0;
SET GLOBAL innodb_autoinc_lock_mode = 2;
Enter fullscreen mode Exit fullscreen mode

Add to my.cnf for a more permanent performance gain:

innodb_buffer_pool_size = 4G          # 70-80% of available RAM
innodb_log_file_size = 512M
innodb_flush_method = O_DIRECT
innodb_io_capacity = 2000
innodb_io_capacity_max = 4000
bulk_insert_buffer_size = 64M
Enter fullscreen mode Exit fullscreen mode

Important: Reset innodb_flush_log_at_trx_commit = 1 in production for durability. Only relax it during controlled import windows.

5. Skip URL Rewrites When Possible

URL rewrite generation is one of the most expensive parts of import. If you're doing a backend-only import (products not yet live, or you're updating prices/stock only), disable it:

In app/etc/env.php, temporarily set:

'catalog' => [
    'frontend' => [
        'flat_catalog_product' => 0,
    ],
],
Enter fullscreen mode Exit fullscreen mode

Or better — import only the columns you need. A CSV with just sku, price, and qty skips URL rewrite generation entirely because Magento only regenerates rewrites when URL-key fields change.

Pro tip: Split your import file into two CSVs:

  1. products_core.csv — sku, name, description, attributes (triggers URL rewrites)
  2. products_stock_price.csv — sku, price, qty only (no URL rewrites)

Import stock/price updates separately and save significant time.

6. Batch Size Tuning

Magento imports in batches. The default batch size is typically 100–200 rows. Increasing this reduces the number of database round-trips:

In vendor/magento/module-import-export/Model/Import.php, the constant DEFAULT_SIZE controls this — but modifying core is bad practice. Instead, create a plugin:

// app/code/Vendor/ImportOptimizer/Plugin/ImportPlugin.php

public function afterGetBunchSize(\Magento\ImportExport\Model\Import $subject, int $result): int
{
    return 500; // Increase from default ~100
}
Enter fullscreen mode Exit fullscreen mode

Test with 500–1000 depending on your row width and available memory. Larger batches reduce overhead but increase memory per batch.

7. Disable Non-Essential Observers During Import

Third-party modules frequently attach observers to product save events. During a CLI import, you almost certainly don't need newsletter triggers, ERP sync hooks, or marketing tag updates firing 50,000 times.

Create a custom CLI command that wraps the import and temporarily disables specific observers, or use Magento's areas configuration to exclude them from crontab area context.

At minimum, identify your heaviest observers:

bin/magento dev:events:list | grep catalog_product_save
Enter fullscreen mode Exit fullscreen mode

Then check the module list and disable non-critical ones during import windows.

8. Image Import Optimization

If your CSV includes images, Magento processes them synchronously — download (if URL), copy to pub/media/catalog/product, and add to the queue for resizing.

Best practice: Import product data first (without images), then import images separately in a second pass. This allows you to:

  • Use a dedicated import window for image processing
  • Run image resize asynchronously via queue consumers
  • Retry failed image downloads without re-importing all product data

Enable async image resizing:

bin/magento config:set catalog/product/flat_catalog_product 0
bin/magento config:set system/upload_configuration/enable_resize_file 0
Enter fullscreen mode Exit fullscreen mode

Then process the image resize queue separately:

bin/magento queue:consumers:start media.storage.catalog.image.resize
Enter fullscreen mode Exit fullscreen mode

9. Export Optimization

Exports in Magento 2 are often even slower than imports because they join across many EAV tables.

Enable flat catalog before large exports:

bin/magento config:set catalog/frontend/flat_catalog_product 1
bin/magento indexer:reindex catalog_product_flat
Enter fullscreen mode Exit fullscreen mode

With flat catalog enabled, exports read from a single denormalized table instead of joining dozens of EAV tables.

Use direct database queries for custom exports: If you need a non-standard export (e.g., just SKU + price + stock for an ERP), writing a custom script that queries catalog_product_entity + cataloginventory_stock_item directly will always outperform the Magento export framework.

10. Monitor and Benchmark

Before and after your optimizations, measure:

# Time a full import
time bin/magento import:run --entity=catalog_product ...

# Watch MySQL in real-time during import
mysqladmin -u root -p extended-status -r -i 1 | grep -E 'Queries|Threads'

# Check slow query log
SET GLOBAL slow_query_log = 'ON';
SET GLOBAL long_query_time = 1;
Enter fullscreen mode Exit fullscreen mode

The combination of schedule-mode indexing, MySQL tuning, and disabled URL rewrites on price-only imports typically yields 5–10x speedups on large catalogs.

Putting It All Together: The Import Checklist

Before running a large import:

  • [ ] Switch relevant indexers to schedule mode
  • [ ] Disable non-essential observers
  • [ ] Tune MySQL buffer pool and log settings
  • [ ] Split CSV: content vs. price/stock
  • [ ] Set max_execution_time = 0 in CLI php.ini
  • [ ] Use CLI import, not Admin UI
  • [ ] Disable image processing if possible

After import:

  • [ ] Run bin/magento indexer:reindex
  • [ ] Run bin/magento cache:flush
  • [ ] Switch indexers back to realtime if desired
  • [ ] Process image resize queue
  • [ ] Verify product visibility in frontend

Conclusion

Magento 2 import performance is a layered problem — no single fix solves everything. The biggest wins come from schedule-mode indexing (eliminates mid-import reindex), MySQL buffer tuning (faster bulk inserts), and splitting CSV files by operation type (avoids unnecessary URL rewrite generation).

Once you've implemented these optimizations, a 50,000-product import that took 3 hours should complete in 15–30 minutes. That's the kind of improvement that makes a real difference in release windows, data migrations, and daily sync pipelines.

Have a specific import bottleneck you're hitting? The techniques above cover the vast majority of cases — but profiling with EXPLAIN on the slow queries during import is always a good next step.

Top comments (0)