DEV Community

SUAD CUMA
SUAD CUMA

Posted on Edited on

Duplicate CSV rows are not the same as duplicate product SKUs

#python #csv #tutorial #ecommerce

Duplicate CSV rows are not the same as duplicate product SKUs

Suppose a product export contains these rows:

Type,SKU,Name,Regular price
simple,TSHIRT-S,Shirt S,19.00
simple,TSHIRT-S,Shirt S,19.00
simple,MUG-1,Ceramic Mug,12.00
simple,MUG-1,Ceramic Mug updated,12.00
Enter fullscreen mode Exit fullscreen mode

There are two different problems:

  1. Rows 1 and 2 are identical after CSV parsing.
  2. Rows 3 and 4 are different rows but share the same non-empty SKU.

A generic row deduplication tool may catch the first issue while missing the second. Before a bulk import, it helps to check both.

A small standard-library Python check 

import csv
from collections import Counter
from io import StringIO

source = """Type,SKU,Name,Regular price
simple,TSHIRT-S,Shirt S,19.00
simple,TSHIRT-S,Shirt S,19.00
simple,MUG-1,Ceramic Mug,12.00
simple,MUG-1,Ceramic Mug updated,12.00
"""

reader = csv.DictReader(StringIO(source))
rows = list(reader)
header = reader.fieldnames or []
exact_keys = [tuple(row.get(name) for name in header) for row in rows]
sku_keys = [row["SKU"].strip() for row in rows if row.get("SKU", "").strip()]

print("Duplicate row occurrences:", sum(v - 1 for v in Counter(exact_keys).values() if v > 1))
print("Duplicate SKU values:", sorted(k for k, v in Counter(sku_keys).items() if v > 1))
Enter fullscreen mode Exit fullscreen mode

Expected output:

Duplicate row occurrences: 1
Duplicate SKU values: ['MUG-1', 'TSHIRT-S']
Enter fullscreen mode Exit fullscreen mode

The first number counts additional repeated rows beyond the first instance. The SKU list represents duplicate identifiers, not the number of duplicates.

Import hygiene matters more than the counter

Before applying an automatic fix, consider whether identifiers are intended to be unique in your target workflow. For variations, check the expected parent relationships and the WooCommerce import schema separately. Preserve a copy of the source and review the import preview before writing to production.

This example deliberately uses fabricated data and does not access anyone's store.

A free, browser-local CSV audit is available for experimenting with a sample before importing. It does not replace testing business-specific SKU rules.

Optional next step: a reproducible local calculator

The Python snippet above is free and sufficient for reproducing the example. For repeatable checks on your own files, USA-1 also offers a separate Duplicate Row Rate Checker ZIP for 49 Telegram Stars as an optional one-time digital download. The ZIP is a local Python calculator, not a WooCommerce plugin and not a replacement for validating SKU uniqueness, parent/variation mapping, or a staging import. The product page describes the files; Telegram shows the charge before any payment. No purchase is required to use the free example or browser-local audit.

Disclosure: This tutorial was prepared with AI assistance. The Python example was run against the synthetic data shown above, and its output was verified before publication.

Top comments (0)