Two records use the same product code, but that does not make the records duplicates. This read-only checker groups repeated values in one CSV column and reports their positions. It keeps leading zeros and spaces, reports empty cells separately and never deletes a row.
1. Make a practice export
CSV means comma-separated values. An identifier names something rather than an amount to calculate. Save this as MakeExamples.py in a practice folder:
from pathlib import Path
Path("Codes.csv").write_bytes(
b'Code,Note\n'
b'00123,first record\n'
b'00123,second record\n'
b'123,different text\n'
b' 00123,leading space\n'
b',empty code\n'
)
Run:
python3 MakeExamples.py
Use your installation's Python 3 command if it is not named python3. This creates or replaces Codes.csv, so use a folder without a file you need to keep.
The first two data records share 00123 but have different notes. The third uses 123, the fourth starts its code with a space, and the last has an empty code. The saved bytes preserve those differences without a spreadsheet converting the codes to numbers.
2. Add the reviewer
Save this as FindRepeatedCodes.py:
import csv as Csv
import io as Io
import json as Json
import sys as Sys
from pathlib import Path
def FindRepeatedCodes(FilePath, Column):
with Path(FilePath).open("rb") as Input:
Data = Input.read(1048577)
if len(Data) > 1048576:
raise ValueError("input exceeds 1 MiB teaching limit")
Csv.field_size_limit(65536)
Reader = Csv.reader(Io.StringIO(Data.decode("utf-8-sig"), newline=""), strict=True)
try:
Header = next(Reader)
except StopIteration:
raise ValueError("empty input") from None
if not Header or any(not Name.strip() for Name in Header):
raise ValueError("blank header name")
if len(Header) != len(set(Header)):
raise ValueError("repeated header name")
if Column not in Header:
raise ValueError("selected column not found")
Index = Header.index(Column)
Groups = {}
Empty = []
Records = 0
for Records, Fields in enumerate(Reader, start=1):
if Records > 1000:
raise ValueError("more than 1000 data records")
if len(Fields) != len(Header):
raise ValueError(f"record {Records} has wrong field count")
Position = {"Record": Records, "EndingLine": Reader.line_num}
Value = Fields[Index]
if Value == "":
Empty.append(Position)
else:
Groups.setdefault(Value, []).append(Position)
Repeated = [Positions for Positions in Groups.values() if len(Positions) > 1]
return {"Records": Records, "Column": Index + 1,
"RepeatedGroups": Repeated, "EmptyCells": Empty}
def Main():
if len(Sys.argv) != 3:
print("Usage: python3 FindRepeatedCodes.py input.csv COLUMN", file=Sys.stderr)
return 2
try:
Report = FindRepeatedCodes(Sys.argv[1], Sys.argv[2])
except (OSError, ValueError, Csv.Error) as Problem:
print(f"Review stopped: {Problem}", file=Sys.stderr)
return 2
print(Json.dumps(Report, indent=2))
return 1 if Report["RepeatedGroups"] or Report["EmptyCells"] else 0
if __name__ == "__main__":
Sys.exit(Main())
The program uses Python's default comma-separated CSV dialect and accepts UTF-8 with or without an initial signature. Quotes can keep a comma or line break inside a field. strict=True stops on parser errors it detects, but it does not recognise every other CSV dialect or validate all possible quoting rules.
The header must contain nonblank, unique names. Your selected column must match exactly: Code and code are different. Every data record must have the same number of fields as the header. A wrong field count or parser error stops the whole review, with no partial JSON report.
The teaching limits are 1 MiB, roughly one million bytes, 1,000 data records and 65,536 characters per parsed field. The file read includes one extra byte to detect oversized input before decoding. Use a saved file that will not change during the read.
Groups is a dictionary that maps each original parsed value to its positions. Nothing is trimmed, lowercased or converted to a number. An exactly empty string goes into EmptyCells rather than a repeated group. A spaces-only value is not empty under this rule and can form a repeated group.
3. Read the repeated group
Run:
python3 FindRepeatedCodes.py Codes.csv Code
The output is:
{
"Records": 5,
"Column": 1,
"RepeatedGroups": [
[
{
"Record": 1,
"EndingLine": 2
},
{
"Record": 2,
"EndingLine": 3
}
]
],
"EmptyCells": [
{
"Record": 5,
"EndingLine": 6
}
]
}
Records 1 and 2 share the same parsed code. They appear together in one group, even though their notes differ. Record 3's 123 and record 4's space-prefixed code are different strings from 00123, so neither joins that group. Record 5 appears only in EmptyCells.
The output uses JSON, a text format for named values and lists. It omits codes, notes and header names. Positions and counts can still reveal information, so keep the report private when it describes work or personal data.
Record starts at the first data record, excluding the header. Column starts at one. EndingLine is the last physical line consumed for the record; a quoted multiline field can make it larger than the record number plus one. Groups follow the first appearance of each value, and positions within each group follow file order.
The exit code, a small result number another script can check, is 1 when repeated groups or empty cells need review, 0 when neither is found and 2 for a command, file, decoding, parsing or limit error. A header-only file returns 0 with zero records; an empty file stops. No finding is not proof that the export is complete or suitable for import.
4. Choose the column deliberately
Run the same file with a different column:
python3 FindRepeatedCodes.py Codes.csv Note
The output is:
{
"Records": 5,
"Column": 2,
"RepeatedGroups": [],
"EmptyCells": []
}
The exit code is 0: the five notes are nonempty and different. The program checks the column you name, not which one ought to identify a record. A clean report for Note cannot stand in for checking Code.
Now try lowercase code in the command. It stops with selected column not found and exit code 2, rather than guessing a nearby header. A header containing spaces needs quoting in your command window.
5. Compare quoted values and physical lines
Try the rules together in a second file. Save this as MakeQuoted.py in the same practice folder:
from pathlib import Path
Name = Path("Quoted.csv")
with Name.open("xb") as Output:
Output.write(
b'Code,Note\n'
b'00123,plain field\n'
b'"00123","two physical\nlines"\n'
b'" ",one space\n'
b' ,same space without quotes\n'
b',empty field\n'
b'"",quoted empty field\n'
)
Run:
python3 MakeQuoted.py
python3 FindRepeatedCodes.py Quoted.csv Code
The setup opens Quoted.csv in exclusive binary mode, so it stops if that name already exists rather than replacing the file. Use a fresh practice folder if you want to repeat it. The checker reports:
{
"Records": 6,
"Column": 1,
"RepeatedGroups": [
[
{
"Record": 1,
"EndingLine": 2
},
{
"Record": 2,
"EndingLine": 4
}
],
[
{
"Record": 3,
"EndingLine": 5
},
{
"Record": 4,
"EndingLine": 6
}
]
],
"EmptyCells": [
{
"Record": 5,
"EndingLine": 7
},
{
"Record": 6,
"EndingLine": 8
}
]
}
The exit code is 1. Records 1 and 2 share 00123: quote marks belong to the CSV syntax, not the parsed code. Record 2's note spans two physical lines, so that record ends on line 4. The next record ends on line 5, not line 4.
Records 3 and 4 each contain exactly one space in the selected column. Quoting that space does not change it. They form a second repeated group because the checker does not trim values. Records 5 and 6 both parse to an empty string, with and without quotes, so they appear separately in EmptyCells, not as a repeated group.
Only the selected Code fields decide the groups. The notes explain the fixture but do not take part in the comparison. Nothing in either practice file is changed by the checker.
6. Do not turn a repeat into a deletion rule
The same identifier can occur on several legitimate rows: a product can appear on multiple order lines, or an account can have multiple entries. This check compares one column, not all fields, event identifiers or business meaning. Two empty cells are reported as separate empty positions, not evidence of the same entity.
Decide from the receiving system's rules whether the selected identifier must be unique. If matching should ignore case or surrounding spaces, review that rule separately before comparing transformed values. Do not silently turn 00123 into 123 or remove spaces just to make groups line up.
The parsed value is different from its CSV spelling. Quotes around a field are syntax, so a quoted "00123" and an unquoted 00123 join the same group. This is not a byte-for-byte row comparison or a general duplicate-record detector.
Keep the original export and inspect the reported records before changing anything.
Top comments (1)
Official Platform Update
Security protocols have been updated for all developer accounts.