Split a large CSV into smaller files and keep the header in each (Python, no dependencies)

Some tools (older import forms, email services, spreadsheet apps) choke on big CSV files. A common fix is to split the file into smaller parts, with the header row repeated in every part. Here is a small script that does exactly that using only Python's standard library.

Why not just split -l?

The Unix split command cuts by line count. That breaks in two ways:

• The header only ends up in the first file.
• A quoted field that contains a line break can be cut in half.

Python's csv module reads whole records, so multi-line fields stay intact.

The script

import csv
import sys
from pathlib import Path


def split_csv(src, rows_per_file, out_dir="."):
    src = Path(src)
    out_dir = Path(out_dir)
    out_dir.mkdir(parents=True, exist_ok=True)
    written = []
    with src.open(newline="", encoding="utf-8-sig") as f:
        reader = csv.reader(f)
        header = next(reader)
        writer = None
        out = None
        count = 0
        part = 0
        for row in reader:
            if writer is None or count >= rows_per_file:
                if out:
                    out.close()
                part += 1
                path = out_dir / f"{src.stem}_part{part:03d}.csv"
                out = path.open("w", newline="", encoding="utf-8")
                writer = csv.writer(out)
                writer.writerow(header)
                written.append(path)
                count = 0
            writer.writerow(row)
            count += 1
        if out:
            out.close()
    return written


if __name__ == "__main__":
    if len(sys.argv) < 3:
        sys.exit("Usage: python split_csv.py input.csv rows_per_file [out_dir]")
    for p in split_csv(sys.argv[1], int(sys.argv[2]), *(sys.argv[3:4])):
        print(p)

Usage

python split_csv.py products.csv 1000 parts/

This writes products_part001.csv, products_part002.csv and so on into parts/, each with at most 1000 data rows plus the header.

Quick test

I tested it with a 25-row file that included fields containing commas and line breaks, split at 10 rows. Result: three files with 10, 10 and 5 data rows, every file starting with the same header, and 25 rows in total, so nothing was lost or duplicated.

Notes

• utf-8-sig on input strips a BOM if Excel added one; the output is plain UTF-8.
• The file is streamed row by row, so memory use stays small even for large files.
• If your target tool needs a BOM (some Excel versions do), change the output encoding to utf-8-sig.

If you only need the rows count per part, adjust the second argument. That is all there is to it.