pandas chunks fixed my memory error, then quietly broke my averages

Checked against official sources on October 30, 2026

Here is a bug that does not crash. It just gives you a wrong number and lets you ship it.

You have a CSV that is too big for pandas. You switch to chunksize. The memory error goes away, the script finishes, and you move on. But if one ID's rows sit on both sides of a chunk edge, some per-ID results are now wrong. Nothing tells you.

We built a test file, broke it on purpose, and fixed it two ways. Every output block below was pasted from our own runs with pandas 3.0.6 and Python 3.13.5.

Where this came from

A Python learner asked about it on Reddit. Their file was too large for a 16 GB machine. They needed stats for each identifier, such as a row count per ID. They saw the trap right away: with chunks, one ID can be split across two reads. Even loading only the ID column was too much memory for them.

So there are two jobs here. Make it fit in memory. Then make sure the answer is still right. The first job is easy. The second one is where people get hurt.

The test file

We made fake patient vital signs, sorted by patient ID. No real patient data was used.

• 4,000,000 rows in total (the file is 172 MB)
• 50,000 patients
• Five columns: patient_id, ts, heart_rate, spo2 and note

Each method ran in a fresh Python process, so the "peak RSS" number is clean. Peak RSS is the most memory the process used at any moment. Python's resource.getrusage gave us that number.

Chunks were 500,000 rows each. That gives 8 chunks and 7 edges, and all 7 edges landed in the middle of a patient.

First, read less

Before you reach for chunks, shrink what you load. The pandas docs page "Scaling to large datasets" makes two points that matter here. Load only the columns you need. Use smaller data types.

One handy type is category. It stores each distinct text value once and keeps a small number per row. For a column like patient ID, that saves a lot.

def full_load():
    df = pd.read_csv(PATH)
    g = df.groupby("patient_id")["heart_rate"]
    return pd.DataFrame({"rows": g.size(), "mean_hr": g.mean()})

def full_load_slim():
    df = pd.read_csv(PATH, usecols=["patient_id", "heart_rate"],
                     dtype={"patient_id": "category", "heart_rate": "int16"})
    g = df.groupby("patient_id", observed=True)["heart_rate"]
    return pd.DataFrame({"rows": g.size(), "mean_hr": g.mean()})

full_load       peak RSS     614 MB   time   4.1s   patients 50000
full_load_slim  peak RSS     178 MB   time   0.9s   patients 50000

Both gave the same answer. The slim load used under a third of the memory. If your file is only a bit too big, stop here. You may not need chunks at all.

The chunk code most of us write first

Group each chunk. Stack the small results. Group again.

def chunks_wrong():
    parts = []
    for chunk in pd.read_csv(PATH, usecols=["patient_id", "heart_rate"],
                             chunksize=500_000):
        g = chunk.groupby("patient_id")["heart_rate"]
        parts.append(pd.DataFrame({"rows": g.size(), "mean_hr": g.mean()}))
    out = pd.concat(parts)
    return out.groupby(level=0).agg(rows=("rows", "sum"),
                                    mean_hr=("mean_hr", "mean"))

It is fast and small:

chunks_wrong    peak RSS     123 MB   time   1.1s   patients 50000

Then we checked it against the full load:

chunks_wrong    rows match: True   patients with wrong mean: 7   worst error: 2.827

Row counts were fine. A count from one chunk plus a count from the next is the true count.

The means were not fine. Seven patients were wrong, which is one per chunk edge. The code took an average of two averages. That gives a small piece of data the same weight as a big piece.

A quick example. One patient has 40 readings with an average of 80 in chunk one. The same patient has 2 readings with an average of 120 in chunk two. The average of 80 and 120 is 100. The true average of all 42 readings is about 82.

Seven bad rows out of 50,000 is the scary part. Random spot checks will almost never land on them.

Fix one: carry sums and counts

Only carry numbers that you can safely add. For an average, that means the sum and the count. Add them up across all chunks. Divide once at the very end.

def chunks_right():
    total = None
    for chunk in pd.read_csv(PATH, usecols=["patient_id", "heart_rate"],
                             dtype={"heart_rate": "int16"}, chunksize=500_000):
        part = chunk.groupby("patient_id")["heart_rate"].agg(["size", "sum"])
        total = part if total is None else total.add(part, fill_value=0)
    return pd.DataFrame({"rows": total["size"].astype("int64"),
                         "mean_hr": total["sum"] / total["size"]})

total.add(part, fill_value=0) matches rows by patient ID. If a patient is missing from one side, that side counts as 0. The same pandas docs page uses this add with fill_value=0 pattern to count values across many files.

chunks_right    peak RSS     107 MB   time   1.1s   patients 50000
chunks_right    rows match: True   patients with wrong mean: 0   worst error: 0.000

Every patient was right, and this was the lowest memory we measured. The running total holds one row per patient, so it does not grow with the length of the file.

The same trick works for more stats. Min and max combine safely. For variance, carry three things: the count, the sum, and the sum of squares.

Fix two: medians need the rows

Some results cannot be built from parts you add up. The median is one. "The last reading per patient" is another. For these, you need all of an ID's rows in one place.

Our file is sorted by ID, and that makes it doable. At the end of each chunk, only the last ID might spill into the next chunk. So hold back that ID's rows and stick them on the front of the next chunk. Every other ID in the chunk is complete, so finish it now.

def median_carry():
    results, carry = [], None
    for chunk in pd.read_csv(PATH, chunksize=500_000, **COLS):
        if carry is not None:
            chunk = pd.concat([carry, chunk], ignore_index=True)
        last_id = chunk["patient_id"].iloc[-1]
        is_last = chunk["patient_id"] == last_id
        carry = chunk[is_last]          # may continue in the next chunk
        done = chunk[~is_last]
        results.append(done.groupby("patient_id")["heart_rate"].median())
    results.append(carry.groupby("patient_id")["heart_rate"].median())
    return pd.concat(results)

median_full   peak RSS   233 MB   time  1.5s   patients 50000   duplicate ids 0
median_carry  peak RSS   123 MB   time  1.4s   patients 50000   duplicate ids 0
medians identical: True

Same medians as the full load, about half the memory, and no patient counted twice.

Two warnings before you copy this:

• It only works if the file is truly sorted by ID. Check that first. If it is not, sort it once with a tool made for big files.
• If one ID has more rows than a whole chunk, the carried rows keep growing until that ID ends. Only a giant ID makes that a problem.

When pandas is the wrong tool

The pandas docs say chunking works best when the chunks need little or no coordination. They also say groupby is much harder to do in chunks. If your job needs joins, lots of group stats, or distinct counts, we find two other routes are often simpler:

• Load it into a database once. SQLite ships with Python. Write each chunk into a table, add an index on the ID, then use plain SQL.
• Use a library that streams for you. Some DataFrame libraries can scan a CSV lazily and handle the chunk work inside.

Before you trust a chunked number

1. Load only the columns you need. Try category and small int types.
2. Measure peak memory, one method per process.
3. Use chunks only if you still have to.
4. Carry values that add up safely: counts, sums, min and max.
5. Never average averages. Keep the sum and the count, and divide at the end.
6. For medians or "last value", sort by ID and carry the edge rows forward.
7. Run the chunked version and a full load on a small sample, and compare them.

Who wrote this

TalentOp Systems LLC is an IT staffing and IT software consulting company in Omaha, Nebraska, and we have been working since February 2025. We place data and software people on projects for clients in banking, insurance, healthcare and retail, where a number that is quietly wrong can cost real money. We hire across the US on W2 and C2C, often for 12+ month contracts.

TalentOp Systems is a direct vendor for Kroger, whose offices are in Blue Ash, Ohio.
We are an E-Verified employer.
Need H-1B sponsorship? TalentOp Systems LLC provides it for hires who meet the role's requirements and eligibility.
People on OPT or STEM OPT can apply too.

See our open roles at www.talentopsystems.com, or write to [e-mail].