Check a CSV's missing-data contract before drawing a line

#python

A parser can read every row successfully and still pass the wrong meaning to a plotting step. Consider six time records with response values 2, 3, NA, 0, 4, 5. The third value is missing. The fourth is an observed zero. Replacing one with the other changes the scientific meaning; deleting the missing record can also change the continuity of a line plot.

This is a self-written synthetic teaching example, not a dataset from a study, a customer result, or the output of an installed research Skill. We are the RigorMed team. The calculation below was actually run locally with Python's standard library; it does not use a model, paid service, or production research task.

Freeze what the file means

For this file only, the schema is time_h,response_au. Time is in hours; response is in arbitrary units. The exact token NA means missing. Numeric zero means observed zero. Rows are in strictly increasing time order. They describe one sequence, so the number of rows is not automatically an independent sample size.

Those rules are declared inputs, not guesses learned by the parser. If a source uses -999, an empty field, or another code, confirm its meaning and change the contract explicitly. Do not extend this example into a universal CSV cleaner.

Inspect without rewriting the source

The code retains each original token alongside a parsed value. Missing values become None in the inspection structure; the source CSV remains unchanged. It also rejects non-finite numbers and time reversals under this particular declared schema.

import csv
import io
import math

raw = """time_h,response_au
0,2
1,3
2,NA
3,0
4,4
5,5
"""
reader = csv.DictReader(io.StringIO(raw))
assert reader.fieldnames == ["time_h", "response_au"]
source_rows = list(reader)
checked = []
previous_time = None
for row in source_rows:
    t = float(row["time_h"])
    assert math.isfinite(t)
    assert previous_time is None or t > previous_time
    previous_time = t
    token = row["response_au"]
    value = None if token == "NA" else float(token)
    assert value is None or math.isfinite(value)
    checked.append((t, token, value))

missing_times = [t for t, token, value in checked if value is None]
observed = [value for t, token, value in checked if value is not None]
zero_times = [t for t, token, value in checked if value == 0]
assert len(source_rows) == 6
assert missing_times == [2.0]
assert zero_times == [3.0]
print({"records": len(checked), "observed": len(observed),
       "missing_times_h": missing_times, "zero_times_h": zero_times})

Actual local output:

{'records': 6, 'observed': 5, 'missing_times_h': [2.0], 'zero_times_h': [3.0]}

There are six records and five observed responses. Missingness occurs at hour two; the observed zero occurs at hour three. No row was imputed or deleted. Python assertions here are deliberately small teaching checks; a production validator should report explicit structured errors and must not rely on assertions that optimized execution can disable.

Pass the contract to the plotting step

Keep all six time coordinates. In the separate minimum Matplotlib teaching example used for today's video, the missing response is represented by a plotting NaN, so the line breaks at hour two. The zero stays on the response axis at hour three. A missingness marker sits in a separate strip; it must not masquerade as a measured response.

This behavior agrees with Matplotlib's documented masked-and-NaN example: removing a point can connect the remaining line, while masked or NaN values can leave a gap. A visible gap is a display choice bound to the data; it does not explain why the measurement is missing or establish that missingness is ignorable.

The handoff should include the original table, variable definitions, units, missing-code rule, exact plotting code and caption. Ask a reviewer to trace the count of records, the missing position and the observed zero back to the source. Do not add confidence intervals without an estimand, an appropriate sampling structure and an uncertainty method.

Where the two recommended Skills fit

The K-Dense-AI public exploratory-data-analysis source provides an inspection-oriented task, while scientific-visualization addresses figure production and export. Both source files reviewed on 2026-11-04 identify version 1.4. The tabular core requires Python 3.11 or later; other formats and plotting/export dependencies need checking for the actual task.

Neither recommended Skill was installed or run for this article. Our CSV check and minimum plot are separate local teaching work, not evidence that either complete Skill installation succeeds, and not a RigorMed integration. Source review, an individual Python calculation, a figure export and a complete workflow acceptance test are different states.

For an authorized research-material task, you can describe the materials and required reviewable output in the RigorMed browser workspace. This article does not claim the two third-party Skills are available there; actual model use follows the displayed credit terms.

Sources and scope

Python standard library, csv and math: the executable example above uses DictReader, isfinite and ordinary collection operations.
Matplotlib, Plotting masked and NaN values, source reviewed with the day's plotting package.
K-Dense-AI exploratory-data-analysis source and scientific-visualization source, reviewed on 2026-11-04; version statements refer to those files.

The example does not estimate a treatment effect, validate a measurement method, perform inference, or recommend a missing-data analysis strategy. It checks a declared material contract before drawing a figure.

DEV Community