DEV Community: Hyunyong Park

I Built a CSV Cleaner That Explains Every Deleted Row

Hyunyong Park — Thu, 30 Jul 2026 21:41:18 +0000

Deleting a duplicate row is easy. Explaining why the output has fewer rows is
the harder product problem.

I ran into that distinction while packaging a small Windows CSV cleaner. A
typical cleanup script can trim whitespace, drop empty rows, and remove
duplicates in a few lines. But when the input has four rows and the output has
two, the person using it still needs answers:

Was the original file changed?
Which rule removed each row?
Did every file finish?
Can I inspect the result without rerunning the job?

That changed the design. The audit report became a first-class output rather
than an optional log written after the "real" work.

Start with a reconciliation rule

The most useful invariant is simple:

input rows = output rows + blank rows removed + duplicate rows removed

The header is tracked separately. A duplicate can mean either an exact repeated
row or a repeated value in one or more key columns chosen by the operator.

If that equation does not balance, the run should not quietly display
"success." It has lost the ability to explain the output.

For each CSV, the cleaner records the input count, output count, blank-row
count, duplicate count, failure state, detected encoding, and original file
hash. A run summary then aggregates success and failure across the batch.

This is intentionally narrower than a general data-cleaning platform. It does
not guess what a broken address should be, rewrite phone numbers, edit Excel
workbooks, or use AI to infer missing values. The rules are deterministic so
the evidence can be checked.

Never make the original the rollback plan

An undo button is not a substitute for leaving the source file alone.

The program creates a new run directory and writes cleaned files there. The
input CSV remains untouched. Before processing, it calculates the original
SHA-256 and includes that value in the report.

That hash is not proof that the content is correct. It answers a narrower,
useful question: "Is this the same input file I intended to clean?"

The output is written as UTF-8 with a BOM because it opens predictably in common
Windows spreadsheet tools. The input reader accepts UTF-8, UTF-8 with BOM, and
CP949. Unsupported formats fail explicitly instead of being guessed into
plausible-looking text.

Treat the report as a product output

The run folder contains three different kinds of evidence:

report.html for a person reviewing the run;
summary.json for another program reading aggregate results;
run_log.jsonl for chronological success and failure events.

The HTML report is the first thing I open after a run. It shows which file was
processed, the input and output counts, the removal reasons, the encoding, and
whether the file failed. The JSON files exist for automation, but the human
report keeps verification from requiring Python or a log viewer.

Open the actual fixed-sample HTML report.

That public report uses bundled test data. It is safe to inspect, but it is not
a claim that unrelated CSV files will produce the same numbers.

Isolate failures in a batch

Batch processing creates another easy failure mode: one bad file can hide the
results of all the good files.

The cleaner processes each file independently. If one file cannot be decoded,
has an invalid duplicate-key column, or exceeds a product limit, that failure is
recorded for the file while eligible files continue. The final summary reports
both counts.

This matters because "the window stayed open" and "every file succeeded" are
different facts. A useful batch result has to make partial success visible.

I also avoided writing a cleaned file before the input contract is accepted.
For example, the free edition checks its row limit before writing an output.
Rejecting an oversized file after creating a partial result would make the
folder harder to interpret.

Test the packaged program, not only the source tree

A passing development folder does not prove that the downloadable ZIP contains
the right executable, samples, documentation, and runtime files.

For the public sales build, I extracted the ZIP into a fresh directory and ran
the packaged Windows executable. The release contract passed 32/32 automated
tests, including UTF-8, UTF-8 BOM, CP949, exact duplicates, key-column
duplicates, empty rows, original preservation, failure isolation, and the free
edition's 101-row rejection path.

I then ran one fixed local sample through the packaged product:

1 CSV succeeded and 0 failed;
4 input rows became 2 output rows;
1 blank row was removed;
1 duplicate was removed using order_id;
the original stayed untouched;
the HTML report reconciled the counts.

The 51-second clip starts from a fresh ready state, selects the bundled sample,
runs the cleaner, and opens the generated report. It demonstrates that one
reproducible path; it is not a benchmark or a promise about every CSV.

Try the same workflow for free

I published a free Lite edition so the first test does not require a purchase:

Download CSV Audit Cleaner Lite from the public GitHub release.

It processes up to 100 data rows per CSV and uses the same cleanup and evidence
workflow. The ZIP includes the Windows executable, a sample CSV, quick-start
instructions, license, and checksum. It does not require installation, Python,
a cloud account, a subscription, or a paid API.

The README publishes the release size and SHA-256 so the download can be checked
before use. As with any data tool, keep backups and review the report before
putting cleaned files into a production workflow.

What the paid edition changes

The full edition removes the per-CSV row limit and includes the complete source
code and a commercial-use license. It is a finished download, not custom
development or a remote setup service.

The full product page
offers Personal for $15 one time and Team for $49 one time. It does not
include custom cleaning rules, Excel editing, ongoing support, or a future-update
commitment.

There are no customer reviews or market-validation claims yet. The evidence I
can publish today is narrower: the packaged program, the fixed sample, the
report, the downloadable Lite edition, and the test contract.

That is also the main lesson from building it: a cleanup tool becomes easier to
trust when deletion is not the final output. The result needs an explanation
that survives after the window closes.

Retry Is Not Resume: Building a Web-to-CSV Pipeline That Survives Interruptions

Hyunyong Park — Thu, 30 Jul 2026 04:45:48 +0000

The first run is the easy part.

The real test starts after record 70 of 100 succeeds, the process stops, and you
run it again. If the pipeline fetches the first 70 pages again, appends duplicate
rows, or trusts a half-written state file, it is not resumable. It is only
restartable.

While building a small Python + Playwright source kit, I separated retry from
resume and gave the output data, progress state, and run evidence different
responsibilities. This post walks through the decisions that made the biggest
difference.

Retry and resume solve different failures

A retry handles a temporary failure inside one run:

a request times out;
a page returns incomplete markup;
a required field is temporarily missing.

Resume handles a different event: the run itself ends and a later process must
continue from durable evidence.

Putting both concerns inside one retry loop hides the distinction. Increasing
the retry count does nothing after a terminal closes or a machine restarts.
Instead, I used an item-level retry loop and wrote a completed-record map only
after parsing succeeded.

The essential flow is small:

completed = read_state(config_fingerprint)

for item in input_rows:
    if item.record_id in completed:
        log_skip(item.record_id, reason="completed")
        continue

    parsed = retry(lambda: parse(fetch(item.url)), max_retries=2)
    write_result(item.record_id, item.url, parsed)
    completed[item.record_id] = hash_output(parsed)

write_state_atomically(completed, config_fingerprint)

Retry decides whether the current item gets another attempt. Resume decides
whether the item should be fetched at all in a future run.

The CSV is not the progress database

It is tempting to inspect the last CSV row and treat it as a cursor. That works
until input order changes, a blank line appears, or the output schema changes.
The data file then has two jobs: representing results and controlling execution.

I kept them separate:

normalized.csv contains the current result rows;
state.json stores completed record IDs and output hashes;
run_log.jsonl records succeeded, failed, retrying, and skipped events;
report.html gives a human-readable result.

The input contract requires record_id,url. A seen set catches duplicate IDs
inside the current input. The durable completed map catches IDs that succeeded
in an earlier run.

This also makes a repeated run understandable. A row is not silently absent; the
log says whether it was skipped because it was a duplicate or because it had
already completed.

Old state must not silently match a new configuration

Resume state becomes dangerous when extraction rules change.

Suppose the first run extracted title and price. Later, the configuration
adds availability. If the old completed IDs are accepted without question,
the pipeline skips pages that have never been processed with the new rule.

The kit hashes a canonical representation of the field configuration:

fingerprint = sha256(
    json.dumps(
        canonical_fields,
        sort_keys=True,
        separators=(",", ":"),
    ).encode("utf-8")
).hexdigest()

That fingerprint is written beside the completed map. When a later run points
to the same output directory with a different configuration, the pipeline
raises an explicit error instead of combining incompatible results.

The fix is intentionally boring: use a new output directory for a new
extraction contract. Quietly guessing would be more convenient and less safe.

Atomic writes protect the mechanism that enables recovery

A state file is useful only if it survives the interruption it is supposed to
help recover from.

Writing directly to state.json can leave truncated JSON if the process stops
at the wrong moment. The next run then fails before it can resume. The same risk
applies to the normalized CSV.

The pipeline writes a complete temporary file and replaces the destination:

temporary = path.with_suffix(path.suffix + ".tmp")
temporary.write_text(value, encoding="utf-8")
temporary.replace(path)

The implementation uses the same pattern for JSON, CSV, and the append-style
run log. It does not make every filesystem failure impossible, but it removes
the common partial-overwrite window.

Evidence should be useful without leaking page content

Debug logs often become accidental data stores. Dumping raw HTML, full query
strings, or local paths would make troubleshooting easier and distribution less
safe.

The public log records the record ID, status, attempt number, and exception type.
It does not record page HTML or local filesystem paths. The HTML report escapes
output values before rendering them.

That boundary matters for a reusable source kit: the person running it controls
the permitted target pages, while the library keeps operational evidence
focused on the run.

I tested the extracted delivery archive, not only the development folder

A passing development environment is not proof that a buyer's ZIP contains the
same working pieces.

I extracted the delivery archive into a clean directory and ran its included
test suite. The result was 10/10 tests passed. One regression test forces
KeyboardInterrupt immediately after the first record is durably checkpointed. A
second process then resumes and proves that the completed record is not fetched
again.

I also ran the offline fixture demo twice:

2 records succeeded;
0 records failed;
1 duplicate ID was skipped safely;
the next run read the completed state instead of fetching finished records again.

The fixture demo uses local HTML and requires no browser, account, proxy, cloud
service, or paid API.

Where Playwright fits

The state, retry, parsing, reporting, and fixture paths do not import Playwright.
The browser dependency is loaded only for permitted http:// or https://
pages.

That keeps the offline demo deterministic and lets the same pipeline accept a
real browser fetcher when JavaScript-rendered pages are in scope.

This is not a universal scraper. Selectors are deliberately simple, and the
operator must define them for pages they own or are allowed to automate. The kit
does not bypass login, CAPTCHA, access controls, rate limits, robots rules, or
website terms.

The finished source kit

I packaged the implementation as the
Resumable Web-to-CSV Pipeline Kit.
It includes the full Python source, tests, configuration examples, offline
fixtures, and run documentation.

Personal: $18 launch price (normally $24) for one user
Team: $44.25 launch price (normally $59) for up to five users in one legal entity

The 25% launch discount is automatically applied through August 1, 20:13 KST for the first 20 uses.

The public
delivery repository
shows the actual product screens and verification evidence before purchase.
Custom selectors, custom development, remote setup, ongoing support, and future
updates are not included.

Watch the crash window in 38 seconds

The short proof reel shows the exact failure path: interrupt after useful work, rerun with retry-only logic, then resume from durable state without refetching completed records.

Watch the 38-second crash-proof Web-to-CSV demo

The reel uses the same fixed offline fixture documented above. It is evidence for this included example, not a promise about every website.

You can also
run the interruption walkthrough in your browser
without installing Python. Run 1 saves the first record and stops; after a real
page reload, Resume skips that completed record before reading its fixture and
finishes the second. This is a fixed offline walkthrough, not an arbitrary-site
benchmark.

Prefer the guided lab instead?

If you want the failure map and the interruption exercise without buying source
code, I also published the 20-page
Retry Is Not Resume field manual.
It includes a 30-minute offline run → stop → resume lab, checkpoint crash windows,
a four-file responsibility map, and a production checklist.

The launch price is $9, and the product page includes a free 6-page preview. The
v1.1 purchase ZIP contains the PDF and a self-contained lab that runs with
python lab/run_demo.py. It is a self-serve download: no custom setup, remote
installation, ongoing support, or future updates are included.

The larger lesson is independent of this kit: retries recover an attempt;
durable state recovers a workflow. Treating those as separate responsibilities
turns "run it again" from a gamble into a defined operation.