I built a real Spark event-log parser instead of faking it with sample data

Dor Farber — Tue, 11 Aug 2026 08:17:58 +0000

Quick context if you didn't see the earlier post: opti-pipe compares your pipeline's config against its actual run metrics and tells you what's over/under-provisioned — executor memory, shuffle partitions, cluster size — with the reasoning shown, not a black-box score. Rule-based, not ML, on purpose: I can point at the exact threshold that fired.

dbt had a real integration from day one (reads your actual target/run_results.json). Spark didn't — the rules existed, fully tested, but ran against hand-typed fixtures because there was no ingestion path. That's fixed now.

Why an event log, not a live REST API reader

Two real options existed: a live reader against Spark's REST API / History Server, or an uploaded event log (the file Spark writes when spark.eventLog.enabled=true). The live path is more precise but needs a reachable driver or History Server — friction most solo setups don't want. The event log is self-serve, same shape as dbt's upload, ships without needing cluster connectivity. Picked the second one; the live reader stays on the roadmap.

The parsing itself

parse_spark_event_log walks the real listener event stream — SparkListenerApplicationStart/End, SparkListenerEnvironmentUpdate, SparkListenerStageSubmitted, SparkListenerTaskEnd with its nested Task Metrics/Task End Reason — line by line (it's JSON-lines, one event per line, not a single document). From it: task runtime, GC pause time, record counts, OOM detection via matching Task End Reason against OOM error signatures, queue delay from stage-submission-vs-task-launch timestamps, and heap usage as a percentage of configured executor memory.

A bug worth mentioning

Spark's event log has zero concept of CPU utilization — no listener emits it, it's an infra-level metric. My first pass defaulted that field to 0.0. That's a real bug: 0% CPU reads to the "overprovisioned cluster" rule as genuinely idle, so it would've spuriously recommended shrinking your cluster on every single real import regardless of actual load. Fixed with an explicit CPU_UTIL_UNAVAILABLE sentinel that the aggregation and rule logic both treat as "no data," not "0%." Only surfaced because I stopped testing against fixtures I'd hand-written with plausible non-zero values and pointed it at something real.

Try it against your own Spark job (upload guide included, walks through finding the event log on EMR/Databricks/Kubernetes/standalone): opti-pipe.onrender.com. Feedback — especially "this recommendation is wrong" — is genuinely the most useful thing anyone can send me right now.

Why I stopped guessing at Spark and dbt config values

Dor Farber — Sat, 08 Aug 2026 21:24:07 +0000

I've spent more than a decade building data pipelines, and the part nobody warns you about isn't the pipeline logic. It's the tuning. Executor memory, shuffle partitions, cluster size, thread counts. You pick numbers, ship it, and a few weeks later something breaks in a way that's obviously tuning-related but not obviously what to change.

The pattern repeats enough times that you start recognizing it before you've even opened the logs. Job's slow, thousands of tiny shuffle tasks, someone way overestimated the partition count. Job dies on OOM, memory's set for last quarter's data volume, nobody updated it since. Cloud bill jumps, a cluster's been sized for peak load and just sits there mostly idle the other 20 hours a day. Every senior data engineer has this pattern-matching running in their head. It's tribal knowledge, and it lives in one or two people's heads on most teams, which means it doesn't scale and it definitely doesn't survive someone leaving.

So I built a small tool to make that pattern-matching explicit instead of tribal: it reads your pipeline's config alongside its actual run metrics and tells you what's likely wrong, with the reasoning shown, not just a suggested number.

Why rules instead of a model

The obvious move in 2026 is to reach for an ML model. I didn't, and it wasn't because I don't think ML has a place here eventually. It's that for this specific problem, a handful of threshold rules already gets you most of the value, and they're something you can actually audit.

If a rule fires, I can point at the exact condition and the exact number: average heap usage 28%, peak 47%, five runs, no OOM errors, therefore memory's over-provisioned, shrink it by roughly a fifth. That's checkable. You can look at your own metrics and see whether the reasoning holds. A model's confidence score doesn't give you that, and for something that's about to change a production config, I want the person approving it to be able to say "yes, I see why" rather than "the model said so."

There are seven rules right now: memory over/under-provisioned, shuffle partition sizing, cluster instance count over/under-provisioned, and two for dbt (thread count and tasks-per-thread). Each one is a pure function: current config plus recent run metrics in, a recommendation or nothing out. No side effects, no state, fully unit tested.

Nothing applies itself

This was a deliberate line, not a limitation I plan to remove later. Every recommendation requires a human click before anything changes. I don't trust an agent to touch a production Spark config unattended, and I don't think you should either, no matter how good the underlying reasoning is. The tool's job is to surface the diff and the reasoning behind it. Approving it is still yours.

What's actually real today, and what isn't

dbt is the one framework with a genuine integration. Upload your actual target/run_results.json and you get recommendations computed from your real run history, not a fixture. Spark's rules exist and are fully tested, but there's no live reader yet, no Spark REST API or event-log ingestion, so it works on sample data today rather than your own cluster. I'd rather say that plainly than let the pitch imply more than the tool does.

Try it

There's a live dashboard with three real sample pipelines you can click into right now, no signup, no upload required to look around: opti-pipe.onrender.com. If you run dbt, you can upload your own run_results.json and get recommendations against your actual data.

I'd genuinely like to hear from anyone who tries it against a real pipeline, especially if a recommendation is wrong. That's the fastest way I'll find the next rule worth adding.

DEV Community: Dor Farber