Written 2026-09-03, about porting ScaleDP (Python/Spark) to @stabrise/scaledp@0.1.1 (TypeScript/browser). Assumes you have seen a Spark DataFrame; assumes no deep Spark knowledge.
TL;DR: Porting a Spark-based document library to the browser sounded like it meant replacing a distributed execution engine. It did not. Every stage was a pure Transformer, and the only DataFrame surface stages used was withColumn / drop / select. That reduces to an array of plain objects and a 215-line runner. The generalisable move: audit the surface a framework's users actually touch before assuming you need the framework.
The assumption going in
ScaleDP is a document-processing library built on Apache Spark. The pipelines look like this:
pipeline = PipelineModel(stages=[
PdfDataToImage(resolution=300),
TesseractOcr(inputCol="image", outputCol="text", keepFormatting=True),
Ner(inputCol="text", outputCol="ner"),
])
result = pipeline.transform(df)
The plan was to run the same pipelines in a browser. The obvious reading of "built on Spark" is that Spark is doing something load-bearing — partitioning, shuffles, a query planner, lazy evaluation, a distributed scheduler — and that replacing it means either reimplementing a scheduler or accepting a much smaller feature set.
So before writing anything, we read every stage and made two lists: which Spark APIs are actually called, and which Spark semantics are actually depended on.
What the audit found
Finding one: there are no Estimators. Spark ML's core abstraction is the Estimator/Transformer split — fit() produces a model from data, transform() applies it. Half the complexity of a pipeline framework exists to model that duality: fitting in dependency order, caching fitted models, distinguishing a Pipeline from a PipelineModel.
ScaleDP has none. Every stage is a pure Transformer. Nothing is fitted, because OCR models and NER models arrive pre-trained. The entire fit half of the framework was unused.
Finding two: the DataFrame surface is three methods. Grepping every stage for DataFrame operations turned up withColumn, drop and select. No joins. No groupBy. No window functions. No aggregations. No SQL.
posexplode appears, for turning a PDF into one row per page — which is the one genuinely interesting operation, and it is "return several rows instead of one".
Finding three: no stage depends on laziness. Nothing relied on the query planner reordering work, pushing down a predicate, or fusing stages. The pipeline is executed in the order written.
Three findings, one conclusion: the Spark coupling is one abstraction thick. Underneath is an ordered list of pure functions over records with named fields.
What was left after removing it
/** One record flowing through the pipeline. Stages read and write named fields. */
export type Row = Record<string, unknown>
That is the data model. withColumn becomes { ...row, [outputCol]: value }. drop becomes delete row[inputCol]. select is not needed at all, because a plain object already lets you read whatever you want.
And the stage:
export abstract class Stage<P extends BaseStageParams = BaseStageParams> {
abstract readonly name: string
constructor(readonly params: P) {}
/**
* Transform one row's input value into this stage's output value.
* Throwing is fine and expected -- `transform` converts it into the output
* schema's `exception` field unless `propagateError` is set.
*/
protected abstract apply(input: unknown, row: Row, ctx: StageContext): Promise<unknown>
/** Value written to `outputCol` when `apply` throws. */
protected abstract onError(message: string, row: Row): unknown
/** A stage may emit several rows per input row -- PDF page explosion. */
protected async expand(input: unknown, row: Row, ctx: StageContext): Promise<Row[] | null> {
return null
}
}
Two abstract methods and one optional override. apply for the single-value case, expand for posexplode, onError for the error contract. The base class handles column wiring, error capture, keepInputData and timing — so those behave identically across all eighteen stages instead of being reimplemented eighteen times.
The runner is a nested loop:
for (const [index, stage] of this.stages.entries()) {
options.signal?.throwIfAborted()
await stage.init()
rows = await stage.transform(rows, { index, signal: options.signal })
}
215 lines including the timing columns, the input normalisation and the docs comments.
What Spark was genuinely providing
This is the honest other half, and skipping it would make the post a sales pitch.
Horizontal scale. Forty executors, forty PDFs, wall-clock time divided by forty. One tab is one machine, and there is no story here beyond "open more tabs", which is not a story.
Spilling to disk. A dataset larger than memory. The browser runner holds its rows in memory. A 300 DPI A4 page is roughly 35 MB of raw pixels, so a long document is a real constraint rather than a theoretical one.
Fault tolerance across a cluster. A lost executor is retried elsewhere. A crashed tab is a crashed tab.
Being the thing your data platform already runs. Not a technical property, and often the deciding one.
If your workload needs any of those, use the Python library. The pipeline you wrote in TypeScript ports back nearly line for line, which is exactly why the stage and parameter names were kept identical.
The generalisable part
The framework question is usually asked as "can we replace X?" — which invites an answer scoped to everything X can do. The more useful question is "which of X's capabilities does our code actually call, and which of its semantics do we actually depend on?"
Those are two different lists and the second is the harder one. Method calls you can grep for. Semantic dependencies — laziness, ordering guarantees, transactionality, at-least-once delivery — hide in the absence of code. The way to find them is to ask, per stage, "what breaks if this ran eagerly / in a different order / twice?"
In this case both lists came back nearly empty, and the port became a transcription rather than a rewrite. That is not the usual outcome, and the audit is what told us which one we were in. It cost about a day and would have been worth it either way: had the answer come back the other way, we would have known before writing code instead of three weeks in.
Trade-offs
- We gave up the ability to say "just add executors". The ceiling is one machine, permanently.
- 215 lines of runner is 215 lines we now maintain. Spark's scheduler is battle-tested; ours is not. It is small enough to read in one sitting, which is the mitigation.
-
Matching Spark-era names locks in API decisions we did not make.
inputCol/outputColis not what you would design fresh for TypeScript. Portability was worth more than taste — a pipeline should read the same in both. - The audit is only as good as the reading. We could have missed a semantic dependency. The parity test suites, which diff against goldens generated by running the real Python, are what would catch it.
Top comments (0)