DEV Community

Mykola Melnyk
Mykola Melnyk

Posted on

The same pipeline, minus the cluster

#ocr #machinelearning #typescript #opensource

The same pipeline, minus the cluster

Written 2026-09-05. Compares @stabrise/scaledp@0.1.1 with ScaleDP on Python/Spark. Assumes you can read both languages; assumes no Spark expertise.

TL;DR: The browser library mirrors the Python one stage for stage — same names, same parameters, same schemas — so porting a pipeline is mechanical rather than a rewrite. What does not carry over is Spark, and the surprising part of the port was how little of Spark was actually load-bearing.

The same pipeline, twice

Python, on Spark:

pipeline = PipelineModel(stages=[
    PdfDataToImage(resolution=300),
    TesseractOcr(inputCol="image", outputCol="text", keepFormatting=True),
    Ner(inputCol="text", outputCol="ner"),
])
result = pipeline.transform(df)
Enter fullscreen mode Exit fullscreen mode

TypeScript, in a tab:

const pipeline = new Pipeline([
  new PdfToImage({ resolution: 300 }),
  new TesseractOcr({ inputCol: 'image', outputCol: 'text', keepFormatting: true }),
  new GlinerNer({ inputCol: 'text', outputCol: 'ner' }),
])
const rows = await pipeline.transform(file)
Enter fullscreen mode Exit fullscreen mode

The parameter names match because they are the same parameters. resolution, inputCol, outputCol, keepFormatting — none of these were renamed to feel more idiomatic in TypeScript, and that was a deliberate choice. A pipeline should read the same in both, so that porting a notebook is a transcription job and not a design job.

The same three stages with the same parameter names, once on Spark and once in a browser tab

What is actually different

Options objects instead of positional column arguments. The column names are still options with the same defaults.

Rows, not a DataFrame. transform returns Row[] — plain objects. A stage that produces several rows per input (a PDF becoming one row per page) does exactly what posexplode does.

Everything is async. Model loading, decoding and inference are all promises. In Python they are blocking calls.

Different NER architecture. Python's Ner is BERT token classification with a fixed tag set. GlinerNer is GLiNER — zero-shot, where the entity labels are the prompt. That is a genuine capability difference, not a naming one: adding a new entity type is an edit rather than a training run.

The part that surprised us

Going in, "port a Spark library to the browser" sounds like it means replacing a distributed execution engine. It did not, and the reason is worth writing down because it generalises.

Reading ScaleDP's stages, two things stood out:

  1. Every stage is a pure Transformer. There are no Estimators anywhere in the library, so there is no fit()/transform() duality to model. A stage is a function from a row to a row.
  2. The only DataFrame surface stages use is withColumn, drop and select. No joins, no groupBy, no window functions, no UDF registration beyond the obvious.

Those two facts together mean the Spark coupling is one abstraction thick. Strip it and what remains is: an ordered array of stages, each reading a named field and writing a named field, over an array of plain objects.

That is the entire runner, and it is 215 lines:

export abstract class Stage<P extends BaseStageParams = BaseStageParams> {
  abstract readonly name: string

  /** Transform one row's input value into this stage's output value. */
  protected abstract apply(input: unknown, row: Row, ctx: StageContext): Promise<unknown>

  /** Value written to `outputCol` when `apply` throws. */
  protected abstract onError(message: string, row: Row): unknown

  async transform(rows: Row[], ctx: StageContext): Promise<Row[]> {
    // column wiring, error capture, keepInputData and timing, once, for everyone
  }
}
Enter fullscreen mode Exit fullscreen mode

The generalisable lesson: audit the surface a framework's users actually touch before assuming you need the framework. ScaleDP looked Spark-shaped from the outside. From the inside, Spark was providing partitioning and shuffles — neither of which a single-document browser pipeline needs — plus a withColumn API that is three lines of object spread.

What Spark was genuinely providing

Not nothing. Being honest about this matters, because it is exactly the list of things the browser version cannot do:

  • Horizontal scale. Forty executors reading forty PDFs. One tab is one machine.
  • Spill to disk. A dataset larger than memory. The browser pipeline holds its rows in memory and that is that.
  • Fault tolerance across a cluster. A lost executor is retried. A crashed tab is a crashed tab.

If your workload needs any of those, the Python library is the answer and the pipeline you wrote for the browser will port back almost unchanged. That symmetry is the point of matching the names.

Horizontal scale, spill to disk and fault tolerance are gone; the stage model and column wiring carry over

What carries over unchanged

  • The non-throwing error contract. Failures land in the output schema's exception field and the pipeline completes. propagateError opts into throwing. Identical in both.
  • The Box convention. x/y is the top-left of the axis-aligned box of the same size centred on the rotated rect's centre, angle is degrees about that centre, and width is always the longer side. Getting this wrong shifts every downstream consumer silently, so it is reproduced exactly.
  • Layout-preserving text reconstruction under keepFormatting, including the per-line indent and blank-line rules.
  • Parameter names and defaults, wherever a Python equivalent exists.

The last two are not asserted, they are tested. test/fixtures/*.py run the real ScaleDP code and write JSON goldens; the parity suites diff against them. Verifying a port against its own expectations just encodes your own misunderstanding twice.

The builder will generate the TypeScript for any pipeline you assemble, which makes the correspondence easy to check against a notebook you already have:

Generated TypeScript showing PdfToImage, DbnetOnnxDetector, TesseractRecognizer and ImageDrawBoxes composed into a Pipeline

PdfToImage, DbnetOnnxDetector, TesseractRecognizer — the same stage names, the same parameter names, the same order. The imports are the only genuinely new thing to learn.

What has no browser equivalent

Python stage Status
CraftTextDetector PyTorch only
LayoutDetector Needs the PaddleOCR Python runtime
EasyOcr, SuryaOcr, DocTROcr No browser builds — use PaddleTextRecognizer
LLMOcr, LLMNer, LLMExtractor Any fetch-based OpenAI client works
TextSplitter, TextEmbeddings Not yet ported

And one stage that goes the other way: PaddleRecognizer has no Python equivalent. It is PP-OCR's recognition model reading another detector's boxes, which the Python API cannot express.

Trade-offs

  • Matching names locks you to someone else's API decisions. Some of ScaleDP's parameter names are not what we would pick starting fresh. Portability was worth more than taste.
  • Matching behaviour means reproducing quirks. The DBNet path feeds the model BGR channels normalised against RGB ImageNet statistics, because the reference does and the model was trained that way. "Fixing" it shifts every box.
  • It also means deciding, bug by bug, when not to match. Five Python bugs are deliberately not reproduced. Each one is listed publicly, because a silent divergence is worse than either choice.

Try it 

npm install @stabrise/scaledp pdfjs-dist onnxruntime-web ppu-paddle-ocr
Enter fullscreen mode Exit fullscreen mode

Open the ported pipeline in the builder and compare the generated TypeScript against the Python you already have. If porting a notebook is not close to mechanical, that is a bug worth reporting.

Further reading

Top comments (0)