DEV Community

Mykola Melnyk
Mykola Melnyk

Posted on

Reading the results: text, boxes, entities and how they line up

#ocr #webdev #tutorial #opensource

Reading the results: text, boxes, entities

Written 2026-09-05. Targets @stabrise/scaledp@0.1.1. Assumes you have run a pipeline; assumes nothing about the schemas.

TL;DR: transform() returns plain objects. Four schemas cover everything — ScaleDpImage, Document, DetectorOutput, NerOutput — every one carries an exception field, and every entity carries the boxes its characters fell inside. That last property is what lets you draw a redaction rectangle over a name the model found in a string.

A row is just an object 

const rows = await pipeline.transform(file)
console.log(Object.keys(rows[0]))
// [ 'path', 'page', 'image', 'text', 'ner', 'row_time', 'execution_time' ]
Enter fullscreen mode Exit fullscreen mode

There is no result class, no accessor API, no .get(). A stage reads a named field and writes another one, and what you get back is whatever the stages wrote. path and page come along from the reader; row_time and execution_time are added by the runner.

That means destructuring works, JSON.stringify works, and passing a row to another function does not require importing anything.

The four schemas

ScaleDpImage — a rendered page 

interface ScaleDpImage {
  path: string
  resolution: number      // DPI it was rendered at; 0 when unknown
  data: Uint8Array        // encoded bytes, PNG unless imageType says otherwise
  imageType: 'png' | 'webp' | 'jpeg'
  height: number
  width: number
  exception: string
}
Enter fullscreen mode Exit fullscreen mode

Encoded bytes, not raw pixels — so row.image.data is a PNG you can hand to createImageBitmap or a Blob.

Document — text plus the boxes it came from 

interface Document {
  path: string
  text: string
  type: string            // 'ocr' | 'pdf' | an engine name
  bboxes: Box[]
  exception: string
}
Enter fullscreen mode Exit fullscreen mode

text is the whole page as one string. bboxes is every word (or region — see boxLevel) with its geometry and its own text.

DetectorOutput — boxes with no text 

interface DetectorOutput {
  path: string
  type: string            // 'dbnet-onnx' | 'paddle' | 'yolo' | …
  bboxes: Box[]
  exception: string
}
Enter fullscreen mode Exit fullscreen mode

The difference from Document is exactly the absence of text. A detector found regions; nobody has read them yet.

NerOutput — entities 

interface Entity {
  entity_group: string
  score: number
  word: string
  start: number           // character offset into the document text
  end: number
  boxes: Box[]            // the boxes those characters fall inside
  source?: 'model' | 'propagated'
}
Enter fullscreen mode Exit fullscreen mode

The Box convention, which is not what you expect

This one catches people, so it is worth stating precisely:

interface Box {
  text: string
  score: number
  x: number       // top-left of the AXIS-ALIGNED box of the same size,
  y: number       // centred on the ROTATED rect's centre
  width: number   // always the LONGER side
  height: number
  angle: number   // degrees about that same centre, normalised to (-90, 270]
}
Enter fullscreen mode Exit fullscreen mode

A Box is not xyxy and it is not a polygon. For an unrotated box, x, y, width, height behave exactly as you would hope and ctx.strokeRect(box.x, box.y, box.width, box.height) is correct. For a rotated one it is not — you need the centre and the angle:

function strokeBox(ctx: CanvasRenderingContext2D, box: Box) {
  const cx = box.x + box.width / 2
  const cy = box.y + box.height / 2

  ctx.save()
  ctx.translate(cx, cy)
  ctx.rotate((box.angle * Math.PI) / 180)
  ctx.strokeRect(-box.width / 2, -box.height / 2, box.width, box.height)
  ctx.restore()
}
Enter fullscreen mode Exit fullscreen mode

Two consequences of "width is always the longer side" worth knowing: a genuinely tall, narrow region is reported as a wide box rotated 90°, and that is correct rather than a bug. And coordinates are in the rendered page's pixel space — the same space PdfToImage produced — so they line up with row.image without conversion. That is a deliberate divergence from the Python library, which leaves text-layer boxes in PDF points.

A Box stores the centre, the size and an angle, not two corners, and width is always the longer side

Entities carry their boxes, and that is the useful part

An NER model operates on a string. It returns character offsets. On its own that gives you a table, not a redaction.

for (const entity of row.ner.entities) {
  console.log(entity.entity_group, entity.word, entity.score)
  for (const box of entity.boxes) {
    strokeBox(ctx, box)   // draw it on the page
  }
}
Enter fullscreen mode Exit fullscreen mode 
person       Marta Feldmann        0.99
email        accounts@acme.de      0.97
iban         DE89 3704 0044 0532…  0.94
Enter fullscreen mode Exit fullscreen mode

Getting from start/end to boxes is not trivial and is worth knowing about, because it was a real bug. The text was reconstructed in reading order — boxes clustered by y, then sorted by x — while bboxes keeps the detector's order. A forward-only cursor walking the boxes therefore found that boxes routinely sat behind it, gave up, and left the entity with no boxes at all: listed in the table, invisible on the page. For a redaction tool that is the worst possible failure, since nothing looks wrong.

An entity carries character offsets into the page text, which map back to the boxes those characters fell inside

If you need this machinery yourself, it is exported:

import { buildCharToBoxMap, boxesForRange } from '@stabrise/scaledp/ner'
Enter fullscreen mode Exit fullscreen mode

The builder renders bboxes directly, which is the fastest way to see what the schema actually contains:

A table of recognised words with score, x, y, width, height and angle for each: ScaleDP 0.948, scalable 0.997, info@stabrise.com 1.000

One row per Box. The text it carries, the score the model gave it, and the geometry — x, y, w, h, angle. Note info@stabrise.com at 1.000 and the ScaleDP wordmark at 0.948: confidence is per box, so you can threshold on it per box.

Displaying it

Four helpers, all returning a DOM element rather than a string, all mirroring ScaleDP's notebook helpers:

import {
  renderInto, showImage, showText, showNer, visualizeNer, showBoxes,
} from '@stabrise/scaledp/display'

renderInto('#page',     showImage(row.image))
renderInto('#text',     showText(row.text))              // layout-preserving
renderInto('#entities', showNer(row.ner))                // a table
renderInto('#inline',   visualizeNer(row.text, row.ner)) // highlighted in the text
renderInto('#boxes',    showBoxes(row.text))             // geometry, first 20
Enter fullscreen mode Exit fullscreen mode

Three things they do that are easy to miss:

showText renders monospace, and that is load-bearing. Layout-preserving text encodes the page's layout in spaces and blank lines. In a proportional font the columns do not line up and it looks like broken output rather than preserved layout.

Failures render as failures. Every helper checks exception first and returns a red error block. A stage that failed shows up as a visible error rather than an empty panel, which is the difference between "something went wrong" and "there was nothing on the page".

visualizeNer drops overlapping entities rather than nesting them — highest score wins — because two spans cannot occupy the same characters in a flat text run.

Annotating the page is a stage, not a helper

Drawing boxes onto the image is a pipeline stage, as in ScaleDP. The annotated page is just another image column:

import { ImageDrawBoxes } from '@stabrise/scaledp'

pipeline.stages.push(
  new ImageDrawBoxes({
    inputCols: ['image', 'text', 'ner'],
    outputCol: 'annotated',
  })
)
Enter fullscreen mode Exit fullscreen mode

One caveat: image_with_boxes is an Image, so it typechecks everywhere an image does — including as the input to a detector. Wiring a detector after an annotation pass means it will faithfully detect the rectangles and label text you just drew. The stage catalogue marks ImageDrawBoxes as terminal for this reason, rather than forbidding it, because chaining two annotation passes for two colours is a genuinely useful idiom.

Trade-offs

  • Plain objects mean no runtime validation. You get TypeScript types and nothing else. A typo in a column name produces undefined, not an error.
  • bboxes can be large. A dense page at word level is hundreds of boxes per page; forty pages is tens of thousands of objects. Drop columns you have consumed.
  • The display helpers are deliberately minimal. They exist so you can see a result in three lines, not so you can ship a viewer. For anything real, read the schemas and render them yourself.

Try it 

npm install @stabrise/scaledp pdfjs-dist onnxruntime-web ppu-paddle-ocr
Enter fullscreen mode Exit fullscreen mode

Open this pipeline in the builder and switch the result panel between Text and Boxes. Those two views are row.text.text and row.text.bboxes — the same two fields this post has been describing, rendered.

Further reading

Top comments (0)