DEV Community: Unsiloed AI

Unsiloed Leads ChartParse-Bench in Chart Extraction Accuracy

Unsiloed AI — Thu, 29 Oct 2026 08:56:17 +0000

By Adnan Abbas and Jishu Sengupta. Originally published on Unsiloed AI.

Every industry runs on chart data. Some of the most decision-critical numbers in quarterly earnings reports, clinical trial results, insurance filings, and engineering specifications live not in text or tables, but inside bar charts, line plots, and scatter diagrams.

Vision language models (VLMs) continue to improve at document parsing, but charts present a distinct challenge. Their meaning depends on the relationship between marks, labels, legends, and axes. A parser can recover the overall shape while changing the values or omitting most of the underlying points, passing those errors to quantitative analysis, chart question answering, and retrieval-augmented generation (RAG) applications.

In this article, we present the Unsiloed chart parsing pipeline, compare leading VLMs and document-parsing pipelines on complex real-world charts, and evaluate them with ChartParse-Bench, a hand-curated dataset designed to test how accurate our and current state of the art chart parsing pipelines are.

View full-resolution image

Why LLM Applications Need Structured Chart Data?

An LLM can describe the apparent trend in a chart without recovering the observations behind it. That may be enough for a summary, but it is not enough to answer questions such as "What was the value in 2019?", identify a crossover, calculate a rate of change, or compare two series.

RAG works best when the retrieval layer can find explicit values, labels, and relationships. Converting the chart into structured series data gives downstream systems a stable representation they can search, filter, and calculate over.

How Unsiloed's Chart Parsing Pipeline Handles Different Chart Types?

A single general-purpose model has to handle different visual problems. A dense line chart requires the parser to trace overlapping series through crossings, while a bar chart requires it to locate discrete marks and bind each one to the correct category and scale.

Unsiloed uses an agentic harness to identify the chart and route it to agents specialized for different chart families. Separate agents are tuned to handle charts of various types like line and area charts, bar charts, and pie and scatter plots with special tools to aid in the process. Each specialist can apply the image-processing approach suited to that geometry instead of forcing every chart through the same extraction path.

The harness coordinates the result so downstream applications receive one structured representation rather than a collection of unrelated model responses.

The diagram below shows how the agentic harness selects one chart-specific extraction route:

View full-resolution image

Comparing Chart Parsing Pipelines on Complex Real-World Charts

Real-world charts are important since they prove that our pipeline works well on messy charts found in production scenarios across various industries. Across the following qualitative examples, we display the 2 main groups of failures many chart parsing pipelines face which our agentic system is robust to.

The Output Has The Correct Outline But Incorrect Chart Value Extracted

A parser can preserve a chart's outline while changing its coordinate system or attaching the wrong values to it. In the following chart below Unsiloed is the only parser to maintain chart accuracy and structure very perfectly.

View full-resolution image

Errors are harder to notice when a reconstruction remains visually plausible. In the three-fund comparison, Unsiloed follows the source curves closely. GPT-5.4 adds sharp fluctuations that are not present in the source, while several other outputs smooth the curves into a smaller set of points.

View full-resolution image

Sparse Output Hides Missing Data

Stacked area charts make missing structure especially damaging because each series is defined by both its upper and lower boundary. Unsiloed returns the formats as bands in their original order and scale. LlamaParse and Extend separate the bands into independent lines, changing what the values mean.

View full-resolution image

The 10-series comparison makes the missing points visible. Unsiloed traces all 10 series across approximately 250 trading days.

View full-resolution image

These examples demonstrate the failure modes, but they cannot establish the size of the error. Quantitative evaluation requires synthetic charts whose underlying values are known.

ChartParse-Bench: A Chart Parsing Benchmark

Understanding the gaps of qualitative evaluation and driven by our belief in a robust evaluation, we present ChartParse-Bench. ChartParse-Bench is a synthetic chart parsing benchmark that provides ground truth for errors that can only be observed qualitatively in real documents. It contains 58 charts and ground-truth points generated with Matplotlib. Each chart isolates a predefined failure mode, such as swapping series at a crossing, interpolating through a gap, or treating a hatched fill as data. This dataset will help us have a quantitative evaluation of our chart parsing pipeline. It is hosted on Hugging Face at Unsiloed/chart-parse-bench.

The benchmark covers five chart families:

Line charts (18): Series identity through crossings, resampling, and axis binding
Scatter plots (11): Recall under overplotting, size and shape channels, and outliers
Area charts (nine): Stacking, computed baselines, and composited fills
Compound charts (12): Non-Cartesian frames, linked panels, and mixed geometries
Market charts (eight): Correlated series that move together

View full-resolution image

Quantitative Measurement of Chart Parsing Pipelines

We utilise 2 important metrics to effectively measure how accurate our chart parsing baseline is.

1. Mean absolute error (MAE) measures how far each extracted value is from the value used to generate the chart. The benchmark averages these differences across the evaluated points and expresses the result as a percentage of the axis range. Lower MAE means the reconstructed values sit closer to the ground truth.

MAE however only scores the points a parser emits. A model can return a dozen well-placed landmarks from a 500-point series and receive a reasonable error score even though it omitted most of the data.

2. Point density exposes that omission by comparing the volume of structured points produced by each pipeline. It acts as a completeness check alongside MAE. Both numbers are needed because dense output can be inaccurate, while accurate output can be too sparse for interpolation, crossover detection, trend fitting, or point lookup.

Benchmark Results

The first result compares the mean absolute error of all seven pipelines across the benchmark.

View full-resolution image

The second compares how many structured coordinates each pipeline returned from the same 58 charts.

View full-resolution image

Reliable Chart Data Makes Downstream Answers More Reliable

Charts can look correctly reconstructed while their values, series identities, or sampling resolution have changed. Those errors become difficult to detect after the original image has passed through a document pipeline and only the extracted representation reaches the application.

For RAG, chart question answering, and quantitative analysis, parsing quality therefore depends on both accuracy and completeness. Preserving the values is necessary, but so is preserving enough of the series for downstream systems to calculate over the data rather than infer from a simplified outline.

Charts, Plots, and Figures: Getting Visual Data APIs Skip (July 2026)

Unsiloed AI — Thu, 29 Oct 2026 08:52:32 +0000

By Aman Mishra. Originally published on Unsiloed AI.

Your parser can return clean text, neat chunks, and complete metadata while still missing the most important value on the page: the number inside a chart.

That is where document extraction pipelines quietly fail. A chart is more than an image in the middle of a report. It may contain the revenue trend, clinical threshold, model benchmark, or risk score the downstream system needs. If the parser detects the figure but does not recover the data inside it, every RAG query, agent workflow, or compliance check built on that output inherits the gap.

This post explains why chart extraction fails, what real figure extraction has to recover, and how a vision-model pipeline can return chart data as structured, cited output.

TL;DR

Most document parsers detect charts but do not extract the values inside them.
Chart extraction means recovering axis labels, tick values, legends, series names, and data points as structured output.
Different chart types need different decoding logic: bar charts use height, pie charts use arc angles, heatmaps use color scales, and dual-axis charts use multiple coordinate systems.
A reliable pipeline detects the figure, classifies the chart type, decodes the visual encoding, and links every value back to its source region.
Unsiloed AI returns extracted figure output with confidence scores and bounding-box citations, so high-confidence results can pass through automatically and borderline results can be routed for review.

Why Charts Are a Data Trap in Documents

Charts, plots, and figures often carry the densest information in a document. A quarterly revenue chart, a trial outcome plot, or a model benchmark figure can contain the answer a downstream system is supposed to retrieve.

Most document parsers are still built around text. They read embedded PDF text, OCR scanned words, detect tables, and return layout regions. A chart breaks that model because its data lives in visual relationships: bar height, line position, axis scale, legend color, point radius, and spatial grouping. NVIDIA's overview of PDF data extraction for information retrieval explains why closing this gap takes page-element detection and specialized extraction models.

The failure is hard to spot. A parser may return a bounding box labeled figure, or include a screenshot-like image reference, and the output looks complete. But the values inside the chart are missing, so a question like "Which product line drove Q3 growth?" returns nothing if the answer only appeared in a stacked bar chart.

What Chart Extraction Actually Means

Chart extraction means reconstructing the data encoded inside a visual. A useful extraction returns axis titles, tick labels, legend entries, series names, units, and data-point values. Researchers often call this "chart-to-table" or "derendering" because the goal is to recover the table that produced the chart. Google Research's work on foundation models for chart reasoning shows how image-to-text models can learn to generate a table-like representation of chart data from an image.

Figure detection is not the same thing. Detection tells you where a chart sits on the page. Extraction tells you what the chart says. For pipelines that need numbers, captions, labels, and traceable evidence, a bounding box alone is not enough.

Why Different Chart Types Need Different Decoders

Each chart type encodes data differently, so a generic image-to-text pass is not enough.


Chart Type	How Data Is Encoded	Why Standard Parsers Fail
Bar and column charts	Value mapped to bar height in a raster image	The parser sees the bars as shapes, not values
Line charts	Points connected across x/y axes	Tick spacing, units, and point-to-axis mapping are not recovered
Pie and donut charts	Values encoded as arc angles	Legend text may be read, but slice values are not tied to geometry
Scatter and bubble charts	Two or three encoded dimensions per point	Points need coordinate mapping against axis ranges
Heatmaps and matrix charts	Magnitude encoded as color	The color scale has to be calibrated before values can be recovered
Combined and dual-axis charts	Multiple series across two coordinate systems	A single-coordinate extraction model mixes or drops series

This is why chart extraction has to classify the figure before decoding it. A bar chart, heatmap, and dual-axis chart may all sit inside a figure bounding box, but they need different logic to recover their values.

How a Chart Extraction Pipeline Works

A production chart extraction pipeline usually runs in four stages:

Detect: A vision model scans each page and marks figure regions with bounding boxes.
Classify: The system identifies the figure type, such as bar, line, scatter, pie, stacked, or dual-axis.
Decode: A chart-specific model reads axis scales, tick labels, legends, series, and visual marks, then rebuilds the values as structured data.
Validate: Extracted figure output comes back with confidence scores and bounding-box citations, so the pipeline can tell which results to trust and which to route for review.

Classification is where text-first parsers usually stop. They can detect that a visual region exists, but they do not know whether the region contains a chart, diagram, scanned table, or decorative image. Without that distinction, the parser either skips the region or returns pixels that still need manual interpretation.

Decoding is the harder part. A pipeline has to handle logarithmic axes, irregular ticks, overlapping series, rotated labels, legends outside the plot area, and low-resolution scans. Layout-aware OCR helps recover nearby text, but chart values still require visual inference.

Validating Chart Data With Confidence Scores

Chart extraction quality changes with chart complexity. Clean bar charts with visible labels parse more reliably. Dense line charts, overlapping series, small tick labels, handwritten annotations, and low-resolution scans produce more uncertainty.

That uncertainty has to be visible in the output. Unsiloed returns confidence scores and bounding-box citations with extracted figure output. The confidence score tells the pipeline how much to trust the extraction. The citation tells a reviewer exactly where to verify it.

That makes automated routing practical. A common routing policy is to pass high-confidence values straight through, flag borderline values for review, and retry, escalate, or exclude low-confidence and partial extractions before they pollute a database, RAG index, or agent workflow.

Localizing Figures and Linking Captions

Reliable figure extraction depends on two related tasks: localization and association. Localization finds the figure boundary. Association links the figure to its caption, title, surrounding paragraph, or nearby table.

Template-based extractors struggle here because they expect figures to appear at fixed coordinates. That works until a report template changes, a caption wraps onto another line, or a figure moves across pages. Caption association also breaks when the caption is a separate text block instead of part of the image.

A vision-first parser uses the full page to detect figure boundaries, then uses spatial reasoning to connect the figure to nearby caption text and section context. That context matters for retrieval. A chart value is much more useful when it is attached to "Figure 3: Q3 revenue by product line" than when it appears as an isolated number.

Chart Extraction at Scale

At scale, chart extraction becomes a pipeline design problem. The model has to extract values, preserve provenance, and expose enough metadata for review and retrieval.

Common failure points include:

Cross-page context: Figures that span pages or reference a legend elsewhere break parsers that process each page independently, the same issue behind why multi-page tables break extraction pipelines.
Coordinate systems: Bounding boxes in PDF point space need to map onto rasterized image coordinates. If the API does not return the coordinate system and page dimensions, verification becomes harder.
Confidence distribution: A parser that performs well on clean charts may degrade on scans, compressed images, and unusual aspect ratios. Per-figure confidence scores let teams route weaker extractions to review.
Storage format: Structured JSON with axis labels, series names, units, and data-point arrays indexes cleanly for retrieval. Flat text summaries do not preserve enough structure for reliable downstream use.
Provenance: Linking every extracted value to a page and bounding box keeps audit trails intact when a number is disputed, and supports a stronger chunking strategy for RAG systems.

How Unsiloed AI Handles Chart and Figure Extraction

Most document data extraction software either skips figures or returns an image reference with no structured data attached. Unsiloed AI runs vision-model inference on detected figure regions and returns structured values instead of a raw visual placeholder.

For bar charts and line plots, that means axis labels, series names, units, and data-point values. For annotated figures, it means caption text, callout labels, and the spatial relationships between annotations and the regions they mark. These are the figure-level signals that multimodal data extraction tools need before they can support reliable AI agents or document workflows.

Extracted figure output includes confidence scores and bounding-box citations back to source regions, so downstream systems can verify what was extracted and where it came from.

When Chart Extraction Matters

Chart extraction matters whenever a document pipeline has to answer questions from visual evidence, beyond body text alone.

Use a chart-aware parser when your documents include:

Financial reports with revenue, forecast, or variance charts
Scientific papers with plots, benchmarks, or experimental results
Medical and clinical documents with threshold charts or outcome figures
Legal, insurance, or compliance documents with timelines, risk matrices, or annotated diagrams
Product, operations, or manufacturing reports with trend charts and dashboards

If those figures are skipped, the pipeline does not merely lose nice-to-have context. It loses source data.

Final Thoughts on Building a Reliable Chart Extraction Pipeline

Skipped charts are systematic data loss. They are hard to spot because the parser may still return a clean-looking document structure, but the values that mattered never enter the pipeline.

A reliable chart extraction workflow detects figures, classifies chart types, decodes visual encodings, returns structured data, and ties extracted output to confidence scores and source regions. To see how Unsiloed handles chart extraction in production document workflows, book a demo.

FAQ

What's the difference between figure detection and chart extraction?

Figure detection locates where a chart or figure sits on a page and returns its bounding-box coordinates. Chart extraction goes further by recovering the encoded data inside that figure, including axis labels, tick values, legends, series names, and plotted quantities.

Can OCR extract data from charts in a PDF?

OCR can read text labels in and around a chart, but it cannot recover the chart values by itself. Chart values are encoded visually through geometry, color, position, and scale. A vision pipeline has to detect the figure, classify the chart type, and decode the visual encoding into structured data.

What should a chart extraction API return?

A useful chart extraction API should return structured data, not a bare image crop. Look for axis labels, units, tick values, legend entries, series names, data-point values, confidence scores, and source citations such as page numbers and bounding boxes.

How do I validate extracted chart data at scale?

Route extractions by confidence score. High-confidence values can pass through automatically, while borderline values should be flagged for review. Bounding-box citations make review faster because each value points back to the exact source region in the document.

How does Unsiloed AI handle chart and figure extraction?

Unsiloed AI runs vision-model inference on detected figure regions, returning structured values for charts and spatial context for annotated figures. Extracted figure output includes confidence scores and bounding-box citations, so downstream systems can verify the result before using it in search, analytics, or automation.