DEV Community: Milliseconds.dev

pypdf vs PdfPig: Text Extraction at Scale

Milliseconds.dev — Sun, 31 May 2026 18:20:15 +0000

Overview

PDF text extraction is a common pre-processing step in data pipelines — ingesting research papers, legal documents, or reports before embedding or indexing. Both pypdf and PdfPig are pure managed-code parsers: no native binaries, no OCR, no system PDF renderer. They implement the same PDF specification operations in their respective languages.

This makes the benchmark unusually clean: the performance difference is entirely due to language execution speed, not library architecture differences.

Benchmark Setup

200 recent arXiv PDFs (mixed technical papers, 5–40 pages each). Tested on subsets of 10, 50, 100, and 200 files. Both libraries extract all text from all pages; output is validated for page-count agreement and character-count agreement within 15% (pypdf and PdfPig decode whitespace and encoding tables slightly differently).

Results

PDFs	Pages	Python (pypdf)	.NET (PdfPig)	Speedup
10	~120	~0.9 s	~230 ms	3.9×
50	~600	~4.2 s	~810 ms	5.2×
100	~1,200	~8.5 s	~1.4 s	6.1×
200	~2,400	~17 s	~2.7 s	6.2×

The speedup grows slightly with corpus size, suggesting pypdf has a per-document startup cost that compounds as PdfPig's JIT gets warmer.

Why PdfPig Is Faster

PDF parsing is byte-heavy: every page is a stream of PostScript-like operators (move, show text, set font, etc.). Each operator must be lexed, looked up in a dispatch table, and executed against a graphics state machine.

In Python, each operator dispatch is a Python method call — the CPython bytecode interpreter has overhead per call regardless of what the method does. In .NET, the JIT compiles the dispatch loop to native code the first time it runs; subsequent pages pay only the cost of the actual work.

Additionally, PdfPig's content-stream parser operates on ReadOnlySpan<byte> — zero-copy slicing through the raw page bytes with no intermediate string allocations. pypdf builds Python string objects for each token.

Key Code

// PdfPig — zero-copy span-based page extraction
public Result Extract(string path)
{
    using var doc = PdfDocument.Open(path);
    long chars = 0;
    foreach (var page in doc.GetPages())
        chars += page.Text.Length;
    return new Result(chars, doc.NumberOfPages, Ok: true);
}

# pypdf — Python object per token
reader = pypdf.PdfReader(path)
chars  = 0
for page in reader.pages:
    chars += len(page.extract_text() or "")

The structure is identical. The performance difference is pure language execution speed on the same PDF parsing logic.

Diagrams

Both scale linearly in page count, but PdfPig's slope is 6× shallower. At 200 files (a typical daily batch job), .NET finishes in under 3 seconds; Python takes 17 seconds.

The 10-file speedup (3.9×) is lower because .NET's JIT incurs one-time compilation overhead for the PDF parser code paths. By 50 files the JIT is fully warm and the speedup stabilises around 6×.

NetworkX vs CSR + TensorPrimitives: PageRank on 28M Edges

Milliseconds.dev — Sun, 31 May 2026 18:20:14 +0000

Overview

PageRank is the canonical graph algorithm. NetworkX implements it in pure Python — its dict-of-dict adjacency representation means every power-iteration step dispatches millions of Python attribute lookups. When the graph has 1.8 million nodes and 28.5 million edges (Wikipedia category hyperlinks), those lookups dominate the runtime.

The .NET replacement uses a CSR (Compressed Sparse Row) matrix — two flat int[] arrays for the graph structure — and TensorPrimitives for the SIMD-accelerated normalization step inside each iteration.

Benchmark Setup

Five SNAP datasets of increasing size:

Dataset	Nodes	Edges
wiki-Vote	7,115	103,689
soc-Epinions1	75,879	508,837
web-Stanford	281,903	2,312,497
web-Google	875,713	5,105,039
wiki-topcats	1,791,489	28,511,807

Algorithm: power-iteration PageRank, damping=0.85, tol=1e-6. Both implementations converge to identical top-10 node rankings.

Results

Dataset	Python (NetworkX)	.NET (CSR)	Speedup
wiki-Vote (103k edges)	~0.8 s	~100 ms	~8×
soc-Epinions1 (508k edges)	~8 s	~600 ms	~13×
web-Stanford (2.3M edges)	~120 s	~5 s	~24×
web-Google (5.1M edges)	~5.5 min	~12 s	~28×
wiki-topcats (28.5M edges)	~47 min	~60 s	~47×

The speedup grows with graph size because NetworkX's Python dispatch cost scales with edge count, while the CSR inner loop is a tight JIT-compiled SIMD pass.

Why CSR Beats NetworkX

NetworkX represents each node's neighbors as a Python dict. Iterating the adjacency in one power-iteration step means:

Calling G.neighbors(node) — a Python method call
Iterating a dict — unboxing int keys, chasing heap pointers
Accumulating a float into another dict value — another boxing step

That happens for every edge, every iteration, roughly 50–80 times to convergence.

CSR collapses the graph to two arrays: rowPtr[n+1] (where each node's neighbors start) and colIdx[edges] (the neighbor list). Iterating neighbors of node v is a tight C loop from rowPtr[v] to rowPtr[v+1]. No Python objects, no dict hashing, no pointer chasing.

Key Code

// Power iteration — one full pass over all edges
for (int v = 0; v < n; v++)
{
    double contrib = rank[v] / (rowPtr[v + 1] - rowPtr[v]);
    for (int e = rowPtr[v]; e < rowPtr[v + 1]; e++)
        next[colIdx[e]] += contrib;
}
// Damping + dangling-node correction
double base = (1.0 - alpha) / n + alpha * dangling;
TensorPrimitives.MultiplyAdd<double>(next.AsSpan(), alpha, base, next.AsSpan());

# NetworkX power iteration (simplified excerpt)
for nbrs in G.adjacency():
    for nbr, _ in nbrs[1].items():
        xlast[nbr] += x[nbrs[0]] / G.out_degree(nbrs[0])
x = {n: alpha * xlast[n] + danglesum + (1.0 - alpha) / N for n in x}

Every dict access in the Python version maps to multiple bytecode instructions. The .NET version compiles to a loop over two contiguous integer arrays — the CPU's prefetcher handles the rest.

Diagrams

The speedup curve is almost linear in edge count. NetworkX's dispatch overhead is proportional to edges processed, while .NET's JIT loop has near-zero per-edge overhead past the first iteration.

At wiki-topcats scale Python takes ~47 minutes; .NET takes ~60 seconds. The same convergence, the same top-10 ranking, 47× faster.

textdistance vs ArrayPool: Edit Distance Without the Allocations

Milliseconds.dev — Sat, 30 May 2026 23:39:36 +0000

Overview

Levenshtein edit distance is used everywhere strings need to be compared: spell checking, fuzzy record matching, DNA sequence alignment, and plagiarism detection. Python's textdistance library implements it cleanly, but with external=False (pure Python, no C extension) it allocates a full O(m × n) matrix on every call.

The .NET replacement uses the Wagner-Fischer algorithm reduced to a single row: O(min(m, n)) space instead of O(m × n), backed by ArrayPool<int> so even that row is never garbage-collected — it's rented from a pool and returned after each call.

Benchmark Setup

Random word pairs from a 10,000-word dictionary, tested at:

10,000 pairs (average length 8 characters)
50,000 pairs
100,000 pairs

Python uses textdistance.Levenshtein(external=False) — pure Python, no Cython or C backend. .NET uses a static Levenshtein.Distance(ReadOnlySpan<char>, ReadOnlySpan<char>, int[]) with the row buffer passed in from the caller.

Results

Pairs	Python (textdistance)	.NET (ArrayPool)	Speedup
10,000	~1.4 s	~115 ms	12×
50,000	~7.1 s	~200 ms	36×
100,000	~14.2 s	~200 ms	71×

The .NET runtime barely grows past 50k — the preallocated row means the GC is idle and the loop is hot in L1 cache. Python's time grows linearly because each pair triggers a fresh matrix allocation.

Why the Speedup Compounds

Allocation. textdistance builds a list of lists for the full DP table every call. At average word length 8, that's a 9×9 Python list — 9 list objects plus 81 boxed integers, each on the heap. At 100k pairs: 9 million list allocations, 810 million boxed integers created and GC'd.

Row-only Wagner-Fischer. The full matrix is never needed — only the previous row to compute the current row. The .NET implementation keeps a single int[] of length min(m, n) + 1, rented once from ArrayPool<int> and reused for the entire batch.

ReadOnlySpan<char>. No string copies. a.AsSpan() and b.AsSpan() give zero-allocation views; the character comparison a[i-1] == b[j-1] compiles to a single native compare instruction.

Key Code

// Single row, rented once — O(min(m,n)) space
// Python equivalent: textdistance.Levenshtein(external=False).distance(a, b)
public static int Distance(ReadOnlySpan<char> a, ReadOnlySpan<char> b, int[] row)
{
    if (a.Length < b.Length) { var t = a; a = b; b = t; }
    int m = a.Length, n = b.Length;
    for (int j = 0; j <= n; j++) row[j] = j;

    for (int i = 1; i <= m; i++)
    {
        int prev = i;
        for (int j = 1; j <= n; j++)
        {
            int cost = a[i - 1] == b[j - 1] ? 0 : 1;
            int cur  = Math.Min(Math.Min(row[j] + 1, prev + 1), row[j - 1] + cost);
            row[j - 1] = prev;
            prev = cur;
        }
        row[n] = prev;
    }
    return row[n];
}

# textdistance — allocates full matrix per call
algo = Levenshtein(external=False)
for a, b in pairs:
    dist = algo.distance(a, b)

The Python call allocates the full table, fills it, reads table[m][n], and then abandons the entire allocation to the GC. The .NET call overwrites the same row buffer in-place.

Diagrams

The speedup grows because Python's GC pressure increases with pair count while .NET's stays constant. At 100k pairs Python's GC is collecting tens of millions of short-lived list objects; .NET's GC has nothing to collect.

.NET's time barely changes from 50k to 100k pairs — the preallocated row fits in L1 cache and stays there for the entire batch. Python's time grows linearly with allocation cost.