๐ The developers found room to improve on latency, so we profiled. We initially expected the sorting algorithm (XY-Cut++) to be the bottleneck, but it turned out to be less than **1% **of the total time. The real cost was hiding in content filtering (55%) and preprocessing (25%).

๐๏ธ3 fixes applied
๐ฅPage-level parallel processing
๐ฅHidden text detection โ opt-in
๐ฅText-only fast path
๐ขOutput is byte-for-byte identical before and after optimization. Only the speed changed results stay the same.
๐๏ธOpenDataLoader PDF highlights
๐#1 in latency ๐ฅ(585 pages in 1.10s)
๐๏ธ#1 in memory efficiency ๐ฅ(7.4MB)
๐ขJava ยท Python ยท Node.js SDK
๐ขMultiple output formats (text, markdown, HTML, JSON, PDF)
Check out the benchmark below for latency and memory usage results. See the PR for full details on what changed and how we got here. We'd love your feedback if you try it out!
GitHub: http://github.com/opendataloader-project/opendataloader-pdf?utm_source=x&utm_medium=social&utm_campaign=perf_update
Benchmark: http://github.com/opendataloader-project/opendataloader-bench?utm_source=x&utm_medium=social&utm_campaign=perf_update
PR: https://github.com/opendataloader-project/opendataloader-pdf/pull/362?utm_source=x&utm_medium=social&utm_campaign=perf_update
Top comments (0)