The latest articles on DEV Community by Yevhenii Molchanov (@yevh). https://dev.to/yevh https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F1010482%2Ffe712272-056b-4bd3-a15b-f73d4ded53e3.jpeg https://dev.to/yevh en Yevhenii Molchanov Sun, 07 Dec 2025 11:00:10 +0000 https://dev.to/yevh/i-built-an-open-source-pipeline-to-convert-documents-into-llm-training-data-37pb https://dev.to/yevh/i-built-an-open-source-pipeline-to-convert-documents-into-llm-training-data-37pb <p>Every time I wanted to fine-tune an LLM or build a RAG system, I hit the same wall: I have documents, how do I turn them into training data?</p> <p>PDFs, HTML pages, JSON files, CSVs, LaTeX papers... Each project meant new scripts, no reproducibility, bloated contexts wasting tokens, and numbers silently getting corrupted.</p> <p>So I built 3DCF/doc2dataset to fix this.</p> <h2> What It Does </h2> <ul> <li> <strong>30+ Document Formats Supported</strong> <ul> <li>PDF, Markdown, Plain Text, HTML, XML, JSON, YAML, TOML, CSV, TSV, LaTeX, BibTeX, images with OCR (PNG, JPG, GIF, WebP), RTF, and more.</li> </ul> </li> <li> <strong>5-6x Token Compression</strong> <ul> <li>Instead of dumping raw text, 3DCF creates macro-cells with layout preservation and importance scoring. Same information, fraction of the tokens.</li> </ul> </li> <li> <strong>NumGuard: Numeric Integrity</strong> <ul> <li>When processing financial or legal documents, numbers can get corrupted. NumGuard extracts every number, computes a SHA-1 hash, and tracks it through the pipeline. If anything changes, you know immediately.</li> </ul> </li> <li> <strong>Multi-Framework Export</strong> <ul> <li>Process once, export to HuggingFace, LLaMA-Factory, Axolotl, OpenAI fine-tuning format, and RAG triples.</li> </ul> </li> <li> <strong>Built in Rust</strong> <ul> <li>Fast parallel processing with Python and Node.js bindings available.</li> </ul> </li> </ul> <h2> Evaluation Results </h2> <p>We tested on policy documents, financial reports, technical docs, and scientific papers.</p> <ul> <li>QA Accuracy: 98.0% (vs 91.3% baseline)</li> <li>Average Context Tokens: 35.9 (vs 206 baseline)</li> <li>Numeric Corruption Detection: 100% recall on 18,501 test cases</li> </ul> <h2> Who Is This For </h2> <p>Building RAG systems on your documents. Fine-tuning LLMs on domain-specific content. Processing financial or legal docs where numbers matter. Anyone tired of writing ad-hoc document scripts.</p> <h2> Links </h2> <ul> <li>See the <a href="https://github.com/3DCF-Labs/doc2dataset" rel="noopener noreferrer">GitHub repo link</a> </li> <li>See the <a href="https://github.com/3DCF-Labs/doc2dataset/blob/main/docs/doc2dataset_paper.pdf" rel="noopener noreferrer">research paper link</a> </li> <li>License: Apache-2.0 (fully open source)</li> </ul> <h2> Try It Out </h2> <p>Install with cargo or pip, check the repo for documentation.<br> Star on GitHub if you find it useful!<br> Questions? Drop a comment or open an issue on GitHub.</p> machinelearning ai rag rust Yevhenii Molchanov Wed, 18 Jan 2023 11:02:50 +0000 https://dev.to/yevh/vulnplanet-vulnerable-code-examples-and-fixes-for-web2-web3-apietc-fam https://dev.to/yevh/vulnplanet-vulnerable-code-examples-and-fixes-for-web2-web3-apietc-fam <p>Link: <a href="https://github.com/yevh/VulnPlanet">https://github.com/yevh/VulnPlanet</a></p> webdev security code programming