The latest articles on DEV Community by Sayantan Patra (@sayantanpatra). https://dev.to/sayantanpatra https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F1178137%2Faa145856-d7ae-4413-9436-25ecc273ae17.jpeg https://dev.to/sayantanpatra en Sayantan Patra Wed, 06 May 2026 15:39:07 +0000 https://dev.to/sayantanpatra/i-built-dfxpy-to-reduce-repetitive-pandas-ml-preprocessing-workflows-142i https://dev.to/sayantanpatra/i-built-dfxpy-to-reduce-repetitive-pandas-ml-preprocessing-workflows-142i <p><a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fn9kf3wbrtswirem1mh1u.png" class="article-body-image-wrapper"><img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fn9kf3wbrtswirem1mh1u.png" alt=" " width="800" height="333"></a><br> Every data project starts with excitement.</p> <p>Then comes:</p> <ul> <li>missing values</li> <li>duplicate rows</li> <li>inconsistent column names</li> <li>encoding</li> <li>leakage checks</li> <li>skew analysis</li> <li>outlier handling</li> <li>repetitive preprocessing pipelines</li> </ul> <p>After rebuilding the same workflow across notebooks and projects, I decided to create something reusable.</p> <p>So I built <code>dfxpy</code> — an open-source Python package focused on accelerating DataFrame workflows for machine learning, analytics, and research.</p> <h2> What dfxpy does </h2> <h3> Automated Cleaning </h3> <ul> <li>smart type inference</li> <li>missing value imputation</li> <li>duplicate removal</li> <li>snake_case normalization</li> <li>currency/percentage/date detection</li> <li>categorical encoding</li> </ul> <h3> ML Preparation </h3> <ul> <li>feature/target splitting</li> <li>optional scaling</li> <li>target encoding</li> <li>date feature extraction</li> <li>class balancing</li> </ul> <h3> Diagnostics &amp; Research </h3> <ul> <li>leakage detection</li> <li>skewness + multicollinearity audits</li> <li>statistical profiling</li> <li>dataset lineage hashing</li> <li>publication-ready LaTeX exports</li> </ul> <h3> Workflow Utilities </h3> <ul> <li>reusable transformation pipelines</li> <li>dataframe comparison tools</li> <li>schema validation</li> <li>standalone HTML EDA reports</li> <li>built-in CLI support</li> </ul> <h2> Example </h2> <div class="highlight js-code-highlight"> <pre class="highlight python"><code><span class="kn">from</span> <span class="n">dfxpy</span> <span class="kn">import</span> <span class="n">auto</span><span class="p">,</span> <span class="n">prepare</span> <span class="n">df</span> <span class="o">=</span> <span class="nf">auto</span><span class="p">(</span><span class="n">df</span><span class="p">)</span> <span class="n">X</span><span class="p">,</span> <span class="n">y</span> <span class="o">=</span> <span class="nf">prepare</span><span class="p">(</span> <span class="n">df</span><span class="p">,</span> <span class="n">target</span><span class="o">=</span><span class="sh">"</span><span class="s">sales</span><span class="sh">"</span><span class="p">,</span> <span class="n">scale</span><span class="o">=</span><span class="bp">True</span> <span class="p">)</span> </code></pre> </div> <p>CLI:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code>dfxpy analyze dataset.csv </code></pre> </div> <h2> One design goal I cared about </h2> <p>I specifically didn’t want this to feel like a thin wrapper around Pandas.</p> <p>The focus became:</p> <ul> <li>workflow automation</li> <li>preprocessing acceleration</li> <li>diagnostics</li> <li>reproducibility</li> <li>research-friendly tooling</li> </ul> <p>rather than simply renaming Pandas functions.</p> <h2> Open Source </h2> <p>The project includes:</p> <ul> <li>automated GitHub workflows</li> <li>PyPI publishing</li> <li>modular architecture</li> <li>active development roadmap</li> </ul> <p>I’d genuinely appreciate feedback from the Python/data community — especially around:</p> <ul> <li>API design</li> <li>architecture</li> <li>performance</li> <li>production-readiness</li> </ul> <p>GitHub: <a href="https://github.com/sayantancodex/dfxpy" rel="noopener noreferrer">https://github.com/sayantancodex/dfxpy</a><br> PyPI: <a href="https://pypi.org/project/dfxpy/" rel="noopener noreferrer">https://pypi.org/project/dfxpy/</a></p> datascience machinelearning python showdev