DEV Community

Satyam Kumar
Satyam Kumar

Posted on

ExperimentPilot AI: Stop Wasting Hours on EDA and Debugging ML Models

Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle Submission πŸ’Έ

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle


ExperimentPilot AI: On-Device Dataset Behavior Discovery & ML Experiment Analysis Platform

"Upload your data. Discover how it behaves. Understand why the patterns matter. Decide what to investigate next."

Transforming raw tabular datasets and ML experiment predictions into empirical discoveries, statistical evidence, and AI-grounded hypotheses β€” 100% locally and privately.


What I Built

1. The Core Problem: The Silent Bottleneck in Data Science & ML

Every data scientist, machine learning engineer, and quantitative analyst faces the exact same time-consuming ritual whenever they receive a new dataset or evaluate a new model iteration:

  1. Repetitive, Manual Exploratory Data Analysis (EDA): Spending hours writing boilerplate Pandas, Seaborn, and Scikit-learn scripts to check missing values, calculate skewness, generate correlation matrices, run ANOVA tests, and plot PCA clusters.
  2. The "Why Did My Model Fail?" Black Box: Standard experiment trackers (like MLflow or Weights & Biases) excel at logging loss curves and hyperparameters, but they leave engineers stranded when answering: Why did Model B fail on samples Model A got right? What specific feature subgroups caused the failure?
  3. The Cloud AI Dilemma (Privacy & Hallucinations): Uploading proprietary enterprise data, healthcare records, or financial figures to closed cloud APIs (OpenAI, Anthropic) directly violates corporate data governance, HIPAA, and GDPR. Furthermore, asking generic cloud LLMs to analyze data leads to fabricated numbers and non-reproducible statistical claims.

2. The Solution: ExperimentPilot AI v2.0

ExperimentPilot AI is an on-device, privacy-preserving Dataset Behavior Discovery and ML Model Experiment Workstation. It pairs deterministic mathematical computation (Pandas, NumPy, SciPy, Scikit-learn) with local open-weight AI inference (via Ollama) to autonomously profile data, discover hidden behavioral patterns, isolate model disagreements, and generate actionable investigation hypotheses.

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                                 EXPERIMENTPILOT AI WORKFLOW                            β”‚
β”‚                                                                                        β”‚
β”‚   [Raw Tabular CSV] ──► [Deterministic Math Engine] ──► [Compact Evidence JSON]       β”‚
β”‚                               (SciPy / Scikit-learn)               β”‚                   β”‚
β”‚                                                                    β–Ό                   β”‚
β”‚   [Interactive Q&A & Reports] ◄── [React Dashboard] ◄── [Local Open-Weight LLM]       β”‚
β”‚        (Recharts Visuals)           (100% On-Device)        (Ollama: Qwen 2.5 0.5B)    β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
Enter fullscreen mode Exit fullscreen mode

The platform delivers two interconnected operational engines:

Engine A: Automated Dataset Behavior Discovery

  • Fine-Grained Semantic Type Inference: Automatically categorizes columns into numeric_continuous, numeric_discrete, categorical_nominal, categorical_ordinal, datetime, identifier, free_text, and constant.
  • 0–100 Data Quality Health Score: Evaluates missingness penalties, duplicate row rates, constant features, and extreme distribution skews.
  • Automated Target Outcome Detection: Detects candidate target variables using keyword semantics (churn, target, price, status, grade, risk), variance, and cardinality.
  • Multivariate Relationship Discovery:
    • Continuous pairs: Pearson $r$, Spearman rank correlation $\rho$, and $p$-value significance.
    • Categorical pairs: CramΓ©r's V and Chi-Square $\chi^2$ independence tests.
    • Mixed pairs: One-way ANOVA $F$-test with $p < 0.05$ validation.
    • Collinearity & Leakage Alerts: Highlights redundant features ($|r| \ge 0.85$) that risk leaking target information.
  • Natural Subpopulations (PCA + K-Means): Standardized continuous dimensions mapped via 2D Principal Component Analysis (PCA) with K-Means clustering ($k \in [2, 3]$) to surface distinct behavioral customer/record personas.
  • Multivariate Anomaly Detection: Isolation Forest algorithm detecting multi-dimensional outliers, detailing per-record top deviating features with exact $z$-scores.
  • Predictive Baseline: Supervised Mutual Information ranking and a 3-fold cross-validated Random Forest baseline (Accuracy for classification, $R^2$ / MAE for regression).
  • Interactive Conversational Q&A: A natural language chat assistant that routes analytical intent (target_association, outlier_investigation, group_comparison, cluster_profiles, data_quality), computes exact mathematical slices, and dynamically renders interactive Recharts charts inside the conversation bubble.

Engine B: ML Model Experiment Comparison & Iteration Engine

  • Deterministic Comparative Metrics: Side-by-side evaluation of baseline vs. candidate models (Macro Accuracy, Precision, Recall, Macro-F1, Weighted-F1, Confusion Matrix heatmaps).
  • Interactive Error Explorer: Isolates sample disagreement quadrants (Both Correct, Both Incorrect, Model A Advantage, Model B Advantage, and Disagreements).
  • Persistent SQLite Experiment Planner: Local hypothesis iteration proposal tracker storing hypotheses, proposed changes, expected metric deltas, and statuses (planned, running, completed).
  • Comprehensive Dual Report Exporter: 1-click generation of fully styled Markdown (.md) and standalone HTML audit reports.

3. Commercialization & Side Hustle Model: Who Would Pay For It?

ExperimentPilot AI is architected as an Open-Core Micro-SaaS / Desktop Pro Product targeting a massive and underserved niche:

  1. Solo ML Engineers & Freelance Data Consultants: Saves 10+ hours of boilerplate EDA and error analysis on every client engagement. They can hand clients interactive, polished HTML behavior reports on Day 1.
  2. Kaggle Competitors & AI Researchers: Quickly spots ensemble stacking opportunities by isolating sample quadrants where Model A and Model B disagree.
  3. Applied ML Teams in Regulated Sectors (Healthcare, FinTech, Legal, Defense): Organizations with strict data privacy mandates (HIPAA, SOC2, GDPR) who are strictly forbidden from uploading datasets to closed cloud APIs.

Sustainable Freemium Pricing Tiers

Tier Price Target Audience Features
Community Core (Open Source) $0 / Free Students, hobbyists, solo developers 100% Local execution, unlimited CSV imports, full statistical suite, local Ollama integration, SQLite planner, Markdown/HTML reports.
Pro Researcher $19 / mo or $149 / yr Freelance consultants, Kaggle Grandmasters Multi-model ensemble stacking simulation (voting on disagreements), automated CI/CD pipeline integration (fails PRs on metric regression), custom metrics scripts.
Team & Studio $79 / seat / mo Startups, enterprise ML squads (3–20 devs) Centralized local team database sync, collaborative hypothesis backlog, compliance audit logging, Slack/Discord notification webhooks.

Demo

Pre-Packaged 1-Click Synthetic Benchmarks

ExperimentPilot AI includes 5 pre-packaged synthetic datasets in examples/ for zero-configuration testing:

  1. Retail Customer Segmentation (retail): 400 rows, 9 columns β€” RFM loyalty, average order value, CLV, and churn outcome.
  2. Student Academic Performance (student): 450 rows, 9 columns β€” Study hours, attendance rate, parental education, and exam scores.
  3. Residential Real Estate Valuation (housing): 500 rows, 9 columns β€” Square footage, bedroom counts, build year, and price regression.
  4. IoT Turbine Vibration Sensors (sensor): 600 rows, 9 columns β€” Rotational vibration, bearing temperature, voltage, and hardware anomalies.
  5. Customer Retention Benchmark (churn): 500 rows, 8 columns β€” Customer tenure, monthly charges, support interactions, and binary retention.

Quickstart: Try It Locally in Under 3 Minutes

# 1. Clone repository
git clone https://github.com/satyam-257/ExperimentPilotAI.git
cd ExperimentPilotAI

# 2. Launch Local Open-Weight AI (Ollama)
ollama pull qwen2.5:0.5b
ollama serve

# 3. Start FastAPI Backend
cd backend
python -m venv venv
# Windows: .\venv\Scripts\Activate.ps1 | macOS/Linux: source venv/bin/activate
pip install -r requirements.txt
python -m uvicorn app.main:app --host 127.0.0.1 --port 8000

# 4. Start React Frontend (in another terminal)
cd ../frontend
npm install
npm run dev
Enter fullscreen mode Exit fullscreen mode

Navigate to http://127.0.0.1:5173 and click any sample dataset to see the platform profile, discover, and explain the data in real time!


Code

ExperimentPilot AI v2.0: AI-Powered Dataset Behavior Discovery & ML Experiment Analysis Platform

"Upload your data. Discover how it behaves. Understand why the patterns matter. Decide what to investigate next."
Transforming raw datasets and ML experiment predictions into empirical discoveries, statistical evidence, and AI-grounded hypotheses.
Built for the Hacktoberfest Open-Source AI Challenge 2026: Week 2, Side Hustle Track.

Python FastAPI React TailwindCSS Ollama Tests License: MIT


1. Core Vision & Problem Statement

Data scientists, machine learning engineers, and analysts frequently receive unfamiliar tabular datasets. Today, they either spend days manually writing repetitive exploratory code (histograms, correlation matrices, outlier checks, clustering scripts) or paste raw column names into cloud chatbots that produce generic hallucinations.

ExperimentPilot AI bridges this gap by combining deterministic mathematical tools (Pandas, NumPy, SciPy, Scikit-learn) with local open-weight AI models (Ollama).

The user experience is effortless $$\textbf{Upload Any CSV} \longrightarrow \textbf{Automatic Profiling} \longrightarrow \textbf{Behavior Discovery} \longrightarrow \textbf{AI Explanation} \longrightarrow \textbf{Interactive Q&amp;A}$$

The user does not need to…

The codebase is built with strict modularity, separation of concerns, and full test coverage:

  • Backend Architecture (backend/app/):
    • dataset_profiler.py: Heuristic semantic typing, null counts, duplicate analysis, and 0–100 health scoring.
    • statistical_analyzer.py: SciPy & Scikit-learn computation for skewness, Pearson/Spearman correlation matrices, ANOVA $F$-tests, CramΓ©r's V, 2D PCA, K-Means clustering, Isolation Forest outliers, and Mutual Information baselines.
    • ai_investigator.py: Compact statistical evidence synthesis and Ollama prompt generation with JSON schema validation.
    • interactive_query_engine.py: Natural language intent classifier that dynamically generates answer narratives and Recharts chart payloads.
    • metrics_engine.py & error_analyzer.py: Deterministic model evaluation, confusion matrix generation, and sample quadrant error classification.
  • Frontend Architecture (frontend/src/):
    • React 18, Vite, TypeScript, and Tailwind CSS.
    • Interactive Recharts components for dynamic histograms, 2D PCA scatter plots, correlation tables, and inline chat visualizations.
  • Testing & Quality Assurance:
    • 20 / 20 unit and integration tests passing (pytest tests/ -v).
    • Covers CSV parsing, statistical calculations, Ollama JSON extraction, offline fallback handling, and REST endpoints.

How I Built It

The Open-Source AI Architecture

The central design philosophy of ExperimentPilot AI is "Deterministic Math First, Open-Source AI Second."

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                          HYBRID INTELLIGENCE PIPELINE                       β”‚
β”‚                                                                             β”‚
β”‚   1. Raw Data Ingestion (Pandas / NumPy / SciPy / Scikit-learn)            β”‚
β”‚      β”œβ”€β”€ Exact Distributions, Medians, Skewness, Kurtosis                   β”‚
β”‚      β”œβ”€β”€ Exact Correlations (Pearson, Spearman, ANOVA, CramΓ©r's V)          β”‚
β”‚      β”œβ”€β”€ Exact 2D PCA Coordinate Projection & K-Means Centroids             β”‚
β”‚      β”œβ”€β”€ Exact Isolation Forest Outlier Scores & Z-Score Deviations         β”‚
β”‚      └── Exact Mutual Information Feature Rankings                          β”‚
β”‚                                β”‚                                            β”‚
β”‚                                β–Ό                                            β”‚
β”‚   2. Evidence Synthesis Layer (Strict JSON Facts)                           β”‚
β”‚      └── Pre-formats only verifiable statistical facts (Zero Raw Data Sent) β”‚
β”‚                                β”‚                                            β”‚
β”‚                                β–Ό                                            β”‚
β”‚   3. Local Open-Weight Inference (Ollama)                                   β”‚
β”‚      β”œβ”€β”€ Model: Qwen 2.5 0.5B (Instruction-Tuned, 397MB footprint)          β”‚
β”‚      β”œβ”€β”€ System Role: Specialized Senior Data Science Investigator          β”‚
β”‚      β”œβ”€β”€ Task: Explain WHY patterns matter & propose testable hypotheses     β”‚
β”‚      └── Output: Structured JSON validated against Pydantic schemas         β”‚
β”‚                                β”‚                                            β”‚
β”‚                                β–Ό                                            β”‚
β”‚   4. Deterministic Graceful Fallback Engine                                 β”‚
β”‚      └── If Ollama is offline or uninstalled, algorithmic rules generate     β”‚
β”‚          a complete, structured narrative report with zero disruption       β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
Enter fullscreen mode Exit fullscreen mode

Why qwen2.5:0.5b via Ollama?

  • Ultra-Compact Footprint: At only 397MB, qwen2.5:0.5b can be pulled in seconds over any internet connection and runs comfortably in under 1GB of RAM.
  • Sub-Second Local Latency: Generates comprehensive multi-section behavioral interpretations and recommendations in 1–2 seconds on standard consumer laptop CPUs without requiring dedicated GPU hardware.
  • Strict Instruction Following: Consistently produces valid JSON conforming to our schema when provided with compact statistical evidence prompts.
  • Model Flexibility: Seamlessly swappable with larger open-weight models (llama3.2:1b, gemma2:2b, mistral, or deepseek-r1) via Ollama simply by selecting them in the UI status drawer.

Why Does Open Innovation Matter?

Open innovation is not just an implementation detail for ExperimentPilot AI β€” it is the fundamental reason the product can exist.

  1. Complete Data Sovereignty in Enterprise & Regulated Industries:
    In finance, healthcare, insurance, and legal sectors, uploading customer records, proprietary tabular data, or model prediction failures to cloud APIs (like OpenAI or Anthropic) violates HIPAA, SOC2, and non-disclosure agreements. By running open-weight models locally via Ollama, ExperimentPilot AI guarantees that 0 bytes of data ever leave the user's workstation.

  2. Zero Marginal Inference Cost for Bootstrapped Side Hustles:
    Closed APIs charge per token. A user analyzing multiple large datasets or asking dozens of questions would quickly rack up prohibitive API costs, forcing a developer to implement restrictive rate limits or charge high subscription fees. With open-weight local models, inference cost is $0.00 forever, creating unmatched operating margins for an open-core side hustle.

  3. Scientific Reproducibility & Immutable Auditing:
    Closed proprietary APIs are moving targets β€” models are updated, weights shift, and API endpoints are deprecated without notice. In scientific machine learning and regulatory model audits, results must be deterministic and verifiable. Open-weight models and open-source code ensure that analyses run today will yield identical results years into the future.

  4. Resilience & Offline Independence:
    Whether an engineer is on a flight, in a secure air-gapped facility, or working in an area with unstable connectivity, ExperimentPilot AI operates completely offline without degradation.


My Agent Session

Developing ExperimentPilot AI v2.0 involved architecting a multi-faceted system spanning statistical pipelines, unsupervised ML algorithms, local LLM prompt engineering, and reactive UI design.

Pairing with an agentic AI coding assistant accelerated this process exponentially:

  • Accelerated Test-Driven Development: Assisted in generating comprehensive unit tests for edge cases (e.g., zero-variance columns, missing values, high-cardinality IDs, and malformed CSVs), achieving a flawless 20/20 test passing record.
  • Mathematical Pipeline Hardening: Rapidly validated mathematical formulas across SciPy ANOVA implementations, CramΓ©r's V contingency calculations, and PCA variance projection formulas.
  • Interactive Component Crafting: Accelerated the construction of dynamic Recharts configurations that allow chat bubbles to render inline bar charts and scatter plots based on analytical intent routing.

My First User

User Profile: Alex R. (Independent ML Consultant & Data Strategist)

Alex is a freelance data consultant who builds predictive models and customer retention systems for mid-market e-commerce brands.

"Before ExperimentPilot AI, I had a custom Jupyter notebook that I'd copy-paste for every client engagement. It took me at least 4 to 6 hours just to profile the data, calculate correlations, check for leakage, and write up an initial brief.

I tested ExperimentPilot AI on a messy 12,000-row customer churn dataset. Within 20 seconds, the Behavior Discovery engine flagged an $r = +0.82$ collinearity alert between average order value and lifetime spend that I had missed β€” which would have leaked my model down the road. The PCA cluster personas gave me three distinct customer tiers right off the bat.

Even better: I exported the styled HTML report and presented it directly in my client kickoff call. The client was blown away by the clarity. This tool easily saved me half a day of tedious work. I would happily pay $19/month for the Pro tier if it integrates directly with my Git CI/CD pipelines to catch model regressions automatically."


Prize Categories

Primary Category:

  • Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle Track (An innovative, practical AI-powered product and open-core micro-SaaS built on open-source AI with a viable, realistic commercialization model, solving real-world pain points for freelance engineers, researchers, and regulated enterprise teams).

Focus & Partner Tracks:

  • Open-Weight Models & Local Inference: Powered entirely by qwen2.5:0.5b (and Ollama-compatible open weights) with 100% on-device privacy, zero cloud transmission, and deterministic fallback reliability.
  • Developer Tools & Productivity: Automates end-to-end dataset profiling, unsupervised behavior discovery, model disagreement analysis, and hypothesis formulation in under 5 minutes.

Thank you to the DEV Community, Hacktoberfest, and open-source AI contributors worldwide for fostering open innovation!

Top comments (0)