Data Autopsy: Finding What's Wrong With Your Dataset Before It Breaks Your ML Model

The Problem: A Dataset Can Look Fine and Still Be Dangerous

Most machine learning workflows start with a familiar routine: load a dataset, inspect a few rows, check its shape, handle missing values, and start training a model.

That workflow is fine for getting started, but it can leave behind problems that are not immediately obvious.

A dataset might contain duplicate records that distort its distribution. A numerical column might contain extreme values that deserve investigation. A categorical feature might represent the same value in several different ways. A classification target might be severely imbalanced.

And sometimes, a feature contains information that would not actually be available when making predictions in the real world.

That last problem is particularly dangerous: target leakage.

A model affected by leakage can produce impressive evaluation scores while failing badly on genuinely unseen data. It may not have learned the underlying pattern at all; it may have learned information that should never have been available.

These issues are often discovered through manual inspection, scattered notebook cells, or debugging sessions after a model has already been trained.

I wanted to build something that brings these investigations into one place.

That is the idea behind Data Autopsy, an open-source, AI-assisted forensic investigator for tabular datasets.

Instead of treating a dataset as just rows and columns, the goal is to investigate its structure, identify suspicious patterns, inspect the evidence, and understand what might be going wrong.

What Is Data Autopsy?

Data Autopsy is designed to inspect tabular data, identify potential data-quality and machine-learning risks, and present findings in a format developers can investigate.

The current focus is CSV-based datasets.

The workflow is straightforward:

1. Upload a CSV file.
2. Profile its structure and contents.
3. Run deterministic forensic checks.
4. Review findings by severity.
5. Inspect affected columns, measurements, and example rows.
6. Understand why a finding matters and what limitations apply.
7. Export a report for further analysis.
8. Optionally use a local language model to explain findings in plain language.

The objective is not to replace data scientists or automate every decision about data quality.

It is to reduce the manual work required to identify suspicious patterns and make the evidence behind those patterns easier to understand.

A useful investigation tool should answer more than "What is wrong?"

It should also help answer:

• Where did the issue appear?
• How many rows or columns are affected?
• What measurement triggered the warning?
• Why might this matter for downstream analysis?
• What should be checked next?
• Under what circumstances could this finding be harmless?

That distinction is central to the design of Data Autopsy.

Overall System Architecture

The project follows a simple architecture. A dataset enters through the loader, passes through profiling and independent forensic checks, and produces structured findings that can be displayed or exported.

The optional language model sits alongside this workflow rather than controlling the detection process.

flowchart TD
    A["CSV Upload"] --> B["CSV Loader"]
    B --> C["Dataset Validation"]
    C --> D["Dataset Profiler"]

    D --> E["Forensic Engine"]

    E --> F["Missing Values"]
    E --> G["Duplicate Records"]
    E --> H["Statistical Anomalies"]
    E --> I["Categorical Issues"]
    E --> J["Correlation and Leakage"]
    E --> K["Target Imbalance"]

    F --> L["Structured Findings"]
    G --> L
    H --> L
    I --> L
    J --> L
    K --> L

    L --> M["Streamlit Dashboard"]
    L --> N["Report Exporter"]
    L --> O["Optional Local LLM"]

    O --> P["Plain-Language Explanations"]
    M --> Q["Evidence Inspection"]
    N --> R["Markdown or JSON Reports"]

    classDef input fill:#e8f0fe,stroke:#4285f4,color:#172554
    classDef engine fill:#d1fae5,stroke:#047857,color:#064e3b
    classDef output fill:#fef3c7,stroke:#d97706,color:#78350f

    class A,B,C input
    class D,E,F,G,H,I,J,K,L engine
    class M,N,O,P,Q,R output

The important part is the separation of responsibilities.

The loader handles incoming data. The profiler describes the dataset. The forensic engine checks for suspicious patterns. Structured findings carry the evidence to the interface and reporting layer.

This makes the project easier to test and extend without tying every detector directly to the UI.

Why Deterministic Analysis Comes First

One of the most important architectural decisions is separating detection from explanation.

Data Autopsy uses Python-based statistical and rule-based checks to identify potential problems. These checks operate on the dataset and return structured findings containing measurements, thresholds, affected columns, and supporting evidence.

The language model is an optional explanation layer. It can translate technical findings into understandable language, but it does not decide whether a detector has found an issue.

This separation matters for several reasons.

Reproducibility: Given the same dataset, configuration, and detector version, deterministic checks should produce reproducible results.

Transparency: A finding should be traceable to a measurement or explicit rule rather than an unexplained model-generated statement.

Testing: Detection logic can be validated using synthetic datasets with known properties.

Privacy: The core investigation does not need an external AI API.

Graceful degradation: The application should remain useful when the local model is unavailable.

AI can make a technical report easier to understand, but it should not be responsible for inventing the underlying evidence.

Inside the Forensic Engine

The forensic engine is the core of the project. It runs a collection of checks across the dataset and produces structured findings that the interface and reporting system can consume.

The current implementation covers ten categories of investigation.

1. Missing Values

Missing data is one of the first things to inspect in any dataset, but the percentage of missing values is only part of the story.

A column with a few missing values may be easy to handle. A column with half its values missing may require a more deliberate decision. A column that is entirely empty may be unusable for its intended purpose.

Data Autopsy categorises missing-value findings using configurable thresholds, with higher severity for columns that are completely or substantially empty.

The report can include the affected column, missing count, missing percentage, and the threshold responsible for the finding.

Missing values are not automatically mistakes. In some datasets, the absence of a value carries meaning. The appropriate treatment depends on the source, the feature, and the downstream task.

2. Duplicate Records

Duplicate data can affect counts, distort distributions, and cause evaluation problems when related records appear across training and test sets.

The engine checks for exact duplicate rows and potential duplicate candidates based on selected non-identifier features.

These are different situations.

An exact duplicate means the relevant row values match. A potential duplicate means records resemble one another under a particular comparison rule.

Potential duplicates need more caution. Two people may share the same age and city. Two transactions may have identical amounts. Repeated measurements may be expected in time-series data.

For that reason, the findings should be treated as candidates for investigation rather than proof that records must be deleted.

3. Constant and Near-Constant Columns

A constant column contains only one distinct value. It usually provides no useful variation for a predictive model.

A near-constant column contains a dominant value in almost every row, with very little variation elsewhere.

Data Autopsy flags constant columns and columns where a single value dominates at least 99% of observations.

These features may be redundant, incorrectly populated, or simply part of the dataset's design. A nearly constant operational flag may still carry meaning in a particular application, so the tool reports the finding instead of silently removing the feature.

4. Potential Identifier Columns

Columns such as record IDs, unique codes, and generated identifiers can be mistaken for meaningful predictive features.

The current detector flags columns with very high uniqueness under defined minimum-sample conditions.

A unique identifier might be appropriate for joining tables or tracing a record. It may be inappropriate as a predictive input if it simply memorises individual observations.

The goal is to help users ask whether a feature represents meaningful information or merely identifies the row.

5. Numerical Outliers

Outlier detection is not as simple as identifying values far away from the mean.

Different datasets have different distributions. A high transaction amount might be suspicious in one context and completely ordinary in another. A naturally heavy-tailed variable may contain extreme values without containing errors.

Data Autopsy uses two statistical approaches:

• The interquartile range (IQR) method, using a configurable multiplier of 1.5.
• The modified Z-score, based on the median absolute deviation (MAD), with a threshold of 3.5.

Using more than one method provides different perspectives on unusually distant values.

Findings can include the number and percentage of flagged observations, relevant bounds or thresholds, summary measurements, and examples of extreme values.

These methods are screening techniques. They do not establish that a value is incorrect, fraudulent, or safe to remove.

6. High-Cardinality Categorical Features

Categorical columns with many distinct values can create problems for some preprocessing and modelling approaches.

A column containing thousands of different labels might represent free-text input, a product code, a user identifier, or a poorly normalised category.

The engine flags categorical features using configurable unique-value count and uniqueness-ratio thresholds.

High cardinality is not necessarily bad. A product catalogue may legitimately contain many product codes, and a geographic field may have many valid categories.

The finding serves as a prompt to inspect how the feature should be represented or encoded.

7. Inconsistent Categorical Representation

Data collected from multiple forms, systems, or manual inputs often contains variations that represent the same apparent category.

Examples include:

• Mumbai and mumbai
• Active and active
• Completed and COMPLETED

The current check looks for conflicting string representations after trimming surrounding whitespace and normalising case.

Such inconsistencies can increase the number of apparent categories and complicate grouping or encoding.

However, normalisation should not be applied blindly. Some datasets are case-sensitive by design, and different strings are not always interchangeable.

Data Autopsy identifies the inconsistency so the user can decide how to handle it.

8. Class Imbalance

In classification tasks, the target distribution can strongly affect training and evaluation.

A dataset with a large majority class and a small minority class may produce a model that appears accurate while failing to identify the cases that matter most.

Data Autopsy checks the distribution of a selected target and flags severe imbalance using configurable ratios. The current rules include thresholds of 4:1 and 20:1 for different severity levels.

These values are screening thresholds rather than universal definitions of an imbalanced dataset.

Class imbalance is common in fraud detection, medical screening, anomaly detection, and many other legitimate applications. The right evaluation strategy depends on the problem, the cost of different errors, and the number of available examples.

9. Highly Correlated Numerical Features

Strong correlations between numerical features can indicate redundancy or relationships worth investigating.

The engine identifies highly correlated pairs using Pearson correlation thresholds. To keep analysis manageable on larger datasets, the current implementation limits pairwise analysis to a subset of high-variance numerical columns and caps the number of rows considered.

This is a practical compromise between usefulness and computational cost.

Correlation is not proof of causation, and a high correlation is not automatically a data-quality problem. Some features are expected to move together because they measure related properties.

The report helps users locate strong relationships and decide whether they are expected, redundant, or potentially problematic.

10. Potential Target Leakage

Target leakage deserves particular attention because it can make model evaluation misleading.

Leakage occurs when a feature contains information about the target that would not legitimately be available at the time of prediction, or when information from the target or evaluation process enters training in an invalid way.

The current checks look for several suspicious signals, including features identical to the target, very strong numerical relationships, unusually strong categorical class purity, and high ROC-AUC scores in applicable cases.

These signals can identify features that deserve closer inspection.

They cannot prove leakage on their own.

For example, a feature may legitimately be a strong predictor. A numerical relationship may be expected in a scientific dataset. A categorical variable may nearly determine the target because the underlying process genuinely works that way.

A more complete leakage investigation should also consider how the data was collected, when each feature becomes available, how preprocessing is performed, and whether the evaluation setup reflects the real prediction scenario.

The following flowchart shows how a suspicious signal can lead to a more careful investigation rather than an automatic conclusion.

flowchart TD
    A["Select Prediction Target"] --> B["Identify Candidate Features"]
    B --> C["Run Leakage Screening"]

    C --> D["Check Target Copies"]
    C --> E["Measure Numerical Relationships"]
    C --> F["Inspect Categorical Class Purity"]
    C --> G["Evaluate Predictive Signal"]

    D --> H["Collect Evidence"]
    E --> H
    F --> H
    G --> H

    H --> I{"Suspicious Signal Found?"}

    I -- No --> J["Report Checks Performed"]
    I -- Yes --> K["Flag Feature for Investigation"]

    K --> L["Inspect Feature Meaning and Provenance"]
    L --> M["Check Feature Availability at Prediction Time"]
    M --> N["Review Preprocessing and Data Splitting"]
    N --> O{"Evidence Supports Leakage?"}

    O -- Yes --> P["Document Leakage Risk"]
    O -- No --> Q["Record Alternative Explanations"]
    O -- Unclear --> R["Mark as Unresolved"]

    P --> S["Recommend Further Action"]
    Q --> S
    R --> S

    classDef process fill:#e8f0fe,stroke:#4285f4,color:#172554
    classDef decision fill:#fef3c7,stroke:#d97706,color:#78350f
    classDef warning fill:#fee2e2,stroke:#dc2626,color:#7f1d1d
    classDef safe fill:#d1fae5,stroke:#047857,color:#064e3b

    class A,B,C,D,E,F,G,H,L,M,N,S process
    class I,O decision
    class K,P warning
    class J,Q,R safe

The long-term goal is to make this investigation more systematic without presenting suspicious correlations as definitive proof.

How a Dataset Moves Through the Investigation

The complete workflow begins with validating the input and ends with an evidence-backed report.

The diagram below illustrates the relationship between dataset profiling, detection, evidence inspection, optional AI explanations, and report generation.

flowchart TD
    A["Upload CSV"] --> B{"Dataset Readable?"}

    B -- No --> C["Show Validation Error"]
    C --> A

    B -- Yes --> D["Inspect Rows, Columns and Types"]
    D --> E["Profile Missing Values and Distributions"]
    E --> F["Run Deterministic Detectors"]
    F --> G["Collect Measurements and Evidence"]

    G --> H{"Findings Detected?"}

    H -- No --> I["Show Profile and Checks Completed"]
    H -- Yes --> J["Assign Severity and Recommendations"]

    J --> K["Group and Rank Findings"]
    K --> L["Inspect Affected Columns and Example Rows"]

    I --> M["Prepare Report"]
    L --> N{"Local LLM Available?"}

    N -- Yes --> O["Generate Evidence-Based Explanation"]
    N -- No --> P["Use Deterministic Fallback"]

    O --> Q["Review Findings"]
    P --> Q

    Q --> M
    M --> R["Export Markdown or JSON"]
    R --> S["Investigate and Decide Next Steps"]

    classDef process fill:#e8f0fe,stroke:#4285f4,color:#172554
    classDef decision fill:#fef3c7,stroke:#d97706,color:#78350f
    classDef output fill:#d1fae5,stroke:#047857,color:#064e3b

    class A,D,E,F,G,J,K,L,O,P,Q,M,R process
    class B,H,N decision
    class C,I,S output

This workflow also highlights an important distinction: a dataset that produces no findings is not necessarily problem-free. It means only that the implemented checks did not identify any issues under the current rules and configuration.

Findings Should Include Evidence, Not Just Severity

A dashboard that displays a red warning without explaining why is not particularly useful.

Data Autopsy uses structured findings designed to carry the context needed for investigation.

Depending on the check, a finding can include:

• A stable identifier and the name of the check
• A severity classification
• The affected columns
• Relevant row indices
• Measured values and detection thresholds
• An explanation of why the issue matters
• A recommendation for further investigation
• Known limitations of the detection method

This structure separates the result from the way it is displayed.

The dashboard can use it to filter and organise findings. The report exporter can use it to generate documentation. Tests can verify the measurements and thresholds. The explanation layer can use the evidence to produce a more readable summary.

Severity and confidence should remain distinct concepts. A potentially critical issue is not necessarily a confirmed issue, and a detector's strength depends on the evidence it can actually observe.

The Role of the Local Language Model

I wanted the project to explore practical AI integration without making a paid cloud API a requirement.

Data Autopsy is designed to use Ollama with Qwen2.5 1.5B as its default local explanation model, with a smaller model as a possible alternative.

The model is intended to explain existing findings in plain language. For example, it could describe why a high percentage of missing values may matter, what an extreme-value detector has measured, or which follow-up questions a developer should consider.

The underlying measurements still come from the deterministic engine.

This design has several advantages:

• Dataset analysis does not depend on a remote language-model service.
• The project can remain useful without the model.
• Model output can be checked against structured evidence.
• Users can experiment with local inference on supported hardware.
• An external AI provider is not mandatory for the core workflow.

A small model also has limitations. It may produce vague explanations, misunderstand context, or fail to follow a requested output structure. Local inference performance depends on the machine, model, and dataset size.

For that reason, model responses should be validated, kept within reasonable time limits, and backed by a deterministic fallback.

Local inference is a design choice, not a guarantee that every part of an application is automatically private. The application must still be checked for external requests, telemetry, logging, and other data-handling behaviour.

Local AI Explanation Architecture

The architecture keeps the deterministic findings independent from the optional AI explanation layer.

flowchart TD
    A["Uploaded Dataset"] --> B["Python Forensic Engine"]
    B --> C["Structured Findings"]

    C --> D["Evidence Validation"]
    D --> E["Dashboard and Report"]

    C --> F{"Ollama Available?"}

    F -- Yes --> G["Prepare Bounded Evidence Summary"]
    G --> H["Qwen2.5 1.5B"]
    H --> I["Validate Model Response"]
    I --> J{"Response Usable?"}

    J -- Yes --> K["Display Explanation"]
    J -- No --> L["Deterministic Fallback"]

    F -- No --> L

    L --> M["Display Baseline Explanation"]
    K --> N["User Reviews Evidence"]
    M --> N

    classDef process fill:#e8f0fe,stroke:#4285f4,color:#172554
    classDef decision fill:#fef3c7,stroke:#d97706,color:#78350f
    classDef local fill:#d1fae5,stroke:#047857,color:#064e3b

    class A,B,C,D,E,G,I,K,L,M,N process
    class F,J decision
    class H local

The diagram represents the intended behaviour. Model-output validation and fallback paths should be tested in the implementation before being treated as guaranteed features.

Technology Stack

The project uses a deliberately familiar Python stack.

• Python: Core analysis logic.
• Pandas and NumPy: Tabular data handling and numerical operations.
• SciPy and scikit-learn: Statistical utilities and modelling metrics where needed.
• Streamlit: Interactive application interface.
• Plotly: Visual exploration of distributions and relationships.
• Ollama and Qwen2.5 1.5B: Optional local language-model explanations.
• Pytest: Testing detectors, data handling, reporting, and related behaviour.

The aim is to keep the project approachable for contributors who already work with Python and data science tools, without requiring a complicated distributed architecture.

Keeping the Interface Useful

A forensic engine can produce many findings, but displaying everything at once creates a different problem: information overload.

The interface is being designed around the investigation workflow rather than a collection of decorative charts.

The main areas include:

• Overview: A summary of the dataset and the most important findings.
• Dataset Profile: Column types, missingness, distributions, and other structural information.
• Forensic Findings: A prioritised view of detected issues.
• Leakage Investigation: A focused view of target-related risks.
• Evidence Inspector: Measurements, thresholds, affected rows, and limitations.
• AI Investigator: Optional explanations generated from existing findings.
• Report Export: A way to preserve results for later review.

The visual direction is restrained and information-focused. Severity should be clear without making the entire interface look like an alert screen.

Charts should answer specific questions. A distribution plot should help someone understand a variable; a correlation view should make relationships easier to inspect.

Visualisations should not exist simply to make the application look more complicated. The same principle applies to metrics: show values calculated from the uploaded dataset, not fabricated example statistics presented as real results.

Testing the Investigation Engine

Deterministic checks are only useful if they behave predictably across ordinary and unusual inputs.

Testing therefore needs to cover more than one clean CSV.

Synthetic datasets are particularly useful because they let us introduce known conditions and verify whether the engine detects them.

Examples include:

• A dataset with a completely empty column
• Rows with exact duplicates
• A constant feature
• A feature containing extreme numerical values
• Categories represented using inconsistent casing
• A target with severe class imbalance
• A feature that duplicates the target
• A dataset containing legitimate strong predictors
• Boolean and datetime columns
• Small datasets and wide tables
• Columns with missing values or unexpected types

Tests should also verify that analysis does not silently mutate the original dataset and that edge cases do not crash the application.

Target-leakage checks need especially careful tests. A detector should not flag every legitimate strong predictor as leakage simply because it correlates with the target.

Report generation should be tested independently as well, so findings can be exported consistently even if the language model is unavailable.

The goal is not to claim that a test suite proves the application is flawless. It is to make detector behaviour explicit and catch regressions as the project evolves.

Performance and Practical Limits

A useful local tool should be responsive enough to run on ordinary development hardware.

Some checks are relatively inexpensive, while pairwise correlations and model-based leakage signals can become more expensive as the dataset grows.

Data Autopsy already uses bounds for some correlation calculations. Additional work should focus on identifying expensive operations, handling unsupported or mixed data types gracefully, and providing clear feedback when a dataset is too large for a particular analysis.

Potential improvements include sampling strategies where appropriate, configurable limits, caching reusable profile results, and clearer progress indicators.

Sampling must be handled carefully. It can improve performance, but it may miss rare anomalies or change the apparent distribution of a target. Any sampling strategy should be documented so users understand what was actually examined.

The project is focused on tabular data. It is not intended to replace specialised systems for image forensics, streaming anomaly detection, or large-scale distributed data processing.

What Data Autopsy Cannot Prove

A good forensic tool should be honest about its boundaries.

A missing value is not necessarily an error. A duplicate is not always redundant. An outlier is not necessarily invalid. A highly correlated feature is not necessarily leakage. A high-cardinality feature is not necessarily poorly designed.

Likewise, a dataset can pass every implemented check and still contain problems the engine cannot detect.

Some issues require domain knowledge, source-system context, temporal reasoning, or an understanding of how the data will be used in production. Statistical patterns can highlight suspicious behaviour, but they cannot always explain its cause.

Data Autopsy is therefore intended as an investigation assistant rather than an automatic data-cleaning system.

It should help developers find the places that deserve attention while leaving consequential decisions visible and reviewable.

Running It Locally

The project is intended to run locally with Python and a virtual environment.

A typical setup looks like this:

git clone <your-repository-url>
cd data-autopsy

python3 -m venv .venv
source .venv/bin/activate

pip install -r requirements.txt

Run the test suite:

.venv/bin/pytest -v

Start the application:

.venv/bin/streamlit run app.py

The optional explanation layer requires Ollama and a supported local model. For the current default model, the setup command is:

ollama pull qwen2.5:1.5b

The application should still support its deterministic analysis and reporting workflow when the model is not installed or running.

Use the repository's actual setup instructions as the source of truth if the commands or configuration change during development.

What's Next?

The next stage is about making the findings more useful and the investigation more trustworthy.

Stronger leakage analysis: Investigate suspicious features with more context, including data availability, target encoding, and evaluation methodology.

Better evidence inspection: Make it easier to move from a warning to the exact rows, measurements, and thresholds behind it.

Reproducible reports: Include enough metadata about the dataset and checks to make results easier to review later, while avoiding unnecessary exposure of sensitive row-level data.

Useful demo datasets: Provide small datasets that demonstrate specific problems without relying on hardcoded findings or misleading screenshots.

More robust testing: Expand coverage for multiclass targets, small samples, unusual types, and legitimate but suspicious-looking relationships.

Contributor-friendly detectors: Keep the detector interface consistent so new checks can be added without tightly coupling them to the UI.

These are directions for improvement, not claims that every capability is already complete.

Why I'm Building This as Open Source

I like projects that solve a concrete engineering problem and can be understood by someone reading the code for the first time.

Data Autopsy brings together statistical checks, evidence-oriented reporting, a local AI explanation layer, and an interface for investigating tabular datasets.

It is also a useful way to explore an important question in AI engineering:

How do we use language models to make tools more understandable without letting them become an unverified source of truth?

I want the deterministic parts to be inspectable, the limitations to be documented, and the project to remain useful without a paid service.

If you work with machine learning datasets, I'd be interested in hearing which checks you would add, which thresholds you would change, and what evidence you need before trusting a warning.

Contributions, issues, and constructive criticism are welcome.

Repository: https://github.com/Daksh-create349

The project is still evolving. The goal is not to promise that every dataset problem can be found automatically; it is to make investigating those problems more systematic, transparent, and accessible.

Sometimes, before training a better model, we need to understand the data we already have.