This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine
What I Built
Ghost Records is a local-first AI data quality auditor that hunts down the ghosts hiding in your datasets.
Real-world data is rarely perfect. Duplicate records, orphaned foreign keys, missing values, and broken relationships can quietly cause problems in applications and databases. Ghost Records helps uncover these issues and understand what went wrong.
The application scans CSV files and SQLite databases, detects anomalies using deterministic data-quality rules, and uses a locally running AI model to explain findings in plain language and suggest possible fixes.
Instead of treating every anomaly as an error to delete, Ghost Records provides evidence, remediation suggestions, and a human-reviewed workflow.
I interpreted the Ghost in the Machine theme literally: sometimes the ghosts aren't supernatural entities—they're the records nobody knows are still there.
Demo
Code
yashr5120
/
Ghost-Records
Ghost Records is a local-first AI data quality auditor that detects duplicate rows, orphaned records, broken relationships, and suspicious values in CSV files and SQLite databases. Powered by local Gemma inference via Ollama, it explains anomalies and suggests safe fixes while keeping your data private.
👻 Ghost Records
“Hunt down the ghosts haunting your data.”
Ghost Records is a local-first AI data quality auditor that unmasks duplicate records, orphaned foreign keys, colliding primary keys, invalid dates, and statistical anomalies in CSV files and SQLite databases.
A locally running open-weight LLM (Gemma via Ollama) provides evidence-grounded forensic explanations and suggests safe, reviewable remediation without transmitting a single row to a third-party cloud API.
📌 Problem Statement
Real-world datasets are plagued by integrity flaws:
- Duplicate records inflating transaction metrics.
- Orphaned foreign keys referencing customer or product IDs that no longer exist.
- Colliding primary keys causing silent row overwrites.
- Missing or whitespace-only values in required operational columns.
- Illegal negative numbers or malformed date strings.
Traditional data quality platforms demand complex configurations, external cloud data warehouses, or expensive hosted AI APIs that violate data confidentiality.
Ghost Records solves this by bringing deterministic forensic audits and open-weight AI explanation…
How I Built It
Ghost Records combines traditional data engineering with open-weight AI.
Technology stack:
Python: Core application logic and data-quality checks.
Streamlit: Interactive investigation dashboard.
Pandas: Data profiling and anomaly detection.
SQLite: Local metadata and scan history.
Ollama + Gemma: Local AI inference for anomaly explanations.
pytest: Automated testing of detection rules and application behavior.
The architecture separates detection from explanation. Deterministic rules identify anomalies and produce evidence; the local language model uses that evidence to explain the problem and suggest appropriate remediation.
This separation makes the results more reproducible and helps prevent the model from inventing missing identifiers or treating unverified assumptions as facts.
Why Does Open Innovation Matter?
Data quality tools often work with sensitive information, including student records, HR data, financial transactions, and customer databases.
Sending that information to a hosted AI API may be unnecessary or unacceptable.
Ghost Records explores a different approach: bring the AI to the data, rather than sending the data to the AI.
Using an open-weight model through local inference makes it possible to:
Keep data local: Avoid sending datasets to an external generative AI provider in the local configuration.
Reduce dependency on paid APIs: Run AI explanations without requiring a hosted LLM subscription.
Customize the experience: Modify prompts, detection rules, and model behavior.
Encourage community contributions: Add new anomaly detectors, database adapters, and model integrations.
Improve transparency: Show the evidence behind each finding instead of presenting an AI-generated answer as unquestionable truth.
Open innovation also means that developers can inspect, extend, and adapt the system to their own data-quality needs.
The goal isn't to replace deterministic validation with AI. It's to combine reliable rules with a locally available model that makes data-quality investigations easier to understand.
The Spirit of It
Every dataset has a story. Some records are valid, some are duplicates, and some point to entities that no longer exist.
Ghost Records is about finding those forgotten pieces of data before they turn into bigger problems.
The idea behind the project is simple: data problems shouldn't remain mysterious. They should be detectable, explainable, and fixable—with the user in control.
Sometimes, the scariest thing in a database is the record nobody remembers creating.
Prize Categories
Best Use of Open Source AI
Best use of gemma
Top comments (0)