DEV Community: Muhammed Rasin O M

Half the answer keys in text-to-SQL benchmarks are wrong. So I generated the database from the answer key.

Muhammed Rasin O M — Fri, 10 Jul 2026 13:02:47 +0000

Earlier this year a team at UIUC audited the benchmarks the text-to-SQL field ranks itself on. The numbers in the paper (Jin et al., VLDB 2026) stopped me cold: 52.8% of BIRD Mini-Dev annotations and 62.8% of Spider 2.0-Snow annotations are wrong. Not ambiguous. Wrong, as adjudicated by human SQL experts after an agent flagged them. When the authors corrected just 100 examples and re-evaluated 16 open-source agents, relative performance swung by as much as 31% and leaderboard positions moved by up to 9 places.

Every team that picked a data agent off those leaderboards was, to a measurable degree, choosing based on noise.

I build synthetic data tooling as a side quest, and my whole library is organised around one guarantee: declared numbers hold exactly in generated data. Reading that paper, it was hard to miss that the two problems are mirror images. Benchmarks struggle to attach correct answers to existing databases. My engine attaches a database to declared answers. So I spent this week building the inversion, and this post walks through the mechanism, the verification, and the parts that did not survive their own verification, because those are the most instructive.

Why annotation-after-the-fact fails structurally

The error rates are not annotator laziness. Look at how every benchmark in this lineage gets built. A database exists first, scraped or donated or assembled. Then question writers read schemas they did not design and write natural-language questions. Then annotators produce the gold SQL and expected answers by interrogating data full of null conventions, encoding quirks, and domain definitions they can only infer.

The UIUC audit's taxonomy shows exactly where that leaks. Semantic mismatches between question and SQL, like a strict inequality in the question compiled as an inclusive BETWEEN. Schema misunderstandings: missed aggregations, wrong joins, a filter on the wrong status column. Domain knowledge errors, their example being an annotator treating K-12 as equivalent to grades 1 through 12. And genuinely ambiguous questions with several defensible readings.

Strip away the details and all of these share one structural property: they can only exist because the answer was derived after the data, by a human reading it. The answer key is a claim about the database, and claims can be wrong.

Inverting the construction

So make the answer key not a claim but a constraint.

Declare the ground truth first, as a specification: exact monthly revenue for calendar 2025, a fraud rate rising from 2% to 3.5%, two foreign-key relationships that must have zero orphans. Then generate a database that satisfies the specification. Questions are derived from the declaration. The expected answer for "total order amount in November 2025" is not something anyone computed by reading rows. It is $750,000.00 because the data was constructed so that it would be.

The generation math matters here, so briefly: for each declared period the engine allocates a row count from the target and the average transaction value, draws per-row amounts from a lognormal around that average, then rescales the period so it sums to the target exactly. Two stages, allocation then scaling, no learned model anywhere in the path, fully deterministic given a seed. The method is written up in the Misata paper; the engine is open source. The property that makes it usable for evaluation is that the aggregate guarantee survives all the way to files on disk: modern float serialization round-trips exactly, so the CSV a grader reads carries the same cent-exact sums the generator produced.

Generators don't get to grade their own homework

Here is the part I consider non-negotiable, because "the generator says the generator is right" should convince nobody.

After the CSVs are written, every candidate question's gold SQL is executed by DuckDB against those files, the ones that actually ship, and the observed answer is compared with the declared one. DuckDB shares no code with my engine. The two systems agree on nothing except the files. Each shipped question looks like this, from the actual pack:

{
  "id": "q011",
  "question": "In the orders table, what is the total of amount during November 2025 (rows whose timestamp is on or after 2025-11-01 and strictly before 2025-12-01)? Give a number rounded to 2 decimal places.",
  "gold_sql": "SELECT ROUND(SUM(\"amount\"), 2) FROM \"orders\" WHERE \"order_date\" >= TIMESTAMP '2025-11-01 00:00:00' AND \"order_date\" < TIMESTAMP '2025-12-01 00:00:00'",
  "expected_answer": 750000.0,
  "answer_type": "number",
  "round_decimals": 2
}

Notice the question states its window as half-open and names its rounding. That is deliberate. Most of what the audit classified as ambiguity comes from questions that leave boundaries and output format to the reader's imagination. When you author the question from the declaration instead of from the data, you can simply say what you mean.

The demonstration pack is public: github.com/rasinmuhammed/evalpacks. A 73,629-row orders table plus customers and products, 35 shipped questions, and a certificate recording the DuckDB version, the seed, the spec hash, and every per-question verification result. Re-checking it needs one dependency and about thirty seconds:

pip install duckdb
python pack/verify.py
# 35/35 verified exactly

Edit one amount in one CSV and the script exits nonzero. That is the entire trust model. There is no leaderboard to take on faith, just a check anyone can run against the artifact itself.

The questions that did not survive

The pack shipped 35 questions. The spec produced 40 candidates. The other five are my favorite part of the artifact.

Declared rates collide with integer arithmetic. January was declared at a 2% fraud rate, and the January allocation came out to 4,824 orders. Two percent of 4,824 is 96.48 orders, and you cannot flag half an order. The achievable neighbors are 96 flagged rows, which is 1.99%, or 97, which is 2.01%. The verification gate measured 1.99% against a declared 2.00% and refused to ship the question. Same story for four other months:

period	declared rate	achieved rate	shipped?
2025-01	2.00%	1.99%	dropped
2025-06	2.68%	2.69%	dropped
2025-08	2.95%	2.96%	dropped
2025-10	3.23%	3.22%	dropped
2025-11	3.36%	3.37%	dropped

Seven other monthly rate anchors happened to be exactly achievable at four decimal places, and those shipped. All five failures are recorded in the manifest with their observed values, next to the full generation spec.

I want to be precise about what this section is, because a reader could mistake it for an admission. A benchmark that publishes what it refused to include, with the measured reason, is making a stronger claim than one that ships everything: it is telling you the shipping bar is exact verification and showing you the bar rejecting things. The alternative, quietly shipping a question whose true answer is 1.99% with an answer key that says 2%, is precisely the failure mode the UIUC paper documented at scale.

Regeneration, or why contamination stops mattering

BIRD and Spider are in every frontier model's training corpus by now. That is not an accusation, it is arithmetic: benchmark questions get indexed, discussed, and absorbed. Static benchmarks depreciate.

An evalpack is a spec plus a seed, so it regenerates. Change the seed and you get a different database that satisfies the same declared answers. From my own run while writing this post:

seed 20260710: November total = 750,000.00 over 73,629 orders
seed 99:       November total = 750,000.00 over 73,629 orders
shared order_ids between the two databases: 544

Two databases, essentially disjoint rows (the 544 shared IDs are random collisions in a ten-million-value ID space), identical declared answers, and the seed-99 pack re-verifies 35/35 on its own certificate. If you suspect a model has seen your eval data, you rotate the seed and the answer key survives. No annotation cycle, no re-adjudication.

There is a second contamination defense hiding in plain sight: every entity in the data is plausible fiction. No model can answer "which month had the highest revenue" from memorized world knowledge about a retailer that does not exist. It has to query. For evaluation data, fictional values are not a compromise. They are the point.

Rewards for RL, where wrong labels hurt twice

If you train data agents with reinforcement learning, ground-truth noise is worse than in evaluation, because the policy optimizes toward the noise. The large synthetic SQL corpora used for training measure around 86% full correctness by their own published human evals, and RL on a 14%-wrong reward signal is a strange thing to spend GPUs on.

A verified evalpack is, from an RL perspective, a reward function with a certificate: exact-match rewards whose correctness was checked by an independent engine before training started, generable at whatever scale and difficulty the spec dials in. Join depth, aggregation complexity, and temporal structure are parameters, not accidents of whichever database was available. I think this is where answer-key-first construction ends up mattering most, and it is the part I would most like to compare notes on with people building agent environments.

Design choices a skeptic should ask about

Three decisions define the current scope, and each was made with eyes open.

Question families cover declared aggregates, per-period counts, rates, extrema, and referential integrity, not arbitrary SQL. That is what the spec language can declare today. The right way to extend coverage is to extend the declaration language, keeping the property that every shipped answer traces to a declared constraint, rather than annotating free-form questions and reintroducing the original disease.

Natural-language ambiguity is reduced, not eliminated. Authoring questions from the declaration lets every question state its window, its rounding, and its tie-breaking rule explicitly, which removes the largest documented ambiguity sources. Wording is still wording, and I will not claim a theorem where I only have a discipline.

And this complements human-authored benchmarks rather than replacing them. Real-world schema messiness, dialect coverage, the long tail of enterprise weirdness: existing benchmarks carry that, and correcting them, as the UIUC team is doing, stays essential. Answer-key-first construction owns the complementary territory: the error classes where annotation-after-the-fact structurally fails, at whatever scale you need, contamination-free by rotation.

Try it

Everything is MIT licensed. The pack, certificate, and verifier are at github.com/rasinmuhammed/evalpacks. Building your own takes one install and one command:

pip install 'misata[evalpack]'
misata evalpack --config your_schema.yaml -o your_pack --seed 42

The declaration language, integrity proofs, and the rest of the engine are documented at misata.studio/docs, and there is a no-code version at misata.studio if you want to go from a plain-English description to a generated database without writing the schema by hand.

If you build evaluations for data agents, or RL environments that contain databases, and this construction is either useful to you or broken in a way I have not noticed, I genuinely want to hear it. Open an issue on the repo. The fastest way to make the answer key trustworthy is to keep trying to break it.

Synthetic Data for Data Engineering: How to test a Pipeline before the real data arrives

Muhammed Rasin O M — Sat, 20 Jun 2026 15:17:34 +0000

There is a quiet absurdity at the center of most data work, and once you notice it you cannot stop seeing it.

You are asked to make sure a pipeline is correct. To do that, you need data to run it on. The only data that would truly prove anything is the real data, the production tables with their odd shapes and their long tails and their one customer in Belgium who somehow has forty thousand orders. And that is exactly the data you are not allowed to have. It sits behind an access request, a compliance review, a privacy policy, and a Slack thread that goes quiet after someone says "let me check with legal."

So you build the thing in the dark. You write twenty rows by hand and tell yourself they represent a million. You run a loop of fake names and random numbers and watch your joins quietly drop half the rows because none of the foreign keys point anywhere. You ship, and you wait, and you find out whether you were right when the real data finally lands and something downstream catches fire.

This is the part nobody puts in the job description. Most of data engineering is reasoning about data you cannot look at.

Synthetic data is usually sold as a machine learning trick. Train a model without touching real records. Get around the privacy rules. Pad out a dataset that is too small. All of that is real. But it hides the use that shows up far more often, in the ordinary Tuesday-afternoon work of building and testing pipelines. And the reframe that makes it click is small and a little beautiful.

The data whose answers you already know

Here is the inversion. Normally data comes first and you spend your life trying to discover its truths. You profile it, you chart it, you write tests that hope to catch it lying. The data is the authority and you are the supplicant.

Generated data flips the direction of that relationship. You state a truth, and the data assembles itself to honor it.

Say you are testing a fraud aggregation. You run your pipeline against some test data and it reports three point one percent fraud last month. Is that correct? You have no idea. You do not know the true fraud rate of your test data, so the number is unfalsifiable. The test can confirm the code ran. It cannot confirm the code is right.

Now turn it around. Generate test data where you decided the fraud rate in advance. One point eight percent in January, climbing to four point one by June. Run the pipeline. If the gold table comes back at one point seven nine and four point oh nine, then your joins held, your filters were honest, your aggregation counted what it was supposed to count. If it comes back at two and a half, something in the middle is dropping or double counting fraud, and you caught it on a laptop, on a branch, before a single real record was at risk.

That is a real test for a data pipeline. Not "did it run" but "is it correct." And you cannot write it with a fake-data loop, because a fake-data loop cannot hand you a target to check against. The value here is not that the data looks real. It is that you wrote the answer key first.

What this actually does on an ordinary day

A few jobs where this earns its place.

Proving a pipeline is correct, on every commit

Declare the tables, the relationships between them, and the outcome you care about. Generate. Assert that your transformation reproduces the outcome. Here is the shape of it, using Misata, an open source library I build (so take the pitch with the appropriate pinch of salt, though the idea holds whatever you reach for):

import misata
from misata import from_dict_schema

schema = from_dict_schema({
    "__rate_curves__": [{
        "table": "transactions", "column": "is_fraud",
        "time_column": "txn_ts", "time_unit": "month", "true_value": True,
        "rate_points": [
            {"period": "2025-01", "rate": 0.018},
            {"period": "2025-06", "rate": 0.041},
        ],
    }],
    "customers": {
        "id": {"type": "integer", "primary_key": True},
        "email": {"type": "email"},
    },
    "transactions": {
        "id": {"type": "integer", "primary_key": True},
        "customer_id": {"type": "integer",
                        "foreign_key": {"table": "customers", "column": "id"}},
        "amount": {"type": "float", "distribution": "lognormal", "mu": 3.4, "sigma": 1.1},
        "txn_ts": {"type": "datetime", "start": "2025-01-01", "end": "2025-06-30"},
        "is_fraud": {"type": "boolean"},
    },
})

tables = misata.generate_from_schema(schema)

Every transaction points at a customer who exists. The monthly fraud rate follows the curve you declared. Run your silver and gold transforms on top, then assert the gold output lands on that curve within a tolerance. Wire it into CI and it runs forever, with no access to production and nothing to leak.

Making a copy of data you have but cannot share

Sometimes the real table is right there on your screen and the problem is the opposite. You need to send it somewhere it should not go. To a vendor, into a public demo, onto the laptop of someone who started on Monday. The job now is a copy that keeps the statistical shape and the relationships between columns while carrying none of the actual rows.

import pandas as pd
import misata

real = pd.read_csv("customers.csv")
synthetic = misata.mimic(real, rows=len(real))["table"]

print(misata.fidelity_report(synthetic, real, target_column="churned").summary())
print(misata.privacy_report(synthetic, real).summary())

The two report lines are the point. A copy is only worth using if you can measure that it matches the real distributions and the relationships inside them. It is only worth sharing if you can measure that it is not quietly memorizing real people. Numbers you can put in a pull request, not a feeling you defend in a meeting.

Honesty belongs here too, because this is the hard part of the whole field. Simple tabular structure reproduces beautifully: distributions, correlations between columns, values that shift by category. Complicated nonlinear or geographic structure is genuinely harder, and the right move is to measure the gap rather than pretend it closed. If your need is deep generative fidelity on messy real data, the heavier synthesizers like SDV have spent years on exactly that.

Filling the time before the data exists

Two more, quickly. Load testing wants volume with a believable shape, so you can watch a pipeline behave at a hundred million rows before the business actually sends them. And teams before launch have no data at all, yet still have to build the dashboard and train the first model. Declared synthetic data covers both, and since you set the dials, you know what every downstream number is supposed to say.

Where it is the wrong tool, plainly

It will not find a bug you did not think to model. If an upstream system encodes dates in a way nobody documented, synthetic data built from a clean schema will be innocent of that sin, and your pipeline will meet it for the first time in production. Use generated data to prove your logic against known inputs, not to discover the unknown.

It does not replace an integration test against the real source systems. It exercises your transformations, not your connectors.

And a copy learned from real data is still a descendant of real data. Measure the privacy distance before you hand it over. Assume nothing.

The small wonder of it

Come back to that two in the morning feeling, the green pipeline you did not trust. The reason you did not trust it was that nothing in the test could have told you it was wrong. The data had no opinion about the right answer.

What changes, when you generate data on purpose, is that the data starts to carry an opinion. You write down what should be true, the rows arrange themselves around it, and then your pipeline has to earn that truth back out the other end. The test can finally fail for the right reasons. That is a strange and slightly wonderful thing to be able to do, to know the answer before the data exists, and it turns the loneliest part of data engineering, reasoning in the dark about tables you cannot see, into something you can actually check.

If you want to try the examples, Misata is pip install misata, and there are runnable Databricks notebooks in the repository. Whatever you use, give your pipelines a test that knows the answer.

The Best Python Library for Generating Quick Synthetic Data in 2026

Muhammed Rasin O M — Sat, 11 Apr 2026 14:22:09 +0000

Misata: Generate Realistic Synthetic Datasets From Plain English Descriptions

Generating synthetic data in Python used to mean one of three things: write random.uniform() loops by hand, use Faker for fake names and emails, or spend a week configuring SDV on top of real data you might not even have. But we have got LLMs now. Still maintaining the logics and the referential integrity is a nightmare.

Misata is none of those things.

One sentence in. Multiple related tables out. Distributions calibrated to real-world statistics. Foreign key integrity guaranteed. Monthly revenue targets hit to the cent.

pip install misata

import misata

tables = misata.generate(
    "A SaaS company with 2000 users. "
    "MRR rises from 80k in January to 320k in June, "
    "drops to 180k in August due to churn, "
    "then recovers to 400k in December.",
    seed=42,
)

That generates two linked tables with 21,000+ rows. Here is what the monthly MRR looks like when you sum the rows:

Jan    $80,000   ✓
Feb   $128,000   ✓
Mar   $176,000   ✓
Apr   $224,000   ✓
May   $272,000   ✓
Jun   $320,000   ✓
Jul   $250,000   ✓
Aug   $180,000   ✓   <- churn dip, as described
Sep   $235,000   ✓
Oct   $290,000   ✓
Nov   $345,000   ✓
Dec   $400,000   ✓

Every target exact. Not approximate. The individual rows still follow a log-normal distribution (median MRR $126, mean $150, p90 $291) because that is what real SaaS revenue looks like. But the monthly totals are pinned to whatever story you gave it.

The core problem: why most synthetic data is useless

There's a gap between what synthetic data generators produce and what you actually need to build, test, or demo a data system.

Uniform distributions lie. Real revenue data is log-normal. Real fraud rates hover around 2%, not 50%. Real product category distributions follow Zipf's law - one category dominates, the others trail off. When your fake data looks nothing like the real thing, your model trains on lies, your dashboards tell wrong stories, and your tests pass cases that would fail in production.

Referential integrity breaks things. If you're testing a JOIN across customers and transactions, orphan foreign keys will silently ruin your results. Most data generators either skip relational structure entirely or produce it inconsistently.

Business targets get ignored. You don't just want data that looks roughly right. You want a dataset where Q3 revenue dips 22% due to a simulated product recall, or where churn spikes in August because your description says so. No general-purpose generator can do this.

Misata was built specifically to close this gap.

Why distributions matter more than people think

Most fake data generators produce values that are uniformly distributed. When you plot them, everything looks flat. Real business data is never flat.

Misata ships calibrated distribution priors for seven domains. Here is what that means in practice.

Fintech: fraud rates, credit scores, and account balances

tables = misata.generate(
    "A fintech company with 2000 customers and banking transactions.",
    seed=42,
)

transactions = tables["transactions"]
print(f"Fraud rate: {transactions['is_fraud'].mean() * 100:.2f}%")

Fraud rate: 2.00%

400 fraudulent transactions out of 20,000. The calibrated real-world baseline for card fraud is around 2%. That is what you get. Not a random number. A calibrated one.

Credit scores follow the actual US distribution:

mean:   679   (real US average: 680-720)
std:     80   (real range: 70-90)
min:    328
max:    850

Account balances follow log-normal because real bank balances do:

median     $1,976
mean       $6,128
p90       $14,260
p99       $62,565

Most customers have under two thousand dollars. A few have tens of thousands. The tail is real. This matters enormously if you're building fraud detection models, credit scoring pipelines, or stress-testing payment infrastructure — a flat distribution would make every one of those models overfit to a distribution that doesn't exist in production.

Healthcare: blood type frequencies, age distributions, and appointment patterns

tables = misata.generate("A hospital with 500 patients and doctors.", seed=42)
patients = tables["patients"]

Blood type    Generated    Real-world
O+               37.9%        38.0%   ✓
A+               33.9%        34.0%   ✓
B+                9.6%         9.0%   ✓
AB+               3.0%         3.0%   ✓
O-                6.5%         7.0%   ✓
A-                6.1%         6.0%   ✓
B-                2.0%         2.0%   ✓
AB-               0.9%         1.0%   ✓

All eight blood types within 0.6% of the actual ABO/Rh frequency distribution. Patient ages center on 45 with a standard deviation of 18, matching a chronic-care hospital population. Nobody configured any of this. It is what the healthcare domain prior knows.

This level of epidemiological accuracy is essential when you're training triage models, testing EHR systems, or building health analytics pipelines that will eventually run on real patient data.

Ecommerce: Zipf categories, seasonal peaks, and return rates

schema = misata.parse(
    "An ecommerce store with 5000 customers and orders. "
    "Revenue grows from 100k in January to 300k in November "
    "then 350k in December.",
    rows=5000,
)
tables = misata.generate_from_schema(schema)

Product categories follow Zipf's law because that is how real shopping behavior works:

electronics      47.1%
clothing         20.0%
home & garden    12.3%
sports            8.7%
books             6.5%
beauty            5.5%

One category dominates. The rest trail off. Uniform would give you ~17% each. Real shopping does not look like that.

Order statuses come with realistic rates:

completed    71.5%
shipped      12.4%
pending       8.2%
returned      5.0%
cancelled     3.0%

Real e-commerce return rates are 8–10%. That is what gets generated. If you're building a returns processing pipeline, this means your test data will actually stress the right code paths.

Referential integrity across all tables

Every child table samples foreign key values from the actual parent pool. This means zero orphan rows by construction, not by luck.

tables = misata.generate(
    "A fintech company with 2000 customers and banking transactions.",
    seed=42,
)

customers    = tables["customers"]     # 2,000 rows
accounts     = tables["accounts"]      # 2,600 rows
transactions = tables["transactions"]  # 20,000 rows

# Both FK edges hold
orphan_accounts = (~accounts["customer_id"].isin(customers["customer_id"])).sum()
orphan_txns     = (~transactions["account_id"].isin(accounts["account_id"])).sum()

print(orphan_accounts)  # 0
print(orphan_txns)      # 0

Tables are generated in topological dependency order. Parents first. Children sample from the completed parent pool. It cannot produce orphans.

This matters for any workflow that involves JOINs. Referential integrity errors in test data produce false negatives — your pipeline looks like it works until it meets real data.

The two-step flow for more control

When you want to inspect or modify the schema before committing to generation:

schema = misata.parse("A hospital with 500 patients and doctors.")
print(schema.summary())

Schema: Healthcare Dataset
Domain: healthcare
Tables (3)
  doctors         25 rows    [doctor_id, first_name, last_name, specialty, years_experience]
  patients       500 rows    [patient_id, name, age, gender, blood_type, registered_at]
  appointments  1500 rows    [appointment_id, patient_id, doctor_id, type, duration_minutes]

Relationships (2)
  patients.patient_id  -> appointments.patient_id
  doctors.doctor_id    -> appointments.doctor_id

Adjust the seed, add columns, change row counts. Then generate. The two-step flow is useful for teams where a data engineer defines the schema and a developer generates data against it — the schema becomes a shared artifact you can version control.

Real-world use cases

Use case 1: Training ML models without access to production data

Privacy regulations — GDPR, HIPAA, CCPA — make it difficult or impossible to use real user data for model training in many industries. The usual workaround is anonymization, but anonymized data often loses the statistical properties that make it useful for training.

Misata generates statistically calibrated data with no PII at all. A fraud detection team can produce 500,000 transactions with a realistic 2% fraud rate, a plausible credit score distribution, and calibrated account balance tails — without touching a single real customer record.

tables = misata.generate(
    "A fintech company with 50000 customers and banking transactions. "
    "Fraud rate is 2%. High-value accounts above 50k balance are 3x more likely to be targeted.",
    seed=42,
)

The model trains on data that behaves like production data. The privacy risk is zero.

Use case 2: Seeding development and staging databases

Every new developer joining a product team hits the same wall: the development database is empty or has three test rows from 2019. You can't build features that depend on realistic data patterns without realistic data.

Misata can seed a full development database in seconds:

from misata import seed_database

tables = misata.generate("A SaaS company with 1000 users.", seed=42)
report = seed_database(tables, "postgresql://user:pass@localhost/mydb", create=True)
print(report.total_rows)  # 12,400

Or from the CLI, which makes it easy to add to a Makefile or docker-compose setup:

misata generate \
  --story "A SaaS company with 1000 users" \
  --db-url postgresql://user:pass@localhost/mydb \
  --db-create --db-truncate

SQLite works too for local-only development:

misata generate \
  --story "A SaaS company with 1000 users" \
  --db-url sqlite:///./dev.db \
  --db-create --db-truncate

A new developer can run make seed-db and have a working dataset in their environment in under 10 seconds.

Use case 3: Building product demos without real customer data

Sales engineering teams routinely need to demo analytics dashboards, CRM systems, and data products to prospects. Using real customer data for demos is a legal and ethical non-starter. Using hand-crafted fake data means someone spends two days building a CSV in Excel.

Misata lets you generate a compelling, internally consistent demo dataset for any domain:

tables = misata.generate(
    "A B2B SaaS company with 800 enterprise customers. "
    "ARR grows from 2M in Q1 to 5M in Q4. "
    "Average contract value is 6000. Churn rate is 8%.",
    seed=42,
)

The result is a dataset where every KPI in the demo dashboard reflects a plausible business trajectory — not a random scatter of numbers.

Use case 4: Testing data pipelines and ETL systems

Data pipeline tests are only as good as the data they run on. Edge cases like NULL foreign keys, skewed distributions, and outlier values are exactly what break pipelines in production — and exactly what hand-crafted test data tends to miss.

Misata's calibrated distributions naturally produce the tail values that stress-test pipelines:

tables = misata.generate(
    "A logistics company with 10000 shipments. "
    "Include delayed deliveries at a 12% rate. "
    "International shipments are 30% of total volume.",
    seed=42,
)

The p99 values in account balances, the occasional NULL in optional fields, the rare blood type AB- at 1% frequency — these are the values that reveal pipeline brittleness.

Use case 5: Generating benchmark datasets for academic and research use

Researchers publishing papers on data systems, query optimizers, or ML benchmarks need datasets that are reproducible, realistic, and free of privacy concerns. Misata's seed parameter makes generation fully deterministic:

tables = misata.generate(
    "A marketplace with 5000 buyers and sellers, orders, and product listings.",
    seed=42,  # Anyone running this gets the exact same dataset
)

Share the seed and description in your paper. Readers can reproduce your exact dataset with a single Python call.

Use case 6: Prototyping data products and BI dashboards

Before you connect a BI tool to production data, you need something to build against. Misata gives you a structurally correct, statistically plausible dataset to prototype on — so you can validate your data model, build your first dashboard, and demo your schema to stakeholders before a single production row exists.

LLM-powered generation for custom domains

The rule-based parser covers SaaS, ecommerce, fintech, healthcare, marketplace, logistics, and pharma. For anything outside those domains, the LLM backend handles arbitrary schema generation:

from misata import LLMSchemaGenerator

gen    = LLMSchemaGenerator(provider="groq")   # or openai, ollama
schema = gen.generate_from_story(
    "A B2B marketplace with vendor tiers, SLA contracts, and quarterly invoices"
)
tables = misata.generate_from_schema(schema)

This works with any LLM provider that supports the OpenAI-compatible API format. Requires GROQ_API_KEY or OPENAI_API_KEY. Retries automatically on rate limits.

The LLM path is useful for:

Industry-specific schemas with unusual entities (clinical trials, commodity trading, fleet management)
Multi-tenant SaaS with complex permission hierarchies
Any domain where the rule-based parser doesn't have calibrated priors

The LLM infers a reasonable schema, column types, and row count ratios from your description. You get back the same DataFrames as the rule-based path — just with the schema derived from a language model instead of hard-coded priors.

How it compares to Faker and SDV

Faker generates individual fake values. One row at a time. It has no concept of tables that reference each other and no domain-specific distributions. Wiring foreign keys and getting log-normal amounts is your job.

SDV (Synthetic Data Vault) learns patterns from real data and generates synthetic copies. It requires actual training data, pulls in heavy ML dependencies, and cannot pin specific business targets like "fraud rate must be 2%." If you don't have real data to train on, SDV is a dead end.

Misata generates from a description. No real data required. No ML training. Distributions are calibrated to domain knowledge. Business targets are exact.

	Faker	SDV	Misata
Multi-table FK integrity	No	Partial	Yes
No real data needed	Yes	No	Yes
Calibrated domain distributions	No	Learned	Yes
Exact monthly aggregate targets	No	No	Yes
Plain-English story input	No	No	Yes
Database seeding	Manual	No	Yes
LLM-powered custom domains	No	No	Yes
Reproducible with seed	No	No	Yes

The key distinction: SDV is a synthetic data replication tool. Misata is a synthetic data generation tool. They solve different problems. SDV needs real data to learn from. Misata generates from scratch.

Installation and quick start

pip install misata pandas numpy

All of these produce full verified output in under 3 seconds:

python examples/saas_revenue_curve.py
python examples/fintech_fraud_detection.py
python examples/healthcare_multi_table.py
python examples/ecommerce_seasonal.py

Or open the Colab notebook and run it without installing anything. No signup, no API key, no configuration.

Design principles

A few constraints Misata holds to that are worth understanding:

Determinism over randomness. Given the same description and seed, you always get the same dataset. This is non-negotiable for reproducible research and CI pipelines where test data needs to be stable across runs.

Statistical realism over convenience. It would be simpler to generate uniformly distributed values. Misata does not do this because uniform distributions produce data that behaves nothing like real data. The extra calibration work is the point.

Aggregate targets are constraints, not approximations. When you say MRR should be $320k in June, the generated data will sum to exactly $320k in June. Not $318k. Not $322k. The individual rows remain statistically realistic while the aggregates are treated as hard constraints.

Referential integrity is structural, not checked. Misata does not generate data and then validate foreign keys. It generates in dependency order so invalid keys cannot occur. This is a stronger guarantee than post-hoc validation.

Frequently asked questions

Can I add custom columns to a generated schema?

Yes. The two-step parse → generate_from_schema flow lets you inspect and modify the schema object before generating. You can add columns, change data types, adjust row counts, and modify relationship cardinality.

How large can generated datasets be?

Misata is DataFrame-based, so the practical limit is your available RAM. For datasets larger than a few million rows, you can generate in chunks and write directly to a database using seed_database. Benchmarks on a standard laptop show ~500k rows/second for most schemas.

Does it support databases other than PostgreSQL and SQLite?

seed_database accepts any SQLAlchemy connection string, which covers PostgreSQL, MySQL, SQLite, MS SQL Server, Oracle, and others. If SQLAlchemy can connect to it, Misata can seed it.

Is there a way to generate time-series data?

Temporal columns are supported. The registered_at, transaction_date, and similar timestamp fields follow realistic distributions relative to one another — a customer's first transaction always comes after their account creation date, for example. You can specify date ranges in your description: "transactions between January 2023 and December 2024."

What if I need data that follows my company's specific distribution?

The LLM path lets you describe distribution constraints in natural language: "30% of accounts are enterprise tier with balances above $50k." For highly specific requirements, the schema object exposes column-level distribution parameters you can override directly.

Misata is open source, MIT licensed, and available now.

GitHub: github.com/rasinmuhammed/misata
PyPI: pypi.org/project/misata
Docs: QUICKSTART.md
Colab: Run the quickstart notebook

How I Built a "Story-to-Data" Engine in Python (Because Faker Wasn't Enough)

Muhammed Rasin O M — Tue, 16 Dec 2025 16:01:49 +0000

The "2 Months of Pain" Origin Story

An year ago, I was working as a Data Science Engineer at a consultancy firm. We needed to build a Tableau dashboard to demonstrate a new business model. The consultants didn't want "random" data; they wanted a specific story:

"Show a _____ failing in Phase 2, causing a 40% revenue dip in Q3, followed by a recovery in Q4 due to a new ____ launch."

I tried at first using standard libraries like Faker and Mimesis. They are fantastic for generating random names and emails, but they failed hard on Business Logic. Then I used python scripting to generate the data, using for loops and all kind of loops.

I ended up with:

Time Travel Bugs: Timesheets dated before an employee's hire date.
Orphaned Rows: Orders linked to non-existent Users.
Flat Curves: Revenue that looked like static noise, not a "Q3 Dip."

I spent 2 months manually hacking Python scripts, hard-coding probabilities, and stitching CSVs together to make the demo look real. It was a nightmare.

I realized: We don't need more random data generators. We need Narrative Data Engines.

So, I built Misata.

What is Misata?
Misata is an open-source Python engine that turns a natural language story into a multi-table, referentially intact dataset.

Instead of writing 500 lines of schema config, you just type:

misata generate --story "A SaaS platform with 50k users, 20% churn in Q3, and usage-based billing" --use-llm

And it generates SQL-ready CSVs where the math actually works.

Under the Hood: The Architecture

Misata isn't just a wrapper around Faker. It uses a Neuro-Symbolic approach to solve the consistency problem.

The Brain (LLM Parser)

First, it uses an LLM (I optimized it for Llama 3.3 via Groq) to parse your story into a strict JSON schema. It extracts:

Entities: Users, Subscriptions, Invoices.

Distributions: "20% churn" becomes a probability weight.

Relationships: "Invoices belong to Subscriptions."

The Logic (Topological Sort)

To prevent "Orphaned Rows," Misata builds a Directed Acyclic Graph (DAG) of your tables. It uses Topological Sorting to ensure parent tables (e.g., Users) are generated before child tables (e.g., Orders).

Python

# Simplified logic from misata/simulator.py
def topological_sort(self):
    graph = defaultdict(list)
    in_degree = {table.name: 0 for table in self.config.tables}

    for rel in self.config.relationships:
        graph[rel.parent_table].append(rel.child_table)
        in_degree[rel.child_table] += 1

    # Standard Kahn's Algorithm...

The Muscle (Vectorized NumPy)

The biggest bottleneck with Python data generation is looping. Generating 10 million rows in a loop is too slow.

Misata uses Vectorized Operations (via NumPy and Pandas) to generate data in blocks. This allows it to hit speeds of ~250,000 rows/second on a standard laptop.

Features for Data Engineers

I built this to solve the specific pains I faced in consulting:

Relational Integrity: It automatically maps Primary Keys to Foreign Keys. No more broken joins in SQL/Tableau.

No "Time Travel": Child tables (like Timesheets) automatically look up their parent's Start Date to ensure events happen chronologically.

Business Constraints: You can define rules like "Employees cannot log > 8 hours/day."

Try it out
It's open source and available on PyPI.

pip install misata

Generate a dataset:

# Needs GROQ_API_KEY (free tier works great)
misata generate --story "E-commerce store with seasonal spikes in December" --use-llm

Why I Open Sourced It
I know there are enterprise tools out there that cost $10k+/year. But for individual consultants, students, and indie hackers, there was no good "middle ground" between Faker and Enterprise Privacy tools.

I want Misata to be that middle ground.

I'm currently working on adding Curve Fitting (so you can draw a chart and get data that matches it). If you're into Data Engineering or Python optimization, I'd love your feedback on the architecture!

Repo: github.com/rasinmuhammed/misata

P.S. If you are a consultant stuck in "Demo Data Hell" right now and need a specific scenario generated, drop a comment or DM me. I'm looking for complex edge cases to stress-test the engine.

I Built a TUI to Visualize RAG Chunking because chunk_size=1000 is a Lie 📉

Muhammed Rasin O M — Wed, 10 Dec 2025 15:38:26 +0000

Let’s be honest for a second. When you are building a RAG (Retrieval-Augmented Generation) pipeline, how do you pick your chunk_size and overlap?

If you are like 90% of us, you copy-paste 1000 and 200 from a tutorial, run it, and hope the LLM doesn't hallucinate.

I realized I was doing "vibes-based engineering". I had no idea if my chunks were cutting sentences in half, if my overlap was actually preserving context, or if my retrieval was failing because of the embedding model or the chunking strategy.

So, I spent my nights and weekends building a tool to fix it.

Meet RAG-TUI.

It is an open-source, terminal-based visual debugger for RAG pipelines. It helps you see what your splitters are doing before you index millions of documents.

The Problem: "The Black Box"

We treat text splitters like black boxes. You feed in a PDF, and out comes a list of strings. But what do those strings look like?

Did you just cut a critical definition in half?
Is your 10% overlap actually capturing the previous sentence?
Are you feeding your embedding model garbage?

I got tired of printing chunks to the console to debug this. I wanted a UI, but I didn't want to leave my terminal.

The Solution: RAG-TUI

RAG-TUI is a lightweight CLI tool built in Python. You point it at a file, and it gives you an interactive dashboard to tune your indexing strategy in real-time.

Key Features (Why you might want this)

1. Real-time Visualization 🎨
Drag a slider to change the chunk_size. Watch the text re-chunk instantly.
The UI uses color-coded cards to show you exactly where one chunk ends and the next begins.

2. Quality Indicators 🚦
I added visual "linters" for your chunks:

🟢 Green: Clean break (ends with ., !, ?).
🟡 Yellow: Mid-phrase break (ends with ,, :).
🔴 Red: Hard cut (ends with a character).
⚠️ Warning: Chunk is too small (<50 tokens) or too large.

3. "Scientific" Batch Testing
Stop guessing. Enter 20 test queries ("What is the refund policy?", "How do I reset my password?"). RAG-TUI runs them against your current settings using local vector search and calculates a Hit Rate.

Hit Rate > 80%? Ship it.
Hit Rate < 60%? Your chunks are wrong. Change the strategy.

4. Privacy First (Ollama Support) 🔒
You don't need to send your private docs to OpenAI just to debug a splitter. RAG-TUI has native support for Ollama. You can run the entire debugging loop offline on your laptop.

💻 Under the Hood

For the Python nerds (like me), here is the stack that makes this possible:

UI: Textual (The best TUI framework, period).
Chunking: Chonkie (Blazing fast token splitting).
Vector DB: Usearch (Lightweight, in-memory vector search).
LLM: Async wrapper for Ollama/OpenAI.

🚀 How to Try It

I tried to make the onboarding as painless as possible.

pip install rag-tui

Then, just run:

rag-tui

(Make sure you have ollama serve running if you want to test embeddings!)

🤝 I Need Your Feedback!

This is currently in v0.0.2 Beta. It works, but I know there are edge cases I haven't found yet.

I am building this in public because I believe RAG tooling needs to get better. If you are learning RAG or building a production pipeline, please give this a spin.

Does it support your weird PDF format?
Do you need a specific splitter I haven't added?
Is the TUI crashing on Windows? (It shouldn't, but you know... Windows).

Star the repo if you think this is useful. It motivates me to keep shipping updates! ⭐️

🔗 GitHub Repo: https://github.com/rasinmuhammed/rag-tui

Happy Chunking! ✂️

I built a FastAPI admin panel that doesn't suck (and here's why it's different)

Muhammed Rasin O M — Sat, 06 Dec 2025 13:56:15 +0000

TL;DR: FastAPI Matrix Admin combines one-line auto-discovery, async-first architecture, and production-grade security in a package that requires zero Node.js.

🟢 Live Demo (Read-only)
💻 Github Repo

The admin panel problem nobody talks about

Every FastAPI project follows the same arc:

You build a great API.
Product wants to "just update a few records manually."
You reluctantly install Django admin (now you have two frameworks).
Or you build a custom React dashboard (6 weeks later...).
Or you use a generic admin and spend days configuring it.

I've done all three. They all sucked for different reasons.

What's wrong with existing FastAPI admin solutions?

I evaluated every major option before building this. Here's what I found:

Library	Issue
FastAPI-Admin	Requires Tortoise ORM (can't use SQLAlchemy).
SQLAdmin	Good, but sync-only. No async support in 2024?
Starlette-Admin	Heavy Starlette dependency, limited FastAPI integration.
Admin-One	Requires Vue.js build step, defeats FastAPI's simplicity.

The pattern: They either force you into specific ORMs, ignore async, or add frontend build complexity.

What I Built Instead

FastAPI Matrix Admin focuses on three non-negotiables:

1. One-Line Auto-Discovery (Because Your Time Matters)

from fastapi_matrix_admin import MatrixAdmin

admin = MatrixAdmin(app, engine=engine, secret_key="...")
admin.auto_discover(Base)  # Done. All models registered.

Under the hood:

Introspects SQLAlchemy models using inspect().
Analyzes column types to generate appropriate form fields.
Detects text columns for search.
Finds timestamp columns for default ordering.
Creates a sensible list display based on column types.

Customization when you need it:

admin.register(
    User,
    list_display=["id", "email", "created_at"],
    searchable_fields=["email", "name"],
    ordering=["-created_at"],
    exclude=["password_hash"]  # Obviously
)

2. Zero Node.js (Seriously)

No npm. No webpack. No package.json.

Stack: Tailwind CSS via CDN, HTMX for dynamic updates, Alpine.js (3KB), and Jinja2 templates.
Why this matters:
- pip install fastapi-matrix-admin → you're done.
- No build step in CI/CD.
- No frontend/backend version mismatches.
- Deploys anywhere Python runs.

3. Production-Grade Security (Not an Afterthought)

Most admin libraries treat security as optional. Here is what is built-in:

Content Security Policy (CSP): Prevents XSS by strictly controlling script sources.
CSRF Protection: Every form gets a signed token automatically.
URL Signing: All admin URLs are cryptographically signed to prevent ID enumeration/tampering.
Type Safety with Pydantic v2: Input validation happens automatically.

Security Comparison:

Feature	FastAPI Matrix Admin	SQLAdmin	FastAPI-Admin
CSP Headers	✅ Built-in	❌ Manual	❌ Manual
CSRF Protection	✅ Automatic	⚠️ Optional	❌ None
URL Signing	✅ Yes	❌ No	❌ No
Pydantic v2	✅ Yes	⚠️ v1	❌ No validation

Performance: Async All The Way Down

Full async support isn't optional in 2024.

# Async SQLAlchemy 2.0
from sqlalchemy.ext.asyncio import create_async_engine, AsyncSession

engine = create_async_engine("postgresql+asyncpg://...")
admin = MatrixAdmin(app, engine=engine)

Simple benchmark (100 concurrent list view requests):

FastAPI Matrix Admin (async): ~50ms avg
SQLAdmin (sync): ~180ms avg (blocks other requests)

The Matrix UI (Yeah, It's Different)

I'm not going to pretend the cyberpunk aesthetic is for everyone. But here's why it exists:

Problem: Every admin panel looks the same. Generic Bootstrap tables. Boring gray sidebars. No personality.
Solution: Terminal-style green/black theme with neon accents.

It makes internal tools feel less corporate, and stakeholders actually remember seeing "that Matrix admin thing." (If you hate it, the CSS variables are customizable).

Quick Start

pip install fastapi-matrix-admin

from fastapi import FastAPI
from sqlalchemy import Column, Integer, String, Boolean
from sqlalchemy.ext.asyncio import create_async_engine
from sqlalchemy.orm import declarative_base
from fastapi_matrix_admin import MatrixAdmin

Base = declarative_base()

class User(Base):
    __tablename__ = "users"
    id = Column(Integer, primary_key=True)
    email = Column(String, unique=True)
    is_active = Column(Boolean, default=True)

app = FastAPI()
engine = create_async_engine("sqlite+aiosqlite:///./database.db")

# That's it
admin = MatrixAdmin(app, engine=engine, secret_key="your-secret-key-min-32-chars")
admin.auto_discover(Base)

Run it:

uvicorn app:app
# Visit http://localhost:8000/admin

What's Next

Current roadmap:

[ ] File/image upload support
[ ] Advanced filters (date ranges, multi-select)
[ ] Export to CSV/Excel
[ ] Custom dashboard widgets
light theme for the Matrix-haters

Final Thoughts

I built this because I kept rebuilding the same admin panel over and over. Auto-discovery saves me hours per project. Zero Node.js means simple deploys. The security features mean I can actually use this in production.

If you try it, let me know what breaks. Or what you wish it did differently.

Questions I'd love feedback on:

Is auto-discovery too magical, or genuinely useful?
What security features am I missing?
Would you actually use this in production?

Drop a comment or open an issue. First-time contributors welcome!

Github: fastapi-matrix-admin