DEV Community: Arian Mokhtariha

Meet data2prompt: The CLI Tool That Finally Makes LLMs Understand Your Data Science Projects

Arian Mokhtariha — Tue, 02 Jun 2026 15:25:43 +0000

Every data scientist has hit this wall.

You are deep in a project — CSVs, SQL dumps, Jupyter notebooks, maybe some Power BI files — and you want to ask an LLM to help you reason across the whole thing. Not just one script. The entire project. You want it to understand your data structure, your pipeline logic, your model decisions all at once.

So you try to package it up and paste it in. And it fails. The context window chokes. The LLM forgets files it saw earlier. The responses stop making sense.

The problem is not your LLM. The problem is that nobody built the right packaging tool for data-heavy projects — until now.

Introducing data2prompt

data2prompt is an open-source CLI tool that packages your entire data science project into a single, optimized, LLM-ready file. Not a generic dump of everything in your folder — a smart, data-aware output that knows how to handle CSVs, SQL, Jupyter notebooks, Excel files, and binary data files the way a data scientist actually needs them handled.

Install it in one command:

pipx install data2prompt

Run it from your project root:

data2prompt

That is it. You get a single PROMPT.md or PROMPT.xml file ready to paste into Claude, ChatGPT, Gemini, or any LLM with a large context window.

The Problem With Generic Tools

There are great tools out there for packaging software projects for LLM context. They work beautifully on codebases full of Python scripts and config files.

But a data science project is not a software project. It contains fundamentally different file types that need fundamentally different handling — and when a generic tool encounters them, it does the worst possible thing: it dumps everything raw.

Here is what that looks like in practice:

A CSV with 50,000 rows gets written to the output in full — 50,000 rows of token-consuming noise when what the LLM actually needs is the schema and a representative sample.

A SQL dump gets included with every single INSERT statement — hundreds of thousands of rows of raw data when what the LLM needs is the CREATE TABLE schema and a handful of example rows per table.

A Jupyter notebook gets written with all its outputs intact — which includes matplotlib charts and styled dataframes stored as base64-encoded image strings. A single notebook visualization can contribute 60,000 tokens of encoded image data that an LLM literally cannot read or use.

A Power BI or pickle file is binary — reading it as text produces corrupted garbage that fills your context window with meaningless characters.

The result is a context file that is technically complete and practically useless. Every token budget gets consumed before the LLM has seen anything worth reasoning about.

How data2prompt Handles Each File Type

data2prompt applies a dedicated strategy to every file type found in a data project.

CSVs and Excel — Intelligent Random Sampling

Instead of dumping all rows, data2prompt takes a random sample:

# Default: 15 random rows per CSV
data2prompt

# Increase sample size when you need more context
data2prompt --csv-sample-size 30

The sampling is true random with a fixed seed — not head/tail. This means the LLM sees a representative spread of your actual data values, not just whatever happened to be at the top of the file. The seed makes the output fully reproducible — same result every time you run it.

The output is clean and annotated:

-- [Sample - Random 15 rows of 52,411 total] --
| order_id | customer   | region | sales  | profit |
|----------|------------|--------|--------|--------|
| CA-2019  | John Smith | West   | 245.00 | 41.65  |
...
-- [CSV truncated: Showing random 15 rows to save context] --

For Excel, each sheet is sampled independently. Sheets that are purely visual dashboards — no tabular data — are detected and noted rather than producing empty tables.

SQL Files — Schema First, Data Sampled

SQL dumps have two distinct parts: the schema (CREATE TABLE definitions) and the data (INSERT statements). data2prompt treats them completely differently.

Schema blocks are always preserved in full. Every CREATE TABLE, every column definition, every constraint and foreign key relationship — this is exactly what the LLM needs to understand your database.

INSERT statements are sampled randomly per table. You get a handful of representative rows for each table, enough to understand what the data looks like, without the thousands of repetitive INSERT lines that collapse your context budget.

Jupyter Notebooks — Logic Without the Noise

Notebooks store both cell source code and cell outputs. The source code is what the LLM needs — your transformations, model definitions, evaluation logic, markdown explanations. The outputs are the problem.

data2prompt extracts source code from every cell and discards outputs entirely. The base64 image strings, printed dataframes, and error tracebacks that bloat notebook files are stripped out. What remains is clean, readable notebook logic that an LLM can actually reason about.

Binary Files — Acknowledged, Not Mangled

.pbix, .pkl, .parquet, .db, .sqlite, .h5, .feather — data2prompt lists these in your project tree so the LLM knows they exist, and skips their content entirely. No corrupted binary strings eating your context window.

Two Output Formats: Markdown and XML

# Clean Markdown (default)
data2prompt

# Structured XML
data2prompt --format xml

The XML format was added based on Anthropic's research showing that XML-style structured tags improve LLM attention and parsing within long context windows. Every file gets semantic tags:

<codebase name="my-project">
  <directory_structure>...</directory_structure>
  <files>
    <file path="data\sales.csv">
      ...sampled table...
    </file>
    <file path="notebooks\analysis.ipynb">
      <cell index="1" type="code">
        <content>
          df = pd.read_csv('data/sales.csv')
          df.head()
        </content>
      </cell>
    </file>
  </files>
</codebase>

Use Markdown for quick analysis sessions. Use XML when you want the LLM to navigate a large project with maximum structural clarity.

The Numbers

Same data science project — dimension and fact CSVs, a SQL dump, Power BI files, ML notebooks, classification and clustering scripts — run through three tools:

Tool	Output Size
Generic tool #1	22,085 KB
Generic tool #2	9,304 KB
data2prompt	241 KB

241 KB versus 22,085 KB. Same project. Same information that actually matters.

That gap exists entirely because of the file-type-specific strategies above — the random CSV sampling, the SQL schema extraction, the notebook output stripping, the binary file handling. Nothing clever, just the right tool for the right files.

Full Usage

# Install (pipx recommended for isolated environment)
pip install data2prompt
pipx install data2prompt

# Basic run — outputs PROMPT.md in project root
data2prompt

# XML format
data2prompt --format xml

# Custom CSV sample size
data2prompt --csv-sample-size 25

# Ignore specific folders
data2prompt --ignore-folders data/raw archive models/checkpoints

# Custom output filename
data2prompt --output my_project_context

Create a .data2promptignore file in your project root for permanent exclusions — same syntax as .gitignore:

data/raw/
*.pkl
archive/

Who This Is Built For

data2prompt is specifically designed for data scientists, data analysts, and data engineers who work with real data files alongside their code. If your project has CSVs, SQL, notebooks, or Excel files, this tool will dramatically improve the quality of your LLM interactions with that project.

If you work in a pure software development context with no data files, a general-purpose packaging tool will serve you well. But if your project looks anything like a real data science workflow, give data2prompt a try.

GitHub: https://github.com/arianmokhtariha/data2prompt
PyPI: https://pypi.org/project/data2prompt

Open to questions, feedback, and contributions — drop them in the comments or open an issue on GitHub.

I Tried Repomix on My Data Science Project. It Generated a 22,000 KB File. So I Built My Own Tool

Arian Mokhtariha — Sun, 31 May 2026 22:31:28 +0000

A few months ago a friend showed me two tools — Repomix and code2prompt. The idea was simple: point them at your project folder, they package everything into one file, you paste it into an LLM and ask questions about your whole codebase at once. For his pure Python projects they worked great.

I was working on a data analytics project at the time — dimension and fact CSVs, a SQL dump, some Power BI files, Jupyter notebooks with ML models. I ran Repomix on it and got a 22,085 KB output file. code2prompt gave me 9,304 KB. I tried pasting either of them into Claude. It choked immediately.

So I opened the files to see what was actually inside them. What I found was the root of the problem.

What These Tools Get Wrong for Data Projects

Repomix and code2prompt are built for code repos. They operate on a simple principle: read every file, dump every file. That works fine when your project is Python scripts and config files. It completely falls apart when your project looks like mine.

Here's what was inflating those files:

Raw CSV dumps. My Fact_Sales.csv had tens of thousands of rows. The tool dumped every single one. An LLM doesn't need 50,000 rows of sales data — it needs to understand the structure and a representative sample.

Endless SQL INSERT statements. My Superstore.sql file had the full database dump including every INSERT INTO statement for every table. The schema — the CREATE TABLE blocks — is what an LLM actually needs. The data rows are mostly noise.

Notebook outputs with base64 images. Jupyter notebooks store cell outputs as JSON inside the .ipynb file. When a cell generates a matplotlib chart, that chart gets saved as a base64-encoded image string inside the notebook. A single chart output can be 50,000+ characters of base64 garbage that an LLM cannot use at all.

Binary files read as text. My .pbix (Power BI) files are binary. These tools attempted to read them as text and produced corrupted garbage that consumed tokens while providing zero information.

There was no tool that understood these problems. So after three months of building — and being very honest that I'm a data scientist not a developer, so this was heavily AI-assisted — I shipped data2prompt.

What data2prompt Does Differently

The core idea is that each file type in a data project needs its own strategy, not a generic "read and dump" approach.

CSVs and Excel: Smart Sampling

Instead of dumping all rows, data2prompt takes a random sample:

df = df.sample(sample_size, random_state=seed)

The default is 15 rows, configurable with --csv-sample-size. Critically it's random sampling with a fixed seed — not head/tail. Random sampling gives the LLM a more representative picture of value diversity across the dataset. The seed (default 42) makes output reproducible across runs.

The output tells the LLM exactly what it's looking at:

-- [Sample - Random 15 rows] --
| order_id | customer_name | sales  | profit |
|----------|--------------|--------|--------|
| CA-2019  | John Smith   | 245.00 | 41.65  |
...
-- [CSV truncated: Showing random 15 rows to save context] --

For Excel files, each sheet is sampled independently. The parser also detects sheets that are purely visual dashboards (charts, images only) and notes them rather than producing an empty table.

SQL Files: Schema Preserved, Data Sampled

This was the hardest parser to get right. SQL dump files typically follow a pattern: CREATE TABLE block defining the schema, followed by hundreds or thousands of INSERT INTO statements loading the data.

data2prompt reads line by line and applies different logic to each part:

# Always preserve the schema
if "CREATE TABLE" in line_upper:
    flush_buffer()
    in_create_block = True
    processed_lines.append(line)

# Buffer INSERT rows for sampling
if is_insert or is_data_row:
    table_data_buffer.append(line)

When it hits a buffer of INSERT rows, it samples them randomly:

rest_indices = sorted(rng.sample(range(1, len(table_data_buffer)), sample_size - 1))
sampled_rows = [first_line] + [table_data_buffer[idx] for idx in rest_indices]

The first line (the INSERT header) is always preserved. The rest are random samples in their original order. The LLM gets the full schema of every table plus a representative data sample — which is exactly what it needs to understand your database.

Jupyter Notebooks: Source Code Only

Notebook cells store three things: source code, execution count, and outputs. The outputs are what bloat the file — printed dataframes, matplotlib charts as base64, error tracebacks.

data2prompt keeps the source code of every cell and strips the outputs entirely. A notebook that was 8MB of JSON becomes a clean sequence of code cells. The parser also handles truncation of unusually long lines and caps output blocks at a configurable line limit for cases where outputs are genuinely useful text.

For the XML format, each cell becomes a structured block:

<cell path="ML/Q1/Q1.ipynb" index="3" type="code">
    <content>
model = XGBRegressor(n_estimators=100, learning_rate=0.1)
model.fit(X_train, y_train)
predictions = model.predict(X_test)
    </content>
</cell>

Binary Files: Listed, Not Read

.pbix, .parquet, .pkl, .db, .sqlite, .feather, .h5 — all listed in the directory tree so the LLM knows they exist, but content is skipped entirely. No garbage bytes consuming your context window.

Two Output Formats: Markdown and XML

data2prompt supports both --format markdown (default) and --format xml.

The XML format was added after Anthropic published research showing that XML-style tags improve LLM attention and parsing. The full project gets wrapped in a structured hierarchy:

<codebase name="superstore-analysis">
  <metadata>
    <generated_on>2025-05-31 09:00</generated_on>
    <total_tokens method="o200k_base">48293</total_tokens>
  </metadata>
  <directory_structure>
    Fact&dim-csv\Fact_Sales.csv
    Superstore.sql
    ...
  </directory_structure>
  <files>
    <file path="Fact&dim-csv\Fact_Sales.csv">
      ...sampled table...
    </file>
  </files>
</codebase>

The Benchmark

Same project, same files, three tools:

Tool	Output Size
Repomix	22,085 KB
code2prompt	9,304 KB
data2prompt	241 KB

That's a 98.9% reduction vs Repomix and 97.4% vs code2prompt on the same data-heavy project, while preserving all the structurally useful information — schemas, sampled data, notebook logic, file tree.

The reduction is so dramatic specifically because of the project type. A pure Python project would show a much smaller gap. That's exactly the point — this tool is built for data projects, not code projects.

Installation and Usage

# Install
pip install data2prompt

# Recommended: use pipx for isolated install
pipx install data2prompt

Basic usage — run inside your project directory:

# Default: Markdown output
data2prompt

# XML output (better for LLM structured parsing)
data2prompt --format xml

# Increase CSV sample size
data2prompt --csv-sample-size 25

# Custom output file name
data2prompt --output my_project_context

# Ignore specific folders
data2prompt --ignore-folders data/raw

The output file (default: PROMPT.md or PROMPT.xml) is ready to paste directly into Claude, ChatGPT, Gemini, or any LLM with a large context window.

You can also create a .data2promptignore file in your project root — same syntax as .gitignore — to exclude specific files or patterns permanently.

Who This Is For

data2prompt is specifically designed for data scientists, data analysts, and data engineers who work with:

CSV/Excel data files
SQL database dumps
Jupyter notebooks
Power BI or other binary analytics files
Mixed projects with both code and data

If your project is purely Python scripts with no data files, Repomix or code2prompt will serve you fine. But if your project looks anything like a real data science workflow, give data2prompt a try.

GitHub: https://github.com/arianmokhtariha/data2prompt

PyPI: https://pypi.org/project/data2prompt

Questions about how the SQL parser works or why random sampling over head/tail? Drop them in the comments.