DEV Community: serada

Stop Passing Raw DataFrames to Your LLM — Here's a Better Way

serada — Wed, 18 Mar 2026 07:36:31 +0000

TL;DR

df.to_string() on a 100K-row DataFrame = millions of tokens, guaranteed failure
df.head() = 5 rows with zero statistical context, useless for real analysis
dfcontext generates a token-budget-aware, column-type-aware summary — no LLM calls required

The Problem Nobody Talks About

You've got a DataFrame with 100,000 rows. You want to ask an LLM about it. What do you do?

Most people try one of two things:

# Option A: Dump everything (will blow the context window)
prompt = df.to_string()

# Option B: Just use head (loses almost all information)
prompt = df.head().to_string()

Option A will hit your token limit instantly. Option B gives the model five rows of data and basically asks it to guess the rest.

There's no obvious middle ground in the standard pandas API — so I built one.

Introducing dfcontext

dfcontext generates a compact, statistically rich summary of your DataFrame that fits within a token budget you specify. It's pure data processing — zero LLM calls, works with any LLM provider.

pip install dfcontext

import pandas as pd
from dfcontext import to_context

df = pd.read_csv("sales.csv")  # 100K rows

ctx = to_context(df, token_budget=2000)
print(ctx)

Here's what that looks like on output:

## Dataset overview

- 100,000 rows × 5 columns

## Schema

| Column    | Type           | Non-null |
| --------- | -------------- | -------- |
| region    | object         | 100%     |
| sales     | float64        | 100%     |
| quantity  | int64          | 100%     |
| date      | datetime64[ns] | 100%     |
| is_return | bool           | 100%     |

## Column statistics

### region (categorical, 4 unique)

Top values: East (28.0%), West (25.8%), North (23.2%), South (23.0%)

### sales (numeric)

Range: 4.64 — 8,172.45 | Mean: 1,010.55 | Std: 1,030.04
Distribution: [█▃▁▁▁▁▁▁]

### date (datetime)

Range: 2024-01-01 — 2024-02-11 | Granularity: hourly

### is_return (boolean)

True: 6.0% | False: 94.0%

## Sample rows (diverse selection)

| region | sales   | quantity | date       | is_return |
| ------ | ------- | -------- | ---------- | --------- |
| East   | 4.64    | 32       | 2024-01-14 | False     |
| South  | 697.55  | 50       | 2024-01-15 | False     |
| West   | 8172.45 | 68       | 2024-01-02 | False     |

2,000 tokens. Full schema. Real distributions. Diverse sample rows. That's the sweet spot.

Why Column-Type-Aware Matters

The key insight behind dfcontext is that different column types need different summaries.

A numeric column like sales needs range, mean, and distribution. A categorical column like region needs value frequencies. A datetime column needs range and granularity. A boolean column needs true/false ratio.

Feeding mean and std for a boolean column is meaningless. Showing "top values" for a float column is wrong. dfcontext handles each type correctly out of the box.

Query Hints: Tell It What You Care About

If you already know the focus of your analysis, pass a hint. dfcontext will allocate more token budget to relevant columns.

ctx = to_context(df, token_budget=2000, hint="regional sales trends")
# "region" and "sales" get richer detail; less budget is spent on other columns

This is useful when you have a wide DataFrame but only care about a few columns for a given question.

Correlation Detection

ctx = to_context(df, token_budget=2000, include_correlations=True)
# Adds: "sales ↔ quantity: r=+0.823 (strong positive)"

Often the most valuable thing you can tell an LLM is how columns relate to each other. This surfaces strong correlations without requiring the model to compute them.

Multiple Output Formats

dfcontext outputs Markdown by default, but it also supports plain text and YAML — useful if your prompt template expects structured data.

ctx_md    = to_context(df, format="markdown")  # default, great for chat models
ctx_plain = to_context(df, format="plain")     # no markdown syntax
ctx_yaml  = to_context(df, format="yaml")      # structured, requires pyyaml

Wiring It Up to Claude (or Any LLM)

Here's a complete example with the Anthropic SDK:

import anthropic
import pandas as pd
from dfcontext import to_context

df = pd.read_csv("sales.csv")
ctx = to_context(df, token_budget=2000, hint="sales trends by region")

client = anthropic.Anthropic()
response = client.messages.create(
    model="claude-sonnet-4-20250514",
    max_tokens=1024,
    messages=[{
        "role": "user",
        "content": f"{ctx}\n\nWhat are the key sales trends? Any anomalies worth investigating?",
    }],
)
print(response.content[0].text)

The same pattern works with OpenAI, Gemini, or any API that accepts a string prompt.

Budget Tuning: More Tokens = Richer Stats

dfcontext adapts to the budget you give it. With a higher budget, it adds percentiles, skewness, and outlier rates automatically.

ctx_tight = to_context(df, token_budget=500)   # overview + schema only
ctx_rich  = to_context(df, token_budget=5000)  # full stats, percentiles, more samples

You can see this in action without any API key by running the budget_tuning.py example in the repo.

Under the Hood: Getting Structured Results

If you want to integrate dfcontext into your own tooling rather than just generating a string, use analyze_columns directly:

from dfcontext import analyze_columns

summaries = analyze_columns(df)
for name, s in summaries.items():
    print(f"{name}: {s.column_type}, {s.unique_count} unique values")
    if s.distribution_sketch:
        print(f"  histogram: [{s.distribution_sketch}]")
    if "outlier_rate" in s.stats:
        print(f"  outlier rate: {s.stats['outlier_rate'] * 100:.1f}%")

Each ColumnSummary object gives you dtype, column_type, non_null_rate, unique_count, stats, sample_values, and distribution_sketch — enough to build your own rendering layer if you need it.

Performance

It handles 100K rows in under a second. The bottleneck is pandas, not dfcontext.

Token counting defaults to a character-based estimate. For accurate counts, install the optional dependency:

pip install dfcontext[tiktoken]

Install Options

pip install dfcontext          # core only (no extra deps)
pip install dfcontext[tiktoken] # accurate token counting
pip install dfcontext[yaml]     # YAML format output
pip install dfcontext[all]      # everything

When to Use This

dfcontext is a good fit when:

You have DataFrames larger than a few hundred rows
You're building LLM pipelines that take tabular data as input
You want consistent, reproducible context across runs (no sampling randomness)
You're working within strict token budgets (API costs, model limits)

It's not trying to replace exploratory analysis. Use pandas profiling or ydata-profiling for that. dfcontext is specifically optimized for the "give an LLM just enough to understand this data" use case.

GitHub: sserada/dfcontext — PRs and issues welcome.

What's your current approach for feeding DataFrames to LLMs? Do you truncate, sample, or something else? Let me know in the comments — I'm curious what patterns people are using in the wild.

[2026 Latest] Pandas 3.0 is Here: Copy-on-Write, PyArrow, and What You Need to Know

serada — Sun, 22 Feb 2026 01:04:48 +0000

Introduction & TL;DR

The long-awaited Pandas 3.0 has officially arrived (released early 2026), bringing some of the most fundamental shifts to the library in years. If you work with data in Python, this upgrade will dramatically affect how your code runs, performs, and occasionally breaks.

TL;DR: The Biggest Changes

Copy-on-Write (CoW) is now the default. Say goodbye to the dreaded SettingWithCopyWarning.
PyArrow String Backend. The old object dtype for strings is gone, replaced by a lightning-fast Apache Arrow backend.
Chained Assignment Errors. If you try to modify a DataFrame via chained indexing (e.g., df[df['A'] > 0]['B'] = 1), it will now throw an error instead of a warning.

Let's dive into what these changes mean for your daily workflows and how to migrate your existing codebase.

1. Copy-on-Write is Now the Standard

Historically, Pandas users have struggled to predict whether an operation returned a view of the original data or a copy. This unpredictability led to the infamous SettingWithCopyWarning.

In Pandas 3.0, Copy-on-Write (CoW) is enabled by default and cannot be turned off.

What does this mean?

Any DataFrame or Series derived from another will behave as an entirely separate object. However, to keep things fast, the actual copying of data is delayed (lazy evaluation) until you explicitly modify one of the objects.

import pandas as pd

df = pd.DataFrame({"A": [1, 2, 3]})
subset = df[df["A"] > 1]  # This doesn't copy data yet!

# Modifying 'subset' will trigger a copy under the hood.
# The original 'df' remains completely unchanged.
subset.iloc[0, 0] = 99

Jargon Explanation: Copy-on-Write (CoW)
CoW is a memory management technique. Instead of duplicating data immediately when a new variable is created, both variables point to the same memory. A separate copy is only created at the exact moment one of the variables is modified.

Breaking Change: Chained Assignment

Because of CoW, chained assignments are formally broken.

- # Pandas 2.x (Warning) or Pandas 3.0 (ChainedAssignmentError)
- df[df['col1'] > 10]['col2'] = 100

+ # Pandas 3.0 Correct Way
+ df.loc[df['col1'] > 10, 'col2'] = 100

Always use .loc for setting values on subsets!

Hands-on: Spotting the Lazy Copy

To see CoW in action without waiting for massive DataFrames to process, you can track memory addresses. When you create a subset, it shares memory with the parent until a mutation occurs.

import pandas as pd
import numpy as np

# Requires pandas >= 3.0
df = pd.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6]})

# Slicing creates a view, perfectly demonstrating CoW
subset = df.iloc[1:3]

# 1. At this point, both dataframes share memory
print(np.shares_memory(df["B"].values, subset["B"].values)) # True

# 2. Mutating the subset forces a lazy copy
subset.iloc[0, 1] = 99

# 3. Now, they are completely separate
print(np.shares_memory(df["B"].values, subset["B"].values)) # False

2. The PyArrow String Backend

If you've ever dealt with massive text datasets, you know that Pandas historically stored strings as generic Python object types. This was highly inefficient in both speed and memory.

In version 3.0, strings are now inferred as a dedicated str dtype, backed by Apache Arrow (if you have pyarrow installed).

The Performance Boost

Switching to the PyArrow backend yields massive performance improvements:

Speed: String operations (.str.contains(), .str.lower()) run 5 to 10 times faster.
Memory: Memory consumption for text-heavy columns drops by up to 50%.

import pandas as pd

# If pyarrow is installed, 'text' is now a PyArrow string array by default.
df = pd.DataFrame({"text": ["apple", "banana", "cherry"]})
print(df.dtypes)
# text    string[pyarrow] 
# dtype: object

Furthermore, this columnar Arrow format allows for zero-copy data sharing with other modern tools like Polars and DuckDB.

3. Other Notable Changes

Microsecond Resolution: The default resolution for datetime data is now microseconds (instead of nanoseconds). This fixes the annoying out-of-bounds date errors for dates outside the 1678–2262 range.
Removed Deprecations: Old friends like DataFrame.applymap() (use .map()), Series.ravel(), and DataFrame.append() (use pd.concat()) have been permanently removed.
Python 3.11+: You must be running at least Python 3.11 and NumPy 1.26.0.

Conclusion

Pandas 3.0 is a massive leap forward, successfully addressing the two biggest pain points of the past decade: memory inefficiency with strings and the unpredictable view vs. copy behavior.

While migrating legacy code (especially removing chained assignments) might take a weekend, the resulting performance and stability are well worth the effort.

Have you encountered any specific migration headaches with Pandas 3.0? Let me know in the comments!