We're going to build a command-line Training Dataset Profiler that ingests a raw JSONL fine-tuning dump, validates structure, flags quality issues, and produces a human-readable summary. If you have ever downloaded a "cleaned" dataset from Hugging Face only to find empty responses and leaked emails inside, this tool saves you from training on garbage.
What you'll need
- Python 3.10 or newer
- An Oxlo.ai API key from https://portal.oxlo.ai
- The OpenAI SDK:
pip install openai
Step 1: Bootstrap the project and Oxlo.ai client
First we initialize the OpenAI-compatible client pointing at Oxlo.ai. I keep my key in an environment variable so it does not end up in shell history.
import os
import json
from openai import OpenAI
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key=os.getenv("OXLO_API_KEY", "YOUR_OXLO_API_KEY")
)Step 2: Create a dirty sample dataset
To make this reproducible without downloading multi-gigabyte files, we will synthesize a small JSONL file that mimics real instruction-tuning data. I have sprinkled in duplicates, PII, and malformed entries so the profiler has something to catch.
def create_sample_dataset(path: str = "sample_data.jsonl"):
records = [
{"instruction": "What is the capital of France?", "input": "", "output": "Paris"},
{"instruction": "Write a Python hello world", "input": "", "output": "print('hello world')"},
{"instruction": "What is the capital of France?", "input": "", "output": "Paris"},
{"instruction": "Email me the report", "input": "", "output": "Sure, I will send it to alice@example.com tomorrow."},
{"instruction": "", "input": "", "output": ""},
{"instruction": "Explain quantum computing", "input": "", "output": "Quantum computing uses qubits. " * 50},
{"instruction": "Fix this bug", "input": "def foo():\n pass", "output": "You should use a better function name."},
]
with open(path, "w", encoding="utf-8") as f:
for r in records:
f.write(json.dumps(r, ensure_ascii=False) + "\n")
return path
create_sample_dataset()Step 3: Define the profiler agent
The profiler is just a system prompt. We treat the LLM as a structured analyst that receives raw JSON and returns findings as JSON. You can edit this prompt to add domain-specific rules, such as rejecting outputs under a certain length or flagging specific keywords.
SYSTEM_PROMPT = """You are a Training Dataset Profiler. Your job is to analyze raw JSONL records from an LLM fine-tuning dataset and return a concise JSON object with exactly these keys:
- schema_detected: string, one of ["alpaca", "sharegpt", "custom", "unknown"]
- total_records_analyzed: integer
- issues: array of objects, each with keys "severity" ("high", "medium", "low"), "category" ("duplicate", "pii", "empty_field", "formatting", "quality"), "record_index": integer, "description": string
- summary: string, a two-sentence human-readable summary
Be strict. If an output contains an email address, phone number, or API key, flag it as pii with high severity. If an instruction-output pair is identical to a previous pair, flag it as duplicate. If any required field is empty, flag it as empty_field."""Step 4: Inspect schema and format
Before we run statistics, we need to know if this is Alpaca, ShareGPT, or a custom format. We send the first three records to Oxlo.ai with a focused user message. I am using Llama 3.3 70B here because it handles general-purpose structured analysis reliably.
def detect_schema(file_path: str) -> str:
with open(file_path, "r", encoding="utf-8") as f:
lines = [json.loads(f.readline()) for _ in range(3)]
user_message = (
"Analyze these sample records and tell me the dataset schema. "
"Return only one word: alpaca, sharegpt, or custom.\n\n"
+ json.dumps(lines, indent=2)
)
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": user_message},
],
)
return response.choices[0].message.content.strip()
schema = detect_schema("sample_data.jsonl")
print(f"Detected schema: {schema}")Step 5: Surface data-quality issues
Now we feed a larger slice of the dataset and ask the agent to flag exact problems. Because Oxlo.ai uses flat per-request pricing, sending a batched context of several records costs the same whether the snippet is 1K or 10K tokens. That makes exploratory profiling predictable when you are scanning long-context training dumps.
def profile_records(file_path: str, limit: int = 20) -> dict:
with open(file_path, "r", encoding="utf-8") as f:
records = [json.loads(line) for line in f][:limit]
user_message = (
f"Analyze these {len(records)} records and return your findings as JSON. "
"Do not wrap the JSON in markdown code fences.\n\n"
+ json.dumps(records, indent=2, ensure_ascii=False)
)
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": user_message},
],
)
raw = response.choices[0].message.content.strip()
if raw.startswith("
```"):
raw = raw.split("```
")[1].replace("json", "").strip()
return json.loads(raw)
report = profile_records("sample_data.jsonl")
print(json.dumps(report, indent=2))Step 6: Generate an executive summary
Finally, we ask the model to synthesize the structured report into a short markdown summary we can paste into a data card or pull request description.
def summarize(report: dict) -> str:
user_message = (
"Turn this structured dataset report into a concise markdown summary "
"suitable for a data card. Include bullet points for high-severity issues.\n\n"
+ json.dumps(report, indent=2)
)
response = client.chat.completions.create(
model="qwen-3-32b",
messages=[
{"role": "system", "content": "You write concise data-quality summaries in markdown."},
{"role": "user", "content": user_message},
],
)
return response.choices[0].message.content.strip()
markdown_summary = summarize(report)
print(markdown_summary)Run it
Putting it all together, the full script loads the dirty sample, detects the schema, profiles the records, and prints the markdown summary. Here is the main entry point and an example of what you should see in your terminal.
if __name__ == "__main__":
create_sample_dataset()
schema = detect_schema("sample_data.jsonl")
print(f"\nSchema: {schema}\n")
report = profile_records("sample_data.jsonl")
print("\nStructured report:")
print(json.dumps(report, indent=2))
print("\nExecutive summary:")
print(summarize(report))Example output:
Schema: alpaca
Structured report:
{
"schema_detected": "alpaca",
"total_records_analyzed": 7,
"issues": [
{"severity": "high", "category": "duplicate", "record_index": 2, "description": "Identical instruction-output pair to record 0"},
{"severity": "high", "category": "pii", "record_index": 3, "description": "Contains email address alice@example.com"},
{"severity": "high", "category": "empty_field", "record_index": 4, "description": "All fields are empty"},
{"severity": "medium", "category": "quality", "record_index": 5, "description": "Output is excessively repetitive"}
],
"summary": "Dataset appears to follow Alpaca schema. Found 1 duplicate, 1 PII leak, 1 empty record, and 1 low-quality repetitive output."
}
Executive summary:
- **Schema:** Alpaca-style instruction tuning
- **High Severity Issues:**
- Record 2 is a duplicate of Record 0
- Record 3 leaks an email address
- Record 4 is completely empty
- **Recommendation:** Remove duplicates and PII before training.Wrap-up and next steps
This profiler is a starting point, not a production pipeline. A concrete next step is to wire it into a CI job that rejects commits when PII or duplicates are detected above a threshold. You could also parallelize it across shards of a large corpus.
If you scale this to massive datasets, Oxlo.ai's request-based pricing becomes a clear win. Profiling a batch of 100K tokens costs the same flat rate as a 1K token ping, so you can stuff large context windows for deep analysis without the bill scaling linearly. For details, see https://oxlo.ai/pricing.
Top comments (0)