Large Language Model Training Datasets: A Comprehensive Overview

#learnai #oxlo #ai

We are building a Dataset Profiler agent that scans raw text samples and flags toxic content, PII leaks, and domain mismatches before they enter a fine-tuning pipeline. It helps ML engineers and data curators audit corpora in minutes instead of manually reviewing thousands of rows.

What you'll need

Python 3.10 or newer
The OpenAI SDK: pip install openai
An Oxlo.ai API key from https://portal.oxlo.ai

Step 1: Set up the client and environment

I start by importing the OpenAI SDK and pointing it at Oxlo.ai. Because Oxlo.ai is fully OpenAI-compatible, the only change is the base_url.

from openai import OpenAI
import os

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ.get("OXLO_API_KEY", "YOUR_OXLO_API_KEY")
)

Step 2: Prepare a sample dataset

I create a local JSONL file that mimics the rows you would pull from a real pre-training or instruction-tuning corpus. Four rows cover a clean instruction, toxic text, a PII leak, and a code snippet.

import json

samples = [
    {"id": 1, "text": "Explain the importance of data cleaning in machine learning pipelines."},
    {"id": 2, "text": "You are stupid and nobody likes you. Go away forever."},
    {"id": 3, "text": "Contact me at john.doe@example.com if you want the dataset. I live at 123 Main St."},
    {"id": 4, "text": "def fib(n):\n    if n < 2: return n\n    return fib(n-1) + fib(n-2)"}
]

with open("sample_dataset.jsonl", "w") as f:
    for s in samples:
        f.write(json.dumps(s) + "\n")

data = [json.loads(line) for line in open("sample_dataset.jsonl")]

Step 3: Write the profiler system prompt

The system prompt turns the model into a structured auditor. I ask for strict JSON so I can parse the result without regex hacks.

SYSTEM_PROMPT = """You are a dataset quality auditor. Analyze the user-supplied text sample and return a single JSON object with these exact keys:
- content_type: one of [instruction, toxic, pii, code, other]
- language: ISO-639-1 code or \"unknown\"
- pii_detected: boolean
- toxicity_risk: one of [low, medium, high]
- recommended_action: one of [keep, review, discard]
- reasoning: one concise sentence

Return only the JSON object. Do not wrap it in markdown fences."""

Step 4: Run the dataset through Oxlo.ai

I loop over the samples and call Llama 3.3 70B through Oxlo.ai. Because Oxlo.ai bills per request, not per token, I can pass the full system prompt and long samples without worrying about ballooning costs for every single row. That makes this audit pattern cheap enough to run on millions of records.

import json
from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

results = []

for row in data:
    user_message = f"Sample ID {row['id']}:\n{row['text']}"

    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
    )

    raw = response.choices[0].message.content.strip()
    if raw.startswith("

```"):
        raw = raw.split("\n", 1)[1].rsplit("```

", 1)[0].strip()

    parsed = json.loads(raw)
    parsed["id"] = row["id"]
    results.append(parsed)
    print(f"Processed ID {row['id']}: {parsed['recommended_action']}")

Step 5: Generate the report

Finally, I aggregate the JSON outputs into a human-readable summary. In production, you would stream these rows into a database, but a printed table is enough to verify the pipeline works.

discard_count = sum(1 for r in results if r["recommended_action"] == "discard")
review_count = sum(1 for r in results if r["recommended_action"] == "review")
languages = set(r["language"] for r in results)

print("\n=== Dataset Audit Report ===")
print(f"Total samples: {len(results)}")
print(f"Discard: {discard_count}")
print(f"Review: {review_count}")
print(f"Languages found: {', '.join(sorted(languages))}")
print("\nDetailed results:")
for r in results:
    print(f"ID {r['id']} | {r['content_type']:8} | risk {r['toxicity_risk']:6} | action {r['recommended_action']:7} | {r['reasoning']}")

Run it

Save the full script as profiler.py, set your key, and execute it.

export OXLO_API_KEY="sk-oxlo.ai-..."
python profiler.py

Expected output looks like this:

Processed ID 1: keep
Processed ID 2: discard
Processed ID 3: discard
Processed ID 4: keep

=== Dataset Audit Report ===
Total samples: 4
Discard: 2
Review: 0
Languages found: en

Detailed results:
ID 1 | instruction | risk low    | action keep    | Clean educational instruction suitable for training.
ID 2 | toxic       | risk high   | action discard | Contains direct insults and harmful language.
ID 3 | pii         | risk low    | action discard | Contains email and street address.
ID 4 | code        | risk low    | action keep    | Valid Python function with no harmful content.

Wrap-up and next steps

This agent gives you a repeatable, programmable way to sanity-check training data before it hits your GPU cluster. If you are processing high-volume crawls, switch the model to deepseek-v3.2 for stronger reasoning on code-heavy subsets, or scale out the loop with a thread pool. Because Oxlo.ai charges a flat rate per request, you can audit long documents or add heavy system prompts without the cost surprises you get from token-based providers. See the latest plans at https://oxlo.ai/pricing.

DEV Community