DEV Community

Nilofer 🚀
Nilofer 🚀

Posted on

Dataset Factory: A Production-Grade Benchmark Dataset Factory for AI Agent Evaluation

#agents #llm #mlops #opensource

Evaluating AI agents requires benchmark datasets that are high-quality, diverse, balanced, and free of duplicates. Building those datasets by hand is slow, inconsistent, and hard to reproduce. The Mercor Dataset Factory automates the entire pipeline: generate, validate, deduplicate, analyze coverage, remediate gaps, audit, govern, and verify, all with a single command.

It is not a simple data generator. It is a complete data engineering platform purpose-built for creating, validating, and releasing benchmark datasets for AI agent evaluation.

Overview

The factory produces a 1,000-record benchmark dataset spanning 10 agent-task categories with balanced difficulty distribution. Each record is a structured evaluation task containing a natural language query, ground-truth plan, expected tools, evaluation criteria, and success conditions.

Every phase has a quality gate. If validation fails, the pipeline halts. If duplicates are found, they are automatically removed and replaced. If coverage gaps exist, they are remediated. The result is a production-ready benchmark with guaranteed quality guarantees.

Features

  • Config-driven architecture: Categories are self-contained drop-in pack files in categories/*.yaml. Global settings live in config.yaml. No Python changes needed to customize.

  • Extensible categories: Add a category by dropping a new pack file into categories/. Remove one by deleting its pack. Packs are auto-discovered and authoritative for the category set.

  • 10 benchmark categories: Coding, Web Research, Data Analysis, System Design, Debugging, Document Processing, Tool Usage, Multi-Step Reasoning, Operations/DevOps, Agent Coordination, each shipped as a pack in categories/

  • Balanced distribution: Exactly 100 records per category, 330/340/330 split across easy/medium/hard

  • Source/config preflight: A pydantic-based validator (factory/config_validator.py) checks the configuration before any work runs, catching incomplete categories, bad/duplicate prefixes, unknown template placeholders, and inconsistent counts

  • Canonical record contract: Every record is validated against a declared JSON Schema (schema/record.schema.json) in addition to the inline validation rules

  • Near-duplicate-aware generation: The generator rejects near-duplicate queries at creation time using a pure-stdlib term-frequency cosine guard, plus a 3-layer dedup pass (exact MD5 hash, TF-IDF cosine, semantic embedding)

  • 100% coverage guarantee: Automatic gap detection and remediation for tools (127) and domains (118)

  • 8-dimensional auditing: Schema quality, diversity, coverage, difficulty balance, tool diversity, duplicate rate, auditability, reproducibility, weighted to a single 0-100 score

  • Independent verification: A separate Python script (verification.py) with zero imports from the factory package validates independently

  • Full governance: Dataset card, release notes, methodology document, lineage tracking, versioning strategy

  • Fully reproducible: Single --seed 42 flag locks every stochastic component

Project Structure 

mercorDatasetFactory/
│
├── README.md                          ← This file
├── CONTRIBUTING.md                    ← Contributor guide (setup, tests, conventions)
├── LICENSE                            ← MIT License
├── pyproject.toml                     ← Packaging metadata + `dataset-factory` entry point
├── requirements.txt                   ← Runtime dependencies
├── requirements-dev.txt               ← Dev/test dependencies (pytest, coverage)
├── dataset_factory.py                 ← Main orchestrator — the CLI entry point
├── verification.py                    ← Independent verification (zero factory imports)
├── config.yaml                        ← GLOBAL configuration only (targets, difficulties, schema constraints, seed)
│
├── categories/                        ← Category packs — one self-contained YAML per category (authoritative)
│   ├── coding.yaml
│   ├── web_research.yaml
│   ├── ...                            ← 10 built-in packs; drop a new file to add a category
│   └── agent_coordination.yaml
│
├── schema/
│   └── record.schema.json             ← Canonical record JSON Schema (the data contract)
│
├── factory/                           ← Core factory package
│   ├── __init__.py
│   ├── config_loader.py               ← Config + category-pack loading, deep-merge, defaults inline
│   ├── config_validator.py            ← Pydantic source/config preflight validator
│   ├── record_schema.py               ← Builds the JSON Schema record contract from config
│   ├── schema.py                      ← Constants, enums, field definitions, schema constraints
│   ├── categories.py                  ← Category registries (loaded from packs via config_loader)
│   ├── templates.py                   ← Query/plan/criteria/success prompt templates
│   ├── generator.py                   ← Stratified sampling, near-duplicate guard & record construction
│   ├── validator.py                   ← Schema + JSON Schema contract validation & distribution analysis
│   ├── deduplicator.py                ← 3-layer dedup (exact, TF-IDF, semantic)
│   ├── coverage.py                    ← Coverage matrices & over/under-representation analysis
│   ├── remediator.py                  ← Automatic gap remediation & supplemental generation
│   ├── auditor.py                     ← 8-dimensional weighted quality scoring
│   └── governance.py                  ← Governance document generation (card, notes, methodology)
│
├── tests/                             ← Test suite (run with `pytest -q`)
├── .github/                           ← CI workflows (must pass on every change)
│
├── artifacts/                         ← All pipeline outputs (git-ignored; regenerated each run)
│   ├── dataset.jsonl                  ★ Final 1,000-record benchmark dataset
│   ├── validation_report.md           ★ Schema, contract & distribution validation results
│   ├── duplicate_analysis.md          ★ 3-layer dedup analysis with examples
│   ├── coverage_report.md             ★ Tool/domain/category coverage matrices
│   ├── remediation_report.md          ★ Gap remediation log (if any gaps found)
│   ├── audit_report.md                ★ 8-dimensional quality score report
│   ├── governance.md                  ★ Dataset lineage, assumptions, versioning
│   ├── release_notes.md               ★ Release documentation
│   ├── dataset_card.md                ★ Dataset card (HuggingFace-style)
│   └── generation_methodology.md      ★ Full methodology for reproducibility
│
├── plans/                             ← Implementation plans
│   ├── plan.md
│   └── config_driven_plan.md
│
└── test_custom_config.yaml            ← Example custom config for testing
Enter fullscreen mode Exit fullscreen mode

Quick Start

Prerequisites

  • Python 3.8+
  • pip (Python package installer)
  • ~4 GB RAM (sentence-transformers model loads to ~800MB, preprocessing on CPU)
  • No GPU required: all phases run on CPU

1. Set Up Environment

# Clone or navigate to the project
cd mercorDatasetFactory

# Create a virtual environment (recommended)
python3 -m venv venv
source venv/bin/activate     # Linux/macOS
# .\venv\Scripts\activate    # Windows
Enter fullscreen mode Exit fullscreen mode

2. Install Dependencies

pip install -r requirements.txt
Enter fullscreen mode Exit fullscreen mode

Alternatively:

pip install scikit-learn sentence-transformers pyyaml pydantic
Enter fullscreen mode Exit fullscreen mode

Note: scikit-learn is used for TF-IDF vectorization in the deduplication layer. sentence-transformers provides the all-MiniLM-L6-v2 embedding model for semantic duplicate detection. pyyaml handles config and category-pack loading. pydantic (v2) powers the source/config preflight validator.

3. Run the Full Pipeline

python3 dataset_factory.py
Enter fullscreen mode Exit fullscreen mode

The pipeline will:

  • Generate 1,000 records using seed=42 (reproducible)
  • Validate every record against schema, field constraints, and distribution targets
  • Deduplicate using 3-layer detection (exact, TF-IDF, semantic), removing and replacing flagged records
  • Analyze coverage of all 127 tools and 118 domains across 10 categories
  • Remediate any gaps automatically with supplemental records
  • Audit the dataset across 8 quality dimensions with weighted scoring
  • Generate governance documents: dataset card, release notes, methodology
  • Run independent verification: 6 checks resulting in PASS/FAIL

All outputs appear in artifacts/.

4. Verify Independence

python3 verification.py
Enter fullscreen mode Exit fullscreen mode

Expected output:

============================================================
OVERALL: PASS (6/6 checks passed)
============================================================
Enter fullscreen mode Exit fullscreen mode

This script has zero imports from the factory/ package. It independently validates the dataset by reading artifacts/dataset.jsonl directly.

CLI Reference

python3 dataset_factory.py [OPTIONS]
Enter fullscreen mode Exit fullscreen mode

Examples

# Full factory run (reproducible default)
python3 dataset_factory.py

# Custom seed and record count
python3 dataset_factory.py --seed 123 --count 500

# Use a custom configuration
python3 dataset_factory.py --config my_custom_config.yaml

# Re-validate and re-audit an existing dataset
python3 dataset_factory.py --skip-generation

# Validate the configuration and category packs without generating anything
python3 dataset_factory.py --validate-config

# Custom config + custom seed + custom count
python3 dataset_factory.py --config my_config.yaml --seed 7 --count 2000
Enter fullscreen mode Exit fullscreen mode

Source/config validation

Before any generation work, the pipeline runs a pydantic-based preflight (factory/config_validator.py) over the merged configuration and the discovered category packs. It aggregates all problems into a single report and refuses to run if there are hard errors: incomplete categories, malformed or duplicate prefixes, unknown template placeholders (allowed: domain, topic, code_snippet, claim, n, stack_trace, task_type). Count drift is surfaced as a non-fatal warning. Run the preflight on its own with --validate-config:

Config validation: PASS — configuration is coherent.
Enter fullscreen mode Exit fullscreen mode

Pipeline Architecture

Each phase has a specific responsibility and a quality gate. If any phase fails its gate, the pipeline halts with a clear error message.

Phase 1: Generation

Engine: factory/generator.py → DatasetGenerator

Stratified sampling across 10 categories x 3 difficulty levels (30 strata). Each record is constructed from templates with randomized domain, tool, and task-type substitutions. The generator maintains running counters to ensure exact balance.

Quality gate: Target distribution (±1 record per category per difficulty). If the generator cannot meet targets after 2x attempts, it reports generation statistics for diagnosis.

Phase 2: Validation

Engine: factory/validator.py → validate_dataset()

Validates every record against the schema definition:

  • All 9 required fields present with correct types
  • ID format: {CATEGORY_PREFIX}_{SEQ:03d} (e.g., COD_001, WEB_042)
  • Query length: 50–500 characters
  • Plan steps: 3–8 (varies by difficulty)
  • Evaluation criteria: ≥2
  • Success conditions: ≥1
  • Tools: ≥1
  • Category distribution: 100 ± 5 per category
  • Difficulty distribution: easy ±5%, medium ±5%, hard ±5%
  • Query diversity: unique queries ≥ 95%

Gate: Pipeline halts if schema validation fails. Distribution warnings are reported but non-fatal.

Phase 3: Deduplication

Engine: factory/deduplicator.py → DatasetDeduplicator

Three-layer duplicate detection, applied in sequence:

Flagged duplicate records are removed and replacement records are automatically generated to maintain exactly 1,000 records with balanced category and difficulty distributions.

Gate: Pipeline proceeds regardless (dedup always succeeds, but high duplicate rates are reflected in the audit score).

Phase 4: Coverage Analysis

Engine: factory/coverage.py → analyze_coverage()

Builds coverage matrices for three dimensions:

  • Tool coverage: All 127 registered tools appear in at least one record
  • Domain coverage: All 118 registered domains appear in at least one record
  • Category x Difficulty: No empty stratum

Also checks for over/under-representation using a ±20% threshold from expected distribution.

Phase 5: Gap Remediation

Engine: factory/remediator.py → remediate()

If any gaps are found in Phase 4, the remediator identifies the specific missing tools/domains, generates supplemental records targeting those gaps, tracks per-category sequence numbers to avoid ID collisions, and merges supplemental records into the dataset.

Gate: Pipeline halts if remediation fails.

Phase 6: Audit

Engine: factory/auditor.py → audit_dataset()

Eight quality dimensions, each scored 0–100 and weighted:

Rating scale: EXCELLENT (≥90), GOOD (≥75), FAIR (≥60), POOR (<60)

Phase 7: Governance

Engine: factory/governance.py → generate_all()

Generates four documents automatically: governance.md, release_notes.md, dataset_card.md, and generation_methodology.md.

Phase 8: Independent Verification

Engine: verification.py (standalone script)

Gate: Final release gate. The dataset must pass 6/6 before it can be considered production-ready.

Configuration System

How Configuration Works
Configuration is split into two concerns:

  • Category packs (categories/*.yaml): one self-contained file per category, auto-discovered and authoritative
  • Global config (config.yaml): globals only: record targets, difficulties, SCHEMA_CONSTRAINTS, DEFAULT_SEED, and difficulty step ranges/tags

The factory uses a layered configuration architecture:

[Python defaults in config_loader.py]
            │
            ▼
[category packs in categories/*.yaml]  ← discovered & merged (authoritative for categories)
            │
            ▼
[config.yaml at project root]  ← deep-merge globals over defaults
            │
            ▼
[--config CLI flag / env var]  ← deep-merge over everything
            │
            ▼
[Module-level globals]
(schema.py, categories.py, templates.py → generator.py, etc.)
Enter fullscreen mode Exit fullscreen mode

The deep-merge algorithm: dictionaries are merged recursively; lists and scalars are replaced entirely by the user config (not appended to).

Using a Custom Config

# Via CLI flag
python3 dataset_factory.py --config my_dataset_config.yaml

# Via environment variable
export DATASET_FACTORY_CONFIG=/path/to/config.yaml
python3 dataset_factory.py

# CLI overrides env var
python3 dataset_factory.py --config override.yaml
Enter fullscreen mode Exit fullscreen mode

What You Can Customize

config.yaml controls the global settings: difficulty ratios, record targets, field bounds, the default seed, and difficulty step ranges and tags. Edit this file to change how many records are generated, how they are split across difficulty levels, or what constraints apply to fields like query length and plan steps.

categories/.yaml controls everything per category: the category name, its unique record ID prefix, the tasks it covers, the tools and domains available to it, the query and plan step templates, evaluation criteria and success condition templates, and an optional topic pool. Add a category by dropping a new pack file. Remove one by deleting it. These packs are the authoritative source for the category set and all per-category data.

Example: Custom Globals Config

# my_dataset_config.yaml — globals only; categories come from categories/*.yaml

TOTAL_RECORDS: 500
RECORDS_PER_CATEGORY: 50
DIFFICULTY_TARGETS:
  easy: 25
  medium: 25
  hard: 0        # Skip hard difficulty entirely
GLOBAL_DIFFICULTY_TARGETS:
  easy: 250
  medium: 250
  hard: 0
SCHEMA_CONSTRAINTS:
  user_query_min_length: 50
  user_query_max_length: 500
Enter fullscreen mode Exit fullscreen mode

Config File Structure

TOTAL_RECORDS: 1000
RECORDS_PER_CATEGORY: 100
SCHEMA_CONSTRAINTS:
  user_query_min_length: 50
  user_query_max_length: 500
  plan_min_steps: 3
  plan_max_steps: 8
  min_tools: 1
  max_tools: 10
  min_criteria: 2
  max_criteria: 6
  min_conditions: 1
  max_conditions: 5
  min_domains: 1
  max_domains: 4
Enter fullscreen mode Exit fullscreen mode

Programmatic Reload

from factory.config_loader import reload_config

# Load a different config at runtime (optionally validate it)
reload_config("/path/to/custom_config.yaml", validate=True)

# All downstream modules immediately reflect the new values
from factory.schema import TOTAL_RECORDS, SCHEMA_CONSTRAINTS
print(TOTAL_RECORDS)                              # Updated record target
print(SCHEMA_CONSTRAINTS['user_query_max_length'])  # Updated field bound
Enter fullscreen mode Exit fullscreen mode

Adding a Category

Categories are self-contained drop-in pack files. To add a category, no Python or config.yaml changes are needed. Drop a new categories/<name>.yaml file. Packs are auto-discovered and authoritative for the category set.

Allowed template placeholders: domain, topic, code_snippet, claim, n, stack_trace, task_type.

1. Drop a pack file

# categories/incident_response.yaml
name: incident_response
order: 11
prefix: INC
tasks:
  - triage
  - root_cause_analysis
  - postmortem
tools:
  - pager
  - log_search
  - metrics_dashboard
  - runbook
domains:
  - payments
  - identity
  - data_platform
topic_pool:
  - a cascading service outage
  - a degraded p99 latency spike
query_templates:
  - "Triage {topic} in the {domain} system and identify the {task_type} steps for {n} on-call engineers."
  - "Write a postmortem for {topic} affecting {domain}, including timeline and remediation."
plan_step_templates:
  - "Acknowledge the alert for {topic} and assemble responders"
  - "Correlate logs and metrics to localize {topic}"
  - "Apply mitigation and verify recovery of {domain}"
evaluation_criteria_templates:
  - "Root cause of {topic} is correctly identified"
  - "Mitigation restores the {domain} service"
success_condition_templates:
  - "Incident is resolved and a postmortem is filed"
Enter fullscreen mode Exit fullscreen mode

2. Validate the config (preflight)

python3 dataset_factory.py --validate-config
Enter fullscreen mode Exit fullscreen mode

3. Run the pipeline as usual

python3 dataset_factory.py --seed 42
Enter fullscreen mode Exit fullscreen mode

NUM_CATEGORIES is derived automatically from the discovered packs, so the new category is picked up with no further changes.

Removing or relocating packs

Remove a category: delete its categories/<name>.yaml pack.

Use a different packs directory:

export DATASET_FACTORY_CATEGORIES_DIR=/path/to/my/packs
python3 dataset_factory.py --validate-config
Enter fullscreen mode Exit fullscreen mode

Programmatic API

Each factory module can be imported independently for custom pipelines or integration into larger systems.

Basic Usage

# Generate a dataset
from factory.generator import DatasetGenerator

gen = DatasetGenerator(seed=42)
records = gen.generate(1000)
gen.save(records, "artifacts/dataset.jsonl")

# Validate
from factory.validator import validate_dataset

result = validate_dataset("artifacts/dataset.jsonl", "artifacts/validation_report.md")
print(f"Status: {result['overall']['status']}")   # "PASS" or "FAIL"
print(f"Checks: {result['overall']['passed']}/{result['overall']['total']}")
Enter fullscreen mode Exit fullscreen mode

Full Pipeline Programmatically

from factory.generator import DatasetGenerator
from factory.validator import validate_dataset
from factory.deduplicator import DatasetDeduplicator
from factory.coverage import analyze_coverage
from factory.remediator import remediate
from factory.auditor import audit_dataset
from factory.governance import generate_all

# Generate
gen = DatasetGenerator(seed=42)
records = gen.generate(1000)
gen.save(records, "artifacts/dataset.jsonl")

# Validate
validation = validate_dataset("artifacts/dataset.jsonl", "artifacts/validation_report.md")
if validation['overall']['status'] != 'PASS':
    exit(1)

# Deduplicate
import json
deduper = DatasetDeduplicator(records)
analysis = deduper.analyze_all()
deduped, dedup_analysis = deduper.deduplicate()      # returns (records, analysis)
with open("artifacts/dataset.jsonl", "w") as f:
    for rec in deduped:
        f.write(json.dumps(rec) + "\n")
print(f"Removed {len(records) - dedup_analysis['remaining_records']} duplicates")

# Coverage
coverage = analyze_coverage("artifacts/dataset.jsonl", "artifacts/coverage_report.md")
print(f"Tool coverage: {coverage['tool_coverage']['global']['overall_coverage_pct']}")

# Remediate
remediation = remediate(
    "artifacts/dataset.jsonl",
    coverage_results=coverage,
    output_path="artifacts/remediation_report.md",
    updated_dataset_path="artifacts/dataset.jsonl",
)
print(f"Gaps: {remediation['gaps_identified']}, Generated: {remediation['supplemental_generated']}")

# Audit
audit = audit_dataset("artifacts/dataset.jsonl", "artifacts/audit_report.md", dedup_results=analysis)
print(f"Score: {audit['overall_score']:.2f}/100 ({audit['rating']})")

# Governance
generate_all("artifacts/dataset.jsonl")
Enter fullscreen mode Exit fullscreen mode

Deduplicator Details

from factory.deduplicator import DatasetDeduplicator

deduper = DatasetDeduplicator(records)

# Run all 3 layers (consolidated)
full_analysis = deduper.analyze_all()
print(f"Total duplicates found: {full_analysis['total_duplicates_found']}")

# Or run layers individually
exact_results = deduper.find_exact_duplicates()
tfidf_results = deduper.find_tfidf_duplicates()
semantic_results = deduper.find_semantic_duplicates()

# Remove duplicates → (deduplicated_records, analysis)
deduped, analysis = deduper.deduplicate()
print(f"Remaining after dedup: {analysis['remaining_records']}")
Enter fullscreen mode Exit fullscreen mode

Coverage Analyzer

from factory.coverage import analyze_coverage

result = analyze_coverage("artifacts/dataset.jsonl", "artifacts/coverage_report.md")

# Per-category tool coverage
for cat, cov in result['tool_coverage']['per_category'].items():
    used = cov['unique_tools_used']
    total = cov['tools_available']
    pct = cov['coverage_pct']
    print(f"{cat}: {used}/{total} tools ({pct})")

# Global coverage stats
global_cov = result['tool_coverage']['global']
print(f"Overall tool coverage: {global_cov['overall_coverage_pct']}")
Enter fullscreen mode Exit fullscreen mode

Output Artifacts

The pipeline writes 10 artifacts to the artifacts/ directory (git-ignored, regenerated by every run):

dataset.jsonl is the final 1,000-record benchmark dataset, one JSON object per line.

validation_report.md contains per-check PASS/FAIL results from schema, JSON Schema contract, and distribution validation.

duplicate_analysis.md contains the 3-layer deduplication results with examples of detected duplicates.

coverage_report.md contains tool, domain, and category coverage matrices with representation analysis.

remediation_report.md logs gap identification and supplemental generation if any gaps were found.

audit_report.md contains the 8-dimensional quality score with weighted overall score and rationale.

governance.md documents dataset lineage, assumptions, limitations, versioning, and release process.

release_notes.md contains semantic versioning release notes.

dataset_card.md is the dataset card in HuggingFace-style format.

generation_methodology.md contains the full methodology for reproducibility.

Dataset Record Schema

{
  "id": "COD_001",
  "category": "coding",
  "difficulty": "medium",
  "user_query": "Implement a function that validates JSON schema compliance...",
  "ground_truth_plan": [
    "Parse the input JSON string into a Python dictionary",
    "Load the JSON Schema definition from the provided schema file",
    "Validate the dictionary against the schema using the jsonschema library",
    "Collect all validation errors with paths and messages",
    "Return the validation result with pass/fail status and error details"
  ],
  "expected_tools": [
    "code_editor",
    "linter",
    "test_runner",
    "static_analyzer"
  ],
  "evaluation_criteria": [
    "Correctly validates all schemas and reports errors",
    "Handles edge cases (empty objects, nested schemas, $ref)",
    "Tests cover at least 90% of code paths"
  ],
  "success_conditions": [
    "All unit tests pass with >90% code coverage",
    "Code passes linting with zero errors or warnings"
  ],
  "metadata": {
    "domains": ["backend_services", "data_structures"],
    "requires_external_access": false,
    "estimated_steps": 7,
    "tags": ["intermediate", "cross_domain", "independent_steps"]
  }
}
Enter fullscreen mode Exit fullscreen mode

The metadata block always contains domains (list), requires_external_access (bool), estimated_steps (int), and tags (list).

Quality Report

Produced from seed=42, 1,000 records:

Audit Score Breakdown

Extending the Factory

Adding a New Category

Drop a new pack file into categories/, validate, and run:

# 1. Create categories/<name>.yaml
# 2. Preflight the config + packs
python3 dataset_factory.py --validate-config
# 3. Run
python3 dataset_factory.py --seed 42
Enter fullscreen mode Exit fullscreen mode

Adding a New Tool

Edit the relevant category pack and add the tool to its tools list:

# categories/coding.yaml
tools:
  - code_editor
  - linter
  - test_runner
  - my_new_tool   # ← Add it here
Enter fullscreen mode Exit fullscreen mode

Then re-run python3 dataset_factory.py --validate-config and the pipeline.

Creating a Custom Pipeline

Import the factory modules directly in Python (see Programmatic API above).

Reproducibility

To reproduce the exact same dataset:

python3 dataset_factory.py --seed 42 --count 1000
Enter fullscreen mode Exit fullscreen mode

The --seed flag propagates to every stochastic component:

Full methodology is documented in artifacts/generation_methodology.md.

Governance and Maintenance

Versioning

Semantic versioning (MAJOR.MINOR.PATCH):

Release Process

  1. Run the config preflight: python3 dataset_factory.py --validate-config → PASS
  2. Run the full pipeline
  3. Verify: python3 verification.py → 6/6 PASS
  4. Check audit score ≥ 90 (EXCELLENT)
  5. Review release_notes.md for accuracy
  6. Tag release in version control

Known Limitations

Documented in artifacts/governance.md:

  • Synthetic queries: All queries are generated from templates, not collected from real users
  • Template bias: Domain/tool distributions reflect template structure
  • Difficulty calibration: Based on step count ranges, not human evaluation
  • No ground-truth execution: Plans are text-based and not executable

Extending for a New Version

# 1. Add/edit category packs in categories/ and/or globals in config.yaml
# 2. Preflight the config + packs
python3 dataset_factory.py --validate-config
# 3. Regenerate
python3 dataset_factory.py --seed 42 --count 1000
# 4. Verify
python3 verification.py
# 5. Check audit score ≥ 90
Enter fullscreen mode Exit fullscreen mode

How I Built This Using NEO

This project was built using NEO. NEO is a fully autonomous AI engineering agent that can write code and build solutions for AI/ML tasks including AI model evals, prompt optimization and end to end AI pipeline development.

The requirement was a production-grade benchmark dataset factory that could generate, validate, deduplicate, audit, and release high-quality evaluation datasets for AI agents with a single command. NEO planned and produced the files in this repository: the main orchestrator, an independent verification script, 14 factory modules covering generation, validation, deduplication, coverage analysis, remediation, auditing, and governance, the canonical record schema, the global config, all 10 category pack files, the test suite, and the CI workflows. The plans/ directory in the repo documents the build run directly.

The result is a fully working 8-phase pipeline that produces a 1,000-record benchmark dataset with a 94.94/100 audit score, 100% tool and domain coverage, zero duplicates, and 6/6 independent verification checks passed.

How You Can Extend This Further With NEO

Use it as the evaluation layer in a CI/CD pipeline for AI agents.
The factory outputs dataset.jsonl and verification.py exits 0 only when 6/6 checks pass. That exit code becomes a CI gate: if the benchmark does not meet quality thresholds, the pipeline fails before the agent is evaluated against bad data. Plugging python3 dataset_factory.py --seed 42 and python3 verification.py as pipeline steps gives every agent deployment a reproducible, audited evaluation layer.

Use --skip-generation to re-audit a dataset after agents modify or filter it.
If a downstream team removes records, rebalances difficulty, or adds domain-specific records to dataset.jsonl, running with --skip-generation re-runs all validation, coverage analysis, deduplication, and auditing against the modified file without regenerating. The audit score immediately shows whether the changes degraded data quality across any of the 8 dimensions.

Use the programmatic API to build a custom evaluation loop.
Every factory module is independently importable. DatasetGenerator, validate_dataset, DatasetDeduplicator, and audit_dataset can be imported directly into an agent testing script, generating a fresh benchmark for each experiment and feeding records to the agent under test, all in one Python process without touching the CLI or the filesystem.

Use a custom category pack to benchmark agents against your own task taxonomy.
If the 10 built-in categories do not match the agent's actual workload, a custom categories/<name>.yaml with domain-specific tasks, tools, and query templates can be dropped in. The factory runs the custom benchmark through the same 8-phase pipeline the built-in categories go through, including deduplication, 100% coverage validation, and governance document generation.

Final Notes

Most benchmark datasets are generated once, checked loosely, and never audited again. The Dataset Factory treats dataset quality the same way production software treats code quality: every run is validated, every gap is remediated, every dimension is scored, and the result is independently verified before it is considered production-ready.

The code is at https://github.com/dakshjain-1616/DataSet-Factory
You can also build with NEO in your IDE using the VS Code extension or Cursor.
You can use NEO MCP with Claude Code: https://heyneo.com/claude-code

Top comments (0)