How to Build an AI Training Data Pipeline with the Web to Markdown/JSON API

Every AI engineer knows the bottleneck: getting clean, structured training data from the web. Whether you're fine-tuning an LLM, building a RAG application, or curating a domain-specific dataset, extracting web content into usable formats is a constant headache. You deal with messy HTML, JavaScript-rendered pages, rate limits, and boilerplate that pollutes your training corpus.

Enter the Web to Markdown/JSON API — a lightweight, no-nonsense service that converts any webpage into clean Markdown or structured JSON in a single API call. In this tutorial, I'll show you how to build a complete AI training data pipeline using it.

Why Your LLM Data Pipeline Needs This

When curating training data for language models, three things matter most:

1. Clean text — No HTML tags, no nav bars, no cookie banners. Just the content.
2. Consistent format — Every document in your pipeline should have the same structure.
3. Reproducibility — You need to re-scrape sources when content updates.

The Web to Markdown/JSON API gives you all three. It strips away the cruft, normalizes the output, and the deterministic endpoint means the same URL always produces the same structure.

The API at a Glance

Detail	Value
Endpoint	POST https://web2md-api-production-d822.up.railway.app/extract
Free Tier	50 requests/day
RapidAPI	Web to Markdown/JSON API

Request body:

{
  "url": "https://en.wikipedia.org/wiki/Transformer_(machine_learning_model)",
  "format": "markdown",
  "max_length": 50000
}

Response (Markdown mode):

{
  "title": "Transformer (machine learning model)",
  "content": "# Transformer (machine learning model)\n\nThe **transformer** is a deep learning architecture...",
  "url": "https://en.wikipedia.org/wiki/Transformer_(machine_learning_model)",
  "format": "markdown",
  "length": 28456
}

Three format options: markdown (headers, lists, code blocks intact), json (structured with sections), and text (plain, stripped-down).

Step 1: Building the Ingestion Pipeline

Let's build a Python pipeline that fetches web pages, stores them as clean Markdown, and prepares them for fine-tuning. First, install dependencies:

pip install requests tqdm datasets

Now, the core ingestion script:

import requests
import json
import time
from pathlib import Path
from typing import Optional

API_URL = "https://web2md-api-production-d822.up.railway.app/extract"

def extract_webpage(url: str, fmt: str = "markdown", max_length: int = 50000) -> Optional[dict]:
    """Extract a single webpage into clean Markdown or JSON."""
    resp = requests.post(
        API_URL,
        json={"url": url, "format": fmt, "max_length": max_length},
        timeout=30
    )
    if resp.status_code == 200:
        return resp.json()
    print(f"Failed {url}: {resp.status_code}")
    return None

def build_training_corpus(
    urls: list[str],
    output_dir: str = "./training_data",
    format: str = "markdown",
    delay: float = 1.0
) -> list[dict]:
    """Ingest a list of URLs and save as clean documents."""
    output_path = Path(output_dir)
    output_path.mkdir(parents=True, exist_ok=True)
    documents = []

    for i, url in enumerate(urls):
        print(f"[{i+1}/{len(urls)}] Extracting: {url[:80]}...")
        result = extract_webpage(url, fmt=format)

        if result and result.get("content"):
            doc = {
                "source": url,
                "title": result.get("title", ""),
                "content": result["content"],
                "length": result.get("length", 0)
            }
            documents.append(doc)

            # Save individual file
            safe_name = url.replace("https://", "").replace("/", "_")[:120]
            ext = "json" if format == "json" else "md"
            file_path = output_path / f"{i:04d}_{safe_name}.{ext}"
            file_path.write_text(result["content"], encoding="utf-8")

        time.sleep(delay)  # Respect rate limits

    # Save manifest
    manifest = {
        "total_documents": len(documents),
        "total_chars": sum(d["length"] for d in documents),
        "format": format,
        "sources": [d["source"] for d in documents]
    }
    (output_path / "manifest.json").write_text(json.dumps(manifest, indent=2))
    print(f"\nDone! {len(documents)} documents saved to {output_dir}/")
    return documents

# Example usage
if __name__ == "__main__":
    urls = [
        "https://en.wikipedia.org/wiki/Attention_Is_All_You_Need",
        "https://arxiv.org/abs/1706.03762",
        "https://huggingface.co/blog/llama2",
        "https://lilianweng.github.io/posts/2023-01-27-the-transformer-family-v2/",
    ]
    build_training_corpus(urls)

Run it and watch your training_data/ directory fill with clean, ready-to-use Markdown files — no HTML soup in sight.

Step 2: Preparing for LLM Fine-Tuning

Now that we have clean documents, let's format them for fine-tuning with Hugging Face's datasets library:

from datasets import Dataset
import json
from pathlib import Path

def prepare_finetuning_dataset(
    input_dir: str = "./training_data",
    instruction_template: str = None
) -> Dataset:
    """Convert ingested Markdown files into a HuggingFace Dataset for fine-tuning."""
    if instruction_template is None:
        instruction_template = (
            "You are a knowledgeable AI assistant. Read the following document "
            "and answer questions about it accurately.\n\nDocument:\n{content}"
        )

    records = []
    for md_file in sorted(Path(input_dir).glob("*.md")):
        content = md_file.read_text(encoding="utf-8")
        records.append({
            "source": md_file.name,
            "content": content,
            "instruction": instruction_template.format(content=content),
            "length": len(content)
        })

    # Filter out very short or empty documents
    records = [r for r in records if r["length"] > 500]

    dataset = Dataset.from_list(records)
    return dataset

# Load and inspect
dataset = prepare_finetuning_dataset("./training_data")
print(f"Dataset size: {len(dataset)} documents")
print(f"Total tokens (approx): {sum(dataset['length']) // 4:,}")

# Push to HuggingFace Hub (uncomment when ready)
# dataset.push_to_hub("your-username/your-domain-corpus")

Step 3: Building a RAG Ingestion Pipeline

For Retrieval-Augmented Generation, the JSON format shines. Here's how to build a chunking pipeline:

import requests
import json
from langchain.text_splitter import RecursiveCharacterTextSplitter

def ingest_for_rag(urls: list[str], chunk_size: int = 1000) -> list[dict]:
    """Fetch webpages as JSON and split into RAG-ready chunks."""
    text_splitter = RecursiveCharacterTextSplitter(
        chunk_size=chunk_size,
        chunk_overlap=200,
        separators=["\n\n", "\n", ".", " ", ""]
    )
    all_chunks = []

    for url in urls:
        resp = requests.post(
            "https://web2md-api-production-d822.up.railway.app/extract",
            json={"url": url, "format": "json", "max_length": 50000}
        )
        if resp.status_code != 200:
            continue

        data = resp.json()
        content = data["content"]

        # If JSON-format, content may be a dict with sections
        text = content if isinstance(content, str) else json.dumps(content)

        chunks = text_splitter.split_text(text)
        for chunk in chunks:
            all_chunks.append({
                "source": url,
                "title": data.get("title", ""),
                "chunk": chunk,
                "chunk_length": len(chunk)
            })

    return all_chunks

Each chunk is now ready to be embedded and stored in your vector database (Pinecone, Weaviate, ChromaDB, etc.).

Why Not Scrape It Yourself?

You could write your own scraper with BeautifulSoup and Trafilatura. But then you'd need to:

• Handle JavaScript-rendered pages (headless browser overhead)
• Deal with different site structures (no two websites use the same HTML)
• Manage proxies and IP rotation for scale
• Maintain extraction logic as sites change their markup

The API abstracts all of that. One endpoint. Predictable output. 50 free requests per day to test and prototype — more than enough for most experimentation.

Real-World Use Cases

Here's how developers are already using this API:

Use Case	Format	Why It Works
Domain-specific LLM fine-tuning	Markdown	Clean docs without boilerplate improve training quality
RAG knowledge bases	JSON	Structured sections make chunking smarter
Competitive intelligence	Text	Monitor competitor pages with diffable plain text
Academic research corpora	Markdown	ArXiv papers → readable training data
Website change monitoring	Text	Compare extracted text to detect content updates

Production Tips

1. Batch responsibly: The free tier gives you 50 requests/day. Space them out with time.sleep(1).
2. Validate URLs: Pre-check that URLs return 200 before sending them to the API.
3. Deduplicate: Hash your extracted content and skip near-duplicates (common across doc pages).
4. Store raw responses: Always save the full API response — you might want to re-chunk later.

def deduplicate_documents(documents: list[dict]) -> list[dict]:
    """Remove near-duplicate documents using content hashing."""
    import hashlib
    seen = set()
    unique = []
    for doc in documents:
        h = hashlib.md5(doc["content"][:1000].encode()).hexdigest()
        if h not in seen:
            seen.add(h)
            unique.append(doc)
    return unique

Getting Started

1. Head to the RapidAPI page and subscribe (free tier available).
2. Copy the endpoint: https://web2md-api-production-d822.up.railway.app/extract
3. Run the pipeline script above against your target URLs.
4. Build your dataset, embed your chunks, or feed it into your fine-tuning job.

The API handles the messy part of web data extraction so you can focus on what actually matters: training better models and shipping faster.

---

Got a cool use case? Drop a comment below — I'd love to see what datasets you're building!