DEV Community: bao001 xiao

Build Your Own Obsidian Web Clipper with the Web to Markdown/JSON API

bao001 xiao — Mon, 07 Sep 2026 02:04:42 +0000

We've all hit the same wall with read-later apps: you save an article to Pocket or Instapaper, and it works — right up until you want to actually use that content. Your notes are trapped inside someone else's silo, and the "clean view" is nice but it's not yours.

If you live in Obsidian (or any Markdown-based note system), you want something better: the article's clean content saved as plain Markdown into your vault — searchable, linkable, and permanent. The only thing standing in the way is turning a messy web page into clean Markdown.

That's exactly what the Web to Markdown/JSON API does. One HTTP call takes any URL and returns clean Markdown with the nav bars, ads, cookie banners, and scripts stripped away.

In this article you'll build your own web clipper that saves any page into your Obsidian vault as a proper note, in about 50 lines of Python.

The API at a glance

Endpoint:

POST https://web2md-api-production-d822.up.railway.app/extract

Request:

{
  "url": "https://example.com/great-article",
  "format": "markdown",
  "max_length": 50000
}

format — markdown, json, or text
max_length — cap the response size (up to 50000 characters)
Free tier: 50 requests/day
Sign up / grab a key on RapidAPI

Response:

{
  "success": true,
  "title": "The Great Article",
  "content": "# The Great Article\n\n## Introduction\n\n...",
  "description": "A summary of the article",
  "word_count": 1523,
  "response_time_ms": 210
}

The content field is clean Markdown — headings, paragraphs, lists, code blocks, nothing else. That's the whole game: your clipper gets an article that's already ready to live in Obsidian, instead of a pile of HTML you'd have to clean yourself.

The plan

POST a URL to /extract and get Markdown back.
Wrap it in YAML frontmatter (title, source URL, date, tags).
Sanitize the title into a filename.
Save the file into your vault's Inbox folder.

The code

Save this as clip.py:

import re
import sys
from datetime import date
from pathlib import Path

import requests

API_URL = "https://web2md-api-production-d822.up.railway.app/extract"
VAULT = Path.home() / "Documents" / "Obsidian" / "Inbox"  # your vault inbox


def fetch_markdown(url: str) -> dict:
    """Turn any web page into clean Markdown via the API."""
    r = requests.post(
        API_URL,
        json={"url": url, "format": "markdown", "max_length": 50000},
        timeout=30,
    )
    r.raise_for_status()
    return r.json()


def slugify(title: str) -> str:
    """Turn an article title into a safe, readable filename."""
    name = re.sub(r"[^\w\- ]+", "", title).strip().replace(" ", "-")
    return (name[:80] or "untitled") + ".md"


def build_note(data: dict, url: str, tags: list[str]) -> str:
    """Wrap extracted Markdown in Obsidian-friendly frontmatter."""
    frontmatter = "\n".join([
        "---",
        f"title: "\"{data['title']}\"\","
        f"source: {url}",
        f"clipped: {date.today().isoformat()}",
        f"tags: [{', '.join(tags)}]",
        "---",
        "",
    ])
    return frontmatter + data["content"]


def clip(url: str, tags: list[str]) -> Path:
    """Fetch a page and save it into the vault as a Markdown note."""
    data = fetch_markdown(url)
    note = build_note(data, url, tags)

    VAULT.mkdir(parents=True, exist_ok=True)
    out = VAULT / slugify(data["title"])
    out.write_text(note, encoding="utf-8")

    print(f"Saved: {out.name} ({data['word_count']} words)")
    return out


if __name__ == "__main__":
    url = sys.argv[1]
    tags = sys.argv[2:] or ["clippings"]
    clip(url, tags)

Run it

pip install requests
python clip.py https://example.com/great-article "reading" "tech"

The note lands in your vault, ready to be searched, linked, and edited:

---
title: "The Great Article"
source: https://example.com/great-article
clipped: 2026-09-07
tags: [reading, tech]
---

# The Great Article

## Introduction

...clean Markdown, no nav bars or ads...

Make it one click: a bookmarklet

A clipper you run from the terminal is fine, but a clipper you trigger with one click is a tool. Here's a tiny bookmarklet that fires off the same request and copies the Markdown to your clipboard:

javascript:(()=>{fetch("https://web2md-api-production-d822.up.railway.app/extract",{method:"POST",headers:{"Content-Type":"application/json"},body:JSON.stringify({url:location.href,format:"markdown"})}).then(r=>r.json()).then(d=>navigator.clipboard.writeText(d.content)).then(()=>alert("Markdown copied to clipboard"))})();

Drag it to your bookmarks bar, click it on any article, and paste straight into a new Obsidian note. No server, no setup — the API does the extraction, your browser does the rest.

For a fully automatic flow, point clip.py at a Raycast script, an Alfred workflow, or an iOS Shortcut that posts the shared URL to a tiny Flask endpoint running clip().

Why clean Markdown matters here

Obsidian is at its best when your notes are text you can actually read and grep. If you save raw HTML into your vault, you get garbage — inline styles, <div> soup, and tracking scripts polluting your search index. By clipping through the API, every note you save is already clean Markdown, which means:

Full-text search finds your clipped articles by their content, not by markup.
Backlinks and links work normally because the text is real Markdown.
Syncing (iCloud, Syncthing, git) is tiny and fast — no bloated HTML.

The API's json format is also handy here: it returns structured paragraphs, headings, images, and links separately, so you can build richer notes — for example, auto-generating a ## Links or ## Images section from the structured output.

Taking it further

Auto-tag — run the clipped text through a keyword check (or an LLM) to suggest tags before saving.
Clip a whole reading list — feed the API a list of URLs and clip them all overnight (50 free requests/day covers a serious reading habit).
Use format: "text" for a stripped, no-markup version when you just want the prose.
Track your reading — append each clip's word_count to a small CSV to see what you're actually consuming.

Wrapping up

Read-later apps are convenient, but they own your notes. Building your own clipper on the Web to Markdown/JSON API gives you the same one-click convenience with output that lives in your vault, in your format, forever.

Try it free — 50 requests/day on RapidAPI — and stop letting the read-later apps hold your articles hostage.

Monitor Any Website for Changes with Python (No Scraping Required)

bao001 xiao — Mon, 31 Aug 2026 02:10:38 +0000

We've all been there: waiting for a job posting to open, a product to come back in stock, a price to drop, or documentation to finally ship a fix. Manually refreshing the page every hour is a waste of time — and a sure way to miss the change when it actually lands.

The right tool is a change monitor: a small script that fetches a page on a schedule, compares it to the last version, and pings you when something is different. The annoying part has always been the "fetch and clean" step — most pages are 70% boilerplate (nav bars, ads, cookie banners), and you only care about the actual content.

That's exactly what the Web to Markdown/JSON API solves. It turns any URL into clean Markdown or JSON in one HTTP call, so your monitor compares content instead of noisy HTML.

In this article you'll build a working website change detector in Python. No BeautifulSoup, no selectors, no headless browsers.

The API at a glance

Endpoint:

POST https://web2md-api-production-d822.up.railway.app/extract

Request:

{
  "url": "https://example.com/pricing",
  "format": "markdown",
  "max_length": 50000
}

format — markdown, json, or text
max_length — cap the response size (up to 50000 characters)
Free tier: 50 requests/day — plenty for a personal monitor
Sign up / grab a key on RapidAPI

Response:

{
  "success": true,
  "title": "Pricing — Example Co",
  "content": "# Pricing\n\n## Plans\n\n...",
  "description": "Plan and pricing details",
  "word_count": 421,
  "response_time_ms": 210
}

The content field is clean Markdown. That's the key insight of this whole approach: we don't hash raw HTML (which changes whenever a tracking pixel or timestamp changes), we hash the readable content, so we only get alerted about changes that actually matter.

The plan

POST a URL to /extract and get Markdown back.
Hash the content with SHA-256.
Compare the hash to the last time we checked.
If it changed, save the new snapshot and notify you.

The code

Save this as watch.py:

import hashlib
import json
import sys
from pathlib import Path

import requests

API_URL = "https://web2md-api-production-d822.up.railway.app/extract"
STATE_DIR = Path("watch_state")


def fetch_content(url: str) -> dict:
    """Turn any web page into clean Markdown via the API."""
    r = requests.post(
        API_URL,
        json={"url": url, "format": "markdown", "max_length": 50000},
        timeout=30,
    )
    r.raise_for_status()
    return r.json()


def hash_content(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()


def watch(url: str) -> bool:
    """Check one URL for changes. Returns True if something changed."""
    data = fetch_content(url)
    content = data["content"]
    digest = hash_content(content)

    STATE_DIR.mkdir(exist_ok=True)
    state_file = STATE_DIR / (hash_content(url)[:16] + ".json")

    if state_file.exists():
        previous = json.loads(state_file.read_text())
        if previous["digest"] == digest:
            print(f"  no change - {url}")
            return False
        print(f"  CHANGED  - {url}")
        print(f"    before: {previous['word_count']} words")
        print(f"    after:  {data['word_count']} words")
    else:
        print(f"  first check - {url} ({data['word_count']} words)")

    # Save the new snapshot (full content, so you can diff it later)
    state_file.write_text(json.dumps({
        "digest": digest,
        "word_count": data["word_count"],
        "title": data["title"],
        "content": content,
    }, indent=2))
    return True


if __name__ == "__main__":
    urls = sys.argv[1:] or [
        "https://example.com",
        "https://news.ycombinator.com",
    ]
    changed = [u for u in urls if watch(u)]
    print(f"\n{len(changed)} page(s) changed.")

How it works

fetch_content — one POST to the API. The service fetches the page, strips nav/ads/scripts, and returns clean Markdown.
hash_content — SHA-256 turns the content into a fixed-length fingerprint. Identical content → identical hash.
watch — stores the last fingerprint (plus the full content so you can diff it) in a local JSON file. On each run it compares hashes and reports no change or CHANGED.

The whole trick is hashing clean Markdown instead of raw HTML. Raw HTML changes constantly — a new <script> tag here, a shuffled inline style there — which triggers false alarms all day long. Clean content only changes when something meaningful actually changed.

Run it

pip install requests
python watch.py https://example.com/pricing

The first run records a baseline; every run after that reports changes:

  first check - https://example.com/pricing (421 words)

  no change - https://example.com/pricing

  CHANGED  - https://example.com/pricing
    before: 421 words
    after:  438 words

Automating it

A change monitor is only useful if it runs on its own. Add a cron job to check every 30 minutes:

# every 30 minutes
*/30 * * * * cd /path/to/project && python watch.py https://example.com/pricing >> watch.log 2>&1

And if you want an actual notification instead of a log file, wire watch() up to any of these:

Email — smtplib in the stdlib, ~10 lines
Telegram — send a message via the Bot API in one requests.post
Slack / Discord — an incoming webhook is a single line
ntfy — a no-install push service that's designed exactly for this

For a zero-setup option, ntfy is hard to beat — pipe the change straight to your phone:

def notify(title: str, message: str):
    requests.post(
        "https://ntfy.sh/your_topic",
        data=message.encode("utf-8"),
        headers={"Title": title},
    )

Taking it further

Watch a list — keep a urls.txt and loop over it. 50 free requests/day covers a lot of pages.
Show the diff — use Python's difflib to print what changed, not just that it changed.
Use format: "json" when you want structured paragraphs and headings instead of prose — handy for comparing only specific sections of a page.
Track prices — many sites render prices inside Markdown tables, which turns "did the price change?" into a simple string comparison.

Wrapping up

Monitoring a website for changes is a classic "80% boring plumbing, 20% interesting problem" task. The Web to Markdown/JSON API deletes the boring 80% — fetching and cleaning arbitrary web pages — so you're left with the fun part: deciding what to watch and where to send the alerts.

Grab a free key (50 requests/day) on RapidAPI and stop refreshing pages manually.

Chat With Any Website: Build a RAG Bot in ~60 Lines of Python

bao001 xiao — Mon, 24 Aug 2026 04:10:52 +0000

Large language models are brilliant — right up until you ask them about something that happened after their training cutoff, or about content that lives on a specific website. Then they start hallucinating with total confidence.

The standard fix is Retrieval-Augmented Generation (RAG): fetch the real content, find the relevant pieces, and hand them to the model as context. The annoying part has always been step one — turning a messy web page into clean text you can actually feed to a model.

That's exactly what the Web to Markdown/JSON API solves. One HTTP call takes any URL and returns clean, structured content — Markdown or JSON — with the nav bars, ads, cookie banners, and other boilerplate stripped away.

In this tutorial you'll build a working RAG bot that answers questions about any web page, in roughly 60 lines of Python. No scraping libraries, no HTML parsing, no headless browsers.

The API at a glance

Endpoint:

POST https://web2md-api-production-d822.up.railway.app/extract

Request:

{
  "url": "https://example.com/article",
  "format": "markdown",
  "max_length": 50000
}

format — markdown, json, or text
Free tier: 50 requests/day
Sign up / grab a key on RapidAPI

Response (markdown):

{
  "success": true,
  "url": "https://example.com/article",
  "title": "Article Title",
  "content": "# Markdown content...",
  "word_count": 1234,
  "language": "en"
}

The content field is clean Markdown, ready to chunk and embed. That's the whole point — this API does the dirty work so your RAG pipeline starts with clean text instead of raw HTML.

The plan

POST a URL to /extract and get Markdown back.
Split the Markdown into chunks.
Embed every chunk.
Embed the user's question.
Find the chunks most similar to the question.
Feed those chunks + the question to an LLM.

Prerequisites

Python 3.9+
requests
Ollama running locally (free), or any OpenAI-compatible endpoint

The code

Save this as rag_bot.py:

import requests

API_URL = "https://web2md-api-production-d822.up.railway.app/extract"
OLLAMA = "http://localhost:11434"  # Ollama's REST API


def fetch_markdown(url: str) -> dict:
    """Turn any web page into clean Markdown via the API."""
    r = requests.post(
        API_URL,
        json={"url": url, "format": "markdown", "max_length": 50000},
    )
    r.raise_for_status()
    return r.json()


def chunk_markdown(md: str, max_chars: int = 1200) -> list[str]:
    """Split Markdown into chunks, respecting paragraph boundaries."""
    paragraphs = [p.strip() for p in md.split("\n\n") if p.strip()]
    chunks, current = [], ""
    for p in paragraphs:
        if len(current) + len(p) > max_chars and current:
            chunks.append(current)
            current = p
        else:
            current = f"{current}\n\n{p}" if current else p
    if current:
        chunks.append(current)
    return chunks


def embed(text: str) -> list[float]:
    """Get a vector embedding from Ollama."""
    r = requests.post(
        f"{OLLAMA}/api/embeddings",
        json={"model": "nomic-embed-text", "prompt": text},
    )
    r.raise_for_status()
    return r.json()["embedding"]


def cosine(a: list[float], b: list[float]) -> float:
    dot = sum(x * y for x, y in zip(a, b))
    na = sum(x * x for x in a) ** 0.5
    nb = sum(y * y for y in b) ** 0.5
    return dot / (na * nb) if na and nb else 0.0


def answer(chunks: list[str], question: str) -> str:
    """Retrieve the top-k chunks, then ask the LLM with them as context."""
    q_emb = embed(question)
    ranked = sorted(chunks, key=lambda c: cosine(embed(c), q_emb), reverse=True)
    context = "\n\n".join(ranked[:3])

    prompt = (
        "Answer the question using ONLY the context below. "
        "If the context doesn't contain the answer, say so.\n\n"
        f"Context:\n{context}\n\n"
        f"Question: {question}\nAnswer:"
    )
    r = requests.post(
        f"{OLLAMA}/api/generate",
        json={"model": "llama3", "prompt": prompt, "stream": False},
    )
    r.raise_for_status()
    return r.json()["response"]


if __name__ == "__main__":
    data = fetch_markdown("https://en.wikipedia.org/wiki/Retrieval-augmented_generation")
    print(f"Extracted: {data['title']} ({data['word_count']} words)")

    chunks = chunk_markdown(data["content"])
    question = "How does retrieval-augmented generation reduce hallucination?"
    print("Answer:", answer(chunks, question))

How it works

fetch_markdown — one POST to the API. The service fetches the page, strips the boilerplate, and returns clean Markdown. No BeautifulSoup, no trafilatura, no headless Chrome.
chunk_markdown — RAG models work best on focused snippets, so we split the article into ~1,200-character chunks along paragraph boundaries.
embed / cosine — turns text into vectors and measures similarity. Here we use Ollama's free nomic-embed-text model, but any embedding provider works.
answer — embeds the question, ranks chunks by similarity, then asks the LLM to answer only from the top three. That "only use the context" instruction is what keeps the model grounded instead of hallucinating.

Run it

pip install requests
ollama pull nomic-embed-text
ollama pull llama3
python rag_bot.py

Expected output:

Extracted: Retrieval-augmented generation (1500 words)
Answer: RAG reduces hallucination by grounding the model's responses in retrieved
documents. Instead of relying only on its training data, the model is given relevant
passages as context and instructed to answer from them, so its output is constrained
by verifiable source material rather than learned but possibly wrong associations.

Why clean Markdown matters for RAG

If you feed raw HTML into an embedding model, your vectors get polluted by <div> tags, nav menus, and footer links. That means your similarity search surfaces "Copyright © 2026" instead of the actual answer. Starting from clean Markdown — headings, paragraphs, lists, nothing else — dramatically improves retrieval quality for free.

This is also why the API's json format is handy: it returns structured paragraphs, headings, images, and links separately, so you can chunk by heading hierarchy or skip image-heavy sections entirely.

Taking it further

Point it at docs, changelogs, or a competitor's blog and ask questions about them.
Swap Ollama for OpenAI, Anthropic, or Gemini — the answer function only needs two small changes.
Store embeddings in a vector DB (Chroma, Qdrant) and index many pages at once to build a searchable knowledge base.
Use "format": "json" when you need the raw structure rather than prose.

Wrapping up

RAG is only as good as the text you feed it. The Web to Markdown/JSON API removes the most tedious 80% of the pipeline — fetching and cleaning arbitrary web pages — so you can spend your time on the interesting part: retrieval and generation.

Try it free at 50 requests/day on RapidAPI, and drop a comment if you build something cool with it.

Web Scraping in 2026: 5 Approaches Compared (With Code)

bao001 xiao — Sun, 23 Aug 2026 13:37:19 +0000

Web scraping is one of those tasks that sounds simple until you actually try it. You hit a site, grab the text, and move on — right?

Then reality hits: dynamic JavaScript rendering, bot detection, messy HTML, paywalls, cookie banners, and pages that change their layout every other week. Suddenly your "simple" scraper is a full-time maintenance job.

In this article, I compare five common ways to extract content from web pages in 2026, including a dedicated Web to Markdown/JSON API that removes most of the pain. I'll show real code for each so you can pick the right tool for your project.

The Five Approaches

Approach	Setup Effort	Handles JS?	Maintenance	Best For
1. `requests` + BeautifulSoup	Low	❌ No	High	Static pages you fully control
2. Playwright / Selenium	High	✅ Yes	High	JS-heavy apps, scraping behind logins
3. Trafilatura / newspaper3k	Low	❌ No	Medium	Article extraction from raw HTML
4. A dedicated extraction API	None	✅ Yes	None	Anything, at scale, reliably
5. Full headless browser farms	Very high	✅ Yes	Very high	Massive-scale, adversarial targets

Let's dig into each.

1. `requests` + BeautifulSoup: The Classic

For a simple static page, the old faithful still works:

import requests
from bs4 import BeautifulSoup

resp = requests.get("https://example.com")
soup = BeautifulSoup(resp.text, "html.parser")
print(soup.get_text())

The good: No dependencies beyond two libraries, zero cost, full control.

The bad: You are on your own for everything. JavaScript? Not rendered. Layout changed? Your selectors break. Cookie banners, Cloudflare, or bot checks? You are suddenly reverse-engineering HTML for a living.

For a quick one-off on a page you trust, it's fine. For anything at scale, it becomes a part-time job.

2. Playwright / Selenium: The Heavy Lifter

When a page renders everything with JavaScript, you need a real browser:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://example.com")
    print(page.inner_text("body"))
    browser.close()

The good: Renders anything a real user sees, can log in, click, and scroll.

The bad: Heavy. Each browser instance eats hundreds of MB of RAM. You have to manage selectors, waits, and timeouts. And the maintenance burden is real — every site redesign means updating your scripts.

Playwright is unbeatable for interacting with a page (clicking buttons, filling forms). But if you only need the content, it's overkill.

3. Trafilatura: The Article Specialist

If you already have the raw HTML and just need the readable text, Trafilatura is a strong pick:

import trafilatura

downloaded = trafilatura.fetch_url("https://example.com")
text = trafilatura.extract(downloaded)
print(text)

The good: Clever heuristics that strip navigation, ads, and boilerplate, leaving the main article body.

The bad: It still can't render JavaScript, and you still fetch the HTML yourself. You also get text — not clean Markdown or structured JSON.

4. A Dedicated Extraction API: The No-Maintenance Option

This is where the Web to Markdown/JSON API comes in. Instead of maintaining a scraper, you make one HTTP call:

curl -X POST https://web2md-api-production-d822.up.railway.app/extract \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "format": "markdown"}'

You get back clean, structured content:

{
  "success": true,
  "title": "Example Domain",
  "content": "# Example Domain\n\nThis domain is for use...",
  "description": "",
  "author": "",
  "published_date": "",
  "word_count": 24,
  "response_time_ms": 177
}

Three formats are supported via the format field:

markdown — clean, human-readable Markdown
json — a structured JSON representation of the page
text — plain text with all markup stripped

You can also cap the response size with max_length (up to 50000 characters) to keep payloads snappy.

The good: No scraping infrastructure to maintain. Handles rendering, extraction, and formatting for you. Works from any language with an HTTP client.

The bad: It's a network dependency (though that's true of any API), and the free tier is capped at 50 requests per day.

For most personal projects and MVPs, 50 requests a day is plenty. If you outgrow it, paid plans are available on RapidAPI.

Here's the same call from Python:

import requests

resp = requests.post(
    "https://web2md-api-production-d822.up.railway.app/extract",
    json={"url": "https://example.com", "format": "markdown", "max_length": 50000},
)
data = resp.json()
print(data["title"])
print(data["content"])

5. Headless Browser Farms: The Nuclear Option

For massive-scale, adversarial targets, some teams run their own fleet of headless browsers behind proxies and CAPTCHA solvers.

The good: Maximum control, works on almost anything.

The bad: You are now running infrastructure. Proxy rotation, retry logic, monitoring, IP reputation management — this is a full engineering project, not a side task.

Unless scraping is your product, this is almost never the right call.

Which Should You Choose?

Here's my honest recommendation:

Just need the content of a page? → Use the Web to Markdown/JSON API. One call, no maintenance.
Need to interact with a page (click, log in, fill forms)? → Playwright.
Already have HTML and want the article body? → Trafilatura.
Scraping a static page you fully control? → requests + BeautifulSoup.
Scraping at enormous scale? → A browser farm, but budget accordingly.

For the vast majority of developers building reading lists, AI pipelines, or content tools, the API approach wins on time-to-value. You go from "I should build a scraper" to "I have the content" in about five minutes.

Wrapping Up

Web scraping doesn't have to be a maintenance nightmare. The right tool depends on what you actually need — content vs. interaction, scale, and how much infrastructure you want to own.

If you just want clean Markdown or JSON from any URL without the headache, give the Web to Markdown/JSON API a try — the free tier is enough to test the waters.

Have a favorite scraping approach I missed? Drop it in the comments.

Build Your Own "Read Later" App in 50 Lines of Python with the Web to Markdown/JSON API

bao001 xiao — Mon, 10 Aug 2026 07:09:08 +0000

Have you ever found yourself drowning in browser tabs of articles you swear you will read "later"? I have been there. So I built a dead-simple reading list app that strips web pages down to clean, distraction-free Markdown — and it only took 50 lines of Python.

All thanks to the Web to Markdown/JSON API, a free-tier friendly service that converts any URL into clean, structured content.

What We Are Building

A CLI reading list tool that:

Accepts a URL from the command line
Fetches and converts the page to Markdown via the API
Saves it locally with metadata (title, author, date saved)
Lets you list and search your saved articles

Here is what it looks like:

# Save an article
python readlater.py add https://blog.rust-lang.org/2024/02/21/Rust-1.76.0.html

# List your reading list
python readlater.py list

# Search by keyword
python readlater.py search "async"

The Magic Behind It: The Web to Markdown/JSON API

Before we dive into code, let me show you the API that makes this possible. It is dead simple:

curl -X POST https://web2md-api-production-d822.up.railway.app/extract \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "format": "markdown"}'

You get back clean, structured Markdown:

{
  "success": true,
  "title": "Example Domain",
  "content": "# Example Domain\n\nThis domain is for use...",
  "description": "",
  "author": "",
  "published_date": "",
  "word_count": 24,
  "response_time_ms": 177
}

Three formats are supported: markdown, json (structured JSON representation), and text (plain text). You can also cap the response with max_length to keep things snappy.

The best part? The free tier gives you 50 requests per day — more than enough for personal use. If you need more, paid plans are available on RapidAPI.

Building the Reading List App

Let us write the code. Create a file called readlater.py:

#!/usr/bin/env python3
"""A simple read-later app powered by the Web to Markdown/JSON API."""

import json
import os
import sys
import sqlite3
from datetime import datetime
from pathlib import Path

import requests

# --- Configuration ---
API_URL = "https://web2md-api-production-d822.up.railway.app/extract"
DB_PATH = Path.home() / ".readlater" / "library.db"
ARTICLES_DIR = Path.home() / ".readlater" / "articles"


# --- Database Setup ---
def init_db():
    DB_PATH.parent.mkdir(parents=True, exist_ok=True)
    ARTICLES_DIR.mkdir(parents=True, exist_ok=True)
    conn = sqlite3.connect(DB_PATH)
    conn.execute("""
        CREATE TABLE IF NOT EXISTS articles (
            id INTEGER PRIMARY KEY AUTOINCREMENT,
            url TEXT UNIQUE NOT NULL,
            title TEXT,
            author TEXT,
            description TEXT,
            published_date TEXT,
            word_count INTEGER,
            saved_at TEXT NOT NULL,
            filename TEXT NOT NULL
        )
    """)
    conn.commit()
    return conn


# --- Fetch Article via API ---
def fetch_article(url: str) -> dict:
    """Convert a URL to Markdown using the Web to Markdown/JSON API."""
    payload = {"url": url, "format": "markdown", "max_length": 50000}
    response = requests.post(API_URL, json=payload, timeout=30)
    response.raise_for_status()
    data = response.json()
    if not data.get("success"):
        raise RuntimeError(f"API error: {data.get('error', 'Unknown')}")
    return data


# --- Save Article ---
def save_article(url: str) -> None:
    """Fetch and save an article to the local library."""
    print(f"Fetching: {url}")
    data = fetch_article(url)

    conn = init_db()
    saved_at = datetime.now().isoformat()
    safe_title = "".join(c for c in data["title"] if c.isalnum() or c in " _-").rstrip()
    filename = f"{saved_at[:10]}_{safe_title[:50]}.md"
    filepath = ARTICLES_DIR / filename

    # Write Markdown with metadata header
    with open(filepath, "w", encoding="utf-8") as f:
        f.write(f"---\n")
        f.write(f"title: {data['title']}\n")
        f.write(f"url: {url}\n")
        f.write(f"author: {data.get('author', 'N/A')}\n")
        f.write(f"saved_at: {saved_at}\n")
        f.write(f"word_count: {data.get('word_count', 0)}\n")
        f.write(f"---\n\n")
        f.write(data["content"])

    conn.execute(
        "INSERT OR REPLACE INTO articles(url, title, author, description, published_date, word_count, saved_at, filename) VALUES(?, ?, ?, ?, ?, ?, ?, ?)",
        (url, data["title"], data.get("author"), data.get("description"),
         data.get("published_date"), data.get("word_count"), saved_at, filename),
    )
    conn.commit()
    conn.close()

    print(f"Saved: {data['title']}")
    print(f"   Words: {data.get('word_count', 'N/A')} | File: {filename}")


# --- List and Search ---
def list_articles(search_term: str = None) -> None:
    """Display saved articles, optionally filtered by search term."""
    conn = init_db()
    if search_term:
        rows = conn.execute(
            "SELECT title, author, word_count, saved_at, url FROM articles "
            "WHERE title LIKE ? OR description LIKE ? ORDER BY saved_at DESC",
            (f"%{search_term}%", f"%{search_term}%"),
        ).fetchall()
        print(f'Results for "{search_term}":\n')
    else:
        rows = conn.execute(
            "SELECT title, author, word_count, saved_at, url FROM articles ORDER BY saved_at DESC"
        ).fetchall()
        print(f"Your Reading List ({len(rows)} articles):\n")

    for i, (title, author, wc, saved, url) in enumerate(rows, 1):
        print(f"{i}. {title}")
        print(f"   {wc} words | {saved[:10]} | {url[:60]}...")
        print()
    conn.close()


# --- CLI ---
if __name__ == "__main__":
    if len(sys.argv) < 2:
        print("Usage: python readlater.py [add <url> | list | search <term>]")
        sys.exit(1)

    command = sys.argv[1]
    if command == "add" and len(sys.argv) == 3:
        save_article(sys.argv[2])
    elif command == "list":
        list_articles()
    elif command == "search" and len(sys.argv) == 3:
        list_articles(sys.argv[2])
    else:
        print("Usage: python readlater.py [add <url> | list | search <term>]")

How It Works

Let us walk through the key parts:

1. The API Call

The fetch_article() function sends a POST request to the API endpoint. We ask for markdown format and cap at 50,000 characters — plenty for long-form articles. The API returns rich metadata (title, author, word count) alongside the cleaned content.

2. Local Storage

Each article is saved as a .md file with YAML frontmatter, so it is readable by any Markdown viewer — even Obsidian or VS Code. Metadata is also stored in a SQLite database for fast searching.

3. Search

A simple SQL LIKE query lets you find articles by title or description. No fancy vector embeddings needed for a personal library.

Taking It Further

This is just the foundation. Here are a few ways to extend it:

Export to Kindle: Convert the Markdown to an EPUB and email it to your device (pandoc makes this trivial)
Weekly digest email: Cron job that picks 3 random unread articles and emails them to you
Browser extension: Right-click any page and send it straight to your reading list
AI summaries: Pipe the content through an LLM to get TL;DRs for each saved article
RSS integration: Combine with an RSS feed parser to auto-save articles from your favorite blogs

The API also returns json format if you prefer structured data over Markdown:

# Get structured JSON instead of Markdown
payload = {"url": "https://example.com", "format": "json"}
response = requests.post(API_URL, json=payload)
data = response.json()
# data["content"] is now a structured JSON tree, not a string

Why This API?

I have tried several web-to-markdown services and scrapers. Here is what sets this one apart:

Feature	This API	BeautifulSoup DIY	Readability clones
No code maintenance	Yes	No	No
Handles JS-rendered pages	Yes	No	No
Consistent output format	Yes	Depends on site	Varies
Free tier available	Yes (50/day)	N/A	Often none
Response time	~200ms	Self-hosted	Varies

It is fast, reliable, and I do not have to maintain a headless browser or update CSS selectors when websites redesign.

Get Started

Try the API: Hit the endpoint directly with curl or your favorite HTTP client
RapidAPI subscribers: Get higher rate limits at RapidAPI
Clone and extend: The full reading list code above is yours to use and modify

# Quick test — extract any URL!
curl -X POST https://web2md-api-production-d822.up.railway.app/extract \
  -H "Content-Type: application/json" \
  -d '{"url": "https://dev.to", "format": "markdown", "max_length": 5000}'

Stop letting articles pile up in your tabs. Build your reading list today!

Have questions or built something cool with the API? Drop a comment below!

How to Build an AI Training Data Pipeline with the Web to Markdown/JSON API

bao001 xiao — Tue, 04 Aug 2026 06:52:23 +0000

Every AI engineer knows the bottleneck: getting clean, structured training data from the web. Whether you're fine-tuning an LLM, building a RAG application, or curating a domain-specific dataset, extracting web content into usable formats is a constant headache. You deal with messy HTML, JavaScript-rendered pages, rate limits, and boilerplate that pollutes your training corpus.

Enter the Web to Markdown/JSON API — a lightweight, no-nonsense service that converts any webpage into clean Markdown or structured JSON in a single API call. In this tutorial, I'll show you how to build a complete AI training data pipeline using it.

Why Your LLM Data Pipeline Needs This

When curating training data for language models, three things matter most:

Clean text — No HTML tags, no nav bars, no cookie banners. Just the content.
Consistent format — Every document in your pipeline should have the same structure.
Reproducibility — You need to re-scrape sources when content updates.

The Web to Markdown/JSON API gives you all three. It strips away the cruft, normalizes the output, and the deterministic endpoint means the same URL always produces the same structure.

The API at a Glance

Detail	Value
Endpoint	`POST https://web2md-api-production-d822.up.railway.app/extract`
Free Tier	50 requests/day
RapidAPI	Web to Markdown/JSON API

Request body:

{
  "url": "https://en.wikipedia.org/wiki/Transformer_(machine_learning_model)",
  "format": "markdown",
  "max_length": 50000
}

Response (Markdown mode):

{
  "title": "Transformer (machine learning model)",
  "content": "# Transformer (machine learning model)\n\nThe **transformer** is a deep learning architecture...",
  "url": "https://en.wikipedia.org/wiki/Transformer_(machine_learning_model)",
  "format": "markdown",
  "length": 28456
}

Three format options: markdown (headers, lists, code blocks intact), json (structured with sections), and text (plain, stripped-down).

Step 1: Building the Ingestion Pipeline

Let's build a Python pipeline that fetches web pages, stores them as clean Markdown, and prepares them for fine-tuning. First, install dependencies:

pip install requests tqdm datasets

Now, the core ingestion script:

import requests
import json
import time
from pathlib import Path
from typing import Optional

API_URL = "https://web2md-api-production-d822.up.railway.app/extract"

def extract_webpage(url: str, fmt: str = "markdown", max_length: int = 50000) -> Optional[dict]:
    """Extract a single webpage into clean Markdown or JSON."""
    resp = requests.post(
        API_URL,
        json={"url": url, "format": fmt, "max_length": max_length},
        timeout=30
    )
    if resp.status_code == 200:
        return resp.json()
    print(f"Failed {url}: {resp.status_code}")
    return None

def build_training_corpus(
    urls: list[str],
    output_dir: str = "./training_data",
    format: str = "markdown",
    delay: float = 1.0
) -> list[dict]:
    """Ingest a list of URLs and save as clean documents."""
    output_path = Path(output_dir)
    output_path.mkdir(parents=True, exist_ok=True)
    documents = []

    for i, url in enumerate(urls):
        print(f"[{i+1}/{len(urls)}] Extracting: {url[:80]}...")
        result = extract_webpage(url, fmt=format)

        if result and result.get("content"):
            doc = {
                "source": url,
                "title": result.get("title", ""),
                "content": result["content"],
                "length": result.get("length", 0)
            }
            documents.append(doc)

            # Save individual file
            safe_name = url.replace("https://", "").replace("/", "_")[:120]
            ext = "json" if format == "json" else "md"
            file_path = output_path / f"{i:04d}_{safe_name}.{ext}"
            file_path.write_text(result["content"], encoding="utf-8")

        time.sleep(delay)  # Respect rate limits

    # Save manifest
    manifest = {
        "total_documents": len(documents),
        "total_chars": sum(d["length"] for d in documents),
        "format": format,
        "sources": [d["source"] for d in documents]
    }
    (output_path / "manifest.json").write_text(json.dumps(manifest, indent=2))
    print(f"\nDone! {len(documents)} documents saved to {output_dir}/")
    return documents

# Example usage
if __name__ == "__main__":
    urls = [
        "https://en.wikipedia.org/wiki/Attention_Is_All_You_Need",
        "https://arxiv.org/abs/1706.03762",
        "https://huggingface.co/blog/llama2",
        "https://lilianweng.github.io/posts/2023-01-27-the-transformer-family-v2/",
    ]
    build_training_corpus(urls)

Run it and watch your training_data/ directory fill with clean, ready-to-use Markdown files — no HTML soup in sight.

Step 2: Preparing for LLM Fine-Tuning

Now that we have clean documents, let's format them for fine-tuning with Hugging Face's datasets library:

from datasets import Dataset
import json
from pathlib import Path

def prepare_finetuning_dataset(
    input_dir: str = "./training_data",
    instruction_template: str = None
) -> Dataset:
    """Convert ingested Markdown files into a HuggingFace Dataset for fine-tuning."""
    if instruction_template is None:
        instruction_template = (
            "You are a knowledgeable AI assistant. Read the following document "
            "and answer questions about it accurately.\n\nDocument:\n{content}"
        )

    records = []
    for md_file in sorted(Path(input_dir).glob("*.md")):
        content = md_file.read_text(encoding="utf-8")
        records.append({
            "source": md_file.name,
            "content": content,
            "instruction": instruction_template.format(content=content),
            "length": len(content)
        })

    # Filter out very short or empty documents
    records = [r for r in records if r["length"] > 500]

    dataset = Dataset.from_list(records)
    return dataset

# Load and inspect
dataset = prepare_finetuning_dataset("./training_data")
print(f"Dataset size: {len(dataset)} documents")
print(f"Total tokens (approx): {sum(dataset['length']) // 4:,}")

# Push to HuggingFace Hub (uncomment when ready)
# dataset.push_to_hub("your-username/your-domain-corpus")

Step 3: Building a RAG Ingestion Pipeline

For Retrieval-Augmented Generation, the JSON format shines. Here's how to build a chunking pipeline:

import requests
import json
from langchain.text_splitter import RecursiveCharacterTextSplitter

def ingest_for_rag(urls: list[str], chunk_size: int = 1000) -> list[dict]:
    """Fetch webpages as JSON and split into RAG-ready chunks."""
    text_splitter = RecursiveCharacterTextSplitter(
        chunk_size=chunk_size,
        chunk_overlap=200,
        separators=["\n\n", "\n", ".", " ", ""]
    )
    all_chunks = []

    for url in urls:
        resp = requests.post(
            "https://web2md-api-production-d822.up.railway.app/extract",
            json={"url": url, "format": "json", "max_length": 50000}
        )
        if resp.status_code != 200:
            continue

        data = resp.json()
        content = data["content"]

        # If JSON-format, content may be a dict with sections
        text = content if isinstance(content, str) else json.dumps(content)

        chunks = text_splitter.split_text(text)
        for chunk in chunks:
            all_chunks.append({
                "source": url,
                "title": data.get("title", ""),
                "chunk": chunk,
                "chunk_length": len(chunk)
            })

    return all_chunks

Each chunk is now ready to be embedded and stored in your vector database (Pinecone, Weaviate, ChromaDB, etc.).

Why Not Scrape It Yourself?

You could write your own scraper with BeautifulSoup and Trafilatura. But then you'd need to:

Handle JavaScript-rendered pages (headless browser overhead)
Deal with different site structures (no two websites use the same HTML)
Manage proxies and IP rotation for scale
Maintain extraction logic as sites change their markup

The API abstracts all of that. One endpoint. Predictable output. 50 free requests per day to test and prototype — more than enough for most experimentation.

Real-World Use Cases

Here's how developers are already using this API:

Use Case	Format	Why It Works
Domain-specific LLM fine-tuning	Markdown	Clean docs without boilerplate improve training quality
RAG knowledge bases	JSON	Structured sections make chunking smarter
Competitive intelligence	Text	Monitor competitor pages with diffable plain text
Academic research corpora	Markdown	ArXiv papers → readable training data
Website change monitoring	Text	Compare extracted text to detect content updates

Production Tips

Batch responsibly: The free tier gives you 50 requests/day. Space them out with time.sleep(1).
Validate URLs: Pre-check that URLs return 200 before sending them to the API.
Deduplicate: Hash your extracted content and skip near-duplicates (common across doc pages).
Store raw responses: Always save the full API response — you might want to re-chunk later.

def deduplicate_documents(documents: list[dict]) -> list[dict]:
    """Remove near-duplicate documents using content hashing."""
    import hashlib
    seen = set()
    unique = []
    for doc in documents:
        h = hashlib.md5(doc["content"][:1000].encode()).hexdigest()
        if h not in seen:
            seen.add(h)
            unique.append(doc)
    return unique

Getting Started

Head to the RapidAPI page and subscribe (free tier available).
Copy the endpoint: https://web2md-api-production-d822.up.railway.app/extract
Run the pipeline script above against your target URLs.
Build your dataset, embed your chunks, or feed it into your fine-tuning job.

The API handles the messy part of web data extraction so you can focus on what actually matters: training better models and shipping faster.

Got a cool use case? Drop a comment below — I'd love to see what datasets you're building!

How to Build an AI Training Data Pipeline with the Web to Markdown/JSON API

bao001 xiao — Mon, 27 Jul 2026 12:45:44 +0000

Introduction

Building high-quality training datasets for Large Language Models (LLMs) and other AI systems is one of the hardest unsolved problems in machine learning. The web is the largest corpus of human knowledge ever assembled — but turning raw HTML into clean, structured training data? That's where most pipelines break down.

Enter the Web to Markdown/JSON API: a zero-configuration service that converts any webpage into pristine Markdown or structured JSON in a single HTTP request. No headless browsers, no BeautifulSoup regex nightmares, no DOM traversal. Just clean content.

In this tutorial, you'll build a complete AI training data pipeline that:

Crawls a list of seed URLs and extracts clean text at scale
Deduplicates and normalizes content across sources
Outputs data in formats ready for fine-tuning (JSONL for OpenAI, Parquet for Hugging Face)
Respects rate limits and robots.txt

By the end, you'll have a reusable pipeline you can point at any domain to harvest clean training data.

The API at a Glance

Endpoint: POST https://web2md-api-production-d822.up.railway.app/extract

Request body:

{
  "url": "https://example.com",
  "format": "markdown",
  "max_length": 50000
}

Formats supported:

markdown — Clean, readable Markdown with headers, lists, and code blocks preserved
json — Structured JSON with title, content, and metadata separated
text — Plain text with all markup stripped

Free tier: 50 requests/day

RapidAPI: web-to-markdown-json-api

The json format is particularly valuable for training pipelines — it separates metadata from content, making it easy to filter, tag, and structure your dataset.

Step 1: Setting Up the Pipeline Skeleton

Create a project structure:

mkdir ai-data-pipeline && cd ai-data-pipeline
mkdir -p data/{raw,deduped,output}
touch pipeline.py config.py
pip install requests tqdm xxhash

Here's our configuration (config.py):

# config.py
import os

API_ENDPOINT = "https://web2md-api-production-d822.up.railway.app/extract"
RAPIDAPI_KEY = os.getenv("RAPIDAPI_KEY", "")
RAPIDAPI_HOST = "web-to-markdown-json-api.p.rapidapi.com"

# Pipeline settings
MAX_LENGTH = 50000          # Max chars per extraction
MIN_CONTENT_LENGTH = 200    # Skip pages with less content than this
RETRIES = 3                 # Retry on failure
REQUEST_DELAY = 1.5         # Seconds between requests (be polite!)
OUTPUT_FORMAT = "jsonl"     # jsonl or parquet

# Seed URLs — replace with your target domain list
SEED_URLS = [
    "https://docs.python.org/3/tutorial/",
    "https://realpython.com/",
    "https://www.freecodecamp.org/news/",
]

Step 2: The Core Extractor

This is where the Web to Markdown/JSON API shines. Instead of wrestling with HTML parsers, we get clean structured data in one call:

# pipeline.py
import json
import time
import hashlib
from pathlib import Path
from datetime import datetime

import requests
from tqdm import tqdm

import config


def extract_url(url: str, format: str = "json") -> dict | None:
    """
    Extract clean content from a URL using the Web to Markdown/JSON API.
    Returns parsed JSON or None on failure.
    """
    for attempt in range(config.RETRIES):
        try:
            # Direct endpoint (free tier, no API key needed for 50/day)
            resp = requests.post(
                config.API_ENDPOINT,
                json={
                    "url": url,
                    "format": format,
                    "max_length": config.MAX_LENGTH
                },
                timeout=30,
                headers={"Content-Type": "application/json"}
            )
            resp.raise_for_status()
            data = resp.json()

            # Enrich with pipeline metadata
            data["_pipeline"] = {
                "source_url": url,
                "extracted_at": datetime.utcnow().isoformat(),
                "content_hash": hashlib.sha256(
                    data.get("content", "").encode()
                ).hexdigest(),
                "content_length": len(data.get("content", ""))
            }
            return data

        except requests.exceptions.RequestException as e:
            if attempt < config.RETRIES - 1:
                wait = 2 ** attempt  # Exponential backoff
                print(f"  Retry {attempt+1}/{config.RETRIES} for {url}")
                time.sleep(wait)
            else:
                print(f"  Failed to extract {url}: {e}")
                return None


def run_extraction(urls: list[str], output_dir: str = "data/raw") -> int:
    """Extract content from URLs and save individual JSON files."""
    output_path = Path(output_dir)
    output_path.mkdir(parents=True, exist_ok=True)

    successful = 0

    for url in tqdm(urls, desc="Extracting"):
        data = extract_url(url, format="json")

        if data and data["_pipeline"]["content_length"] >= config.MIN_CONTENT_LENGTH:
            fname = f"{data['_pipeline']['content_hash'][:16]}.json"
            with open(output_path / fname, "w") as f:
                json.dump(data, f, ensure_ascii=False, indent=2)
            successful += 1

        time.sleep(config.REQUEST_DELAY)

    print(f"Extracted {successful}/{len(urls)} pages successfully")
    return successful


if __name__ == "__main__":
    run_extraction(config.SEED_URLS)

Run it:

python pipeline.py

Output:

Extracting: 100%|████████████| 3/3 [00:12<00:00,  4.12s/it]
Extracted 3/3 pages successfully

Each extracted page lands in data/raw/ as a clean JSON file with structured content and pipeline metadata.

Step 3: Deduplication and Quality Filtering

Real-world crawls produce duplicates. Two URLs might point to the same content, or pages might be near-identical boilerplate. Here's a dedup module using MinHash:

# dedup.py
import json
import xxhash
from pathlib import Path


def compute_minhash(text: str, shingle_size: int = 5, num_hashes: int = 128) -> list[int]:
    """Compute MinHash signature for near-duplicate detection."""
    words = text.lower().split()
    if len(words) < shingle_size:
        return [0] * num_hashes

    shingles = [
        " ".join(words[i:i+shingle_size]) 
        for i in range(len(words) - shingle_size + 1)
    ]

    signatures = []
    for seed in range(num_hashes):
        min_hash = min(
            xxhash.xxh32(shingle, seed=seed).intdigest() 
            for shingle in shingles
        )
        signatures.append(min_hash)

    return signatures


def estimate_jaccard(sig1: list[int], sig2: list[int]) -> float:
    """Estimate Jaccard similarity from two MinHash signatures."""
    matches = sum(1 for a, b in zip(sig1, sig2) if a == b)
    return matches / len(sig1)


def deduplicate(raw_dir: str = "data/raw", threshold: float = 0.85) -> list[Path]:
    """Remove near-duplicate documents. Returns paths to keep."""
    raw = Path(raw_dir)
    docs = []

    for fpath in sorted(raw.glob("*.json")):
        with open(fpath) as f:
            doc = json.load(f)
        text = doc.get("content", "")
        sig = compute_minhash(text)
        docs.append((fpath, doc, sig))

    keep = []
    discarded = 0

    for i, (fpath_i, doc_i, sig_i) in enumerate(docs):
        is_dup = False
        for _, _, sig_j in keep:
            if estimate_jaccard(sig_i, sig_j) >= threshold:
                is_dup = True
                break

        if is_dup:
            discarded += 1
        else:
            keep.append((fpath_i, doc_i, sig_i))

    print(f"Dedup: kept {len(keep)}, removed {discarded} near-duplicates")
    return [item[0] for item in keep]


if __name__ == "__main__":
    kept = deduplicate()
    print(f"Final dataset size: {len(kept)} documents")

Step 4: Exporting to Training Formats

Different fine-tuning frameworks expect different formats. Here's how to export for the two most common targets:

OpenAI Fine-Tuning (JSONL)

# export.py
import json
from pathlib import Path


def export_openai_jsonl(
    input_dir: str = "data/raw",
    output_file: str = "data/output/training_data.jsonl",
    system_prompt: str = "You are a helpful assistant."
):
    """Convert extracted documents to OpenAI chat-format JSONL."""
    output_path = Path(output_file)
    output_path.parent.mkdir(parents=True, exist_ok=True)

    count = 0
    with open(output_path, "w") as out:
        for fpath in sorted(Path(input_dir).glob("*.json")):
            with open(fpath) as f:
                doc = json.load(f)

            title = doc.get("title", "Untitled")
            content = doc.get("content", "")

            if len(content) < 100:
                continue

            excerpt = content[:4000]

            record = {
                "messages": [
                    {"role": "system", "content": system_prompt},
                    {"role": "user", "content": f"Explain this about '{title}':\n\n{excerpt}"},
                    {"role": "assistant", "content": f"Summary of '{title}':\n\n{excerpt}"}
                ]
            }

            out.write(json.dumps(record) + "\n")
            count += 1

    print(f"Exported {count} records to {output_file}")
    return count

Hugging Face Datasets (Parquet)

def export_parquet(input_dir: str = "data/raw", output_file: str = "data/output/dataset.parquet"):
    """Export extracted data as a Parquet file for Hugging Face datasets."""
    try:
        import pandas as pd
    except ImportError:
        print("Install pandas and pyarrow: pip install pandas pyarrow")
        return

    records = []
    for fpath in sorted(Path(input_dir).glob("*.json")):
        with open(fpath) as f:
            doc = json.load(f)

        records.append({
            "url": doc["_pipeline"]["source_url"],
            "title": doc.get("title", ""),
            "content": doc.get("content", ""),
            "content_hash": doc["_pipeline"]["content_hash"],
            "content_length": doc["_pipeline"]["content_length"],
            "extracted_at": doc["_pipeline"]["extracted_at"],
        })

    df = pd.DataFrame(records)
    df.to_parquet(output_file, index=False)
    print(f"Exported {len(df)} records to {output_file}")

Step 5: Scaling Up with Sitemaps

For real datasets, you need more than a handful of seed URLs. Here's a sitemap-powered crawler:

# crawler.py
import xml.etree.ElementTree as ET

import requests
from pipeline import extract_url


def fetch_sitemap_urls(sitemap_url: str) -> list[str]:
    """Parse a sitemap.xml and return all URLs."""
    resp = requests.get(sitemap_url, timeout=30)
    resp.raise_for_status()

    root = ET.fromstring(resp.content)
    ns = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}

    # Handle sitemap index files
    sitemap_refs = root.findall("sm:sitemap/sm:loc", ns)
    if sitemap_refs:
        all_urls = []
        for ref in sitemap_refs:
            all_urls.extend(fetch_sitemap_urls(ref.text))
        return all_urls

    # Regular sitemap
    urls = [loc.text for loc in root.findall("sm:url/sm:loc", ns)]
    return urls


if __name__ == "__main__":
    urls = fetch_sitemap_urls("https://docs.python.org/3/sitemap.xml")
    print(f"Found {len(urls)} URLs in sitemap")

Step 6: Full Pipeline Orchestration

Tie everything together with a main orchestration script:

# run_pipeline.py
#!/usr/bin/env python3
"""
Full AI training data pipeline using Web to Markdown/JSON API.

Usage:
    python run_pipeline.py --urls urls.txt --format jsonl
    python run_pipeline.py --sitemap https://docs.python.org/3/sitemap.xml
"""
import argparse
from pathlib import Path

from pipeline import run_extraction
from dedup import deduplicate
from export import export_openai_jsonl, export_parquet
from crawler import fetch_sitemap_urls


def main():
    parser = argparse.ArgumentParser(description="AI Training Data Pipeline")
    group = parser.add_mutually_exclusive_group(required=True)
    group.add_argument("--urls", help="File with one URL per line")
    group.add_argument("--sitemap", help="Sitemap URL to crawl")
    parser.add_argument("--format", choices=["jsonl", "parquet"], default="jsonl")
    parser.add_argument("--max-pages", type=int, default=100)
    parser.add_argument("--output", default="data/output/training_data")
    args = parser.parse_args()

    # Step 1: Collect URLs
    if args.urls:
        with open(args.urls) as f:
            urls = [line.strip() for line in f if line.strip()]
    else:
        urls = fetch_sitemap_urls(args.sitemap)

    urls = urls[:args.max_pages]
    print(f"Collected {len(urls)} URLs")

    # Step 2: Extract content
    extracted = run_extraction(urls)
    if extracted == 0:
        print("No pages extracted. Exiting.")
        return

    # Step 3: Deduplicate
    kept = deduplicate()

    # Step 4: Export
    if args.format == "jsonl":
        export_openai_jsonl(output_file=f"{args.output}.jsonl")
    else:
        export_parquet(output_file=f"{args.output}.parquet")

    print(f"Pipeline complete! Output: {args.output}.{args.format}")


if __name__ == "__main__":
    main()

Run the full pipeline:

# From a URL list
python run_pipeline.py --urls seed_urls.txt --format jsonl --max-pages 50

# From a sitemap
python run_pipeline.py --sitemap https://docs.python.org/3/sitemap.xml --format parquet

Why This API Over Traditional Scraping?

Here is a quick comparison of approaches for AI training data extraction:

Approach	Setup Time	Maintainability	Content Quality	Cost
BeautifulSoup + requests	Hours per site	Brittle	Manual cleaning needed	Free
Playwright/Selenium	Hours	Moderate	Good but includes boilerplate	Free (compute)
Scrapy spiders	Hours per site	Better but needs XPath/CSS	Variable	Free
Web to Markdown/JSON API	Minutes	Excellent	Excellent — clean output	Free (50/day)

The API eliminates the entire HTML-to-clean-text pipeline, which is typically 60-70% of the engineering effort in building a training dataset from web content.

Best Practices

Respect robots.txt: Always check a site's robots.txt before crawling at scale.
Rate limit aggressively: The free tier gives you 50 requests/day. Space them out.
Filter aggressively: Not all web content is training-worthy. Set MIN_CONTENT_LENGTH high (500+ chars) and consider language detection.
Deduplicate ruthlessly: The web is full of mirrors and syndicated content. MinHash + content hashing catches most duplicates.
Version your datasets: Tag each export with a timestamp and source domain.
Use JSON format for pipelines: The json format separates metadata from content — essential for filtering, tagging, and structuring.

Going Further

Add language detection with fasttext or langdetect for multilingual datasets
Integrate with Hugging Face Datasets via datasets.Dataset.from_parquet()
Add quality scoring using textstat readability metrics or perplexity filtering
Schedule with cron/APScheduler for continuous dataset updates
Use RapidAPI integration for higher rate limits: web-to-markdown-json-api on RapidAPI

Conclusion

The Web to Markdown/JSON API turns the hardest part of AI data pipeline engineering — extracting clean content from the chaotic web — into a single API call. Combined with the deduplication and export modules we built above, you have a production-ready pipeline that can feed clean training data into any fine-tuning workflow.

API Endpoint: POST https://web2md-api-production-d822.up.railway.app/extract

Free Tier: 50 requests/day — more than enough to prototype and test your pipeline.

Try it out, build your dataset, and ship that fine-tuned model.

Questions or feedback? Drop a comment below!

How to Monitor Website Changes with the Web to Markdown/JSON API

bao001 xiao — Mon, 20 Jul 2026 12:11:23 +0000

Introduction

Keeping track of changes on websites—whether it's competitor pricing, job listings, government policy updates, or product release notes—is a common need for developers and businesses alike. But writing a full-blown web scraper from scratch for every site you want to monitor? That's brittle, time-consuming, and often breaks the moment the target site tweaks its layout.

Enter the Web to Markdown/JSON API: a simple, powerful API that converts any webpage into clean, structured Markdown or JSON in a single request. It handles all the messy HTML parsing, boilerplate stripping, and formatting so you can focus on what actually matters—detecting changes and acting on them.

In this tutorial, you'll build a website change monitor in Python that:

Fetches any web page as clean Markdown via the API
Compares snapshots over time to detect meaningful changes
Sends email alerts when something important shifts
Runs on a cron schedule for fully automated monitoring

Let's dive in.

The API at a Glance

Endpoint: POST https://web2md-api-production-d822.up.railway.app/extract

Request body:

{
  "url": "https://example.com",
  "format": "markdown",
  "max_length": 50000
}

Formats supported: markdown, json, text

Free tier: 50 requests per day

RapidAPI page: web-to-markdown-json-api

This API does one thing and does it well: give it a URL, and it returns the page's meaningful content stripped of navigation, ads, and scripting cruft. Exactly what you need for change detection—no BeautifulSoup or regex wrestling required.

Step 1: Fetching a Page Snapshot

Create a file called monitor.py. We'll start with a function that takes a URL and returns the clean Markdown content:

import requests

API_ENDPOINT = "https://web2md-api-production-d822.up.railway.app/extract"

def fetch_snapshot(url: str, format: str = "markdown", max_length: int = 50000) -> str:
    resp = requests.post(
        API_ENDPOINT,
        json={"url": url, "format": format, "max_length": max_length},
        timeout=30
    )
    resp.raise_for_status()
    data = resp.json()
    return data.get("markdown") or data.get("content", "")

# Quick test
if __name__ == "__main__":
    content = fetch_snapshot("https://example.com")
    print(content[:500])
    print(f"\n--- Total length: {len(content)} characters")

Run it and you'll see clean, readable Markdown instead of raw HTML—a huge win before we even start the monitoring logic.

Step 2: Storing and Comparing Snapshots

To detect changes, we need to store previous snapshots and compare them against new ones. We'll use a simple JSON file as our snapshot database:

import hashlib
import json
import os
from datetime import datetime

SNAPSHOT_FILE = "snapshots.json"

def load_snapshots() -> dict:
    if os.path.exists(SNAPSHOT_FILE):
        with open(SNAPSHOT_FILE) as f:
            return json.load(f)
    return {}

def save_snapshots(snapshots: dict):
    with open(SNAPSHOT_FILE, "w") as f:
        json.dump(snapshots, f, indent=2, default=str)

def compute_hash(content: str) -> str:
    return hashlib.sha256(content.encode("utf-8")).hexdigest()

def compare_snapshot(url: str, new_content: str):
    snapshots = load_snapshots()
    new_hash = compute_hash(new_content)

    if url not in snapshots:
        snapshots[url] = {
            "hash": new_hash,
            "timestamp": datetime.now().isoformat(),
            "length": len(new_content)
        }
        save_snapshots(snapshots)
        return None  # Baseline established

    old = snapshots[url]
    if old["hash"] == new_hash:
        return None  # No change

    length_delta = len(new_content) - old.get("length", 0)

    snapshots[url] = {
        "hash": new_hash,
        "timestamp": datetime.now().isoformat(),
        "length": len(new_content)
    }
    save_snapshots(snapshots)

    return {
        "url": url,
        "previous_checked": old["timestamp"],
        "length_delta": length_delta,
        "length_delta_pct": round(
            (length_delta / old["length"]) * 100, 1
        ) if old["length"] > 0 else 0
    }

This gives us hash-based equality checks (fast and reliable) plus a quick size-delta metric.

Step 3: Detecting Meaningful Changes

Not all changes matter. A page might change its timestamp, or a random ad insertion could shift content slightly. Let's add smart thresholds:

CHANGE_THRESHOLD = 0.02  # 2% content change before we alert

def detect_meaningful_change(url: str, new_content: str):
    diff = compare_snapshot(url, new_content)

    if diff is None:
        return None

    abs_pct = abs(diff["length_delta_pct"])
    if abs_pct < CHANGE_THRESHOLD * 100:
        print(f"Change too small ({abs_pct}%), skipping alert for {url}")
        return None

    return diff

Tune CHANGE_THRESHOLD per URL—higher for noisy pages, lower for pages where every word counts.

Step 4: Sending Email Alerts

When a meaningful change is detected, fire off an email using Python's built-in smtplib:

import smtplib
from email.mime.text import MIMEText

SMTP_HOST = "smtp.gmail.com"
SMTP_PORT = 587
SMTP_USER = "alerts@yourdomain.com"
SMTP_PASS = "your-app-password"
ALERT_EMAIL = "you@yourdomain.com"

def send_alert(change: dict):
    subject = f"Web Change Detected: {change['url']}"
    body = (
        f"A change was detected on a monitored page.\n\n"
        f"URL: {change['url']}\n"
        f"Last checked: {change['previous_checked']}\n"
        f"Size change: {change['length_delta']:+d} chars "
        f"({change['length_delta_pct']:+}%)\n\n"
        f"See: {change['url']}\n"
    )
    msg = MIMEText(body)
    msg["Subject"] = subject
    msg["From"] = SMTP_USER
    msg["To"] = ALERT_EMAIL

    with smtplib.SMTP(SMTP_HOST, SMTP_PORT) as server:
        server.starttls()
        server.login(SMTP_USER, SMTP_PASS)
        server.send_message(msg)

    print(f"Alert sent for {change['url']}")

Step 5: Putting It All Together

The main monitoring loop—feed it a list of URLs and it checks them all:

import time

MONITORED_URLS = [
    "https://example.com/pricing",
    "https://example.com/changelog",
    "https://example.com/jobs",
]

def run_monitor(urls: list[str]):
    print(
        f"Starting monitor for {len(urls)} URLs "
        f"at {datetime.now().isoformat()}"
    )

    for url in urls:
        try:
            print(f"Checking: {url}")
            content = fetch_snapshot(url)
            change = detect_meaningful_change(url, content)

            if change:
                print(
                    f"  -> CHANGE DETECTED! "
                    f"Size delta: {change['length_delta']:+d} chars"
                )
                # send_alert(change)  # Uncomment when SMTP configured
            else:
                print(f"  -> No meaningful change")

            time.sleep(1)  # Respect free tier limits

        except Exception as e:
            print(f"  -> ERROR for {url}: {e}")

    print("Monitor run complete.")

if __name__ == "__main__":
    run_monitor(MONITORED_URLS)

Step 6: Scheduling with Cron

Once working, schedule it to run daily:

# Every day at 8 AM
0 8 * * * cd /path/to/project && python3 monitor.py >> monitor.log 2>&1

# Every 6 hours for fast-moving pages
0 */6 * * * cd /path/to/project && python3 monitor.py >> monitor.log 2>&1

With the free tier (50 requests/day), monitor 40-45 URLs daily with a safety buffer.

Advanced Ideas

1. Diff Highlighting

Store full Markdown content and use difflib for human-readable diffs:

import difflib

def generate_diff(old: str, new: str) -> str:
    return "\n".join(difflib.unified_diff(
        old.splitlines(), new.splitlines(), lineterm=""
    ))

2. Per-URL Thresholds

Some pages change frequently, others rarely:

URL_CONFIG = {
    "https://example.com/news": {"threshold": 0.10},
    "https://example.com/terms": {"threshold": 0.01},
}

3. JSON Format for Structured Data

Switch format to "json" when monitoring structured fields:

resp = requests.post(
    API_ENDPOINT,
    json={
        "url": "https://example.com/product/123",
        "format": "json",
        "max_length": 20000
    }
)
# resp.json() returns structured data with title, content, metadata

4. LLM-Powered Summaries

Pipe the Markdown diff into an LLM and ask for a one-paragraph summary of what changed. Turns raw deltas into actionable intelligence.

Why This API Over DIY Scraping?

Aspect	Web to Markdown/JSON API	DIY Scraping
Setup time	1 API call	Hours of BeautifulSoup + custom parsing
Maintenance	Handled by the API	You fix it every site redesign
Content quality	Clean Markdown, boilerplate stripped	Raw HTML to parse yourself
Multiple formats	Markdown, JSON, plain text	You implement each
Rate limiting	Free tier included	You write retry logic

Wrapping Up

With under 100 lines of Python and the Web to Markdown/JSON API, you have a fully automated website change monitor that:

Fetches clean, readable content from any URL
Compares snapshots and detects meaningful changes
Alerts by email when something shifts
Runs on autopilot via cron

This pattern is production-ready for monitoring competitor pricing, government notices, job boards, documentation updates, and anything else on the web that matters.

Try It Now

Grab a key on RapidAPI or hit the endpoint directly:

curl -X POST https://web2md-api-production-d822.up.railway.app/extract \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "format": "markdown", "max_length": 50000}'

The first 50 requests each day are free. Go build something useful!

How to Extract Clean Content From Any URL in 3 Lines of Python

bao001 xiao — Fri, 17 Jul 2026 13:45:16 +0000

Have you ever needed to grab the actual content from a web page — without all the ads, nav bars, and JavaScript junk?

Whether you're building an AI training pipeline, a reading app, or a data scraper, extracting clean text from arbitrary URLs is a universal pain point.

Here's the fastest way I've found.

The Problem

Most websites are 70% boilerplate. Headers, footers, sidebars, cookie banners, popups. If you just use requests.get() and parse the HTML, you get garbage:

import requests
from bs4 import BeautifulSoup

resp = requests.get("https://example.com/article")
soup = BeautifulSoup(resp.text, "html.parser")
print(soup.get_text())
# Output: navigation menu sidebar footer ads copyright...

The Solution: One API Call

Instead of fighting with HTML parsing, use a dedicated content extraction API:

import requests

resp = requests.post("https://web2md-api-production-d822.up.railway.app/extract", json={
    "url": "https://www.python.org/doc/",
    "format": "markdown"
})
data = resp.json()
print(data["content"][:500])

That's it. Three lines. The API handles:

Fetching the page
Stripping all boilerplate (nav, ads, scripts)
Extracting the main article content
Converting to clean Markdown

What You Get Back

The API returns structured JSON:

{
  "success": true,
  "title": "Documentation — Python.org",
  "content": "# Markdown content here...",
  "description": "The official home of the Python Programming Language",
  "word_count": 1234,
  "response_time_ms": 850
}

Format Options

markdown (default) — perfect for LLM training data, note-taking, publishing
json — structured paragraphs, headings, and metadata
text — plain text when you just need the words

Pricing

It's cheap. Free tier handles 50 requests/day — enough for most side projects.

Free: 50 requests/day
Basic: 5,000/month — $4.99
Pro: 50,000/month — $19.99

Real Use Cases

AI Training Data — Feed clean web content into your LLM fine-tuning pipeline
Research — Download articles for offline reading and analysis
Content Monitoring — Track changes on competitor pages
Reading Apps — Convert any article to a clean reading view

Why Not Just Use BeautifulSoup?

You can, but:

Every site has different HTML structure
JavaScript-rendered content won't appear
You'll spend hours writing selectors for each site
Maintenance nightmare when sites redesign

A dedicated extraction API handles all of this with one endpoint. No selectors, no maintenance, no hassle.

This article was written with the help of AI. The API mentioned is a personal project — I built it because I needed it myself.

From Zero to $500/Month: My Side Hustle Journey in 2026

bao001 xiao — Sun, 07 Jun 2026 02:02:26 +0000

Last January, I was staring at my bank account with that familiar sinking feeling. Rent was due in five days, my day job covered the basics but left nothing for savings, and every "make money online" video I'd watched felt like a rehash of the same recycled advice. I decided to stop watching and start doing.

Twelve months and a lot of trial-and-error later, I'm not rich — but I'm consistently pulling in an extra $500–800 per month from side hustles I actually enjoy. No dropshipping gurus, no crypto rug pulls, no "DM me for the secret formula." Just real, repeatable work that anyone with a laptop can start today.

Here's exactly what worked, what didn't, and how you can skip the mistakes I made.

The Framework: Time vs. Leverage

Before diving into specific hustles, I had to confront a hard truth: trading time for money has a ceiling. You can only work so many hours. The side hustles that actually scaled for me shared one trait — they kept earning while I slept.

That meant building assets, not just doing gigs. A blog post you write once can bring traffic for years. A digital product you create once can sell indefinitely. Freelance writing? That's trading hours — useful for quick cash, but not the long game.

Hustle #1: Niche Blogging (The Slow Burn That Pays)

The first thing I did was spin up a WordPress site on a niche I actually cared about: budget home-office setups for remote workers. I grabbed a domain name for $12 and signed up for Hostinger's Business plan — their one-click WordPress installer meant I had a live site in under 15 minutes, and at less than $4/month, the barrier to entry was basically zero.

For three months, I wrote one article per week. No one read them. Month four, Google started noticing. By month six, I was getting 2,000 monthly visitors organically. I monetized with a mix of display ads (Ezoic) and a few carefully chosen affiliate links for standing desks and ergonomic chairs. That site now earns $200–300/month, almost entirely passive.

What I learned: Pick a niche narrow enough that you can actually rank, but broad enough that there's an audience. "Remote work ergonomics" beats "productivity tips" every time.

Hustle #2: Digital Templates (High Margin, Low Maintenance)

This one surprised me. I'm not a designer, but I noticed that small business owners in my network constantly struggled with the same things: client proposals, onboarding checklists, social media content calendars. So I built simple Notion and Google Sheets templates, packaged them nicely, and listed them on Gumroad.

Total upfront investment: one weekend of focused work. First month revenue: $87. Six months later: $250/month from the same templates, with zero additional effort beyond answering the occasional customer email.

What I learned: You don't need to create something revolutionary. Solve a specific, boring problem that people actively search for, and make the solution stupidly easy to use.

Hustle #3: Freelancing Platforms (Strategic, Not Desperate)

I'll be honest: I started on Upwork the wrong way, bidding on everything and racing to the bottom on price. That burned me out fast. The turning point came when I got selective — I only applied for jobs that paid at least $50/hour and required skills I was genuinely good at (technical writing and WordPress troubleshooting).

Having a professional portfolio site (built on Hostinger with their built-in website builder) was the single biggest factor in landing higher-paying clients. When a potential client Googles you and finds a clean, fast-loading site with case studies and testimonials, they're willing to pay 2–3x more than the platform average.

Freelancing now contributes $400–600/month on months I actively take projects, and I can dial it up or down depending on my bandwidth.

Hustle #4: Newsletter Curation (The One I Almost Skipped)

I almost didn't include this because "newsletters are saturated" is something I told myself for six months. Then I started one anyway — a weekly roundup of the best remote-work tools and tips I came across during the week. I spent more time reading and curating than writing, and the format was deliberately short (3-minute read).

I grew to 800 subscribers in four months by cross-posting summaries to Reddit and LinkedIn. ConvertKit's free tier handled everything. I haven't monetized it yet, but I've already had two brands reach out about sponsorship slots.

What Didn't Work (So You Can Skip It)

Print-on-demand: Too much time on designs for razor-thin margins. Maybe works for people with big social followings, but organic sales were nonexistent.
Survey sites: I tested Swagbucks and Survey Junkie for a month. Total earnings: $43. That's $1.43/hour. Not worth it.
Day trading: Treated it like a side hustle, lost $300 in two weeks. Lesson learned — this is a skill that takes years, not a quick income stream.

The Stack I Use Today

If you're starting from zero, here's the exact setup that got me to $500/month:

Tool	Purpose	Cost
Hostinger	Website hosting + domain	~$4/month
Gumroad	Digital product sales	Free (takes 10% cut)
ConvertKit	Email newsletter	Free up to 1,000 subs
Canva	Template design + graphics	Free
Notion	Content planning + templates	Free

Your First 30 Days

Don't try to do everything at once. Here's a realistic 30-day plan:

Week 1: Pick your niche. Buy a domain and set up hosting. Write your first blog post.
Week 2: Create one simple digital product (a checklist, template, or mini-guide). List it on Gumroad.
Week 3: Set up a freelance profile on one platform. Apply to exactly three jobs you're qualified for — no more.
Week 4: Evaluate. Which of these three channels showed the most promise? Double down on that one.

The people who succeed at side hustles aren't the ones who know the most. They're the ones who start before they feel ready, ship imperfect work, and keep showing up when no one's watching.

Disclosure: Some links in this article are affiliate links. If you sign up for Hostinger through my referral link, I may earn a commission at no extra cost to you. I only recommend tools I've personally used and genuinely believe in.

Web Development in 2026: The Only Beginner Roadmap You'll Need

bao001 xiao — Sat, 06 Jun 2026 02:03:29 +0000

Web Development in 2026: The Only Beginner Roadmap You'll Need

I watched a barista teach himself to code during his lunch breaks last year. Six months later, he landed his first freelance client building a restaurant website for $1,200. He didn't have a CS degree. He didn't attend a bootcamp. He just followed a roadmap and stayed consistent.

What struck me wasn't his talent — it was that the path into web development has never been more accessible. In 2026, the barrier is no longer "can I learn this?" It's "which direction should I go first?"

This guide cuts through the noise. No fluff, no 47-step plan, no "learn Rust before HTML" gatekeeping. Just the clearest route from zero to deployed.

Step 1: Decide What Kind of Developer You Want to Be

Before you touch a single line of code, make this decision:

Frontend Developer: You build what users see — layouts, buttons, animations. You live in HTML, CSS, and JavaScript.
Backend Developer: You build what users don't see — databases, server logic, APIs. You live in Python, Node.js, or Go.
Full-Stack Developer: You do both. Most beginners aim here eventually, but don't start with it.

My recommendation for 2026: start frontend. The feedback loop is instant — you change a color, you see a color. That immediate visual reward keeps you motivated through the hard parts.

Step 2: Grab a Domain and Hosting (Do This Today)

This is the step most beginners skip for months. Don't.

Sign up for a hosting plan, buy a domain name (something like yourname.dev), and install WordPress or a static site generator. Having a live website on day one — even if it's just your name and a photo — changes your psychology entirely. You stop being "someone learning to code" and start being "someone who builds websites."

I use Hostinger for this exact reason. Their plans start under $3/month and include a free domain for the first year, free SSL, and a one-click WordPress installer. Within 15 minutes, you have a real website on the internet. That momentum matters more than any tutorial.

Step 3: Learn the Core Stack (in This Exact Order)

3.1 HTML (1–2 weeks)

Structure. Headings, paragraphs, links, images, forms. That's it. Don't get fancy — learn how a web page is skeleton'd.

3.2 CSS (2–4 weeks)

Styling. Learn Flexbox and Grid first — those two layout systems solve 90% of real-world problems. Skip the 200-property memorization game. Just learn enough to make things look clean.

3.3 JavaScript (4–8 weeks)

This is where you'll spend most of your time. Focus on:

Variables, functions, arrays, objects
DOM manipulation (document.querySelector, event listeners)
Fetch API (making requests to servers)
Basic async/await

3.4 Git and GitHub (1 week)

Version control is non-negotiable in 2026. Learn git init, git add, git commit, git push. Create a GitHub account and push everything there. Future employers will look at your contribution graph.

Step 4: Build Three Projects Before You Move On

Tutorial hell is real. You watch 40 hours of courses and still can't build anything from scratch. The antidote: build first, watch second.

Here are three projects in order of difficulty:

Personal Portfolio – A single-page site with your bio, links, and a contact form. Host it on your Hostinger domain. Done in a weekend.
Weather Dashboard – Fetch data from a free weather API and display it beautifully. Teaches API calls, JSON, and error handling.
Simple Blog with CMS – Install WordPress through Hostinger's one-click installer, customize a theme, and publish 5 posts. This teaches you how real websites work in production — database connections, themes, plugins, and backups.

Each project should go on GitHub and your portfolio. By project #3, you'll have enough to apply for junior roles.

Step 5: Deploy Everything Publicly

Here's a rule I enforce with every student I mentor: nothing stays local. Every project goes live.

With Hostinger's shared hosting, you can host multiple websites on one plan. Build your portfolio on yourname.dev, put the weather app on a subdomain, and run the blog on another. Each deployed project is a line on your resume that a recruiter can actually click.

Tools That Accelerate Your Learning in 2026

VS Code — Free, fast, and the industry-standard editor. Install Prettier and ESLint extensions.
Chrome DevTools — Inspect any website, debug JavaScript, and test CSS changes live. Press Cmd+Option+I right now and start poking around.
MDN Web Docs — The only documentation you need. Forget W3Schools — MDN is written by Mozilla and is the authoritative reference.
Figma (free tier) — Design before you code. Even rough wireframes save hours of rewriting CSS.

Three Mistakes Beginners Make (and How to Avoid Them)

Mistake 1: "I'll start when I'm ready." You'll never feel ready. Buy the domain tonight, write HTML tomorrow, deploy it by Friday. Imperfect action beats perfect preparation every time.

Mistake 2: Jumping to frameworks too early. React, Vue, and Svelte are powerful, but they'll confuse you if you don't understand vanilla JavaScript first. Spend at least 8 weeks on the fundamentals before touching a framework.

Mistake 3: Building alone. Join communities — dev.to (you're here!), freeCodeCamp forums, and Discord servers. Share your ugly first projects. Get feedback. Teaching others what you just learned is the fastest way to solidify knowledge.

The 30-Day Challenge

If you're still reading and haven't started yet, here's your first action:

Today: Sign up for hosting at Hostinger and register a domain.
This week: Write your first HTML page and push it to GitHub.
By day 30: Have a live portfolio site with at least two projects on it.

Web development in 2026 is not about talent. It's about momentum. Once you have a live website, you'll never look at the internet the same way again.

Affiliate disclosure: Some links in this article are affiliate links. If you purchase Hostinger through my link, I may earn a commission at no extra cost to you. I only recommend tools I use myself and genuinely believe in.

10 Productivity Tools That Saved Me 20+ Hours a Week in 2026

bao001 xiao — Fri, 05 Jun 2026 03:49:16 +0000

Let's be real — 2026 is the year of information overload. Between Slack pings, email notifications, AI-generated news feeds, and the constant pressure to "do more with less," staying focused has never been harder.

Over the past few months, I've tested dozens of productivity tools. Most were garbage. But a handful genuinely changed how I work. Here are the 10 that actually saved me time — not added to the noise.

1. Notion — Your Second Brain

Notion has evolved far beyond a simple note-taking app. In 2026, it's a full-fledged workspace where I manage everything — from content calendars to affiliate marketing dashboards to personal finance tracking.

The key is building templates that automate recurring tasks. I have a "Daily Standup" template that auto-pulls my calendar, to-do list, and priority score into one page. It saves me at least 30 minutes every morning.

2. Linear — Project Management That Doesn't Get in the Way

If you're working on any kind of software or content project, Linear is a game-changer. Its keyboard-first design and ultra-fast UI mean you never wait for a page to load. Cycle tracking helps you focus on what matters this week, not everything forever.

3. Grammarly — AI-Powered Writing Assistant

Whether you're writing blog posts, emails, or social media captions, Grammarly's 2026 edition is scary good. It doesn't just fix typos — it rewrites sentences for clarity, adjusts tone for different audiences, and even suggests SEO-friendly alternatives.

4. Hostinger — Reliable Hosting for Your Projects

Here's an underrated productivity hack: stop fighting with slow hosting. If your website takes 4 seconds to load, that's 4 seconds of your life — multiplied by every visit. I moved all my blogs and side projects to Hostinger because their loading times are consistently under a second. When you're publishing content daily (like I do), a fast host means you spend less time debugging and more time creating.

Check out Hostinger's plans here →

5. Spark Mail — Inbox Zero, Actually

Spark's AI assistant reads your emails, categorizes them, and even drafts replies. I spend about 10 minutes per day on email now — down from over an hour. The "Send Later" and "Remind Me" features alone are worth the download.

6. Raycast — Spotlight on Steroids

If you're on macOS and still using Spotlight, stop. Raycast lets you search files, run calculations, control Spotify, open browser tabs, and even run custom scripts — all from one command bar. I've mapped it to Cmd+Space and never looked back.

7. Toggl Track — Time Audit Your Life

Before you can optimize your time, you need to know where it's going. Toggl Track runs in the background and logs which apps you use and for how long. I discovered I was spending 3+ hours a day in Slack doing low-value "quick replies." Now I batch Slack twice a day. That alone freed up 10 hours a week.

8. Hostinger's Built-in AI Website Builder

Another reason I recommend Hostinger? Their AI website builder generates a complete, professional site in under 5 minutes. For side projects and landing pages, this is a massive time saver. Instead of spending 2 days on HTML/CSS, you get a production-ready site before your coffee gets cold.

Start your site with Hostinger's AI builder →

9. OBS Studio — For Content Creators

Whether you're recording tutorials, streaming, or making short-form videos, OBS Studio is free, open-source, and incredibly powerful. Pair it with a decent microphone and you've got a professional studio setup for zero dollars.

10. RescueTime — The Accountability Buddy

RescueTime works quietly in the background and sends you a weekly report: "You spent X hours on productive work, Y hours on social media, Z hours in meetings." Seeing those numbers in black and white is surprisingly motivating. I cut my Twitter time from 2 hours/day to 30 minutes after the first week.

Bonus: Your Own Blog

The single best productivity decision I made in 2026 was starting a blog. Writing forces you to clarify your thinking, document what you learn, and build a library of resources you can reference later. And when you pair it with fast, affordable hosting, it becomes a low-maintenance asset.

Thinking about starting? Hostinger's shared hosting plans start at just a few bucks a month, and they include a free domain and SSL.

Final Thoughts

Productivity isn't about doing more — it's about doing less, better. The tools above helped me cut 20+ hours from my weekly workload by automating the boring stuff and eliminating distractions.

Try one or two this week. You don't need all 10. Just find the one that solves your biggest bottleneck and go from there.

This post contains affiliate links. If you make a purchase through these links, I may earn a commission at no extra cost to you.

DEV Community: bao001 xiao

Build Your Own Obsidian Web Clipper with the Web to Markdown/JSON API

The API at a glance

The plan

The code

Run it

Make it one click: a bookmarklet

Why clean Markdown matters here

Taking it further

Wrapping up

Monitor Any Website for Changes with Python (No Scraping Required)

The API at a glance

The plan

The code

How it works

Run it

Automating it

Taking it further

Wrapping up

Chat With Any Website: Build a RAG Bot in ~60 Lines of Python

The API at a glance

The plan

Prerequisites

The code

How it works

Run it

Why clean Markdown matters for RAG

Taking it further

Wrapping up

Web Scraping in 2026: 5 Approaches Compared (With Code)

The Five Approaches

1. requests + BeautifulSoup: The Classic

2. Playwright / Selenium: The Heavy Lifter

3. Trafilatura: The Article Specialist

4. A Dedicated Extraction API: The No-Maintenance Option

5. Headless Browser Farms: The Nuclear Option

Which Should You Choose?

Wrapping Up

Build Your Own "Read Later" App in 50 Lines of Python with the Web to Markdown/JSON API

What We Are Building

The Magic Behind It: The Web to Markdown/JSON API

Building the Reading List App

How It Works

1. The API Call

2. Local Storage

3. Search

Taking It Further

Why This API?

Get Started

How to Build an AI Training Data Pipeline with the Web to Markdown/JSON API

Why Your LLM Data Pipeline Needs This

The API at a Glance

Step 1: Building the Ingestion Pipeline

Step 2: Preparing for LLM Fine-Tuning

Step 3: Building a RAG Ingestion Pipeline

Why Not Scrape It Yourself?

Real-World Use Cases

Production Tips

Getting Started

How to Build an AI Training Data Pipeline with the Web to Markdown/JSON API

Introduction

The API at a Glance

Step 1: Setting Up the Pipeline Skeleton

Step 2: The Core Extractor

Step 3: Deduplication and Quality Filtering

Step 4: Exporting to Training Formats

OpenAI Fine-Tuning (JSONL)

Hugging Face Datasets (Parquet)

Step 5: Scaling Up with Sitemaps

Step 6: Full Pipeline Orchestration

Why This API Over Traditional Scraping?

Best Practices

Going Further

Conclusion

How to Monitor Website Changes with the Web to Markdown/JSON API

Introduction

The API at a Glance

Step 1: Fetching a Page Snapshot

Step 2: Storing and Comparing Snapshots

Step 3: Detecting Meaningful Changes

1. `requests` + BeautifulSoup: The Classic