How I Built a Self-Healing Study Planner: Catching Paywalled "Free" Courses with Cheerio and Schema.org

Sree Naresh A — Thu, 12 Nov 2026 17:56:39 +0000

Most AI study path generators fail in a predictable way: they hallucinate dead links, ignore real-world weekly time constraints, or link to "free" courses that suddenly demand a credit card on Week 3.

I built Episteme AI (live demo) to solve this at the pipeline level. Here is how the paywall detection and constraint enforcement work under the hood.

1. Why HTTP 200 OK Isn't Enough

Many course pages return HTTP 200 while locking content behind a paywall, subscription, or 7-day trial.

Rather than trusting status codes, we fetch the target page using rotating browser headers and inspect the DOM directly with Cheerio:

// From scripts/check-url.mjs & src/lib/paywallInspector.mjs
const $ = cheerio.load(html);

// 1. Check Schema.org JSON-LD for explicit open access markers
let isAccessibleForFree = false;
$('script[type="application/ld+json"]').each((_, el) => {
  try {
    const data = JSON.parse($(el).html());
    if (data.isAccessibleForFree === true || data.offers?.price === 0) {
      isAccessibleForFree = true;
    }
  } catch {}
});

Next, we strip scripts, layout navs, and footers, scoring paywall regexes against free-audit indicators:

const PAYWALL_PATTERNS = [
  /\b(subscribe\s+to\s+unlock|subscription\s+required)\b/i,
  /\b(start\s+(your\s+)?(\d+-day\s+)?free\s+trial)\b/i,
  /\b(billed\s+(monthly|annually))\b/i,
  /\$\s*\d+(\.\d{2})?\s*(usd|\/mo|\/month)?/i,
];

If paywall markers are detected or confidence falls below threshold, the link is flagged and replaced with a verified topic-matched alternative.

2. Testing Any URL via CLI or API

We packaged this into a standalone utility developers can run directly:

# In the cloned repo:
npm run check-url -- https://docs.python.org/3/tutorial/

# Or via public API:
curl "https://episteme-ai-one.vercel.app/api/verify-url?url=https://docs.python.org/3/tutorial/"

3. Strict Time Budgets via Constrained Gemini Validation

When generating a 26-week curriculum for a working professional (5h/week = 130h budget), standard LLMs often drift.

Episteme AI restricts Gemini 2.0 Flash to an explicit allowlist of approved URLs, then runs post-generation validation (validatePath in src/lib/pathGenerator.ts):

Exact week count matches horizon
Each week's hours ≤ weekly budget cap
Sum of hours matches total budget exactly
Zero unapproved URLs allowed

If validation fails or no API key is set, it falls back to a deterministic 52-week domain curriculum.

Source Code & Contributing

The full engine is MIT-licensed and open source:
⭐ GitHub: https://github.com/SreeNaresh1/episteme-ai

🚀 Live Demo: https://episteme-ai-one.vercel.app

We have open starter issues for Rust curricula and search adapters—feedback and PRs are welcome!

DEV Community: Sree Naresh A

How I Built a Self-Healing Study Planner: Catching Paywalled "Free" Courses with Cheerio and Schema.org

1. Why HTTP 200 OK Isn't Enough

2. Testing Any URL via CLI or API

3. Strict Time Budgets via Constrained Gemini Validation

Source Code & Contributing