DEV Community

Cover image for I build Apify Actors for RAG pipelines — before/after: one Actor turns 41K of HTML soup into 3.6K of clean Markdown
Abdou
Abdou

Posted on

I build Apify Actors for RAG pipelines — before/after: one Actor turns 41K of HTML soup into 3.6K of clean Markdown

#ai #llm #rag #webdev

I build Actors on Apify — small, single-purpose tools that turn websites into structured data you can call from any pipeline. My latest one converts web pages to Markdown for RAG ingestion, and I want to show you what it actually returns, not a feature list.

I ran it tonight on an Apify documentation page, exactly as a customer would (the Store's default input).

Before — the page as your ingestion code sees it (41,286 characters):

<script src=https://cdn.cookielaw.org/scripttemplates/otSDKStub.js type=text/javascript charset=UTF-8 data-domain-script=7a8d334b-...>
<script type=text/javascript>function OptanonWrapper(){}</script>
<script src=https://cdn.jsdelivr.net/npm/@segment/analytics-consent-wrapper-onetrust@latest/dist/umd/analytics-onetrust.umd.js>
<script>!function(){var e="analytics",t=window[e]||[]; ...Segment snippet... }
Enter fullscreen mode Exit fullscreen mode

Cookie banners, consent managers, analytics loaders — thousands of characters of it before the first word of content.

After — what the Actor returned (3,643 characters):

---
title: "Develop Actors | Platform | Apify Documentation"
language: en
word_count: 500
---

# Develop Actors

This section will guide you through the whole story of Actor development.
Enter fullscreen mode Exit fullscreen mode

One honest wart: a "Copy for LLM" widget label from the docs site leaked in right after the heading. One stray line — it's on my list.

The harder test — a page in Arabic. I ran the Arabic Wikipedia article on artificial intelligence. Back came 236,855 characters of Markdown, language detected as ar, word_count 35,755, the article body clean from the first line. Fair caveat: Wikipedia's references and navboxes ride along at the tail — the body is clean, the apparatus isn't stripped.

How it works, briefly: static fetch first, Playwright rendering when a page is JavaScript-heavy, and pages that fail or come back thin are not charged. It also runs as a standby HTTP API if you want to call it like an endpoint instead of a batch job.

Price: $0.003 per page (static) or $0.004 (browser-rendered), plus platform usage of roughly $0.50–0.70 per 1,000 pages, measured on real runs.

The Actor: https://apify.com/abdou_009/apify-web-to-markdown

What are you using to get web content into your RAG pipelines today — and what does its output actually look like?

Top comments (0)