I Built a Page Importer That Clones Any URL Into an Editor

"Import from URL" is a button in my page editor. You paste any public page's address, and a few seconds later an editable copy of it is open in the canvas, with every image, stylesheet and script already served from your own CDN.

The feature description is one sentence. The implementation is a fetch, a bot-wall detector, an HTML parser, a bounded upload pool, and a network egress policy. Here's what each of those is defending against.

Two tiers, and only one of them fails loudly

Plain fetch gets you the HTML the server sent. It does not get you anything a framework rendered client-side, which on modern marketing pages is often the hero, the tabs and the countdown. So there's a second tier using a headless browser API that returns the DOM after load.

The interesting part is the fallback policy, because "fall back on error" is wrong here:

async function captureHtml(url: string): Promise<{ html: string; tier: ImportTier }> {
  try {
    const html = await fetchRenderedHtml(url);
    return { html, tier: 'browser-rendering' };
  } catch (err) {
    if (err instanceof BrowserRenderingNotConfiguredError) {
      logger.info('Browser Rendering not configured, falling back to fetch', { url });
      return { html: await plainFetchHtml(url), tier: 'fetch' };
    }
    if (err instanceof BrowserRenderingHttpError) throw err;
    if (err instanceof UnsafeUrlError) throw err;

    logger.warn('Browser Rendering network failure, falling back to fetch', { url, error: err });
    return { html: await plainFetchHtml(url), tier: 'fetch' };
  }
}

Three different treatments for three different causes:

• Not configured is a normal state, not an error. Most self-hosted installs never set up the browser API. Degrade silently, log at info.
• Network failure is transient. Degrade, log at warn.
• An HTTP error from the browser service propagates as a 502. This is the one that matters: if the upstream said 500 or 429, silently falling back means the user gets a visibly worse import and no explanation of why. They'd file a bug about missing images, and the real cause would be a quota.

The result carries the tier that produced it, so the UI can say which path ran. Any automatic degradation the user can't see is a support ticket with the wrong title.

Detecting a bot wall before you save it

Fetch a page behind a challenge and you get 200 OK with a body containing a spinner and a JavaScript puzzle. Store that and you've created a "page" whose content is someone else's captcha, which will confuse the user far more than an error would.

So there's a detector, and it's deliberately a pile of string checks:

const TITLE_MARKERS = ['<title>Just a moment...</title>', '<title>Just a moment…</title>'];
const CLASS_MARKERS = ['class="cf-browser-verification"', 'class="cf-injected-html"'];
const TEXT_MARKERS  = ['Checking your browser before accessing'];
const META_REFRESH_RE = /<meta\s+http-equiv=["']refresh["'][^>]*__cf_chl_tk/i;

const CF_WRAPPER_RE = /id=["']cf-wrapper["']/;
const CF_WRAPPER_MAX_BYTES = 50 * 1024;

Two things I'd point out. The title list contains both an ASCII ellipsis and a Unicode one, because the real page has shipped both and matching one of them is a detector that works until it doesn't.

And the last rule is size-gated: an element id alone is too weak a signal, since a real page could legitimately use it, so it only counts as a challenge if the whole document is under 50 KB. A challenge page is tiny; a real page that happens to contain that id is not. When a heuristic is individually too weak, pairing it with a size or structure constraint is usually cheaper than finding a stronger one.

Rewriting the assets without a serial upload loop

A captured page references 50 to 150 assets. Each needs downloading and re-uploading to the user's own storage, then the HTML needs rewriting to point at the new URLs.

The naive version does that one at a time inside the traversal, and blows the request timeout. The structure that works is three phases:

// Collect every absolute source URL first, then upload uniques with a bounded pool, then rewrite
// synchronously off the resolved map.
const pending = new Set<string>();

Collect (a Set, so a logo referenced twelve times uploads once), upload with a concurrency limit of 5, then rewrite synchronously from the resolved map. The rewrite phase touching no network at all is what makes it comprehensible: after the pool drains, it's pure string substitution over a parsed tree.

The attribute list is longer than you'd guess, and each entry is a bug someone would otherwise report:

{ tag: 'img', attr: 'srcset' }, { tag: 'link[rel="apple-touch-icon"]', attr: 'href' },
{ tag: 'video', attr: 'poster' }, { tag: 'source', attr: 'srcset' },
{ tag: 'object', attr: 'data' }, { tag: 'embed', attr: 'src' },

plus a regex pass for url(...) inside inline styles. srcset needs its own parsing branch, since it's a comma-separated list of URL-plus-descriptor pairs rather than a single URL.

The security problem this feature is, by definition

A form that takes a URL and makes the server fetch it is server-side request forgery with a nice UI. Left unguarded, someone types http://169.254.169.254/... or an internal hostname and my server dutifully retrieves it.

So every outbound fetch in this path goes through a wrapper that validates the destination and then, critically, pins the connection to the address it validated. Checking DNS and then calling fetch is not sufficient: the name can resolve differently on the second lookup, which is the DNS rebinding attack. Every redirect hop is re-validated for the same reason.

The runtime split is worth stating plainly, because it's the honest version:

• On Node, the connection is pinned to the validated IP while preserving SNI and the Host header, so certificate validation is unaffected.
• On workerd there is no node:http and its fetch can't be told which address to use. The request proceeds with a debug log, relying on the platform's egress not reaching a private network.
• Any other runtime, with neither DNS nor pinning available, is refused outright.

That last one is the part I'd argue for hardest. The tempting default is to allow when you can't check. A guard that silently disables itself on unknown infrastructure is worse than no guard, because it's the one you stop thinking about. Fail closed and let the deployment tell you it needs support.

There are also byte and time ceilings on the fetch itself (20 MB, 30 seconds), enforced by capping the response body stream rather than trusting the Content-Length header, since a hostile server will happily lie about that.

Two things I tried that made it worse

Scrolling the page before capture. It loads lazy images. It also looks exactly like automation and reliably triggered challenges on sites that had let a plain load through. Net negative.

Inlining computed styles. This gives you a pixel-accurate snapshot and destroys the page, because computed styles are resolved at one viewport width. Every media query stops mattering and the import is unusable on mobile. Snapshot fidelity and editability are in direct conflict, and for something that opens in an editor, editability has to win.