DEV Community: Alphasec

Turn Any Website into LLM-Ready Markdown with Firecrawl and Python

Alphasec — Tue, 10 Nov 2026 08:20:07 +0000

If you're feeding web content to an LLM, raw HTML wastes most of your context window on navigation, scripts and markup. What you want is the article text as markdown, with headings and code blocks intact. Firecrawl does that conversion as an API, including JavaScript-rendered pages, so you don't maintain a scraper per site.

This post walks through three jobs: converting a single page, saving a list of URLs as markdown files, and crawling a whole docs site into a folder you can index for RAG.

Setup

Install the SDK. I pin the version so the examples keep working:

pip install firecrawl==4.55.0

You can try scrape without an API key, at a low per-IP rate limit. For anything beyond a quick test, create a free account (1,000 credits a month, one credit per page) and export the key:

export FIRECRAWL_API_KEY="fc-YOUR_API_KEY"

Convert a Single Page to Markdown

from firecrawl import Firecrawl

firecrawl = Firecrawl()  # reads FIRECRAWL_API_KEY

doc = firecrawl.scrape(
    "https://docs.firecrawl.dev/introduction",
    formats=["markdown"],
    only_main_content=True,
)
print(doc.markdown)

only_main_content=True is the setting that matters most for LLM use. It strips headers, footers, cookie banners and sidebars, and on a typical docs page that's a large share of the text. It's on by default, but I set it explicitly so nobody reading the code has to guess.

Save a List of URLs as Markdown Files

For a handful of known pages, batch_scrape sends them in one job instead of a loop of single calls:

from pathlib import Path
from urllib.parse import urlparse

from firecrawl import Firecrawl

urls = [
    "https://docs.firecrawl.dev/features/scrape",
    "https://docs.firecrawl.dev/features/crawl",
    "https://docs.firecrawl.dev/features/map",
]

firecrawl = Firecrawl()
job = firecrawl.batch_scrape(urls, formats=["markdown"], poll_interval=2, timeout=120)

out = Path("pages")
out.mkdir(exist_ok=True)
for page in job.data:
    slug = urlparse(page.metadata.source_url).path.strip("/").replace("/", "_") or "index"
    (out / f"{slug}.md").write_text(page.markdown or "", encoding="utf-8")
    print(f"saved {slug}.md")

Naming files after the URL path keeps the mapping obvious when you index them later, and makes it easy to cite the source page in an answer.

Crawl an Entire Docs Site into a Folder

Before crawling, map the site. map returns the URLs it can find, fast and cheaply, so you know what the crawl will cost before you spend the credits:

from firecrawl import Firecrawl

firecrawl = Firecrawl()
links = firecrawl.map("https://docs.firecrawl.dev", limit=500)
print(links)

If the count looks right, crawl with a limit and exclude the sections you don't need. Changelogs and blog archives are the usual ones to skip:

from pathlib import Path
from urllib.parse import urlparse

from firecrawl import Firecrawl

firecrawl = Firecrawl()
job = firecrawl.crawl(
    "https://docs.firecrawl.dev",
    limit=50,
    exclude_paths=["blog/.+", "changelog.*"],
    scrape_options={"formats": ["markdown"], "only_main_content": True},
)

out = Path("docs")
out.mkdir(exist_ok=True)
for page in job.data:
    slug = urlparse(page.metadata.source_url).path.strip("/").replace("/", "_") or "index"
    (out / f"{slug}.md").write_text(page.markdown or "", encoding="utf-8")

print(f"{job.status}: {len(job.data)} pages")

crawl blocks until the job finishes. For big sites, use start_crawl to get a job ID back immediately and check it with get_crawl_status, so a long crawl doesn't tie up your script.

Tips Before You Index Anything

Set limit on every crawl. Without it, the crawl follows every link it finds, and each page costs a credit. Excluding paths up front matters for the same reason: pages you filter out after the crawl have already been paid for.

Check for near-empty pages. Login walls and bot challenges can come back as a few lines of text, and indexing them adds noise to every search. Skipping files under a few hundred characters catches most of them.

When you chunk the files for embeddings, split on headings rather than a fixed character count. Firecrawl keeps the page's heading structure, so splitting on ## gives you chunks that each cover one topic.

Where to Go Next

Once the markdown is on disk, any embedding pipeline can index it. If you want structured fields instead of text, Firecrawl's scrape also takes a JSON schema. I cover that and the crawl, map and agent endpoints in more detail in this Firecrawl guide on alphasec.io.

This post includes a referral link: signing up through it gives us both extra credits.

Stop Hiring, Start Orchestrating: Running an AI Agent Company with Paperclip on Railway

Alphasec — Sat, 16 May 2026 09:02:51 +0000

This is a cross-post from alphasec.io. Original post here.

The idea of a one-person company isn't new. What's new though is the idea of a one-person company where the person isn't doing most of the actual work. AI agents are. You set the direction, define the roles and budgets, and the agents handle the execution — writing, researching, analysing, reporting — on a schedule, without being asked twice. And tools like Paperclip are making it real enough to run today.

What is Paperclip?

Paperclip describes itself as "the human control plane for AI labor." You define a company — give it a mission, a budget, a set of agent personas — and Paperclip runs it. Agents get heartbeats on a schedule, pick up open tasks, call their assigned large language model (LLM), and post results back as issue comments. It's effectively a project management tool where every team member is an AI agent.

Paperclip started as an experiment in running fully autonomous agentic operations at scale — "open-source orchestration for zero-human companies." It's evolving into something broader: a full agent orchestration platform for building companies that run on autonomous agents rather than headcount. The pivot is still in motion, but the core idea is already compelling enough to deploy and explore.

The unit of organisation is a company. Inside a company you have agents (personas with roles, adapters, and heartbeat schedules), projects, and issues (tasks). An agent wakes up when its heartbeat fires, looks at what's assigned to it, does the work, and goes back to sleep. The idea of a pre-built company is quite powerful, and the ecosystem is slowly growing around companies.sh — pre-built company definitions you can import into Paperclip with a single command. Imagine an external penetration testing team, a content agency, or a software development shop, fully staffed with AI agents, importable in seconds.

To make it concrete, here's what a lean content agency could look like:

Agent	Role	Heartbeat
Editor in Chief	Reviews briefs, assigns stories, sets editorial direction	Daily
Research Analyst	Monitors sources, surfaces stories, drafts research notes	Every 6h
Writer	Drafts posts from research notes and briefs	On demand
SEO Analyst	Reviews drafts for keyword coverage and structure	On demand
Publisher	Formats and schedules final output	Daily

Every agent is backed by an LLM of your choice. The Editor in Chief might run on Claude for judgment-heavy decisions; the Research Analyst on Gemini Flash for speed and cost. Each agent has its own adapter, its own system prompt, its own task queue. Paperclip keeps them coordinated.

Deploying Paperclip on Railway

This section covers the deployment of Paperclip on Railway — a managed app hosting platform that handles infrastructure, TLS, and deployments without requiring server management. If you want a VPS approach with a fixed monthly cost, the DigitalOcean guide covers that instead.

Prerequisites: A Railway account, and a valid API key from at least one LLM provider (Anthropic, OpenAI, or Gemini).

Paperclip runs from a custom Dockerfile that wraps the official Paperclip image without replacing its startup chain, so tini, the upstream entrypoint, and the production server all run exactly as upstream ships them. A small entrypoint.sh handles first-boot configuration, writing a minimal config.json, printing the admin invite to the logs, and closing sign-up once the admin account exists. An earlier version ran Paperclip's development server, which idled at about 3GB and returned 403 to Railway's health check; the production server idles at about 1GB and answers /api/health, which the template now uses as its health check, with the restart policy set to Always.

The template provisions Paperclip with a managed Railway Postgres database already wired to DATABASE_URL, a persistent volume at /paperclip, and a public domain. Expect about $10-15/month at idle, more while agents are working.

Post-Deployment Setup and Connection

Railway routes traffic through its edge to port 3100 internally — no port configuration needed in the public URL. Once the service is running, the admin invite URL appears in the service logs. In the Railway dashboard, go to your deployed service → View logs → Deploy logs and search for invite.

◆  Created bootstrap CEO invite.
│  Invite URL: https://paperclip-xxxx.up.railway.app/invite/pcp_bootstrap_...
│  Expires: <72 hours after first boot>

Open the URL to create your admin account through Paperclip's native setup screen. Sign-up closes automatically once your admin account exists: the service restarts once, and the logs show entrypoint: admin claimed, restarting to disable sign-up. To keep sign-up open for teammates, set PAPERCLIP_AUTH_DISABLE_SIGN_UP=false in Railway Variables.

A couple of adapters work out of the box with environment variables — no interactive login required:

Provider	Adapter in Paperclip	Environment Variable
Anthropic	`claude_local`	`ANTHROPIC_API_KEY`
OpenAI	`opencode_local`	`OPENAI_API_KEY`
Gemini	`opencode_local`	`GEMINI_API_KEY`

For Gemini, select opencode_local as the adapter and choose your model (e.g. gemini-3.8-flash) in the agent settings. The gemini_local adapter requires interactive OAuth and won't work in a Railway container. Similarly, the codex_local adapter needs WebSocket-based authentication that can't be completed headlessly — stick with opencode_local for OpenAI models.

What's Next

The deployment covered here is just the start — Paperclip running cleanly, agents wired up, ready to receive tasks. The real interesting (and fun) part lies in what you build on top of it. The GitHub repo also includes a digitalocean/ folder with a single-script VPS deployment if you prefer a fixed-cost server. I'm fascinated by the companies.sh concept, and I'm exploring a purpose-built agent company - a threat intelligence team that monitors security advisories, extracts IOCs and TTPs, and generates detection rules - as a follow up. If that sounds exciting, it'll be next.

A Developer Built an Open-Source Dropbox on a Dare. Here's How to Self-Host It.

Alphasec — Tue, 21 Apr 2026 16:23:55 +0000

A few weeks ago, @swyx nerd-sniped @zachmeyer into building an open-source Dropbox. Zach took it seriously, and the result is Locker: a self-hostable file storage platform that covers most of what you'd actually use Dropbox or Google Drive for, without the subscription or lock-in.

I came across the thread on X, spent some time getting Locker running on Railway, and figured the deployment notes were worth writing up — especially since the setup has a few non-obvious pieces that trip you up if you're looking to self-host.

What Locker Actually Is

Locker is a Dockerized Next.js application backed by PostgreSQL. The GitHub repo is worth a look — the tech stack is modern and clean: Next.js 16 App Router, tRPC for end-to-end type safety, Drizzle ORM, BetterAuth, and Tailwind CSS, organized as a Turborepo monorepo with pnpm workspaces.

Feature-wise it covers the things you'd actually miss from the commercial alternatives:

File and folder management — upload, rename, move, delete, with a familiar explorer UI
Share links — password protection, expiry dates, and download limits per link
Upload links — let anyone send you files without an account, useful for collecting documents from clients or collaborators
Storage provider agnostic — switch between local disk, AWS S3, Cloudflare R2, or Vercel Blob via a single environment variable, no code changes required
Locker as an S3 bucket — generate Locker API keys and use them in other applications to write data directly to your Locker instance; it speaks S3-compatible protocol
Per-user storage quotas — set limits per user with usage tracking
Workspace teams — invite team members with role-based access and organise files across workspaces
QMD semantic search (optional plugin) — search inside the content of your files, not just filenames
FTS full-text search (optional plugin) — full-text search across your stored documents
Email/password and Google OAuth — authentication handled by BetterAuth, sessions managed server-side
API keys — programmatic access for building integrations and automating workflows
Virtual bash shell — navigate your file tree with ls, cd, find, cat, and grep via a terminal panel; reads your actual stored files lazily from the configured storage provider

The last one is either delightful or unnecessary depending on your personality.

The Storage Provider Thing Is the Key Insight

Most self-hosted file storage tools tie you to a specific backend. Locker doesn't. You set BLOB_STORAGE_PROVIDER in your environment and point it at wherever you want files to live:

Provider	Value	What you need
Local disk	`local`	Just a directory path
AWS S3	`s3`	Access key, secret, region, bucket
Cloudflare R2	`r2`	Account ID, keys, bucket
Vercel Blob	`vercel`	A read/write token

If you already have an S3 bucket, you can point Locker at it and immediately have a UI over your existing data. If you're starting fresh, local disk works out of the box. Switching later is one variable change.

Deploying Locker

Locker is designed to run with Docker Compose — a migrate container runs the database migrations first, then the web container starts once migrations complete. That's the intended flow.

Deploying to Railway takes a bit more work because Railway runs a single container rather than orchestrating multiple services. I spent some time getting this right: the key issue is that migrations need to run before the app starts, and the Dockerfile in the repo builds a Next.js standalone output that doesn't include the migration tooling in the final image by default.

The solution was a custom Dockerfile.railway that copies the migration dependencies into the runner stage and runs them as part of the startup command:

CMD ["sh", "-c", "cd /app/packages/database && pnpm drizzle-kit migrate && cd /app && node apps/web/server.js"]

Drizzle tracks which migrations have already been applied, so this is idempotent — subsequent deploys only apply new migrations and skip the rest.

To deploy it yourself, use the Dockerfile change above with Railway’s GitHub deploy, or the project’s Docker Compose setup.

What It Looks Like in Practice

After deployment you get a clean file management interface, workspace support for teams, and the ability to generate share links for any file or folder. The virtual bash shell is accessible via a terminal panel and lets you navigate your file tree with standard Unix commands — which turns out to be genuinely useful when you want to script something against your stored files.

Authentication supports email/password out of the box, and you can add Google OAuth by dropping in GOOGLE_CLIENT_ID and GOOGLE_CLIENT_SECRET.

Is It Ready for Production?

For personal use and small teams, maybe yes. For a large organisation relying on it as primary infrastructure, I'd want to see more production mileage first — the project is still relatively young. That said, the tech choices are solid, the codebase is readable, and the maintainer is active.

The hosted version at locker.dev is available if you want to try it before committing to self-hosting.