DEV Community: InsForge

Web Scraper Integration

InsForge — Sun, 02 Aug 2026 01:58:39 +0000

Originally published on the InsForge blog, written by Carmen Dou (Software Engineer). Reposted here with permission.

InsForge now supports Apify for web scraping. Connect once and your coding agent can pull data from the web on demand, with no scraper infrastructure to stand up and no API keys to paste around.

One click in the dashboard authorizes your own Apify account, links it to your InsForge project, and fills a Web Scraper page with your actors, runs, and datasets. The access token stays server-side and refreshes on its own, so the connection keeps working without you managing it.

How It Works

Open Web Scraper in the InsForge dashboard, click Connect Apify, and approve in your own Apify account. The page reloads with your data in seconds.

What You See After Connecting

The page opens on three tabs, each a searchable, paginated table that works like the Users grid:

Actors: the scrapers you have created or used, most-recently-run first, with run counts and last-run time.
Runs: every run with status, start time, and cost, color-coded by outcome.
Dataset: your stored datasets with item counts, traceable back to the actor that produced them.

An icon on every row opens that actor, run, or dataset straight in the Apify console.

The connection itself lives behind the gear icon next to Web Scraper, a Web Scraper Config dialog with your Apify account, plan, status, and data-retention window, plus disconnect.

Setup for Agents

Connecting the dashboard is half the job. The point is letting your agent scrape. The same page shows a copyable prompt:

Use the insforge webscraper apify skill to scrape <what you want> and return the results.

Paste it into Claude Code, Cursor, or any agent that runs the InsForge skill, replace the placeholder with what you want, and the agent drives Apify through the skill. No manual apify login, no keys to copy around. For headless or CI setups, insforge webscraper apify connect runs the same connect flow from the terminal.

Where to Host a Full-Stack App in 2026

InsForge — Sun, 02 Aug 2026 01:58:38 +0000

Originally published on the InsForge blog, written by Carmen Dou (Software Engineer). Reposted here with permission.

TL;DR

A full-stack app is more than a frontend. To host one you need a home for the frontend, the backend/API, a database, auth, file storage, background jobs, and often an AI/model gateway. There is no single winner — the right host depends on the shape of your app:

Vercel / Netlify / Cloudflare Pages — host the frontend. Great at it; not where your long-running backend lives.
Railway / Render / Fly.io / Heroku — host the backend (containers, long-running processes) plus managed databases. You still assemble auth, storage, and the rest.
Supabase / Appwrite / Firebase — all-in-one BaaS: database, auth, storage, functions in one platform.
InsForge — agent-native cloud infrastructure: an all-in-one backend-as-a-service (database, auth, storage, functions, AI gateway) an AI coding agent can build, deploy, and verify end to end.

This guide breaks down what full-stack includes, what self-hosting actually costs, and which host to pick for which situation.

What "full-stack" actually includes

Before choosing where to host, be precise about what you're hosting. A modern full-stack app is a set of layers, and every layer needs a home:

Layer	What it does	What hosts it
Frontend	UI the user sees (React, Next.js, Vue, static assets)	Frontend hosts / CDNs
Backend / API	Business logic, request handling, server-side code	Container PaaS or serverless functions
Database	Persistent app data (usually Postgres)	Managed database services
Authentication	Login, sessions, OAuth, access control	Auth service or BaaS
File storage	Uploads, documents, media	Object storage (S3-compatible)
Background jobs	Cron, queues, async work	Container PaaS with workers
AI / model gateway	Calls to LLMs and embedding models	Backend functions + a gateway
Realtime	Live updates, subscriptions, streaming	Websocket-capable backend

The mistake beginners make is treating "hosting" as one decision. In practice you're hosting layers, and many platforms only cover some of them. The question isn't "which host is best" — it's "which combination covers all my layers with the fewest moving parts."

The pitfalls of self-hosting (rolling your own on a VM)

The tempting move is to rent a raw VM (an EC2 instance, a droplet) and run everything yourself. It looks cheap and maximally flexible. Here's what it actually costs:

You become the ops team. OS patches, security updates, TLS certificates, firewall rules, and backups are now your job, forever.
No autoscaling by default. A raw VM has a fixed size. Traffic spikes mean either overpaying for idle capacity or falling over under load.
Databases are the hard part. Running production Postgres yourself — replication, backups, point-in-time recovery, failover — is a specialty. Most self-host projects quietly move the database to a managed service anyway.
Secrets and environments sprawl. Dev, staging, and prod on hand-managed VMs drift apart, and secrets end up in the wrong places.
It's opaque to AI agents. An AI coding agent can't reliably deploy to and reason about a bespoke VM setup — there's no structured interface for it to read the environment, apply changes, and verify them.

Self-hosting is the right call when you have strict data-residency or compliance requirements, or a genuine ops team. For almost everyone else in 2026, a managed platform is cheaper once you count the hours. For a concrete look at how much lives inside a raw instance, see What's Inside an EC2 Instance Your Agent Should Read.

The three categories of hosting

Platforms fall into three groups. Knowing which group you're shopping in is half the decision.

1. Frontend hosts

Optimized for static assets and frontend frameworks, served from a global CDN with instant deploys.

Platform	Best at	Watch out for
Vercel	Next.js and frontend DX; edge/serverless functions	Serverless functions are request-scoped — wrong model for long-running backends
Netlify	Static sites, Jamstack, edge functions	Same serverless limits for heavy backend work
Cloudflare Pages/Workers	Edge performance, global reach, cheap	Workers runtime has constraints vs. a full Node/container

Use a frontend host when: you have a separate backend/BaaS and just need the UI served fast and globally. Pair it with one of the categories below.

2. Backend / container PaaS

Run long-running processes, workers, and containers, usually with managed databases attached.

Platform	Best at	Watch out for
Railway	Great DX, managed Postgres/Redis, usage-based pricing, agent-friendly API	You still wire up auth, storage, and app logic yourself
Render	Web services, cron, background workers, managed DB	Fewer regions than the hyperscalers
Fly.io	Running containers close to users, global deploys, Fly Postgres	More infra knobs; you manage more
Heroku	The classic PaaS, simple `git push` deploys	Pricier; less momentum than newer entrants
Northflank / Koyeb	Container platforms with more control / global edge	More configuration than the batteries-included options

Use a container PaaS when: you have a real backend (a Node/Python/Go server, workers, long-running jobs) and want it hosted without managing VMs. You'll add auth, storage, and a database service around it.

3. All-in-one backend platforms (BaaS)

Cover most backend layers — database, auth, storage, functions, realtime — in a single platform, so you don't stitch services together.

Platform	Best at	Watch out for
Supabase	Open-source Postgres BaaS: auth, storage, edge functions, realtime	Configuration (RLS, policies) assumes a human reasoning about it
Appwrite	Self-hostable open-source BaaS with a broad feature set	Self-hosting is real ops work
Firebase	Mobile-first BaaS, realtime NoSQL, Google Cloud integration	NoSQL + proprietary lock-in
InsForge	Agent-native all-in-one backend: Postgres, auth, storage, functions, AI gateway, realtime	Agent-native layer is wasted if you never use AI agents

Use an all-in-one BaaS when: you want the fewest moving parts — database, auth, storage, and functions from one platform, often with a frontend host in front of it.

Full comparison at a glance

Platform	Category	Frontend	Backend/API	Database	Auth	Storage	Best for
Vercel	Frontend host	✅	Serverless only	—	—	—	Frontend + Next.js
Cloudflare	Frontend/edge	✅	Workers	D1 (edge)	—	R2	Edge-first apps
Railway	Container PaaS	✅	✅	Managed	—	—	Backends + workers
Render	Container PaaS	✅	✅	Managed	—	—	Web services + cron
Fly.io	Container PaaS	✅	✅	Fly Postgres	—	—	Global low-latency
Supabase	BaaS	—	Edge functions	Postgres	✅	✅	Human-driven full-stack
Appwrite	BaaS	—	Functions	Postgres/NoSQL	✅	✅	Self-hosted full-stack
Firebase	BaaS	✅ Hosting	Cloud functions	Firestore	✅	✅	Mobile-first apps
InsForge	Agent-native BaaS	✅	Edge functions	Postgres	✅	✅	Agent-built full-stack

Decision guide: which host for which app

Read this as "if your situation is the left column, start with the right column."

Your situation	Where to host
Frontend-heavy app, backend already handled by a BaaS	Vercel / Netlify / Cloudflare for the frontend
A real backend server with long-running processes and workers	Railway / Render / Fly.io
You want global low-latency and don't mind infra knobs	Fly.io
You want the whole backend from one platform, human-driven	Supabase (or Appwrite if you self-host)
Mobile-first app already in Google Cloud	Firebase
An AI coding agent is building and deploying the app	InsForge
Strict data-residency / compliance, with an ops team	Self-host (or self-hosted Appwrite)

A very common, clean 2026 stack: Vercel for the frontend + an all-in-one backend behind it. That covers every layer with two decisions instead of eight.

The rise of agent-native cloud infrastructure

Here's the shift that reframes the whole question. A growing share of full-stack apps in 2026 aren't hand-written — they're built by AI coding agents like Claude Code, Cursor, and ChatGPT. And the layers we listed at the top — frontend, backend, database, auth, storage — were all designed for a human to click through a dashboard and read documentation.

When an AI agent is assembling the whole stack, it isn't enough for one layer to be agent-friendly. The cloud infrastructure as a whole has to be legible to the agent:

The agent needs to discover the current state across layers — schema, auth rules, storage config, deployed functions — not infer it from prose docs.
The agent needs to apply changes through a structured interface (MCP, a CLI), not screenshots of a console.
The agent needs to verify what it built against real state, so AI-generated apps stop drifting into "works in the demo, broken in production."

That's the case for agent-native cloud infrastructure: not just a backend an agent can call, but a single cloud platform — database, auth, storage, functions, AI gateway, realtime, and hosting — that an agent can read, change, and check end to end. InsForge is built this way: an all-in-one backend-as-a-service exposed through MCP and a CLI, so an agent can stand up and deploy the full stack, then confirm it actually works. You can still drive it by hand — but when an agent is your primary builder, agent-native infrastructure is the difference between shipping and debugging.

Individual hosts are inching toward this — container platforms like Railway now expose agent-friendly deployment APIs, for instance. But making one layer agent-friendly isn't the same as an agent being able to reason about the entire stack. The fewer platforms the agent has to stitch together, the fewer places its mental model can drift — which is why, for agent-built apps, the winning shape is one cloud platform that spans every layer.

For more on that shift, see The Rise of Agent-Native Backends.

FAQ

Where should I host a full-stack app in 2026?
Split the decision by layer. Host the frontend on Vercel, Netlify, or Cloudflare. Host a long-running backend on Railway, Render, or Fly.io. Or cover the whole backend with an all-in-one BaaS — Supabase, Appwrite, Firebase, or InsForge. The fewest-moving-parts stack is a frontend host plus one all-in-one backend.

Can I host the frontend and backend on the same platform?
Yes. All-in-one BaaS platforms (Supabase, Appwrite, Firebase, InsForge) cover most backend layers, and several also serve a frontend. Container PaaS like Railway and Render can host both a frontend and a backend service.

Is Vercel a full-stack host?
Vercel is excellent for frontends and request-scoped serverless functions, but its serverless model isn't built for long-running backend processes. Most Vercel apps pair it with a separate backend or BaaS.

What's the cheapest way to host a full-stack app?
For small apps, a generous BaaS free tier plus a frontend host often costs nothing to start. Raw VMs look cheap but cost you in ops time. Scale-to-zero databases (like Neon) and usage-based PaaS (like Railway) keep idle costs low.

Which host is best if an AI agent is building my app?
Choose agent-native cloud infrastructure the agent can inspect and verify. The best backend for AI agents is one like InsForge, where the agent can discover schema and rules, apply changes through MCP/CLI, and confirm the result rather than guessing.

Self-Hosted InsForge Now Supports S3 Storage

InsForge — Sun, 02 Aug 2026 01:54:57 +0000

Originally published on the InsForge blog, written by Can Lyu (Principal Software Engineer). Reposted here with permission.

Self-hosted InsForge now supports bundled MinIO, bundled RustFS, and any external S3-compatible object store as storage backends.

Run an object store next to InsForge with one Docker Compose overlay, or point InsForge at infrastructure you already operate. The existing Storage REST API, SDK, Dashboard, and S3-compatible gateway all use the same files. The full configuration matrix is in the self-hosted storage guide.

Three Storage Backends

Self-hosted deployments now have three storage paths:

Local filesystem: still the default. It needs no configuration and stores files in the storage-data Docker volume.
Bundled MinIO or RustFS: one Compose overlay starts the store, creates the backing bucket, and configures InsForge automatically.
External S3-compatible storage: set environment variables for AWS S3, Wasabi, Cloudflare R2, Tencent COS, Aliyun OSS, or a remote MinIO, RustFS, Garage, or Ceph deployment.

Setting S3_BUCKET selects the S3 storage provider. It also enables InsForge's S3-compatible gateway at /storage/v1/s3.

Run MinIO or RustFS with One Overlay

Start InsForge with MinIO:

docker compose -f docker-compose.prod.yml -f docker-compose.minio.yml up -d

Or use RustFS:

docker compose -f docker-compose.prod.yml -f docker-compose.rustfs.yml up -d

Each overlay starts the object store, waits for it to become ready, creates an insforge-storage bucket, and points the InsForge backend at it. The store stays on the internal Docker network with no host ports. Uploads and downloads stream through InsForge, so the object store does not need its own public domain or TLS certificate.

Both overlays include development defaults. Set MINIO_ROOT_USER and MINIO_ROOT_PASSWORD, or RUSTFS_ACCESS_KEY and RUSTFS_SECRET_KEY, to unique values before running them in production.

Bring Your Own S3-Compatible Store

For an existing object store, add its connection details to .env:

S3_BUCKET=my-insforge-bucket
S3_REGION=us-east-1
S3_ACCESS_KEY_ID=...
S3_SECRET_ACCESS_KEY=...
S3_ENDPOINT_URL=https://s3.my-provider.example
S3_FORCE_PATH_STYLE=true

Leave S3_ENDPOINT_URL empty for AWS S3. The backend then uses the standard AWS endpoint and can read credentials from an IAM role. The S3_* names are provider-neutral; legacy AWS_S3_BUCKET and AWS_REGION values still work as fallbacks.

Path-style addressing is the default and works with MinIO, RustFS, and Wasabi. Set S3_FORCE_PATH_STYLE=false for providers such as Tencent COS and Aliyun OSS that require virtual-hosted-style requests.

Why Bundled Stores Use Proxy Mode

The bundled MinIO and RustFS overlays intentionally keep the storage container on InsForge's internal Docker network. The backend can reach http://minio:9000 or http://rustfs:9000, but a browser outside that network cannot. A presigned URL containing either internal address would fail.

The overlays therefore set S3_USE_PRESIGNED_URLS=false. Uploads and downloads pass through the public InsForge endpoint while the storage container stays private.

This is a property of the bundled deployment, not a MinIO or RustFS limitation. If your object store has a browser-accessible endpoint and supports S3 POST policies, configure it as an external S3-compatible store and leave S3_USE_PRESIGNED_URLS=true. Clients then upload and download directly with the store.

Proxy mode is still required when an external endpoint is private or does not support S3 POST policies. Cloudflare R2 is the notable public-endpoint case: set S3_USE_PRESIGNED_URLS=false for browser uploads. Proxy downloads stream instead of buffering and support Range requests for media seeking.

Use the Same Storage from Any S3 Client

Once any S3 backend is configured, open Storage → Settings → S3 Configuration and create a project access key. Then point the AWS CLI, an AWS SDK, rclone, or Terraform at:

https://your-domain.example/storage/v1/s3

Gateway uploads appear immediately in the InsForge Dashboard and Storage API. The gateway uses project credentials and path-style addressing, while the backing store remains an implementation detail.

Moving from Local Storage

Changing the backend does not migrate files already stored in the local storage-data volume. Copy those objects to the new bucket with mc mirror, aws s3 sync, or another migration tool before switching, or start with an empty bucket.