DEV Community

Amritasha Agrawal
Amritasha Agrawal

Posted on

Self-Hosting Langfuse v4 on Railway: ClickHouse, S3 and the Things That Fail Silently

#llm #selfhosted #observability #clickhouse

Langfuse is an open-source LLM engineering platform: tracing for every LLM call, tool call and agent step, plus prompt management, evals and cost tracking. The core is MIT-licensed and self-hostable, which is the main reason people pick it — prompts and model outputs often contain exactly the data you don't want in a third-party SaaS.

Langfuse v4 changed the data model: everything is now an observation in a wide, mostly immutable ClickHouse table (events_full, with a lighter events_core projection for tables and charts). It's faster, and it raised the infrastructure bar — most v3-era Docker Compose files and templates won't run it unchanged.

This guide covers running v4 on Railway, a usage-billed container platform with managed volumes, private networking and S3-compatible buckets, so there's no VM to look after. Most of it applies to any single-node setup.

The six parts

Component Job v4 minimum
langfuse-web UI and ingestion API (public) —
langfuse-worker Processes queued events into ClickHouse, runs evals and exports —
PostgreSQL Users, projects, prompts, API keys, config 15 (16+ recommended)
ClickHouse All traces/observations/scores 25.12
Redis / Valkey Queues and cache 7.0
S3-compatible storage Raw event payloads, media, batch exports required

Ingestion is asynchronous: the web container writes the raw event to S3, puts a pointer on a Redis queue, and the worker reads it back and inserts into ClickHouse. That's why three different components can each make "my trace never appeared" happen without any request returning an error.

Langfuse's own scaling page lists production minimums of 2 CPU / 4 GiB for each container and Postgres, 1 CPU / 1.5 GiB for Redis, and 2 CPU / 8 GiB for ClickHouse. On Railway those are limits, not reservations — you're billed for what's actually used — but they tell you where the memory goes.

What breaks on a single node

1. ClickHouse assumes a cluster

CLICKHOUSE_CLUSTER_ENABLED defaults to true, which makes the migrations run ON CLUSTER statements. A single ClickHouse server with no cluster config rejects them, the web container never finishes migrating, and the health check eventually kills it. Set:

CLICKHOUSE_CLUSTER_ENABLED=false
Enter fullscreen mode Exit fullscreen mode

on both web and worker.

2. ClickHouse doesn't listen on IPv6

The stock clickhouse/clickhouse-server image listens on 0.0.0.0 only. Railway's private network (*.railway.internal) has historically been IPv6-only, and on such an environment web and worker get connection refused even though both services are "up". Add a config drop-in to the ClickHouse image:

<!-- /etc/clickhouse-server/config.d/railway-listen.xml -->
<clickhouse>
    <listen_host>::</listen_host>
    <listen_host>0.0.0.0</listen_host>
    <listen_try>1</listen_try>
</clickhouse>
Enter fullscreen mode Exit fullscreen mode

Binding both covers IPv4 and IPv6 environments; listen_try stops ClickHouse from refusing to start on a host where one address family is missing.

3. ClickHouse 25.12 or newer

v4 needs ClickHouse 25.12 for lightweight updates, the JSON type and full-text search. Templates and compose files written for v3 often pin 24.x; v4's migrations fail on it. Pin a current LTS (26.3 at the time of writing) rather than latest, so a redeploy can't jump a major version under you.

4. S3 is not optional

Langfuse's docs mark LANGFUSE_S3_EVENT_UPLOAD_BUCKET as required — raw events go there before the worker processes them. You need three sets of variables (event upload, media upload, batch export), which can all point at one bucket with different prefixes:

LANGFUSE_S3_EVENT_UPLOAD_BUCKET=<bucket>
LANGFUSE_S3_EVENT_UPLOAD_PREFIX=events/
LANGFUSE_S3_MEDIA_UPLOAD_BUCKET=<bucket>
LANGFUSE_S3_MEDIA_UPLOAD_PREFIX=media/
LANGFUSE_S3_BATCH_EXPORT_ENABLED=true
LANGFUSE_S3_BATCH_EXPORT_BUCKET=<bucket>
LANGFUSE_S3_BATCH_EXPORT_PREFIX=exports/
# plus _REGION, _ENDPOINT, _ACCESS_KEY_ID, _SECRET_ACCESS_KEY and
# _FORCE_PATH_STYLE=true for each of the three
Enter fullscreen mode Exit fullscreen mode

Prefixes must end in /. One subtlety: the media endpoint must be reachable from browsers and SDKs, not just from the server, because Langfuse hands out presigned URLs for uploads and downloads. An internal-only MinIO breaks image/audio attachments while text traces keep working. A Railway storage bucket is S3-compatible with a public endpoint, so it avoids running MinIO as a seventh service.

5. Node's default heap is smaller than your container

Node.js caps its heap at about 1.7 GiB by default, regardless of how much memory the container may use. Under load the web or worker process dies with an out-of-memory error long before it hits the platform limit. Set it explicitly on both:

NODE_OPTIONS=--max-old-space-size=3072
Enter fullscreen mode Exit fullscreen mode

6. First boot is slow

On a fresh install, langfuse-web runs the Postgres and ClickHouse migrations before it answers HTTP. That takes minutes, not seconds. Give the web service's health check (/api/public/health) a long timeout — 600 seconds is safe — or the platform restarts it mid-migration.

7. Old SDKs and "where are my traces?"

This one isn't infrastructure, but it's the easiest one to hit after a v4 deploy. v4's default ingestion mode rejects Python SDK v2 and older and JS/TS SDK v3 and older. Use Python SDK v4+ or JS/TS SDK v5+, or plain OpenTelemetry.

And within those: below Python SDK 4.7.0 or JS/TS SDK 5.4.0, traces still arrive but can show up with a delay of around 15 minutes. If your dashboard is empty right after sending a test trace, check your SDK version before you start debugging ClickHouse.

8. Email is optional — but SMTP is blocked on Railway Hobby

Langfuse only needs email for invites and password resets, and works fine without it. If you want it: Railway blocks outbound SMTP ports on the Free, Trial and Hobby plans. Langfuse has a way around that which doesn't involve SMTP at all — it can send through the AWS SES v2 API over HTTPS:

SMTP_CONNECTION_URL=ses://us-east-1
EMAIL_FROM_ADDRESS=no-reply@yourdomain.com
Enter fullscreen mode Exit fullscreen mode

(with AWS credentials in the environment and a verified sender identity). Regular smtp:// URLs work on Railway's Pro plan.

Deploying it

I maintain a Railway template with all of the above already set: web and worker on the unmodified langfuse/langfuse:4 and langfuse/langfuse-worker:4 images, ClickHouse 26.3 with the IPv6 drop-in and a volume, Postgres 18, Redis 8.2 with a volume, and a Railway bucket wired into all three S3 configurations. SALT, ENCRYPTION_KEY and NEXTAUTH_SECRET are generated at deploy time.

Deploy on Railway

(Disclosure: I earn a share of Railway usage from deployments of this template. The source is public at Amritasha/langfuse-railway, and everything above works the same if you wire it up yourself.)

After the deploy goes green:

  1. Open the web service's public URL and sign up, then create an organisation and project. Anyone who finds the URL can also sign up until you set AUTH_DISABLE_SIGNUP=true on the web service — do that once your own account (and any teammates') exist.
  2. In Project settings → API keys, create a key pair.
  3. Verify UTC. Langfuse requires Postgres and ClickHouse to run in UTC. Both images default to it, but check once: SHOW timezone; in Postgres and SELECT timezone(); in ClickHouse.
  4. Send a test trace: 
pip install "langfuse>=4.7"
export LANGFUSE_PUBLIC_KEY=pk-lf-...
export LANGFUSE_SECRET_KEY=sk-lf-...
export LANGFUSE_BASE_URL=https://<your-langfuse-domain>
Enter fullscreen mode Exit fullscreen mode 
from langfuse import get_client

langfuse = get_client()

with langfuse.start_as_current_observation(as_type="span", name="smoke-test") as span:
    with langfuse.start_as_current_observation(
        as_type="generation", name="fake-llm-call", model="gpt-4o"
    ) as gen:
        gen.update(input="ping", output="pong")
    span.update(output="ok")

langfuse.flush()  # short-lived scripts must flush before exiting
Enter fullscreen mode Exit fullscreen mode

It should show up in the Observations view within seconds. If it doesn't: check the worker's logs first (queue processing errors land there), then the SDK version, then the S3 variables.

Disk: the thing that will eventually page you

Observability data grows with traffic, and nearly all of it lands on the ClickHouse volume. When ClickHouse runs out of space, inserts fail with NOT_ENOUGH_SPACE and ingestion stops.

Check what's using the space:

SELECT table,
       formatReadableSize(sum(bytes_on_disk)) AS size,
       sum(rows) AS rows
FROM system.parts
WHERE active
GROUP BY table
ORDER BY sum(bytes_on_disk) DESC;
Enter fullscreen mode Exit fullscreen mode

Plan for it up front:

  • Know your plan's limit. Railway's Hobby plan caps storage at 5 GB; Pro raises it to 1 TB. A busy project can fill 5 GB of ClickHouse quickly.
  • Alert before it's full. Langfuse's docs suggest alerting at 80%. Railway's volume metrics show usage per service.
  • Use retention. Langfuse supports per-project data retention policies; check whether they're available on your edition.

Backups

Two stores matter: Postgres (accounts, projects, prompts, API keys) and ClickHouse (all observability data). Redis holds only queues; S3 holds raw event payloads and media.

  • Railway volume backups. Each volume's Backups tab schedules daily (kept 6 days), weekly (kept 27 days) and monthly (kept 89 days) incremental backups. Turn them on for the Postgres and ClickHouse volumes. They restore only within the same project and environment, so they don't protect against deleting the project.
  • Off-platform copies. pg_dump for Postgres (use a client at least as new as the server — 18 for the template). For ClickHouse, the native BACKUP command can write straight to S3: 
  BACKUP DATABASE default
  TO S3('https://<endpoint>/<bucket>/backups/2026-11-06', '<key>', '<secret>');
Enter fullscreen mode Exit fullscreen mode

Restoring is the mirror image, RESTORE DATABASE default FROM S3(...).

Losing Postgres is worse than losing ClickHouse: without it, your API keys, prompts and users are gone, even if every trace survives.

What it costs

Railway bills per second: about $10 per GB of RAM per month, $20 per vCPU per month, and $0.16 per GB-month of volume storage. Hobby is $5/month including $5 of usage; Pro is $20/month including $20.

Langfuse is RAM-heavy at idle — two Node.js processes, ClickHouse, Postgres and Redis all stay resident — so memory is most of the bill, and ClickHouse disk grows with what you ingest. A small instance runs on a few GB of RAM in total; every GB is roughly $10/month. After a week of real traffic, the Metrics tab for each service gives you the actual number for your workload.

When not to self-host it

  • You'd ingest a lot and nobody owns the database. At high volume, ClickHouse needs real attention: disk, merges, memory. Langfuse itself recommends ClickHouse Cloud for larger deployments.
  • You want it purely to save money at low volume. Compare against Langfuse Cloud's pricing first; self-hosting pays off on data control first, cost second.
  • Your team can't upgrade SDKs. v4 requires current SDKs; if you're pinned to Python SDK v2, stay on a v3 server until you can move.

Checklist

  • [ ] ClickHouse ≥ 25.12, pinned to a version tag
  • [ ] CLICKHOUSE_CLUSTER_ENABLED=false on web and worker
  • [ ] ClickHouse listens on :: as well as 0.0.0.0
  • [ ] S3 event, media and export variables set; media endpoint reachable from browsers
  • [ ] NODE_OPTIONS=--max-old-space-size=... on web and worker
  • [ ] Long health-check timeout for the first boot
  • [ ] AUTH_DISABLE_SIGNUP=true once your accounts exist
  • [ ] Postgres and ClickHouse on UTC
  • [ ] Python SDK ≥ 4.7.0 / JS SDK ≥ 5.4.0
  • [ ] Disk alert on the ClickHouse volume
  • [ ] Volume backups + off-platform pg_dump and ClickHouse BACKUP

If something here is wrong or out of date, say so in the comments — I'll fix it.

Top comments (0)