DEV Community

Cover image for TLS for Postgres on Railway with Let's Encrypt
Rakesh Paul
Rakesh Paul

Posted on Originally published at binaryroute.com

TLS for Postgres on Railway with Let's Encrypt

#postgres #railway #security #devops

Originally published on BinaryRoute.

The Problem: A Public Port Needs Verified TLS

Railway gives each service a private network. Your app talks to Postgres over it, and nothing leaves the project. Then one day you want to reach the database from your laptop: a pg_dump, a SQL client, a sync script. You switch on TCP Proxy, get a something.proxy.rlwy.net:NNNNN address, and it just works.

The proxy forwards raw bytes; it does not add PostgreSQL TLS. My custom pgvector deployment did not offer TLS either. Asking it for TLS returned no:

Probing the proxy

$ openssl s_client -starttls postgres -connect xxxx.proxy.rlwy.net:NNNNN
...
# Postgres answers 'N' to the SSLRequest: no TLS on offer
Enter fullscreen mode Exit fullscreen mode

On a server with TLS disabled, everything that crosses that port is in clear text:

  • Your data. Queries, results and whole dumps.
  • Your login, in a sense. scram-sha-256 never sends the password itself. But a recorded exchange can be brute-forced offline, and without TLS there's no channel binding, so a man in the middle can relay the exchange and keep the session.

This post is the runbook I used on a custom pgvector service. Postgres gets a Let's Encrypt certificate, and TLS becomes required for every connection from outside the private network. Railway's current official Postgres image already enables TLS and includes pgvector, so start with the image check below before changing a start command.

Everything below uses placeholders: db.example.com for the hostname, xxxx.proxy.rlwy.net:NNNNN for the proxy, and fd12:3456:789a::/48 for the private range. Measure and substitute your own.

First: Check Your Image

In Railway, record the service's image tag and digest, entrypoint, and start command. In its console, inspect the running configuration:

Railway console: identify the TLS setup

psql -U postgres -d railway -c 'SHOW ssl;' -c 'SHOW ssl_cert_file;' -c 'SHOW ssl_ca_file;' -c 'SHOW hba_file;'
Enter fullscreen mode Exit fullscreen mode

Using the official Railway Postgres template?

It already offers TLS with its own CA and includes pgvector. Public roots do not trust that CA, and its generated certificate names localhost and, when configured, the private service domain—not your public hostname. Connecting to your public hostname with public roots and sslmode=verify-full will therefore fail. Its startup renewal can also replace the CA, so copying that CA once is not a lasting setup.

Keep its entrypoint and wrapper. See Railway’s Postgres documentation and the image repository for that image. The runbook below covers custom images.

The Custom pgvector Runbook

This is the setup I tested: a pgvector/pgvector image using the official Docker Postgres entrypoint, with TLS initially off and no additional lifecycle wrapper. Confirm that shape before using these commands. Keep your existing PostgreSQL major version and image.

The Shape of the Fix

A laptop connects through the Railway TCP proxy to Postgres with TLS. DNS resolves the public hostname; the app uses a separate private-network connection.

The dotted link is a DNS lookup, not a hop in the TLS session. The CNAME resolves to the TCP proxy; database traffic flows from the laptop through that proxy to Postgres. The app's separate connection reaches Postgres over Railway's encrypted private network.

The Runbook at a Glance

  • STEP 0: Record the current state. Start command and pg_hba rules: your rollback values.
  • STEP 1: Issue the certificate. lego + Cloudflare DNS-01, on your own machine.
  • STEP 2: Store it in Railway. Base64 cert and key in two service variables.
  • STEP 3: Offer TLS. New start command, then measure where the app connects from.
  • STEP 4: Require TLS. A boot-time pg_hba: plaintext only on the private range.
  • STEP 5–6: Open the proxy, then verify. CNAME, chain check, plaintext refused, real login.

TLS goes on in two deploys, offer and then require, on purpose. Between them you measure the addresses your app really connects from, so the rule that keeps it working is based on data, not a guess.

Four decisions make this work:

Decision Why
A hostname you own (db.example.com, a CNAME to the proxy) A public CA will only certify a domain you control, and you don't control *.proxy.rlwy.net.
Let's Encrypt via DNS-01 Nothing serves HTTP on a database hostname, so HTTP-01 can't work. DNS-01 proves control with a TXT record instead.
Cert and key in Railway variables, written to /tmp/ssl at boot Nothing touches the data volume. Rollback is just configuration.
A boot-time pg_hba.conf ssl=on only offers TLS. A client with sslmode=disable still connects. The pg_hba rules are what require it.

Let's Encrypt or your own CA?

Both work with the same steps. The difference is where the work goes:

Let's Encrypt Your own CA
Trust on the client Public roots, explicitly enabled where needed Copy ca.crt to every client
Certificate life 90 days, and getting shorter Your choice, e.g. 2 years
Renewal Based on expiry/ARI: renew, update 2 variables, redeploy Before the leaf certificate expires
Exposure Hostname appears in public Certificate Transparency logs Nothing public
Prerequisites A DNS provider with an API, a scoped token, lego Just openssl

I picked Let's Encrypt because clients can use public roots, with the appropriate trust configuration. A private CA reuses the server setup but needs its root installed on every client and its own issuance/renewal procedure; skip the lego commands in that case.

Before You Start

These terminal examples target macOS with zsh, lego v5, and OpenSSL 3. Check lego --version and openssl version; Homebrew OpenSSL may need its full path, $(brew --prefix openssl@3)/bin/openssl. Existing lego v4 users should follow the v5 migration guide. Linux users need their distribution's CA-bundle path and clipboard command; pbcopy and base64 -i are macOS-specific.

  • A domain on Cloudflare (or any DNS provider lego supports). Pick a plain name like db.example.com. There's no point making it random: every Let's Encrypt name lands in public Certificate Transparency logs anyway.
  • A scoped Cloudflare API token, limited to that one zone, with two permissions:
    • Zone → DNS → Edit, to write the challenge TXT record;
    • Zone → Zone → Read, so lego can find the zone. Without it, issuance fails.
  • lego and a modern OpenSSL (for -starttls postgres): 
  brew install lego openssl
Enter fullscreen mode Exit fullscreen mode
  • A backup. On plans without a Backups tab, take one from the service console and keep it until you've restored a copy somewhere else:

Railway console (Postgres service)

  pg_dump -U postgres -d railway -Fc -f /var/lib/postgresql/pre-tls.dump
  chmod 600 /var/lib/postgresql/pre-tls.dump
  pg_restore --list /var/lib/postgresql/pre-tls.dump | head
Enter fullscreen mode Exit fullscreen mode

pg_restore --list only reads the catalog. It doesn't prove the dump restores, and a dump on the database's own volume won't survive losing that volume.

No psql on your laptop? The runbook uses the official Docker image for the public probes. Mount an explicit CA bundle so verification is independent of what that image ships: -v /etc/ssl/cert.pem:/certs/roots.pem:ro on macOS.

Step 0: Record the Current State

Two things to write down before changing anything. First, the service's start command from Settings → Deploy. On the pgvector template it looks like this, and it's your rollback value:

Original start command

/bin/sh -c "unset PGPORT; docker-entrypoint.sh postgres --port=5432"
Enter fullscreen mode Exit fullscreen mode

Second, the current pg_hba.conf rules, from the service console:

Railway console

grep -vE '^\s*(#|$)' "$(psql -U postgres -Atc 'show hba_file')"
Enter fullscreen mode Exit fullscreen mode

The image default is trust for local socket and loopback connections, the matching replication lines, and host all all all scram-sha-256. Any extra line you find must carry over into step 4.

Step 1: Issue the Certificate

Run this in a normal terminal, not inside an AI assistant or anything else that logs, so the token never lands in a transcript. read -s keeps it out of your shell history too:

Issue (zsh)

read -s 'CLOUDFLARE_DNS_API_TOKEN?Cloudflare token: ' && export CLOUDFLARE_DNS_API_TOKEN
lego run --accept-tos --email you@example.com --dns cloudflare \
  --dns.resolvers <ns1>.ns.cloudflare.com:53,<ns2>.ns.cloudflare.com:53 \
  --domains db.example.com --key-type EC256 --path ~/.lego-db
unset CLOUDFLARE_DNS_API_TOKEN
Enter fullscreen mode Exit fullscreen mode

In bash, use read -rsp 'Cloudflare token: ' CLOUDFLARE_DNS_API_TOKEN instead. Replace the <ns1>/<ns2> names with your zone's two assigned nameservers, from Cloudflare's DNS page.

lego requests a certificate, creates a Cloudflare DNS TXT challenge, waits for validation by Let’s Encrypt, and removes the TXT record.

Two things that cost me time:

  • 9109: Cannot use the access token from location. The token's IP filter didn't match the address my terminal actually used. Command-line tools often go out over native IPv6 while the browser shows something else (iCloud Private Relay, a VPN). Check with curl -s -6 https://api.cloudflare.com/cdn-cgi/trace | grep ip= (and -4), and allow the IPv6 address as a /64.
  • NXDOMAIN until the 2-minute timeout. My router's resolver had cached "no such record" for the zone's negative-cache TTL, 30 minutes. --dns.resolvers makes lego poll Cloudflare's authoritative nameservers, which don't cache.

Then check what you got. The certificate is public, so this is safe to print. Never print the .key:

Inspect the certificate

openssl x509 -in ~/.lego-db/certificates/db.example.com.crt \
  -noout -subject -issuer -enddate -ext subjectAltName
Enter fullscreen mode Exit fullscreen mode

Expect the SAN to contain db.example.com, a Let's Encrypt issuer, and a future end date. lego v5 omits the subject Common Name by default; an empty subject is not a failure. The .crt holds the full chain (leaf plus intermediates), which is what Postgres should serve. See lego's certificate output.

Step 2: Store It in Railway

Railway variables are single strings, and PEM files are multi-line. Base64 them so the newlines survive, straight to the clipboard:

Copy cert → PG_SSL_CERT

base64 -i ~/.lego-db/certificates/db.example.com.crt | pbcopy
Enter fullscreen mode Exit fullscreen mode

Copy key → PG_SSL_KEY

base64 -i ~/.lego-db/certificates/db.example.com.key | pbcopy
Enter fullscreen mode Exit fullscreen mode

Paste each into a variable on the Postgres service. Adding variables stages a redeploy, so don't deploy yet. Step 3's start command needs to be in place first.

Step 3: Offer TLS

Replace the start command with this. It has to be one line:

Start command: TLS offered

/bin/sh -c "unset PGPORT; set -e; (umask 077; mkdir -p /tmp/ssl; printf %s $PG_SSL_CERT | base64 -d > /tmp/ssl/server.crt; printf %s $PG_SSL_KEY | base64 -d > /tmp/ssl/server.key); chown -R postgres:postgres /tmp/ssl; exec docker-entrypoint.sh postgres --port=5432 -c ssl=on -c ssl_cert_file=/tmp/ssl/server.crt -c ssl_key_file=/tmp/ssl/server.key -c ssl_min_protocol_version=TLSv1.3"
Enter fullscreen mode Exit fullscreen mode

Taken apart:

(umask 077; …)

Writes owner-only files. The parentheses isolate the umask: in my image, a global umask carried into the entrypoint and caused Permission denied.

printf %s $PG_SSL_CERT | base64 -d

Decodes the variables. They are unquoted to avoid nesting quotes in the field; this relies on each variable containing only single-line base64, without whitespace or shell metacharacters.

chown -R postgres:postgres

Makes the files readable by Postgres while the key remains owner-only.

exec docker-entrypoint.sh postgres …

Starts the custom image's normal entrypoint with TLS settings. The certificate files stay outside the data directory.

ssl_min_protocol_version=TLSv1.3

Requires TLS 1.3. Check all client runtimes support it before deploying; PostgreSQL's default minimum is TLS 1.2.

Deploy. Expect a short interruption; most connection pools retry, but check your application's reconnect behavior. Then check:

  1. The logs say database system is ready to accept connections, with no SSL or private key errors.
  2. Your app still works.
  3. Measure where your app connects from. This is the input to step 4:

Railway console

psql -U postgres -Atc "select distinct client_addr, s.ssl from pg_stat_activity a
  left join pg_stat_ssl s using (pid) where backend_type = 'client backend' and client_addr is not null"
Enter fullscreen mode Exit fullscreen mode

My app connected over Railway's private IPv6 network from an address like fd12:3456:789a:1:…, with ssl = f. The example /48 is based on that setup. Railway documents an isolated, encrypted private network, but does not promise that prefix length.

Check every app, worker, and cron service, including fresh connections after redeploys. Compare their IPv4 and IPv6 source addresses with your environment's interface, route, and allocation information before choosing a range. One address alone cannot tell you its subnet. A /128 allows just one IPv6 address, but it may change on redeploy.

If you cannot establish a suitable private range, require TLS everywhere. In step 4, omit the entire echo 'hostnossl …' >> /tmp/ssl/pg_hba.conf; statement. Before enforcing that variant, enable and test TLS in every private client during this offer phase. A certificate for db.example.com will not pass hostname verification for postgres.railway.internal; verified private clients need an appropriate certificate name and trusted CA, or must use the public verified endpoint. Do not solve that mismatch by disabling certificate validation.

Step 4: Require TLS Outside the Private Network

Now the start command also writes a pg_hba.conf into /tmp/ssl and points Postgres at it with hba_file. The rules are checked top to bottom, and the first match wins:

Connections are checked in order: trust local connections, authenticate approved private plaintext connections, authenticate TLS connections, and reject all other connections.

Start command: TLS required

/bin/sh -c "unset PGPORT; set -e; (umask 077; mkdir -p /tmp/ssl; printf %s $PG_SSL_CERT | base64 -d > /tmp/ssl/server.crt; printf %s $PG_SSL_KEY | base64 -d > /tmp/ssl/server.key; echo 'local all all trust' > /tmp/ssl/pg_hba.conf; echo 'host all all 127.0.0.1/32 trust' >> /tmp/ssl/pg_hba.conf; echo 'host all all ::1/128 trust' >> /tmp/ssl/pg_hba.conf; echo 'hostnossl all all fd12:3456:789a::/48 scram-sha-256' >> /tmp/ssl/pg_hba.conf; echo 'hostssl all all all scram-sha-256' >> /tmp/ssl/pg_hba.conf); chown -R postgres:postgres /tmp/ssl; exec docker-entrypoint.sh postgres --port=5432 -c ssl=on -c ssl_cert_file=/tmp/ssl/server.crt -c ssl_key_file=/tmp/ssl/server.key -c ssl_min_protocol_version=TLSv1.3 -c hba_file=/tmp/ssl/pg_hba.conf"
Enter fullscreen mode Exit fullscreen mode

Replace fd12:3456:789a::/48 only with a private range you established in step 3, or omit that statement for the TLS-everywhere variant. Never copy the example range unchanged. The replication lines are gone because my service had no replicas; preserve any replication and custom rules your service needs. Those rules must not reintroduce a public plaintext path. PostgreSQL documents the first-match rule and connection types.

⚠️ Avoid backslashes in this Railway start command

My first attempt wrote the rules with a single printf '%s\n' …. Railway's field strips backslashes, so \n became n, the five rules ran together into one line, and Postgres refused to start: invalid authentication method "trustnhost". That's why each rule now gets its own echo. If a deploy fails, put the step 3 command back; it's known to work.

Test it locally first. Use a disposable database with the same image tag/digest and an empty volume. Strip backslashes to check for the field behavior that caused my outage. Confirm that pg_hba_file_rules shows the intended rules without errors. From outside the allowed private range, check that plaintext is refused and verified TLS succeeds. The TLS-everywhere variant has four rules rather than five.

Deploy, then check in the console:

Railway console

psql -U postgres -Atc "show hba_file" -c "select line_number, type, address, netmask, error from pg_hba_file_rules"
Enter fullscreen mode Exit fullscreen mode

Expect /tmp/ssl/pg_hba.conf and an empty error column for every rule. Open fresh app connections and rerun step 3's query to check their source address and TLS state; a cached successful request does not prove a rule matches. Watch each cron job's next run too. A no pg_hba.conf entry error names the rejected address: investigate its origin before changing the policy.

Step 5: Open the Proxy and Point DNS at It

  1. Postgres service → Settings → Networking → TCP Proxy, target port 5432. Railway assigns xxxx.proxy.rlwy.net:NNNNN.
  2. In Cloudflare DNS, add CNAME db → xxxx.proxy.rlwy.net, set to DNS only (grey cloud). The orange-cloud proxy only carries HTTP.

Connect with the hostname and the proxy's port: db.example.com:NNNNN.

Step 6: Verify, Without Sending a Password Until the End

Set two shell variables, then run the checks in order. The first two never send a credential.

H=db.example.com
P=NNNNN
Enter fullscreen mode Exit fullscreen mode

a. The chain and hostname verify, strictly:

6a: TLS handshake

(
  set -o pipefail
  openssl s_client -brief -starttls postgres -connect $H:$P -servername $H \
    -verify_hostname $H -verify_return_error -CAfile /etc/ssl/cert.pem </dev/null 2>&1 \
    | grep -E 'Protocol version|Verification|Peer certificate|Verified peername'
)
Enter fullscreen mode Exit fullscreen mode

You want Verification: OK, Protocol version: TLSv1.3, and a successful command exit status. The pipefail setting keeps a failed OpenSSL check from being masked by a successful grep. Any verify error is a stop: remove the proxy and investigate. See Railway's TCP proxy and custom-domain documentation.

b. Plaintext is refused. -w means psql never prompts for a password:

6b: plaintext must fail

docker run --rm postgres:18 psql -w "host=$H port=$P dbname=railway user=postgres sslmode=disable" -c 'select 1'
Enter fullscreen mode Exit fullscreen mode

You want FATAL: no pg_hba.conf entry for host "…", … no encryption. Note the source address in that message and check that it falls outside your private plaintext exception. In my setup it was in 100.64.0.0/10. The plaintext connection matches no permitted rule and is refused; the hostssl rule can match only an encrypted connection.

If you get fe_sendauth: no password supplied instead, plaintext was accepted: proxied connections are landing inside your hostnossl range. Remove the proxy and rethink. The fallback is to require TLS on the private network too.

Side effect: since every proxied connection comes from the proxy's address, Postgres never sees the real client. Its logs can't tell you who tried to connect.

c. A real login, fully verified. Type the password at the prompt:

6c: verify-full login

docker run --rm -it -v /etc/ssl/cert.pem:/certs/roots.pem:ro postgres:18 psql \
  "host=$H port=$P dbname=railway user=postgres sslmode=verify-full sslrootcert=/certs/roots.pem" \
  -c "select ssl, version from pg_stat_ssl where pid = pg_backend_pid();"
Enter fullscreen mode Exit fullscreen mode

Expected

 t | TLSv1.3
Enter fullscreen mode Exit fullscreen mode

verify-full checks the chain and the hostname before authenticating. If anything is wrong, the client stops before the password exchange starts: it fails closed.

Using It From Real Clients

Every client gets the same rule: sslmode=verify-full, nothing weaker.

Connection URL

postgresql://postgres:<password>@db.example.com:NNNNN/railway?sslmode=verify-full
Enter fullscreen mode Exit fullscreen mode
  • Percent-encode the password if it contains @ / : # ? %. Otherwise the URL parses wrong.
  • Node postgres.js reads sslmode=verify-full from the URL and checks against Node's built-in roots. No extra config.
  • psql in Docker needs &sslrootcert=/certs/roots.pem plus the mount from step 6c.
  • DataGrip / JetBrains: enable SSL, select Full Verification, and explicitly enable Use truststore with System, Java, or IDE roots that trust the issuer. No client certificate is needed. Use the same db.example.com hostname as the certificate. See JetBrains' SSL instructions. Consider marking the connection read-only.

Better still, make your scripts refuse a remote URL that doesn't say sslmode=verify-full. sslmode=require encrypts but doesn't check who's on the other end, so one missing parameter quietly downgrades you to "encrypted to anyone". A guard catches that before it happens:

connection guard

export function remoteTlsRefusal(connectionString: string): string | null {
  const url = new URL(connectionString)
  if (!['postgres:', 'postgresql:'].includes(url.protocol)) {
    return 'Expected a PostgreSQL connection URL'
  }
  if (url.searchParams.has('host') || url.searchParams.has('hostaddr')) {
    return 'Pass the destination in the URL authority, without host overrides'
  }
  const local = ['localhost', '127.0.0.1', '::1', '[::1]'].includes(url.hostname)
  if (local) return null
  const modes = url.searchParams.getAll('sslmode')
  return modes.length === 1 && modes[0] === 'verify-full'
    ? null
    : `Refusing ${url.hostname}: remote connections need sslmode=verify-full`
}
Enter fullscreen mode Exit fullscreen mode

This guard is a configuration check, not a substitute for the driver's TLS verification. Use the same validated URL when connecting and do not override its SSL settings afterward. Invalid URLs throw and should stop the script. Keep hostname-mismatch and untrusted-certificate tests in scripts that depend on this guarantee.

To read the password without it touching your shell history, percent-encode it before building the URL. encodeURIComponent handles the reserved characters listed above; passing it through the environment keeps the value out of the Node command's arguments:

One-off session (zsh)

read -s 'PGPW?postgres password: '
PGPW_ENCODED=$(PGPW="$PGPW" node -p 'encodeURIComponent(process.env.PGPW)')
export DATABASE_URL="postgresql://postgres:${PGPW_ENCODED}@db.example.com:NNNNN/railway?sslmode=verify-full"
# ... run your script ...
unset PGPW PGPW_ENCODED DATABASE_URL
Enter fullscreen mode Exit fullscreen mode

Renewal: Follow the Certificate's Lifetime

As of this post, Let's Encrypt's default certificates last 90 days. The announced schedule reduces that to 64 days on February 10, 2027 and 45 days on February 16, 2028. Do not hard-code a 60-day renewal interval.

Check certificate expiry and the token IP filter, run lego, update both Railway variables, redeploy, and repeat the verification checks.

  1. Record NotBefore and NotAfter. Schedule renewal around two-thirds through the actual lifetime, with repeat reminders until the deployed certificate is confirmed updated. Monitor the served endpoint's expiry independently of the local certificate.
  2. Check your terminal's IP still passes the token's filter (curl -s -6 https://api.cloudflare.com/cdn-cgi/trace | grep ip=).
  3. Run step 1's three lines again with lego v5. It supports ACME Renewal Information (ARI) and a dynamic renewal threshold; see its renewal flags. A successful no-op does not mean a new certificate was issued: check the local expiry. If your manual deployment window requires an early renewal, use --renew-force deliberately.
  4. Update both variables (step 2) and redeploy.
  5. Re-run all of step 6, and confirm the new NotAfter.

If you miss a renewal, Postgres keeps serving the expired certificate and verified clients refuse to connect. Private clients using the plaintext exception are unaffected; verified private clients fail too. Encryption does not silently downgrade, but the resulting outage can still be serious.

Automating this means a scheduled job holding both a Railway token and a DNS token. That's worth its own security review before you build it.

Rollback

Nothing above touches the data or the volume's own pg_hba.conf, so rolling back only undoes configuration:

  1. Remove the TCP proxy, which closes public access immediately.
  2. Restore the step 0 start command and delete PG_SSL_CERT / PG_SSL_KEY.
  3. Redeploy Postgres and check the app.
  4. Delete the CNAME.

Takeaways

  • A TCP proxy is not a security boundary. If the protocol behind it doesn't encrypt, neither does the proxy.
  • ssl=on offers; pg_hba requires. Until a hostssl rule is the only way in from outside, a client can still pick plaintext.
  • Measure, then restrict. Observed addresses give you evidence for a private exception, not proof of a permanent subnet. Check all clients across redeploys; require TLS everywhere if you cannot establish a suitable range.
  • Make every check fail closed. verify-full on the client, a refused plaintext probe, and scripts that won't run without the right sslmode.
  • Respect the field you're typing into. A start-command box isn't a shell. One stripped backslash took the database down for two minutes.

Top comments (0)