DigitalOcean Postgres Has a Firewall Trap. We Walked Right Into It.

This article was originally published on Jo4 Blog.

We added a database backup pipeline. The pipeline temporarily allowlists the runner IP, runs pg_dump, then revokes the IP. Standard pattern, used by basically every CI-driven DB job.

The first run did all of those things correctly. It also took down production.

Here's what nobody tells you about DigitalOcean managed Postgres.

The Setup

Our managed Postgres had no trusted-source rules at all. By default this means "open to the world" — or rather, open to anything that has the connection string and credentials. The droplet running our app talked to it from a public IP, no allowlist required.

The new pipeline added one rule:

doctl databases firewalls add "$DB_ID" --rule "ip_addr:${RUNNER_IP}"

The runner IP. Just the runner. We expected: "OK, the runner can also reach the DB now."

What actually happened: DigitalOcean flipped the database into enforcement mode the moment a single rule was added. From "anyone with credentials" to "only IPs on the allowlist." Our droplet wasn't on the allowlist — it had never needed to be. So the application instantly lost its database connection.

The backup pipeline was running. The dump succeeded. The app was 500-ing for every request the entire time.

The Quiet Bit

The semantics are documented somewhere — eventually we found a docs page that mentions trusted sources are enforced "when configured." That phrasing is doing a lot of work. "Configured" means "any rule exists." Adding the first rule is the moment your enforcement model changes from "off" to "on."

There is no warning in the dashboard. No prompt. No "are you sure?" You add a rule, the rule appears in the list, and your previously-public-but-credentialed database becomes private to that one rule. Everything else that wasn't on the list — including the production app — is now blocked.

If you remove all rules, the DB returns to open mode. So you can dig out of the hole. But during the window, all your traffic is dropped at the network layer, not at the auth layer. From the app's perspective, the database has vanished.

The Fix

Two parts: an immediate ops fix, and a permanent guard.

Immediate ops fix

Add the production droplet's IP to the allowlist before the pipeline runs again. Once your droplet is on the list, the pipeline adding another runner IP is fine — it's just one more entry in an existing allowlist, no enforcement-mode flip.

DROPLET_IP=$(doctl compute droplet list --format Name,PublicIPv4 --no-header \
  | awk -v n="${PROJECT_NAME}-server" '$1==n {print $2}')
doctl databases firewalls append "$DB_ID" --rule "ip_addr:${DROPLET_IP}"

append, not add — same effect for a single rule, but append makes it explicit you're not setting the entire firewall.

Permanent guard

The real fix is preflight in every workflow that touches the firewall. Refuse to run if the production droplet isn't already on the list:

- name: Verify droplet IP is in DB trusted-sources (preflight)
  run: |
    # Adding ANY trusted-source rule to a previously-empty list flips DO Postgres
    # into enforcement mode. If the droplet IP isn't already on the list, the prod
    # app loses DB access the moment we open the runner IP. Fail loudly here so we
    # never strand prod again — operator must add the droplet IP permanently.

    DB_ID="${{ steps.database.outputs.db_id }}"
    DROPLET_NAME="${PROJECT_NAME}-server"

    DROPLET_IP=$(doctl compute droplet list --format Name,PublicIPv4 --no-header \
      | awk -v n="$DROPLET_NAME" '$1==n {print $2}')

    if [ -z "$DROPLET_IP" ]; then
      echo "❌ Could not resolve public IP for droplet '$DROPLET_NAME'"
      exit 1
    fi

    if ! doctl databases firewalls list "$DB_ID" \
        | awk 'NR>1 && $3=="ip_addr" {print $4}' \
        | grep -qxF "$DROPLET_IP"; then
      echo "❌ Droplet IP $DROPLET_IP is NOT in DB trusted-sources for $DB_ID"
      echo ""
      echo "Add it once (operator) before re-running this workflow:"
      echo "  doctl databases firewalls append $DB_ID --rule \"ip_addr:$DROPLET_IP\""
      echo ""
      echo "Refusing to proceed — opening the runner IP without the droplet rule"
      echo "would lock prod out of the database."
      exit 1
    fi

    echo "✅ Droplet IP confirmed in trusted-sources"

This is a hard guard, not a hint. Without the droplet rule, the workflow refuses to run. The operator has to add the rule once, manually, with full knowledge of what they're doing. After that, every subsequent workflow run sails through the preflight.

We added the same preflight to every other workflow that touches the DB firewall — backup, seed, restore. One pattern, replicated everywhere we mutate trusted-sources.

The Footnote on doctl

While we were in there, another doctl quirk: doctl databases firewalls list doesn't support --no-header or --format. Both flags exist for other doctl subcommands, but not this one. We were parsing column 3 expecting (UUID, Type, Value) and getting back (UUID, ClusterUUID, Type, Value) because the un-suppressible header line tipped our awk indexing off by one.

# Wrong (silently parses nothing):
doctl databases firewalls list "$DB_ID" --no-header --format UUID,Type,Value \
  | awk -v ip="$RUNNER_IP" '$2=="ip_addr" && $3==ip {print $1}'

# Right:
doctl databases firewalls list "$DB_ID" \
  | awk -v ip="$RUNNER_IP" 'NR>1 && $3=="ip_addr" && $4==ip {print $1}'

Skip the header with NR>1, accept that columns are positional, and remember that doctl flag support is per-subcommand. There's no global contract.

Lessons Learned

• Empty allowlist is not the same as no enforcement. A managed-DB firewall with zero rules behaves as "off." A firewall with one rule behaves as "deny by default." Adding the first rule is a state change, not an addition.
• Test firewall changes against a non-prod database first if your platform doesn't have explicit "enforcement: on" / "enforcement: off" toggles. The blast radius of the first rule is the entire system.
• Hard preflights beat warnings. A workflow comment that says "be sure the droplet IP is on the allowlist" is wishful thinking. A preflight that exits non-zero unless the droplet IP is on the allowlist is a control.
• doctl flag support is per-subcommand. Don't assume --format works just because the next subcommand over supports it. Read the actual --help for the exact subcommand you're calling.
• When ops touches network ACLs, fence both ends. Open the rule, run the work, close the rule, and preflight that the persistent rules required by prod are present before you do any of that.

---

Have you been bitten by an "off-by-default" cloud setting that turned on the moment you configured anything? Drop the story in the comments.

Building jo4.io — a URL shortener whose ops scripts now treat firewall rules like loaded weapons.