DEV Community

Cover image for Production broke due to un-synced secrets
Arka Chakraborty
Arka Chakraborty

Posted on

Production broke due to un-synced secrets

I run a Next.js side project deployed on Vercel, with all secrets managed in Doppler and synced to Vercel through the native Doppler↔Vercel integration. No .env files anywhere: secrets get injected at runtime locally, and synced as Sensitive env vars in Vercel for preview and production.

This week the production sync silently broke. Here's the whole thing, from discovery to fix, because the root cause is a quirk almost nobody documents.


The setup

Two syncs configured in the Doppler dashboard (Integrations → Vercel), each with import option Prefer Doppler:

Vercel scope Doppler config
Preview stg
Production prd

Locally, scripts run through doppler run so secrets are injected without a file:

npm run dev    # doppler run -- next dev
npm run build  # doppler run -- next build
Enter fullscreen mode Exit fullscreen mode

Vercel's build command is plain next build: no CLI, no DOPPLER_TOKEN. The integration owns the env vars.


How I noticed

I rotated a secret in Doppler, redeployed production, and the app was still running on the old value. Classic "did I actually deploy?" moment.

Checked the Vercel dashboard: the variable still had the old value. Checked Doppler: the new value was there. The sync between them had failed, and the failure mode was the interesting part. The integration couldn't write to the variables that already existed in Vercel.


Root cause: Vercel Sensitive env vars are write-once via the API

Once an env var exists in Vercel, especially a Sensitive one, the API can't:

  • update its value
  • convert a non-sensitive var to Sensitive
  • read it back to compare (that's the point of Sensitive: Vercel won't return the value, even to integrations)

So the integration can't reconcile existing vars. It doesn't patch around it; the sync just fails for those keys.

Why did my vars already exist? Because when I first set the project up, I created a few env vars by hand in the Vercel dashboard before wiring up the Doppler sync. They were in the wrong state (some non-sensitive, some sensitive) and the integration had been quietly unable to touch them ever since.


The fix

Low-tech, slightly nerve-wracking, but clean:

1. Delete the conflicting variables from Vercel.

Via the Vercel CLI, or in the dashboard under Settings → Environment Variables:

vercel link                      # if not already linked to the project
vercel env ls production         # see what's there
vercel env rm SECRET_NAME production
# repeat for each conflicting var
Enter fullscreen mode Exit fullscreen mode

Note: vercel env ls shows sensitive vars exist but can't show their values. Consistent with the whole write-once model.

2. Re-sync from Doppler.

Dashboard → Integrations → Vercel → your sync → hit sync again. I went one step further and deleted the whole integration, then re-created it as a full reset. That works, but it means redoing the config→scope mapping. Deleting just the vars and re-syncing is the lighter path; the integration recreates them itself as Sensitive.

3. Redeploy.

Env var changes don't apply retroactively: existing deployments keep the values they were built with.

git push          # or: vercel --prod
Enter fullscreen mode Exit fullscreen mode

4. Verify.

App healthy, new secret values live, and the vars in the Vercel dashboard now show as Sensitive, created and owned by the integration this time.

Screen recording of the full cleanup + re-sync


Why I'll never pre-create env vars again

The whole failure traces back to one decision: creating vars in the Vercel dashboard "temporarily" before setting up the integration. Once they existed, the sync could never reconcile them, and the failure was silent enough that I only noticed when a rotated secret didn't take effect.

The rule going forward: let the integration own your env vars from day one. If a secrets sync is going to manage a variable, never create it manually, not even temporarily. If you inherit a project where vars already exist, delete them first and let the sync recreate them cleanly.

And operationally: know which secrets your deploys actually depend on before deleting anything. Deleting production env vars by hand is fine, as long as you can recreate them from your source of truth. That's the entire point of having one.


If you've hit this same wall with Doppler↔Vercel, would love to hear how you handled it.

Top comments (0)