We moved Terraform Enterprise to a new home on EKS. The cutover went smoothly. Two days later, every pipeline in the organisation stopped producing runs at the same moment, and nobody got paged.
Every workspace was fine except for one thing. The whole platform reached GitHub through a single OAuth token, which belonged to a single personal account, and GitHub had quietly revoked it.
The architecture we actually had
On paper we had about 275 workspaces with dozens of owning teams. In practice, they all went through one token.
flowchart TD
A["GitHub ā hundreds of repos"] -->|"push / merge webhook"| B["Terraform Enterprise"]
B -->|"clone code"| C{"OAuth token ā one personal account"}
C -->|"valid"| D["Plan and apply"]
C -->|"revoked"| E["Failed to clone ā every workspace"]
F["GitHub App ā owned by the org"] -.->|"after the fix"| BWebhooks kept arriving and GitHub got a 200 for every push. The failure was in the next step, when TFE tried to download the code:
Failed to clone repository: Invalid username or token
Repository doesn't exist or isn't accessible
The second message is the one that cost us time. It showed up when someone saved workspace settings or when the projects pipeline created a new workspace. It reads like a permissions problem on a single repo, but in reality the credential was dead for every repo.
Why the token died (probably)
Re-authorising the OAuth connection fixed it straight away: same token ID, same webhooks, and runs went green. So the token was the cause. What we couldn't prove is why GitHub revoked it.
The most likely explanation is a GitHub limit that's easy to miss. An OAuth app can hold at most 10 tokens per user. Every "Reauthorize" click creates a new token, and once there are more than 10, GitHub silently revokes the oldest. During a migration you click Reauthorize a lot: once on the old instance, once on the new one, again when testing, and again when the first one looks stuck. Sooner or later, one of those clicks revokes the token production is using.
We ruled out rate limits, repository permissions, SSO de-authorisation, and the old instance sharing the app. What remained was the 10-token limit.
What hurt most was that it took two days to notice. TFE has no alert for "I can't read GitHub". Runs just didn't start, and an empty run list doesn't look broken.
The real fix: stop depending on a person
Re-authorising got us running again, but it would happen again the next time anyone clicked that button. The durable fix is a GitHub App owned by the organisation:
| OAuth (before) | GitHub App (after) | |
|---|---|---|
| Owner | One person's GitHub account | The GitHub organisation |
| Can a reauthorise click break every run? | Yes | No |
| Webhooks | One per repo, per connection | One for the whole app |
| New repos | Need a working personal token | Covered by "all repositories" |
Setting it up in TFE took about 15 minutes: create the app at org level with read access to contents and metadata, write access to commit statuses, read access to pull requests, and subscriptions to push, pull_request, create and delete. Then install it on all repositories. The org now owns the connection, so nobody's personal account is in the path.
Moving 276 workspaces onto it was the bigger job.
Switching workspaces with the API
Pointing a workspace at the app means one PATCH to its vcs-repo block, setting github-app-installation-id in place of the OAuth token ID. One catch: you must send the whole block back, or you lose the branch, the submodule setting, and the tag regex.
def switch_workspace(cfg, name, dry_run, allow_auto_apply):
ws = get_workspace(cfg, name)
vcs = ws["attributes"].get("vcs-repo")
if not vcs:
return row(name, "skipped_no_vcs", ws)
if vcs.get("github-app-installation-id") == cfg.installation_id:
return row(name, "already_switched", ws)
# Changing the VCS connection queues a run. With auto-apply on,
# that run applies whatever is on the branch, unreviewed.
if ws["attributes"].get("auto-apply") and not allow_auto_apply:
return row(name, "skipped_auto_apply", ws)
if dry_run:
return row(name, "would_switch", ws)
error = patch_vcs_repo(cfg, name, {
"identifier": vcs["identifier"],
"branch": vcs.get("branch") or "",
"ingress-submodules": bool(vcs.get("ingress-submodules")),
"tags-regex": vcs.get("tags-regex"),
"github-app-installation-id": cfg.installation_id,
})
return row(name, "failed" if error else "switched", ws, error=error)
The script writes a CSV report that includes the old OAuth token ID, so --revert-from report.csv can roll back any workspace. We backed up every workspace's vcs-repo before touching anything. Never run a bulk change without a way back.
One gotcha: the API token needs to belong to a user who has personally authorised the GitHub App in TFE. Without that, TFE returns the same unhelpful Repository doesn't exist or isn't accessible error. Same message, completely different cause.
How I triggered 272 runs in five minutes
This is the funny part, at least looking back.
Changing a workspace's VCS connection queues a run. The docs don't mention it. I ran the switch in parallel waves and was watching for errors, not run queues. A teammate messaged me:
"Man, you are triggering all the runs!!"
They were right. Roughly 272 runs had been queued across the organisation, all landing on the shared agent pool. We cancelled 192 that were still waiting before anything applied. We'd already left out the workspaces with auto-apply on, which in hindsight was the most important decision of the day. Twenty-three of them were in ACC, where a surprise auto-apply would have deployed whatever happened to be on master.
For those 23 we turned auto-apply off first, switched them, let the owners review the plans, and turned auto-apply back on through code afterwards.
The lesson: for any bulk TFE change, the number to report up front is how many runs this will start, not how many workspaces it touches.
Infrastructure as code tried to undo the fix
Our workspaces are created by a Terraform project that uses a shared module, which contained:
vcs_repo {
identifier = "org/${var.repo}"
oauth_token_id = local.github_oauth_key
}
The first apply after the switch would have pointed every workspace back at OAuth and quietly undone the day's work. So the projects workspace stayed locked until the module was fixed:
vcs_repo {
identifier = "org/${var.repo}"
github_app_installation_id = local.github_app_installation_id
}
Two details made this trickier than a one-line change:
-
Provider version.
github_app_installation_idarrived inhashicorp/tfe0.43.0, and0.44.0fixed the provider sending both attributes at once. We were pinned to~> 0.40.0, in two separateversions.tffiles, which we found out the hard way when the first plan failed oninit. -
Old module versions. Not every project was on the latest major. Pinned constraints like
~> 9.0.11never pick up10.8.0. So we pushed a patch tag on every minor line that was still in use: 11 tags, fromv7.2.4tov10.7.2, all carrying the same change.
# The private registry publishes from tag pushes. GitHub skips the
# push event when more than 3 tags arrive at once, so push them one by one.
for tag in v7.2.4 v8.0.2 v8.1.2 v9.0.12 v10.0.7; do
git push origin "$tag"
until registry_status "$tag" | grep -q '^ok$'; do sleep 5; done
done
The verification plan showed 0 add, 57 change, 0 destroy, and no vcs_repo diffs. Exactly what we wanted.
What else broke after the cutover
The GitHub token was the headline problem, but the new instance had a few smaller issues of its own:
- Runs couldn't reach an SFTP server. The new TFE egresses through different NAT gateways, and the target's security group only allowed the old ones. We added one ingress rule per environment and the timeouts went away.
- Provider tokens stored as workspace variables didn't come across in the migration. We moved them to Secrets Manager with a data source, so the next migration has one less thing to copy.
-
A run marked
erroredthat had applied everything. All 58 changes went through, but a second-instance provider got a503while configuring. The resources were fine. Only the run status was misleading. -
Drift that never goes away. The verification plan kept showing about 30 changes. CloudTrail showed why: an hourly tag-manager job rewrites the
ownertag that Terraform sets, and two project modules both manage the same GitHub repo with different descriptions. Both were there before the migration, and both will flip again on every apply until one tool owns each field.
Field notes
- Find out which identity your platform depends on. If every CI run goes through one human's token, you have a single point of failure on someone's personal account.
- GitHub OAuth apps keep at most 10 tokens per user. Repeated Reauthorize clicks during a migration can revoke the token production is still using, without any warning.
- "Repository doesn't exist or isn't accessible" almost never means the repo is missing. Check the credential first.
- Changing a workspace's VCS connection queues a run. For bulk changes, count the runs, run in small waves, and have a cancel command ready.
- Treat auto-apply workspaces differently in bulk operations. Turn auto-apply off, make the change, review, then turn it back on through code.
- Lock the workspace that manages your workspaces until its code matches the change you made by hand. Otherwise the next apply undoes it.
- Pinned module constraints don't follow your latest release. If a fix has to reach everyone, tag it on every minor line still in use.
- Push registry tags one at a time. GitHub doesn't send a push event when more than three tags are pushed together.
- Alert on absence. "No VCS-triggered runs in the last N hours" would have caught this two days earlier.
The connection to GitHub now belongs to the organisation instead of a person. All 276 workspaces use the GitHub App and none use OAuth. The old OAuth connection is still there for rollback and will be removed after a quiet period. The most valuable outcome is the alert we'll add for runs that should have started and didn't.
Top comments (0)