Why Your Terraform Cloud Runs Are Stuck

Workspaces, queues, locks, and the concurrency ceiling nobody documents.

Our Terraform Cloud organization has 270 workspaces. One morning, 20 of them had a run waiting on someone to approve it, and 46 more runs queued behind those. The dashboard's "needs attention" count said twenty; the real number of things wrong was one. A change merged two days earlier, never approved, sat at the head of fifteen of those queues with two days of ordinary merges piled up behind it. Clearing it was straightforward. What cost the time was finding out the next morning that seventeen more workspaces held another 67 runs in the same state, going back nine days — and had never shown up in any sweep we ran, because the tooling searched four component names, which matched 54 workspaces. The other 216 were invisible to it.

If you manage more than a handful of Terraform Cloud (TFC) or Terraform Enterprise (TFE) workspaces, you know the moment: the dashboard shows a wall of queued, errored, and "needs attention" runs, and the honest answer to "why is this stuck" is that nobody on the team knows the model.

This is that model. Six layers — workspace, run, queue, lock, concurrency ceiling, trigger — and a triage script at the end that turns a wall of runs into a five-minute answer.

The Workspace Is the Unit of Serialization

A workspace isn't a folder of Terraform code. It's the unit of state isolation (one state file, always), variable scope, execution mode, and — the part people miss — run concurrency.

A workspace processes one apply-track run at a time. Not one apply: one run, plan through apply. Every other run that could eventually apply queues behind it, strictly FIFO, until the current one reaches a terminal status.

There's one exemption. Plan-only runs bypass both the queue and the workspace lock entirely. That covers every speculative plan — PR-triggered plans, CLI speculative plans, any run against a configuration version marked speculative — plus the planning stage of saved-plan runs. They can never apply, so they can never corrupt state, so TFC starts them immediately even while an apply-track run holds the workspace's one slot. They don't become the current run and they don't block anything behind them.

Exclude plan-only runs before you compute queue depth. On an active repo they're the majority of what you see — in the organization these examples come from, 72 of the 100 most recent runs were plan-only — and counting them turns a healthy workspace into a phantom incident.

Two structural notes before you scale:

• Group workspaces by volatility. HashiCorp's workspace guidance recommends sizing state files to minimize blast radius and separating long-lived resources (databases, networking) from frequently-changing ones (compute). The side effect is queue hygiene — volatile workspaces churn through their FIFO queue fast, and shouldn't be stuck behind something that changes twice a year.
• The anchor workspace is a serialization point. If you're assuming roles into multiple AWS accounts from a single anchor workspace — the standard Control Tower and AFT pattern — every downstream workspace consuming its outputs is rate-limited by how fast that one queue drains. One anchor is one slot.

Past a few dozen workspaces, the answer isn't more workspaces, it's Projects — a grouping layer below the org with per-project team permissions.

The Run Is a State Machine

Yellow is where a run waits on a person. Red is where it is writing, or has failed. Grey and green are terminal.

pending → plan_queued → planning → planned
   → [cost_estimated] → [policy_checked | policy_override]
   → confirmed → apply_queued → applying → applied

A common misconception: a plan with no changes never waits for you. It terminates as planned_and_finished, a final state that exists for plan-only runs and runs whose plan contains nothing to apply. TFC closes it out itself. There's nothing to confirm and nothing to discard, so no-op plans are not a source of dashboard noise. (The allow_empty_apply option overrides this if you deliberately want a no-op apply — useful after a Terraform version bump or a batch of moved blocks.)

"Queued," "errored," and "needs attention" aren't different failure modes. They're different resting points in one state machine, and the resting point is the diagnosis.

• PLANNED — real changes, awaiting confirmation or auto-apply.
• POLICY_OVERRIDE — a soft-mandatory policy failed. Needs a human decision, not a retry.
• ERRORED — root-cause the failure class first. IAM/auth, state lock, provider crash, and resource conflict need four different fixes, and blind retry only helps one of them.
• APPLYING — leave it alone. It's mid-write to real infrastructure and real state.

One resting point catches people because it looks like progress: cost_estimated. If cost estimation is enabled org-wide, a run awaiting approval sits here rather than at planned. Filter on planned alone and you miss every run that is waiting for you.

Queues Are Per-Workspace, Not Per-Org

There is no global run queue. There's one queue per workspace, and one stuck run at the head of it can accumulate an arbitrary number of runs behind it — every one of which looks like an independent problem in a dashboard count.

Across workspaces, runs execute in parallel, up to the org ceiling.

That gives you the first fork in triage: do the stuck runs cluster in a few workspaces (queue-head problem — fix the run at the front and the rest drain by themselves), or are they spread thin, one per workspace across dozens (systemic — auto-apply disabled broadly, or a new Sentinel policy shipped without a grace period)?

Locks Are State-Level, Not Run-Level

Terraform holds a state lock during plan and apply so two processes can't write the same state file. Inside TFC the run queue already prevents concurrent apply-track runs, so locks matter in two places:

1. Local CLI applies against a TFC-backed state, running alongside CI-driven runs. The queue doesn't govern locally-initiated applies; the lock is the only thing standing between you and a corrupted state file. If anything besides TFC can reach the state, workspaces alone are not sufficient isolation.
2. A run that crashed hard enough to leave the lock held without transitioning to a terminal status. This is the classic "state locked" error, it needs a deliberate manual unlock, and it is the last thing you should ever script an automatic recovery for. Confirm nothing is mid-write first.

Depth and Ceiling Are Two Different Numbers

Conflating these is where most "why won't my runs start" confusion lives.

Per-workspace depth is 1, always, on every tier — apply-track runs only.

The org concurrency ceiling caps how many runs are actively planning or applying across the entire organization. On SaaS, this is set by your plan tier and nothing else — adding agents does not raise it. HCP Terraform now has five tiers (Free, Essentials, Standard, Premium, Enterprise); the legacy free plan reached end-of-life on March 31, 2026 and was replaced by a 500-managed-resource free tier. HashiCorp's support KB publishes concurrency figures for exactly two of them: 1 concurrent run on Free, 3 on Standard. Essentials and Premium integers aren't published anywhere — check your org's plan page rather than trusting a blog post, including this one. Standard and Premium orgs can also purchase additional remote and agent runs in increments of five, up to fifty additional runs.

The concurrency number is not exposed through the API. GET /organizations/:org?include=entitlement-set returns operations, agents, policy limits and much else, but no run-concurrency field. The plan page in the UI is the only place it appears.

Self-hosted TFE is the opposite situation: the ceiling is a dial you own. On Kubernetes, TFE_CAPACITY_CONCURRENCY defaults to 10 concurrent jobs per agent pod and tunes up to 50, so it becomes capacity planning against cluster autoscaling.

This is the limit that quietly bites teams as workspace count grows without a plan or agent-pool upgrade. Zero conflicts, zero locks, zero errors — and runs still sitting in PENDING, because the org is at capacity. It presents identically to "something is stuck," but the fix isn't debugging a workspace.

What Actually Creates Runs

VCS webhooks, CLI plans and applies, direct API calls, and run triggers — a downstream workspace firing automatically after an upstream workspace applies successfully. That last one is the anchor-workspace fan-out, and it's capped at 20 source workspaces per workspace. HashiCorp warns about cascading failures and recommends short trigger chains with failure notifications. One anchor apply spawning fifteen downstream runs is exactly when you hit the org ceiling while every individual workspace queue looks perfectly healthy.

The sneakier backlog source: a superseded commit doesn't kill its predecessor's run. Push commit A, then commit B thirty seconds later, and you get two runs. The first isn't auto-discarded for being stale — it sits in PLANNED awaiting confirmation until something clears it.

Half of this has a built-in fix: "Automatically cancel speculative plans for outdated commits" (Settings → Version Control → General) cancels in-flight and pending plan-only runs when new commits land on the same branch. Turn it on. It does nothing for apply-track runs, which is the gap you have to close yourself.

And that gap is worse than "some wasted compute." A queued run is pinned to the configuration version it was created from, and that never updates. Approving a nine-day-old run applies nine-day-old code — which can be the opposite of what your current main says, with nothing in the plan's summary counts to warn you.

Policy Enforcement, Briefly

Sentinel has exactly three enforcement levels: advisory (logs a warning, blocks nothing), soft-mandatory (blocks, overridable by permitted users), and hard-mandatory (blocks unconditionally; the only way past it is removing the policy from the set).

If you're staring at a pile of POLICY_OVERRIDE runs, don't override them N times. Find the shared policy and stage it properly — advisory, then soft-mandatory, then hard-mandatory — so teams get a grace period instead of a wall.

Confirm the policy is attached before you trust it. A policy set committed to a repo enforces nothing until it's wired to the org; GET /organizations/:org/policy-sets returning an empty array is the whole answer, and it's a five-second check.

Triage, Automated

Three checks replace clicking into runs one at a time. All are read-only, and all go through the organization runs endpoint — GET /organizations/:org/runs — rather than iterating workspaces. Iterating workspaces is one request each; on a few hundred workspaces that's minutes, against seconds for the org endpoint.

Three things about this endpoint that will silently break naive code:

• Its pagination metadata omits total-count and total-pages. You get current-page, next-page, prev-page and page-size. Follow next-page until it's empty; don't compute page counts.
• Run objects carry a workspace ID but not a name. include=workspace solves it — the workspaces come back in .included on each page, so you can attach names as you stream rather than building a separate lookup.
• plan_only runs are not excluded by default. On an active org they'll be most of what you get back — filter them out explicitly for anything queue-related.

On rate limits: the general API limit is 30 requests/second — X-RateLimit-Limit: 30, with a sub-second X-Ratelimit-Reset. Some endpoints carry tighter limits depending on tier, so check the headers rather than assuming. Sequential pagination needs no sleeps; concurrent fan-out needs 429 retry.

Shared setup:

#!/usr/bin/env bash
set -euo pipefail
: "${ORG:?set ORG to your Terraform Cloud organization name}"
TF_TOKEN="${TF_TOKEN:-$(jq -r '.credentials["app.terraform.io"].token' \
  ~/.terraform.d/credentials.tfrc.json)}"
API="https://app.terraform.io/api/v2"

# Emits one run object per line, with .ws_name attached.
# include=workspace returns the workspace in .included, so there's no second
# lookup pass and no temp file. $1 is an extra query string.
tfc_runs() {
  local qs="${1:-}" page=1 resp
  while :; do
    resp=$(curl -sf -H "Authorization: Bearer $TF_TOKEN" \
      "$API/organizations/$ORG/runs?include=workspace&page%5Bsize%5D=100&page%5Bnumber%5D=$page&$qs")
    jq -c '
      (reduce (.included[]? | select(.type == "workspaces")) as $w
        ({}; .[$w.id] = $w.attributes.name)) as $ws
      | .data[]
      | .ws_name = ($ws[.relationships.workspace.data.id]
                    // .relationships.workspace.data.id)
    ' <<<"$resp"
    page=$(jq -r '.meta.pagination["next-page"] // empty' <<<"$resp")
    [[ -z "$page" ]] && break
  done
}

1. Queue-head detection. filter[status_group]=non_final returns what's running, queued, or awaiting confirmation. Filter plan-only out explicitly — they skip the queue, so counting them inflates every depth number.

tfc_runs 'filter%5Bstatus_group%5D=non_final' \
| jq -s -r '
    map(select(.attributes["plan-only"] != true))
    | group_by(.relationships.workspace.data.id)
    | map(sort_by(.attributes["created-at"]))
    | map({
        ws:       .[0].ws_name,
        head:     .[0].id,
        status:   .[0].attributes.status,
        since:    .[0].attributes["created-at"],
        blocking: (length - 1)
      })
    | sort_by(-.blocking)[]
    | "\(.blocking)\t\(.ws)\thead=\(.head)\tstatus=\(.status)\tsince=\(.since)"'

2   aft-prod-apps   head=run-a1  status=errored  since=2026-09-01T10:00:00.000Z
2   prod-network          head=run-b1  status=planned  since=2026-09-02T09:00:00.000Z
0   prod-eks              head=run-c2  status=applying since=2026-09-03T08:10:00.000Z

Sorting ascending by created-at and taking .[0] picks the oldest non-terminal run — the one holding the lock. Leading column is how many runs are stacked behind it. One workspace in double digits is a queue-head problem: fix that run and the rest drain. Every workspace showing 0 is systemic.

2. Concurrency-ceiling check. Here you want plan-only runs, because speculative plans skip the workspace queue but still consume org concurrency. Name every operation explicitly rather than relying on defaults.

OPS='plan_only,plan_and_apply,save_plan,refresh_only,destroy,empty_apply'

tfc_runs "filter%5Bstatus_group%5D=non_final&filter%5Boperation%5D=$OPS" \
| tee /tmp/nonfinal.ndjson \
| jq -r '.attributes.status' | sort | uniq -c | sort -rn

echo -n 'live concurrency: '
jq -s 'map(select(.attributes.status | IN(
    "planning","applying","plan_queued","apply_queued","cost_estimating",
    "policy_checking","post_plan_running","pre_plan_running","queuing","fetching"
  ))) | length' /tmp/nonfinal.ndjson

IN() needs jq 1.6 or later. If live concurrency sits at your tier's cap while runs pile up in pending, stop debugging — you're waiting on capacity. The histogram costs nothing on the way past: a large errored count and a large pending count are different mornings.

3. Supersession sweep. filter[status_group]=discardable is the built-in definition of "paused and safe to clear." Within each workspace, only the newest reflects current config.

tfc_runs 'filter%5Bstatus_group%5D=discardable' \
| jq -s -r '
    map(select(.attributes.actions["is-discardable"]))
    | group_by(.relationships.workspace.data.id)
    | map(sort_by(.attributes["created-at"]) | reverse | .[1:])
    | flatten | .[]
    | "\(.id)\t\(.ws_name)\t\(.attributes["created-at"])"'

Discard with POST /runs/:id/actions/discard, which accepts an optional comment — use it, so the audit trail says why.

Two verbs, and using the wrong one fails: a run that has finished planning is discardable; a run that never started is only cancelable, and discard answers 409 on it. Route on the run's own actions.is-discardable / actions.is-cancelable flags rather than matching status names, and you can't get it wrong.

Two caveats on this sweep. Newest-per-workspace is a proxy for "newest configuration version," and it's wrong for a manually-queued run pinned to an older config, so review the list before wiring it to a cron. And half this problem has a native fix already — turn on "Automatically cancel speculative plans for outdated commits" and this sweep only has to handle apply-track runs.

One ordering detail if you clear a deep queue: work newest-first, lock-holder last. Discard the head first and TFC immediately starts the next queued run, so you end up fighting a plan that's building while you're still clearing.

Run all three and the flat scary number resolves into three buckets: fix the head of the queue, wait on or scale capacity, clear the superseded noise. The dashboard gives you counts. The state machine gives you causes.

---

Further reading

• Remote operations: ordering and timing — the plan-only and saved-plan queue exemption
• Run states and stages — every status and what leaves it
• Manage your HCP Terraform plan — concurrency by tier, buying additional runs
• /runs API reference — attributes and actions used above

Comments

[Arjun Sharma]

The resting point is the diagnosis is a framing I will carry into every queue I debug, not just Terraform. Routing the discard-versus-cancel decision on the actions flags instead of status names is the kind of API detail that separates a script that works from one answering 409 at 3am. And the newest-first ordering caveat for clearing a deep queue is the sort of thing you only learn by fighting a plan storm once.