Trino. Understanding Memory Errors

Memory management is one of the most common sources of confusion for Trino operators and users. When a query fails with a memory error, the specific error code tells you a great deal about what went wrong, where it went wrong, and who enforced the limit. Yet the three most common memory errors are frequently conflated:

• EXCEEDED_GLOBAL_MEMORY_LIMIT
• EXCEEDED_LOCAL_MEMORY_LIMIT
• CLUSTER_OUT_OF_MEMORY

This article breaks down each error, explains the architectural distinctions between them, describes the scenarios that trigger them, and provides concrete remediation strategies.

---

A Quick Primer on Trino's Memory Model

Before diving into the errors, it helps to understand two dimensions Trino uses to track memory:

• Scope — Memory can be measured per node (on a single worker) or cluster-wide (summed across every worker running a query).
• Category — Trino distinguishes between:
  • User memory — heap allocated by operators to satisfy the semantics of a query (e.g., building hash tables for joins, holding aggregation state).
  • System / revocable memory — memory used by the engine's internal machinery (e.g., buffers). Total memory = user + system.

Two components enforce limits along these dimensions:

• The coordinator runs the ClusterMemoryManager, which periodically sums a query's memory across all workers and enforces cluster-wide, per-query caps. It also hosts the low-memory killer that reacts when the cluster physically runs out of memory.
• Each worker enforces per-node, per-query caps locally via its QueryContext.

With that framing, the three errors fall neatly into place.

---

1. EXCEEDED_GLOBAL_MEMORY_LIMIT

Definition

A coordinator-enforced, per-query, cluster-wide limit. The ClusterMemoryManager periodically sums the memory a single query is consuming across all worker nodes and compares that total against a configured cap. If the cap is exceeded, the query is killed immediately.

There are two sub-variants:

• exceededGlobalUserLimit — the query's aggregated user memory exceeded query.max-memory. > "Query exceeded distributed user memory limit of %s"
• exceededGlobalTotalLimit — the query's aggregated total memory (user + system) exceeded query.max-total-memory. > "Query exceeded distributed total memory limit of %s"

Root Causes

• A single query legitimately needs more distributed memory than the configured cap — typically large joins, high-cardinality GROUP BY aggregations, ORDER BY over huge result sets, or window functions over wide partitions.
• The cluster-wide caps (query.max-memory / query.max-total-memory) are set conservatively relative to the workload.

Fixes & Mitigations

Raise the limits (in config.properties on the coordinator, or per session):

Property	Session property	Default	Controls
query.max-memory	query_max_memory	20GB	Max distributed user memory per query
query.max-total-memory	query_max_total_memory	2 × query.max-memory	Max distributed total (user + system) memory per query

Optimize the query — often the better long-term fix:

• Rewrite expensive joins/aggregations and push down more selective filters to reduce data volume.
• Select only the columns you need.
• Use effective partitioning/distribution strategies so data spreads evenly.

Enable spilling so large intermediate results are written to disk instead of held in memory:

• spill-enabled (config) / SPILL_ENABLED (session).
• Tune aggregation-operator-unspill-memory-limit for spilling aggregations.

---

2. EXCEEDED_LOCAL_MEMORY_LIMIT

Definition

A worker-enforced, per-node, per-query limit. It fires when a single query's memory usage on one specific worker node exceeds the per-node cap. Crucially, this check happens locally on the worker (in QueryContext), not on the coordinator.

"Query exceeded per-node memory limit of %s [%s]"

Root Causes

• Data skew is the classic trigger: even if a query's total memory is well within global limits, a skewed key can pile a disproportionate share of data onto one node, blowing the per-node cap.
• Too much parallelism on a node (high concurrency / many drivers per task) increasing per-node pressure.
• A per-node cap (query.max-memory-per-node) that is small relative to the work assigned to each node.

Fixes & Mitigations

Raise the per-node limit (in config.properties on each worker — and the coordinator if it runs tasks):

Property	Session property	Default	Controls
query.max-memory-per-node	query_max_memory_per_node	30% of JVM heap	Max memory a single query may use on one node
memory.heap-headroom-per-node	—	30% of JVM heap	Heap reserved for untracked allocations

Constraint: query.max-memory-per-node + memory.heap-headroom-per-node must not exceed the available JVM heap, or the node fails to start.

Reduce per-task pressure:

• Lower task_concurrency (session) to reduce concurrent operator instances per task.
• Lower max-drivers-per-task / MAX_DRIVERS_PER_TASK to reduce parallelism on a node.

Attack skew and data volume:

• Improve partitioning/distribution so no single node receives an outsized share of data.
• Apply the same query-optimization and spilling techniques described above.

---

3. CLUSTER_OUT_OF_MEMORY

Definition

Not a pre-configured per-query limit. This error fires reactively when the cluster's memory pool is physically exhausted — specifically, when at least one node has blocked allocations (pool.getBlockedNodes() > 0). At that point the coordinator's low-memory killer selects a victim query (or task) and kills it to free memory.

"Query killed because the cluster is out of memory. Please try again in a few minutes."

Note that the query which receives this error may simply be the victim chosen by the killer — it is not necessarily the query that misbehaved.

Root Causes

• The cluster is undersized or overloaded for the current concurrent workload.
• Too many queries running simultaneously, each consuming memory close to their allowed limits, collectively exhausting the pool.
• Insufficient heap headroom, or a few "rogue" queries monopolizing cluster resources.

Fixes & Mitigations

This error signals genuine cluster-wide memory pressure, so the remedies are more structural:

1. Scale the cluster — add more workers, or increase JVM heap (-Xmx) on existing workers. This is the most direct fix.

2. Tune the low-memory killer to reclaim memory more effectively:

| Property | Default | Options |
|---|---|---|
| query.low-memory-killer.policy | total-reservation-on-blocked-nodes | none, total-reservation, total-reservation-on-blocked-nodes |
| task.low-memory-killer.policy | total-reservation-on-blocked-nodes | none, total-reservation-on-blocked-nodes, least-waste |

1. Cap individual queries by lowering query.max-memory / query.max-memory-per-node so no single query can exhaust the pool before being stopped.

2. Manage the workload — use resource groups to isolate and throttle workloads, preventing a few queries from monopolizing the cluster; stagger or reduce query concurrency.

3. Reserve headroom & enable revoking:

   • memory.heap-headroom-per-node reserves heap for untracked allocations, guarding against system-level OOM.
   • Tune memory-revoking-threshold and memory-revoking-target (used by the MemoryRevokingScheduler) to reclaim revocable memory more aggressively before the pool is exhausted.

4. Enable spilling to offload memory pressure to disk.

---

Side-by-Side Comparison

	EXCEEDED_GLOBAL_MEMORY_LIMIT	EXCEEDED_LOCAL_MEMORY_LIMIT	CLUSTER_OUT_OF_MEMORY
Scope	Per-query, cluster-wide	Per-query, per-node	Cluster-wide
Trigger	Query's total memory across all nodes exceeds a configured cap	Query's memory on a single node exceeds a configured cap	Physical memory pool exhausted (a node has blocked allocations)
Enforced by	Coordinator (ClusterMemoryManager)	Worker node (QueryContext)	Coordinator (low-memory killer)
Nature of limit	Pre-configured, proactive	Pre-configured, proactive	Reactive, no fixed per-query threshold
Is the failing query at fault?	Yes — it exceeded its own cap	Yes — it exceeded its own per-node cap	Not necessarily — it may be a chosen victim
Key knobs	query.max-memory, query.max-total-memory	query.max-memory-per-node, memory.heap-headroom-per-node	Cluster sizing, query.low-memory-killer.policy, task.low-memory-killer.policy
Primary fix	Raise cap or optimize/spill the query	Raise per-node cap, reduce skew/parallelism, or spill	Add capacity, tune killer, cap queries, manage workload

---

Choosing the Right Response

A simple decision guide when you encounter one of these errors:

• EXCEEDED_GLOBAL_MEMORY_LIMIT → One query is too big overall. Decide whether to grant it more memory (query.max-memory / query.max-total-memory) or make the query cheaper (filters, projections, spilling).

• EXCEEDED_LOCAL_MEMORY_LIMIT → One query is too big on one node. Suspect data skew first. If the data is genuinely large per node, raise query.max-memory-per-node or reduce per-node parallelism.

• CLUSTER_OUT_OF_MEMORY → The whole cluster is out of room. This is a capacity and workload-management problem, not a single-query tuning problem. Scale out, tune the killer policy, and use resource groups to keep the workload within the cluster's means.

---

Summary

The three errors live at different layers of Trino's memory hierarchy:

• EXCEEDED_GLOBAL_MEMORY_LIMIT and EXCEEDED_LOCAL_MEMORY_LIMIT are proactive, per-query limits — the former cluster-wide (coordinator-enforced), the latter per-node (worker-enforced). They protect the cluster by capping how much any single query may consume.
• CLUSTER_OUT_OF_MEMORY is a reactive, cluster-wide safeguard — it fires only when physical memory is exhausted and the low-memory killer must sacrifice a query to keep the cluster alive.

Understanding which boundary was crossed — a query's global budget, a query's per-node budget, or the cluster's physical capacity — points you directly at the right fix, whether that's tuning a configuration property, optimizing a query, enabling spilling, or scaling the cluster.