DEV Community

Cover image for Docker Swarm Said “Insufficient Resources.” The Running Service Was Using 480 KiB
Praneeth Kodumagulla
Praneeth Kodumagulla

Posted on

Docker Swarm Said “Insufficient Resources.” The Running Service Was Using 480 KiB

#devops #docker #debugging #tutorial

I expected this task to schedule.

The running service was barely doing anything:

MemoryUsage=480KiB / 5.772GiB
MemoryPercent=0.01%
CPU=0.00%
Enter fullscreen mode Exit fullscreen mode

Then I created another Docker Swarm service.

Swarm responded with:

0/1
Pending
Enter fullscreen mode Exit fullscreen mode

and:

no suitable node (insufficient resources on 1 node)
Enter fullscreen mode Exit fullscreen mode

At first glance, those observations seem incompatible.

They aren't.

The missing piece is that runtime memory usage and scheduler reservations answer different questions.

I built a small single-node Swarm lab to make that difference reproducible.

The result: a service consuming roughly 480 KiB can coexist with a scheduler that refuses another task for insufficient memory — because Swarm is making the placement decision from configured reservations, not from that service's current memory consumption.

The lab

I used a disposable single-node Docker Swarm with:

Docker client: 29.4.3
Docker server: 29.2.1
CPUs:          4
Node memory:   5.772 GiB
Enter fullscreen mode Exit fullscreen mode

There were no existing Swarm services.

The workload itself was deliberately boring: an Alpine container sleeping in a loop.

The interesting part was the reservation.

For portability, the reproduction script calculates reservation sizes from whatever memory the Swarm node reports:

Service A       60%
Service B       50%   initially
Service B       20%   after the update
Enter fullscreen mode Exit fullscreen mode

So the first placement attempt asks Swarm to accommodate:

60% + 50% = 110%
Enter fullscreen mode Exit fullscreen mode

The second asks for:

60% + 20% = 80%
Enter fullscreen mode Exit fullscreen mode

That gives us a controlled way to flip the scheduler decision by changing one reservation.

Phase 1: reserve 60%, consume almost nothing

On my 5.772 GiB node, Service A received a reservation of:

3.463 GiB
Enter fullscreen mode Exit fullscreen mode

Swarm scheduled it successfully:

swarm-reservation-a   1/1
Enter fullscreen mode Exit fullscreen mode

Then I checked its actual runtime usage:

Service=swarm-reservation-a
MemoryUsage=480KiB / 5.772GiB
MemoryPercent=0.01%
CPU=0.00%
Enter fullscreen mode Exit fullscreen mode

That distinction matters.

The 3.463 GiB reservation is not saying:

this container is currently consuming 3.463 GiB.

It is part of the resource requirement Swarm considers when deciding where the task may be placed.

Docker's documentation makes another important distinction: --reserve-memory is not a runtime memory limit. If you need to enforce a maximum amount of memory a task can consume, that is what --limit-memory is for.

Phase 2: add another 50%

Next I created Service B with a reservation equal to 50% of node memory.

On this node:

Node capacity:          5.772 GiB

Service A reservation:  3.463 GiB
Service B reservation:  2.886 GiB
                        ---------
Combined reservation:  ~6.349 GiB
Enter fullscreen mode Exit fullscreen mode

The workload itself had not suddenly become memory-hungry.

But the declared reservations no longer fit.

Swarm showed:

NAME                    REPLICAS
swarm-reservation-a     1/1
swarm-reservation-b     0/1
Enter fullscreen mode Exit fullscreen mode

And docker service ps gave the important line:

Pending
"no suitable node (insufficient resources on 1 node)"
Enter fullscreen mode Exit fullscreen mode

Meanwhile, the running lab service was still around:

480 KiB
0.01%
Enter fullscreen mode Exit fullscreen mode

That is the contradiction I wanted the lab to isolate.

It is possible to look at runtime usage and think:

There is hardly any workload here.

while the scheduler is effectively answering:

I cannot satisfy the resource commitments you declared for another task.

Both statements can be true.

Phase 3: change one number

I didn't add memory.

I didn't add another Swarm node.

I didn't restart Docker.

I didn't change the application.

I changed only Service B's reservation:

50% → 20%
Enter fullscreen mode Exit fullscreen mode

Now the scheduler math became:

Service A:  60%
Service B:  20%
            ---
Total:      80%
Enter fullscreen mode Exit fullscreen mode

On the tested node:

Service A reservation:  3.463 GiB
Service B reservation:  1.154 GiB
                        ---------
Combined reservation:  ~4.617 GiB
Enter fullscreen mode Exit fullscreen mode

That fits inside 5.772 GiB.

The result immediately changed to:

swarm-reservation-a   1/1
swarm-reservation-b   1/1
Enter fullscreen mode Exit fullscreen mode

Service B moved from Pending to Running.

And now both lab containers were still tiny at runtime:

swarm-reservation-a   ~480 KiB
swarm-reservation-b   ~480 KiB
Enter fullscreen mode Exit fullscreen mode

Why this isn't a Docker bug

Swarm is doing exactly what it was configured to do.

A reservation is information given to the scheduler about resources a task requires.

Docker documents the behavior directly: if no node has enough memory available to satisfy a service's reservation, the task stays pending.

So this:

docker stats
Enter fullscreen mode Exit fullscreen mode

and this:

docker service ps
Enter fullscreen mode Exit fullscreen mode

are giving us information about different layers of the system.

One tells us something about runtime consumption.

The other tells us whether the orchestrator can currently satisfy the task's scheduling requirements.

That's why:

low runtime usage
Enter fullscreen mode Exit fullscreen mode

does not imply:

enough schedulable reserved capacity
Enter fullscreen mode Exit fullscreen mode

The autoscaling trap

This becomes particularly interesting when infrastructure scaling and workload scheduling use different signals.

Imagine an external autoscaler whose scale-out rule is based on things such as:

host CPU utilization
host runtime memory utilization
Enter fullscreen mode Exit fullscreen mode

while the workload scheduler is deciding placement from:

configured CPU reservations
configured memory reservations
placement constraints
available nodes
Enter fullscreen mode Exit fullscreen mode

You can end up with:

Scheduler:
"insufficient resources"

Autoscaler:
"utilization is below the scale-out threshold"
Enter fullscreen mode Exit fullscreen mode

Those answers do not necessarily contradict each other.

They may simply be measuring different kinds of pressure.

Important caveat: the lab in this article tests Docker Swarm scheduling and memory reservations. It does not implement or benchmark a particular infrastructure autoscaler. The autoscaling scenario is the operational implication of the scheduling behavior, not another result claimed by this experiment.

Reproduce it yourself

I packaged the experiment into a small public lab.

The script:

  • requires a disposable single-node Swarm
  • refuses to run if existing Swarm services are present
  • calculates reservation values from node memory
  • verifies the expected Pending state
  • captures the insufficient resources error
  • lowers only Service B's reservation
  • verifies the resulting Running state
  • removes only the services created by the lab

Run the reproducible Docker Swarm lab

A successful run ends with:

60% + 50% => Pending: VERIFIED
Reason => insufficient resources: VERIFIED
60% + 20% => Running: VERIFIED

LAB_RESULT=PASS
Enter fullscreen mode Exit fullscreen mode

The repository also contains sanitized captured output, SHA-256 manifests, and a separate safety test proving that the script refuses to run in a non-empty Swarm without deleting the existing service.

The debugging lesson

When an orchestrated task refuses to schedule, don't reduce the question to:

How much CPU or memory is the machine using right now?

Also ask:

What did the workload request?
What did the scheduler reserve?
What capacity does the scheduler think remains?
What other placement requirements must be satisfied?
What signal is my infrastructure autoscaler actually watching?
Enter fullscreen mode Exit fullscreen mode

The mental model I now use is:

Scheduler pressure ≠ runtime utilization.

That distinction can save a lot of time when an application looks lightweight but the orchestrator still says there is nowhere to put it.

One question for people running orchestrators

Have you run into the equivalent mismatch with Docker Swarm, Kubernetes requests, Nomad, ECS, or an external autoscaler — where scheduler pressure said one thing while runtime utilization said another?

I'm especially interested in what signal you eventually chose for scale-out when unschedulable work and host utilization disagreed.

References

AI assistance disclosure: I used AI to help structure and edit this article. I ran the lab myself, reviewed the commands and outputs, and verified the technical claims against Docker's documentation. The reproducible script and captured evidence are linked above.

Top comments (0)