"Too many levels of symbolic links" with no symlink in sight: autofs, a bind mount and a cron container

In September, our PHP crons started failing with an error that sounded specific:

Too many levels of symbolic links

That is ELOOP, the error a symlink loop gives you, so the first move was to look for a loop on /srv/apps/app-a/task.php. There was no symlink on that path. Nothing to untangle.

Paths and component names below are generic examples, not ours.

By the time we looked again, on September 10, the file was readable. The autofs service had just restarted, and the same file answered on /opt/code/app-a/task.php and on /srv/apps/app-a/task.php. That left a question the successful ls could not answer: what had to work for the second path to reach the first?

For context: I run engineering at GoodBarber, an app platform, and these crons run its background jobs. The short version of what follows: the code was already bind-mounted into the container, the paths the crons used still went through autofs, so we mounted the code directly at those paths and took them out of the automount map. If ELOOP sends you hunting for symlinks that are not there, look at your automounts, and at what the container's mount namespace can actually see.

The autofs layer inside Docker

The cron container's Compose configuration already mounted the application code from the host:

host /srv/code/app-a -> container /opt/code/app-a
host /srv/code/app-b -> container /opt/code/app-b

The application expects /srv/apps/app-a and /srv/apps/app-b. The image supplied those two paths through an autofs direct map:

# /etc/auto.master
/- /etc/auto.apps --ghost,--timeout=30

# Two entries in /etc/auto.apps
/srv/apps/app-a -fstype=bind :/opt/code/app-a
/srv/apps/app-b -fstype=bind :/opt/code/app-b

Any access to either path could trigger the automounter. The other entries in that map were NFS shares, managed by the same daemon.

Two layers of mounting for code that was already local. Opening a PHP file depended on the service that manages our network mounts.

Why an automount can return ELOOP

The error name is narrower than the mechanism behind it. In Linux v6.1, follow_automount() increments nd->total_link_count, the same counter symlink traversal spends, and returns -ELOOP once it reaches MAXSYMLINKS, which is 40. During a path lookup, every automount crossed counts against the symlink limit. No symlink required.

The autofs documentation describes the other half. When an autofs filesystem is visible in several places and the mounts created by its daemon do not propagate to the others, access from those other places "will likely result in the ELOOP error". The caller keeps meeting a trigger and never sees the mount it is waiting for.

That is how you get ELOOP without a symlink. Which of the two our incident took, we cannot say: the investigation tied the failure to an NFS incident and an unhealthy automounter, and we did not capture a kernel trace. What the configuration did show is the dependency itself, and it had no reason to exist, whatever upset the daemon first.

The fix stayed in configuration

On September 10 we added direct Docker binds at the paths the crons actually use. The relevant Compose fragment:

volumes:
  - type: bind
    source: /srv/code/app-a
    target: /srv/apps/app-a
  - type: bind
    source: /srv/code/app-b
    target: /srv/apps/app-b

We also gave the cron container its own copy of the automount map, with those two entries removed. It takes both changes: leave the keys in the map and autofs installs its triggers over the direct mounts again.

One detail of our image: the map path, shown here as /etc/auto.apps, is a symlink to the real map file. The Compose override mounts the cron-specific map over that target. The web containers keep the map they had.

The application teams maintain and deploy that code. This was an operations fix, so it stayed in the mount configuration and the application paths did not move. No PHP patch, no local checkout drifting away from what the application teams deploy.

Proving the dependency is gone

Roy from The IT Crowd has a suggestion: "Have you tried turning it off and on again?" We already had a readable file after a restart. The useful test was whether it still needed the automounter.

Inside the cron container, check each mount on its own:

findmnt -o TARGET,SOURCE,FSTYPE /srv/apps/app-a
findmnt -o TARGET,SOURCE,FSTYPE /srv/apps/app-b
grep -cE '^/srv/apps/app-(a|b)[[:space:]]' /etc/auto.apps

Both mounts should resolve to the host-backed code, with no autofs trigger underneath. The count should be 0, and grep exits with status 1 when nothing matches. If mounts are stacked, read /proc/self/mountinfo as well: the filesystem on top hides what sits below it.

Then check the cron runs after the deployment. The jobs have to actually run, and their new logs have to be free of ELOOP. Silence from a job that never started proves nothing.

The deployment was confirmed working on September 10. One check we have not run yet: in a maintenance window, with the NFS-dependent jobs paused, stop autofs inside the cron container, read the PHP entry point, then restart autofs and confirm it is active. The direct bind should stay readable. That tests the dependency removal, not an NFS hang.

The maintenance we kept

We now have two maps, and any change to their shared NFS entries has to land in both. Removing a runtime dependency left us a configuration chore.

The code was local all along. We changed how the crons reach it, and opening a local PHP file no longer needs the automounter to cooperate.

Have you met ELOOP with no symlink in sight? What was underneath on your side: autofs, a bind mount, something stranger? Genuinely curious.