DEV Community

Jeansen
Jeansen

Posted on

Move docker folder to external storage

Quick note to me and the world:

Docker by design stores everything (except some certs and the daemon config) in one folder. By default, that's '/var/lib/docker'.

That folder can grow quite big over time. Yes, one could do some housekeeping. But especially when using Docker Volumes for persistent data, this folder quickly grows. Aside from symlinks or mount binds, it is also possible to directly tell the Docker Daemon where to store all the files.

For instance, I have the folder /mnt/srv mounted to an external disk /dev/sdb1.

To make Docker use, e.g. /mnt/srv/docker, I put the following in /etc/docker/daemon.json. If the file does not exist, simply create it!

{
  "data-root": "/mnt/srv/docker"
}
Enter fullscreen mode Exit fullscreen mode

You do not have to create the docker folder. If /mnt/srv/docker does not exist, Docker will create it. But normally, you do not want to have a blank start, so you could simply mv /var/lib/docker /mnt/srv/.

Make sure you first stop the Docker Dameon/Service. This can be done by executing sudo systemctl stop docker.socket.

When done, simply start the service: sudo systemctl start docker.service.

The first command makes sure docker really shuts down. If you only stop the service, the socket will still be active! On the other hand, if you start the service, the socket will be activated, too.

Important

If you mount a disk partition, like I did, it might happen that some containers fail. For instance, I tried to run sonatype/nexus3 and it always failed. Here's an excerpt from the logs:

Caused by: java.lang.UnsatisfiedLinkError: /nexus-data/tmp/jna4777440640643979514.tmp: /nexus-data/tmp/jna4777440640643979514.tmp: failed to map segment from shared object
        at java.lang.ClassLoader$NativeLibrary.load(Native Method)
        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1934)
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1817)
        at java.lang.Runtime.load0(Runtime.java:782)
        at java.lang.System.load(System.java:1100)
        at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:1045)
        at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:1015)
        at com.sun.jna.Native.<clinit>(Native.java:221)
        at com.orientechnologies.common.jna.ONative.instance(ONative.java:31)
        at com.orientechnologies.orient.core.engine.OMemoryAndLocalPaginatedEnginesInitializer.configureDefaultDiskCacheSize(OMemoryAndLocalPaginatedEnginesInitializer.java:167)
        at com.orientechnologies.orient.core.engine.OMemoryAndLocalPaginatedEnginesInitializer.configureDefaults(OMemoryAndLocalPaginatedEnginesInitializer.java:77)
        at com.orientechnologies.orient.core.engine.OMemoryAndLocalPaginatedEnginesInitializer.initialize(OMemoryAndLocalPaginatedEnginesInitializer.java:62)
        at com.orientechnologies.orient.core.engine.local.OEngineLocalPaginated.startup(OEngineLocalPaginated.java:56)
        at com.orientechnologies.orient.core.Orient.startEngine(Orient.java:930)
        at com.orientechnologies.orient.core.Orient.loadStorage(Orient.java:523)
        at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.<init>(ODatabaseDocumentTx.java:173)
        at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.<init>(ODatabaseDocumentTx.java:154)
        at com.orientechnologies.orient.server.OSystemDatabase.init(OSystemDatabase.java:151)
        at com.orientechnologies.orient.server.OSystemDatabase.<init>(OSystemDatabase.java:44)
        at com.orientechnologies.orient.server.OServer.initSystemDatabase(OServer.java:1309)
        at com.orientechnologies.orient.server.OServer.activate(OServer.java:367)
        at org.sonatype.nexus.internal.orient.DatabaseServerImpl.doStart(DatabaseServerImpl.java:196)
        at org.sonatype.nexus.common.stateguard.StateGuardLifecycleSupport.start(StateGuardLifecycleSupport.java:69)
        at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
        at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:193)
        at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:57)
        at org.sonatype.nexus.internal.orient.OrientBootstrap.doStart(OrientBootstrap.java:71)
        at org.sonatype.nexus.common.stateguard.StateGuardLifecycleSupport.start(StateGuardLifecycleSupport.java:69)
        at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
        at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:193)
        at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:57)
        ... 7 common frames omitted

Enter fullscreen mode Exit fullscreen mode

The reason was, I had the following in my /etc/fstab:

UUID=a3d2ba5a-dbbd-40e9-b6d7-a4349987e6f8   /mnt/srv        ext4    defaults,nofail,user  0       2
Enter fullscreen mode Exit fullscreen mode

The user argument implies the options noexec, nosuid, and nodev. With one or more of these implied options in place, I get the above error. Removing the user argument from my mount resolved it!

Top comments (1)

Collapse
 
scsoi profile image
疏影 •

Useful when the local C: drive is filling up — and 'external storage' usually means an SMB share or an external USB drive in practice. Three patterns I keep returning to:

  • For an SMB NAS as the target: mount it as a Windows drive letter first (ScsDriver covers SMB alongside S3 / Blob / SFTP / WebDAV), then point Docker's data-root at the mounted path. The mount survives sleep / VPN and Docker sees a normal local folder.
  • For an external USB drive: NTFS-format it if you'll only use it from Windows; exFAT if you also want it on macOS. Don't use FAT32 — the 4GB file limit bites sooner than you'd think.
  • For the registry / images themselves: keep them on the local SSD. The external mount adds latency that shows up at startup time across all containers. Operational note: don't symlink the Docker folder from C: into the external mount — Docker Desktop specifically has problems with that. Use the documented data-root change instead.