File exists or not, this is the question

Today I ran into an interesting issue. I downloaded a binary and when tried to run Linux surprised me: No such file or directory.

Wait! What the heck?

echo $?
127

What does it mean?

Of course, the file exists, executable and ownership is ok:

ls -la ./ignite-cntr
-rwxr-xr-x 1 mhmxs users 43785128 May 16 08:43 ./ignite-cntr

It is an ELF binary, the same as my host:

file ./ignite-cntr
./ignite-cntr: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, Go BuildID=2ysLdCmI_2jEgu-V5h1c/SP9Ydh3EJWttepF73BdP/89qBX7R3MEzeLn0NZARQ/PDW1WPguwGiACmtUAl2M, BuildID[sha1]=1954e0539099fce0e813bc5ce5017ab0e8128468, not stripped

What about dynamic dependencies:

ldd ./ignite-cntr
        linux-vdso.so.1 (0x00007ffcbd5ca000)
        libpthread.so.0 => /nix/store/qjgj2642srlbr59wwdihnn66sw97ming-glibc-2.33-123/lib/libpthread.so.0 (0x00007fc896577000)
        libdl.so.2 => /nix/store/qjgj2642srlbr59wwdihnn66sw97ming-glibc-2.33-123/lib/libdl.so.2 (0x00007fc896572000)
        libc.so.6 => /nix/store/qjgj2642srlbr59wwdihnn66sw97ming-glibc-2.33-123/lib/libc.so.6 (0x00007fc89639d000)
        /lib64/ld-linux-x86-64.so.2 => /nix/store/qjgj2642srlbr59wwdihnn66sw97ming-glibc-2.33-123/lib64/ld-linux-x86-64.so.2 (0x00007fc896599000)

Seems everything is all right. Let's try to debug the issue somehow:

strace ./ignite-cntr
execve("./ignite-cntr", ["./ignite-cntr"], 0x7ffcb65d4f30 /* 133 vars */) = -1 ENOENT (No such file or directory)
strace: exec: No such file or directory
+++ exited with 1 +++

Nothing new :)

ltrace ./ignite-cntr
Can't execute `./ignite-cntr': No such file or directory
failed to initialize process 3885: No such file or directory
couldn't open program './ignite-cntr': No such file or directory

???

gdb -q ./ignite-cntr
(gdb) run
Starting program: ./ignite-cntr
/nix/store/gqicc6kxah7f8j54hp80qxpk16898lj9-bash-interactive-5.1-p8/bin/bash: line 1: ./ignite-cntr: No such file or directory

Ok, I don't have any idea, let's search on Google:
https://github.com/NixOS/nixpkgs/issues/107375

Have you noticed I'm a NixOS user?

nix-shell -p patchelf --run "patchelf --print-interpreter ./ignite-cntr"
/lib64/ld-linux-x86-64.so.2

So NixOS has its own package manager and builds system. With this package manager, it is relatively easy to bind a binary with its dependency by git hash. This allows us to use a different versions for each binary on the same machine.

But this binary has been built outside of NixOS, and the interpreter is pointing to an interpreter which isn't exist on my machine. Instead of playing with symlinks I simply patched the downloaded ELF binary with the same patchelf command:

nix-shell -p glibc patchelf --run "patchelf --set-interpreter $(find /nix/store -name ld-linux-x86-64.so.2 | head -1) ./ignite-cntr"

And finally, it works:

./ignite-cntr
ignite-cntr is an ignite VM image creation and container application
running tool. The VM images are built to support running container applications.

Nice to know, thank you NixOS to taught me this!