YugabyteDB in a Micro VM with no network

In a previous post I mentioned how to start YugabyteDB fast for a quick short-life lab. This can run on Micro VM (like firecracker) and in some cases you have no network interfaces with it. You don't need it if you connect with ysqlsh from the MicroVM but you may encounter name resolution issues. Note that you can reproduce it with docker run --network none.

This post is about the following errors and their resolution:

ysqlsh: FATAL:  Timed out: OpenTable RPC (request call id 2) to 0.0.0.0:9100 timed out after 120.000s

or, in the yb-tserver.ERROR:

E0830 11:32:44.518383    66 async_initializer.cc:100] Failed to initialize client: Network error (yb/util/net/net_util.cc:447): Could not determine local host names: Unable to lookup FQDN (699cdfc74424), getaddrinfo returned -11 (EAI_SYSTEM): Connection refused (system error 111)

This has to do with name resolution. YugabyteDB is a distributed SQL database and must have a matching hostname and IP address to identify the nodes. The last message is explicit: YugabyteDB calls the Linux getaddrinfo() system call to get the IP of 699cdfc74424 which is my container hostname

Basically, it does the same as:

sh-4.2# getent ahosts  $(hostname)

127.0.0.2       STREAM 699cdfc74424
127.0.0.2       DGRAM
127.0.0.2       RAW
::1             STREAM
::1             DGRAM
::1             RAW

But why 127.0.0.2? I'm trying to start YugabyteDB on 127.0.0.1 which is the only interface I have on my no-network container:

/home/yugabyte/bin/yb-master --fs_data_dirs=/var/tmp --master_addresses=127.0.0.1:7100 --replication_factor=1 --default_memory_limit_to_ram_ratio=0.30 &
/home/yugabyte/bin/yb-tserver --fs_data_dirs=/var/tmp --tserver_master_addrs=127.0.0.1:7100 --default_memory_limit_to_ram_ratio=0.30 &

The reason is in /etc/nsswitch.conf:

sh-4.2# grep hosts: /etc/nsswitch.conf

#hosts:     db files nisplus nis dns
hosts:      files dns myhostname
sh-4.2#

The host name resolution goes first to files, which in this case is /etc/hosts:

sh-4.2# cat /etc/hosts

127.0.0.1       localhost
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

My hostname is unknown there.

Then it goes to dns but I've no DNS in my no-network container (reason for the Connection refused error I guess)

Then it goes to myhostname which resolves the hostname, but to 127.0.0.2. It is used to avoid patching the /etc/hosts but in my case cannot be used because 127.0.0.2 is not known.
I tried to start yb-master --master_addresses=127.0.0.2:7100 but it fails with None of the local addresses are present in master_addresses 127.0.0.2:7100

Then, my solution is simply to add the hostname to the /etc/hosts:

echo "127.0.0.1 localhost $(hostname)"  > /etc/hosts

With this before starting yb-master--fs_data_dirs=/var/tmp --master_addresses=127.0.0.1:7100 --replication_factor=1 and yb-tserver --fs_data_dirs=/var/tmp --master_addresses=127.0.0.1:7100 --replication_factor=1 you should be able to connect with a simple ysqlsh which connects by default to localhost

Of course, a distributed SQL database with no network makes no sense except for a small container used for continuous integration for example.