Automating Cloud Storage Deployments With MinIO and SystemD

By AJ, MinIO

MinIO is one of the most widely implemented object stores because it offers high-performance, massive scalability, high-availability and adheres to the industry-standard S3 protocol. MinIO is capable of tremendous performance - a recent benchmark achieved 325 GiB/s (349 GB/s) on GETs and 165 GiB/s (177 GB/s) on PUTs with just 32 nodes of off-the-shelf NVMe SSDs. No matter where you run MinIO – bare metal, virtual instances or Kubernetes – MinIO’s performance and scalability provide an excellent foundation for such cloud-native applications as data lakes, analytics, AI/ML and more.

In addition to Kubernetes, customers run MinIO on virtual instances and bare metal, frequently relying on Supermicro hardware in the datacenter and clouds like AWS, GCP and Azure. Linux’s small footprint coupled with efficient resource utilization make it a versatile and flexible choice for running MinIO. Yet, the greater numbers of Linux machines and instances require automation to reduce administrative burden. There is a need for managing MinIO servers as an init service using SystemD because it helps to automate the service lifecycle, particularly during startup, shutdown and restart.

SystemD is a service manager that maintains services in Linux systems during boot up, shutdown, initialization and more. Think of it as the successor to init scripts where you had to write the whole thing from scratch, but with SystemD you just fill in a few parameters and the commands and logging are standardized.

One of the several challenges of using the older init.d service management was that you had to write the entire file from scratch. Each init.d file was unique and had a different way of managing the lifecycle of the service. SystemD takes the lessons learned from init.d and standardizes them. For instance, the SystemD service files will work any service as long as it's written to run – only the paths, names, and semantics are different – but the basic structure, how to restart services, reading logs, gracefully un-mounting filesystems, waiting for network connections to come up, among other things, are now common among all services, so no matter which service you run you know exactly where to look for its logs.

Not only is SystemD versatile among different services, but you can take the same SystemD service file and use it across multiple operating systems, as long as the operating system is also using SystemD to manage the lifecycle of its services. This simplifies automation as you can be working on Ubuntu while creating the initial file, but the same file can then be deployed to CentOS/RedHat operating system as long as the paths and other things remain the same, which in most cases they do.

There are several main components to SystemD but a few that you will come across more often are:

• systemctl: This is what is used to control the processes, whether to stop, start and restart mostly. The most common commands used are:

  • systemctl start
  • systemctl stop
  • systemctl restart

• journalctl: The logs for the services when they do their startup, shutdown and restart operations. It’s a good way to get insight if something is not working properly. The most common command I use, which shows logs enough to fill up your terminal window and scrolls to the very end:

  • journalctl -e -u

To run most of the commands in this guide you need to have sudo or root access because SystemD config files need to be permissioned as root.

You can also find additional instructions for MinIO and SystemD in our repository.

MinIO Binary

• Fetch the MinIO binary from upstream

# wget https://dl.min.io/server/minio/release/linux-amd64/minio


--2022-07-24 11:31:33--  https://dl.min.io/server/minio/release/linux-amd64/minio
Resolving dl.min.io (dl.min.io)... 132.68.11.115, 138.118.66.212
Connecting to dl.min.io (dl.min.io)|132.68.11.115|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 96649216 (92M) [application/octet-stream]
Saving to: 'minio'

minio                       100%[==========================================>]  92.17M  50.2MB/s    in 1.8s

2022-07-24 11:31:35 (50.2 MB/s) - 'minio' saved [96649216/96649216]

• Make it executable so SystemD can run MinIO as a service

# chmod +x minio

• Move it to a location that is executable under $PATH

# mv minio /usr/local/bin/

Service File
MinIO can be installed in a few different ways on Linux based on the operating system. RedHat/CentOS and its derivatives rely on the.rpm package and Debian/Ubuntu use the .deb package. In both cases, the following SystemD service file is included in the installation package.

However, we manually fetched the MinIO binary from upstream, so we’ll manually create the SystemD service file.

• Use your favorite text editor to create a SystemD file at /etc/systemd/system/minio.service with the following contents.

[Unit]
Description=MinIO
Documentation=https://docs.min.io
Wants=network-online.target
After=network-online.target
AssertFileIsExecutable=/usr/local/bin/minio

[Service]
WorkingDirectory=/usr/local

User=minio-user
Group=minio-user
ProtectProc=invisible

EnvironmentFile=-/etc/default/minio
ExecStartPre=/bin/bash -c "if [ -z \"${MINIO_VOLUMES}\" ]; then echo \"Variable MINIO_VOLUMES not set in /etc/default/minio\"; exit 1; fi"
ExecStart=/usr/local/bin/minio server $MINIO_OPTS $MINIO_VOLUMES

# Let systemd restart this service always
Restart=always

# Specifies the maximum file descriptor number that can be opened by this process
LimitNOFILE=65536

# Specifies the maximum number of threads this process can create
TasksMax=infinity

# Disable timeout logic and wait until process is stopped
TimeoutStopSec=infinity
SendSIGKILL=no

[Install]
WantedBy=multi-user.target

# Built for ${project.name}-${project.version} (${project.name})

• Group=minio-user: The Linux system group the Minio daemon will run as. Create it using the following command:
  • groupadd -r minio-user
• User=minio-user: The Linux system user the MinIO daemon will run as. Create the user using the following command:
  • useradd -M -r -g minio-user minio-user
    • -M: This prevents creating a home directory for the user since this is a service.
    • -r: Systems users have a separate UID/GID range for tracking purposes, this flag will create the user from the predetermined range.
    • -g : Group to add the user under.

MinIO

Distributed Setup

There are a couple of prerequisites that are required to set up on the bare metal node before MinIO service can be started.

• Create a new disk and ensure it is not on the same partition as root volume, to avoid the following message:
  • Error: Disk /mnt/disk1/minio is part of root disk, will not be used
• Create 4 directories on your local host where the new disk was just mounted, in this case /mnt/data:

mkdir -p /mnt/data/disk1 \
mkdir -p /mnt/data/disk2 \
mkdir -p /mnt/data/disk3 \
mkdir -p /mnt/data/disk4

• chown the directories with the MinIO user and group
  • chown minio-user:minio-user /mnt/data/disk1 /mnt/data/disk2 /mnt/data/disk3 /mnt/data/disk4

Environment Service Files

• Update this file /etc/default/minio with the following content

# Set the hosts and volumes MinIO uses at startup

# The command uses MinIO expansion notation {x...y} to denote a

# sequential series.

#

# The following example covers four MinIO hosts

# with 4 drives each at the specified hostname and drive locations.

# The command includes the port that each MinIO server listens on

# (default 9000)


MINIO_VOLUMES="https://minio1.example.com:9000/mnt/data/disk{1...4}/minio"


# Set all MinIO server options

#

# The following explicitly sets the MinIO Console listen address to

# port 9001 on all network interfaces. The default behavior is dynamic

# port selection.


MINIO_OPTS="--console-address :9001"


# Set the root username. This user has unrestricted permissions to

# perform S3 and administrative API operations on any resource in the

# deployment.

#

# Defer to your organizations requirements for superadmin user name.


MINIO_ROOT_USER=minioadmin


# Set the root password

#

# Use a long, random, unique string that meets your organizations

# requirements for passwords.


MINIO_ROOT_PASSWORD=minioadmin


# Set to the URL of the load balancer for the MinIO deployment

# This value *must* match across all MinIO servers. If you do

# not have a load balancer, set this value to to any *one* of the

# MinIO hosts in the deployment as a temporary measure.

MINIO_SERVER_URL="https://minio.example.net:9000"

Start the MinIO Process

• We have all the pieces we need to enable and start the service

# systemctl enable minio.service

# systemctl start minio.service

• Check the status of the service and logs

# systemctl status minio.service
# journalctl -e -u minio.service

• Verify that the MinIO service has come up. In the logs you should see something similar to this

Aug 01 13:27:06 aj-test-3 systemd[1]: Starting MinIO...

Aug 01 13:27:06 aj-test-3 systemd[1]: Started MinIO.

Aug 01 13:27:07 aj-test-3 minio[3241]: Formatting 1st pool, 1 set(s), 4 drives per set.

Aug 01 13:27:07 aj-test-3 minio[3241]: WARNING: Host minio1.example.com:9000 has more than 2 drives of set. A host fai>

Aug 01 13:27:07 aj-test-3 minio[3241]:  You are running an older version of MinIO released 4 days ago

Aug 01 13:27:07 aj-test-3 minio[3241]:  Update: Run `mc admin update`

Aug 01 13:27:07 aj-test-3 minio[3241]: MinIO Object Storage Server

Aug 01 13:27:07 aj-test-3 minio[3241]: Copyright: 2015-2022 MinIO, Inc.

Aug 01 13:27:07 aj-test-3 minio[3241]: License: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>

Aug 01 13:27:07 aj-test-3 minio[3241]: Version: RELEASE.2022-07-26T00-53-03Z (go1.18.4 linux/amd64)

Aug 01 13:27:07 aj-test-3 minio[3241]: Status:         4 Online, 0 Offline.

Aug 01 13:27:07 aj-test-3 minio[3241]: API: https://minio.example.net:9000

Aug 01 13:27:07 aj-test-3 minio[3241]: Console: http://10.128.0.4:9001 http://127.0.0.1:9001

Aug 01 13:27:07 aj-test-3 minio[3241]: Documentation: https://docs.min.io

Log in to Console

Using a browser, log in to the MinIO console using the MINIO_ROOT_USER and MINIO_ROOT_PASSWORD from the configuration we made earlier.

http://<server_host_ip>:9001/

Note the above setup is for you to quickly get up and running with MinIO. You can expand from a single node to a multi-node distributed configuration for additional testing. If you would like to deploy and configure MinIO in a production environment, please refer to the documentation.

SystemD Simplifies MinIO Virtual and Bare Metal Deployments

Integration with SystemD is very versatile.

• The syntax of the service file is the same across all services
• The same service file will work across any OS that supports SystemD
• You can use SystemD with bare metal or on VMs in any cloud such as AWS, GCP and Azure.
• This can help with automation for DevOps by standardizing the format and making it simple to automate deployments.

Automation goes a long way towards maintaining availability when running a mission critical service like object storage. Automation is a requirement for operating at scale across multiple clouds and environments. With the help of SystemD and MinIO you can automate your cloud object storage deployments and ensure the service lifecycle is managed smoothly and successfully.

Got questions? Want to get started? Reach out to us on Slack.