DEV Community

murat cabuk
murat cabuk

Posted on

Setting Up Ceph Storage the Hard Way

#ceph #storage #linux #devops

Hello friends,

In this article, we'll try to install Ceph, one of Red Hat's open source distributed storage solutions, without using any deployment tool, entirely as described in its documentation.

I recommend doing this kind of installation only for educational purposes. Our goal is to try to understand the installations and configurations done during the installation of Ceph. Trying to set up and manage real production environments this way would be very exhausting. Also, since it'll be an installation that's very prone to mistakes, it makes more sense to use deployment tools.

Our prerequisite is knowing a bit of Vagrant and Linux.

Happy reading, I hope it'll be useful.

Setting Up the Lab Environment

We'll use 3 machines. The specs and the ceph components to be installed will be as follows

  • Ubuntu 20.04, an extra disk at /dev/sdb
  • The Ceph MON, MGR,MDS, OSD components will be installed on all three machines.

You can find both methods for setting up the lab environment with Vagrant below.

  • libvirt (kvm-qemu): Can only run on Linux
  • Virtualbox: Can run on Mac, Windows and Linux

Setting Up the Vagrant Lab Environment with Libvirt

If you don't want to deal with too much, prefer the Vagrant-VirtualBox installation below.

  • we install the kvm-qemu packages 
sudo apt update && sudo apt install -y qemu qemu-kvm libvirt-daemon bridge-utils virt-manager virtinst libvirt-clients

sudo systemctl status libvirtd

sudo systemctl enable --now libvirtd
Enter fullscreen mode Exit fullscreen mode

We install Vagrant.

curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
sudo apt-get update && sudo apt-get install vagrant
Enter fullscreen mode Exit fullscreen mode

Then we install the vagrant-libvirt plugin.

sudo apt-get install libxslt-dev libxml2-dev libvirt-dev zlib1g-dev ruby-dev

vagrant plugin install vagrant-libvirt
Enter fullscreen mode Exit fullscreen mode

WARNING: If you get an error about not being able to access the https://rubygems.org page while installing the plugin, apply the steps below. For details, visit this page.

Open the /etc/gai.conf file and add the line "precedence 2a04:4e42::0/32 5" at the very bottom, removing the quotes. Then restart the network service.

WARNING: if you get an error saying the st.h file doesn't exist, run the command below. Pay attention to your version number.

sudo ln -s /opt/vagrant/embedded/include/ruby-3.0.0/ruby/st.h /opt/vagrant/embedded/include/ruby-3.0.0/st.h
Enter fullscreen mode Exit fullscreen mode

WARNING: If you keep getting errors with the command above in the Vagrant-libvirt installation, you can also try installing it with the command below.

sudo apt install ruby-libvirt
Enter fullscreen mode Exit fullscreen mode

Now we're ready to bring up our lab environment. We run the system with the command below. Downloading the first box will take a while.

mkdir -p ~/ceph/vagrant && cd ~/ceph/vagrant

cat <<EOF | Vagrantfile

# -*- mode: ruby -*-
# vi: set ft=ruby :


# if you want to install centops, it's enough to write generic/centos8 instead of generic/ubuntu2004

# https://github.com/vagrant-libvirt/vagrant-libvirt
ENV['VAGRANT_DEFAULT_PROVIDER'] = 'libvirt'

Vagrant.configure("2") do |config|

# node1
config.vm.define "node1" do |node1|
    node1.vm.box = "generic/ubuntu2004"
    node1.vm.hostname="node1"
    node1.vm.network "private_network", ip: "10.200.10.10"

    node1.vm.provider :libvirt do |lv|
       lv.memory=2048
       lv.cpus=1
       #lv.host = "node1"
       lv.storage :file, :size => '10G'
# the disk will be added to the machine as /dev/vdb
    end 
 end


# node2
config.vm.define "node2" do |node2|
    node2.vm.box = "generic/ubuntu2004"
    node2.vm.hostname="node2"
    node2.vm.network "private_network", ip: "10.200.10.11"

    node2.vm.provider :libvirt do |lv|
       lv.memory=2048
       lv.cpus=1
       #lv.host = "node2"
       lv.storage :file, :size => '10G'
# the disk will be added to the machine as /dev/vdb
    end
 end


# node3
config.vm.define "node3" do |node3|
    node3.vm.box = "generic/ubuntu2004"
    node3.vm.hostname="node3"
    node3.vm.network "private_network", ip: "10.200.10.12"

    node3.vm.provider :libvirt do |lv|
       lv.memory=2048
       lv.cpus=1
       #lv.host = "node3"
       lv.storage :file, :size => '10G'
# the disk will be added to the machine as /dev/vdb
    end 
 end

end


EOF

vagrant up
Enter fullscreen mode Exit fullscreen mode

Setting Up the Vagrant Lab Environment with Virtualbox

After installing VirtualBox and then Vagrant on our machine, we create a file named Vagrantfile in a folder we opened on our computer and copy the lines below into it.

# -*- mode: ruby -*-
# vi: set ft=ruby :


# if you want to install centops, it's enough to write generic/centos8 instead of generic/ubuntu2004


Vagrant.configure("2") do |config|

# node1
config.vm.define "node1" do |node1|
    node1.vm.box = "generic/ubuntu2004"
    node1.vm.hostname="node1"
    node1.vm.network "private_network", ip: "10.200.10.10"

    node1.vm.provider :virtualbox do |lv|
       lv.memory=2048
       lv.cpus=1
       #lv.host = "node1"
       lv.storage :file, :size => '10G'
# the disk will be added to the machine as /dev/vdb
    end 
 end


# node2
config.vm.define "node2" do |node2|
    node2.vm.box = "generic/ubuntu2004"
    node2.vm.hostname="node2"
    node2.vm.network "private_network", ip: "10.200.10.11"

    node2.vm.provider :virtualbox do |lv|
       lv.memory=2048
       lv.cpus=1
       #lv.host = "node2"
       lv.storage :file, :size => '10G'
# the disk will be added to the machine as /dev/vdb
    end
 end


# node3
config.vm.define "node3" do |node3|
    node3.vm.box = "generic/ubuntu2004"
    node3.vm.hostname="node3"
    node3.vm.network "private_network", ip: "10.200.10.12"

    node3.vm.provider :virtualbox do |lv|
       lv.memory=2048
       lv.cpus=1
       #lv.host = "node3"
       lv.storage :file, :size => '10G'
# the disk will be added to the machine as /dev/vdb
    end 
 end

end
Enter fullscreen mode Exit fullscreen mode

then, on the terminal or command-com, while in the directory where the Vagrantfile is located, we run the vagrant up command

After doing the installation with either method, we connect to our machines with ssh

  • Username: vagrant
  • Password: vagrant

while our terminal is still in the same directory as the Vagrantfile, we can connect with the command below.

vagrant ssh node1
Enter fullscreen mode Exit fullscreen mode

When we connect to all the machines this way and ping the (10.200.10.10,11 and 12) ips, we should get a response.

We're Starting the Ceph Installation

We'll run some commands on all machines, so being able to run commands in more than one terminal at the same time will save time. One of the best known is Tmux; you can click the link and examine it in detail.

1. First, we need to make sure the NTP service is running on all machines.

For this, we run the timedatactl command.

timedatectl

# result
               Local time: Wed 2021-07-14 16:20:38 UTC
           Universal time: Wed 2021-07-14 16:20:38 UTC
                 RTC time: Wed 2021-07-14 16:20:39    
                Time zone: UTC (UTC, +0000)           
System clock synchronized: yes                        
              NTP service: active                     
          RTC in local TZ: no
Enter fullscreen mode Exit fullscreen mode

In the end, as above, we need to be able to see that the NTP service is active and synced. If you brought up Vagrant with Ubuntu 20.04, it's installed, but if you still have problems, you can install it by following this link.

2. We install Ceph on all machines

wget -q -O- 'https://download.ceph.com/keys/release.asc' | sudo apt-key add -

release=lsb_release -sc

sudo apt-add-repository 'deb https://download.ceph.com/debian-octopus/ focal main'

sudo apt update && sudo apt upgrade

sudo apt install ceph
Enter fullscreen mode Exit fullscreen mode

We'll need about 1 GB of space for the installation.

When the installation is finished, we do a version check.

ceph -v

# as a result, the octopus version should have been installed

ceph version 15.2.13 (c44bc49e7a57a87d84dfff2a077a2058aa2172e2) octopus (stable)
Enter fullscreen mode Exit fullscreen mode

3. We edit the hosts file of all machines

we add the lines below

sudo vim /etc/hosts

# we add the lines below

10.200.10.10 node1
10.200.10.11 node2
10.200.10.12 node3
Enter fullscreen mode Exit fullscreen mode

4. We change the password of the root user

We change the root password on all machines

sudo passwd root
# we changed it to vagrant
Enter fullscreen mode Exit fullscreen mode

MON (Monitor) Installation

Before starting the installation, let's examine the Monitor components below.

  • Unique Identifier: the fsid is a unique identifier for the cluster.
  • Cluster Name: It's a descriptive name with no spaces for the Ceph cluster. The default cluster name is ceph, but a different name can also be specified. For example, when you run more than one cluster in a multi-region configuration, the cluster name (e.g. Us-west, us-east) identifies the cluster for the current CLI session. Note: To identify the cluster name on the command line interface, specify the Ceph configuration file with the cluster name (e.g. Ceph.conf, us-west.conf, us-east.conf, etc.). Also see the ceph --cluster {cluster-name} command for CLI usage.
  • Monitor Name (mon): Each monitor instance within a cluster has a unique name. In common practice, the CephMonitor name is the host name (one Ceph Monitor per host is recommended. It's also recommended not to mix Ceph Monitors with Ceph OSDDaemons). You can use the hostname -s command to get the host's short name.
  • Monitor Map: To bootstrap the monitors, you need to create a monitor map. The monitor map requires the fsid, the cluster name (or uses the default) and at least one host name and IPaddress.
  • Monitor Keyring: Monitors communicate with each other via a secret key (keyring). You should create a keyring with the monitor secret and use it in the initial bootstrap of the monitors.
  • Administrator Keyring: to use the ceph CLI tools, you need to have a client.admin user. So you need to create the admin user and keyring and also add the client.admin user to the monitor keyring.

1. We create the ceph.conf file on all machines

We'll create the cluster id only on our node1 machine and use it on the others too.

we run the command below only on the node1 machine.


uuidgen

# in the end, a GUID like the one below was created.

7fa60b7c-9d9c-46c2-a57b-e537ab26a58f
Enter fullscreen mode Exit fullscreen mode

we add this code, our monitor nodes and all the necessary settings to the ceph.conf files on all machines

cat <<EOF | sudo tee -a /etc/ceph/ceph.conf
[global]
fsid = 7fa60b7c-9d9c-46c2-a57b-e537ab26a58f
mon initial members = node1, node2, node3
mon host = 10.200.10.10, 10.200.10.11, 10.200.10.12
public_network = 10.200.10.0/24

auth cluster required = none # if auth is wanted, cephx should be written
auth service required = none # if auth is wanted, cephx should be written
auth client required = none # if auth is wanted, cephx should be written
osd journal size = 1024
osd pool default size = 3
osd pool default min size = 2
osd pool default pg num = 50
osd pool default pgp num = 50
osd crush chooseleaf type = 1
EOF
Enter fullscreen mode Exit fullscreen mode

2. We create the Keyrings

We continue from node1 again; when we're done here, we'll distribute what we did on node1 to the others too. We create the Secret Keys.

# monitor secret key.
sudo ceph-authtool --create-keyring /tmp/ceph.mon.keyring --gen-key -n mon. --cap mon 'allow *'

# we create the adminsitrator keyring, create the client.admin user and add the user to the keyring
sudo ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *'

# we create the bootstrap-osd keyring, create the client.bootstrap-osd user and add the user to the keyring.
sudo ceph-authtool --create-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring --gen-key -n client.bootstrap-osd --cap mon 'profile bootstrap-osd' --cap mgr 'allow r'

# we add all the keys we created to ceph.mon.keyring

sudo ceph-authtool /tmp/ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyring
sudo ceph-authtool /tmp/ceph.mon.keyring --import-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring
Enter fullscreen mode Exit fullscreen mode

3. We Create the Monitor Maps

We're still continuing from the node1 machine.

monmaptool --create --add node1 10.200.10.10 --add node2 10.200.10.11 --add node3 10.200.10.12  --fsid 7fa60b7c-9d9c-46c2-a57b-e537ab26a58f /tmp/monmap

# Command result

monmaptool: monmap file /tmp/monmap
monmaptool: set fsid to 7fa60b7c-9d9c-46c2-a57b-e537ab26a58f
monmaptool: writing epoch 0 to /tmp/monmap (3 monitors)
Enter fullscreen mode Exit fullscreen mode

4. We move the keys and monitor map files we created to the other servers

while on the node1 machine

If you get an error while running the commands below, run them as root with the sudo su command.

scp /tmp/monmap root@node2:/tmp/
scp /tmp/monmap root@node3:/tmp/


scp /etc/ceph/ceph.client.admin.keyring root@node2:/etc/ceph/
scp /etc/ceph/ceph.client.admin.keyring root@node3:/etc/ceph/

scp /tmp/ceph.mon.keyring root@node2:/tmp/
scp /tmp/ceph.mon.keyring root@node3:/tmp/
Enter fullscreen mode Exit fullscreen mode

we'll use the files later.

5. We create the data folder for the Monitor and start the Monitors

We run this command on all machines. But on each machine we should do it using its own hostname.

For example, for the node1 machine

sudo  mkdir /var/lib/ceph/mon/ceph-node1

sudo  ceph-mon --mkfs -i node1 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring

sudo ls /var/lib/ceph/mon/ceph-node1

# result
kv_backend  store.db
Enter fullscreen mode Exit fullscreen mode

for node2

sudo  mkdir /var/lib/ceph/mon/ceph-node2
sudo  ceph-mon --mkfs -i node2 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring

sudo ls /var/lib/ceph/mon/ceph-node2

# result
kv_backend  store.db
Enter fullscreen mode Exit fullscreen mode

for node3

sudo  mkdir /var/lib/ceph/mon/ceph-node3
sudo  ceph-mon --mkfs -i node3 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring

sudo ls /var/lib/ceph/mon/ceph-node3

# result
kv_backend  store.db
Enter fullscreen mode Exit fullscreen mode

We change the ownership of the files and directories.

sudo chown -R ceph:ceph /var/lib/ceph/mon
sudo chown -R ceph:ceph /var/log/ceph
sudo chown -R ceph:ceph /var/run/ceph
sudo chown ceph:ceph /etc/ceph/ceph.client.admin.keyring
sudo chown ceph:ceph /etc/ceph/ceph.conf
sudo chown ceph:ceph /etc/ceph/rbdmap
Enter fullscreen mode Exit fullscreen mode

We restart the Monitor on all machines.

for node1

sudo systemctl start ceph-mon@node1
sudo systemctl status ceph-mon@node1
Enter fullscreen mode Exit fullscreen mode

for node2

sudo systemctl start ceph-mon@node2
sudo systemctl status ceph-mon@node2
Enter fullscreen mode Exit fullscreen mode

for node3

sudo systemctl start ceph-mon@node3
sudo systemctl status ceph-mon@node3
Enter fullscreen mode Exit fullscreen mode

After seeing that the monitor is running on all machines, we continue with the Manager installation.

MGR (Manager) Installation

We should run it separately on all machines. The example for node1 is below; on the others, instead of node1, we should write the hostname of whichever machine we'll run it on (node,node3).

for node1

sudo -u ceph mkdir /var/lib/ceph/mgr/ceph-node1
sudo ceph auth get-or-create mgr.node1 mon 'allow profile mgr' osd 'allow *' mds 'allow *' -o /var/lib/ceph/mgr/ceph-node1/keyring
sudo chown -R ceph:ceph /var/lib/ceph/mgr
sudo systemctl enable ceph-mgr.target
sudo systemctl enable ceph-mgr@node1
sudo systemctl restart ceph-mgr@node1
ceph mon enable-msgr2
Enter fullscreen mode Exit fullscreen mode

for node2

sudo -u ceph mkdir /var/lib/ceph/mgr/ceph-node2
sudo ceph auth get-or-create mgr.node2 mon 'allow profile mgr' osd 'allow *' mds 'allow *' -o /var/lib/ceph/mgr/ceph-node2/keyring
sudo chown -R ceph:ceph /var/lib/ceph/mgr
sudo systemctl enable ceph-mgr.target
sudo systemctl enable ceph-mgr@node2
sudo systemctl restart ceph-mgr@node2
ceph mon enable-msgr2
Enter fullscreen mode Exit fullscreen mode

for node3

sudo -u ceph mkdir /var/lib/ceph/mgr/ceph-node3
sudo ceph auth get-or-create mgr.node3 mon 'allow profile mgr' osd 'allow *' mds 'allow *' -o /var/lib/ceph/mgr/ceph-node3/keyring
sudo chown -R ceph:ceph /var/lib/ceph/mgr
sudo systemctl enable ceph-mgr.target
sudo systemctl enable ceph-mgr@node3
sudo systemctl restart ceph-mgr@node3
ceph mon enable-msgr2
Enter fullscreen mode Exit fullscreen mode

we check the system by running the command below on any of the nodes

ceph -s

# as a result, in the services section we can see that node1 is active and the other two are waiting in standby.

  cluster:
    id:     7fa60b7c-9d9c-46c2-a57b-e537ab26a58f
    health: HEALTH_WARN
            mons are allowing insecure global_id reclaim
            3 monitors have not enabled msgr2
            OSD count 0 < osd_pool_default_size 3

  services:
    mon: 3 daemons, quorum node1,node2,node3 (age 80m)
    mgr: node1(active, starting, since 1.35779s), standbys: node2, node3
    osd: 0 osds: 0 up, 0 in

  data:
    pools:   1 pools, 1 pgs
    objects: 0 objects, 0 B
    usage:   0 B used, 0 B / 0 B avail
    pgs:     100.000% pgs unknown
             1 unknown
Enter fullscreen mode Exit fullscreen mode

By the way, it's possible to install many modules on the manager. for example, for the web ui dashboard, the link below can be checked.

We can activate the dashboard by running the commands below on node1.

ceph mgr module enable dashboard
ceph mgr module ls


# result

   1 {
   2     "always_on_modules": [
   3         "balancer",
   4         "crash",
   5         "devicehealth",
   6         "orchestrator",
   7         "pg_autoscaler",
   8         "progress",
   9         "rbd_support",
  10         "status",
  11         "telemetry",
  12         "volumes"
  13     ],
  14     "enabled_modules": [
  15         "dashboard",
  16         "iostat",
  17         "restful"
  18     ],
  19     "disabled_modules": [
  20     ...
Enter fullscreen mode Exit fullscreen mode

If the dashboard settings are also made before the monitors start, there won't be problems on restarts. The setting below can be added to the /etc/ceph/ceph.conf file.

[mon]
mgr initial modules = dashboard balancer
Enter fullscreen mode Exit fullscreen mode

There are many more modules.

  • Dashboard module
  • Ceph RESTful API
  • Alerts module
  • DiskPrediction module
  • Local pool module
  • RESTful module
  • Zabbix module
  • Prometheus module
  • Influx module
  • Hello module
  • Telegraf module
  • Telemetry module
  • Iostat module
  • Crash module
  • Insights module
  • Orchestrator module
  • Rook module

OSD (Object Storage Daemon) Installation

  • First, we copy the ceph.keyring file under the bootstrap-osd folder on node1 to the other nodes.

If there's a problem in this step, this link should be checked. the keyring may not exist and may need to be recreated. see item 4 in the link.

while on node1

sudo scp /var/lib/ceph/bootstrap-osd/ceph.keyring root@node2:/var/lib/ceph/bootstrap-osd/
sudo scp /var/lib/ceph/bootstrap-osd/ceph.keyring root@node3:/var/lib/ceph/bootstrap-osd/
Enter fullscreen mode Exit fullscreen mode
  • Then we set up the extra disk attached to our machines (nodes) as an OSD.

For this, on all nodes we get the list of our disks with the fdisk -l command.

Most likely the one that's /dev/vdb is the extra disk attached later.

We create the OSD. While on Node1, we run the command below. We do the same on all nodes.

sudo ceph-volume lvm create --data /dev/vdb
Enter fullscreen mode Exit fullscreen mode

When the commands are run, we can see that the OSDs are given ids like 0,1.2 etc in order.

If you mix up these ids, you'll be running a service that isn't on the machine, and naturally it won't work. to run the correct id on the correct machine, when we check the sudo ls /var/lib/ceph/osd directory on each machine, we'll see one of the folders named ceph-0, ceph-1, ceph-2. For example, if the machine we check has the ceph-1 folder, then we'll run the sudo systemctl restart ceph-osd@1 service on that machine.

On all nodes, we restart the OSD with whichever OSD id it has.

for example, on node1 the OSD id was 0

sudo systemctl restart ceph-osd@0
Enter fullscreen mode Exit fullscreen mode

for node 2

sudo systemctl restart ceph-osd@1
Enter fullscreen mode Exit fullscreen mode

for node 3

sudo systemctl restart ceph-osd@2
Enter fullscreen mode Exit fullscreen mode
  • the volume list can be retrieved. it should be run separately on each machine

for example, the osd list on node1 in my cluster

sudo ceph-volume lvm list

====== osd.2 =======

  [block]       /dev/ceph-98012ae9-ce95-436d-b2f9-b109ce86e4c9/osd-block-13243528-50a7-4ca7-9181-15e8df77573f

      block device              /dev/ceph-98012ae9-ce95-436d-b2f9-b109ce86e4c9/osd-block-13243528-50a7-4ca7-9181-15e8df77573f
      block uuid                Mvi6eR-7O5X-kYjw-uoYT-duMq-p1rT-JHFJtM
      cephx lockbox secret      
      cluster fsid              7fa60b7c-9d9c-46c2-a57b-e537ab26a58f
      cluster name              ceph
      crush device class        None
      encrypted                 0
      osd fsid                  13243528-50a7-4ca7-9181-15e8df77573f
      osd id                    2
      osdspec affinity          
      type                      block
      vdo                       0
      devices                   /dev/vdb
Enter fullscreen mode Exit fullscreen mode
  • let's also look at the state of our cluster 
ceph -s

  cluster:
    id:     7fa60b7c-9d9c-46c2-a57b-e537ab26a58f
    health: HEALTH_WARN
            mons are allowing insecure global_id reclaim

  services:
    mon: 3 daemons, quorum node1,node2,node3 (age 2h)
    mgr: node1(active, since 2h), standbys: node2, node3
    osd: 3 osds: 3 up (since 13m), 3 in (since 3h)

  data:
    pools:   1 pools, 1 pgs
    objects: 0 objects, 0 B
    usage:   6.4 GiB used, 24 GiB / 30 GiB avail
    pgs:     1 active+clean
Enter fullscreen mode Exit fullscreen mode

MDS (Metadata Server) Installation

Source: https://docs.ceph.com/en/latest/install/manual-deployment/#adding-mds

  • The commands below are run on all 3 nodes.

node1 example

sudo -u ceph mkdir -p /var/lib/ceph/mds/ceph-node1
sudo ceph-authtool --create-keyring /var/lib/ceph/mds/ceph-node1/keyring --gen-key -n mds.node1
sudo ceph auth add mds.node1 osd "allow rwx" mds "allow" mon "allow profile mds" -i /var/lib/ceph/mds/ceph-node1/keyring
sudo chown -R ceph:ceph /var/lib/ceph/mds
sudo systemctl restart ceph-mds@node1
Enter fullscreen mode Exit fullscreen mode

node2 example

sudo -u ceph mkdir -p /var/lib/ceph/mds/ceph-node2
sudo ceph-authtool --create-keyring /var/lib/ceph/mds/ceph-node2/keyring --gen-key -n mds.node2
sudo ceph auth add mds.node2 osd "allow rwx" mds "allow" mon "allow profile mds" -i /var/lib/ceph/mds/ceph-node2/keyring
sudo chown -R ceph:ceph /var/lib/ceph/mds
sudo systemctl restart ceph-mds@node2
Enter fullscreen mode Exit fullscreen mode

node3 example

sudo -u ceph mkdir -p /var/lib/ceph/mds/ceph-node3
sudo ceph-authtool --create-keyring /var/lib/ceph/mds/ceph-node3/keyring --gen-key -n mds.node3
sudo ceph auth add mds.node3 osd "allow rwx" mds "allow" mon "allow profile mds" -i /var/lib/ceph/mds/ceph-node3/keyring
sudo chown -R ceph:ceph /var/lib/ceph/mds
sudo systemctl restart ceph-mds@node3
Enter fullscreen mode Exit fullscreen mode
  • the lines below are added to the end of the /etc/ceph/ceph.conf file on all nodes. 
[mds.node1]
host = node1

[mds.node2]
host = node2

[mds.node3]
host = node3
Enter fullscreen mode Exit fullscreen mode
  • then all services are restarted from scratch on all nodes.

node1 example (osd 2 is on the node1 machine for me)

sudo systemctl restart ceph-mon@node1
sudo systemctl restart ceph-mgr@node1
sudo systemctl restart ceph-mds@node1
sudo systemctl restart ceph-osd@2
Enter fullscreen mode Exit fullscreen mode

node2 example

sudo systemctl restart ceph-mon@node2
sudo systemctl restart ceph-mgr@node2
sudo systemctl restart ceph-mds@node2
sudo systemctl restart ceph-osd@1
Enter fullscreen mode Exit fullscreen mode

node3 example (osd 0 is on the node3 machine for me)

sudo systemctl restart ceph-mon@node3
sudo systemctl restart ceph-mgr@node3
sudo systemctl restart ceph-mds@node3
sudo systemctl restart ceph-osd@0
Enter fullscreen mode Exit fullscreen mode
  • then the cluster is checked 
ceph mds stat

#result
 3 up:standby
Enter fullscreen mode Exit fullscreen mode

another check

ceph -s

  cluster:
    id:     7fa60b7c-9d9c-46c2-a57b-e537ab26a58f
    health: HEALTH_WARN
            mons are allowing insecure global_id reclaim

  services:
    mon: 3 daemons, quorum node1,node2,node3 (age 18s)
    mgr: node1(active, since 17s), standbys: node3, node2
    mds:  3 up:standby
    osd: 3 osds: 3 up (since 16s), 3 in (since 5h)

  task status:

  data:
    pools:   1 pools, 1 pgs
    objects: 0 objects, 0 B
    usage:   6.3 GiB used, 24 GiB / 30 GiB avail
    pgs:     1 active+clean
Enter fullscreen mode Exit fullscreen mode

final check for the OSDs

ceph osd tree


ID  CLASS  WEIGHT   TYPE NAME       STATUS  REWEIGHT  PRI-AFF
-1         0.02939  root default                             
-3         0.00980      host node1                           
 2    hdd  0.00980          osd.2       up   1.00000  1.00000
-5         0.00980      host node2                           
 1    hdd  0.00980          osd.1       up   1.00000  1.00000
-7         0.00980      host node3                           
 0    hdd  0.00980          osd.0       up   1.00000  1.00000
Enter fullscreen mode Exit fullscreen mode

up to here, we've built the bottom layer in the image

ceph.png

now object (amazon s3), block (disk) and file storage (nfs, smb) can be installed on top of this cluster

stack.png

Services That Need to Be Running

  • services that need to run with the same name on all machines.

The first of these services, that is, ceph.target, makes it possible to start and stop the target services below all together. For the differences between a target and a service, you can take a look at the resources below

simply put, while the system boots up, it runs the services expected to run at certain levels at the related level and grouped together. We can also write our own target file. For example, for 2 or more services that need to run together at the same time, we can specify which one will run first and which after, and which target will kick in in case of a conflict.

sudo systemctl enable ceph.target
sudo systemctl enable ceph-mgr.target
sudo systemctl enable ceph-mon.target
sudo systemctl enable ceph-osd.target

sudo systemctl enable ceph-radosgw.target
Enter fullscreen mode Exit fullscreen mode

If we look at all the services and targets running on node1, we can read from their descriptions what the targets do.

systemctl  | grep  ceph

  var-lib-ceph-osd-ceph\x2d2.mount   loaded active mounted   /var/lib/ceph/osd/ceph-2                                                     
  ceph-crash.service                 loaded active running   Ceph crash dump collector                                                    
  ceph-mds@node1.service             loaded active running   Ceph metadata server daemon                                                  
  ceph-mgr@node1.service             loaded active running   Ceph cluster manager daemon                                                  
  ceph-mon@node1.service             loaded active running   Ceph cluster monitor daemon                                                  
  ceph-osd@2.service                 loaded active running   Ceph object storage daemon osd.2                                             
  system-ceph\x2dmds.slice           loaded active active    system-ceph\x2dmds.slice                                                     
  system-ceph\x2dmgr.slice           loaded active active    system-ceph\x2dmgr.slice                                                     
  system-ceph\x2dmon.slice           loaded active active    system-ceph\x2dmon.slice                                                     
  system-ceph\x2dosd.slice           loaded active active    system-ceph\x2dosd.slice                                                     
  system-ceph\x2dvolume.slice        loaded active active    system-ceph\x2dvolume.slice                                                  
  ceph-fuse.target                   loaded active active    ceph target allowing to start/stop all ceph-fuse@.service instances at once  
  ceph-mds.target                    loaded active active    ceph target allowing to start/stop all ceph-mds@.service instances at once   
  ceph-mgr.target                    loaded active active    ceph target allowing to start/stop all ceph-mgr@.service instances at once   
  ceph-mon.target                    loaded active active    ceph target allowing to start/stop all ceph-mon@.service instances at once   
  ceph-osd.target                    loaded active active    ceph target allowing to start/stop all ceph-osd@.service instances at once   
  ceph.target                        loaded active active    ceph target allowing to start/stop all ceph*@.service instances at once
Enter fullscreen mode Exit fullscreen mode

If we examine ceph-mgr.target, it can be understood a bit better.


cat /lib/systemd/system/ceph-mgr.target

[Unit]
Description=ceph target allowing to start/stop all ceph-mgr@.service instances at once
PartOf=ceph.target
After=ceph-mon.target
Before=ceph.target
Wants=ceph.target ceph-mon.target

[Install]
WantedBy=multi-user.target ceph.target
Enter fullscreen mode Exit fullscreen mode
  • each of the lines below should run on a different machine. so how will we know which service will run on which machine?

When we check the sudo ls /var/lib/ceph/osd directory on each machine, we'll see one of the folders named ceph-0, ceph-1, ceph-2. For example, if the machine we check has the ceph-1 folder, then we'll run the sudo systemctl restart ceph-osd@1 service on that machine.

sudo systemctl start ceph-osd@0
sudo systemctl start ceph-osd@1
sudo systemctl start ceph-osd@2
Enter fullscreen mode Exit fullscreen mode
  • Services to run on each machine according to its hostname 
sudo systemctl enable ceph-mgr@node1
sudo systemctl start ceph-mgr@node1

sudo systemctl enable ceph-mgr@node2
sudo systemctl start ceph-mgr@node2

sudo systemctl enable ceph-mgr@node3
sudo systemctl start ceph-mgr@node3
Enter fullscreen mode Exit fullscreen mode
  • Services to run on each machine according to its hostname 
sudo systemctl enable ceph-mon@node1
sudo systemctl start ceph-mon@node1

sudo systemctl enable ceph-mon@node2
sudo systemctl start ceph-mon@node2

sudo systemctl enable ceph-mon@node3
sudo systemctl start ceph-mon@node3
Enter fullscreen mode Exit fullscreen mode
  • Services to run on each machine according to its hostname 
sudo systemctl restart ceph-mds@node1
sudo systemctl restart ceph-mds@node2
sudo systemctl restart ceph-mds@node3
Enter fullscreen mode Exit fullscreen mode

Troubleshooting

  • dashboard-related errors
    1. make sure the modules are installed. They should be among the Available or always on modules 
  ceph mgr module ls
Enter fullscreen mode Exit fullscreen mode
  1. make sure the services are running 
  sudo systemctl status ceph-mgr@node1
Enter fullscreen mode Exit fullscreen mode

in the end, if there's still an error, it'll show up.

  1. dashboard-related commands 

  # ceph mgr module disable dashboard

  #  dashboard
  ceph config set mgr mgr/dashboard/server_addr 0.0.0.0
  ceph config set mgr mgr/dashboard/server_port 9000 # 443 or 8443 is used for ssl

  #  dashboard 
  # ceph dashboard create-self-signed-cert # to create a certificate. if we want to use our own certificate, the documentation should be checked


  #  dashboard  ssl
  ceph config set mgr mgr/dashboard/ssl false # it should be true for ssl

  #  dashboard 
  ceph mgr module enable dashboard # it already needs to be enabled to be able to run the commands above, that is, for ssl, ip and port

  #  dashboard 
  ceph dashboard set-login-credentials username_gelmeli -i sifrenin_oldugu_dosya_adresi


  #  mgr - if everything is fine, the dashbord should appear under the mgr services.
  ceph mgr services
  echo :
  {
    "dashboard": "https://0.0.0.0:9000/"
  }
Enter fullscreen mode Exit fullscreen mode

By the way, we also need to pay attention to the following: all the machines should have a dns record on our own host machine, that is, for this example, we should make the hosts file settings we specified at the beginning on our host machine too.

sudo vim /etc/hosts

# we add the lines below

10.200.10.10 node1
10.200.10.11 node2
10.200.10.12 node3
Enter fullscreen mode Exit fullscreen mode

Dashboard

when we go to the https://10.200.10.10:8443 address in the browser, we'll see a page like the one below.

db1.png

CRUSH map

db2.png

Resources

Top comments (0)