Daemons: From Physics (?) to Linux System Control

If there's one set of concepts and stories I've always liked, it's those related to physics, mythology, and computing. While going about my business on the interwebs and in my daily work, I once stumbled upon DAEMONS. Well, the name immediately grabs your attention, and in my opinion, the concept do too. So, let's take a closer look.

In 1871 (although he had cited it earlier in 1867), James Clerk Maxwell, to explain the second law of thermodynamics, proposed a thought experiment to refute it. He imagined a finite being, capable of manipulating molecules in a closed system, who observed and made decisions about opening or closing a door to allow the passage of fast molecules—basically, a being he considered impossible to exist. Later, Lord Kelvin called this being a "daemon," not in the negative sense (he wasn't talking about a kid's backpack, 7-skins, or cold synthetic... This are brazilian ways to call the devil), but referring to a supernatural creature that works behind the scenes, the concept of the daemon from Greek mythology: invisible beings to humans, carrying out important tasks.

This "daemon" concept is often associated with computing, although the link might not even exist, as, in the context of computers, the term has taken on its own specific technical meaning (but let's suppose the link exists; it makes everything cooler).

But why would there be this connection?

Well, the general idea is the same, but no longer as a thought experiment, but as something virtual. In operating systems, we have to keep the complex system under constant observation and action, without direct user intervention—that is, a true Greek daemon.

And voilà, there we have it: Linux Daemons. They remain in a state of slumber (they aren't actually off, but I'll explain this better later), being executed by a trigger, at regular intervals (usually seconds or milliseconds), or continuously processing queues. They exist for all types and functions, but the constant is: they operate autonomously, without depending on user action, maintaining the operation of the operating system or managing functions that may eventually be used by interactive processes.

One important example of a daemon management system, widely adopted in many modern distributions, is systemd. It functions as a central manager: it starts and controls many other services, analyzing .service files to determine the execution order, the location of the process binary, and how to manage its inputs and outputs.

In practice, daemons are the foundation of Linux, operating at various levels, from functions close to the hardware to user applications. Almost everything we use or see in our workflow has a daemon behind it. And the best part is, we can create our own daemons to execute tasks that don't require our direct attention, running in the background.

Today, in many modern systems, systemd is a widely used daemon manager, although alternatives like OpenRC, runit, and s6 have their own merits and proponents. With systemd, we can, for example, create a service that monitors hardware temperature and automatically generates logs, or run "one-shot" applications under its control, managing their execution, status, and eventual errors.

Of course, there are many other daemons—as I said, they are everywhere, even where we might not expect (?). Take the Apache HTTP Server (httpd), a popular web server that operates as a daemon (I didn't know until I looked into it). There's also crond, responsible for scheduled tasks present in cronjobs, and this one has an interesting point: if we think about it directly, it executes tasks we create and ask to run at certain times or at regular intervals, and when they're not running, they are idle—very similar to daemons themselves!

So, are my schedulers daemons too? The line can be thin, as they share characteristics. The main difference is that many daemons are not actually "dormant"; they are actually assuming different "states." (I said I'd mention this again)

• Interruptible Sleep (S): In this state, they are actively executing, monitoring events, interfaces, or ports. For example, sshd is always waiting for connections on port 22, or cupsd is waiting for someone to send something for printing.
• Uninterruptible Sleep (D): In this state, it is waiting for an I/O operation to complete. For example, during a large file copy, rsync might appear in state D.

In other words, some use polling, others operate based on system events or task queues, etc. In fact, both daemons and schedulers can face delays due to system conditions, although daemons are typically designed to maximize efficiency and responsiveness.

It's also interesting to know that there are several ways to create a daemon—some old and conventional, others modern and very resilient, or somewhere in between.

A traditional way would be a script in init.d, which is part of the SysV initialization system. This model preceded systemd and was the standard in Linux distributions for many years. SysV init uses scripts in /etc/init.d/ to start, stop, and manage services in different runlevels, offering a simpler but less parallelized system than systemd. This approach is still supported in many distributions for compatibility reasons.

We can also create direct scripts with a loop or by observing queues without any control—possibly the easiest way, but the most problematic: the application runs without any control, management, etc.

But not everything is so black and white. Within the Linux community, there are those who see systemd as an unacceptable paradigm shift. The Unix philosophy preaches to "Do One Thing and Do It Well." Systemd does many things; for some, this is practical and a very favorable application design; for others, it's complexity and increased risk.

---

Hands On - Systemd

---

Well, we've seen the theory, and although I find the idea very cool, let's move on to some practice now.

I must warn you that I will only cover systemd in this Hands-On; I have nothing against the other ways of creating a daemon, it's just for the practicality and visualization we'll have.

Before getting our hands dirty and creating a daemon, let's analyze some existing ones within a small lab. So, let's go.

Lab:

Troubleshooting Laboratory with Systemd and Journal

The idea in this lab is to practice how to create a daemon using systemd and perform troubleshooting using journalctl and systemd itself.

Part 1:

1.

Create a simple script that will serve as our service:

sudo vi /usr/local/bin/my-service.sh

And inside the service, let's add a small loop made with bash:

#!/bin/bash
echo "Starting my-service"
logger "my-service started"
count=1
while true; do
    echo "my-service is running: iteration $count"
    logger "my-service is running: check #$count"
    count=$((count+1))
    sleep 15
done

NOTE: Any content can be used, but you can use this if you want something ready.

2.

Now let's allow this service to run:

sudo chmod +x /usr/local/bin/my-service.sh

3.

Great, if everything is correct so far, now let's create a systemd config file for the service:

sudo vi /etc/systemd/system/my-service.service

Add the following content:

[Unit]
Description=My test service
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/my-service.sh
Restart=always
RestartSec=5
StandardOutput=journal
StandardError=journal

SyslogIdentifier=my-service

[Install]
WantedBy=multi-user.target

4.

Okay, now let's start everything. To do this, follow these steps:

sudo systemctl daemon-reload
sudo systemctl enable my-service.service
sudo systemctl start my-service.service

5.

To check if the service is running correctly, we can execute the following command:

sudo systemctl status my-service.service

If you see it's running, congratulations! We've successfully created our first daemon using systemd.

6.

Now let's check the logs of our service:

sudo journalctl -u my-service.service -f

The -f (follow) parameter shows the messages in real time.

Part 2:

Ah, great, our daemon is running, everything is fine and peaceful. But it's time to break things...

Let's break the service in various ways to practice troubleshooting.

Scenario 1: Corrupt the service script

Let's access the script code again:

sudo vi /usr/local/bin/my-service.sh

Let's insert some syntax error:

Restart the service:

sudo systemctl restart my-service.service

Scenario 2: Modify script permissions

sudo chmod -x /usr/local/bin/my-service.sh
sudo systemctl restart my-service.service

Part 3:

Now let's use the systemd and journal tools to diagnose the problems.

1. Check service status

sudo systemctl status my-service.service

Observe the error codes, messages, and status.

# View all service logs
sudo journalctl -u my-service.service

# View only error logs
sudo journalctl -u my-service.service -p err

# View logs from a specific period
sudo journalctl -u my-service.service --since "10 minutes ago"

# View logs from the current boot
sudo journalctl -u my-service.service -b

# Check service dependencies
sudo systemctl list-dependencies my-service.service

# Check service configuration
sudo systemctl cat my-service.service

# Check processes
ps aux | grep my-service

All these commands will help you in troubleshooting. I recommend running them and seeing what each one does, trying to grasp every nuance. Now, try to find the errors within the logs, break it again (or more), correct them, and restart the daemon. In the end, that's the idea...

Practice so that if one day you have to create one and work on it, everything will be smooth.