DEV Community

Dharamraj Yadav
Dharamraj Yadav

Posted on

Mastering systemd, SSH, and Package Management

#linux #devops #devsecops #networking

Week 5 of My DevSecOps Journey — systemd, SSH & Package Management

This week was all about how a Linux server actually works behind the scenes. Three big things: how Linux runs services in the background (systemd), how you securely log into remote servers (SSH), and how you manage software on a server (package management). I also did real hands-on stuff on my own EC2 — building a service, setting up tunnels, and hardening SSH. Here's everything in my own words.

systemd: The Boss Process (PID 1) — what it is and why it exists

systemd is the first process that starts when linux boots, and it runs as PID 1. It's the parent of every other process in the system. systemd supervises them while the machine is on and restarts them if they crash. Because it's designed never to fail, if it died the whole system would go down with it.

That's the "why it exists" part — when a computer boots, something has to bring the whole system up (networking, logging, ssh) and keep it alive with no human sitting there. systemd is that something.

You can see it yourself:

ps -p 1        # this shows PID 1, and it will say systemd
Enter fullscreen mode Exit fullscreen mode

Units and Their Four Types

A unit is one thing that systemd manages — "unit" is the general word, and there are several types. The four common ones:

  • .service = a running program / background job (e.g. a web server). This is where you declare what to run and how. The main type you'll use.
  • .target = a checkpoint that groups other units together. It runs nothing itself, it just means "this whole group of services should be on by now" (like multi-user.target).
  • .timer = a scheduler that starts another unit at a set time or interval. A modern replacement for cron.
  • .socket = a doorway that starts a service on demand, only when a connection actually arrives, instead of keeping it running all the time.

The way I think about it: a service is a doer (a program doing a job), and a target is a checkpoint (a milestone that just says a group of stuff is up).

You can list them:

systemctl list-units --type=service
systemctl list-units --type=target
systemctl status <name>
Enter fullscreen mode Exit fullscreen mode

Old Init Systems vs systemd

The old init system — called SysV init — had several problems. It started services one at a time, sequentially, by running startup scripts, which made booting slow. It also didn't have any auto-restart mechanism — once a script ran, init forgot about it, so if the service crashed nobody noticed. And it could only control the order things started, not real dependencies.

New systemd fixed all of this. It runs services in parallel (much faster), it supervises them and restarts them if they crash, and it understands real dependencies. That's why the world moved from old systems to systemd.

Writing Your First Custom Service File

To register your own program with systemd, you write a small .service file. It's basically a form with three sections:

  • [Unit] = what it is
  • [Service] = how to run it
  • [Install] = when it starts

I created one for my own monitor script (it checks CPU, memory and disk usage). The file goes in /etc/systemd/system/:

sudo vim /etc/systemd/system/monitor.service
Enter fullscreen mode Exit fullscreen mode

Inside:

[Unit]
Description=System monitor script (CPU, memory, disk)

[Service]
Type=oneshot
ExecStart=/home/aadi/system-monitoring/monitor.sh

[Install]
WantedBy=multi-user.target
Enter fullscreen mode Exit fullscreen mode

One important thing here is Type=oneshot. My script runs once, does its job, and exits — it's not a forever-running program like a web server. oneshot tells systemd "this one runs once and finishes, that's normal, don't think it crashed."

After creating or editing the file, you have to tell systemd to re-read it (it doesn't notice new files automatically):

sudo systemctl daemon-reload        # always run this after editing a unit file
sudo systemctl start monitor        # run it now
systemctl status monitor            # check it
sudo systemctl enable monitor       # make it start at boot
Enter fullscreen mode Exit fullscreen mode

Note: start and enable are two different things. start runs it right now. enable makes it start automatically at boot. When you enable a service, systemd just creates a symlink inside multi-user.target.wants/ — that folder is the target's "guest list," and at boot the target starts everything in it.

Also — use full paths in your script. systemd doesn't run inside your folder, so a relative path like ./logs will break. Always use the absolute path.

Dependencies — After, Requires, Wants

This is the part that confuses people, so I want to get it right. In a service file, order and requirement are two completely separate things:

  • After=X → start this after X. This is only about order, it doesn't mean you need X.
  • Wants=X → weak dependency: pull X in too, but start even if X fails.
  • Requires=X → strong dependency: if X fails or stops, stop this too.

The trap: Requires= says nothing about order. So Requires=postgresql.service on its own will start your app and the database at the same time — and your app might try to connect before the DB is ready. That's why you usually pair them:

Requires=postgresql.service
After=postgresql.service
Enter fullscreen mode Exit fullscreen mode

Requirement says whether, ordering says when. You normally need both.

Reading Logs with journalctl

Every service that systemd runs writes its logs to one central place, and journalctl is the tool to read them. Think of it like the machine's diary.

The whole skill is filtering — you never read everything. The most useful command is -u, which shows only one service:

journalctl -u monitor.service      # only this service's logs
journalctl -u ssh -f               # follow live (Ctrl+C to exit)
journalctl -u ssh --since today    # only today's logs
journalctl -n 20                   # last 20 lines
Enter fullscreen mode Exit fullscreen mode

Without -u, you get every program mixed together and it's unreadable. So -u <service> is what actually makes journalctl useful.

Timers — the Modern Cron

A timer is like an alarm clock that starts a service on a schedule. It comes as a pair: a .timer file (the when) and a .service file (the work). And if they have the same name — monitor.timer and monitor.service — systemd links them automatically.

I made a timer to run my monitor script every 2 minutes:

[Unit]
Description=Run system monitor every 2 minutes

[Timer]
OnBootSec=2min
OnUnitActiveSec=2min

[Install]
WantedBy=timers.target
Enter fullscreen mode Exit fullscreen mode

OnBootSec = first run after boot. OnUnitActiveSec = repeat interval. For calendar-based scheduling (like "every night at 2am") you'd use OnCalendar instead.

One important thing: you start the timer, not the service:

sudo systemctl daemon-reload
sudo systemctl start monitor.timer
systemctl list-timers               # shows next fire time + which service it triggers
Enter fullscreen mode Exit fullscreen mode

The cool part was opening journalctl -u monitor.service -f and watching my script fire on its own every 2 minutes, with nobody touching anything. That's when it clicked — my machine was monitoring itself automatically.

SSH — Key-Based Authentication

SSH (Secure Shell) lets you get a terminal on another machine over the network. Instead of logging in with a password (which bots can guess and which travels over the network), you use a key pair.

The way I understand it — think of a padlock and its key:

  • Public key (.pub) = the padlock. You can share it freely, and it goes on the server.
  • Private key = the secret key. It stays on your laptop and you never share it.

The server locks a challenge with your padlock, and only your private key can open it — and it does that on your own machine. So the secret never travels over the network. That's the whole win over passwords.

Generate a key:

ssh-keygen -t ed25519 -C "my laptop key"
ssh-copy-id user@host        # copies your public key onto the server
Enter fullscreen mode Exit fullscreen mode

Fun fact: on AWS EC2, the .pem file AWS gives you is a private key. AWS already put the public half on the instance. So ssh -i key.pem ubuntu@IP is already key-based auth — I was doing it without realizing it.

The SSH Config File

Typing the full command every time is painful:

ssh -i ~/Downloads/testawsmachine.pem ubuntu@16.170.254.209
Enter fullscreen mode Exit fullscreen mode

So you save all of that under a nickname in ~/.ssh/config:

Host testaws
    HostName 16.170.254.209
    User ubuntu
    IdentityFile ~/Downloads/testawsmachine.pem
Enter fullscreen mode Exit fullscreen mode

Now I just type:

ssh testaws
Enter fullscreen mode Exit fullscreen mode

It's not magic — SSH reads the config, expands the nickname back into the full command, and logs in with the key. Host is the nickname you type, HostName is the real address, User is the username, IdentityFile is the key.

SSH Tunneling — -L, -D, -R

This one bends your brain a bit. The core idea: one SSH connection can do two jobs at once — carry your terminal and carry other network traffic through the same encrypted connection.

The trick to remembering the flags: local = your laptop, remote = the server.

-L (local forwarding) → your laptop reaches into the server. Good for reaching a service on the server that isn't open to the internet (like a database).

ssh -L 8080:127.0.0.1:8080 testaws
Enter fullscreen mode Exit fullscreen mode

I ran a tiny web server on my EC2 bound to 127.0.0.1 (so only the EC2 could see it), and using -L I opened it in my laptop's browser — reaching a "locked inside" service through SSH.

-D (dynamic forwarding = SOCKS proxy) → route anything through the server, so the world sees the server's IP instead of yours.

ssh -D 1080 testaws
curl --socks5 127.0.0.1:1080 https://api.ipify.org
Enter fullscreen mode Exit fullscreen mode

When I ran that curl, it showed my EC2's IP, not my home IP — proof my traffic was going out through the server in Europe. This is how you'd route your browser through your own server.

-R (remote forwarding) → the mirror of -L. The server reaches back into your laptop. Good for exposing something running on your laptop through the server's public address.

ssh -R 9090:127.0.0.1:9090 testaws
Enter fullscreen mode Exit fullscreen mode

I ran a server on my laptop and fetched it from inside the EC2 — a server in a data center reading files off my laptop, through the tunnel. Mind-bending but cool.

Quick summary:

  • -L → laptop into server (one fixed target)
  • -D → laptop to anywhere through the server (proxy)
  • -R → server back into laptop

There's also ssh-agent (holds your unlocked key in memory so you type the passphrase once) and the -A flag (agent forwarding, to carry your key identity to another server) — but only use -A with servers you trust.

Package Management — apt & dnf/yum

A server has no app store with buttons, so you install software by command. On Ubuntu that command is apt.

The thing that confused me at first is that it's a two-step process:

  • apt update = refresh the list of available software. This installs nothing.
  • apt upgrade = actually update the software you already have.
  • apt install X = install a new program. 
sudo apt update              # do this FIRST
sudo apt upgrade             # update everything
sudo apt install tree        # install a package
sudo apt remove tree         # uninstall
Enter fullscreen mode Exit fullscreen mode

If you skip apt update, apt might say "Unable to locate package" even for real software — because its list is stale. So always update first. (This actually happened to me.)

Other distros use different tools for the same job. Red Hat family (Amazon Linux, CentOS, Fedora) use yum or dnf:

  • apt update → dnf check-update
  • apt install X → dnf install X
  • apt upgrade → dnf upgrade

One gotcha: in dnf/yum, update means "install upgrades" — the opposite of apt's update. And dnf is just the newer version of yum.

SSH Hardening

This is where DevSecOps really starts — locking down the SSH server so attackers can't get in. The settings live in a file called sshd_config.

⚠️ The golden rule (learned this the important way): never lock yourself out. Always keep your working session open, test the config before restarting, and test a new login in a second terminal — and only close the original session once the new login works.

The two main changes:

  • PasswordAuthentication no → only key logins allowed. This kills password-guessing bots completely.
  • PermitRootLogin no → block direct login as root. "root" is the one username every attacker tries, so blocking it removes the obvious target.

The clean way I did it was to create my own config file (so I don't mess with what AWS set up). Files in /etc/ssh/sshd_config.d/ are read in order and the first setting wins, so a lower number wins:

sudo vim /etc/ssh/sshd_config.d/10-hardening.conf
Enter fullscreen mode Exit fullscreen mode 
PermitRootLogin no
PasswordAuthentication no
Enter fullscreen mode Exit fullscreen mode

Then the safe workflow:

sudo sshd -t                      # test config (silent output = OK)
sudo systemctl restart ssh
# in a SECOND terminal, confirm you can still log in:
ssh testaws                       # should work
ssh root@testaws                  # should be DENIED — proof it worked
Enter fullscreen mode Exit fullscreen mode

When I tried ssh root@testaws and got "Permission denied," that was the proof my hardening actually worked.

One real lesson: always check the current settings before changing them. On my EC2, AWS had already turned off password login by default — so I would've been "fixing" something that was already done. Check first, don't assume.

fail2ban — the Automatic Bouncer

Even with hardening, bots constantly hammer a public server trying to log in. fail2ban is an automatic bouncer for exactly this: it watches the login logs, and if one IP fails to log in too many times too fast, it automatically bans that IP at the firewall.

Its decision comes down to a few settings:

  • maxretry = how many failures before a ban
  • findtime = the time window those failures must happen in
  • bantime = how long the ban lasts
  • ignoreip = IPs that are never banned (put your own home IP here so you don't lock yourself out)

So the rule is basically: X failures within Y minutes → ban for Z minutes. A typical default is 5 failures in 10 minutes → ban.

The clever bit is that it's really rate-limiting — a human fumbling a login once in a while never trips it, but a bot firing thousands of attempts a second trips it instantly. Hardening makes attacks fail; fail2ban makes attackers go away. Two layers = defense in depth.

sudo apt install fail2ban
sudo systemctl enable --now fail2ban
sudo fail2ban-client status sshd
Enter fullscreen mode Exit fullscreen mode

Wrap-up

That was Week 5. By the end of it I'd built a service that monitors my own machine, put it on a timer so it runs automatically, learned to read logs with journalctl, set up key-based SSH with config aliases, tunneled traffic three different ways, managed packages, and hardened my EC2 so root and password logins are blocked.

The biggest lesson wasn't any single command — it was that all this abstract stuff finally made sense when I did it with my own hands and watched it work on a real server. On to Week 6.

Top comments (0)