The Server Was Up. It Was Also Compromised.

Jabed Hossain — Mon, 19 Oct 2026 09:20:22 +0000

The sites were still loading. Forms worked, pages opened, nothing was throwing errors. From the outside, everything looked fine.

It wasn't. Someone else had been living on that server.

What I found

A rootkit hiding processes from the usual tools
Backdoor files planted in web directories, made to look like ordinary PHP files
An SSH key I had never added, sitting in an authorized_keys file
Cron jobs quietly running malicious scripts on a schedule

None of it broke anything. That was the point. Attackers who break things get noticed and kicked out. Attackers who stay quiet keep access for months.

Cleaning it up

I removed the backdoors, the unknown key and every cron job I couldn't explain, got rid of the rootkit, and audited the installed packages for changed files.

Then I worked on making sure it couldn't happen the same way again:

SSH: keys only, no passwords, no root login
Firewall: UFW on, with only the ports that need to be open
Integrity checks and login audits, so changes and strange logins get noticed
Least privilege: services run as their own users, not as root

Watching it

A clean server only stays clean if someone notices when it changes. So I added off site backups with rclone, a watchdog that runs every 5 minutes, uptime and SSL checks, and alerts to Discord and email.

That setup now watches 29 client sites.

The checks I run now

Nothing advanced. That's exactly why they get skipped.

Who can log in?

cat /root/.ssh/authorized_keys /home/*/.ssh/authorized_keys

What runs on a schedule?

for u in $(cut -d: -f1 /etc/passwd); do crontab -l -u "$u" 2>/dev/null; done

What is listening?

ss -tulpn

Which PHP files changed this week?

find /var/www -name "*.php" -mtime -7

One warning: if a rootkit is already on the machine, the tools on that machine can lie to you. That's why integrity checks and backups stored somewhere else matter so much.

What I learned

"The site is up" says nothing about whether the server is safe. Uptime monitoring and security monitoring are different jobs.

Attackers don't break things. They move in and stay quiet. The most dangerous compromise looks like a normal day.

If you don't know what normal looks like, you can't spot what isn't. Write down the expected SSH keys, cron jobs and open ports for every server you run, and check against that list.

I'm Jabed, a DevOps engineer who also builds the Laravel, React and Next.js apps I deploy.

One Pipeline, Many Servers: How I Deploy 19+ Repos With One GitHub Actions Template

Jabed Hossain — Mon, 19 Oct 2026 08:12:35 +0000

Every deploy at my job used to be a small ritual. SSH into the right server, pull the code, install dependencies, build the assets, run the migrations, fix the permissions, clear the cache, reload PHP. Then do it again for the next project.

Today I look after 19+ repositories across 8 products, running on several Linux servers, and none of them gets deployed by hand. They all use the same GitHub Actions template.

This post covers how that template works, the small bot I run next to it, and the honest list of what it still can't do.

Why manual deploys stop working

The steps themselves are easy. The problem is that they live in someone's memory.

Forget composer install after a dependency change and the site throws errors. Skip a migration and a page breaks. Run the commands in a terminal connected to the wrong server and you've just updated the wrong client's site. And when something goes wrong, nobody else knows what was run, or when.

With one project that's manageable. With dozens of client sites spread across several servers, it isn't.

One template, two environments

Every repository gets the same workflow file. Two branches matter:

develop deploys to the staging environment
main deploys to the live environment

Each one is a GitHub Environment with its own secrets: the server host, the SSH user, the deploy key, the app path and a Discord webhook. That's what makes one template work across many servers. The workflow file never changes. The secrets decide where the code lands.

name: Deploy

on:
  push:
    branches: [develop, main]

concurrency:
  group: deploy-${{ github.ref_name }}
  cancel-in-progress: false

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: ${{ github.ref_name == 'main' && 'live' || 'staging' }}
    steps:
      - name: Deploy over SSH
        uses: appleboy/ssh-action@v1
        with:
          host: ${{ secrets.SERVER_HOST }}
          username: ${{ secrets.SERVER_USER }}
          key: ${{ secrets.SERVER_SSH_KEY }}
          script: |
            set -e
            cd ${{ secrets.APP_PATH }}
            git pull origin ${{ github.ref_name }}
            composer install --no-dev --optimize-autoloader
            npm ci && npm run build
            php artisan migrate --force
            php artisan config:cache && php artisan route:cache
            sudo chown -R www-data:www-data storage bootstrap/cache
            sudo systemctl reload php8.3-fpm

      - name: Notify Discord
        if: always()
        run: |
          curl -s -H "Content-Type: application/json" \
            -d "{\"content\": \"Deploy ${{ job.status }}: ${{ github.repository }} (${{ github.ref_name }})\"}" \
            "${{ secrets.DISCORD_WEBHOOK }}"

A few details that matter more than they look:

set -e stops the script at the first command that fails, instead of carrying on and leaving the app half updated.
concurrency makes sure two pushes never deploy to the same environment at the same time.
if: always() on the Discord step means the team hears about failures, not only successes.
Each server has its own deploy user and SSH key. GitHub never holds a root login.

WordPress repos use the same file with the Laravel lines removed. Adding a new project means copying the template, creating the two environments and filling in their secrets.

A bot for the problems that show up before deploy

Once deploys were automated, a different problem became visible. With this many repositories, pull requests got left open, branches drifted, and merge conflicts only surfaced when someone finally tried to merge, often right before a release.

So every repo also gets a small scheduled workflow. Once a day it checks the open pull requests and posts to Discord if any of them has a merge conflict or has been waiting too long.

name: PR check

on:
  schedule:
    - cron: "0 3 * * 1-5"
  workflow_dispatch:

permissions:
  pull-requests: read

jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/github-script@v7
        env:
          DISCORD_WEBHOOK: ${{ secrets.DISCORD_WEBHOOK }}
        with:
          script: |
            const { owner, repo } = context.repo;
            const prs = await github.paginate(github.rest.pulls.list, { owner, repo, state: "open" });
            const lines = [];
            for (const pr of prs) {
              const { data } = await github.rest.pulls.get({ owner, repo, pull_number: pr.number });
              const days = Math.floor((Date.now() - new Date(pr.created_at)) / 86400000);
              if (data.mergeable === false) lines.push(`Conflict: #${pr.number} ${pr.title}`);
              else if (days >= 3) lines.push(`Open ${days} days: #${pr.number} ${pr.title}`);
            }
            if (lines.length) {
              await fetch(process.env.DISCORD_WEBHOOK, {
                method: "POST",
                headers: { "Content-Type": "application/json" },
                body: JSON.stringify({ content: `**${repo}**\n` + lines.join("\n") }),
              });
            }

It's not clever. It just makes sure someone notices while the problem is still small.

What changed

A deploy is a git push. Nobody has to remember the steps, because the steps are written down in the workflow.
Staging gets every change first. Clients see work on staging before it reaches their live site.
New projects and new servers are a template, not a process.
There's a record. The Actions log shows every deploy: which commit, which environment, and whether it worked. Discord shows the same thing to the whole team.

Automation isn't about being fast. It's about making the boring steps impossible to forget.

What it still can't do

Imagine the migration fails on live. set -e stops the script and Discord tells me something broke. But by then, git pull has already replaced the code. The live site is now running new code against an old database. Stopping early is better than carrying on, but it doesn't put anything back.

Two things are missing:

No backup right before the deploy. The servers have nightly off site backups, but those can be almost a day old.
No rollback. Going back means reverting the commit and deploying again, while the site is broken.

What I'm adding next

1. A database dump before every live deploy

The code is already in git. What can't be recreated is the data. So the first step on the server becomes a timestamped dump, keeping only the last five. Credentials come from a protected ~/.my.cnf, not the script.

TS=$(date +%Y%m%d-%H%M%S)
DIR=/var/backups/deploys/my-app
mkdir -p "$DIR"
mysqldump --single-transaction my_app_db | gzip > "$DIR/db-$TS.sql.gz"
ls -1t "$DIR"/db-*.sql.gz | tail -n +6 | xargs -r rm

2. Release folders and a symlink

Instead of pulling into the live folder, each deploy builds into its own folder under releases/. Files that must survive deploys, like .env, Laravel's storage and WordPress uploads, live in shared/ and get linked in. Nginx serves current, which is just a symlink. The new release only goes live when that link moves, and moving it back is the rollback.

set -e
APP=/var/www/my-app
REL=$APP/releases/$(date +%Y%m%d-%H%M%S)

git clone --depth 1 --branch main git@github.com:your-org/my-app.git "$REL"
ln -s $APP/shared/.env "$REL/.env"
rm -rf "$REL/storage" && ln -s $APP/shared/storage "$REL/storage"

cd "$REL"
composer install --no-dev --optimize-autoloader
php artisan migrate --force
php artisan config:cache

ln -sfn "$REL" $APP/current.tmp && mv -Tf $APP/current.tmp $APP/current
sudo systemctl reload php8.3-fpm

ls -1dt $APP/releases/* | tail -n +6 | xargs -r rm -rf

I'll probably use Deployer for this rather than maintaining the script myself. It does the release and symlink pattern out of the box, has recipes for Laravel and WordPress, and rolls back with dep rollback.

3. A health check that rolls back on its own

After the switch, the pipeline requests a health endpoint. If it doesn't get a success response within ten seconds, it points current back at the previous release and fails the job, which sends the Discord alert.

if ! curl -fsS --max-time 10 https://example.com/health > /dev/null; then
  PREV=$(ls -1dt $APP/releases/* | sed -n 2p)
  ln -sfn "$PREV" $APP/current.tmp && mv -Tf $APP/current.tmp $APP/current
  sudo systemctl reload php8.3-fpm
  echo "Health check failed, rolled back to $PREV"
  exit 1
fi

One limit to keep in mind: the symlink rolls back code, not the database. A migration that drops a column won't be undone by moving a link. That's why the dump comes first, and why migrations should stay backward compatible. Add the new column in one release, stop using the old one in the next, and remove it only after that.

The takeaway

The first version of a pipeline doesn't need to be perfect. Mine started as "run the same commands I used to type," and that alone removed most of the mistakes. The next version is about making failure cheap: keep a copy of the data, and make going back a single command.

How do you handle deploys across many small projects? I'd like to hear what other setups look like, especially for WordPress.

DEV Community: Jabed Hossain