DEV Community: Dinesh Wijethunga

Connect Jenkins to GitHub: Deploy Keys, PAT and Webhooks

Dinesh Wijethunga — Sun, 13 Sep 2026 02:25:05 +0000

In this tutorial, I will show you step by step how to connect Jenkins to a private GitHub repository with least-privilege credentials — a per-repo deploy key, a read-only fine-grained PAT, and a push webhook — including the two mistakes that each cost a failed build in a real setup.

The three credentials and their jobs

Credential	Owner	Used for
SSH deploy key	deploy user on the server	`git pull` during deploys
Fine-grained PAT	Jenkins	checkout + GitHub API (scan, statuses)
Webhook	GitHub → Jenkins	instant builds on push

Step 1 — Per-repo deploy key (a convention worth stealing)

One key per repo, named accordingly, with an SSH host alias so multiple repo keys coexist on one server:

# as the deploy user on the server
ssh-keygen -t ed25519 -f ~/.ssh/id_repo_myapp -N "" -C "deploy@myapp"

cat >> ~/.ssh/config <<'EOF'
Host github-repo-myapp
    HostName github.com
    User git
    IdentityFile ~/.ssh/id_repo_myapp
    IdentitiesOnly yes
EOF

Add the public key in GitHub → repo → Settings → Deploy keys (read-only), then verify and clone through the alias:

ssh -T git@github-repo-myapp        # "successfully authenticated"
git clone github-repo-myapp:acme/myapp.git

Step 2 — The fine-grained PAT (minimal, but watch the expiry)

GitHub → Settings → Developer settings → Fine-grained tokens:

Repository access: Only select repositories → your repo.
Permissions: Contents: Read-only (Metadata is added automatically). That's all a checkout needs.
Expiration: the default is 30 days — your pipeline will die silently in a month. Choose Custom and set ~1 year, with a reminder before it lapses.

Store it in Jenkins as Username with password: username = your GitHub username, password = the token, ID = github-token.

Gotcha #1 — credential ID is not the username

We also store the CI database login in Jenkins. First build failed every single test with:

SQLSTATE[HY000] [1045] Access denied for user 'ci-mysql'@'localhost'

ci-mysql was the credential's ID, not the database username — the dialog had been filled with the ID in the username field. When a Jenkinsfile does credentials('ci-mysql'), the ID is only the lookup key; the username/password fields are what get injected. If your logs show your credential ID where a username should be, that's the tell.

Step 3 — The multibranch job, and gotcha #2

New Item → Multibranch Pipeline → Branch Sources → GitHub → credentials github-token, HTTPS repo URL. Save triggers the first scan — which immediately failed:

ERROR: Could not fetch branches from source
"message":"Resource not accessible by personal access token","status":"403"
… Failed to retrieve …/pulls?state=open

The default behaviors include Discover pull requests from origin/forks, which call the PR API — and a Contents-only token can't. Least privilege collided with default assumptions. Two clean fixes:

No PR workflow (our case): in Behaviors, delete both Discover pull requests entries and set Discover branches → All branches (the default "exclude branches filed as PRs" strategy also touches the PR API).
PR workflow: add Pull requests: Read-only to the PAT instead.

Rescan: 'Jenkinsfile' found — Scheduled build for branch: master.

Step 4 — The webhook

GitHub repo → Settings → Webhooks → Add webhook:

Payload URL: https://jenkins.example.com/github-webhook/ (trailing slash matters)
Content type: application/json
Events: Just the push event

GitHub sends a ping on save — look for the green tick under Recent Deliveries. From now on every push builds within seconds; keep the periodic scan off unless you need branch-deletion cleanup.

Next in the series: post #6 is the Jenkinsfile itself — the full stage graph for a Laravel + React app, the CI MySQL user with its dual-host grant quirk, and why withoutVite() belongs in your base TestCase.

Originally published at dineshstack.com — read the full version with code samples and updates there.

How to Run Jenkins Behind Nginx with SSL on a Subdomain

Dinesh Wijethunga — Thu, 10 Sep 2026 01:00:06 +0000

In this tutorial, I will show you step by step how to run Jenkins behind nginx with SSL on its own subdomain — on a server that already hosts a production app on a wildcard domain, which is exactly where the sharp edges are.

Setup: one VPS, an existing Laravel SaaS serving *.example.com (multi-tenant subdomains), and Jenkins on 127.0.0.1:8080 that we want at https://jenkins.example.com.

Step 1 — The nginx vhost (exact name beats wildcard)

Key nginx fact: when a request arrives, an exact server_name always wins over a wildcard. So even though *.example.com would swallow jenkins.example.com, a dedicated vhost takes precedence:

# /etc/nginx/sites-available/jenkins
server {
    listen 80;
    server_name jenkins.example.com;

    location / {
        proxy_pass         http://127.0.0.1:8080;
        proxy_set_header   Host $host;
        proxy_set_header   X-Real-IP $remote_addr;
        proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $scheme;
        proxy_http_version 1.1;
        proxy_request_buffering off;   # long-running build log streams
        proxy_read_timeout 90s;
    }
}

sudo ln -sf /etc/nginx/sites-available/jenkins /etc/nginx/sites-enabled/jenkins
sudo nginx -t && sudo systemctl reload nginx

Gotcha from the field: if ln says File exists, there's a stale symlink from an earlier attempt shadowing your new config — rm it and re-link. And if the subdomain still shows your main app, the wildcard vhost is winning because your new vhost isn't actually enabled; verify with nginx -T | grep -A2 "server_name jenkins".

Step 2 — TLS. Read this before running certbot

Here is the mistake, so you don't repeat it. I ran the "friendly" installer mode:

certbot --nginx -d jenkins.example.com   # ← DON'T, on a multi-site server

certbot issued the certificate fine — and then deployed it into the wrong vhost: the production wildcard site. Its ssl_certificate lines suddenly pointed at the Jenkins cert, and every tenant subdomain on *.example.com started throwing certificate-mismatch errors. Production TLS, broken by a CI chore.

The fix was to point those two lines back at the wildcard cert and reload — but the lesson is permanent:

# On any server with more than one vhost: issue only, wire manually
sudo certbot certonly --nginx -d jenkins.example.com

Then add the 443 block yourself, where you decide which file changes:

server {
    listen 443 ssl;
    server_name jenkins.example.com;

    ssl_certificate     /etc/letsencrypt/live/jenkins.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/jenkins.example.com/privkey.pem;
    include /etc/letsencrypt/options-ssl-nginx.conf;
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

    location / {
        proxy_pass       http://127.0.0.1:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

The Cloudflare shortcut: maybe you need no cert at all

If Cloudflare proxies your zone, check what your origin certificate already covers before issuing anything:

echo | openssl s_client -servername app.example.com -connect YOUR_SERVER_IP:443 2>/dev/null \
  | openssl x509 -noout -ext subjectAltName
# X509v3 Subject Alternative Name:
#     DNS:*.example.com, DNS:example.com

A wildcard SAN like that means any new first-level subdomain can simply reuse the existing cert files in its 443 block — zero certbot, zero new renewals, and Cloudflare's edge certificate covers the browser side. We used exactly this later in the series for the staging vhost.

Step 3 — Lock the box down

# Jenkins should only be reachable through nginx now
sudo ufw deny 8080/tcp

Also set Manage Jenkins → System → Jenkins URL to https://jenkins.example.com/ so redirects and webhook URLs generate correctly.

Next in the series: post #5 connects Jenkins to GitHub — per-repo deploy keys, a minimal fine-grained PAT, the webhook, and the 403 that breaks multibranch scanning when your token is (correctly) minimal.

Originally published at dineshstack.com — read the full version with code samples and updates there.

How to Install Jenkins on Ubuntu 24.04 Step by Step

Dinesh Wijethunga — Sun, 06 Sep 2026 23:00:07 +0000

In this tutorial, I will show you step by step how to install Jenkins on Ubuntu 24.04 with a complete, production-tested command sequence — including the two traps that produce "Package 'jenkins' has no installation candidate", both of which I hit on a real server.

Step 0 — Add swap if you have none

Check first. A box with Swap: 0B will eventually OOM-kill Jenkins mid-build once your pipeline runs composer, npm and a database-backed test suite at once:

free -h   # if Swap shows 0B:
sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile && sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
echo 'vm.swappiness=10' | sudo tee /etc/sysctl.d/99-swappiness.conf
sudo sysctl -p /etc/sysctl.d/99-swappiness.conf

swappiness=10 keeps swap as an emergency parachute, not a performance drag.

Step 1 — Java 21

sudo apt update
sudo apt install -y openjdk-21-jre
java -version   # openjdk 21.x

Step 2 — The repo, and the two traps

Trap #1: the keyrings directory doesn't exist. Every tutorial pipes the key with wget -O /etc/apt/keyrings/… — but wget -O does not create parent directories. On a fresh Ubuntu 24.04, /etc/apt/keyrings may not exist, wget silently writes nothing usable, and apt later reports no installation candidate. Create it explicitly:

sudo mkdir -p /etc/apt/keyrings

Trap #2: the signing key rotated. The widely-copied tutorials reference jenkins.io-2023.key. That key has been rotated — with it, apt update fails signature verification and the package is invisible. Use the current key:

sudo wget -O /etc/apt/keyrings/jenkins-keyring.asc \
    https://pkg.jenkins.io/debian-stable/jenkins.io-2026.key

echo "deb [signed-by=/etc/apt/keyrings/jenkins-keyring.asc]" \
    "https://pkg.jenkins.io/debian-stable binary/" | \
    sudo tee /etc/apt/sources.list.d/jenkins.list > /dev/null

If you're reading this later, check pkg.jenkins.io/debian-stable for the current key filename — rotation will happen again.

Step 3 — Install and verify

sudo apt update
apt-cache policy jenkins   # must show a candidate version — if not, revisit step 2
sudo apt install -y jenkins
sudo systemctl enable --now jenkins
systemctl status jenkins   # active (running)

Step 4 — The setup wizard, opinionated

sudo cat /var/lib/jenkins/secrets/initialAdminPassword

Open http://your-server-ip:8080, paste the password, then:

Plugins: "Install suggested plugins" is fine. You specifically need: Git, GitHub, Pipeline, Credentials Binding, SSH Agent. Anything missing installs later in two clicks.
Create the admin user — don't keep running as the unlock account.
Instance URL: set it to the HTTPS subdomain you'll create in post #4 (e.g. https://jenkins.example.com/), not the bare IP.

One UI note that cost me ten minutes: in current Jenkins (2.5xx), the # of executors setting is no longer on the System page — it moved to Manage Jenkins → Nodes → Built-In Node → Configure. For a VPS that also serves production traffic, set executors to 1 so two pushes can't run two heavy builds simultaneously.

Verification checklist

systemctl is-active jenkins          # active
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:8080  # 403 = up, auth required
free -h                              # swap present

Next in the series: right now Jenkins is exposed on :8080 over plain HTTP. Post #4 puts it behind nginx on its own HTTPS subdomain — and covers the wildcard-vhost mistake that briefly broke TLS on a production domain.

Originally published at dineshstack.com — read the full version with code samples and updates there.

Laravel 13 Git Hooks with Husky: Pint on Every Commit

Dinesh Wijethunga — Sat, 05 Sep 2026 19:00:08 +0000

In this tutorial, I will show you step by step how to set up git hooks in Laravel 12 with Husky, with a complete example from a production Laravel + React (Inertia) repo.

The goal: nobody can commit unformatted PHP, and nobody can push broken JavaScript — while keeping every hook fast enough that no one is ever tempted to bypass it.

Step 1 — Install Husky

npm install --save-dev husky
npx husky init

This creates a .husky/ directory and adds a prepare script to package.json, so every developer gets the hooks automatically on npm install:

"scripts": {
    "prepare": "husky"
}

Step 2 — The pre-commit hook: Pint on staged files only

The naive version runs Pint on the whole project. Don't — it reformats files your teammate is mid-way through, and it's slow. The correct version formats only the PHP files in this commit, then re-stages them so the fixes ride along:

# .husky/pre-commit
STAGED_PHP=$(git diff --cached --name-only --diff-filter=ACM | grep '\.php$' || true)

if [ -n "$STAGED_PHP" ]; then
    echo "husky: pint on staged PHP files…"
    ./vendor/bin/pint $STAGED_PHP
    git add $STAGED_PHP
fi

Three details that matter:

--diff-filter=ACM — added, copied, modified. Deleted files are excluded, otherwise Pint errors on paths that no longer exist.
|| true — grep exits non-zero when nothing matches, which would abort every PHP-free commit.
git add $STAGED_PHP — Pint's fixes are re-staged, so what lands in the commit is the formatted version. No follow-up "apply pint" commits, ever.

Step 3 — The pre-push hook: the fast test wall

# .husky/pre-push
echo "husky: JS tests…"
npm run test:js

echo "husky: pint check…"
./vendor/bin/pint --test

# Optional: uncomment to also run a critical PHP test group (keep it FAST)
# composer test:critical

On our repo the JS suite runs in about 3 seconds and pint --test (check-only mode, no writes) in about 2. A failed push looks like this and never leaves the machine:

husky: JS tests…
 FAIL  resources/js/__tests__/invoice-totals.test.ts
error: failed to push some refs

Why the full PHP suite is deliberately NOT in the hooks

Our full Laravel suite is 340+ tests against a real MySQL server — several minutes. Here is the uncomfortable truth from every team I've taught this to: any hook slower than about ten seconds gets bypassed. Developers discover git push --no-verify on day two, and now your "safety net" is theatre.

So the contract is: hooks catch the cheap mistakes instantly, and the full suite runs in Jenkins on every push — where it blocks the deploy, not the developer. That pipeline is exactly what the rest of this series builds.

Verify it works

Deliberately mangle a file and try to commit:

echo 'class   Foo{public function bar( ){return 1 ;}}' >> app/Models/Scratch.php
git add app/Models/Scratch.php && git commit -m "test hook"
# → pint reformats it, re-stages, commit lands clean
git show --stat HEAD

Next in the series: post #3 installs Jenkins on Ubuntu 24.04 — including the two apt traps that produce the infamous "Package 'jenkins' has no installation candidate" error in 2026.

Originally published at dineshstack.com — read the full version with code samples and updates there.

Husky vs Jenkins vs Ansible: When to Use Which for CI/CD?

Dinesh Wijethunga — Thu, 03 Sep 2026 19:00:08 +0000

In this guide, we will cover everything you need to know about choosing between Husky, Jenkins and Ansible for a Laravel CI/CD pipeline — and why the real answer is a layered setup, not a single winner.

I recently built the full pipeline for a production Laravel 12 + React (Inertia) multi-tenant SaaS running on a single Ubuntu VPS. This series documents that build step by step, including everything that broke along the way. But before touching a terminal, we had to answer the question every team hits: which tool?

The trap: these are not competing tools

The comparison is a category error, and understanding why is the whole decision:

Tool	Where it runs	What it's for	When it fires
Husky	Developer's machine	Git-hook checks (lint, quick tests)	Before commit / push
Jenkins	Your server	Pipelines: test → build → deploy	After push
Ansible	Your machine → many servers	Server configuration management	When you provision servers

Husky cannot deploy. Jenkins cannot stop a bad commit from being created. Ansible does neither — it makes servers identical.

The layered architecture we chose

Developer Mac
 ├─ pre-commit (Husky): Laravel Pint on staged files only  → ~1s
 ├─ pre-push  (Husky): JS test suite + pint --test         → ~5s
 ▼ git push
Jenkins on the VPS
 ├─ Lint → PHP tests (full suite) → Vite build → JS tests
 ├─ Package artifact → deploy to STAGING → smoke test
 ├─ Manual approval gate
 └─ Deploy to PRODUCTION → health check → auto-rollback on failure

Two principles drive this:

Fast checks live locally, slow checks live on the server. A pre-commit hook that takes 30 seconds gets bypassed with --no-verify within a week — I've watched it happen on every team. Pint on staged files takes about a second; the full PHPUnit suite (340+ tests against MySQL) belongs in Jenkins where nobody is waiting with their fingers on the keyboard.
The server is the source of truth. Husky is a courtesy that keeps embarrassing commits out of history. Jenkins is the enforcement layer — it runs the suite you can't skip.

Why Jenkins over GitHub Actions?

Fair question, and for many teams Actions is the right call. We picked Jenkins because:

The VPS had spare capacity (31 GB RAM) — self-hosted CI costs nothing extra, and there are no per-minute charges for a 340-test suite that runs on every push.
The deploy target is the CI machine. No SSH-from-cloud-runner complexity, no secrets leaving the box.
Existing Jenkins experience on the team. The best tool is frequently the one you already know how to debug at 1 a.m.

Why we skipped Ansible

One VPS. Ansible's payoff is "make 10 servers identical" — with a single server, a version-controlled runbook (every command we ran, committed to the repo in docs/) gives the reproducibility without the abstraction tax. If we grow to multiple app servers, that runbook becomes the Ansible playbook spec.

Decision rule you can steal: Husky always (it's a 10-minute setup, post #2). Jenkins or Actions for CI — Jenkins if you own idle hardware, Actions if you don't. Ansible only at 3+ servers.

What's next in the series

Post #2 sets up Husky properly for a Laravel + React repo — including the trick that runs Pint only on staged files and re-stages the fixes automatically, and why the full PHP suite deliberately stays out of the hooks.

This series documents a real production build. The domains and IPs are anonymised; the mistakes are not.

Originally published at dineshstack.com — read the full version with code samples and updates there.

From Zero to Staging: A Real-World Laravel + Node.js Docker Deployment (With Every Bug We Hit)

Dinesh Wijethunga — Tue, 01 Sep 2026 19:00:08 +0000

From Zero to Staging: A Real-World Laravel + Node.js Docker Deployment (With Every Bug We Hit)

A step-by-step walkthrough of setting up a clean staging server from scratch — SSH hardening, Docker orchestration, Let's Encrypt SSL, database seeding, and debugging a login that refused to work for six different reasons.

The Setup

We had a production server (api.dineshstack.ae) running a Laravel 12 + Node.js microservices stack inside Docker. We needed a staging environment that was an exact mirror of production for mobile testing. The staging server existed but had been used by five different developers over time — it was a graveyard of stale projects, orphaned Docker volumes, and broken nginx configs.

Goal: One clean server. One Docker network. Three domains. Zero stale state.

Teaching note: Before touching anything on a shared server, always survey first. Never assume you know what's there. A five-minute df -h and docker ps -a saves hours of debugging phantom behaviour caused by leftover containers or conflicting ports.

Phase 1 — SSH Hardening

The first step before anything else: lock down the door.

On your local machine

# Generate a dedicated key for this server

ssh-keygen -t ed25519 -C "taggo_staging" -f ~/.ssh/dstack_staging


  
  
  Copy the public key to the server


ssh-copy-id -i ~/.ssh/dstack_staging.pub taggo@<SERVER_IP>

Then add a clean alias to ~/.ssh/config:

Host dstack_staging


    HostName <SERVER_IP>


    User taggo


    IdentityFile ~/.ssh/dstack_staging


    IdentitiesOnly yes

Now you connect with just ssh dstack_staging. No password, no ambiguity about which key to use.

On the server — disable password auth

sudo sed -i 's/^#\?PasswordAuthentication./PasswordAuthentication no/' /etc/ssh/sshd_config


sudo sed -i 's/^#\?PubkeyAuthentication./PubkeyAuthentication yes/' /etc/ssh/sshd_config


sudo systemctl reload ssh

Install fail2ban

sudo apt install fail2ban -y


sudo systemctl enable fail2ban --now

Teaching note: IdentitiesOnly yes is the flag most tutorials skip. Without it, SSH will try every key in your agent, which can confuse servers with strict attempt limits. Always be explicit.

Phase 2 — Server Survey

Before deleting anything, map exactly what exists.

# Who's taking up disk?


du -sh /var/www/* | sort -rh


  
  
  What containers are running?


docker ps -a --format "table {{.Names}}\t{{.Status}}\t{{.Image}}"


  
  
  What networks exist?


docker network ls


  
  
  What volumes exist?


docker volume ls

What we found:

49 GB across 8 stale project folders in /var/www/
Three different Docker Compose setups from three different developers, all creating their own networks
11 nginx site configs, only 1 needed
An erpdstack-mysql container with live data that must not be touched

Teaching note: docker ps -aq --filter name=dstack matches substrings. erpdstack contains dstack. Always preview your filter before passing it to docker rm. We learned this the hard way — the erpdstack container was removed. The data volume survived, but the container had to be recreated. Always verify, never assume.

Phase 3 — Full Server Reset

We removed everything dstack-related except the erpdstack data volume and network.

# Stop and remove all containers (carefully — verified list first) docker stop $(docker ps -aq) && docker rm $(docker ps -aq) Remove stale named volumes docker volume rm dstack_sail-mysql dstack_sail-redis dstack_sail-kafka \ dstack_sail-grafana dstack_sail-prometheus Remove stale networks docker network rm dstack_sail Remove all project folders sudo rm -rf /var/www/dstack /var/www/dstack_api /var/www/dstack-services \ /var/www/dinesh-dstack-testing /var/www/nextjs \ /var/www/dsstack-admin-dash /var/www/dstack-admin-dashboard \ /var/www/dstack-dashboard Purge unused Docker images docker image prune -af Clean apt cache and journal logs

sudo apt-get clean sudo journalctl --vacuum-size=100M

Result: Disk dropped from 70 GB used (73%) to 21 GB (22%). 49 GB recovered in under ten minutes.

Phase 4 — Fresh Deployment

Clone the projects

cd /home/dstack dstack (Laravel) — uses GitHub deploy key 1 git clone git@github-main:orions-it/dstack_api.git dstack cd dstack && git checkout docker-standalone dstack-services (Node.js) — uses GitHub deploy key 2

cd /home/dstack git clone git@github-second:DishKief/taggo-services.git dstack-services

Configure staging .env files

Key differences from production:

# dstack/.env — staging overrides


APP_ENV=staging


APP_DEBUG=true


APP_URL=https://api-staging.dstack.ae


DB_DATABASE=dstack_staging


  
  
  dstack-services/.env


NODE_ENV=staging


LARAVEL_UPSTREAM_URL=http://laravel_dstack:81

Start the stack — order matters

Kafka and MySQL are dependencies. Start the infrastructure side first.

# Start dstack-services first (Kafka, Redis, MySQL, monitoring) cd /home/dstack/dstack-services docker compose up -d --build Wait ~60 seconds for MySQL and Kafka to become healthy, then start Laravel

cd /home/dstack/dstack docker compose --profile kafka up -d --build

Bug we hit: The queue, scheduler, and Reverb containers all have depends_on: service_healthy on MySQL. If you run docker compose up before MySQL is healthy, those containers start and immediately exit. The fix: wait for MySQL to show (healthy) in docker ps, then run docker compose up -d again. Docker Compose is idempotent — it only starts what isn't running.

External volumes must be pre-created

The dstack-services compose declares Prometheus and Grafana volumes as external: true. Docker will refuse to start if they don't exist.

docker volume create --name dstack_sail-prometheus


docker volume create --name dstack_sail-grafana

Teaching note: external: true in a Compose file means "this volume was created outside this file — don't manage its lifecycle." It's a design choice that prevents accidental deletion during docker compose down -v. The trade-off: you must remember to create it manually when deploying fresh.

Run migrations and bootstrap Laravel

cd /home/dstack/dstack Create the staging database docker compose exec mysql_dstack mysql -uroot -p -e \ "CREATE DATABASE IF NOT EXISTS dstack_staging CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;" Install PHP dependencies (without --no-dev — Telescope needs dev packages) docker compose exec laravel_dstack composer install Run all migrations docker compose exec laravel_dstack php artisan migrate --force Bootstrap docker compose exec laravel_dstack php artisan config:cache docker compose exec laravel_dstack php artisan storage:link Generate Passport keys and create password grant client

docker compose exec laravel_dstack php artisan passport:keys docker compose exec laravel_dstack php artisan passport:client --password --name="DStack Password Grant" --no-interaction

Bug we hit: Running composer install --no-dev failed because Laravel Telescope is in require-dev and the TelescopeServiceProvider is registered in bootstrap/providers.php. If the package isn't installed, every artisan command throws a fatal error. Lesson: on staging, always run a full composer install. Only strip --no-dev on production where Telescope genuinely isn't needed.

Phase 5 — SSL with Let's Encrypt

This is the step most tutorials get wrong. You cannot have an nginx config referencing SSL certificates that don't exist yet — nginx will refuse to start.

The correct order:

Step 1 — HTTP-only config first

server {


    listen 80;


    listen [::]:80;


    server_name api-staging.dstack.ae;


    location /.well-known/acme-challenge/ { root /var/www/certbot; }


    location / { return 301 https://$host$request_uri; }


}

sudo ln -s /etc/nginx/sites-available/api-staging.dstack.ae /etc/nginx/sites-enabled/


sudo nginx -t && sudo systemctl reload nginx

Step 2 — Issue the certificate

sudo certbot certonly --nginx -d api-staging.dstack.ae

Step 3 — Replace with full HTTPS config

server {


    listen 443 ssl http2;


    listen [::]:443 ssl;


    server_name api-staging.dstack.ae;

ssl_certificate     /etc/letsencrypt/live/api-staging.dstack.ae/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/api-staging.dstack.ae/privkey.pem;
include             /etc/letsencrypt/options-ssl-nginx.conf;
ssl_dhparam         /etc/letsencrypt/ssl-dhparams.pem;

add_header X-Robots-Tag "noindex, nofollow" always;
client_max_body_size 100M;

location /realtime/ {
    proxy_pass         http://127.0.0.1:3010/;
    proxy_http_version 1.1;
    proxy_set_header   Upgrade    $http_upgrade;
    proxy_set_header   Connection "upgrade";
    proxy_set_header   Host              $host;
    proxy_set_header   X-Real-IP         $remote_addr;
    proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header   X-Forwarded-Proto $scheme;
    proxy_read_timeout 86400s;
}

location /app  { proxy_pass http://127.0.0.1:6001; ... }
location /apps { proxy_pass http://127.0.0.1:6001; ... }

location / {
    proxy_pass http://127.0.0.1:3006;
    proxy_http_version 1.1;
    proxy_set_header Host              $host;
    proxy_set_header X-Real-IP         $remote_addr;
    proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
}


}

Teaching note: Notice that / proxies to port 3006, not directly to Laravel. The Node.js gateway sits in front of Laravel and handles auth token validation, correlation ID injection, and request routing. Laravel is never directly exposed to the internet. This is the production architecture mirrored exactly.

Phase 6 — Staging Identity Headers

We added two more subdomains — grafana-staging.dstack.ae (monitoring) and realtime-staging.dstack.ae (WebSocket service). To make it immediately obvious these are staging, every nginx config adds identification headers:

add_header X-Robots-Tag  "noindex, nofollow"    always;


add_header X-Environment "staging"              always;


add_header X-Backend     "api-staging.taggo.ae" always;

You can verify this on any response:

curl -sI https://grafana-staging.dstack.ae/login | grep -E "x-environment|x-backend"


  
  
  x-environment: staging




  

  

  x-backend: api-staging.dstack.ae

Teaching note: X-Robots-Tag: noindex, nofollow prevents search engines from indexing staging. Without it, a staging endpoint once appeared in Google search results — exposing internal API structure. Defence in depth: even if Cloudflare DNS accidentally proxies staging, these headers are a last line.

Phase 7 — Database Seeding

With the stack running, seed the database so there's something to log in with.

cd /home/dstack/dstack Core: permissions, roles, admin + dummy users docker compose exec laravel_dstack php artisan db:seed Module seeders: booking config, locations, currency docker compose exec laravel_dstack php artisan db:seed \ --class="Modules\Locations\Database\Seeders\LocationsDatabaseSeeder" docker compose exec laravel_dstack php artisan db:seed \ --class="Modules\Currency\Database\Seeders\CurrencyDatabaseSeeder"

docker compose exec laravel_dstack php artisan db:seed <br> --class="Modules\BookingConfig\Database\Seeders\BookingConfigDatabaseSeeder"

Bug: Role assignment silently failed

After seeding, the admin user had no role. This is because Spatie Permission in this project is team-scoped — every role assignment requires a team_id. The seeder was running $user->assignRole('sys_admin') without setting the team context first.

// Wrong — team_id would be null, MySQL throws a NOT NULL constraint


$user->assignRole('sys_admin');

// Correct — set team context first


app(PermissionRegistrar::class)->setPermissionsTeamId($user->current_team_id);


$user->assignRole('sys_admin');

You can verify the actual state by checking model_has_roles directly:

docker compose exec laravel_dstack php artisan tinker --execute "


DB::table('model_has_roles')


    ->join('roles', 'roles.id', '=', 'model_has_roles.role_id')


    ->select('model_has_roles.model_id', 'roles.name as role', 'model_has_roles.team_id')


    ->get()->each(fn(\$r) => print(\$r->model_id.' | '.\$r->role.' | team='.\$r->team_id.PHP_EOL));


"

Teaching note: When a relationship returns empty in Eloquent but you're sure the data should be there, always go one level deeper and query the pivot table directly. Eloquent's $user->roles will silently return an empty collection if the team scope isn't set — no error, no warning. Direct SQL never lies.

Phase 8 — The Login That Refused to Work

This phase deserves its own section. Getting the admin login working required diagnosing six separate issues in sequence. Each one was a lesson.

Bug 1: Wrong endpoint

We called POST /api/v1/public/auth/login with the sys_admin credentials and got messages.invalid_user.

Reading the frontend login controller revealed:

$allowedRoles = [


    RoleEnum::agent_admin->name,


    RoleEnum::customer->name,


    RoleEnum::dealer->name,


    RoleEnum::driver->name,


    // sys_admin is intentionally NOT here


];

if (! $user->hasAnyRole($allowedRoles)) {


    return self::error(__('messages.invalid_user'), 401);


}

The admin has a dedicated endpoint: POST /api/v1/admin/auth/login.

Lesson: Before debugging credentials, always verify you're hitting the right endpoint. The route file is the source of truth — grep -n "login" routes/apiAdmin.php takes five seconds.

Bug 2: artisan serve was in FATAL state

The admin endpoint returned upstream_unavailable from the gateway. Testing directly inside the container confirmed port 81 was not listening. The Laravel container showed as running but no PHP process was active.

docker logs dstack-laravel_dstack-1 --tail 20


  
  
  WARN exited: php (exit status 255; not expected)




  

  

  WARN gave up: php entered FATAL state, too many start retries too quickly

Supervisor had given up on the PHP process because it crashed on startup before composer install ran (vendor/autoload.php didn't exist at container first boot). After we ran composer install, the vendor directory appeared — but supervisor had already stopped retrying.

Fix:

docker compose restart laravel_dstack



  

  

  Supervisor retries the PHP process fresh — this time vendor exists → success

Lesson: A container showing as "running" only means PID 1 (supervisor) is alive. The actual application process inside might be dead. Always verify with docker logs before assuming the app is up.

Bug 3: artisan serve visible, but login still returns 500

After restarting, the server was up but login returned "An error occurred. Please try again later.". The global exception handler was swallowing the real error:

// bootstrap/app.php

if (! app()->environment(['local', 'testing']) && $response->getStatusCode() === 500) {


    return response()->json(['status' => false, 'message' => 'An error occurred. Please try again later.'], 500);


}

On staging, every 500 becomes this generic message. We couldn't see the real error. Even Log::error() in the controller wasn't writing because:

Bug 4: Log file was root-owned, sail user couldn't write

ls -la storage/logs/


  
  
  -rw-r--r-- 1 root root 6310 Jul 18 10:57 laravel-2026-07-18.log




  

  

  ^^^^ sail cannot write here

The log file was created by an artisan command that ran as root (inside the container). The artisan serve process runs as the sail user. 644 means owner-write only. The sail user silently failed to write any logs.

docker exec dstack-laravel_dstack-1 chown sail:sail /var/www/html/storage/logs/laravel-2026-07-18.log

Now the real error appeared in the log:

[2026-07-18 16:44:06] staging.ERROR: Admin login token issue

{"error":"Key path \"file:///var/www/html/storage/passport/oauth-private.key\" does not exist or is not readable"}

Lesson: When debugging a silent 500, the very first thing to check is whether the log file is writable by the process that's serving requests. A log that can't write is worse than no log — it creates the illusion that nothing is wrong.

Bug 5: Passport private key was root-owned

passport:keys ran as root and created the keys with -rw------- root ownership. The sail user who runs artisan serve couldn't read the private key to sign JWTs.

docker exec dstack-laravel_dstack-1 ls -la /var/www/html/storage/passport/ -rw------- 1 root root 3322 oauth-private.key ← sail cannot read -rw-rw---- 1 root root 812 oauth-public.key Fix ownership and permissions docker exec dstack-laravel_dstack-1 chown sail:sail \ /var/www/html/storage/passport/oauth-private.key \ /var/www/html/storage/passport/oauth-public.key

docker exec dstack-laravel_dstack-1 chmod 600 /var/www/html/storage/passport/oauth-private.key docker exec dstack-laravel_dstack-1 chmod 644 /var/www/html/storage/passport/oauth-public.key

Teaching note: Passport (and any RSA/EC key-based system) requires strict permissions on the private key. Passport v13 specifically checks and will reject 644 on the private key — it wants 600 or 660. The public key can be world-readable (644) since it's meant to be distributed.

Bug 6: Missing Passport personal access client

After fixing the key permissions, login failed again with a different error. Running the token issuance code manually in tinker revealed:

RuntimeException: Personal access client not found for 'users' user provider. Please create one.

The passport:client --password we ran earlier created a password grant client. But $user->createToken() (which the app uses internally) requires a separate personal access client.

docker compose exec laravel_taggo php artisan passport:client <br>

  --personal <br>

  --name="DStack Personal Access Client" <br>

  --no-interaction

Verify the client was created with the correct grant type:

docker compose exec laravel_dstack php artisan tinker --execute "


DB::table('oauth_clients')->get()->each(fn(\$c) =>


    print(\$c->id.' | '.\$c->name.' | '.\$c->grant_types.PHP_EOL)


);


"




  

  

  019f7515... | DStack Personal Access Client | ["personal_access"]

Teaching note: In Passport v13, there are three distinct client types: password grant (for mobile apps using username+password), personal_access (for server-side token creation via $user->createToken()), and authorization_code (for OAuth flows). Most apps need all three. Running only passport:client --password is a common mistake that only surfaces when you try to programmatically issue tokens.

Final Verification

curl -s -X POST https://api-staging.dstack.ae/api/v1/admin/auth/login \

-H "Content-Type: application/json" <br>

  -H "Accept: application/json" <br>

  -d '{"email":"admin@dineshstack.com","password":"Admin@dstack2027"}'

{


    "status": true,


    "data": {


        "account_status": "active",


        "token": {


            "access_token": "eyJ0eXAiOiJKV1Qi...",


            "token_type": "Bearer",


            "access_token_expires_in": 86400


        },


        "user": {


            "email": "admin@dineshstack.com",


            "role": "sys_admin",


            "permissions": [...]


        }


    }


}

What the Final State Looks Like

URL	Service	Port
https://api-staging.dstack.ae	Laravel API (via Node gateway)	3006 → Laravel:81
https://grafana-staging.dstack.ae	Grafana monitoring dashboard	3003
https://realtime-staging.dstack.ae	WebSocket / node-realtime	3010

Docker: 19 containers, all on a single dstack_sail network.
Disk: 31 GB / 96 GB (32%) — recovered 39 GB from the old state.
SSL: Let's Encrypt on all three domains, auto-renewing.

Lessons Condensed

Survey before you touch anything. df -h, docker ps, docker network ls — know the battlefield.
Docker filter substrings bite. --filter name=dstack matches erpdstack. Always echo your list before piping to rm.
Order matters for SSL. HTTP-only config → certbot → HTTPS config. Never the other way.
depends_on: healthy is not a guarantee. If dependencies weren't healthy when up first ran, run it again. Idempotent.
"Running" container ≠ working app. Supervisor alive ≠ PHP alive. Check docker logs.
Log file permissions are invisible failures. A root-owned log file silently eats every error. Always verify ls -la storage/logs/ against the user running your web process.
Passport needs two clients. --password for mobile auth flows. --personal for server-side $user->createToken(). Both. Always.
RSA private keys must be readable by the web process user — and nothing else. 600 on the private key, owned by the process user. Non-negotiable.
Team-scoped permissions require team context on every operation. setPermissionsTeamId() before any role assign or check. Spatie fails silently without it.
When stuck on a generic 500, reproduce in tinker step by step. Narrow the failure to a single line before reading any framework source.

How to Set Up SSH Key Authentication on Mac (and Why You Should Never Use Passwords)

Dinesh Wijethunga — Sun, 30 Aug 2026 06:40:07 +0000

How to Set Up SSH Key Authentication on Mac (and Why You Should Never Use Passwords)

If you've been logging into servers with a username and password, you're one leaked credential away from losing that server. SSH key authentication isn't just "more secure" — it changes the entire threat model. Once you understand how it works, you'll never go back.

This guide walks you through the full setup on a Mac, step by step, with the reasoning behind every decision.

First, understand what you're actually building

SSH key authentication works like a padlock you put on a door. You keep the key (private key) on your laptop. You put the matching padlock (public key) on the server. When you connect, SSH proves you hold the key — without ever sending it over the network. Even if someone intercepts every byte of traffic, they learn nothing useful.

A password, by contrast, travels to the server on every login. One phishing attempt, one leaked database, one shoulder-surf — and it's gone.

Step 1 — Generate a dedicated key pair for this server

Most people generate one SSH key and reuse it everywhere. Don't. Use a separate key per server or per purpose. That way, if one key is ever compromised, you revoke it in one place without touching anything else.

ssh-keygen -t ed25519 -C "you@server-name" -f ~/.ssh/server_name

Breaking this down:

-t ed25519 — the algorithm. Ed25519 is the modern standard: shorter keys, faster, and cryptographically stronger than the older RSA 2048. Always use this unless the server is ancient.
-C "you@server-name" — a comment embedded in the public key. It doesn't affect security, but six months from now when you're looking at a server's authorized_keys file and see five entries, you'll be grateful you labeled them.
-f ~/.ssh/server_name — the filename. Be precise here. A typo (server-name vs server_name) means the next command will fail with "No such file or directory." Run ls ~/.ssh to confirm the file was created before moving on.

When prompted for a passphrase, set one. Your private key file is essentially a master password in a file — if someone copies it off your laptop, the passphrase is the only thing stopping them from using it. macOS Keychain will remember the passphrase for you, so you only type it once per reboot.

Step 2 — Copy the public key to the server

This is the step that "puts the padlock on the door." You're appending your public key to ~/.ssh/authorized_keys on the server — a file SSH checks on every connection attempt.

ssh-copy-id -i ~/.ssh/server_name.pub user@SERVER_IP

Notice the .pub extension — you're sending the public key, not the private one. Your private key (~/.ssh/server_name without .pub) never leaves your machine. Ever. If anyone asks you to send them your private key, that's a red flag.

This command will ask for your password one last time — it needs it to log in and place the key. After this step, you'll never need the password again.

Step 3 — Create a named alias in your SSH config

You could type ssh -i ~/.ssh/server_name user@192.168.1.100 every time. But you won't — you'll forget the flags, use the wrong key, and end up debugging authentication errors at 2am. Instead, write it down once in ~/.ssh/config:

Host server-alias

    HostName SERVER_IP

    User your_username

    IdentityFile ~/.ssh/server_name

    IdentitiesOnly yes

The line most tutorials skip explaining: IdentitiesOnly yes. Without it, SSH will try every key in your ~/.ssh/ folder, one by one, before trying the right one. On hardened servers that lock you out after a few failed attempts, this can get you banned by your own SSH client before the correct key is even tried. IdentitiesOnly yes says: use only the key I specified, nothing else.

Now test it:

ssh server-alias

If you get a shell prompt, the key is working. Do not proceed to Step 4 until this works.

Step 4 — Lock the door: disable password authentication on the server

Key login working is not enough. As long as password authentication is enabled, your server is still accepting passwords — and still vulnerable to brute-force attacks. This step removes that attack surface entirely.

Before you make any changes: open a second terminal and verify you can still ssh server-alias in. If you misconfigure sshd_config and lock yourself out, your existing session stays alive. That's your lifeline to fix it. Never edit SSH config with only one active session.

On the server, edit /etc/ssh/sshd_config:

PasswordAuthentication no

PermitRootLogin no

PubkeyAuthentication yes

ChallengeResponseAuthentication no

UsePAM no

Each line matters:

PasswordAuthentication no — the main goal. No more passwords accepted.
PermitRootLogin no — attackers always try root first. Remove the target entirely.
ChallengeResponseAuthentication no and UsePAM no — these close side doors that can sometimes re-enable password prompts even when PasswordAuthentication is off. Set all three.

Apply the changes:

sudo systemctl restart sshd

Then, from a new terminal on your laptop, run ssh server-alias. If it connects without asking for a password, you're done. If it asks for a password, something in the config didn't apply — go back and check for typos, then restart sshd again.

Step 5 — Add a final layer of protection against bots

Even with passwords disabled, bots will hammer port 22 all day long. It wastes resources and pollutes your logs. Two quick fixes:

# Auto-ban IPs that repeatedly fail authentication

sudo apt install fail2ban -y

sudo systemctl enable fail2ban --now

And optionally, change the SSH port in /etc/ssh/sshd_config:

Port 2222

Changing the port doesn't stop a determined attacker — a port scan finds it in seconds. But it eliminates 99% of automated bot traffic, which never scans beyond port 22. Update your ~/.ssh/config to match:

Host server-alias

    HostName SERVER_IP

    User your_username

    IdentityFile ~/.ssh/server_name

    IdentitiesOnly yes

    Port 2222

What you've actually built

You now have a server that:

Only accepts cryptographic proof of identity — no passwords, no brute-force surface
Refuses root login entirely
Auto-bans IPs that probe it repeatedly
Is reachable from your laptop with a single memorable alias

The mental model to take away: your private key is a physical key you never duplicate or hand to anyone. Your public key is a lock you can put on as many doors as you want. The config file is your keychain — it tells SSH which key to use for which door, without you having to think about it every time.

Do this for every server you ever touch. It takes five minutes and it will save you from the kind of incident that ends careers.

Originally published at dineshstack.com — read the full version with code samples and updates there.

The Messy Reality: 5 Dead Ends the AI Tutorials Skip

Dinesh Wijethunga — Wed, 26 Aug 2026 15:57:04 +0000

The bonus 6th part of the series on using Claude AI to run a real production server. Parts 1–5 told the clean story. This one tells the truth: the dead ends, the wrong guesses, and the moments the AI hit a wall — because that's what real AI-assisted work actually looks like.

The Messy Reality: 5 Dead Ends the AI Tutorials Skip (Part 6)

Every "I used AI to fix my server" post makes it look like a straight line: prompt, answer, done. The five parts before this one are cleaner than the real day was, because teaching needs a clear narrative. But if you're a new developer about to try this yourself, the polished version sets a false expectation — and then the first time your AI session hits a wall, you think you're doing it wrong.

You're not. Walls are normal. What separates a good outcome from a frustrating one isn't avoiding dead ends — it's how you and the AI recover from them. Here are five real ones from that day, and what each taught me.

Dead End 1: The AI Confidently Blamed the Wrong Thing

When a deploy failed with an SSH timeout, I asked the AI why. Its first theory was detailed and plausible: a firewall tool called fail2ban had probably banned the deploy server's IP after all my failed password attempts. It even gave me the command to check.

The command returned: fail2ban: command not found. fail2ban wasn't even installed. The entire theory was built on an assumption about the server that happened to be wrong.

The lesson: AI reasoning is only as good as its assumptions about your system. It wasn't lying — it was reasoning correctly from a false premise. The fix wasn't to trust the confident answer; it was to check it. The real cause turned out to be a transient network blip, found by actually reading the logs instead of accepting the first explanation. When an AI gives you a confident diagnosis, the move is to verify the premise before acting on the conclusion.

Dead End 2: The Terminal "Froze" and I Panicked

Early on, I pressed a key combination to scroll up through some output, and suddenly my terminal stopped responding. I typed — nothing appeared. For a solid minute I genuinely thought I'd broken the session and lost the AI's work.

I hadn't. I'd entered tmux's scroll mode, where the keyboard scrolls instead of typing. The escape was a single key: q.

The lesson: the scariest-feeling problems are often the most trivial. Nothing was frozen, nothing was lost — I was just in a mode I didn't recognise. Before assuming disaster, ask "am I in a different state than I think?" Half of "it's broken" moments for beginners are actually "it's in a mode you didn't know about."

Dead End 3: The AI Couldn't Do the Thing I Asked

Several times, the AI simply couldn't run a command — anything needing a root password failed, because its shell had no way to type one interactively. My first reaction was frustration: what good is an AI assistant that can't run sudo?

But this turned into the single most valuable pattern of the day. Instead of the AI having root power, the workflow became: the AI writes a careful, verified script; I run it in my own terminal where sudo works; I paste the results back for the AI to check.

The lesson: the AI's limitation was actually the safety feature. An AI that couldn't silently run root commands meant I stayed in control of everything that mattered, while still getting the AI's speed on the thinking. What felt like a bug was the guardrail working. Reframe the limitation and it becomes the reason the whole thing is safe.

Dead End 4: The Fix Worked, Then Immediately Broke Again

While fixing the login, I kept hitting the same class of error — "permission denied" writing a file — over and over, in different places. Fix one, another appears. It felt like whack-a-mole.

Eventually the pattern clicked (and the AI named it): the deploy created files as one user, but the app runs as a different user. Every file the app tried to write at runtime failed, and I was fixing them one at a time instead of seeing the shared root cause.

The lesson: when you're fixing the "same" error repeatedly in different spots, stop patching individual symptoms and look for the structural cause. The real fix wasn't ten chown commands — it was one reconciliation step in the deploy pipeline so it never happens again. If you're playing whack-a-mole, you're fixing symptoms, not the disease.

Dead End 5: The Data Was Perfect, and It Still Didn't Work

The final bug was the most maddening. Login worked, but every page returned "403 Forbidden — you don't have permission." So I checked the database: my user had the admin role, the role had all 60 permissions, every link was correct. The data was flawless. And it still failed.

The answer wasn't in the data at all — it was in the code. A route demanded a permission named manage-users, but the permission that actually existed was called users.view. A single naming mismatch. The check was looking for something that was never created.

The lesson: "the data looks correct" doesn't mean "the code agrees with the data." When everything should work but doesn't, the bug is often in the gap between two things that were each built correctly but don't match — a name, a version, a guard. That's where pointing the AI at the actual source code (not just the database) cracked it open.

What the Messy Version Actually Teaches

The clean series shows you what's possible. This one shows you what's normal. Every one of these dead ends ended fine — not because the AI was perfect, but because the process had a recovery move built in: verify the assumption, check the mode, reframe the limitation, find the pattern, read the code. AI didn't remove the friction of real engineering. It gave me a faster, sharper partner to work through the friction with.

If you take one thing from this whole series into your own AI-assisted work, let it be this: the AI is a brilliant collaborator, not an oracle. It will sometimes be confidently wrong, occasionally unable to act, and frequently one step away from the answer rather than holding it outright. Treat it like the sharpest junior engineer you've ever worked with — check its assumptions, keep the keys, and you'll get results that neither of you would reach alone.

Key Takeaways

A confident AI answer can rest on a false assumption — verify the premise before acting.
The scariest-feeling problems (a "frozen" terminal) are often trivial state confusion.
The AI's inability to run root commands is a safety feature, not a flaw — it keeps you in control.
Repeating the same error in different places means you're patching symptoms; find the structural cause.
"The data is correct" and "the code works" are different claims — bugs hide in the gap between them.

That's the honest end of the series. If the clean parts inspired you and this messy one reassured you, it's done its job. What's a dead end you hit that turned out to be a one-line fix? Share it in the comments — the war stories are where we all actually learn.

Originally published at dineshstack.com — read the full version with code samples and updates there.

The Login That Never Worked: Debugging 5 Layers Deep

Dinesh Wijethunga — Mon, 24 Aug 2026 18:54:04 +0000

Part 5, the finale of a 5-part series on using Claude AI to run, secure, and ship a real production server. The pipeline shipped (Part 4). Then I tried to log in.

The Login That Never Worked — Debugging Five Layers Deep with AI (Part 5)

The pipeline was green. Both releases were live. I opened my SaaS to see it working — and got a 502. Then, after fixing that, a "CORS error." Then a 500. Then an OAuth failure. Then a 403. Each fix revealed the next problem underneath, like peeling an onion that makes you cry five times.

This is the most instructive post in the series, because it shows what AI debugging actually looks like on a real system: not one magic answer, but a disciplined peeling of layers, where each error's real cause is hidden behind a misleading symptom. Here's all five, in order.

Layer 1: The 502 — A Stopped Process

The frontend returned 502 Bad Gateway. nginx proxies the site to a Node process on port 3003, and Claude's first check — pm2 list — showed the visa-saas process was stopped. The deploy's process-reload had left it in a bad state.

The fix wasn't a blind restart (that can hide a crash loop). Claude had me delete the corrupted process entry and start it fresh:

pm2 delete visa-saas
pm2 start ecosystem.config.js
pm2 save   # so it survives reboots

Process online, port 3003 answering. Lesson learned: the deploy's "reload" logic couldn't recover a stopped process — a real pipeline bug we noted for a follow-up fix. A green deploy that leaves the site down is worse than one that fails honestly.

Layer 2: The "CORS Error" That Wasn't CORS

Now the site loaded but login threw what the browser called a CORS error. Here's the trap Claude flagged that saves hours: when a Laravel API throws a 500, the error response often has no CORS headers — so the browser reports "CORS policy" when the real problem is the API crashing. The CORS message was a symptom, not the disease.

And the disease connected back to our Part 3 work. Remember the storage-permissions fix? On this new release, the storage folder was owned by the deploy user, but PHP-FPM runs as the web user — so PHP couldn't write logs or sessions, threw 500s, and the browser painted them as CORS. Claude spotted it because it read the actual error, not the browser's guess. One chown and the 500s vanished.

Layer 3: The OAuth Failure — A Wrong Client ID

"Client authentication failed"
League\OAuth2\Server\Exception\OAuthServerException

Laravel Passport couldn't issue a token. Claude traced it into the database. There were two OAuth clients:

ID	Client	Type
1	Password Grant Client	personal = 0
2	Personal Access Client	personal = 1

But the app's config pointed PASSPORT_PERSONAL_ACCESS_CLIENT_ID=1 — the wrong type. It was trying to issue a personal access token using a password-grant client. Changing the ID to 2 and rebuilding the config cache fixed it. Almost certainly a bug that had existed since setup — because with login broken, nobody had ever successfully authenticated on this deployment. Our work didn't break it; it was the first time anyone got far enough to hit it.

Layer 4: The Permission-Write 500s

Logged in! Then the dashboard threw 500s on loading modules. Same root cause as Layer 2, different file: the app writes a modules_statuses.json at runtime, owned by the wrong user, so the web process couldn't write it. Another targeted chown. This was the moment the deeper pattern became clear: the deploy creates files as one user, the app runs as another — the real fix is a permission-reconciliation step in the pipeline itself, so every future deploy doesn't reintroduce it.

Layer 5: The 403 — A One-Word Typo

The final boss. Every data endpoint returned 403 Forbidden — "You do not have the required permissions." But I was logged in as a super-admin. Claude checked the database: my user had the super-admin role, the role had all 60 permissions, everything linked correctly. The data was perfect. So why 403?

This is where I pointed Claude Code at the actual codebase, and it traced the exact chain. The route guarding /users required a permission named manage-users. But the seeder that creates permissions created users.view — the permission name the route demanded had never been created. A single naming mismatch between the route and the seeder. Every user in the system was locked out of that endpoint, and had been forever.

Route wants:    permission:manage-users   ← doesn't exist
Seeder creates: users.view                ← the real name

The fix: change the route guard to the permission that actually exists. And because AI is good at exactly this kind of grind, I had it grep the entire route file for other mismatches — so we could fix them all in one pass instead of discovering them one 403 at a time.

Why This Story Matters More Than a Clean Success

Five layers. A blank page that was a stopped process. A CORS error that was a permissions problem. An OAuth failure that was a config typo. A 403 that was a naming mismatch. Not one of these symptoms pointed at its own cause. That's what real debugging is — and it's exactly where an AI partner earns its place: reading the actual error instead of the misleading one, tracing a symptom to its true source, and grepping a codebase for every instance of a bug in seconds.

The AI didn't replace my judgment — I decided what to fix by hand, what to route through the pipeline, and when to stop and think. But it turned a bewildering cascade into a solvable sequence. And critically: bugs 3, 4, and 5 were pre-existing — they'd been in that codebase since setup, invisible because nobody could log in to trigger them. We didn't cause them. We were the first to reach them, and the AI helped root-cause each one instead of flailing.

The Whole Journey, Start to Finish

Over one day, with Claude AI as an investigate-and-verify partner, this server went from: an internet-exposed financial API, world-readable secrets across 20 projects, password-based SSH open to the world, no deployment pipeline, and a frontend that had never worked — to: a locked-down box, key-only SSH, a zero-downtime CI/CD pipeline, and a working login on a live dashboard.

Key Takeaways for New Developers

The symptom is almost never the cause. A 502, a CORS error, and a 403 all lied about what was really wrong.
Read the actual error, not the browser's interpretation of it.
AI is exceptional at tracing a symptom to its source and grepping a whole codebase for every instance of a bug.
Keep the human in charge of what to fix and where it belongs (hand-fix vs. pipeline); let the AI do the tracing and the grind.
A working system revealing old bugs isn't a regression — it's progress. You can only find the login bug once login gets far enough to fail.

That's the series. One developer, one AI partner, one very real production server — audited, secured, shipped, and debugged, the honest way, with a paper trail at every step. If you're nervous about letting AI near your infrastructure, I hope this showed you the pattern that makes it not just safe but genuinely powerful: the AI gets the brains, you keep the keys.

Thanks for reading all five parts. If this helped, the whole series is built to be followed step by step on your own server. What would you point an AI agent at first — a security audit, or that one bug you've been avoiding?

Originally published at dineshstack.com — read the full version with code samples and updates there.

How to Build a Zero-Downtime CI/CD Pipeline with an AI Pair

Dinesh Wijethunga — Sat, 22 Aug 2026 07:52:06 +0000

Part 4 of a 5-part series on using Claude AI to run, secure, and ship a real production server. The server is now secure (Parts 1–3). Time to ship code the right way.

Build a Zero-Downtime CI/CD Pipeline with an AI Pair (Part 4)

Up to now I'd been deploying by hand — SSH in, pull, hope. For my main SaaS project I wanted the real thing: push to GitHub, tests run automatically, and if they pass, the server swaps to a fresh release with a symlink so rollback is instant. In this post, Claude plays two roles: code-review partner for my pipeline, and server-prep engineer to get the box ready — using the same investigate-then-execute pattern from the earlier parts.

The Architecture (Capistrano-Style Releases)

The idea is simple and powerful. Instead of overwriting your live app in place, each deploy lands in a timestamped folder, and a symlink points to the current one:

/var/www/visa-saas/
├── releases/
│   ├── 20260709224220/   ← previous release
│   └── 20260712172016/   ← new release
├── shared/               ← .env, storage (survive across deploys)
├── api  → releases/20260712172016   ← symlink, flipped atomically
└── web  → releases-web/...           ← same idea for the frontend

Deploy = build a new release folder, then flip the symlink. Rollback = flip the symlink back. That's the whole magic: rollback becomes one command instead of a panic.

Claude as Code Reviewer: What It Caught in My Pipeline

I had draft GitHub Actions workflows and asked Claude to review them like a senior engineer. It found real problems I'd have shipped:

Issue Claude flagged	Why it mattered
Web deploy did `rm -rf` then move — no rollback	The old release was destroyed at activation. One bad build = no way back.
Prune step would eventually delete the original backup	"Keep 5 newest" quietly wipes your `initial` safety copy after 5 deploys.
Bare `php` on a server with 6 PHP versions	Migrations could run under the wrong PHP than the site serves.
Migrations with no database backup first	A symlink rollback can't undo a schema change — you need a dump.
Third-party Actions pinned to tags, not commit SHAs	A moved tag could inject malicious code into a job that holds your SSH key.

It also caught the classic PHP-plus-symlink gotcha: without the right nginx setting ($realpath_root), OPcache can keep serving old code after you flip the symlink. That's the kind of subtle, experience-earned detail that makes an AI reviewer genuinely valuable — not because it's magic, but because it's read every variation of this mistake.

The Security Angle Nobody Writes Down

One point Claude raised that I hadn't considered: once GitHub Actions can SSH into production, anyone who can push a workflow change can run code on your server. So the deploy key got its own dedicated keypair, restricted in authorized_keys (no-agent-forwarding,no-port-forwarding), and the deploy job was gated behind a protected GitHub environment. Your CI is only a safety gate if nothing can route around it.

Server Prep: Investigate, Then One Script

Before merging, the server needed preparing — release directories, a database-backup folder, the nginx storage path, a narrow passwordless-sudo rule for reloading PHP. True to the pattern, Claude did a read-only readiness report first, and it changed the plan: three things I'd assumed needed fixing were already correct. Had we skipped straight to scripting, we'd have installed a duplicate sudo rule and "fixed" a setting that was already right. The verify-first phase paid for itself again.

Then Claude generated one prep script with its now-familiar signature: timestamped backups, diff checkpoints, per-step verification, and printed (never auto-run) rollback commands. It even excluded Passport's private keys from a bulk permission change so they'd stay locked at 600 — a detail I'd have missed.

Merge, and the First Real Deploy

All checks green, PR merged. Both API and web pipelines fired. Watching the symlinks flip in real time was the payoff:

watch -n 2 'readlink /var/www/visa-saas/api; readlink /var/www/visa-saas/web'
# api  → releases/20260712172016
# web  → releases-web/20260712174334   ← it flipped!

The First Failure (And Why It Didn't Matter)

The web deploy failed on its first attempt — an SSH i/o timeout uploading the release. My first instinct was "the hardening broke CI." But we checked the logs instead of guessing: the deploy key authenticated fine; it was just a transient network blip on one job. A re-run sailed through.

The crucial point: because of the release architecture, the site never went down during that failure. The old release stayed live until the new one was ready to flip. A failed deploy on this design is a non-event, not an outage. That safety net is the entire reason to build it this way.

Key Takeaways

Timestamped releases + a symlink flip turn rollback from a crisis into one command.
An AI code reviewer shines at catching the subtle, experience-earned mistakes — missing backups, wrong PHP, OPcache traps, supply-chain risks.
When CI can reach production, treat the deploy key like the sensitive credential it is.
Investigate-then-script saved us from "fixing" things that were already fine — twice.
A good pipeline fails safely: the live site stays up until the new release is proven.

The pipeline worked. Both releases were live. And then I opened the site to admire it — and the login was broken. Not a little broken. A cascading, five-layers-deep broken that started as a blank page and ended at a one-word typo buried in a route file. That debugging journey — the most instructive part of the whole day — is Part 5.

👉 Coming up in Part 5: "The Login That Never Worked — Debugging Five Layers Deep with AI." What's the worst production bug you've shipped through a green pipeline?

Originally published at dineshstack.com — read the full version with code samples and updates there.

Laravel queue jobs not processing: wrong connection

Dinesh Wijethunga — Fri, 21 Aug 2026 11:44:04 +0000

If your jobs table is growing while the queue worker sits there reporting healthy, the two processes are reading and writing different queues. Laravel lets you set the connection in two places, and the one in the worker's command line silently wins over the one in your environment file.

Ours disagreed for nineteen days. It cost 55,470 orphaned jobs, every transactional email in that window, and — the part I find hardest to defend — nobody noticed, because from every angle the system looked fine.

What we saw

We were auditing a production database before an unrelated migration when a count came back wrong:

SELECT COUNT(*) FROM jobs;
-- 56588

Fifty-six thousand pending jobs. The oldest was nineteen days old. The newest was thirty seconds old and there was a fresh one every thirty seconds, forever.

The queue worker container had been up for weeks. Its logs showed a clean supervisord boot and nothing else — no errors, no warnings, no processed jobs. Every health check passed. The application served traffic normally. Customers were completing orders.

The arithmetic told us where to look before the configuration did. One job class accounted for 55,470 of the rows, and the scheduler dispatched it every thirty seconds:

2 per minute × 60 × 24 = 2,880 per day
2,880 × 19 days ≈ 54,720

That is not a job failing and retrying. That is a job being enqueued perfectly and never once being read, since the day the scheduler entry went live.

The two places a queue connection is set

The environment file said one thing:

QUEUE_CONNECTION=database

The worker's container definition said another, hardcoded months earlier and never revisited:

SUPERVISOR_PHP_COMMAND: "php /var/www/html/artisan queue:work redis
  --queue=high,default,low --sleep=3 --tries=3 --max-time=3600"

That first argument to queue:work is the connection name, and it overrides QUEUE_CONNECTION completely. The behaviour is documented and genuinely useful — it is how you run separate workers against separate backends. It is dangerous only because it is set in a different file, in a different repository concern, from the value it overrides.

So the application dispatched jobs into MySQL. The worker polled Redis, found nothing, slept three seconds, and polled again. It did that several million times without complaint.

Why nothing logged an error

This is the part worth internalising, because it generalises far beyond queues.

Nothing failed. Dispatch wrote a row and returned success — that is what dispatch does. The worker blocked on an empty list and returned success — that is what polling an empty queue does. An error requires some component to attempt something impossible, and neither component ever attempted anything impossible. Each half was working correctly. The system was broken only in the relationship between them, and nothing in the stack is responsible for that relationship.

Every monitor we had was pointed at a component. Container health: passing. Error rate: zero. Database: fine. Not one of them was pointed at the contract.

Why the damage was smaller than it should have been

The dominant job class was an outbox recovery job — a safety net that re-publishes events whose delivery was not confirmed after the transaction committed. So we checked what it would have had to recover:

SELECT status, COUNT(*) FROM outbox_messages GROUP BY status;
-- sent  1668

Every row sent. Zero pending. The primary publish path had a perfect record for the entire window, which is why nineteen days of a dead safety net produced no visible symptom.

We got away with it. Read that sentence as the accusation it is: we did not detect the failure, we were rescued by the fact that the thing it protected never needed protecting. Had the primary path faltered once during those nineteen days, the recovery mechanism would have been sitting in a MySQL table watching it happen.

The rest of the backlog was less lucky. Several hundred registration notifications and booking-status emails were in there. Those never sent, and nobody filed a ticket — which tells you something uncomfortable about how much of that mail anyone was reading.

Clearing it: why we did not just release the backlog

The instinct on finding 56,000 stuck jobs is to point a worker at them and let them drain. We deliberately did not.

Those jobs were nineteen days stale. Releasing them would have delivered hundreds of "welcome, you've registered" emails to people who registered three weeks ago, and status updates for rides that finished long before. Delivering a stale message is a worse outcome than never delivering it, and unlike the silent failure, customers would definitely have noticed that one.

So: group by class, decide per class, back up, then discard.

mysqldump --single-transaction app_prod jobs failed_jobs | gzip > jobs-backup.sql.gz
TRUNCATE TABLE jobs;

The backup makes the decision reversible for the cost of a few megabytes. Take it even when you are confident, because the confidence is about the job classes you identified, not the ones you skimmed past.

Then the fix, which was one line — pointing the application at the connection the worker had been watching all along. Jobs began clearing in two to four milliseconds each.

The same bug is not equally dangerous everywhere

Here is the detail that changes how you should weight this. We fixed the mismatch as part of moving the queue onto Redis, and that move altered the failure's blast radius entirely.

In MySQL, 56,000 orphaned jobs were a large table on a disk with hundreds of gigabytes free. Genuinely harmless — which is precisely why it survived nineteen days.

On Redis, the identical bug consumes memory, and queue entries carry no TTL. They sit there until a worker takes them. On a shared box without swap, unbounded memory growth does not politely degrade; it reaches a limit and something gets killed, and the process the kernel selects is chosen by size rather than by blame — frequently your database rather than the cache that caused it.

Same misconfiguration, same silence, radically different consequence. Moving a queue to a faster substrate also moves it to a less forgiving one. If you are making that migration, fix your queue-depth monitoring first, not afterwards.

You cannot alert on a metric that does not exist

The obvious follow-up is an alert on queue depth. We went to add one and found the gap was one level deeper than expected.

The Redis exporter reports totals — memory, client count, keys per database — but it does not publish the length of any individual list unless you name it explicitly:

REDIS_EXPORTER_CHECK_KEYS: "queues:*"

Until that line existed there was no queue-depth metric in Prometheus at all. A dashboard would have shown a healthy Redis for all nineteen days, because every metric it displayed was genuinely healthy. An absent metric and a good metric look identical on a graph.

The alert we wrote was wrong, and testing caught it

A healthy queue drains in milliseconds, so any depth that survives a long window means nothing is consuming it. That was the rule:

min_over_time(redis_key_size{key=~"queues:.*"}[30m]) > 10

We tested it by planting a synthetic backlog. It went pending, correctly. Then we deleted the key — and it stayed pending.

min_over_time keeps returning samples from its whole window after a key disappears. So any burst that got scraped once would fire this five minutes later and hold it for half an hour: exactly the false-positive noise that teaches a team to ignore alerts. Pairing it with a check for a currently-present sample fixes it, because an emptied Laravel queue deletes its Redis list and the series simply stops:

redis_key_size{key=~"queues:.*"} > 10
  and
min_over_time(redis_key_size{key=~"queues:.*"}[30m]) > 10

Re-tested: pending with a backlog, inactive the moment it drains. An alert is a piece of production code, and an untested one is likelier to erode trust than to protect anything.

What to check on your own system

Two commands, worth running now rather than during an incident. What the application believes:

php artisan tinker --execute="echo config('queue.default');"

And what the worker is actually executing — read the process, not the config file that you believe produced it:

docker compose exec app_queue ps aux | grep queue:work

If those two disagree, you have this bug, and your logs will not tell you. Then confirm something is genuinely draining, rather than that nothing has arrived: a depth of zero and a broken consumer look the same from outside.

The principle

Health checks verify components. This failure lived in the space between two healthy components, where nothing was looking, and the only honest signal available was a number that nobody was collecting.

For any handoff between two processes, monitor the queue between them rather than the processes themselves. Depth over time is the cheapest true statement you can make about a distributed system: it goes up when the producer outruns the consumer, and it stays up when the consumer is gone. Neither of those facts is visible from either end alone.

Originally published at dineshstack.com — read the full version with code samples and updates there.

Locking Down Secrets and SSH with AI (and the Cloud-Init Trap)

Dinesh Wijethunga — Thu, 20 Aug 2026 11:18:04 +0000

Part 3 of a 5-part series on using Claude AI to run, secure, and ship a real production server. Part 2 closed an exposed API. Now we tackle the biggest structural risk the audit found — and lock the front door.

Locking Down Secrets and SSH with AI — and the Trap That Almost Fooled Us (Part 3)

The audit's number-one risk wasn't dramatic, but it was the scariest: across nearly 20 projects, the .env files — the ones holding database passwords and API keys — were world-readable. Any process, any local user, any path-traversal bug in the weakest app could read every other client's secrets. This post is how Claude and I fixed all of them at once, then shut off password-based SSH entirely — and hit a trap that silently tried to undo the whole thing.

Why World-Readable .env Files Are a Slow-Motion Disaster

A .env at mode 644 means "owner can write, everyone can read." On a server with one project, that's sloppy. On a server with 20 unrelated client projects sharing a web user, it means a single vulnerability anywhere gives an attacker every tenant's credentials. The fix is simple — 640, owned by the right user — but doing it across 20 live sites without breaking any of them takes care.

Claude's Approach: One Script, Verify Every Step

I asked Claude to write a single script following the same safety pattern from Part 2: back up, change, verify, and — crucially — check every website still works after each change. The clever part it added on its own: a baseline pass that curls every site before touching anything, so a site that was already broken wouldn't be misreported as something the script broke.

# For each project: capture current perms, fix, verify the live site
for project in "${PROJECTS[@]}"; do
  # backup-aware: record old mode/owner first
  sudo chown www-data:www-data "$project/.env"
  sudo chmod 640 "$project/.env"
  # then curl the site and compare against the pre-change baseline
done

Claude also caught two things a blanket script would have broken:

Next.js apps under pm2 run as a different user than PHP apps — chowning their .env to the web user would lock out the process that reads it. Claude special-cased those.
One .env shared by two apps (a Laravel API and a Next.js frontend) needed split ownership so both could still read it while dropping public access.

That's the difference between "AI runs a chmod loop" and "AI understands the runtime." It ran across 18 projects plus a world-writable storage directory and a trading-bot secret — every one verified against baseline, zero regressions.

The Bonus: Baseline Checks Found Pre-Existing Problems

The baseline pass earned its keep immediately. Before changing a thing, it revealed three sites that were already broken — two APIs returning 500 errors and one domain that wouldn't connect at all. None caused by the script; all surfaced by it. That's a lovely side effect of doing things carefully: you discover problems you didn't know you had.

Then: Locking SSH to Keys Only

Next, the front door. The server still accepted password logins from the entire internet — meaning it was brute-forceable. The plan: install a personal SSH key, prove it works, then disable passwords. Order matters, because a mistake here locks you out of your own server.

# 1. On my Mac: create a key if I didn't have one
ssh-keygen -t ed25519

# 2. Install it on the server
ssh-copy-id deploy_user@SERVER_IP

# 3. PROVE key auth works BEFORE disabling passwords
ssh -o PasswordAuthentication=no deploy_user@SERVER_IP

Only after that login succeeded did we disable passwords in sshd_config. And this is where the trap sprang.

⚠ The Cloud-Init Trap That Almost Fooled Us

I set PasswordAuthentication no in the main SSH config. Clean. Done, right? Claude insisted on one more check — grepping all the config, including the /etc/ssh/sshd_config.d/ drop-in directory. And there it was:

/etc/ssh/sshd_config:65:            PasswordAuthentication no
/etc/ssh/sshd_config.d/50-cloud-init.conf:1: PasswordAuthentication yes   ← !!!

A cloud-init drop-in file was setting PasswordAuthentication yes — and here's the killer detail: SSH reads the "Include" directive near the top of the config, so the drop-in file's setting wins over the main file. Without catching this, my "hardening" would have changed nothing. Password auth would have stayed wide open while I believed it was closed. Claude also caught that root login was still enabled and flagged that too.

We fixed both files, validated with sshd -t (test before reload, like nginx -t), reloaded, and then ran the two-sided proof from my laptop:

ssh deploy_user@SERVER_IP 'echo OK'          # key works
ssh -o PubkeyAuthentication=no deploy_user@SERVER_IP   # password refused
ssh root@SERVER_IP                            # root refused

Result: key works, password bounces, root bounces. Three for three.

The Deeper Lesson: Verify the Baseline, Not Just Your Changes

This was the second time that day a "should already be hardened" assumption turned out false when Claude actually checked. That's the real discipline this whole experience taught me: verify-don't-assume applies to the starting state, not just to the changes you make. The cloud-init file had been silently overriding intent for who knows how long. An AI that grepped everything instead of trusting the obvious file is what caught it.

Key Takeaways

World-readable .env files are the quiet #1 risk on a multi-project box. Fix to 640, but verify each site still runs after.
Always baseline before a bulk change, so you can tell your breakage apart from pre-existing breakage.
Install and prove your SSH key before disabling passwords — keep your current session open as a lifeline.
On Ubuntu, always grep sshd_config.d/ — a drop-in file can silently override your main config.

The server was now genuinely locked down: secrets protected, SSH key-only, root disabled — all verified from outside. With the house secure, it was time to build something: a proper CI/CD pipeline so I could ship code changes with zero downtime and one-command rollbacks. That's Part 4, where Claude becomes a code-review partner and we ship a real deployment pipeline — and hit our first live deploy failure.

👉 Coming up in Part 4: "Building a Zero-Downtime CI/CD Pipeline with an AI Pair." Run sudo grep -r PasswordAuthentication /etc/ssh/ right now — are you sure it says what you think it does?

Originally published at dineshstack.com — read the full version with code samples and updates there.