DEV Community: Alexey

SSH died. Spent 3 hours fixing the wrong thing.

Alexey — Wed, 20 May 2026 15:38:14 +0000

Three or four days ago I deployed a Gitea runner, tested it, closed the terminal and moved on. Today, for some reason I don't even remember, I tried to SSH into the server. Nothing.

Opened the browser to check if the server was alive — course project running, Gitea up, Nextcloud up, everything in the k3s cluster fine. Went into VMmanager, rebooted. Still couldn't get in. Closed The Boys and went to fix it through VNC.

nmap first

nmap -p 22 2.27.42.100

PORT     STATE    SERVICE
22/tcp   filtered ssh

filtered — not closed, not refused. Packets reaching the server and getting silently dropped. Firewall somewhere.

Checked the INPUT chain:

Chain INPUT (policy DROP)
1    KUBE-ROUTER-INPUT
2    ACCEPT     tcp dpt:22
3    KUBE-PROXY-FIREWALL
...
8    ufw-before-input

ACCEPT rule for port 22 at position 2. Looked fine.

Wasn't fine.

Where it actually breaks

nft list ruleset | grep -E "22|drop|DROP"

type filter hook input priority filter; policy drop;
...
tcp dport 22 counter packets 0 bytes 0 accept

Zero packets on the accept rule. Connections weren't reaching it at all.

The server has both iptables-nft (kube-router, UFW) and native nftables running at the same time. They fight. kube-router constantly reconciles and overwrites stuff. The output was already warning about it:

# Warning: table ip filter is managed by iptables-nft, do not touch!

The Gitea runner deploy from a few days ago triggered a reconciliation that messed up the rule ordering. I just hadn't tried SSH since then.

The fix

Native nftables table at priority -150. Runs before any filter chains (priority 0), before kube-router, before everything. kube-router only manages the iptables-nft layer — doesn't touch native nftables tables.

nft add table inet ssh_rescue
nft 'add chain inet ssh_rescue input { type filter hook input priority -150; }'
nft add rule inet ssh_rescue input tcp dport 22 accept

SSH came back. Wrote to my friend: "fixed". He said "eee". Made it a systemd service. Two minutes later wrote to him: "SSH died again".

Round two

Back in VNC. Table still there, rules still there, zero packets on the accept rule again.

systemctl start nftables had loaded the saved /etc/nftables.conf which had kube-router's warning comments in it and corrupted the nftables state. Re-added the table manually. SSHed from the server to itself — worked. sshd on 0.0.0.0:22 — fine. Still couldn't connect from outside.

Then checked fail2ban, which had installed itself as a dependency somewhere during all this. It had banned two IPs — bots hammering port 22 the whole time. My repeated failed attempts got caught in the same drop. Ran fail2ban-client unban --all, connected immediately.

Then SSH died again. Disabled fail2ban completely — still nothing.

The actual reason

I was in the middle of debugging nftables chains when I remembered — my VPN is self-hosted on a server in Finland. The day before I'd been complaining to my friend that the VPN was slow. The provider had sent an email:

Due to an increase in brute-force complaints via SSH (port 22), outgoing traffic on port 22 has been blocked for VPS servers located in Finland.

I had read it. I knew about it. And I spent three hours in VNC chasing nftables, fail2ban and kube-router while the answer was sitting in my inbox the whole time.

Turned off the VPN. SSH connected immediately.

What to do from day one

Add the ssh_rescue table before you need it:

nft add table inet ssh_rescue
nft 'add chain inet ssh_rescue input { type filter hook input priority -150; }'
nft add rule inet ssh_rescue input tcp dport 22 accept

Make it a service so it survives reboots and kube-router reconciliation:

cat > /usr/local/bin/ssh-rescue.sh << 'EOF2'
#!/bin/bash
nft list table inet ssh_rescue 2>/dev/null || {
    nft add table inet ssh_rescue
    nft add chain inet ssh_rescue input { type filter hook input priority -150; }
    nft add rule inet ssh_rescue input tcp dport 22 accept
}
EOF2
chmod +x /usr/local/bin/ssh-rescue.sh

After any deploy that touches networking — run nmap -p 22 <ip> immediately.

And read your damn emails.

I got tired of setting up Go projects from scratch, so I built a scaffolding CLI

Alexey — Sun, 17 May 2026 13:37:39 +0000

Before I wrote any actual business logic in my last project, I spent days just setting up the infrastructure around it. Parsing configs, writing Dockerfiles, wiring up Compose, connecting Postgres and Redis, getting the server to actually start. At some point I just gave up and went to play Dota instead.

I'd done this before in .NET — dotnet new generates a working project from a template in seconds. Go has nothing like that out of the box. So I built it.

It's called gofro.

What it does

One command:

gofro new myapi --postgres --redis --grafana --github johndoe --git

You get:

cmd/main.go with graceful shutdown wired up
Config loading via koanf (TOML + env vars, env wins)
Docker Compose with only the services you asked for
Multi-stage Dockerfile
pgxpool connection + goose migrations runner (if --postgres)
go-redis client (if --redis)
Prometheus scrape config + Grafana in Compose (if --grafana)
OpenAPI spec + oapi-codegen setup so you can define your API and generate typed handlers
Generic storage layer — GetAll, GetOne, Create, Update, Delete over pgx
Module path set to github.com/johndoe/myapi automatically
git init if you pass --git

Then go mod tidy and docker compose up -d and you're writing actual code.

Why I built this instead of using something existing

Coming from .NET, dotnet new was just there. Pick a template, get a project, start working. In Go I kept doing the same dance every time: copy the config parser from the last project, rewrite the Compose file, remember how pgxpool initialization works, set up graceful shutdown again.

The last time I did it was for a college logistics platform. I had a tight deadline, hadn't touched Go in a while after a stressful hackathon, and spent way too long just building the scaffolding before writing a single handler. That's when I decided to just solve it once.

gofro is opinionated. It picks the stack for you — chi for routing, koanf for config, pgx for Postgres, goose for migrations, oapi-codegen for API generation. If you want something different, it's probably not for you. But if you're fine with that stack, you skip a day of setup.

The flags

--postgres    pgxpool + goose migrations + generic storage layer
--redis       go-redis/v9 client
--prometheus  Prometheus scrape config
--grafana     Grafana in Compose (enables --prometheus automatically)
--github      sets module path to github.com/<nick>/<project>
--module      full custom module path
--git         runs git init

You can mix and match. Minimal API with no databases:

gofro new myapi --github johndoe

Full stack with observability:

gofro new myapi --postgres --redis --prometheus --grafana --github johndoe --git

What the generated project looks like

myapi/
├── cmd/
│   └── main.go              # graceful shutdown, signal handling, dependency wiring
├── configs/
│   ├── config.toml          # base config
│   └── prometheus.yml       # only with --prometheus
├── internal/
│   ├── api/
│   │   ├── api.swagger.yaml # define your endpoints here
│   │   ├── gen.go           # go:generate directive for oapi-codegen
│   │   └── oapi-codegen.yaml
│   ├── config/
│   │   └── config.go        # struct + koanf loader + DSN helpers
│   ├── database/
│   │   ├── postgres.go      # only with --postgres
│   │   └── redis.go         # only with --redis
│   └── handler/
│       └── server_impl.go   # Server struct, JSON(), Run()
├── pkg/
│   └── storage/
│       └── storage.go       # only with --postgres
├── migrations/              # only with --postgres
├── docker-compose.yml
├── Dockerfile
├── .env
└── Makefile

The workflow after generation:

Define your API in internal/api/api.swagger.yaml
Run make generate — oapi-codegen produces typed models and the server interface
Implement the interface methods on the Server struct
Run docker compose up -d and go run ./cmd/

The generic storage layer

This is the part I use in every project. It's built on go-sqlbuilder and pgx, works with any struct via generics:

// SELECT with optional filter
users, err := storage.GetAll[User](ctx, "users", db,
    func(sb *sqlbuilder.SelectBuilder) {
        sb.Where(sb.Equal("active", true))
    },
)

// SELECT one
user, err := storage.GetOne[User](ctx, db, "users",
    func(sb *sqlbuilder.SelectBuilder) {
        sb.Where(sb.Equal("id", id))
    },
)
if errors.Is(err, storage.ErrNotFound) { ... }

// INSERT
err = storage.Create(ctx, "users", newUser, db)

// UPDATE — skips fields tagged `immutable` (like created_at)
err = storage.Update(ctx, "users", user, db,
    func(sb *sqlbuilder.UpdateBuilder) {
        sb.Where(sb.Equal("id", user.ID))
    },
)

Fields tagged db:"-" are skipped on insert. Fields tagged immutable are skipped on update. No magic, just generics and struct tags.

Install

go install github.com/anxi0uz/gofro@latest

Make sure $(go env GOPATH)/bin is in your $PATH. Then:

gofro new myproject --postgres --redis --github yourname --git
cd myproject

Source: github.com/anxi0uz/gofro

My friend wanted GitLab. He got Gitea and Nextcloud for Obsidian instead.

Alexey — Thu, 14 May 2026 14:22:41 +0000

A friend sent me an article about GitHub potentially getting blocked in Russia and asked me to spin up GitLab. I suggested Gitea — I'd used it at a college hackathon, knew it was lightweight and wouldn't eat half the server. He agreed.

While the deploy was running, I asked him how he syncs Obsidian. He said — plain WebDAV, nothing fancy. Well, server's already open anyway, so I threw in Nextcloud too. Hour and a half later I had both a git host and a cloud storage.

Why not GitLab

My friend originally wanted GitLab. I opened the docs, looked at the requirements — 4 GB RAM just to start — and said no. We don't have a dedicated git server, there's already a project running on it. GitLab would've eaten everything.

Gitea idles at ~150 MB. Actions are compatible with GitHub Actions syntax, so existing workflows move over without rewriting. I'd already used it at a hackathon, knew it worked fine. Suggested it, got the green light.

On Helm

First time I touched Kubernetes was when I had to deploy a college project for a grade. Wrote manifests by hand — Deployment, Service, Ingress, PVC, repeat. I knew Helm existed but never had a reason to dig into it.

Turns out it's like pacman, but for Kubernetes. One values.yaml instead of five hundred lines of YAML, one command — everything's up. Would've lost my mind doing it the old way.

First though, Helm couldn't see the cluster at all:

Kubernetes cluster unreachable: Get "http://localhost:8080/version"

k3s puts the kubeconfig somewhere Helm doesn't look by default. Fix:

export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
echo 'export KUBECONFIG=/etc/rancher/k3s/k3s.yaml' >> ~/.bashrc

Gitea

The server already had ingress-nginx and cert-manager with a letsencrypt-prod ClusterIssuer. Set the DNS beforehand — A record git.logiflowadvanced.online → 2.27.42.100.

gitea-values.yaml:

ingress:
  enabled: true
  ingressClassName: nginx
  hosts:
    - host: git.logiflowadvanced.online
      paths:
        - path: /
          pathType: Prefix
  tls:
    - secretName: gitea-tls
      hosts:
        - git.logiflowadvanced.online
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod

gitea:
  admin:
    username: admin
    password: yourpassword
    email: your@email.com

persistence:
  size: 10Gi

postgresql-ha:
  enabled: false

postgresql:
  enabled: true

helm repo add gitea-charts https://dl.gitea.com/charts/
helm repo update
helm install gitea gitea-charts/gitea \
  --namespace gitea --create-namespace \
  -f gitea-values.yaml

Pods came up. Opened the browser — invalid certificate, browser complaining. Checked the Ingress:

NAME    CLASS    HOSTS                          ADDRESS   PORTS
gitea   <none>   git.logiflowadvanced.online              80, 443

CLASS: <none> — the nginx controller just ignored this Ingress entirely. cert-manager didn't issue anything either, so nginx was serving its default self-signed cert.

kubectl patch ingress gitea -n gitea \
  --type=json \
  -p='[{"op":"add","path":"/spec/ingressClassName","value":"nginx"}]'

After the patch, cert-manager issued the certificate and the site opened fine.

SSH

gitea-ssh is created as a headless ClusterIP by default — not reachable from outside. Port 22 is taken by the system SSH, so I needed a NodePort. You can't patch a headless service into NodePort — have to delete and recreate:

kubectl delete svc gitea-ssh -n gitea

kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: gitea-ssh
  namespace: gitea
spec:
  type: NodePort
  selector:
    app.kubernetes.io/name: gitea
  ports:
    - port: 22
      targetPort: 2222
      nodePort: 30022
EOF

Remote looks like this:

git remote add gitea ssh://git@git.logiflowadvanced.online:30022/username/repo.git

Nextcloud

Set up the DNS for cloud.logiflowadvanced.online while Gitea was still deploying.

nextcloud-values.yaml:

ingress:
  enabled: true
  ingressClassName: nginx
  hosts:
    - host: cloud.logiflowadvanced.online
      paths:
        - path: /
          pathType: Prefix
  tls:
    - secretName: nextcloud-tls
      hosts:
        - cloud.logiflowadvanced.online
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/proxy-body-size: "0"

nextcloud:
  host: cloud.logiflowadvanced.online
  username: admin
  password: yourpassword

mariadb:
  enabled: true
  auth:
    password: yourdbpassword
    database: nextcloud
    username: nextcloud

postgresql:
  enabled: false

persistence:
  enabled: true
  size: 10Gi

helm install nextcloud nextcloud/nextcloud \
  --namespace nextcloud --create-namespace \
  -f nextcloud-values.yaml

Same ingressClassName issue — same patch, same result.

After the first login it kept redirecting to /login/cleary?=1. Nextcloud didn't know its own external address:

kubectl exec -n nextcloud deploy/nextcloud -- \
  php occ config:system:set overwriteprotocol --value="https"
kubectl exec -n nextcloud deploy/nextcloud -- \
  php occ config:system:set overwrite.cli.url \
  --value="https://cloud.logiflowadvanced.online"

Obsidian

Created separate users for me and my friend. WebDAV URL per user:

https://cloud.logiflowadvanced.online/remote.php/dav/files/username/

In the RemotelySave plugin: type — WebDAV, URL, login, password. Works.

The thing that ate most of my time wasn't Gitea or Nextcloud — it was ingressClassName. Neither chart sets it automatically, and without it nginx just ignores the Ingress completely. cert-manager doesn't issue anything. Browser shows self-signed, you stare at the logs, pods are all Running, no errors anywhere.

Run kubectl get ingress -n <namespace> right after deploy. If CLASS says <none> — that's your problem.

The other non-obvious one: gitea-ssh is headless and you can't patch it into a NodePort — you have to delete and recreate it. Spent a few minutes trying to patch it before actually reading the error.

With Nextcloud and MariaDB — if MariaDB didn't come up on the first deploy or you uninstalled and reinstalled, helm will complain about credential mismatch on upgrade. Just helm uninstall + kubectl delete pvc --all -n nextcloud and start fresh, it's faster than untangling the creds.