Migrating & Upgrading GitLab 15.6.1 to 19.2: New Box, Going Private, Cutover Behind Production

Every company name, domain, IP, account ID, instance ID, and fingerprint in this article has been replaced with fictional values for illustration. The company is called Acme (an ecommerce marketplace), the domain is acme.io, and the platform codename is core. Public IPs use the documentation range 203.0.113.0/24, private IPs have had their ranges changed, and account IDs are 1111..., 2222.... The structure and the approach are real.

This article recounts one night of migrating GitLab EE 15.6.1 to 19.2. Not just a version bump. I also moved GitLab from public to private (reachable only over VPN and PrivateLink), under one hard constraint:

Don't touch the old box. The old box served production right up until the DNS flip. Everything read-only (backup, reading the host key), no config changes, no restarts.

The end result: the new box gitlab.acme.io runs 19.2, fully private, with ArgoCD, the runner, and VPN all able to reach it, and rollback is a single DNS record. The road there had many colors.

---

1. Context and constraints

The starting state:

• Old GitLab: EE 15.6.1 omnibus, running on one EC2 (i-0old..., private 10.10.5.6) in the Dev-core 111111111111 account, VPC 10.10.0.0/16.
• Public via Cloudflare (proxied) then a Classic ELB (internet-facing) then the old box on ports 80/22. TLS terminated at the ELB with an ACM *.acme.io cert. Git over SSH ([e-mail]) went straight to ELB:22 (Cloudflare doesn't proxy port 22), so there was already an internal Route53 private zone pointing gitlab.acme.io at the ELB to bypass Cloudflare for SSH.
• Main consumers: ArgoCD (~570 Applications, cloning over SSH), the GitLab Runner (docker-autoscaler), and developers connecting through 4 OpenVPN boxes.

There was no new EE license key. The first question: install CE or EE? An EE backup can only be restored onto the same EE version, so I had to build a new EE 15.6.1 box, restore, and only then step-upgrade to 19.2. No shortcut.

---

2. Architecture: before and after

Before the cutover, everything funneled through Cloudflare and the public ELB. After the cutover, there is no route to the internet, cross-account traffic goes through PrivateLink at a network hub, and the Route53 private zone decides who reaches it by which path.

The key thing to remember: the new box has no public IP, and sits in the dev VPC (10.20.0.0/16). Consumers in other accounts (prod, ArgoCD) can't route to it directly. Everything goes through a VPC Endpoint Service at the network hub account 333333333333, and spokes reach the hub over a Transit Gateway. This is exactly the model another private service in the org already used, so I cloned it instead of inventing something new.

Resource names (changed):

Component	Value
New box	i-0new..., 10.20.12.34, EE upgraded to 19.2
Internal NLB	gitlab-priv-nlb (443 TLS ACM, 22 TCP)
VPC Endpoint Service	vpce-svc-0abc... (allow principal: hub account)
Interface endpoint (hub)	vpce-0def..., IPs 10.40.10.11, .20.22, .30.33
Private zone	acme.io (hub) and a gitlab.acme.io override per VPC

---

3. Phase 1: backup without touching the old box

The old box is private, no direct SSH, controlled through SSM Run Command. The first backup died for two reasons.

(a) The SSM timeout killed the process. An omnibus backup includes artifacts by default. This repo has 14GB of artifacts. gitlab-backup create ran for 40 minutes and then SSM RunShellScript hit its timeout-seconds and killed the process, leaving a half-finished backup.

(b) A stale PID lock. The next run reported There is a backup and restore task in progress, suggesting I remove backup_restore.pid. The process killed by SSM never got to clean up the PID, so every subsequent backup was blocked.

The fix: clear the dead lock, run the backup detached with systemd-run so it's immune to the SSM timeout, and skip artifacts (14GB — accepting the loss of old artifacts; what matters is code, DB, and the repos on the new box).

# 1) confirm there is NO real process, then remove the dead lock
ps aux | grep -E "rake gitlab:backup" | grep -v grep      # empty
rm -f /opt/gitlab/embedded/service/gitlab-rails/tmp/backup_restore.pid

# 2) write a backup + upload script, run it as a transient systemd unit
cat >/root/gl-migrate-backup.sh <<'SCRIPT'
#!/bin/bash
BUCKET=acme-gitlab-migrate-transfer
gitlab-backup create STRATEGY=copy SKIP=artifacts CRON=1 || { echo "BACKUP_FAILED rc=$?"; exit 1; }
TAR=$(ls -t /var/opt/gitlab/backups/*_gitlab_backup.tar | head -1)
cp /etc/gitlab/gitlab-secrets.json /etc/gitlab/gitlab.rb /var/opt/gitlab/backups/
aws s3 cp "$TAR" "s3://$BUCKET/migrate/" --only-show-errors && echo UP_TAR_OK
aws s3 cp /var/opt/gitlab/backups/gitlab-secrets.json "s3://$BUCKET/migrate/" && echo UP_SECRETS_OK
aws s3 cp /var/opt/gitlab/backups/gitlab.rb "s3://$BUCKET/migrate/" && echo UP_RB_OK
echo "MIGRATE_BACKUP_DONE $(date -u)"
SCRIPT
chmod +x /root/gl-migrate-backup.sh
systemd-run --unit=gl-migrate-backup --collect /root/gl-migrate-backup.sh

systemd-run detaches the process entirely from SSM's process tree. The SSM command returns immediately, and the backup keeps running under systemd. Follow it with journalctl -u gl-migrate-backup:

Three things went up to S3: the 15G tarball, gitlab-secrets.json (mandatory for decrypting CI variables and 2FA), and gitlab.rb. The "don't touch the old box" constraint still held, because I only read data and deleted lock/temp files in the backup directory — not GitLab data, and no config changes.

---

4. Phase 2: restore and step-upgrade — this is the time sink

The new box was installed with EE 15.6.1 to match the backup version (a hard requirement of gitlab-backup restore). Verify the match before restoring, then load the old box's secrets, stop puma and sidekiq, and restore.

The two red ERROR: must be owner of extension pg_trgm lines scare newcomers, but they're benign: the extension already exists, restore just couldn't set the owner. The data still came in fully. The skipped restore ... .wiki.bundle doesn't exist lines are because those projects never created a wiki. It's precisely these warnings that make gitlab-backup restore return exit code 1 even though the restore finished — don't let a script bail there.

4.1. Bugs hit during the step-upgrade

The restore only took ten minutes. The real time sink is the step-upgrade from 15.6.1 to 19.2, and that's where most of the bugs live. Logging each one:

Case 1: required-stops, no leapfrogging. GitLab forces you to stop at specific intermediate versions. The path comes from the official upgrade-path tool, roughly:

15.6.1 -> 15.11.13 -> 16.0.10 -> 16.1.8 -> 16.3.9 -> 16.7.10
       -> 16.11.10 -> 17.1.8 -> 17.3.7 -> 17.5.5 -> 17.8.7 -> 17.11.6
       -> 18.0.x -> 18.2.x -> ... -> 19.2.0

Jumping straight there makes the reconfigure die. Each hop has to install exactly that version, reconfigure, wait for migrations to finish, and only then move on.

Case 2: batched background migrations hang and block the next hop. This is the number-one time demon. After each upgrade, GitLab queues a batch of background migrations that run gradually through Sidekiq (backfilling columns, changing data types, adding indexes on large tables). With a DB of 71k merge requests, 113k pipelines, and 128k notes, these crawl. If you upgrade again before they're finished, the next hop errors out and forces you back to wait. Check them:

gitlab-psql -c "select job_class_name, status from batched_background_migrations
                where status not in (3,6);"   # 3=finished, 6=finalized

Case 3: major PostgreSQL upgrades. GitLab bundles Postgres. Around 16.x you have to move PG 13 to 14, and around 17.x/18.x up to PG 16. Run gitlab-ctl pg-upgrade. The trap: pg-upgrade needs double the disk space of the current data (it copies to a new cluster). A nearly-full disk fails midway — extremely annoying.

Case 4: old settings in gitlab.rb break reconfigure. Across several majors, some keys in gitlab.rb were renamed or removed. Reconfigure stops when it hits an unknown key. You have to read the reconfigure log, remove the deprecated key, and re-run.

Case 5: the runner registration token was retired (16.0+). From 16.0, the old registration_token style of registering runners is disabled by default, replaced by the new runner authentication token (glrt-...). The old runner loses its connection until you create a new token. I hit exactly this when reattaching the runner (section 7.7).

Case 6: reconfigure times out on heavy migrations. Some migrations (adding an index on ci_builds, backfilling a large column) take tens of minutes. reconfigure waits for migrations by default, and if it runs inside SSM it hits the SSM timeout again, just like the backup. Same lesson repeated: run the whole upgrade chain detached (systemd-run or tmux), not inside SSM's process tree.

4.2. How to speed up the upgrade

Left to run "naturally", the upgrade chain can take a whole day, mostly sitting and waiting on background migrations. What I did to shorten it:

1. Force batched background migrations to run now, don't wait for Sidekiq to trickle. Instead of letting them crawl, finalize each hanging migration manually:

# list active migrations
gitlab-rails runner "Gitlab::Database::BackgroundMigration::BatchedMigration.active \
  .each { |m| puts \"#{m.job_class_name} #{m.table_name}.#{m.column_name}\" }"

# finalize synchronously (runs to completion now, instead of waiting forever)
gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_builds,id,'[["id"],["id_convert_to_bigint"]]']

2. Temporarily upsize the box during the upgrade. Migrations are CPU- and IO-heavy. Switch to a larger instance type (more vCPU, RAM, high-IOPS gp3 EBS) during the upgrade night, then downsize afterward. Index build times drop noticeably.

3. Give resources to Sidekiq, starve Puma. The migration box serves no traffic, so raise Sidekiq concurrency and turn off almost all of Puma so Sidekiq chews through migrations faster:

# /etc/gitlab/gitlab.rb  (only during the upgrade phase)
sidekiq['max_concurrency'] = 50
puma['worker_processes']   = 0

4. Tune Postgres for bulk work. Temporarily raise maintenance_work_mem and max_wal_size for faster index builds, and restore the defaults when done.

5. Pre-download packages for every hop up front. Don't sit waiting on downloads between stops:

apt-get download gitlab-ee=15.11.13-ee.0 gitlab-ee=16.0.10-ee.0 gitlab-ee=16.3.9-ee.0 ...

6. Disable cron jobs and integrations during the upgrade so Sidekiq doesn't fight for workers with pipeline schedules, webhooks, and mirrors.

The end of the chain:

$ cat /opt/gitlab/embedded/service/gitlab-rails/VERSION
19.2.0-ee
$ gitlab-rails runner 'puts %Q[projects=#{Project.count} users=#{User.count} MR=#{MergeRequest.count}]'
projects=458  users=192  MR=71854

---

5. Phase 3: going private with NLB and PrivateLink

This is the part that's easy to get wrong. The initial plan was an internal ALB (HTTP/443) in the dev VPC. Wrong, because of two things I found when looking at how ArgoCD actually talks to GitLab:

1. ArgoCD clones over SSH ([e-mail]), not HTTPS. An ALB is L7 HTTP and can't carry port 22.
2. Cross-account has to go over PrivateLink, and a VPC Endpoint Service only sits in front of an NLB, not an ALB.

So I switched to an NLB with two listeners: 443 TLS (terminated with the ACM *.acme.io cert, forwarding TCP:80 to GitLab's nginx) and 22 TCP (passthrough to the box's sshd).

# target groups: HTTPS (TLS terminated at NLB -> box:80) and SSH (TCP -> box:22)
aws elbv2 create-target-group --name gitlab-priv-https-tg --protocol TCP --port 80 \
  --vpc-id vpc-dev --target-type ip \
  --health-check-protocol HTTP --health-check-path /users/sign_in --matcher HttpCode=200
aws elbv2 create-target-group --name gitlab-priv-ssh-tg --protocol TCP --port 22 \
  --vpc-id vpc-dev --target-type ip --health-check-protocol TCP
aws elbv2 register-targets --target-group-arn $TGH --targets Id=10.20.12.34,Port=80
aws elbv2 register-targets --target-group-arn $TGS --targets Id=10.20.12.34,Port=22

# internal NLB + listeners
aws elbv2 create-load-balancer --name gitlab-priv-nlb --type network --scheme internal \
  --subnets subnet-1a subnet-1b subnet-1c
aws elbv2 create-listener --load-balancer-arn $NLB --protocol TLS --port 443 \
  --certificates CertificateArn=$ACM --ssl-policy ELBSecurityPolicy-TLS13-1-2-2021-06 \
  --default-actions Type=forward,TargetGroupArn=$TGH
aws elbv2 create-listener --load-balancer-arn $NLB --protocol TCP --port 22 \
  --default-actions Type=forward,TargetGroupArn=$TGS

# a VPC Endpoint Service in front of the NLB, so the hub account can create an interface endpoint
SVC=$(aws ec2 create-vpc-endpoint-service-configuration \
  --network-load-balancer-arns $NLB --no-acceptance-required \
  --query 'ServiceConfiguration.ServiceId' --output text)
aws ec2 modify-vpc-endpoint-service-permissions --service-id $SVC \
  --add-allowed-principals arn:aws:iam::333333333333:root

On the network hub (333333333333), create an interface endpoint in the shared VPC (10.40.x), with a security group for 22 and 443 from the internal ranges. Test from inside the VPC (the NLB is private, so it can only be tested from within), with the right SNI and Host gitlab.acme.io: https=200, ssl=0, and the banner SSH-2.0-OpenSSH. Cross-account from the prod VPN and from a pod in the ArgoCD cluster to the hub endpoint also worked on both 443 and 22. The PrivateLink tunnel is internal, so there's no need to open up the dev VPC's route segmentation.

---

6. Phase 4: cutover with a Route53 private zone (split-horizon)

The cutover doesn't touch Cloudflare (Cloudflare can't reach the private endpoint). The flip point is the Route53 private zone that every VPC uses to resolve gitlab.acme.io.

The problem: the dev VPN is in the same VPC as the NLB (10.20) but has no route to the hub (10.40). So we need split-horizon:

• The shared (hub) zone gitlab.acme.io points at the hub endpoint, for the prod VPN, ArgoCD, and other accounts.
• A separate override zone for the dev VPC points gitlab.acme.io straight at the NLB, for the dev VPN in the same VPC.

# override zone for the DEV VPC -> NLB (alias works because the record is in the SAME account as the NLB)
Z=$(aws route53 create-hosted-zone --name gitlab.acme.io --caller-reference dev-1 \
     --hosted-zone-config PrivateZone=true \
     --vpc VPCRegion=ap-southeast-1,VPCId=vpc-dev --query 'HostedZone.Id' --output text)
aws route53 change-resource-record-sets --hosted-zone-id $Z --change-batch '{
  "Changes":[{"Action":"UPSERT","ResourceRecordSet":{"Name":"gitlab.acme.io.","Type":"A",
    "AliasTarget":{"HostedZoneId":"'$NLB_ZONE'","DNSName":"gitlab-priv-nlb-...elb.amazonaws.com.","EvaluateTargetHealth":false}}}]}'

# hub zone: change CNAME(old ELB) INTO an A alias -> hub endpoint
aws route53 change-resource-record-sets --hosted-zone-id $HUB_ZONE --change-batch '{
  "Changes":[
   {"Action":"DELETE","ResourceRecordSet":{"Name":"gitlab.acme.io.","Type":"CNAME","TTL":300,
     "ResourceRecords":[{"Value":"acme-gitlab-legacy-lb-...elb.amazonaws.com."}]}},
   {"Action":"CREATE","ResourceRecordSet":{"Name":"gitlab.acme.io.","Type":"A",
     "AliasTarget":{"HostedZoneId":"'$VPCE_ZONE'","DNSName":"vpce-0def-...vpce.amazonaws.com.","EvaluateTargetHealth":false}}}
  ]}'

Rollback is pointing the hub zone's record back at the old ELB. The old box and old ELB are untouched, running in parallel.

---

7. Seven little demons after the cutover

This part is fun. Every bug below happened for real, in between 4 a.m.

7.1. SSM timeout and stale PID lock

Already covered in Phase 1. Lesson: long tasks go detached with systemd-run, not in SSM's process tree. And always SKIP=artifacts if object storage isn't used.

7.2. SSH host key mismatch, ArgoCD frozen

Right after the cutover, ArgoCD reported ssh: handshake failed: knownhosts: key mismatch. The new box generated its own new host key, different from the old box, so ArgoCD's known_hosts (pinning the old key) didn't match. The correct approach for a migration is to keep the host identity: copy the old box's host key onto the new box and restart sshd. After that, ssh-keyscan returns the same old fingerprint, and nothing has to change on the client side.

7.3. Empty authorized_keys after restore, publickey auth fails

Host key matched, and ArgoCD then reported unable to authenticate, attempted methods [none publickey]. The new box uses the authorized_keys file for SSH auth, but after the restore that file was 0 bytes. gitlab-backup restore restores the DB (which has 111 SSH keys) but doesn't regenerate the file itself. Fix it with a single rake task: gitlab-rake gitlab:shell:setup.

7.4. NLB cross-zone not enabled, 2/3 endpoint IPs can't reach port 22

The client resolves gitlab.acme.io to 3 endpoint IPs (round-robin). Testing showed only 1/3 IPs could open port 22. The target box is in a single AZ, and the NLB has cross-zone disabled by default, so an NLB node in another AZ has no target and the connect fails — hit or miss. One-line fix: enable load_balancing.cross_zone.enabled=true.

7.5. A cross-account Route53 alias to a VPC endpoint won't resolve

The override zone for the prod VPN (in the prod account) aliased to the hub endpoint (in the hub account), and dig returned empty. Meanwhile the hub zone (same account as the endpoint) resolved the alias fine. The cause: a Route53 alias to a VPC endpoint only resolves when the record is in the same account as the endpoint. Cross-account, use an A record with the endpoint's real IPs instead of an alias.

7.6. DNS negative caching

After changing the record, dig on the prod box still returned empty, because the earlier failed-alias lookup had been negatively cached in systemd-resolved. Query the Amazon resolver 169.254.169.253 directly to check, then resolvectl flush-caches. When a DNS change doesn't show up, suspect negative cache before you suspect Route53.

7.7. VPN missing a route push, devs report "connected but still can't get in"

Devs reported that after connecting to vpn-core.acme.io they still couldn't open GitLab. This one is an OpenVPN Access Server in the prod VPC. Looking at its config: it pushed DNS correctly (the VPC resolver, resolving to 10.40.x), but the pushed route list had 10.60, 10.10, and a pile of Cloudflare ranges (104.x, 172.64.x, leftovers from when GitLab was public), yet was missing 10.40.0.0/16 (the hub endpoint's network). The client resolved to 10.40.x but had no route there, so traffic went out the default gateway. Push the extra route and have clients reconnect:

sacli --key "vpn.server.routing.private_network.23" --value "10.40.0.0/16" ConfigPut
sacli start   # clients disconnect + reconnect to pick up the new route

Sweep all 4 VPN boxes while you're at it — one of them also had its primary DNS set to 8.8.8.8 (public, can't resolve the private zone); fix it back to the VPC resolver.

---

8. Tip: a debug DaemonSet for instant network probing

Throughout the cutover, the most-repeated action was: "from cluster X, can I resolve gitlab.acme.io, and can I reach :443 and :22?". Running kubectl run --image=netshoot each time is slow — it pulls the image and waits to schedule. Instead, pre-place a netshoot DaemonSet on every node, so there's already a pod to exec into, no waiting.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: netshoot-debug
  namespace: infra-debug
spec:
  selector:
    matchLabels: { app: netshoot-debug }
  template:
    metadata:
      labels: { app: netshoot-debug }
    spec:
      tolerations:
        - operator: Exists          # run even on tainted nodes
      containers:
        - name: netshoot
          image: nicolaka/netshoot:latest
          command: ["sleep", "infinity"]
          resources:
            requests: { cpu: 10m, memory: 32Mi }
            limits:   { cpu: 100m, memory: 128Mi }
      terminationGracePeriodSeconds: 1

Probe instantly from any node:

POD=$(kubectl -n infra-debug get pod -l app=netshoot-debug \
      --field-selector spec.nodeName=$NODE -o jsonpath='{.items[0].metadata.name}')
kubectl -n infra-debug exec $POD -- bash -c '
  getent ahostsv4 gitlab.acme.io
  nc -z -w3 gitlab.acme.io 443 && echo 443_OK
  nc -z -w3 gitlab.acme.io 22  && echo 22_OK
  ssh-keyscan -t ed25519 gitlab.acme.io 2>/dev/null | ssh-keygen -lf -
'

A debug DaemonSet is a double-edged sword — netshoot carries a full network toolkit. Put it in its own namespace, cap its resources, and remove it when done (kubectl delete ds netshoot-debug -n infra-debug). Don't leave it living in prod forever. During the cutover it's also worth enabling GuardDuty Runtime Monitoring (the DaemonSet agent) to have eyes on unusual egress while you're opening and closing network paths repeatedly.

---

9. Results and lessons

Acceptance: all three consumer groups reached the new, private box.

Source	Path	443	22 (git SSH)
Dev VPN	override zone to NLB (same VPC)	200	OK
Prod VPN	override zone to hub endpoint	200	OK
ArgoCD	hub zone to hub endpoint	200	git ls-remote works for real
Runner v2	NLB directly	200	online

A few things that stuck:

1. Don't upgrade the new box to the target version too early. A cross-version backup can't be restored or merged. If the old box keeps producing commits and MRs after the backup, you can't incrementally sync between 15.6.1 and 19.2. Shrink the window between the last backup and the cutover.
2. PrivateLink, not peering. Reaching a private service cross-account should go through a VPC Endpoint Service at the hub — it doesn't break segmentation and needs no direct route into the target VPC.
3. NLB for GitLab, not ALB, because you need SSH port 22 too, and because the endpoint service only sits in front of an NLB. Remember to enable cross-zone if the target is in a single AZ.
4. Keep the host identity when migrating (copy /etc/ssh/ssh_host_*) so clients don't hit a key mismatch. And always run gitlab-rake gitlab:shell:setup after a restore.
5. A Route53 alias to a VPC endpoint only works same-account. Cross-account, use an A record with IPs.
6. When a DNS change doesn't show up, suspect negative cache (resolvectl flush-caches) before suspecting Route53.
7. A network cutover comes with a VPN route cutover. When a service moves from public (Cloudflare ranges) to private (new internal ranges), the VPN has to push the new ranges, otherwise it's "connected but still can't get in".

The biggest time sink wasn't the cutover — it was the step-upgrade chain and the pile of batched background migrations in the middle. Prep a strong box, force migrations to run now instead of waiting, and do everything detached: that's how you cut a day down to a night. And the old box didn't get a single config line touched until it was shut off.

---

Come contribute

Synapse is open source (Apache-2.0): github.com/KKloudTarus/synapse-ce.

• Star it if the "AI proposes, Go and a second model confirm" direction is one you want to follow.
• Run it against your own LLM endpoint and tell me what happened: which model refutes well, which one likes to make things up. Feedback from a real model is the most useful kind.
• The verifier's adversarial prompt, the driver vocabulary, the consensus threshold: it all lives in internal/usecase/fptriage, and it's short and easy to read.

If you turn this on in your own repo, drop a comment with how many false positives the AI held back on the first run, and whether it ever came close to refuting something that turned out to be real. I'm curious.