DEV Community: Huỳnh Lê Nhất Nghĩa

Migrating & Upgrading GitLab 15.6.1 to 19.2: New Box, Going Private, Cutover Behind Production

Huỳnh Lê Nhất Nghĩa — Sat, 25 Jul 2026 13:01:49 +0000

Every company name, domain, IP, account ID, instance ID, and fingerprint in this article has been replaced with fictional values for illustration. The company is called Acme (an ecommerce marketplace), the domain is acme.io, and the platform codename is core. Public IPs use the documentation range 203.0.113.0/24, private IPs have had their ranges changed, and account IDs are 1111..., 2222.... The structure and the approach are real.

This article recounts one night of migrating GitLab EE 15.6.1 to 19.2. Not just a version bump. I also moved GitLab from public to private (reachable only over VPN and PrivateLink), under one hard constraint:

Don't touch the old box. The old box served production right up until the DNS flip. Everything read-only (backup, reading the host key), no config changes, no restarts.

The end result: the new box gitlab.acme.io runs 19.2, fully private, with ArgoCD, the runner, and VPN all able to reach it, and rollback is a single DNS record. The road there had many colors.

1. Context and constraints

The starting state:

Old GitLab: EE 15.6.1 omnibus, running on one EC2 (i-0old..., private 10.10.5.6) in the Dev-core 111111111111 account, VPC 10.10.0.0/16.
Public via Cloudflare (proxied) then a Classic ELB (internet-facing) then the old box on ports 80/22. TLS terminated at the ELB with an ACM *.acme.io cert. Git over SSH (git@gitlab.acme.io) went straight to ELB:22 (Cloudflare doesn't proxy port 22), so there was already an internal Route53 private zone pointing gitlab.acme.io at the ELB to bypass Cloudflare for SSH.
Main consumers: ArgoCD (~570 Applications, cloning over SSH), the GitLab Runner (docker-autoscaler), and developers connecting through 4 OpenVPN boxes.

There was no new EE license key. The first question: install CE or EE? An EE backup can only be restored onto the same EE version, so I had to build a new EE 15.6.1 box, restore, and only then step-upgrade to 19.2. No shortcut.

2. Architecture: before and after

Before the cutover, everything funneled through Cloudflare and the public ELB. After the cutover, there is no route to the internet, cross-account traffic goes through PrivateLink at a network hub, and the Route53 private zone decides who reaches it by which path.

The key thing to remember: the new box has no public IP, and sits in the dev VPC (10.20.0.0/16). Consumers in other accounts (prod, ArgoCD) can't route to it directly. Everything goes through a VPC Endpoint Service at the network hub account 333333333333, and spokes reach the hub over a Transit Gateway. This is exactly the model another private service in the org already used, so I cloned it instead of inventing something new.

Resource names (changed):

Component	Value
New box	`i-0new...`, `10.20.12.34`, EE upgraded to 19.2
Internal NLB	`gitlab-priv-nlb` (443 TLS ACM, 22 TCP)
VPC Endpoint Service	`vpce-svc-0abc...` (allow principal: hub account)
Interface endpoint (hub)	`vpce-0def...`, IPs `10.40.10.11`, `.20.22`, `.30.33`
Private zone	`acme.io` (hub) and a `gitlab.acme.io` override per VPC

3. Phase 1: backup without touching the old box

The old box is private, no direct SSH, controlled through SSM Run Command. The first backup died for two reasons.

(a) The SSM timeout killed the process. An omnibus backup includes artifacts by default. This repo has 14GB of artifacts. gitlab-backup create ran for 40 minutes and then SSM RunShellScript hit its timeout-seconds and killed the process, leaving a half-finished backup.

(b) A stale PID lock. The next run reported There is a backup and restore task in progress, suggesting I remove backup_restore.pid. The process killed by SSM never got to clean up the PID, so every subsequent backup was blocked.

The fix: clear the dead lock, run the backup detached with systemd-run so it's immune to the SSM timeout, and skip artifacts (14GB — accepting the loss of old artifacts; what matters is code, DB, and the repos on the new box).

# 1) confirm there is NO real process, then remove the dead lock
ps aux | grep -E "rake gitlab:backup" | grep -v grep      # empty
rm -f /opt/gitlab/embedded/service/gitlab-rails/tmp/backup_restore.pid

# 2) write a backup + upload script, run it as a transient systemd unit
cat >/root/gl-migrate-backup.sh <<'SCRIPT'
#!/bin/bash
BUCKET=acme-gitlab-migrate-transfer
gitlab-backup create STRATEGY=copy SKIP=artifacts CRON=1 || { echo "BACKUP_FAILED rc=$?"; exit 1; }
TAR=$(ls -t /var/opt/gitlab/backups/*_gitlab_backup.tar | head -1)
cp /etc/gitlab/gitlab-secrets.json /etc/gitlab/gitlab.rb /var/opt/gitlab/backups/
aws s3 cp "$TAR" "s3://$BUCKET/migrate/" --only-show-errors && echo UP_TAR_OK
aws s3 cp /var/opt/gitlab/backups/gitlab-secrets.json "s3://$BUCKET/migrate/" && echo UP_SECRETS_OK
aws s3 cp /var/opt/gitlab/backups/gitlab.rb "s3://$BUCKET/migrate/" && echo UP_RB_OK
echo "MIGRATE_BACKUP_DONE $(date -u)"
SCRIPT
chmod +x /root/gl-migrate-backup.sh
systemd-run --unit=gl-migrate-backup --collect /root/gl-migrate-backup.sh

systemd-run detaches the process entirely from SSM's process tree. The SSM command returns immediately, and the backup keeps running under systemd. Follow it with journalctl -u gl-migrate-backup:

Three things went up to S3: the 15G tarball, gitlab-secrets.json (mandatory for decrypting CI variables and 2FA), and gitlab.rb. The "don't touch the old box" constraint still held, because I only read data and deleted lock/temp files in the backup directory — not GitLab data, and no config changes.

4. Phase 2: restore and step-upgrade — this is the time sink

The new box was installed with EE 15.6.1 to match the backup version (a hard requirement of gitlab-backup restore). Verify the match before restoring, then load the old box's secrets, stop puma and sidekiq, and restore.

The two red ERROR: must be owner of extension pg_trgm lines scare newcomers, but they're benign: the extension already exists, restore just couldn't set the owner. The data still came in fully. The skipped restore ... .wiki.bundle doesn't exist lines are because those projects never created a wiki. It's precisely these warnings that make gitlab-backup restore return exit code 1 even though the restore finished — don't let a script bail there.

4.1. Bugs hit during the step-upgrade

The restore only took ten minutes. The real time sink is the step-upgrade from 15.6.1 to 19.2, and that's where most of the bugs live. Logging each one:

Case 1: required-stops, no leapfrogging. GitLab forces you to stop at specific intermediate versions. The path comes from the official upgrade-path tool, roughly:

15.6.1 -> 15.11.13 -> 16.0.10 -> 16.1.8 -> 16.3.9 -> 16.7.10
       -> 16.11.10 -> 17.1.8 -> 17.3.7 -> 17.5.5 -> 17.8.7 -> 17.11.6
       -> 18.0.x -> 18.2.x -> ... -> 19.2.0

Jumping straight there makes the reconfigure die. Each hop has to install exactly that version, reconfigure, wait for migrations to finish, and only then move on.

Case 2: batched background migrations hang and block the next hop. This is the number-one time demon. After each upgrade, GitLab queues a batch of background migrations that run gradually through Sidekiq (backfilling columns, changing data types, adding indexes on large tables). With a DB of 71k merge requests, 113k pipelines, and 128k notes, these crawl. If you upgrade again before they're finished, the next hop errors out and forces you back to wait. Check them:

gitlab-psql -c "select job_class_name, status from batched_background_migrations
                where status not in (3,6);"   # 3=finished, 6=finalized

Case 3: major PostgreSQL upgrades. GitLab bundles Postgres. Around 16.x you have to move PG 13 to 14, and around 17.x/18.x up to PG 16. Run gitlab-ctl pg-upgrade. The trap: pg-upgrade needs double the disk space of the current data (it copies to a new cluster). A nearly-full disk fails midway — extremely annoying.

Case 4: old settings in gitlab.rb break reconfigure. Across several majors, some keys in gitlab.rb were renamed or removed. Reconfigure stops when it hits an unknown key. You have to read the reconfigure log, remove the deprecated key, and re-run.

Case 5: the runner registration token was retired (16.0+). From 16.0, the old registration_token style of registering runners is disabled by default, replaced by the new runner authentication token (glrt-...). The old runner loses its connection until you create a new token. I hit exactly this when reattaching the runner (section 7.7).

Case 6: reconfigure times out on heavy migrations. Some migrations (adding an index on ci_builds, backfilling a large column) take tens of minutes. reconfigure waits for migrations by default, and if it runs inside SSM it hits the SSM timeout again, just like the backup. Same lesson repeated: run the whole upgrade chain detached (systemd-run or tmux), not inside SSM's process tree.

4.2. How to speed up the upgrade

Left to run "naturally", the upgrade chain can take a whole day, mostly sitting and waiting on background migrations. What I did to shorten it:

1. Force batched background migrations to run now, don't wait for Sidekiq to trickle. Instead of letting them crawl, finalize each hanging migration manually:

# list active migrations
gitlab-rails runner "Gitlab::Database::BackgroundMigration::BatchedMigration.active \
  .each { |m| puts \"#{m.job_class_name} #{m.table_name}.#{m.column_name}\" }"

# finalize synchronously (runs to completion now, instead of waiting forever)
gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_builds,id,'[["id"],["id_convert_to_bigint"]]']

2. Temporarily upsize the box during the upgrade. Migrations are CPU- and IO-heavy. Switch to a larger instance type (more vCPU, RAM, high-IOPS gp3 EBS) during the upgrade night, then downsize afterward. Index build times drop noticeably.

3. Give resources to Sidekiq, starve Puma. The migration box serves no traffic, so raise Sidekiq concurrency and turn off almost all of Puma so Sidekiq chews through migrations faster:

# /etc/gitlab/gitlab.rb  (only during the upgrade phase)
sidekiq['max_concurrency'] = 50
puma['worker_processes']   = 0

4. Tune Postgres for bulk work. Temporarily raise maintenance_work_mem and max_wal_size for faster index builds, and restore the defaults when done.

5. Pre-download packages for every hop up front. Don't sit waiting on downloads between stops:

apt-get download gitlab-ee=15.11.13-ee.0 gitlab-ee=16.0.10-ee.0 gitlab-ee=16.3.9-ee.0 ...

6. Disable cron jobs and integrations during the upgrade so Sidekiq doesn't fight for workers with pipeline schedules, webhooks, and mirrors.

The end of the chain:

$ cat /opt/gitlab/embedded/service/gitlab-rails/VERSION
19.2.0-ee
$ gitlab-rails runner 'puts %Q[projects=#{Project.count} users=#{User.count} MR=#{MergeRequest.count}]'
projects=458  users=192  MR=71854

5. Phase 3: going private with NLB and PrivateLink

This is the part that's easy to get wrong. The initial plan was an internal ALB (HTTP/443) in the dev VPC. Wrong, because of two things I found when looking at how ArgoCD actually talks to GitLab:

ArgoCD clones over SSH (git@gitlab.acme.io), not HTTPS. An ALB is L7 HTTP and can't carry port 22.
Cross-account has to go over PrivateLink, and a VPC Endpoint Service only sits in front of an NLB, not an ALB.

So I switched to an NLB with two listeners: 443 TLS (terminated with the ACM *.acme.io cert, forwarding TCP:80 to GitLab's nginx) and 22 TCP (passthrough to the box's sshd).

# target groups: HTTPS (TLS terminated at NLB -> box:80) and SSH (TCP -> box:22)
aws elbv2 create-target-group --name gitlab-priv-https-tg --protocol TCP --port 80 \
  --vpc-id vpc-dev --target-type ip \
  --health-check-protocol HTTP --health-check-path /users/sign_in --matcher HttpCode=200
aws elbv2 create-target-group --name gitlab-priv-ssh-tg --protocol TCP --port 22 \
  --vpc-id vpc-dev --target-type ip --health-check-protocol TCP
aws elbv2 register-targets --target-group-arn $TGH --targets Id=10.20.12.34,Port=80
aws elbv2 register-targets --target-group-arn $TGS --targets Id=10.20.12.34,Port=22

# internal NLB + listeners
aws elbv2 create-load-balancer --name gitlab-priv-nlb --type network --scheme internal \
  --subnets subnet-1a subnet-1b subnet-1c
aws elbv2 create-listener --load-balancer-arn $NLB --protocol TLS --port 443 \
  --certificates CertificateArn=$ACM --ssl-policy ELBSecurityPolicy-TLS13-1-2-2021-06 \
  --default-actions Type=forward,TargetGroupArn=$TGH
aws elbv2 create-listener --load-balancer-arn $NLB --protocol TCP --port 22 \
  --default-actions Type=forward,TargetGroupArn=$TGS

# a VPC Endpoint Service in front of the NLB, so the hub account can create an interface endpoint
SVC=$(aws ec2 create-vpc-endpoint-service-configuration \
  --network-load-balancer-arns $NLB --no-acceptance-required \
  --query 'ServiceConfiguration.ServiceId' --output text)
aws ec2 modify-vpc-endpoint-service-permissions --service-id $SVC \
  --add-allowed-principals arn:aws:iam::333333333333:root

On the network hub (333333333333), create an interface endpoint in the shared VPC (10.40.x), with a security group for 22 and 443 from the internal ranges. Test from inside the VPC (the NLB is private, so it can only be tested from within), with the right SNI and Host gitlab.acme.io: https=200, ssl=0, and the banner SSH-2.0-OpenSSH. Cross-account from the prod VPN and from a pod in the ArgoCD cluster to the hub endpoint also worked on both 443 and 22. The PrivateLink tunnel is internal, so there's no need to open up the dev VPC's route segmentation.

6. Phase 4: cutover with a Route53 private zone (split-horizon)

The cutover doesn't touch Cloudflare (Cloudflare can't reach the private endpoint). The flip point is the Route53 private zone that every VPC uses to resolve gitlab.acme.io.

The problem: the dev VPN is in the same VPC as the NLB (10.20) but has no route to the hub (10.40). So we need split-horizon:

The shared (hub) zone gitlab.acme.io points at the hub endpoint, for the prod VPN, ArgoCD, and other accounts.
A separate override zone for the dev VPC points gitlab.acme.io straight at the NLB, for the dev VPN in the same VPC.

# override zone for the DEV VPC -> NLB (alias works because the record is in the SAME account as the NLB)
Z=$(aws route53 create-hosted-zone --name gitlab.acme.io --caller-reference dev-1 \
     --hosted-zone-config PrivateZone=true \
     --vpc VPCRegion=ap-southeast-1,VPCId=vpc-dev --query 'HostedZone.Id' --output text)
aws route53 change-resource-record-sets --hosted-zone-id $Z --change-batch '{
  "Changes":[{"Action":"UPSERT","ResourceRecordSet":{"Name":"gitlab.acme.io.","Type":"A",
    "AliasTarget":{"HostedZoneId":"'$NLB_ZONE'","DNSName":"gitlab-priv-nlb-...elb.amazonaws.com.","EvaluateTargetHealth":false}}}]}'

# hub zone: change CNAME(old ELB) INTO an A alias -> hub endpoint
aws route53 change-resource-record-sets --hosted-zone-id $HUB_ZONE --change-batch '{
  "Changes":[
   {"Action":"DELETE","ResourceRecordSet":{"Name":"gitlab.acme.io.","Type":"CNAME","TTL":300,
     "ResourceRecords":[{"Value":"acme-gitlab-legacy-lb-...elb.amazonaws.com."}]}},
   {"Action":"CREATE","ResourceRecordSet":{"Name":"gitlab.acme.io.","Type":"A",
     "AliasTarget":{"HostedZoneId":"'$VPCE_ZONE'","DNSName":"vpce-0def-...vpce.amazonaws.com.","EvaluateTargetHealth":false}}}
  ]}'

Rollback is pointing the hub zone's record back at the old ELB. The old box and old ELB are untouched, running in parallel.

7. Seven little demons after the cutover

This part is fun. Every bug below happened for real, in between 4 a.m.

7.1. SSM timeout and stale PID lock

Already covered in Phase 1. Lesson: long tasks go detached with systemd-run, not in SSM's process tree. And always SKIP=artifacts if object storage isn't used.

7.2. SSH host key mismatch, ArgoCD frozen

Right after the cutover, ArgoCD reported ssh: handshake failed: knownhosts: key mismatch. The new box generated its own new host key, different from the old box, so ArgoCD's known_hosts (pinning the old key) didn't match. The correct approach for a migration is to keep the host identity: copy the old box's host key onto the new box and restart sshd. After that, ssh-keyscan returns the same old fingerprint, and nothing has to change on the client side.

7.3. Empty `authorized_keys` after restore, publickey auth fails

Host key matched, and ArgoCD then reported unable to authenticate, attempted methods [none publickey]. The new box uses the authorized_keys file for SSH auth, but after the restore that file was 0 bytes. gitlab-backup restore restores the DB (which has 111 SSH keys) but doesn't regenerate the file itself. Fix it with a single rake task: gitlab-rake gitlab:shell:setup.

7.4. NLB cross-zone not enabled, 2/3 endpoint IPs can't reach port 22

The client resolves gitlab.acme.io to 3 endpoint IPs (round-robin). Testing showed only 1/3 IPs could open port 22. The target box is in a single AZ, and the NLB has cross-zone disabled by default, so an NLB node in another AZ has no target and the connect fails — hit or miss. One-line fix: enable load_balancing.cross_zone.enabled=true.

7.5. A cross-account Route53 alias to a VPC endpoint won't resolve

The override zone for the prod VPN (in the prod account) aliased to the hub endpoint (in the hub account), and dig returned empty. Meanwhile the hub zone (same account as the endpoint) resolved the alias fine. The cause: a Route53 alias to a VPC endpoint only resolves when the record is in the same account as the endpoint. Cross-account, use an A record with the endpoint's real IPs instead of an alias.

7.6. DNS negative caching

After changing the record, dig on the prod box still returned empty, because the earlier failed-alias lookup had been negatively cached in systemd-resolved. Query the Amazon resolver 169.254.169.253 directly to check, then resolvectl flush-caches. When a DNS change doesn't show up, suspect negative cache before you suspect Route53.

7.7. VPN missing a route push, devs report "connected but still can't get in"

Devs reported that after connecting to vpn-core.acme.io they still couldn't open GitLab. This one is an OpenVPN Access Server in the prod VPC. Looking at its config: it pushed DNS correctly (the VPC resolver, resolving to 10.40.x), but the pushed route list had 10.60, 10.10, and a pile of Cloudflare ranges (104.x, 172.64.x, leftovers from when GitLab was public), yet was missing 10.40.0.0/16 (the hub endpoint's network). The client resolved to 10.40.x but had no route there, so traffic went out the default gateway. Push the extra route and have clients reconnect:

sacli --key "vpn.server.routing.private_network.23" --value "10.40.0.0/16" ConfigPut
sacli start   # clients disconnect + reconnect to pick up the new route

Sweep all 4 VPN boxes while you're at it — one of them also had its primary DNS set to 8.8.8.8 (public, can't resolve the private zone); fix it back to the VPC resolver.

8. Tip: a debug DaemonSet for instant network probing

Throughout the cutover, the most-repeated action was: "from cluster X, can I resolve gitlab.acme.io, and can I reach :443 and :22?". Running kubectl run --image=netshoot each time is slow — it pulls the image and waits to schedule. Instead, pre-place a netshoot DaemonSet on every node, so there's already a pod to exec into, no waiting.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: netshoot-debug
  namespace: infra-debug
spec:
  selector:
    matchLabels: { app: netshoot-debug }
  template:
    metadata:
      labels: { app: netshoot-debug }
    spec:
      tolerations:
        - operator: Exists          # run even on tainted nodes
      containers:
        - name: netshoot
          image: nicolaka/netshoot:latest
          command: ["sleep", "infinity"]
          resources:
            requests: { cpu: 10m, memory: 32Mi }
            limits:   { cpu: 100m, memory: 128Mi }
      terminationGracePeriodSeconds: 1

Probe instantly from any node:

POD=$(kubectl -n infra-debug get pod -l app=netshoot-debug \
      --field-selector spec.nodeName=$NODE -o jsonpath='{.items[0].metadata.name}')
kubectl -n infra-debug exec $POD -- bash -c '
  getent ahostsv4 gitlab.acme.io
  nc -z -w3 gitlab.acme.io 443 && echo 443_OK
  nc -z -w3 gitlab.acme.io 22  && echo 22_OK
  ssh-keyscan -t ed25519 gitlab.acme.io 2>/dev/null | ssh-keygen -lf -
'

A debug DaemonSet is a double-edged sword — netshoot carries a full network toolkit. Put it in its own namespace, cap its resources, and remove it when done (kubectl delete ds netshoot-debug -n infra-debug). Don't leave it living in prod forever. During the cutover it's also worth enabling GuardDuty Runtime Monitoring (the DaemonSet agent) to have eyes on unusual egress while you're opening and closing network paths repeatedly.

9. Results and lessons

Acceptance: all three consumer groups reached the new, private box.

Source	Path	443	22 (git SSH)
Dev VPN	override zone to NLB (same VPC)	200	OK
Prod VPN	override zone to hub endpoint	200	OK
ArgoCD	hub zone to hub endpoint	200	`git ls-remote` works for real
Runner v2	NLB directly	200	online

A few things that stuck:

Don't upgrade the new box to the target version too early. A cross-version backup can't be restored or merged. If the old box keeps producing commits and MRs after the backup, you can't incrementally sync between 15.6.1 and 19.2. Shrink the window between the last backup and the cutover.
PrivateLink, not peering. Reaching a private service cross-account should go through a VPC Endpoint Service at the hub — it doesn't break segmentation and needs no direct route into the target VPC.
NLB for GitLab, not ALB, because you need SSH port 22 too, and because the endpoint service only sits in front of an NLB. Remember to enable cross-zone if the target is in a single AZ.
Keep the host identity when migrating (copy /etc/ssh/ssh_host_*) so clients don't hit a key mismatch. And always run gitlab-rake gitlab:shell:setup after a restore.
A Route53 alias to a VPC endpoint only works same-account. Cross-account, use an A record with IPs.
When a DNS change doesn't show up, suspect negative cache (resolvectl flush-caches) before suspecting Route53.
A network cutover comes with a VPN route cutover. When a service moves from public (Cloudflare ranges) to private (new internal ranges), the VPN has to push the new ranges, otherwise it's "connected but still can't get in".

The biggest time sink wasn't the cutover — it was the step-upgrade chain and the pile of batched background migrations in the middle. Prep a strong box, force migrations to run now instead of waiting, and do everything detached: that's how you cut a day down to a night. And the old box didn't get a single config line touched until it was shut off.

Come contribute

Synapse is open source (Apache-2.0): github.com/KKloudTarus/synapse-ce.

Star it if the "AI proposes, Go and a second model confirm" direction is one you want to follow.
Run it against your own LLM endpoint and tell me what happened: which model refutes well, which one likes to make things up. Feedback from a real model is the most useful kind.
The verifier's adversarial prompt, the driver vocabulary, the consensus threshold: it all lives in internal/usecase/fptriage, and it's short and easy to read.

If you turn this on in your own repo, drop a comment with how many false positives the AI held back on the first run, and whether it ever came close to refuting something that turned out to be real. I'm curious.

A security merge gate is only as good as its false positive rate

Huỳnh Lê Nhất Nghĩa — Thu, 16 Jul 2026 10:06:01 +0000

A hard gate on merge is the right instinct. Security findings are cheapest when they interrupt a change before it becomes shared state, before other branches build on it, before it ships. Blocking the merge is the correct place for that class of control.

But there is a failure mode that is worse than having no gate at all: a gate that blocks too eagerly. The first time it flags something that is obviously fine, people start looking for the bypass flag. The second time, they add it to their muscle memory. After that the gate is theater, and the one real finding it catches next quarter gets waved through with everything else.

So the interesting engineering problem is not "can we block a merge on a security finding." That part is easy. The problem is staying accurate enough that nobody wants to route around you.

Here is how we think about it on Synapse.

The gate blocked a real pull request until an embedded-credential git URL was rejected, then let it through.

The gate has to be explainable

A finding that blocks a merge carries four things by construction:

what failed (a rule id and a CWE),
why it matters (a short rationale),
the evidence it used,
and the exact change that makes it pass (a compliant example).

For a dependency finding the evidence is the scanner and the advisory database version. For a static analysis finding it is the matched pattern or the data flow, at a specific file and line. The evidence is hash chained, so a finding cannot be quietly edited after the fact and still pass the report gate.

The goal is that a developer never has to guess why the gate fired or argue with a black box. If the gate cannot tell you what to change, it has not earned the right to block you.

Precision is a release gate for the rules, not just for your code

This is the part most tools get wrong, and it is the part we care about most.

Our deterministic detectors (dependency scanning, pattern static analysis, taint, secrets, misconfiguration) publish findings with no model in the path. Same input, same finding, every time. There is no AI triage step softening the output. That is a feature, because it makes the gate reproducible and auditable. It is also a constraint, because it means a false positive prone rule is not a nuisance. It is a bug in the tool.

A concrete example. We were reviewing a community pull request that added a rule pack for Jupyter and IPython notebooks. Fifteen new rules, clean build, all tests green. Two of the rules looked completely reasonable on paper:

One flagged <script> markup saved in a notebook's output cell. Reasonable sounding, since script markup in output is an XSS smell.
One flagged local filesystem paths like /home/alice/... saved in a traceback, since those can disclose environment structure.

Both would have been a disaster on real notebooks.

Plotly, Bokeh, ipywidgets, and Altair all embed <script> tags in their normal chart output. Every interactive plot in every data science notebook would have tripped the first rule. And every saved Python exception contains site-packages frames under a home directory, so the second rule would fire on essentially any notebook that ever caught an error.

Neither rule is wrong in theory. Both are wrong in practice, because they cry wolf on ordinary code. So we blocked the merge until they were tightened:

The script rule now requires a real injection signal in the markup, like an event handler attribute, document.write(, or eval(, not just the presence of a <script> tag.
The traceback rule now excludes site-packages, dist-packages, and virtualenv frames, and fires only on a genuine project path.

The rule count stayed the same. The false positive rate went to near zero, with regression tests to keep it there.

The bar we hold is simple: if a rule fires on ordinary, safe code, it does not ship. Precision is a gate the rules have to pass before they are allowed to gate anyone else.

AI generated code and the shared blind spot

A growing share of pull requests are agent generated now, and they read clean at a glance. The dangerous case is not the code that looks sloppy. It is the code where the same agent also wrote the tests, so the suite inherits the exact blind spot the implementation has. Everything is green, and the vulnerable pattern sails through.

This is why the gate is pattern and data flow based, and deliberately independent of the test suite rather than derived from it. If the tests missed it because they shared the author's assumptions, the gate still does not. A gate that trusts the tests is only as good as whoever wrote the tests, which increasingly is the same model that wrote the code.

Let the model propose, do not let it decide

We do use models. We just do not let them block a merge on their own opinion.

Any analytical claim, whether it is reachability, a risk narrative, or a suspected exploit, is a proposal. It cannot gate anything until a separate, deterministic verifier seals a verdict over a threshold. A confident but wrong model output never becomes a merge blocker, because it never gets to be the one deciding.

For dependency findings, reachability and taint analysis decide whether a vulnerable symbol is actually reachable in your code before the finding escalates. "It exists somewhere in the tree" is not automatically a hard stop. That alone removes a large slice of the noise that makes teams resent dependency scanners.

The gate stays green because it is boring

None of this works if the checks are flaky. The detectors are plain, deterministic Go: build with and without CGO, vet, and a fast test suite that runs on every change. Reproducible checks are what let you trust a red result enough to block on it.

Deterministic checks: same input, same result, fast enough to run on every pull request.

The quiet part

Catching real issues is table stakes. Any scanner can produce findings. The actual engineering work is in the negative space: not firing on the Plotly chart, not firing on the site-packages traceback, not blocking a merge over a vulnerable function nobody calls.

A gate earns trust by being right, being specific about the fix, and being quiet when there is nothing wrong. Get that part right and people stop trying to route around it, because there is nothing to route around. Get it wrong and it does not matter how good your detection is, because the gate everyone silences is the same as no gate at all.

Come contribute

Synapse is open source (Apache-2.0): github.com/KKloudTarus/synapse-ce.

Star it if the "AI proposes, Go and a second model confirm" direction is one you want to follow.
Run it against your own LLM endpoint and tell me what happened: which model refutes well, which one likes to make things up. Feedback from a real model is the most useful kind.
The verifier's adversarial prompt, the driver vocabulary, the consensus threshold: it all lives in internal/usecase/fptriage, and it's short and easy to read.

Let an AI clear out your false positives without letting it hide a real bug

Huỳnh Lê Nhất Nghĩa — Sun, 12 Jul 2026 11:07:40 +0000

Last time I wrote about wiring up a security gate that blocks merges in CI with Synapse. Someone left a comment that hit the exact sore spot:

"The gate is the easy part. Two weeks in, the team turns it off because it's red over false positives every single time."

Yep. A gate that cries wolf too often dies on its own. People start hitting "merge anyway," and by the time a real vulnerability shows up, nobody's looking anymore.

The obvious 2020s fix is to hand the whole pile to an LLM and say "delete the junk." Except: the moment a security tool lets an AI delete findings, you've just built a brand new vulnerability. The model guesses wrong once, a real SQL injection quietly drops off the report, and nobody's the wiser.

So Synapse does it differently. The AI is allowed to propose that a finding is a false positive. It is never allowed to confirm that on its own, and it is never allowed to delete anything. This whole post is a real run. Terminal output at the bottom, nothing faked.

Where the actual "gate" is

Three rules, and all three exist so you can trust the output:

1. The model only proposes. It doesn't return prose, and it doesn't return a delete command. It returns exactly one typed verdict: verdict is one of refuted | sound | uncertain, driver is a closed snake_case token (not a sentence), and confidence is 0–100. That grammar is validated in Go. The model can hallucinate a whole paragraph of reasoning and none of it reaches the report.

2. A second, different model has to agree. When the first model proposes "refuted" with confidence ≥ 75, a separate verifier model gets called to assess the same finding independently. The refutation only stands if the verifier also says "refuted" at ≥ 75. And the verifier is prompted adversarially. Its job is to stop a real bug from being waved away. This is the CLI version of the rule Synapse enforces on the server: a claim is only confirmed by a different reviewer's sealed verdict. No confirming your own work.

3. Nothing gets deleted. A "refuted" finding is retain-and-mark: it stays in the report, it stays evidence-sealed, it still shows up in the compliance table. It's just exempt from the gate's exit code. The worst a wrong verdict can do is let one finding skip --fail-on. It can never make a finding disappear.

One more thing, because it matters for the cost: this is the second layer. Before any model runs, a deterministic scope classifier already strips the obvious noise: test files, fixtures, that sort of thing. The model only gets the harder calls: is this sink actually attacker-controlled? Is that interpolated value a constant or user input? Model calls cost tokens, so they only run on production-scope, first-party source findings.

Turning it on

It's opt-in. Four environment variables:

export SYNAPSE_FP_TRIAGE_ENABLED=true                  # turn on AI false-positive triage
export SYNAPSE_LLM_BASE_URL=http://localhost:20128/v1  # any OpenAI-compatible endpoint
export SYNAPSE_FP_TRIAGE_MODEL=<proposer-model>        # a cheap/fast model to propose
export SYNAPSE_VERIFIER_MODEL=<different-model>         # a DIFFERENT model to verify (two-model consensus)

A few notes:

SYNAPSE_LLM_BASE_URL is a plain OpenAI-style endpoint (/v1/chat/completions). Run it locally with Ollama or vLLM, or point it at OpenAI. Doesn't matter. Add SYNAPSE_LLM_API_KEY if yours needs a key.
If you don't set SYNAPSE_VERIFIER_MODEL (or you set it to the same model), you get single-model mode. It still works, but the real gate is the second model being different. Set it to the proposer and Synapse ignores it. That's not a verifier.
The bar is 75, the same threshold every other AI claim in Synapse has to clear.

Then you run the exact same scan command as always:

synapse-cli scan . --fail-on high

What actually happened

I built a small service with both real bugs and deliberate false positives:

payments/store.py: a SQL query built with db.session.execute(f"... {account_id}") where account_id comes straight from request.args. That's a real SQL injection. Same file has a hashlib.md5(...) call used as a cache key for a static asset. The weak-hash rule flags it, but it has nothing to do with security.
payments/settings.py: a hardcoded AWS access key. Real.
reports/ledger.py: another f-string SQL query, but the interpolated value is PERIOD, a server-side constant, not user input. The pattern matcher flags every f-string SQL, so this is a textbook high-severity false positive.

Before: no AI

$ synapse-cli scan . --fail-on high

Synapse SCA dogfood – payments-service
  vulnerabilities: 0
  findings (promoted): 7
    critical  risk  0.00  Hardcoded AWS access key id (payments/settings.py:4)
    high      risk  0.00  SQL execution uses dynamic string construction (payments/store.py:14)
    high      risk  0.00  Python SQLAlchemy/raw SQL uses dynamic string construction (payments/store.py:14)
    medium    risk  0.00  Weak hash: MD5 (payments/store.py:21)
    high      risk  0.00  SQL execution uses dynamic string construction (reports/ledger.py:13)
    high      risk  0.00  Python SQLAlchemy/raw SQL uses dynamic string construction (reports/ledger.py:13)
    high      risk  0.00  AWS access key ID (payments/settings.py:4)

synapse-cli: 6 finding(s) at or above high
$ echo $?
1

Gate is red over 6 findings at or above high. Two of them (the SQL in reports/ledger.py) are false alarms you'd have to triage by hand.

After: AI on, two models

The proposer and the verifier are two different models on my endpoint:

$ export SYNAPSE_FP_TRIAGE_ENABLED=true
$ export SYNAPSE_FP_TRIAGE_MODEL=gpt-5.4-mini      # proposer
$ export SYNAPSE_VERIFIER_MODEL=gpt-5.4            # verifier (a DIFFERENT model)
$ synapse-cli scan . --fail-on high

synapse-cli: AI false-positive triage (gpt-5.4-mini, verified by gpt-5.4): critiqued 7 finding(s), 3 suspected false positive(s) held back from the gate

Synapse SCA dogfood – payments-service
  findings (promoted): 7            # <- still 7, nothing deleted
  ...
  compliance: Synapse AppSec Baseline v1.0 – 3/7 controls passing
    [FAIL] SAB-INJ-1  No injection weaknesses ...
           - SQL ... (payments/store.py:14)
           - SQL ... (reports/ledger.py:13)     # <- the FP still shows in compliance
    ...

synapse-cli: 2 suspected false positive(s) at or above high held back from the gate by AI triage (reported with a verdict; not deleted)
synapse-cli: 4 finding(s) at or above high
$ echo $?
1

Read those lines closely:

critiqued 7 finding(s), 3 suspected false positive(s) held back: the AI looked at 7 findings and proposed 3 as false.
2 ... at or above high held back ... (reported with a verdict; not deleted): two of those were high-severity, held back from the gate but not deleted.
4 finding(s) at or above high: the 4 real bugs still count. Gate is still red (exit 1). The AI couldn't bury a real bug.
findings (promoted): 7 is unchanged, and the false positives are still sitting in the compliance table. Nothing left the report.

Verdict by verdict

This is my favorite part. With --json, every finding carries the AI's verdict next to it:

[critical] Hardcoded AWS access key id (settings.py:4)
    verdict=sound    confidence=98   suspected_fp=false   driver=hardcoded_secret
[high]     SQL execution dynamic string (store.py:14)
    verdict=sound    confidence=99   suspected_fp=false   driver=confirmed_by_review
[high]     SQLAlchemy raw SQL dynamic (store.py:14)
    verdict=sound    confidence=99   suspected_fp=false   driver=dynamic_string_sql
[high]     AWS access key ID (settings.py:4)
    verdict=sound    confidence=99   suspected_fp=false   driver=literal
[medium]   Weak hash: MD5 (store.py:21)
    verdict=refuted  confidence=96   suspected_fp=true    driver=unspecified_refutation
[high]     SQL execution dynamic string (ledger.py:13)
    verdict=refuted  confidence=99   suspected_fp=true    driver=constant_literal
[high]     SQLAlchemy raw SQL dynamic (ledger.py:13)
    verdict=refuted  confidence=98   suspected_fp=true    driver=constant_literal

Look at the driver column. For reports/ledger.py the model worked out on its own that the interpolated value is a constant_literal, a constant, not input. For store.py it's sound: real bug, left alone. And suspected_fp=true is the only thing that exempts a finding from the gate. The key detail: because I configured a different verifier (the log literally says verified by gpt-5.4), a finding only gets suspected_fp=true when both models refute it at ≥ 75. Drop the verifier and these exact three refutations still have to clear a second model to stand.

End result: you're left with 4 real problems to fix instead of 6-with-2-fake mixed in. And more importantly, not one real bug got dimmed.

"But what if the AI is wrong?"

Right question. The whole design is built around it:

AI wrongly refutes a real bug? The verifier is a different model, prompted to push back, and it has to agree before the refutation counts. If the verifier disagrees, is unsure, or errors out, the finding still counts toward the gate (fail-safe). Even if both models are wrong, the finding is still sitting in the report with its verdict attached. You can see it, it didn't vanish.
Model times out / endpoint dies? That critique becomes "uncertain" and the finding gates normally. Triage is best-effort and never fails the scan.
Model tries to smuggle prose into the report? driver is forced into a closed token grammar, verdict has to be from a closed vocabulary, anything else is rejected. No LLM gets prose into the report path.

Put another way: the worst a bad verdict can do is exempt one finding from the exit code while it stays visible in the report. Not hide a bug.

Wiring it into CI

Nothing about the workflow from the last post changes. Add four environment variables to the scan step and you're done:

      - name: Synapse scan (SARIF + gate on high)
        env:
          SYNAPSE_FP_TRIAGE_ENABLED: "true"
          SYNAPSE_LLM_BASE_URL: ${{ secrets.LLM_BASE_URL }}
          SYNAPSE_LLM_API_KEY: ${{ secrets.LLM_API_KEY }}
          SYNAPSE_FP_TRIAGE_MODEL: "your-proposer-model"
          SYNAPSE_VERIFIER_MODEL: "your-verifier-model"   # different from the proposer
        run: synapse-cli scan . --sarif --fail-on high > synapse.sarif

The gate stays as deterministic as before. The pile of false alarms just stops eroding the team's trust in it. And since nothing gets deleted, the SARIF you push to Code scanning is still complete; the false positives are only marked, not dropped.

One honest caveat: this is a noise filter, not a replacement for a human. It gets you looking at the 4 real problems instead of 6 muddled ones. You still have to fix those 4.

Come contribute

Synapse is open source (Apache-2.0): github.com/KKloudTarus/synapse-ce.

Star it if the "AI proposes, Go and a second model confirm" direction is one you want to follow.
Run it against your own LLM endpoint and tell me what happened: which model refutes well, which one likes to make things up. Feedback from a real model is the most useful kind.
The verifier's adversarial prompt, the driver vocabulary, the consensus threshold: it all lives in internal/usecase/fptriage, and it's short and easy to read.

Synapse Feature Demo: Deterministic Software Composition Analysis in Action

Huỳnh Lê Nhất Nghĩa — Sat, 11 Jul 2026 10:49:34 +0000

Hi everyone!

I've just added a short demo video to the Synapse project to showcase how the platform performs deterministic Software Composition Analysis (SCA) from start to finish.

In this demo, you'll see:

Importing a project for analysis
SBOM generation
Dependency and vulnerability analysis
Evidence collection
Findings presented through the web interface

The goal of Synapse is to provide a deterministic-first approach to application security, where every finding is backed by reproducible evidence rather than opaque scanning results.

I'd love to hear your thoughts on:

User experience
Scan workflow
Dashboard design
Missing capabilities
Features you'd like to see next

If you're interested in application security, DevSecOps, or open-source security tooling, your feedback would be greatly appreciated.

GitHub: https://github.com/KKloudTarus/synapse-ce
Youtube: https://www.youtube.com/watch?v=rUdt52em2QQ

Contributions are always welcome! Whether you're interested in backend development, frontend improvements, documentation, parser support, or security research, feel free to open an issue or submit a pull request.

Thank you for taking the time to watch the demo, and I hope it gives you a better idea of where Synapse is heading.

Block the merge when a PR ships a vulnerability: a CI security gate with Synapse

Huỳnh Lê Nhất Nghĩa — Wed, 08 Jul 2026 16:07:09 +0000

I have a simple rule when I work with a team: if a vulnerability makes it into main, the process already failed somewhere upstream. The right place to catch it is on the pull request, before anyone clicks merge.

This post is how I built exactly that gate with Synapse, an open source security scanner written in Go. The end result: open a PR that introduces a vulnerability and CI goes red, a comment lands on the PR telling you what to fix and how, the results show up in the GitHub Code scanning tab, and the merge button stays locked until the issue is dealt with.

Everything below actually runs. There is a public demo repo linked at the end so you can see it, not just read about it.

Why another tool

Fair question. Trivy, Grype and OSV-Scanner are all good and I still use them.

What is different about Synapse is not "it finds more CVEs". Detection is basically a commodity now, since every tool pulls from the same handful of data sources. The difference is the governance layer around a finding: a hash chained chain of custody so results cannot be edited quietly, a separation between who proposes and who confirms for the AI assisted parts, and a report path with no LLM in it. In short, it is built for the case where you have to prove a result, not just print one.

But this post is not about philosophy. It is about one concrete job: blocking the merge.

One thing I like: synapse-cli runs with no database and no server. It is a single static Go binary. It shells out to syft to build the SBOM and to grype as one detection source, and both of those are single static binaries too. That is the same footprint you already know from running Trivy in CI.

What you need

Three binaries:

synapse-cli (built from source, see the workflow below)
syft (builds the SBOM)
grype (one offline detection source)

No Postgres, no external service. If the runner has network access it also uses OSV.dev as a second source. If you are air gapped, add the --offline flag.

The gate itself is one flag:

synapse-cli scan . --fail-on high

That returns a non zero exit code if there is any finding at or above high, and the important part is that it counts across every kind: dependency vulnerabilities (SCA), issues in your own source code (SAST), hardcoded secrets, and misconfigurations in Dockerfiles, Kubernetes, Helm and Terraform. A leaked API key in the code blocks the merge just like a CVE does.

Add --sarif to emit a SARIF 2.1.0 report for the GitHub Code scanning tab:

synapse-cli scan . --sarif --fail-on high > synapse.sarif

SARIF is written to stdout before the gate sets the exit code, so the file is complete even when the command returns 1. A single run both annotates the PR and blocks the merge.

The full workflow

Here is the .github/workflows/synapse.yml I use in the demo repo. It runs the scan, gates on high, uploads SARIF, and posts a remediation report on the PR.

name: Synapse Security Scan

on:
  push:
    branches: [main]
  pull_request:
  workflow_dispatch:

permissions:
  contents: read
  security-events: write   # to upload SARIF
  pull-requests: write      # to comment on the PR

jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-go@v5
        with:
          go-version: '1.26.4'

      - name: Install syft + grype
        run: |
          mkdir -p "$HOME/.local/bin"
          echo "$HOME/.local/bin" >> "$GITHUB_PATH"
          curl -sSfL https://raw.githubusercontent.com/anchore/syft/main/install.sh  | sh -s -- -b "$HOME/.local/bin"
          curl -sSfL https://raw.githubusercontent.com/anchore/grype/main/install.sh | sh -s -- -b "$HOME/.local/bin"

      - name: Build synapse-cli
        env:
          GOTOOLCHAIN: auto
        run: |
          git clone --depth 1 https://github.com/KKloudTarus/synapse-ce.git /tmp/synapse-ce
          cd /tmp/synapse-ce
          go build -o "$HOME/.local/bin/synapse-cli" ./cmd/synapse-cli

      # The gate. The JSON result feeds both the exit code and the PR report.
      - name: Synapse scan (gate on high)
        id: scan
        env:
          SYNAPSE_MISCONFIG_ENABLED: "true"
          SYNAPSE_SAST_ENABLED: "true"
          SYNAPSE_SECRET_SCAN_ENABLED: "true"
        run: synapse-cli scan . --json --fail-on high > synapse.json

      # SARIF for the Code scanning tab. The gate already ran, so ignore the exit here.
      - name: Synapse scan (SARIF for code scanning)
        if: always()
        env:
          SYNAPSE_MISCONFIG_ENABLED: "true"
          SYNAPSE_SAST_ENABLED: "true"
          SYNAPSE_SECRET_SCAN_ENABLED: "true"
        run: synapse-cli scan . --sarif > synapse.sarif || true

      - name: Upload SARIF to code scanning
        if: always()
        uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: synapse.sarif

      - name: Post report on the PR
        if: always() && github.event_name == 'pull_request'
        env:
          GH_TOKEN: ${{ github.token }}
          PR: ${{ github.event.pull_request.number }}
          GATE: ${{ steps.scan.outcome }}
        run: |
          crit=$(jq '[.findings[]|select(.Severity=="critical")]|length' synapse.json)
          high=$(jq '[.findings[]|select(.Severity=="high")]|length' synapse.json)
          med=$(jq '[.findings[]|select(.Severity=="medium")]|length' synapse.json)
          low=$(jq '[.findings[]|select(.Severity=="low" or .Severity=="info")]|length' synapse.json)
          {
            echo "## Synapse security scan"
            echo
            if [ "$GATE" = "failure" ]; then
              echo "**Result: FAILED.** Findings at or above high severity. The merge is blocked until they are resolved."
            else
              echo "**Result: PASSED.** No findings at or above high severity."
            fi
            echo
            echo "| Severity | Count |"
            echo "|---|---|"
            echo "| Critical | $crit |"
            echo "| High | $high |"
            echo "| Medium | $med |"
            echo "| Low | $low |"
            echo
            if [ "$(jq '[.vulnerabilities[].Component]|unique|length' synapse.json)" -gt 0 ]; then
              echo "### Dependencies to upgrade"
              echo
              echo "Edit the manifest and bump each package to at least the version in the \"Upgrade to\" column."
              echo
              echo "| Package | Current | Upgrade to | CVEs | Top severity |"
              echo "|---|---|---|---|---|"
              jq -r '
                .vulnerabilities | group_by(.Component)
                | map({comp:.[0].Component, ver:.[0].Version,
                       fix:((map(.FixedVersion)|map(select(.!=null and .!=""))|max) // "no fix yet"),
                       n:length,
                       sev:(if any(.[];.Severity=="critical") then "critical"
                            elif any(.[];.Severity=="high") then "high"
                            elif any(.[];.Severity=="medium") then "medium" else "low" end)})
                | sort_by(.n) | reverse
                | .[] | "| \(.comp) | \(.ver) | >= \(.fix) | \(.n) | \(.sev) |"' synapse.json
              echo
            fi
            if [ "$(jq '[.findings[]|select(.Kind!="sca")]|length' synapse.json)" -gt 0 ]; then
              echo "### Code and config"
              echo
              jq -r '.findings[] | select(.Kind != "sca")
                | "- **\(.Title)** [\(.Severity)] `\(.DedupKey|split(":")[1])`\n  \(.Description|split("\n")[0])"' synapse.json
              echo
            fi
            echo "_Per line annotations live in the repository Code scanning tab._"
          } > comment.md
          gh pr comment "$PR" --edit-last --body-file comment.md || gh pr comment "$PR" --body-file comment.md

A few things worth calling out:

The gate step (id: scan) has no continue-on-error, so when it fails the whole job fails. That is what turns CI red.
The upload and comment steps use if: always(), so they still run after the gate fails. That is how you get both the merge block and the annotations plus the comment.
security-events: write is required for upload-sarif, and pull-requests: write is required to post the comment.
The report is built from synapse-cli scan --json, which carries the fixed version, EPSS, CVSS and a remediation description per finding. That is what lets the comment say "upgrade to X" and "here is how to fix it", not just "there is a problem".

Actually blocking the merge

The CLI only produces a pass or fail signal. Blocking the merge is GitHub branch protection using that check as a required status.

# require the "scan" check to pass before merging into main
# enforce_admins=true so even an admin cannot bypass it
gh api -X PUT repos/OWNER/REPO/branches/main/protection --input - <<'JSON'
{
  "required_status_checks": { "strict": false, "contexts": ["scan"] },
  "enforce_admins": true,
  "required_pull_request_reviews": null,
  "restrictions": null
}
JSON

From here, any PR where the scan job fails cannot be merged.

What it looks like in practice

I set up a demo repo: a clean main (CI green), then a PR that adds a "payments service" with three deliberate problems. A Dockerfile that runs as root and pulls a script from an external URL, a function that uses MD5 to build a token, and a requirements.txt pinned to old Django, PyYAML and requests full of CVEs.

That PR produced:

Gate FAILED: findings at and above high, so the job went red.
51 alerts in the Code scanning tab, each with a file location: most on requirements.txt, several on Dockerfile, and one on app/payments.py:6 where the MD5 call is.
A single PR comment that reads like a real report:
- a severity summary table
- a "Dependencies to upgrade" table (for example django 2.2.0 needs >= 5.2.8, requests 2.19.0 needs >= 2.33.0), with a collapsible list of every CVE and its fixed version sorted by severity and EPSS
- a "Code and config" section with each issue as file:line, its rule id, and a one line remediation ("MD5 is broken, use SHA-256 or a KDF", "add a non-root USER before the entrypoint", and so on)
The merge blocked. I ran gh pr merge for real and GitHub answered: the base branch policy prohibits the merge.

One run caught all four kinds: vulnerable dependencies, weak source code, and an unsafe Docker config, all rolled into a single pass or fail decision, with the fix spelled out on the PR.

The SARIF gotcha worth knowing

This is the part I most wanted to share, because it only showed up when I ran the real thing.

On the first run the SARIF upload failed with:

Code Scanning could not process the submitted SARIF file:
expected a physical location

It turns out GitHub code scanning rejects the entire file if any result has a location that carries only a logicalLocation and no physicalLocation. The SCA findings were pointing at a "module" (something like django@2.2.0) with no concrete file, so they were all rejected.

The correct fix, which is also what Trivy does, is to point each SCA finding at the manifest that declares it, for example requirements.txt. If the manifest is unknown, leave the result with no location at all, since GitHub still accepts that as a repo level alert. Never emit a location that is logical only.

The lesson if you generate SARIF yourself: every result you want GitHub to annotate on a line needs a physicalLocation, and artifactLocation.uri must be a repo relative path with no leading slash. This is handled inside synapse-cli now, but if you write your own exporter, keep it in mind.

How it compares to Trivy

I ran both side by side on the same large real repo:

Unique CVEs: Synapse 261, Trivy 239, with 235 in common. The four CVEs only Trivy reported were all one package where the two tools resolved a different version, not a database gap.
License detection: Synapse attached a license to 1443 packages, Trivy to 1394.
Misconfig: Trivy has more low severity rules in raw count (270 to 208), but both now cover Dockerfile, Kubernetes, Helm and Terraform.
Plus SAST and secret scanning, which trivy fs does not do.

The short version: detection is at parity with a few areas ahead, and Synapse adds the governance layer that the plain scanners do not have.

Come and contribute

Synapse is open source under Apache-2.0 at github.com/KKloudTarus/synapse-ce.

If you find it useful, I would love for you to jump in:

Drop a star if the project looks worth following. It helps other people find it.
Run it in your own pipeline and open an issue about what breaks or what is missing. Feedback from a real CI run is the most valuable kind.
Pick up an issue labeled good first issue, or add a lockfile parser, a SAST or misconfig rule, or a new ecosystem for SCA. The architecture is cleanly layered, so adding a tool does not touch the core.
There is a CONTRIBUTING.md and a full docs set to get you started.

You can fork the demo repo and play with it right away: github.com/nghiadaulau/synapse-ci-demo (open PR #1 to see the blocked merge and the report).

If you wire this gate into your own project, drop a comment below and tell me what it caught on the first run. My guess is there will be a surprise.

Introducing Synapse: a deterministic-first, open-source SCA and evidence platform

Huỳnh Lê Nhất Nghĩa — Mon, 06 Jul 2026 18:48:33 +0000

We just open-sourced Synapse, a governed control plane for software composition analysis,
recon, evidence, and reporting. It is built for people who have to scan a dependency tree,
prove what they found, and hand over a report that holds up.

Site: https://synapse.kkloudtarus.net/
Code: https://github.com/KKloudTarus/synapse-ce (Apache-2.0)

Why we built it

The usual workflow is fragmented. One tool for the SBOM, another for vulnerabilities, a
spreadsheet for licenses, a folder of screenshots for evidence, and a report you assemble by
hand. Nothing is reproducible, and when a client asks "how do you know this is real," the
answer lives in someone's memory. Adding an LLM that writes your findings only makes that
worse. We wanted the opposite: fast, but provable.

What it is

Synapse runs the assessment lifecycle behind one control plane, in Go, clean architecture. A
few ideas hold it together:

Deterministic-first. Scanning, matching, license classification, and reporting are pure, reproducible Go. There is no model in the report path.
Scope-gated execution. Every engagement carries a scope and an authorization window, enforced server-side before any tool runs. Tools run via argument arrays, never a shell string.
Tamper-evident evidence. Every artifact is hash-chained and append-only. A broken chain blocks the report.
Bounded automation. The optional AI layer only ever proposes. A distinct verifier or a human confirms. The agent can never confirm its own claim.

What it does today: SBOM across 15+ ecosystems, multi-source vulnerability detection with
risk-based prioritization (KEV, then EPSS, then CVSS), license compliance, reachability, and
deterministic reports in CycloneDX, SPDX, SARIF, and OpenVEX.

Try it

git clone https://github.com/KKloudTarus/synapse-ce.git
cd synapse-ce
docker compose -f deploy/docker-compose.full.yml up --build
# open http://localhost:5173

Or gate CI on real risk: ./bin/synapse-cli scan . --fail-on high.

We are looking for contributors

Synapse is open source because this kind of tool gets better with more eyes on it. Good places
to jump in:

New ecosystems and lockfile parsers
Detection quality and false-positive reduction
Reachability and SAST coverage
Reporting and standards (OpenVEX, CSAF, SARIF)
The dashboard (Vite, React, Tailwind) and the docs

Start with the documentation
and CONTRIBUTING, run it
locally, then open an issue or send a pull request.

One note: Synapse is for authorized security testing only. It validates scope data but cannot
verify your legal authorization, so use it only against systems you are allowed to test.

If it looks useful, a star helps other people find it, and if you want to build with us, open
an issue and say hello.

Building a Production Serverless URL Shortener on AWS — 21 Articles, Every Test Run for Real

Huỳnh Lê Nhất Nghĩa — Tue, 26 May 2026 00:40:26 +0000

Most serverless tutorials stop at a hello-world Lambda behind API Gateway. I wanted the opposite: build one real product end to end, run every command on real AWS, and write
down the actual numbers — including the ones that didn't go as planned.

The result is a 21-article series that builds a URL shortener with realtime click analytics, fully serverless, and takes it to production: auth, multi-tenancy, an event
pipeline, realtime push, observability, CI/CD with canary deploys, a cost breakdown, and a load test. Code:
github.com/nghiadaulau/serverless-url-shortener-aws.

The product

       ┌──────── Cognito (JWT) ─────────┐

Browser ─HTTPS──▶ API Gateway (HTTP API) │ auth
│ │ │ │
POST /links ───┼─────▶ create ──▶ DynamoDB (single-table)
GET /{code} ───┼─────▶ resolve ─┬─▶ 301 redirect
│ │ └─▶ EventBridge ─▶ SQS(+DLQ) ─▶ aggregator
│ │ │ count + push
└─WebSocket───────────────────────────────────◀── realtime dashboard

Step Functions (link moderation) · X-Ray · CloudWatch alarms
SAM (IaC) · GitHub Actions CI · CodeDeploy canary + rollback

Nothing here is a server you keep running. Idle cost is effectively zero.

A few findings that surprised me (all measured, not quoted)

Memory is CPU. The same CPU-bound work at 128 MB vs 1769 MB:

128 MB: 2594 ms
1769 MB: 88 ms → ~29x faster, and ~2.25x cheaper (memory × billed time)

At 128 MB a Lambda is CPU-starved, not memory-starved (Max Memory Used stayed ~82 MB in both). The same lever cut a cold start from 1513 ms to 295 ms.

The concurrency ceiling is a quota, not your code. Load testing with k6 at ~554 req/s:

Total requests: 27741
301 (success): 304 (1.1%)
503 (overload): 25 (0.1%)
429 (throttled): ~98.8%
ConcurrentExecutions (max): 10

This account had a reduced Lambda concurrency limit of 10, and an API Gateway rate throttle I'd set deliberately. So under load the system sheds traffic in two layers (429
at the gateway, 503 at Lambda) and stays fast for what it serves — graceful degradation, not a crash. The fix isn't optimizing code; it's raising the quotas.

Idempotency without a framework. Click events are delivered at-least-once, so naive counting double-counts. One DynamoDB TransactWriteItems bumps the counters and
writes a CLICK#<eventId> marker with attribute_not_exists — a duplicate cancels the whole transaction, so it's counted exactly once.

The bill. Building and testing the entire thing — API, database, auth, event bus, queues, realtime, state machine, observability — cost essentially $0, all within
free tier.

What the series covers

Foundations — SAM, the Lambda execution lifecycle, cold starts, arm64
Core — HTTP API vs REST API, DynamoDB single-table design, GSIs and sparse indexes, conditional writes, atomic counters
Auth — Cognito + JWT authorizer, multi-tenancy, blocking IDOR at the data layer
Event-driven — EventBridge, SQS + DLQ, idempotency, partial batch failure, WebSocket realtime push, Step Functions + the saga pattern
Production — Powertools + X-Ray, CloudWatch alarms and SLOs, cold-start optimization, IAM least-privilege, throttling
Operations — CI/CD with canary deploys and rollback, a real cost breakdown, load testing with k6, a Well-Architected review

Every article is grounded in the official AWS docs and ends with cleanup so you never get a surprise bill.

Read it

Full series (English): https://kkloudtarus.net/en/blog/what-is-serverless-when-to-use
Code: https://github.com/nghiadaulau/serverless-url-shortener-aws

If you're learning serverless beyond hello-world, this is the path I wish I'd had. Questions and corrections welcome.

eBPF From Scratch: from the eBPF VM to writing your own tools (tested on a live Cilium cluster)

Huỳnh Lê Nhất Nghĩa — Sun, 24 May 2026 15:08:00 +0000

Right now, on one worker of a Kubernetes cluster I built, 140 eBPF programs are running inside the Linux kernel — routing every packet, controlling device access, collecting metrics. Nobody recompiled the kernel. Nobody loaded a module.

That sentence is the whole reason I wrote eBPF From Scratch: a free, 22-chapter series that takes you from "what even is the eBPF virtual machine" all the way to writing and loading your own eBPF programs — in C with libbpf + CO-RE, and from Go with cilium/ebpf.

👉 Read it free (English): https://kkloudtarus.net/en/blog/series/ebpf-from-scratch
💻 Source code: https://github.com/nghiadaulau/ebpf-from-scratch

What makes it different

Everything is tested on real hardware — a Kubernetes cluster running kernel 6.17 and Cilium 1.19 (kube-proxy-less, hundreds of BPF programs live) is the lab throughout. No hand-wavy diagrams; we dissect programs that are actually running.
Grounded in official docs — ebpf.io, kernel.org, libbpf, cilium. Claims are verified, not vibes.
Deep-dive, not surface — we go down to registers, the verifier's safety proofs, JIT, maps, and the exact lifecycle a program goes through.
It's also fully bilingual (English + Vietnamese).

A quick taste — every eBPF concept is something you can see on a real node:

sudo bpftool prog show id 2871

2871: sched_cls  name tail_no_service_ipv4  tag fe7bcb57c001d434  gpl
    xlated 4920B  jited 2778B  memlock 8192B  map_ids 171,631
    btf_id 758

xlated = bytecode after the verifier accepted it. jited = native machine code. map_ids = how it keeps state. That's eBPF, not on a slide — running.

What you'll learn (7 parts, 22 chapters)

Part I — Foundations

The eBPF Virtual Machine: registers, instruction set, and bytecode
The Verifier: why eBPF doesn't crash the kernel
Maps: memory and the bridge to userspace
Program types and hooks: where you attach, what you see
BTF and CO-RE: compile once, run on every kernel

Part II — Tracing

bpftrace from a one-liner to maps, counting and histograms
uprobe, USDT, and inspecting a pod from the host

Part III — Writing real tools

libbpf + CO-RE: writing an eBPF tool yourself (C)
cilium/ebpf: loading eBPF from Go

Part IV — Networking

XDP: processing packets at the earliest point — writing a firewall
tc/sched_cls and dissecting a live Cilium datapath
Writing a tc program yourself: __sk_buff and the tcx chain

Part V — Security

LSM BPF: enforcing security right inside the kernel
seccomp-bpf: filtering syscalls in every container
The Tetragon way: from observe to enforce with bpf_send_signal

Part VI — Observability

CPU profiling with perf_event (the foundation of flame graphs)
Off-CPU and scheduler latency
Inside Hubble: from eBPF events to cluster-wide network flows

Part VII — Putting it together

Case study: a packet through Cilium's eBPF datapath
Capstone: writing connmon, a node-wide TCP connection monitor

Who it's for

Backend / platform / SRE / security folks who keep hearing "Cilium does that with eBPF" and want to actually understand — and write — the thing. You'll want to be comfortable on the Linux command line; everything else is built up from zero.

If you read any of it, I'd genuinely love feedback — what was clear, what wasn't. And if it's useful, a ⭐ on the repo helps a lot.

Start here → https://kkloudtarus.net/en/blog/series/ebpf-from-scratch

Kubernetes From Scratch: build an HA cluster by hand (no kubeadm), then deep-dive every concept

Huỳnh Lê Nhất Nghĩa — Sun, 24 May 2026 15:06:41 +0000

Most Kubernetes tutorials start with kubeadm init or a managed cluster. You get a working cluster and almost no idea why it works.

So I did the opposite. Kubernetes From Scratch builds a complete, highly-available cluster by hand — no kubeadm, no scripts — starting from the very first TLS certificate, all the way to a real HA control plane. Then it uses that cluster as a lab to deep-dive every concept in Kubernetes: 73 chapters.

👉 Read it free (English): https://kkloudtarus.net/en/blog/series/kubernetes-from-scratch
💻 Manifests & scripts: https://github.com/nghiadaulau/kubernetes-from-scratch

What makes it different

You build it by hand. cfssl-signed certs, etcd quorum, apiserver/controller-manager/scheduler wired up yourself, kubelet + containerd on the workers, pod networking with the CNI bridge + VPC routes, CoreDNS — then a smoke test of the whole thing running together.
Then it goes deep. Once the cluster is up, every concept gets explained from the inside and configured by hand — not just "apply this YAML".
Tested for real on AWS EC2 with Kubernetes v1.36, grounded in the official kubernetes.io docs.
Fully bilingual (English + Vietnamese).

The journey

Part 1 — Build the cluster from nothing (chapters 0–17)
PKI/TLS and why a cluster needs so many certificates → etcd (quorum, Raft) → kube-apiserver and the request pipeline → controller-manager & scheduler (control loops, leader election) → HAProxy in front of 3 API servers → containerd/CRI → kubelet → kube-proxy → the Kubernetes network model → wiring pod networking by hand → CoreDNS → a full smoke test → and finally, the lifecycle of a request: from kubectl apply to a running pod.

Part 2 — Deep-dive every concept (chapters 18–72)
Using the cluster you built:

Workloads: Pod lifecycle, init/sidecar containers, probes, Deployment/StatefulSet/DaemonSet/Job, requests/limits/QoS, PodDisruptionBudgets
Scheduling: the scheduling framework, affinity/taints/tolerations, topology spread, priority & preemption, node-pressure eviction, HPA/VPA
Storage: volumes, PV/PVC, StorageClass + dynamic provisioning (EBS CSI), VolumeSnapshots
Networking: why replace kube-proxy, migrating to kube-proxy-less Cilium (eBPF), NetworkPolicy, Ingress, Gateway API, LB IPAM
Security: authentication → RBAC → ServiceAccounts & bound tokens, Pod Security Standards, seccomp/AppArmor/capabilities, Secrets hardening
Extending K8s: CRDs, admission webhooks, the operator pattern, API aggregation, device plugins
Operations: etcd backup & cert rotation, upgrades & version skew, logging, metrics/traces/APF, CEL admission policy, in-place pod resize, and a full teardown

Who it's for

Anyone who can use Kubernetes but wants to truly understand it — for the CKA/CKS, for debugging production with confidence, or just to finally kill the "it's magic" feeling. Comfortable-with-Linux is the only prerequisite.

If this helps you, a ⭐ on the repo means a lot — and I'd love to hear which chapter clicked for you.

Start here → https://kkloudtarus.net/en/blog/series/kubernetes-from-scratch

Getting Started with EVS CLI: Manage Amazon Elastic VMware Service from the Command Line

Huỳnh Lê Nhất Nghĩa — Wed, 06 Aug 2025 15:06:51 +0000

🚀 Getting Started with EVS CLI: Manage Amazon Elastic VMware Service from the Command Line

Amazon EVS is currently in public preview and subject to change.

Amazon Elastic VMware Service (EVS) is a fully managed service that allows you to run VMware Cloud Foundation (VCF) directly on EC2 bare metal instances inside your Amazon VPC, without refactoring workloads or changing operational tools.

With the launch of the EVS CLI, you now have a powerful command-line interface to automate and manage your EVS infrastructure, ideal for scripting, DevOps pipelines, and quick access tasks.

🧰 What is EVS CLI?

The EVS CLI is an AWS CLI-compatible tool to help manage your Amazon EVS environments programmatically. You can:

🚧 Create or delete VCF environments
🧱 Add or remove ESXi hosts
📡 List VLANs and connected resources
🏷️ Tag and organize environments
📥 Retrieve environment information

It complements AWS Console and offers Infrastructure-as-Code (IaC) like capabilities directly in shell environments.

📋 Available Commands

Here's a quick overview of commands you can run with evs:

Command	Description
`create-environment`	Provision a new VCF environment
`create-environment-host`	Add a new host to an existing EVS environment
`delete-environment`	Remove an entire EVS environment
`delete-environment-host`	Remove a host from a deployed environment
`list-environments`	List all EVS environments
`list-environment-hosts`	View hosts within an environment
`list-environment-vlans`	View VLAN setup in an environment
`get-environment`	Fetch detailed information of an EVS environment
`tag-resource` / `untag-resource`	Manage tags on EVS resources
`list-tags-for-resource`	Display tags on a specific resource

⚙️ Creating Your First EVS Environment

The most essential command is create-environment. It provisions:

vCenter Server
NSX Manager
SDDC Manager
ESXi hosts (min 4)
VLANs: Management, vMotion, vSAN, etc.

🧪 Sample Command

aws evs create-environment \
  --environment-name "prod-vcf-env" \
  --vpc-id vpc-0a12b345cdef67890 \
  --service-access-subnet-id subnet-0123abcd4567ef890 \
  --vcf-version VCF-5.2.1 \
  --terms-accepted \
  --license-info file://license.json \
  --initial-vlans file://vlans.json \
  --hosts file://hosts.json \
  --connectivity-info file://connectivity.json \
  --vcf-hostnames file://vcf-hostnames.json \
  --site-id "SITE-123456"

📝 Note: Provisioning takes several hours. Host count must be 4 to 16, and all subnets must reside in a single Availability Zone.

🛠️ Example: hosts.json

[
  {
    "hostName": "esxi01.mycorp.local",
    "keyName": "evs-key",
    "instanceType": "i4i.metal"
  },
  {
    "hostName": "esxi02.mycorp.local",
    "keyName": "evs-key",
    "instanceType": "i4i.metal"
  },
  ...
]

🧹 Cleaning Up Resources

Once you’re done, it’s critical to clean up properly to avoid ongoing charges.

Delete a host:

aws evs delete-environment-host \
  --environment-id env-abc123456 \
  --host-name esxi01.mycorp.local

⚠️ You must decommission the host in SDDC Manager UI before deleting it via CLI.

aws evs delete-environment-host \
  --environment-id env-abc123456 \
  --host-name esxi01.mycorp.local

Delete the entire environment:

aws evs delete-environment \
  --environment-id env-abc123456

🛡️ Requirements Summary

✅ Minimum 4 hosts, maximum 16
✅ VCF Version: VCF-5.2.1
✅ Valid VCF and vSAN license keys from Broadcom
✅ CIDRs for VLAN subnets must not overlap
✅ All subnets must be in one Availability Zone
🧠 Why Use EVS CLI?
📦 Infrastructure-as-Code: fits easily into Terraform or CI/CD workflows
⚙️ Automation: simplifies provisioning and scaling
🧪 Testing: replicate environments consistently
🔍 Debugging: quick access to environment state
🧩 Additional Tips
Use --generate-cli-skeleton to scaffold input JSON or YAML files
Use --debug for troubleshooting and verbose logs
Secure your credentials using named --profile or environment variables

📌 Final Thoughts
Amazon EVS with CLI support marks a new era for hybrid cloud operations. If your team relies on VMware, this opens a path to modernize workloads with minimal disruption while enjoying the scale and security of AWS.

Try out the CLI and start building your first EVS environment today.

Amazon Elastic VMware Service (EVS) is Now GA — Run VMware Cloud Foundation Natively in AWS

Huỳnh Lê Nhất Nghĩa — Wed, 06 Aug 2025 14:43:06 +0000

🚀 Amazon Elastic VMware Service (EVS) is now generally available. If you're managing VMware workloads, this new service might be the bridge to the cloud you've been waiting for.

🧩 What is Amazon EVS?

Amazon Elastic VMware Service (EVS) lets you deploy and run VMware Cloud Foundation (VCF) environments directly on EC2 bare metal instances inside your Amazon VPC.

No replatforming, no IP reassignments, no retraining. Just your existing VMware stack — now in AWS.

🔧 Key Features

✅ Drop-in migration — Lift-and-shift workloads without changing IPs or refactoring applications.
✅ Full administrative control — Customize and operate your VMware stack just like on-prem.
✅ Self-managed or Partner-operated — Choose your ops model based on team capacity.
✅ License portability — Bring your existing VCF licenses to EVS.
✅ Hybrid-ready — Extend on-prem networks and workloads with zero disruption.
✅ AWS-native integration — Connect with 200+ AWS services including Lambda, S3, Aurora, Bedrock, and more.

🌍 Available Now in 6 Regions

Amazon EVS is now GA in:

🇺🇸 US East (N. Virginia)
🇺🇸 US East (Ohio)
🇺🇸 US West (Oregon)
🇯🇵 Asia Pacific (Tokyo)
🇩🇪 Europe (Frankfurt)
🇮🇪 Europe (Ireland)

Supported:

VCF Version: 5.2.1
Instance Type: i4i.metal
Storage: Compatible with external storage and backup tools

🔄 Use Cases to Explore

Here are some hybrid scenarios Amazon EVS enables:

🏢 Datacenter Exit Strategy

Migrate entire clusters from on-prem to AWS without rewriting infrastructure.

💼 Disaster Recovery & Resilience

Use EVS as your DR site with full VMware feature parity.

⚙️ Dev/Test Bursting

Spin up VCF environments in AWS for short-term workloads, testing, or isolated dev.

🧠 Gradual Modernization

Move VMs to EVS and start layering AWS-native services like S3, RDS, or even Bedrock-based AI.

🧪 Getting Started with EVS

Here's how to spin up your first Amazon EVS environment:

Complete prerequisites (VPC, CIDR, IAM, etc.)
Create your EVS environment via:
- AWS Console
- AWS CLI
- CloudFormation (AWS::EVS::Environment)
Customize VCF in the familiar vSphere UI
Connect & Migrate your workloads

📘 Official Docs – Getting Started

🖥️ EVS Console Access

💸 Pricing

💬 Final Thoughts

If you've been looking for a cloud migration path that doesn't disrupt your VMware expertise, tooling, or licensing, Amazon EVS might be your golden ticket.

You get the operational consistency of VMware + the scale and innovation of AWS.

✅ Try It Today

🔗 Explore Amazon EVS on AWS

📩 Thinking about migrating? Let’s talk about your use case!

🛠 Have a hybrid cloud story? Share it in the comments!

Thanks for reading! Follow me for more AWS deep dives and hybrid cloud updates.

Beyond Auto-Complete: Supercharge Your Workflow with AWS's Newest AI, Amazon Q

Huỳnh Lê Nhất Nghĩa — Wed, 06 Aug 2025 14:14:00 +0000

You've probably used AI code assistants. They're great for boilerplate, auto-completing a function, or even writing a unit test. But what if an AI could do more than just write the code you tell it to? What if it could understand your intent, debug multi-step problems, and even upgrade your application's framework for you?

That's the promise of Amazon Q Developer, AWS's new, supercharged generative AI assistant. It's more than just a chatbot that knows the AWS docs; it's an active participant in your development workflow. If you're an experienced developer looking for the next leap in productivity, this is a tool you need to pay attention to.

This post isn't just about what Amazon Q is. We'll dive into the specific, powerful features that move beyond simple code generation and show you how to leverage them with concrete examples.

From Code Whisperer to True Collaborator

AWS has had CodeWhisperer, its direct competitor to GitHub Copilot. It's solid for inline code suggestions. But Amazon Q represents a fundamental shift. It's less of a passive "whisperer" and more of an active "collaborator."

The biggest leap forward is its agentic capabilities. An "agent" in this context is an AI that can perform a series of actions to achieve a goal. Instead of just suggesting a single block of code, Amazon Q can:

Scan your entire project to understand its structure and dependencies.
Reason about the steps needed to complete a complex task.
Read files, write new code, and apply diffs on your behalf.
Run shell commands to test its own work.

This is where things get interesting for seasoned developers. Think about the time you spend on tasks that are more than just writing a single function.

Killer Feature #1: The `/fix` Command for In-IDE Debugging

Imagine you're running your application and hit a cryptic error message in the console. Your usual workflow might be to copy-paste the error into Google and hunt through Stack Overflow.

With Amazon Q, you streamline this dramatically.

Scenario: A Null Pointer Exception in Java

Let's say your test run fails with this common Java error:

Exception in thread "main" java.lang.NullPointerException
    at com.example.myapp.service.DataProcessor.process(DataProcessor.java:25)
    at com.example.myapp.Main.main(Main.java:10)

Instead of leaving your IDE, you open the Amazon Q chat window and type:

/fix Exception in thread "main" java.lang.NullPointerException at com.example.myapp.service.DataProcessor.process(DataProcessor.java:25)

Amazon Q doesn't just look up the error. It analyzes DataProcessor.java at line 25 within the context of your project. It might see that you're calling a method on an object that wasn't initialized.

It will then propose a specific, multi-file solution with a complete diff that you can review.

Q's Proposed Fix (Diff View):

--- a/src/main/java/com/example/myapp/service/DataProcessor.java
+++ b/src/main/java/com/example/myapp/service/DataProcessor.java
@@ -22,7 +22,9 @@

     public void process(Data data) {
         // The object 'someObject' was not initialized.
-        String result = data.getSomeObject().getValue();
+        if (data != null && data.getSomeObject() != null) {
+            String result = data.getSomeObject().getValue();
+        }
         System.out.println("Processing: " + result);
     }
 }

You get a clear, context-aware suggestion that you can accept with one click. It's like pair-programming with someone who has already diagnosed the problem.

Killer Feature #2: Agentic Framework Upgrades

This is a game-changer for dealing with technical debt. Let's say you have a Java 8 application using Spring Boot 2.x and you need to upgrade to Java 17 and Spring Boot 3.x. This is a notoriously complex task.

With Amazon Q's upgrade agent, you can initiate this from the command line (after installing the Q tools).

Command to Start the Upgrade:

// Make sure you are in your project's root directory
amazon-q-transformation upgrade --from-version 8 --to-version 17

Amazon Q will then:

Analyze Dependencies: It scans your pom.xml or build.gradle and identifies all dependencies that need updating (e.g., Spring Boot, Jakarta EE).
Refactor Code: It systematically refactors code that relies on deprecated APIs. For instance, it knows that javax.persistence.* needs to be changed to jakarta.persistence.*.

Example: Refactoring javax to jakarta

Old Code (Before Q):

import javax.persistence.Entity;
import javax.persistence.Id;

@Entity
public class User {
    @Id
    private Long id;
    // ...
}

New Code (After Q's Transformation):

import jakarta.persistence.Entity;
import jakarta.persistence.Id;

@Entity
public class User {
    @Id
    private Long id;
    // ...
}

This is just one of hundreds of changes it can make automatically. It handles the tedious work, leaving you to focus on testing and logic verification.

Killer Feature #3: Implementing Features with `/dev`

Here's where it gets really powerful for teams. The /dev command acts as a "workspace agent." You can give it high-level tasks.

Scenario: Add a new API endpoint

Imagine you need to add a standard CRUD API for a Product entity. Instead of creating all the files manually, you can instruct Q:

/dev create a new REST API for a 'Product' resource. It should have full CRUD operations (GET all, GET by ID, POST, PUT, DELETE). Use Spring Boot conventions and create a controller, service, and repository layer. The Product entity has 'id', 'name', and 'price' fields.

Amazon Q will then generate a plan and, upon your approval, create the following file structure and content:

Plan Proposed by Q:

I will perform the following actions:
1.  ✅ Create `src/main/java/com/example/myapp/model/Product.java` with id, name, and price.
2.  ✅ Create `src/main/java/com/example/myapp/repository/ProductRepository.java` extending JpaRepository.
3.  ✅ Create `src/main/java/com/example/myapp/service/ProductService.java` with business logic for CRUD.
4.  ✅ Create `src/main/java/com/example/myapp/controller/ProductController.java` with REST endpoints.
5.  ✅ Suggest a new unit test file `ProductControllerTest.java`.

Example Code Generated for ProductController.java:

@RestController
@RequestMapping("/api/products")
public class ProductController {

    @Autowired
    private ProductService productService;

    @GetMapping
    public List<Product> getAllProducts() {
        return productService.findAll();
    }

    @GetMapping("/{id}")
    public ResponseEntity<Product> getProductById(@PathVariable Long id) {
        return productService.findById(id)
                .map(ResponseEntity::ok)
                .orElse(ResponseEntity.notFound().build());
    }

    @PostMapping
    public Product createProduct(@RequestBody Product product) {
        return productService.save(product);
    }

    // ... other methods for PUT and DELETE
}

It takes a high-level developer request and turns it into fully-scaffolded, idiomatic code, saving you a significant amount of setup time.

Conclusion: Is It Worth It?

AI assistants are becoming standard, but Amazon Q Developer is making a strong case for being best-in-class. For intermediate and advanced developers, the value isn't in generating simple loops; it's in automating the complex, time-consuming tasks that bog us down.

Features like context-aware debugging (/fix), automated framework upgrades, and the ability to act as a workspace agent (/dev) are significant productivity multipliers. By understanding your entire codebase and executing multi-step plans, Amazon Q is moving the goalposts for what a developer assistant can be.

If you're already in the AWS ecosystem, it's a no-brainer to integrate it into your IDE and give it a try. The time you save on just one nasty bug or one framework upgrade could easily justify the investment. The future of development is collaborative, and AWS is making a powerful new collaborator available to all of

DEV Community: Huỳnh Lê Nhất Nghĩa

Migrating & Upgrading GitLab 15.6.1 to 19.2: New Box, Going Private, Cutover Behind Production

1. Context and constraints

2. Architecture: before and after

3. Phase 1: backup without touching the old box

4. Phase 2: restore and step-upgrade — this is the time sink

4.1. Bugs hit during the step-upgrade

4.2. How to speed up the upgrade

5. Phase 3: going private with NLB and PrivateLink

6. Phase 4: cutover with a Route53 private zone (split-horizon)

7. Seven little demons after the cutover

7.1. SSM timeout and stale PID lock

7.2. SSH host key mismatch, ArgoCD frozen

7.3. Empty authorized_keys after restore, publickey auth fails

7.4. NLB cross-zone not enabled, 2/3 endpoint IPs can't reach port 22

7.5. A cross-account Route53 alias to a VPC endpoint won't resolve

7.6. DNS negative caching

7.7. VPN missing a route push, devs report "connected but still can't get in"

8. Tip: a debug DaemonSet for instant network probing

9. Results and lessons

Come contribute

A security merge gate is only as good as its false positive rate

The gate has to be explainable

Precision is a release gate for the rules, not just for your code

AI generated code and the shared blind spot

Let the model propose, do not let it decide

The gate stays green because it is boring

The quiet part

Come contribute

Let an AI clear out your false positives without letting it hide a real bug

Where the actual "gate" is

Turning it on

What actually happened

Before: no AI

After: AI on, two models

Verdict by verdict

"But what if the AI is wrong?"

Wiring it into CI

Come contribute

Synapse Feature Demo: Deterministic Software Composition Analysis in Action

Block the merge when a PR ships a vulnerability: a CI security gate with Synapse

Why another tool

What you need

The full workflow

Actually blocking the merge

What it looks like in practice

The SARIF gotcha worth knowing

How it compares to Trivy

Come and contribute

Introducing Synapse: a deterministic-first, open-source SCA and evidence platform

Why we built it

What it is

Try it

We are looking for contributors

Building a Production Serverless URL Shortener on AWS — 21 Articles, Every Test Run for Real

The product

A few findings that surprised me (all measured, not quoted)

What the series covers

Read it

eBPF From Scratch: from the eBPF VM to writing your own tools (tested on a live Cilium cluster)

What makes it different

What you'll learn (7 parts, 22 chapters)

Who it's for

Kubernetes From Scratch: build an HA cluster by hand (no kubeadm), then deep-dive every concept

What makes it different

The journey

Who it's for

Getting Started with EVS CLI: Manage Amazon Elastic VMware Service from the Command Line

🚀 Getting Started with EVS CLI: Manage Amazon Elastic VMware Service from the Command Line

🧰 What is EVS CLI?

📋 Available Commands

⚙️ Creating Your First EVS Environment

🧪 Sample Command

🛠️ Example: hosts.json

🧹 Cleaning Up Resources

Amazon Elastic VMware Service (EVS) is Now GA — Run VMware Cloud Foundation Natively in AWS

🧩 What is Amazon EVS?

🔧 Key Features

🌍 Available Now in 6 Regions

🔄 Use Cases to Explore

7.3. Empty `authorized_keys` after restore, publickey auth fails

Killer Feature #1: The `/fix` Command for In-IDE Debugging

Killer Feature #3: Implementing Features with `/dev`