How We Solved ArgoCD Notifications' Single-Secret Limitation with Kyverno

Note: I wrote this post a while back, before Kyverno introduced its new Mutating Policies (released in Kyverno 1.15). The Kyverno manifests here use the older approach, but the core concept remains the same.

If you run ArgoCD Notifications with multiple integrations (for example, GitLab + Grafana), you will eventually hit the single-secret constraint. This post walks through the options I evaluated, why most of them were not so elegant or risky, and the Kyverno policy pattern that worked cleanly in production.

TL;DR

ArgoCD Notifications is hardcoded to read from a single secret. When you need credentials for multiple services — each managed independently — that becomes a problem fast.

• The constraint: ArgoCD Notifications only reads from argocd-notifications-secret
• The complication: Two Vault Secret Operator (VSO) resources (static or dynamic) cannot safely co-manage the same Kubernetes Secret object — they’ll fight and flap
• The fix: Use a Kyverno mutate policy to merge specific keys from individual source secrets into one target secret

A Bit of Context

We run a GitOps workflow using ArgoCD. App code lives in one repo, rendered manifests are committed to a deploy repo by GitLab CI, and ArgoCD reconciles those manifests across clusters.

After each deployment, ArgoCD Notifications sends a webhook back to the GitLab pipeline with the sync status. This webhook setup tells us whether the deployment actually landed successfully.

For secret management, we use Vault Secret Operator (VSO). It talks to HashiCorp Vault and provisions secrets directly into the cluster.

Below is a very high-level overview of VSO for the context. The internals of VSO are outside the scope of this post.

The Problem

This setup is very common in the industry, and ArgoCD Notifications is a great way to send deployment updates to external systems. It supports Slack, Microsoft Teams, Grafana, custom webhooks (which is how GitLab is integrated in this case), etc.

In our case, we needed two notifications firing after every deployment: a GitLab webhook and a Grafana annotation on our dashboards.

See the Docs: ArgoCD Grafana notification docs for details.

Grafana notification support is already built into ArgoCD — it just needs a Grafana API token and a few configuration changes on the ArgoCD application side (See ArgoCD Grafana notification docs).

However, there is one catch: all service credentials for the ArgoCD notification controller must live in a single Kubernetes Secret named argocd-notifications-secret. That is the only place the ArgoCD Notifications controller will look.

If you are to notify multiple services, all credentials need to be consolidated into that one secret.

At this point, the GitLab token was already being managed by VSO. Now we needed the Grafana token to live alongside it.

So it would look like this,

At least at the time of writing, there was no way to point ArgoCD Notifications to multiple secrets (for example, one for GitLab and one for Grafana), or to another secret name.

So it was one secret, both the tokens had to end up in the same secret, but still be managed separately.

Co-locating unrelated credentials in one place raises both operational and security concerns, worth avoiding.

At that point, a few options were left.

Options Considered

Option 1 — A separate secret for the Grafana token

We already established this that (at the time of writing) this was not possible. ArgoCD Notifications was hardcoded to use argocd-notifications-secret.

Option 2a — Combine both tokens into one VSO secret

Technically possible. Store both tokens in one Vault path (a single location in Vault that contains both keys), then let VSO create one Kubernetes Secret from it.

But that means co-locating unrelated credentials in the same Vault path — not ideal from a security standpoint, and it creates an operational headache too. Especially once you factor in dynamic secrets, where each service should own and rotate its own credentials independently.

Option 2b — Use two VSO resources to manage the same secret

This sounds cleaner: one VSO resource per token, both targeting argocd-notifications-secret.

The issue is that VSO manages the entire Secret resource, not individual fields. If two VSO resources claim the same Secret, they continuously reconcile against each other and overwrite each other’s updates. The result is a flapping Secret that does not stabilise.

Here is a sample secret managed by VSO. Check managedFields to see ownership.

Option 3 — Use an Istio EnvoyFilter to inject the token

Since we use Istio service mesh, an EnvoyFilter could intercept requests to Grafana and inject the API token as an HTTP header, without ArgoCD knowing about it.

A clever workaround, but it comes with trade-offs:

• Wrong abstraction layer: token management becomes network behaviour.
• Coupling risk: upgrades in Istio/Envoy can break filter behaviour.
• Testing complexity: harder to validate compared to policy/controller logic.
• Scope risk: Host-based matches can affect unintended traffic after routing changes.

Option 4 — Write a custom controller or use a templating controller

We already had internal controller tooling, so this was possible. We also considered kluctl as a templating-controller path.

Both are valid, but they add operational overhead.

Option 5 — Use Kyverno

Kyverno was already on our DevOps roadmap, just not yet adopted.

Kyverno is a Kubernetes-native policy engine that can mutate resources. Instead of writing a custom controller, we could use policy-driven mutation to handle secret composition.

So, I decided to go ahead with Kyverno.

The Solution

The plan was straightforward:

1. VSO creates two source secrets (one per token):
   • argocd-notifications-secret-gitlab-merge
   • argocd-notifications-secret-grafana-merge
2. A Kyverno mutate policy watches those source secrets and patches argocd-notifications-secret.

One important detail:

Kyverno cannot watch a Secret and generate another Secret from it — it cannot generate a resource of the same kind as the trigger. What it can do is mutate a resource of the same kind.

So the approach is: use a mutate rule that is triggered by the source secrets and patches argocd-notifications-secret directly. So this means argocd-notifications-secret needs to be provisioned first.

The easiest way is to use the ArgoCD chart’s built-in secret provisioning to create an empty placeholder Secret (resource shell only). Kyverno then updates its data fields when source secrets change.

As we do not want Kyverno mutating a resource that does not exist yet, we can use ArgoCD sync-waves to ensure the empty argocd-notifications-secret is created first (by setting its sync-wave to a value lower than the VSO source secrets).

This is the Kyverno policy that ties it all together. It watches both source secrets (Kubernetes) and patches their keys into argocd-notifications-secret whenever they change:

apiVersion: kyverno.io/v1
kind: Policy
metadata:
  annotations:
    policies.kyverno.io/subject: Secrets
    policies.kyverno.io/title: argocd-notifications-secret
  name: argocd-notifications-secret
  namespace: argocd
spec:
  rules:
  - name: mutate-secret-on-gitlab-secret-update
    match:
      any:
      - resources:
          kinds:
          - Secret
          names:
          - argocd-notifications-secret-gitlab-merge
          operations:
          - CREATE
          - UPDATE
    mutate:
      mutateExistingOnPolicyUpdate: true
      targets:
      - apiVersion: v1
        kind: Secret
        name: argocd-notifications-secret
      patchStrategicMerge:
        data:
          gitlab-token: '{{ request.object.data."gitlab-token" }}'
  - name: mutate-secret-on-grafana-secret-update
    match:
      any:
      - resources:
          kinds:
          - Secret
          names:
          - argocd-notifications-secret-grafana-merge
          operations:
          - CREATE
          - UPDATE
    mutate:
      mutateExistingOnPolicyUpdate: true
      targets:
      - apiVersion: v1
        kind: Secret
        name: argocd-notifications-secret
      patchStrategicMerge:
        data:
          grafana-api-key: '{{ request.object.data."grafana-api-key" }}'

A few useful notes:

• Two rules, one policy: each rule updates a specific field from a specific source secret.
• mutateExistingOnPolicyUpdate: true: reconciles immediately on policy apply/update.
• patchStrategicMerge: merges specific keys instead of replacing the whole data map (Kyverno strategic merge patch docs).
• match: Kyverno watches for CREATE and UPDATE events on the source secrets (argocd-notifications-secret-gitlab-merge and argocd-notifications-secret-grafana-merge). Any change to either of those triggers the corresponding rule.
• targets: this tells Kyverno which resource to patch — in this case argocd-notifications-secret, the empty shell provisioned by the ArgoCD Helm chart.

And that is really it. VSO manages each source secret independently, Kyverno watches for changes and patches the right keys into argocd-notifications-secret, and ArgoCD Notifications picks it up without knowing any of this is happening.

How Kyverno Tracks Ownership — Managed Fields

One thing worth knowing is how Kubernetes handles field ownership here. Because multiple controllers are touching the same Secret, Kubernetes managed fields track which controller owns which keys — so nothing gets blindly overwritten.

In this setup:

• ArgoCD chart owns the shell (empty secret resource)
• Kyverno background controller owns the token fields that it writes

You can verify this directly:

kubectl get secrets argocd-notifications-secret \
  --show-managed-fields -n argocd -o yaml

The output includes a managedFields section showing which manager owns which keys.

Things to Keep in Mind When Running Kyverno in Production

These are some of the points that I make a note of:

1. Install all CRDs. Some Kyverno controllers can error if required CRDs are missing, even when those CRDs are not actively used. I observed this in the version we ran at that time, so check your version and release notes.
2. Give Kyverno enough resources. If the admission controller capacity is insufficient, resource creation can fail cluster-wide.
3. Test policies early. Two solid options: – Kyverno CLI tests (fast, no cluster required): docs – Chainsaw tests (integration style, with real cluster or mock API server): repo

My preference is Chainsaw for realism, while CLI tests are great for quick validation.

Wrapping Up

What started as a “just add one more token” request ended up being a neat Kyverno adoption point for us. Sometimes a constraint in one tool opens the door to doing something smarter in another.

If you are running ArgoCD with multiple notification integrations and have hit this exact wall, hopefully, this gives you a concrete path forward.

And if you have solved it differently — especially with the newer Kyverno mutation features — I would genuinely love to hear how you approached it.