Building a DevSecOps Pipeline on Alibaba Cloud

Part 7 of the Alibaba Cloud Engineering Lab Series.

Architecture

Developer
   ↓
Git
   ↓
CI Pipeline
   ↓
Code Security (SAST, dependency scan)
   ↓
Container Build
   ↓
Image Security (vulnerability scan, sign)
   ↓
Deployment (policy gate)
   ↓
ACK
   ↓
Monitoring (runtime security)

The core argument: security shouldn't be a final gate someone remembers to run before a release — it should be a property of every stage that fails the pipeline early and cheaply, rather than being caught in production expensively.

Before the how, the what — three terms this pipeline leans on:

• SAST (Static Application Security Testing) — scans source code for known-dangerous patterns (SQL injection risk, hardcoded credentials) without running it, catching classes of bugs at the pull-request stage rather than after they're merged and deployed.
• DevSecOps — folding security checks into the same CI/CD pipeline that already runs tests and linting, instead of treating security as a separate audit that happens right before (or worse, after) a release. The name is a statement of when security happens, not a different set of tools.
• Admission controller — a component that sits in front of the Kubernetes API and can approve or reject a resource (like a new pod) before it's ever scheduled. This is what makes a "policy gate" enforceable at deploy time — it's the mechanism, not just a checklist item, that can actually refuse to run an unsigned or unscanned container image.

I built this exact pipeline — dependency scan through cosign signing through the Kyverno admission gate — and hit the wrong-public-key failure in Section 07 for real, not as a hypothetical. The companion repo has the actual workflow, a real (demo) signing keypair, and the policy that enforces it.

---

Problem

The pipeline this replaced ran a single vulnerability scan manually, in an ad hoc way, whenever someone remembered before a major release — dependency CVEs shipped to production for weeks at a time, container images were built from an unpinned base image that silently pulled in new vulnerabilities on every rebuild, and there was no consistent secrets-management story: some credentials lived in CI environment variables, others were hardcoded in a "temporary" config file that had been temporary for eight months.

---

Implementation

Stage 1 — Code security (SAST + dependency scan), fails the build on high-severity findings:

- name: Dependency scan
  run: |
    trivy fs --severity HIGH,CRITICAL --exit-code 1 .
- name: SAST
  run: semgrep --config=auto --error

Stage 2 — Container build, pinned base image, non-root user:

FROM node:20.11.1-alpine3.19@sha256:abc123...
RUN addgroup -S app && adduser -S app -G app
USER app

Pinning by digest, not just tag — a tag can be repointed to different content; a digest cannot.

Stage 3 — Image security scan before push to Alibaba Container Registry (ACR):

- name: Image scan
  run: trivy image --severity HIGH,CRITICAL --exit-code 1 registry.cn-hangzhou.aliyuncs.com/app/api:${{ github.sha }}

Stage 4 — Secrets management, moved entirely off CI environment variables and hardcoded files into Alibaba Cloud KMS Secrets Manager, referenced at runtime rather than baked into any artifact:

resource "alicloud_kms_secret" "db_password" {
  secret_name       = "prod-db-password"
  secret_data       = var.db_password
  version_id        = "v1" # required — bump this string on each rotation
  encryption_key_id = alicloud_kms_key.main.id
}

env:
  - name: DB_PASSWORD
    valueFrom:
      secretKeyRef:
        name: db-credentials
        key: password

Stage 5 — Deployment policy gate, an admission controller (Kyverno) enforcing that only images passing the scan stage (verified by a signed attestation) can be scheduled onto the cluster:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-image-signature
spec:
  validationFailureAction: Enforce
  rules:
    - name: check-signature
      match:
        resources:
          kinds: [Pod]
      verifyImages:
        - imageReferences: ["registry.cn-hangzhou.aliyuncs.com/app/*"]
          attestors:
            - entries:
                - keys:
                    publicKeys: |-
                      -----BEGIN PUBLIC KEY-----...

---

Failure / Challenge

The first version of the admission policy blocked every deployment, including ones from the scanned, signed pipeline — the signature verification was checking against the wrong public key (a leftover test key from the Kyverno documentation example, never swapped for the pipeline's actual signing key). Every deployment failed with an opaque ImagePolicyWebhook denial and no obvious root cause in the pod events.

---

Solution

kubectl logs -n kyverno -l app.kubernetes.io/name=kyverno --tail=50

Kyverno's own controller logs (not the pod events) surfaced the actual signature-mismatch reason. Fixed by generating a real cosign keypair for the pipeline, storing the private key in KMS Secrets Manager for the signing step, and updating the ClusterPolicy with the matching public key. Added a staging-cluster dry-run stage for any admission-policy change going forward — policy changes are exactly the kind of infrastructure change that shouldn't get tested for the first time against production.

---

Cost / Performance

Stage	Added Pipeline Time	Value
SAST + dependency scan	+45 sec	Catches vulnerable code/deps pre-merge
Image vulnerability scan	+90 sec	Catches vulnerable base images pre-push
Signature verification	+5 sec (admission)	Blocks unsigned/tampered images at deploy time
Total pipeline overhead	~2.5 min	Eliminates the class of incident this replaces

Two and a half minutes of pipeline time is a rounding error against the cost of a shipped critical CVE or a compromised base image reaching production.

---

Lessons Learned

• An admission controller that silently blocks everything is worse than no admission controller — always ship policy changes through a dry-run/staging path first, exactly like any other infrastructure change.
• Controller logs, not pod events, are usually where the real denial reason lives for anything enforced by a webhook — check the source, not the symptom.
• Security folded into the pipeline (fails fast, cheap, pre-merge) beats security bolted on before release (fails slow, expensive, post-build) every time.

GitHub Repository: devsecops-pipeline-alibaba-cloud-lab — the full pipeline: dependency scan, SAST, image scan, cosign signing, and the Kyverno admission gate, ready to run.

DevSecOps · Alibaba Cloud · Kyverno · Container Security · Secrets Management · CI/CD

---

Originally published on my portfolio.