<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: DevOpsInterview.Cloud</title>
    <description>The latest articles on DEV Community by DevOpsInterview.Cloud (@devopsinterview_cloud).</description>
    <link>https://dev.to/devopsinterview_cloud</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4204459%2F45a1238f-58ad-480e-a75b-377d37c95fdf.png</url>
      <title>DEV Community: DevOpsInterview.Cloud</title>
      <link>https://dev.to/devopsinterview_cloud</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/devopsinterview_cloud"/>
    <language>en</language>
    <item>
      <title>Argo CD Interview Questions and Answers</title>
      <dc:creator>DevOpsInterview.Cloud</dc:creator>
      <pubDate>Wed, 18 Nov 2026 09:20:03 +0000</pubDate>
      <link>https://dev.to/devopsinterview_cloud/argo-cd-interview-questions-and-answers-4ji0</link>
      <guid>https://dev.to/devopsinterview_cloud/argo-cd-interview-questions-and-answers-4ji0</guid>
      <description>&lt;p&gt;Argo CD interview questions usually start with a definition and end in an incident. "What does Argo CD do?" turns into "why does this app say OutOfSync when the pods are fine?", then "self-heal just reverted a fix someone made by hand during an outage, now what?". The questions below follow that path through sync, self-heal and prune, drift, sync waves and hooks, app-of-apps and ApplicationSets, multi-cluster setups and secrets. Defaults and commands are from the Argo CD 3.x docs. For where GitOps fits in a wider prep plan, see the &lt;a href="https://devopsinterview.cloud/blog/devops-interview-preparation-roadmap" rel="noopener noreferrer"&gt;DevOps interview preparation roadmap&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Argo CD interview questions on the basics
&lt;/h2&gt;

&lt;h3&gt;
  
  
  1. How does Argo CD decide what to deploy, and how often does it check?
&lt;/h3&gt;

&lt;p&gt;Every app is an &lt;code&gt;Application&lt;/code&gt; resource: a source (repo, revision and path), a destination (cluster and namespace) and the project that limits both. The repo server renders the source into plain manifests, whether it holds Helm, Kustomize or plain YAML, and the application controller compares the result with what is running.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;argoproj.io/v1alpha1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Application&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;payments&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;argocd&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;project&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;payments&lt;/span&gt;
  &lt;span class="na"&gt;source&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;repoURL&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;https://github.com/acme/deploy.git&lt;/span&gt;
    &lt;span class="na"&gt;targetRevision&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;main&lt;/span&gt;
    &lt;span class="na"&gt;path&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;apps/payments/prod&lt;/span&gt;
  &lt;span class="na"&gt;destination&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;server&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;https://kubernetes.default.svc&lt;/span&gt;
    &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;payments&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;By default the controller checks every 120 seconds plus up to 60 seconds of jitter, so a merged change can wait up to three minutes. A Git webhook removes that delay. If the API server is public, the docs recommend a webhook secret to stop floods of fake events.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why does an Argo CD app show OutOfSync?
&lt;/h2&gt;

&lt;h3&gt;
  
  
  2. The app shows OutOfSync but the pods look healthy. What does that tell you?
&lt;/h3&gt;

&lt;p&gt;Sync status and health status are separate checks. Sync status asks whether the live objects match what Git renders. Health asks whether they work, using built-in checks for common kinds (a Deployment counts as healthy once its rollout has finished) and Lua checks for custom resources. Healthy and OutOfSync together is common: everything runs, and one field on one resource differs from Git.&lt;/p&gt;

&lt;p&gt;Find the field before deciding anything:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;argocd app diff payments                 &lt;span class="c"&gt;# exit code 1 means a diff was found&lt;/span&gt;
argocd app diff payments --hard-refresh  &lt;span class="c"&gt;# re-render, skipping the manifest cache&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The usual causes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Someone changed the live object&lt;/strong&gt; with &lt;code&gt;kubectl edit&lt;/code&gt; or &lt;code&gt;kubectl scale&lt;/code&gt;. That is real drift: put the change in Git or let a sync undo it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A controller or mutating webhook rewrites the object&lt;/strong&gt; after it is applied. The diffing docs name the HPA controller, which reorders &lt;code&gt;spec.metrics&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Git and an autoscaler both set replicas.&lt;/strong&gt; If the manifest says &lt;code&gt;replicas: 3&lt;/code&gt; and the HPA has scaled to eight, the app is OutOfSync. The Kubernetes docs recommend removing &lt;code&gt;spec.replicas&lt;/code&gt; from the manifest once an HPA manages it, and on a live workload they give migration steps, because simply dropping a previously applied &lt;code&gt;spec.replicas&lt;/code&gt; can scale it down to one replica.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The render changes every time.&lt;/strong&gt; Helm functions such as &lt;code&gt;randAlphaNum&lt;/code&gt; produce new output on each render, so Git never matches.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A resource was deleted from Git and prune is off.&lt;/strong&gt; The app stays OutOfSync until something deletes the object.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Status fields aren't on the list. Since 3.0, Argo CD ignores &lt;code&gt;status&lt;/code&gt; differences on every resource by default; in 2.x the default covered only CustomResourceDefinition objects.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. How do you make Argo CD accept a difference it should ignore?
&lt;/h3&gt;

&lt;p&gt;First decide whether the difference is expected. If it's drift, fix Git or sync. If another controller owns the field and you can't drop it from the manifest, say a third-party chart that always sets replicas, tell Argo CD to ignore it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;ignoreDifferences&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;group&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;apps&lt;/span&gt;
      &lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Deployment&lt;/span&gt;
      &lt;span class="na"&gt;jsonPointers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s"&gt;/spec/replicas&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;group&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;admissionregistration.k8s.io&lt;/span&gt;
      &lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;MutatingWebhookConfiguration&lt;/span&gt;
      &lt;span class="na"&gt;jqPathExpressions&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s"&gt;.webhooks[]?.clientConfig.caBundle&lt;/span&gt;
  &lt;span class="na"&gt;syncPolicy&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;syncOptions&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s"&gt;RespectIgnoreDifferences=true&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;ignoreDifferences&lt;/code&gt; on its own only changes the comparison, and a sync still writes the Git value over what the controller set. &lt;code&gt;RespectIgnoreDifferences=true&lt;/code&gt; carries the rule into the sync, for resources that already exist. A third option, &lt;code&gt;managedFieldsManagers&lt;/code&gt;, ignores every field owned by a named manager such as &lt;code&gt;kube-controller-manager&lt;/code&gt;. Rules for every app go in &lt;code&gt;resource.customizations&lt;/code&gt; in the &lt;code&gt;argocd-cm&lt;/code&gt; ConfigMap.&lt;/p&gt;

&lt;p&gt;The other lever is the diff strategy. The default is a three-way diff of live state, desired state and the &lt;code&gt;last-applied-configuration&lt;/code&gt; annotation. Server-side diff, stable since 3.1, runs a server-side apply in dry-run mode and compares against the predicted result, so admission controllers take part in the calculation. Changes made by mutating webhooks are left out unless you add &lt;code&gt;IncludeMutationWebhook=true&lt;/code&gt;. Enable it per app with the &lt;code&gt;argocd.argoproj.io/compare-options: ServerSideDiff=true&lt;/code&gt; annotation, or globally with &lt;code&gt;controller.diff.server.side: "true"&lt;/code&gt; in &lt;code&gt;argocd-cmd-params-cm&lt;/code&gt; and a restart of the application controller.&lt;/p&gt;

&lt;h2&gt;
  
  
  Sync, self-heal and prune
&lt;/h2&gt;

&lt;h3&gt;
  
  
  4. What do automated sync, self-heal and prune each do?
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Setting&lt;/th&gt;
&lt;th&gt;What it does&lt;/th&gt;
&lt;th&gt;Default&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;automated&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Syncs when the app is OutOfSync because Git or the app's parameters changed&lt;/td&gt;
&lt;td&gt;Off (manual sync)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;selfHeal&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Also re-syncs when live state drifts with no Git change, after a 5-second timeout&lt;/td&gt;
&lt;td&gt;Off&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;prune&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Deletes resources that are no longer in Git&lt;/td&gt;
&lt;td&gt;Off&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;allowEmpty&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Lets an automated sync with prune remove everything when the source renders empty&lt;/td&gt;
&lt;td&gt;Off&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;syncPolicy&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;automated&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;prune&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
    &lt;span class="na"&gt;selfHeal&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
  &lt;span class="na"&gt;retry&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;limit&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;
    &lt;span class="na"&gt;backoff&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;duration&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;10s&lt;/span&gt;
      &lt;span class="na"&gt;factor&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;
      &lt;span class="na"&gt;maxDuration&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;3m&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The usual follow-up is a failed sync. Automated sync makes one attempt per commit SHA and parameter set, and if that attempt fails it doesn't try the same commit again. The &lt;code&gt;retry&lt;/code&gt; block is what retries a failed sync, with exponential backoff.&lt;/p&gt;

&lt;h3&gt;
  
  
  5. Self-heal reverted a fix someone applied with kubectl during an incident. What should happen?
&lt;/h3&gt;

&lt;p&gt;Self-heal did its job: Git said one thing, the cluster another, and Argo CD put Git back. What's missing is a documented emergency procedure.&lt;/p&gt;

&lt;p&gt;The durable path goes through Git: commit the fix or revert the bad change, and let the sync apply it. The &lt;a href="https://devopsinterview.cloud/questions/gitops-rollback-how-to-revert-a-bad-deploy" rel="noopener noreferrer"&gt;GitOps rollback question page&lt;/a&gt; covers revert versus roll forward. If the live change can't wait for a merge, switch that one app to manual first, make the change, land it in Git, then switch automation back on:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;argocd app &lt;span class="nb"&gt;set&lt;/span&gt; payments --sync-policy manual
&lt;span class="c"&gt;# make the emergency change, then merge it to Git&lt;/span&gt;
argocd app &lt;span class="nb"&gt;set&lt;/span&gt; payments --sync-policy automated --self-heal --auto-prune
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That works for a standalone Application. If an ApplicationSet generated the app, its controller patches the Application back to the template, which can switch automation on again mid-incident. Either change the template, or add &lt;code&gt;ignoreApplicationDifferences&lt;/code&gt; for &lt;code&gt;/spec/syncPolicy&lt;/code&gt; to the ApplicationSet ahead of time so a temporary toggle sticks. Confirm automation is off before touching the cluster.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;argocd app rollback&lt;/code&gt; refuses to run while automated sync is on, and afterwards the app shows OutOfSync until Git catches up. Put the procedure in the incident runbook so nobody learns about self-heal during an outage.&lt;/p&gt;

&lt;p&gt;Self-heal only reverts objects the app manages; a Deployment created by hand is invisible to it. Orphaned resource monitoring (&lt;code&gt;orphanedResources: {warn: true}&lt;/code&gt; in the AppProject) flags namespaced objects no app owns. The post on &lt;a href="https://devopsinterview.cloud/blog/kubernetes-rbac-least-privilege-lockdown" rel="noopener noreferrer"&gt;locking down Kubernetes RBAC&lt;/a&gt; makes the same point about bindings created out of band.&lt;/p&gt;

&lt;h3&gt;
  
  
  6. How do you run prune safely in production?
&lt;/h3&gt;

&lt;p&gt;Prune is what makes Git the full record: delete a manifest and the object goes. The risk is a bad commit or broken overlay that deletes things you needed. The guardrails:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Leave &lt;code&gt;allowEmpty&lt;/code&gt; off, so a source that renders to nothing can't empty the app.&lt;/li&gt;
&lt;li&gt;Annotate objects that must not be pruned automatically with &lt;code&gt;argocd.argoproj.io/sync-options: Prune=false&lt;/code&gt;, or &lt;code&gt;Prune=confirm&lt;/code&gt; to require approval. Candidates are PVCs holding data, shared namespaces, and CRDs, since deleting a CRD deletes every custom resource of that type.&lt;/li&gt;
&lt;li&gt;Pruning and app deletion are separate. &lt;code&gt;Prune=false&lt;/code&gt; doesn't stop a cascading delete of the whole Application. &lt;code&gt;Delete=false&lt;/code&gt; keeps a resource, such as a PVC, when its app is deleted; &lt;code&gt;Delete=confirm&lt;/code&gt; deletes it only after approval.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;PruneLast=true&lt;/code&gt; prunes only after every other wave has synced and is healthy, so the replacement is up first.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;FailOnSharedResource=true&lt;/code&gt; fails a sync if another Application already applied the same resource.&lt;/li&gt;
&lt;li&gt;Make deletions visible in review with a CI job that diffs the rendered manifests.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  How do you order a sync with sync waves and hooks?
&lt;/h2&gt;

&lt;h3&gt;
  
  
  7. A database migration has to finish before the new pods roll out. How do you set that up?
&lt;/h3&gt;

&lt;p&gt;A sync runs in phases: PreSync, Sync, then PostSync once the synced resources are healthy. SyncFail runs if the sync fails, and PostDelete (2.10 and later) after an app's resources are deleted. Within a phase, Argo CD orders resources by wave (lower first, default 0, negatives allowed), then kind, then name. It applies one wave, waits until it is synced and healthy, then moves on.&lt;/p&gt;

&lt;p&gt;A migration fits as a PreSync hook:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;batch/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Job&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;payments-migrate&lt;/span&gt;
  &lt;span class="na"&gt;annotations&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;argocd.argoproj.io/hook&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;PreSync&lt;/span&gt;
    &lt;span class="na"&gt;argocd.argoproj.io/hook-delete-policy&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;BeforeHookCreation&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;backoffLimit&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
  &lt;span class="na"&gt;template&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;restartPolicy&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Never&lt;/span&gt;
      &lt;span class="na"&gt;containers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;migrate&lt;/span&gt;
          &lt;span class="na"&gt;image&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ghcr.io/acme/payments-migrate:1.42.0&lt;/span&gt;
          &lt;span class="na"&gt;args&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"migrate"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"up"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If a PreSync hook fails, the sync stops and is marked failed, so the new pods never roll out. The points that come up next:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Hooks run on every sync&lt;/strong&gt;, not only when the migration changed, so it must be safe to run twice. They don't run during a selective sync.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Old pods keep serving&lt;/strong&gt; during the migration and the rollout, so the schema has to work with both versions. That is the expand and contract pattern.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;BeforeHookCreation&lt;/strong&gt; is what Argo CD assumes when no policy is set. It deletes the previous Job just before creating the new one, so the last run's logs stay around until the next sync.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  App-of-apps or ApplicationSets?
&lt;/h2&gt;

&lt;h3&gt;
  
  
  8. When would you use app-of-apps, and what catches people out?
&lt;/h3&gt;

&lt;p&gt;App-of-apps is a parent Application whose source is a directory or chart of Application manifests; syncing the parent creates the children. It suits bootstrapping a cluster's fixed platform set (ingress, cert-manager, monitoring), where every child is a reviewed file. Three catches:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Child health isn't reported by default.&lt;/strong&gt; Health assessment for Application resources was removed in 1.8, so a parent won't wait for wave -1 children to become healthy before creating wave 0. Restore it with a &lt;code&gt;resource.customizations.health.argoproj.io_Application&lt;/code&gt; Lua check in &lt;code&gt;argocd-cm&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Deletion needs a finalizer.&lt;/strong&gt; Add &lt;code&gt;resources-finalizer.argocd.argoproj.io&lt;/code&gt; to each child so deleting it removes its resources, not just the Application object.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The parent's repo is an admin boundary.&lt;/strong&gt; Creating Applications in any project is an admin-level capability, so only admins should push to it.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  9. How do ApplicationSets work, and how do you keep them safe?
&lt;/h3&gt;

&lt;p&gt;An ApplicationSet is a template plus generators. Each generator produces parameter sets, and the controller renders one Application per set. Generators include List, Cluster, Git, Matrix and Merge, SCM Provider and Pull Request. This one deploys payments to every cluster labelled &lt;code&gt;env: prod&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;argoproj.io/v1alpha1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ApplicationSet&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;payments&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;argocd&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;goTemplate&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
  &lt;span class="na"&gt;goTemplateOptions&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"missingkey=error"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
  &lt;span class="na"&gt;generators&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;clusters&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;selector&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;matchLabels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="na"&gt;env&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;prod&lt;/span&gt;
  &lt;span class="na"&gt;template&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'payments-{{.name}}'&lt;/span&gt;
    &lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;project&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;payments&lt;/span&gt;
      &lt;span class="na"&gt;source&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;repoURL&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;https://github.com/acme/deploy.git&lt;/span&gt;
        &lt;span class="na"&gt;targetRevision&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;main&lt;/span&gt;
        &lt;span class="na"&gt;path&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;apps/payments/prod&lt;/span&gt;
      &lt;span class="na"&gt;destination&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;server&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'{{.server}}'&lt;/span&gt;
        &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;payments&lt;/span&gt;
      &lt;span class="na"&gt;syncPolicy&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;automated&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;prune&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
          &lt;span class="na"&gt;selfHeal&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Register a cluster with that label and its Application appears. The &lt;a href="https://devopsinterview.cloud/questions/argocd-application-sets-multi-cluster-deployment" rel="noopener noreferrer"&gt;ApplicationSets question page&lt;/a&gt; covers matrix generators for per-environment config. The controls interviewers probe:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;What the controller may change.&lt;/strong&gt; &lt;code&gt;applicationsSync&lt;/code&gt; takes &lt;code&gt;create-only&lt;/code&gt;, &lt;code&gt;create-update&lt;/code&gt;, &lt;code&gt;create-delete&lt;/code&gt; or the default &lt;code&gt;sync&lt;/code&gt;. A controller-level &lt;code&gt;--policy&lt;/code&gt; flag wins unless policy override is enabled.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;What happens on removal.&lt;/strong&gt; When a cluster drops out of the generator, its Application is deleted and by default its workloads go too. &lt;code&gt;preserveResourcesOnDeletion: true&lt;/code&gt; keeps them running.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Gradual rollout.&lt;/strong&gt; Progressive syncs, beta since 3.3 and off until enabled, use RollingSync to update groups of generated apps by label, with &lt;code&gt;maxUpdate&lt;/code&gt; per group. RollingSync forces autosync off on those apps.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Who can create them.&lt;/strong&gt; Only admins, because an ApplicationSet can create Applications in any project. If &lt;code&gt;project&lt;/code&gt; is templated from a generator, anyone who can write that generator's source can pick a looser project.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  How do you run Argo CD across many clusters?
&lt;/h2&gt;

&lt;h3&gt;
  
  
  10. Would you run one Argo CD for every cluster, or one per cluster?
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;One central Argo CD (hub)&lt;/th&gt;
&lt;th&gt;Argo CD in each cluster&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;UI and RBAC&lt;/td&gt;
&lt;td&gt;One UI, one set of projects&lt;/td&gt;
&lt;td&gt;One per cluster&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Credentials&lt;/td&gt;
&lt;td&gt;Hub holds credentials for every cluster&lt;/td&gt;
&lt;td&gt;Each instance touches only its own cluster&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Network&lt;/td&gt;
&lt;td&gt;Hub must reach every API server&lt;/td&gt;
&lt;td&gt;Each cluster pulls from Git itself&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;If Argo CD is compromised&lt;/td&gt;
&lt;td&gt;Attacker can reach every registered cluster&lt;/td&gt;
&lt;td&gt;Attacker reaches one cluster&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Operations&lt;/td&gt;
&lt;td&gt;One install; shard the controller as it grows&lt;/td&gt;
&lt;td&gt;Many installs to upgrade and monitor&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The compromise row is about Argo CD itself. A bad commit reaches every instance tracking its path, wherever Argo CD runs.&lt;/p&gt;

&lt;p&gt;A common middle ground is one instance per environment or region, so production credentials never sit with dev. On a hub, clusters are Secrets labelled &lt;code&gt;argocd.argoproj.io/secret-type: cluster&lt;/code&gt;; for EKS, use &lt;code&gt;awsAuthConfig&lt;/code&gt; with IRSA or EKS Pod Identity rather than long-lived tokens. When one controller can't keep up, scale the application controller StatefulSet and set &lt;code&gt;ARGOCD_CONTROLLER_REPLICAS&lt;/code&gt; to match, and clusters are spread across the replicas.&lt;/p&gt;

&lt;h2&gt;
  
  
  How do you handle secrets and access in Argo CD?
&lt;/h2&gt;

&lt;h3&gt;
  
  
  11. How do you get secrets into the cluster without committing plaintext to Git?
&lt;/h3&gt;

&lt;p&gt;The Argo CD docs strongly recommend letting the destination cluster populate secrets rather than having Argo CD inject them while rendering manifests. The options:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Sealed Secrets.&lt;/strong&gt; Commit an encrypted SealedSecret that only the controller in the cluster can decrypt.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;External Secrets Operator or Vault Secrets Operator.&lt;/strong&gt; Commit a reference; the operator reads the value from the store and writes a Kubernetes Secret. The &lt;a href="https://devopsinterview.cloud/questions/external-secrets-operator-sync-vault-to-kubernetes" rel="noopener noreferrer"&gt;External Secrets Operator question page&lt;/a&gt; walks through the Vault setup.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Secrets Store CSI Driver.&lt;/strong&gt; Mount values from the store into pods as a volume.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Plugins such as argocd-vault-plugin inject values during manifest generation, and the docs caution against it: Argo CD needs access to the secrets, generated manifests sit in plaintext in its Redis cache, and rotating a secret needs a sync.&lt;/p&gt;

&lt;h3&gt;
  
  
  12. If Git is the deploy authority, how do you lock Git and Argo CD down?
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Protect the tracked branches&lt;/strong&gt; with required reviews and CODEOWNERS on production paths. Anyone who can merge there can deploy.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Keep real workloads out of the &lt;code&gt;default&lt;/code&gt; project&lt;/strong&gt;, which permits any repo, any cluster and every kind. Give each team an AppProject that limits &lt;code&gt;sourceRepos&lt;/code&gt;, &lt;code&gt;destinations&lt;/code&gt; and cluster-scoped kinds (&lt;code&gt;clusterResourceWhitelist&lt;/code&gt;), mirroring the per-team layout in &lt;a href="https://devopsinterview.cloud/blog/kubernetes-multi-tenancy-shared-cluster-design" rel="noopener noreferrer"&gt;designing a shared Kubernetes cluster&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Don't let workload projects target Argo CD's own namespace.&lt;/strong&gt; A project whose &lt;code&gt;destinations&lt;/code&gt; include it effectively grants its Applications admin rights.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Recheck RBAC after a 3.0 upgrade.&lt;/strong&gt; &lt;code&gt;update&lt;/code&gt; and &lt;code&gt;delete&lt;/code&gt; on an Application no longer extend to its managed resources, and logs need an explicit &lt;code&gt;logs, get&lt;/code&gt; permission.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Interview questions this covers
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;An Argo CD app shows OutOfSync but the pods are healthy. What does that mean, and how do you fix it?&lt;/li&gt;
&lt;li&gt;What do self-heal and prune do, and how do you run them safely in production?&lt;/li&gt;
&lt;li&gt;How do you make a database migration run before a rollout with sync waves and hooks?&lt;/li&gt;
&lt;li&gt;When do you use app-of-apps, and when ApplicationSets?&lt;/li&gt;
&lt;li&gt;How do you handle secrets with Argo CD without committing plaintext to Git?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For more practice, browse &lt;a href="https://devopsinterview.cloud/ebooks" rel="noopener noreferrer"&gt;all the DevOps interview ebooks&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Senior GitOps rounds go further: structuring GitOps for dozens of clusters and teams, using it to satisfy audit requirements in a regulated environment, and running canaries that promote or abort themselves on metrics. &lt;a href="https://devopsinterview.cloud/ebooks/modern-cicd-gitops" rel="noopener noreferrer"&gt;CI/CD &amp;amp; GitOps: Pipelines, GitOps &amp;amp; Progressive Delivery Interview Questions&lt;/a&gt; works through 50 senior-level questions with worked answers, including a chapter on GitOps and continuous deployment with Argo CD and Flux. It is one of the five books in the &lt;a href="https://devopsinterview.cloud/ebooks/complete-devops-mastery-bundle" rel="noopener noreferrer"&gt;DevOps Interview Mastery bundle&lt;/a&gt;, and every purchase includes the free Interview-Day Playbook with a 7-day and 24-hour prep plan and a 30-second answer framework.&lt;/p&gt;

&lt;p&gt;Prefer to watch or listen? Answers like these are walked through on &lt;a href="https://www.youtube.com/@DevOpsInterviewCloud" rel="noopener noreferrer"&gt;YouTube&lt;/a&gt; and on the podcast, on &lt;a href="https://open.spotify.com/show/033txr35oY2Rcn6H4CVZ4S" rel="noopener noreferrer"&gt;Spotify&lt;/a&gt; and &lt;a href="https://music.amazon.com/podcasts/49554d87-91cb-4816-9420-cdac8a06ac75/devops-cloud-interview-prep-real-scenarios-answers" rel="noopener noreferrer"&gt;Amazon Music&lt;/a&gt;.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://devopsinterview.cloud/blog/argo-cd-interview-questions" rel="noopener noreferrer"&gt;devopsinterview.cloud&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>cicd</category>
      <category>devops</category>
      <category>interview</category>
      <category>kubernetes</category>
    </item>
    <item>
      <title>AI Infrastructure Engineer Interview Questions and Answers</title>
      <dc:creator>DevOpsInterview.Cloud</dc:creator>
      <pubDate>Sat, 14 Nov 2026 09:20:02 +0000</pubDate>
      <link>https://dev.to/devopsinterview_cloud/ai-infrastructure-engineer-interview-questions-and-answers-1pd7</link>
      <guid>https://dev.to/devopsinterview_cloud/ai-infrastructure-engineer-interview-questions-and-answers-1pd7</guid>
      <description>&lt;p&gt;This guide covers AI infrastructure engineer interview questions from the platform system design round. The prompt sounds simple, something like "design a platform that serves our models to the whole company", and the interviewer spends the rest of the round on what makes it different from serving a web app: GPUs are scarce, expensive and slow to start, and two requests to the same endpoint can differ in cost many times over. The questions follow the order a strong answer takes, from requirements to rollback. Shorter answers to six related questions are on the &lt;a href="https://devopsinterview.cloud/ai-infrastructure-interview-questions" rel="noopener noreferrer"&gt;AI infrastructure interview questions page&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  AI infrastructure engineer interview questions on requirements and sizing
&lt;/h2&gt;

&lt;h3&gt;
  
  
  1. What do you ask before drawing anything?
&lt;/h3&gt;

&lt;p&gt;Requirements, because each one changes the design.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Ask&lt;/th&gt;
&lt;th&gt;Why it matters&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Which models, and how big?&lt;/td&gt;
&lt;td&gt;Sets GPU memory per replica and whether one card is enough&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Interactive or batch?&lt;/td&gt;
&lt;td&gt;Interactive needs a latency target and warm capacity; batch can queue on cheaper capacity&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;What latency target?&lt;/td&gt;
&lt;td&gt;For LLMs, time to first token and time per output token are separate targets&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;What does traffic look like over a day?&lt;/td&gt;
&lt;td&gt;Steady load suits committed capacity; spiky load needs headroom&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;How long are prompts and outputs?&lt;/td&gt;
&lt;td&gt;They drive KV-cache memory, and so how many requests fit on one GPU&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;How many tenants?&lt;/td&gt;
&lt;td&gt;Per-team quotas, rate limits and cost reporting&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Where can the data go?&lt;/td&gt;
&lt;td&gt;Whether prompts may leave a region or go to a hosted API at all&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;If a hosted model API meets the data rules and the budget, self-hosting has to justify itself.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. How do you estimate how many GPUs a model needs?
&lt;/h3&gt;

&lt;p&gt;Start with memory. Weights take parameters times bytes per parameter: an 8B model at 16-bit precision is about 16 GB, and a 70B model is about 140 GB, more than one 80 GB H100 holds. That model gets split across several GPUs with tensor parallelism, usually inside one node where the GPUs talk over NVLink. Quantized to 8 bits it is about 70 GB and fits on one card, with little room left.&lt;/p&gt;

&lt;p&gt;What needs the room is mostly the KV cache, which grows with every token in flight. Per token it is 2 × layers × KV heads × head dimension × bytes per value. For Llama 3 8B (32 layers, 8 KV heads, head dimension 128) at 16 bits:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;2 × 32 × 8 × 128 × 2 bytes = 131,072 bytes = 128 KiB per token
one 8,192-token sequence    = 1 GiB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On a 24 GB L4 the weights already take about 16 GB, so only a few full-length sequences fit beside them. On an 80 GB card the same model has room for dozens.&lt;/p&gt;

&lt;p&gt;The arithmetic picks the card. The capacity number comes from a load test with your real prompt and output lengths: the highest request rate one replica sustains inside the latency target. vLLM ships &lt;code&gt;vllm bench serve&lt;/code&gt; for this, and the autoscaling threshold and cost per request both depend on the result.&lt;/p&gt;

&lt;h2&gt;
  
  
  How would you design the GPU cluster?
&lt;/h2&gt;

&lt;h3&gt;
  
  
  3. Should training and inference share a cluster?
&lt;/h3&gt;

&lt;p&gt;They can, and the simplest version keeps them on separate node pools so a training job can't hurt serving latency. Serving gets its own GPU node pool on committed capacity, sized for the warm floor, and a PriorityClass above training so that serving wins if the two ever compete. Training and batch inference run on a separate pool and queue through Kueue, which holds quotas per team, admits jobs when capacity exists and preempts lower-priority work. The &lt;a href="https://devopsinterview.cloud/blog/kubernetes-gpu-sharing-mig-time-slicing-mps" rel="noopener noreferrer"&gt;Kubernetes GPU sharing post&lt;/a&gt; covers taints, node pools and splitting a card between small models.&lt;/p&gt;

&lt;p&gt;On-demand GPU instances can be unavailable when you need them, so the serving floor belongs on reserved capacity, such as On-Demand Capacity Reservations on AWS. Capacity Blocks end on a fixed date, so they suit a training run better.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. What does a multi-node training job need that a web service doesn't?
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;All pods or none.&lt;/strong&gt; A job with half its workers running holds GPUs and does no work. Kueue admits a job as a whole, and with &lt;code&gt;waitForPodsReady&lt;/code&gt; it evicts and requeues one whose pods don't all start in time.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Placement.&lt;/strong&gt; Workers that exchange gradients every step should sit close together. Kueue's topology-aware scheduling keeps each PodSet inside one rack or block with the &lt;code&gt;kueue.x-k8s.io/podset-required-topology&lt;/code&gt; annotation, and a shared &lt;code&gt;kueue.x-k8s.io/podset-group-name&lt;/code&gt; keeps a leader and its workers in the same domain.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A fast interconnect.&lt;/strong&gt; NVLink inside a node and RDMA between nodes (InfiniBand, RoCE, or EFA on AWS), which NCCL uses for collective operations. On the ordinary pod network, gradient exchange becomes the bottleneck as the job grows.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Checkpoints&lt;/strong&gt; written often to fast shared storage, so a failed GPU or a reclaimed spot node costs the work since the last checkpoint, not the whole run.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  How do you design the model serving path?
&lt;/h2&gt;

&lt;h3&gt;
  
  
  5. What sits between the user and the GPU?
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Layer&lt;/th&gt;
&lt;th&gt;Job&lt;/th&gt;
&lt;th&gt;Common choices&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;API gateway&lt;/td&gt;
&lt;td&gt;Authentication, per-tenant rate limits and token quotas, usage metering&lt;/td&gt;
&lt;td&gt;An Envoy-based gateway, Kong, a cloud API gateway&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Inference-aware routing&lt;/td&gt;
&lt;td&gt;Picks a replica based on its load and cache state&lt;/td&gt;
&lt;td&gt;Gateway API Inference Extension&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Model server&lt;/td&gt;
&lt;td&gt;Batching, KV-cache management, streaming tokens&lt;/td&gt;
&lt;td&gt;vLLM, SGLang, Triton&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Weights&lt;/td&gt;
&lt;td&gt;Versioned, immutable model files&lt;/td&gt;
&lt;td&gt;Object storage or a model registry, cached on the node&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Async path&lt;/td&gt;
&lt;td&gt;Bulk jobs that can wait&lt;/td&gt;
&lt;td&gt;A queue and workers on the batch pool, off the online replicas&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h3&gt;
  
  
  6. Why does round-robin load balancing work badly for LLM traffic?
&lt;/h3&gt;

&lt;p&gt;Because it assumes requests cost about the same. One request carries a short prompt and finishes in a second. The next carries a long document and streams for a minute, holding KV-cache memory the whole time. Round-robin ignores how long a request runs. Least-connections at least counts open streams, but neither sees token counts or KV-cache pressure, so a replica can be full while its connection count looks normal.&lt;/p&gt;

&lt;p&gt;The Gateway API Inference Extension adds an &lt;a href="https://gateway-api-inference-extension.sigs.k8s.io/api-types/inferencepool/" rel="noopener noreferrer"&gt;InferencePool&lt;/a&gt; backend for HTTPRoutes, GA since v1.0.0. Its endpoint picker tracks each model server's KV-cache utilization, pending-request queue length and active LoRA adapters, and sends each request to the replica best placed to serve it.&lt;/p&gt;

&lt;h2&gt;
  
  
  How do you autoscale inference on queue depth?
&lt;/h2&gt;

&lt;h3&gt;
  
  
  7. Which metric should drive inference autoscaling?
&lt;/h3&gt;

&lt;p&gt;Not CPU, and not GPU utilization either. &lt;code&gt;DCGM_FI_DEV_GPU_UTIL&lt;/code&gt; is the share of time any kernel was running, not how much work the GPU did, so it can read high at low concurrency and says little about how many more requests a replica can take.&lt;/p&gt;

&lt;p&gt;Queue depth measures what users feel. vLLM admits requests into its running batch until it hits its sequence limit or runs out of KV-cache space, and after that they wait. &lt;code&gt;vllm:num_requests_waiting&lt;/code&gt; counts them, and every waiting request adds to time to first token. &lt;code&gt;vllm:kv_cache_usage_perc&lt;/code&gt; (a 0 to 1 fraction, despite the name) is the early warning: when it nears full, the queue is about to grow.&lt;/p&gt;

&lt;p&gt;With KEDA and a Prometheus trigger, the ScaledObject looks like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;keda.sh/v1alpha1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ScaledObject&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;chat-llm&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;inference&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;scaleTargetRef&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;chat-llm&lt;/span&gt;            &lt;span class="c1"&gt;# the vLLM Deployment
&lt;/span&gt;  &lt;span class="na"&gt;minReplicaCount&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;          &lt;span class="c1"&gt;# warm floor, see cold starts below
&lt;/span&gt;  &lt;span class="na"&gt;maxReplicaCount&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;12&lt;/span&gt;
  &lt;span class="na"&gt;advanced&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;horizontalPodAutoscalerConfig&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;behavior&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;scaleUp&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;policies&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Pods&lt;/span&gt;
              &lt;span class="na"&gt;value&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;
              &lt;span class="na"&gt;periodSeconds&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;120&lt;/span&gt;
        &lt;span class="na"&gt;scaleDown&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;stabilizationWindowSeconds&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;600&lt;/span&gt;
  &lt;span class="na"&gt;triggers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;prometheus&lt;/span&gt;
      &lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;serverAddress&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;http://prometheus.monitoring.svc:9090&lt;/span&gt;
        &lt;span class="na"&gt;query&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;sum(vllm:num_requests_waiting{model_name="chat"})&lt;/span&gt;
        &lt;span class="na"&gt;threshold&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"4"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;KEDA's default metric type is &lt;code&gt;AverageValue&lt;/code&gt;, so the HPA aims for about four waiting requests per replica: 20 waiting across the pool asks for five replicas. KEDA's &lt;code&gt;cooldownPeriod&lt;/code&gt; &lt;a href="https://keda.sh/docs/2.18/reference/scaledobject-spec/" rel="noopener noreferrer"&gt;only applies when scaling to zero&lt;/a&gt;, so with a floor of two, scale-down pacing comes from the behavior block. The free &lt;a href="https://devopsinterview.cloud/questions/keda-event-driven-autoscaling-for-kubernetes" rel="noopener noreferrer"&gt;KEDA question&lt;/a&gt; covers how KEDA feeds the HPA.&lt;/p&gt;

&lt;h3&gt;
  
  
  8. What goes wrong when you scale on queue depth?
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;It drains to zero.&lt;/strong&gt; With enough replicas the queue empties and the metric says you need none. The ten-minute scale-down window slows the drop, but a queue-only signal can still oscillate once the window passes. Running plus waiting requests per replica, with a target from the load test, is steadier.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It overshoots.&lt;/strong&gt; A new replica takes minutes to load while the queue keeps growing, so the HPA keeps asking for more. Hence the scale-up cap above.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Readiness passes too early.&lt;/strong&gt; A pod that reports ready before its model is loaded takes traffic it can't serve.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The queue moves.&lt;/strong&gt; If the gateway queues requests itself, the model servers look idle while users wait upstream. Scale on wherever requests wait.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Scale-down cuts streams.&lt;/strong&gt; Give endpoint removal time to propagate with a short preStop delay, check that the server drains in-flight requests on SIGTERM, and set &lt;code&gt;terminationGracePeriodSeconds&lt;/code&gt; (30 by default) to cover the preStop delay plus your longest allowed generation.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  How do you handle cold starts?
&lt;/h2&gt;

&lt;h3&gt;
  
  
  9. Where does an inference cold start spend its time?
&lt;/h3&gt;

&lt;p&gt;Count every stage between "the autoscaler wants a replica" and "that replica serves its first token":&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Stage&lt;/th&gt;
&lt;th&gt;What takes the time&lt;/th&gt;
&lt;th&gt;Fixes&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;New GPU node&lt;/td&gt;
&lt;td&gt;VM boot, GPU driver, device plugin registering the GPUs&lt;/td&gt;
&lt;td&gt;A node image with the driver installed, or the GPU Operator's precompiled drivers; a spare warm node&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Image pull&lt;/td&gt;
&lt;td&gt;CUDA and PyTorch images run to several gigabytes&lt;/td&gt;
&lt;td&gt;Images baked into the node image, lazy-loading snapshotters such as SOCI or stargz&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Weights download&lt;/td&gt;
&lt;td&gt;About 16 GB for an 8B model, well over 100 GB for a 70B one&lt;/td&gt;
&lt;td&gt;A node-local NVMe cache (KServe's LocalModelCache), streaming loads with vLLM's &lt;code&gt;--load-format runai_streamer&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Load and warm-up&lt;/td&gt;
&lt;td&gt;Copying weights to GPU memory, torch.compile, CUDA graph capture&lt;/td&gt;
&lt;td&gt;Share vLLM's compile cache between replicas, or trade throughput for startup with &lt;code&gt;--enforce-eager&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;If the pod waits for a new GPU node, its startup time is the node's boot plus the rest of the table. The free &lt;a href="https://devopsinterview.cloud/questions/karpenter-vs-cluster-autoscaler-node-scaling-compared" rel="noopener noreferrer"&gt;Karpenter vs Cluster Autoscaler question&lt;/a&gt; covers that layer.&lt;/p&gt;

&lt;h3&gt;
  
  
  10. Would you scale an LLM service to zero?
&lt;/h3&gt;

&lt;p&gt;For a user-facing service, usually not. The first request after zero waits for the whole table above, which can take minutes. Keep a warm floor sized for baseline traffic and pay for it.&lt;/p&gt;

&lt;p&gt;For internal tools, rarely used fine-tuned models and batch endpoints, it can be worth it if something holds requests while a replica starts. KServe's serverless mode runs on Knative, whose activator buffers requests for a revision scaled to zero, and the KEDA HTTP add-on's interceptor does the same job. Without either, the caller has to retry or go through a queue.&lt;/p&gt;

&lt;h2&gt;
  
  
  How do you work out cost per request?
&lt;/h2&gt;

&lt;h3&gt;
  
  
  11. How do you calculate cost per request for a model?
&lt;/h3&gt;

&lt;p&gt;From what you actually spent, not from one replica's best throughput. GPU spend is the usual starting point, with CPU, storage and networking added for the full figure, and idle warm replicas and headroom belong in it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;GPU cost per 1,000 requests = GPU spend for the period / (requests served / 1,000)

4 replicas × $5 an hour (illustrative) × 24 hours = $480
360,000 requests in the day / 1,000                = 360
$480 / 360                                         = about $1.33 per 1,000 requests
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The load test gives the floor. A replica that sustains 5 requests a second inside the latency target serves 18,000 requests an hour, so at $5 an hour it costs about $0.28 per 1,000 when fully busy. The gap between the two numbers is idle capacity you're paying for.&lt;/p&gt;

&lt;p&gt;For LLMs, report cost per million tokens too, input and output separately: the prompt is processed in parallel during prefill, while output tokens are generated one step at a time. Meter tokens per tenant at the gateway so each team sees its own bill.&lt;/p&gt;

&lt;h3&gt;
  
  
  12. How would you bring cost per request down?
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Right-size the card.&lt;/strong&gt; An 8B model doesn't need an H100 if a smaller card meets the latency target. Small models can share a card with MIG.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Fill the replicas.&lt;/strong&gt; Let each run more sequences at once (vLLM's &lt;code&gt;--max-num-seqs&lt;/code&gt;) while latency stays inside the target. Set &lt;code&gt;minReplicaCount&lt;/code&gt; to the overnight floor and let a KEDA cron trigger raise it during the day.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Quantize where the evals allow it.&lt;/strong&gt; Smaller weights leave more memory for the KV cache, and so more concurrent requests per card.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Commit to the floor, burst on demand.&lt;/strong&gt; Reservations or savings plans cover the always-on replicas.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Use spot for batch, not for chat.&lt;/strong&gt; AWS gives a best-effort two-minute warning before reclaiming a Spot instance. Streams that outlast the drain are cut off, and the replacement may need a cold start. Batch jobs can retry.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The Kubernetes side of the bill is in &lt;a href="https://devopsinterview.cloud/blog/kubernetes-cost-optimization-200-node-cluster" rel="noopener noreferrer"&gt;cutting costs on a 200-node Kubernetes cluster&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  How do you roll out and roll back a model on a GPU fleet?
&lt;/h2&gt;

&lt;h3&gt;
  
  
  13. What makes a model rollout harder on GPUs than an app rollout?
&lt;/h3&gt;

&lt;p&gt;Capacity. A rolling update needs somewhere to put new pods. The defaults allow 25% extra pods (rounded up) and 25% unavailable (rounded down), so a Deployment with four or more replicas can free a GPU by removing an old pod first, running short until the rollout ends. With fewer replicas, or &lt;code&gt;maxUnavailable: 0&lt;/code&gt;, on a pool with no free cards, the new pods stay Pending. The rollout reports &lt;code&gt;ProgressDeadlineExceeded&lt;/code&gt; after its progress deadline (600 seconds by default), and Kubernetes doesn't roll it back for you. Blue-green avoids that but doubles the GPU bill while both versions run.&lt;/p&gt;

&lt;p&gt;The usual answer is a canary: a second Deployment with one or two replicas of the new version behind an HTTPRoute that splits traffic by weight, on GPUs held back as headroom or freed by shrinking the stable Deployment. Then:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Pin weights to an immutable version, a registry version or a commit hash, so a restarted pod can't quietly load a different model.&lt;/li&gt;
&lt;li&gt;Gate on time to first token, error rate and a quality signal. Offline evals, shadow traffic and canary analysis are in &lt;a href="https://devopsinterview.cloud/blog/mlops-interview-questions" rel="noopener noreferrer"&gt;MLOps interview questions&lt;/a&gt;, and the general mechanics in the free &lt;a href="https://devopsinterview.cloud/questions/canary-deployments-test-in-production-safely" rel="noopener noreferrer"&gt;canary deployment question&lt;/a&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  14. How do you make a model rollback fast?
&lt;/h3&gt;

&lt;p&gt;Make it a routing change. If the previous version is still running, even with fewer replicas, rollback is a weight change on the route. If it was scaled away, rollback is a cold start, and the old weights should at least still be in the node cache. The slower failures are compatibility problems:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Prompt templates and tokenizers.&lt;/strong&gt; An app that changed its prompt format for the new model can break against the old one. Version the template with the model and roll back both.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The API contract.&lt;/strong&gt; Clients should call a stable model name that the platform maps to a version, so rollback needs no client change.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Interview questions this covers
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;How would you design a platform that serves LLMs to many internal teams?&lt;/li&gt;
&lt;li&gt;How do you estimate how many GPUs a model needs?&lt;/li&gt;
&lt;li&gt;Which metric would you autoscale an inference service on, and why not GPU utilization?&lt;/li&gt;
&lt;li&gt;How do you cut cold-start time for a model server?&lt;/li&gt;
&lt;li&gt;How do you calculate cost per request, and how do you bring it down?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For more practice, browse &lt;a href="https://devopsinterview.cloud/ebooks" rel="noopener noreferrer"&gt;all the DevOps interview ebooks&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;None of the books covers AI infrastructure end to end, but the &lt;a href="https://devopsinterview.cloud/ebooks/complete-devops-mastery-bundle" rel="noopener noreferrer"&gt;DevOps Interview Mastery bundle&lt;/a&gt; covers the platform underneath it: five books and 252 senior-level questions with worked answers across cloud, Kubernetes, Terraform, CI/CD and SRE. Some are about AI and ML workloads: cutting AI/ML infrastructure costs without sacrificing training throughput or inference latency (Cloud Platforms), designing a custom Kubernetes scheduler for GPU ML workloads and building an MLOps platform on Kubernetes (Containers &amp;amp; Orchestration), and building a pipeline that trains, validates and ships a model (CI/CD &amp;amp; GitOps). Every purchase includes the free Interview-Day Playbook with a 7-day and 24-hour prep plan and a system-design walkthrough scaffold. If a full AI infrastructure book goes ahead, the &lt;a href="https://devopsinterview.cloud/ai-infrastructure-interview-questions" rel="noopener noreferrer"&gt;AI infrastructure interview questions page&lt;/a&gt; is where to get the one email about it.&lt;/p&gt;

&lt;p&gt;Prefer to watch or listen? Answers like these are walked through on &lt;a href="https://www.youtube.com/@DevOpsInterviewCloud" rel="noopener noreferrer"&gt;YouTube&lt;/a&gt; and on the podcast, on &lt;a href="https://open.spotify.com/show/033txr35oY2Rcn6H4CVZ4S" rel="noopener noreferrer"&gt;Spotify&lt;/a&gt; and &lt;a href="https://music.amazon.com/podcasts/49554d87-91cb-4816-9420-cdac8a06ac75/devops-cloud-interview-prep-real-scenarios-answers" rel="noopener noreferrer"&gt;Amazon Music&lt;/a&gt;.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://devopsinterview.cloud/blog/ai-infrastructure-engineer-interview-questions" rel="noopener noreferrer"&gt;devopsinterview.cloud&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>infrastructure</category>
      <category>interview</category>
      <category>systemdesign</category>
    </item>
    <item>
      <title>Terraform State Interview Questions and Answers</title>
      <dc:creator>DevOpsInterview.Cloud</dc:creator>
      <pubDate>Tue, 10 Nov 2026 09:20:03 +0000</pubDate>
      <link>https://dev.to/devopsinterview_cloud/terraform-state-interview-questions-and-answers-3oj5</link>
      <guid>https://dev.to/devopsinterview_cloud/terraform-state-interview-questions-and-answers-3oj5</guid>
      <description>&lt;p&gt;Terraform state interview questions rarely stop at the definition. "What is state?" turns into "where do you keep it?", then "the lock won't release, now what?", then "someone deleted the state file, walk me through the next hour". The questions below follow that path, from backends and locking to splitting a monolith and recovering lost state. For the wider Terraform screen, see &lt;a href="https://devopsinterview.cloud/blog/terraform-interview-questions" rel="noopener noreferrer"&gt;20 Terraform interview questions from real senior interviews&lt;/a&gt;; this post stays on state.&lt;/p&gt;

&lt;h2&gt;
  
  
  Terraform state interview questions on the basics
&lt;/h2&gt;

&lt;h3&gt;
  
  
  1. What does a state file hold, and why can't Terraform just query the cloud?
&lt;/h3&gt;

&lt;p&gt;State maps each resource address in your configuration to the ID of the real object, caches that object's attributes, and records dependencies and output values. Two top-level fields matter in recovery questions: &lt;code&gt;lineage&lt;/code&gt;, a unique ID assigned when the state is first created, and &lt;code&gt;serial&lt;/code&gt;, which goes up every time the state is written. Terraform uses them to tell whether two snapshots share a history and which one is newer.&lt;/p&gt;

&lt;p&gt;You can inspect state without opening the JSON:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;terraform state list                       &lt;span class="c"&gt;# every address Terraform tracks&lt;/span&gt;
terraform state show aws_db_instance.main  &lt;span class="c"&gt;# cached attributes of one object&lt;/span&gt;
terraform state pull &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; snapshot.tfstate    &lt;span class="c"&gt;# raw copy of the current state&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The usual follow-up: why not look everything up in the cloud? Names and tags aren't reliable identity, many types have no natural key, and Terraform must remember objects that have left the configuration, because those are the ones it should destroy.&lt;/p&gt;

&lt;h2&gt;
  
  
  Remote backends and Terraform state locking
&lt;/h2&gt;

&lt;h3&gt;
  
  
  2. Which remote backend would you choose, and how would you configure it?
&lt;/h3&gt;

&lt;p&gt;Use the one that matches where the team runs: S3, Azure Blob Storage, Google Cloud Storage, or HCP Terraform if you also want its runs and policy checks. Whichever you pick, cover locking, versioning, encryption and read access. On S3:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight hcl"&gt;&lt;code&gt;&lt;span class="n"&gt;terraform&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;backend&lt;/span&gt; &lt;span class="s2"&gt;"s3"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;bucket&lt;/span&gt;       &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"acme-tfstate-prod"&lt;/span&gt;
    &lt;span class="n"&gt;key&lt;/span&gt;          &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"network/terraform.tfstate"&lt;/span&gt;
    &lt;span class="n"&gt;region&lt;/span&gt;       &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"eu-west-1"&lt;/span&gt;
    &lt;span class="n"&gt;encrypt&lt;/span&gt;      &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
    &lt;span class="n"&gt;kms_key_id&lt;/span&gt;   &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"arn:aws:kms:eu-west-1:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab"&lt;/span&gt;
    &lt;span class="n"&gt;use_lockfile&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The details that matter:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Locking is opt-in on S3.&lt;/strong&gt; &lt;code&gt;use_lockfile = true&lt;/code&gt; makes Terraform write a lock object next to the state, here &lt;code&gt;network/terraform.tfstate.tflock&lt;/code&gt;. S3-native locking was experimental in Terraform 1.10 and has been generally available since 1.11. DynamoDB locking (&lt;code&gt;dynamodb_table&lt;/code&gt;) is deprecated; you can set both while you migrate, then drop the table.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Turn on bucket versioning.&lt;/strong&gt; The backend docs recommend it for state recovery, and it is what saves you in question 8.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Encryption and access.&lt;/strong&gt; &lt;code&gt;encrypt&lt;/code&gt; turns on server-side encryption of the state and lock files. With a customer-managed KMS key, the key policy becomes a second control on who can read state.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No variables in the backend block.&lt;/strong&gt; Terraform's backend block can't reference variables, locals or data sources, so per-environment values go into partial configuration: &lt;code&gt;terraform init -backend-config=prod.s3.tfbackend&lt;/code&gt;. OpenTofu 1.8 and later can evaluate variables and locals there. Supply credentials through the environment, because values passed with &lt;code&gt;-backend-config&lt;/code&gt; end up in the &lt;code&gt;.terraform&lt;/code&gt; directory and in plan files.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  3. An apply fails with "Error acquiring the state lock". What do you do?
&lt;/h3&gt;

&lt;p&gt;Read the Lock Info block in the error first. It shows the lock ID, the operation, who holds the lock and when it was taken. Usually another run is in progress and the fix is to wait. In CI, &lt;code&gt;-lock-timeout=5m&lt;/code&gt; makes a queued run retry instead of failing at once.&lt;/p&gt;

&lt;p&gt;If the holder is dead, for example a CI runner killed mid-apply, first confirm that nothing is still running against that state: check the job, ask the person named in &lt;code&gt;Who&lt;/code&gt;. Then release the lock with the ID from the error:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;terraform force-unlock 9db590f1-b6fe-c5f2-2678-8804f089deba
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The docs reserve force-unlock for your own lock when automatic unlocking failed: unlocking a lock someone else still holds can leave you with multiple writers. &lt;code&gt;-lock=false&lt;/code&gt; skips the protection rather than fixing anything. After a killed apply, run a plan before the next apply and read it. Objects the dead run created may never have made it into state, and a plan that wants to create something that already exists is how you find them.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. What does state locking not protect you from?
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Two states managing the same object.&lt;/strong&gt; A lock covers one state, not one cloud resource. If two root modules declare the same security group with different rules, each apply undoes the other's change, and neither lock prevents it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The gap between plan and apply.&lt;/strong&gt; A lock lasts for a single operation, so nothing holds it between a plan reviewed on Tuesday and an apply on Thursday. Terraform guards that gap a different way: a saved plan records the state it was built from, and if anything writes state in between (another apply, an import, a &lt;code&gt;state rm&lt;/code&gt;), applying it fails with "Saved plan is stale" and you plan again.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  5. How do you secure state, given it stores secrets in plaintext?
&lt;/h3&gt;

&lt;p&gt;State and saved plan files can contain sensitive resource attributes, including generated passwords and private keys. Marking a variable or output &lt;code&gt;sensitive&lt;/code&gt; hides it in plan and apply output; the value is still in state. The controls that work:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Limit who can read the backend with IAM on the bucket and key prefix, plus the KMS key policy, and audit reads.&lt;/li&gt;
&lt;li&gt;Treat saved plan files like state. CI artifacts that keep &lt;code&gt;tfplan&lt;/code&gt; files around are a second copy of your secrets.&lt;/li&gt;
&lt;li&gt;Keep secrets out of state where the provider supports it: ephemeral resources (Terraform 1.10) and write-only arguments (Terraform 1.11) are omitted from state and plan files.&lt;/li&gt;
&lt;li&gt;On OpenTofu, client-side encryption (since 1.7) encrypts state before it reaches the backend, and can encrypt plan files too.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  How do you split a monolithic Terraform state?
&lt;/h2&gt;

&lt;h3&gt;
  
  
  6. One state holds everything, plans are slow and every apply touches it all. How do you split it?
&lt;/h3&gt;

&lt;p&gt;Pick the boundaries first, by blast radius, ownership and how often things change. Networking, shared data stores and each service's resources are the usual cuts. Each piece becomes a root module with its own state key, and its own credentials when the boundary is an account.&lt;/p&gt;

&lt;p&gt;Then move objects without destroying them, using import blocks in the new root and removed blocks in the old one, so every step is a plan someone can review. Check first that each resource type you move supports import; most do, and any that don't stay in the old root for now.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Pause applies on the old root and save a copy: &lt;code&gt;terraform state pull &amp;gt; pre-split.tfstate&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;In the new root, add the resource blocks and an &lt;code&gt;import&lt;/code&gt; block for each one. The provider docs give each type's import ID format, and &lt;code&gt;terraform state show&lt;/code&gt; in the old root has the values.&lt;/li&gt;
&lt;li&gt;In the old root, delete those resource blocks and add &lt;code&gt;removed&lt;/code&gt; blocks with &lt;code&gt;destroy = false&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Plan both. The new root should show only imports and the old one only removals from state. Any create, update or destroy is a configuration mismatch to fix first.&lt;/li&gt;
&lt;li&gt;Apply the new root, then the old one, then resume applies.
&lt;/li&gt;
&lt;/ol&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight hcl"&gt;&lt;code&gt;&lt;span class="c1"&gt;# new root: network/imports.tf&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;to&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;aws_vpc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;main&lt;/span&gt;
  &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"vpc-0a1b2c3d4e5f67890"&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="c1"&gt;# old root: removed.tf&lt;/span&gt;
&lt;span class="n"&gt;removed&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;from&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;aws_vpc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;main&lt;/span&gt;
  &lt;span class="n"&gt;lifecycle&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;destroy&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Import blocks need Terraform 1.5, or 1.7 for &lt;code&gt;for_each&lt;/code&gt; across many instances; removed blocks need 1.7. Applying the import first means that for a few minutes the object sits in both states, which is harmless while applies are paused. The other order leaves it unmanaged, and if the second step gets forgotten you have an orphan.&lt;/p&gt;

&lt;p&gt;The new roots still need each other's values. A reference like &lt;code&gt;aws_vpc.main.id&lt;/code&gt; becomes a cross-stack lookup: the &lt;code&gt;terraform_remote_state&lt;/code&gt; data source reads another stack's outputs, but the docs point out that whoever uses it needs access to the entire state snapshot. Publishing values to SSM Parameter Store, or looking objects up with data sources, avoids that. For one root per account in a single repository, see &lt;a href="https://devopsinterview.cloud/blog/terraform-mono-repo-20-aws-accounts" rel="noopener noreferrer"&gt;Terraform mono-repo design for 20 AWS accounts&lt;/a&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  7. When do moved blocks help, and where do they stop?
&lt;/h3&gt;

&lt;p&gt;Moved blocks (Terraform 1.1 and later) record a refactor inside one state: renaming a resource, moving it into a module, or switching instance keys when you replace &lt;code&gt;count&lt;/code&gt; with &lt;code&gt;for_each&lt;/code&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight hcl"&gt;&lt;code&gt;&lt;span class="n"&gt;moved&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;from&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;aws_instance&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;web&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
  &lt;span class="n"&gt;to&lt;/span&gt;   &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;aws_instance&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;web&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"blue"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="n"&gt;moved&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;from&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;aws_s3_bucket&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;logs&lt;/span&gt;
  &lt;span class="n"&gt;to&lt;/span&gt;   &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;module&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;logging&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;aws_s3_bucket&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;this&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The plan reports each object as moved, with no destroy and create. Terraform looks up the &lt;code&gt;from&lt;/code&gt; address in the state of the configuration that declares the block, so a moved block can't hand an object to a different state; that is the import-and-removed job from question 6. A module can only declare moves for its own objects and its child modules' objects. Changing resource type works only where the provider supports it. The &lt;a href="https://devopsinterview.cloud/questions/terraform-moved-blocks-refactor-without-destroy" rel="noopener noreferrer"&gt;moved blocks question page&lt;/a&gt; walks the rename case end to end.&lt;/p&gt;

&lt;p&gt;For renames, module moves and key changes within one resource type, &lt;code&gt;terraform state mv&lt;/code&gt; does the same job by hand, once per state. A moved block ships with the module, so every configuration using it picks up the move on its next plan. HashiCorp recommends keeping old moved blocks in shared modules, since removing one breaks anyone who hasn't applied the new version yet.&lt;/p&gt;

&lt;h2&gt;
  
  
  Recovering corrupted or lost Terraform state
&lt;/h2&gt;

&lt;h3&gt;
  
  
  8. The state file is corrupted or deleted. How do you recover?
&lt;/h3&gt;

&lt;p&gt;Pause the pipelines first: against empty state, Terraform plans to create every resource again, and an apply would duplicate anything without a unique name.&lt;/p&gt;

&lt;p&gt;Then recover from the backend's history. With S3 versioning, find the last good version and download it to inspect:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;aws s3api list-object-versions --bucket acme-tfstate-prod \
  --prefix network/terraform.tfstate
aws s3api get-object --bucket acme-tfstate-prod \
  --key network/terraform.tfstate --version-id &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;version-id&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; restored.tfstate
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If Terraform can't read the current snapshot, restore through S3 by copying the good version back over the key. &lt;code&gt;terraform state push restored.tfstate&lt;/code&gt; works when the current state is readable or absent. It refuses a state whose lineage differs or whose serial is lower than the remote one; &lt;code&gt;-force&lt;/code&gt; skips those two checks but not read errors. A deliberate rollback always has a lower serial, so check the lineage before you use &lt;code&gt;-force&lt;/code&gt;. One trap on older setups: with DynamoDB locking, the table stores a checksum of the state, and an object restored behind Terraform's back fails that check until the digest is updated. HCP Terraform keeps a version history per workspace, so the same idea applies there.&lt;/p&gt;

&lt;p&gt;A restored snapshot is only as current as its timestamp. Run a plan and read every line. Objects created after the snapshot appear as creates for things that already exist, so import those; refresh picks up later updates and deletions of tracked objects, and each of those diffs needs a decision before anyone applies.&lt;/p&gt;

&lt;p&gt;With no backup at all, rebuild state from the configuration. Write an &lt;code&gt;import&lt;/code&gt; block for every resource address, look up each ID in the console or CLI, and iterate on the plan until it shows only imports. A type that doesn't support import has to be recreated in a controlled change. For resources that have no configuration yet, &lt;code&gt;terraform plan -generate-config-out=generated.tf&lt;/code&gt; writes resource blocks for you; it is still marked experimental, so review what it produces. The &lt;a href="https://devopsinterview.cloud/questions/terraform-import-bring-existing-resources-under-control" rel="noopener noreferrer"&gt;Terraform import question page&lt;/a&gt; covers the adoption workflow.&lt;/p&gt;

&lt;p&gt;Then add the guardrail that would have made this boring. In a versioned bucket, a plain delete only adds a delete marker, and a request to remove a version for good needs &lt;code&gt;s3:DeleteObjectVersion&lt;/code&gt;, so deny that to everyone except a break-glass role. Lifecycle rules can still expire old versions, so check they keep noncurrent state versions as long as you might need them.&lt;/p&gt;

&lt;h3&gt;
  
  
  9. The apply failed with "Failed to persist state to backend". What now?
&lt;/h3&gt;

&lt;p&gt;Terraform changed the infrastructure but couldn't write the new state, often because credentials expired during a long apply or the backend refused the write. It saves the state it couldn't upload to &lt;code&gt;errored.tfstate&lt;/code&gt; in the working directory, and that file is the only record of what the apply just did. Don't run another apply. Fix the access problem, push the file, then review a fresh plan against the failed run's errors and what actually exists:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;terraform state pull &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; before-push.tfstate   &lt;span class="c"&gt;# what the backend holds now&lt;/span&gt;
terraform state push errored.tfstate
terraform plan                               &lt;span class="c"&gt;# reconcile with the failed run&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In CI, a failed job has to keep &lt;code&gt;errored.tfstate&lt;/code&gt; somewhere as locked down as the backend itself, since it holds the same secrets. If the runner's workspace is wiped, the file is gone and you are back to question 8, importing whatever that apply created.&lt;/p&gt;

&lt;h2&gt;
  
  
  Workspaces or a directory per environment?
&lt;/h2&gt;

&lt;h3&gt;
  
  
  10. Should environments use CLI workspaces or separate directories?
&lt;/h3&gt;

&lt;p&gt;CLI workspaces give one configuration several states in the same backend. On S3, a &lt;code&gt;staging&lt;/code&gt; workspace's state lands at &lt;code&gt;env:/staging/network/terraform.tfstate&lt;/code&gt;, so every environment shares the bucket, the backend access and the code path, even when provider roles differ per workspace. Workspaces alone aren't an access boundary. They suit short-lived copies, such as a test environment per branch. For dev and prod, with different accounts, roles and approvals, a root module per environment with its own backend configuration is easier to defend. HCP Terraform workspaces are a different thing, each with its own state, variables and permissions. The &lt;a href="https://devopsinterview.cloud/questions/terraform-workspaces-multi-environment-management" rel="noopener noreferrer"&gt;Terraform workspaces question page&lt;/a&gt; goes through the trade-off.&lt;/p&gt;

&lt;h2&gt;
  
  
  Interview questions this covers
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Which backend would you use for Terraform state, and how do you lock state on S3 without DynamoDB?&lt;/li&gt;
&lt;li&gt;An apply can't acquire the state lock. When is force-unlock safe?&lt;/li&gt;
&lt;li&gt;How do you split a monolithic Terraform state without destroying or recreating anything?&lt;/li&gt;
&lt;li&gt;Your state file is corrupted or deleted. Walk me through the recovery.&lt;/li&gt;
&lt;li&gt;When do you use moved blocks, and when do you need import and removed blocks instead?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For more practice, browse &lt;a href="https://devopsinterview.cloud/ebooks" rel="noopener noreferrer"&gt;all the DevOps interview ebooks&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Senior state questions go further: designing state for 200+ configurations with locking, isolation and disaster recovery, and speeding up a plan that has crept from two minutes to twenty. &lt;a href="https://devopsinterview.cloud/ebooks/infrastructure-automation-mastery" rel="noopener noreferrer"&gt;Infrastructure as Code: Terraform &amp;amp; OpenTofu Interview Questions&lt;/a&gt; works through 50 senior-level questions with worked answers, including a chapter on state and backends. It is one of the five books in the &lt;a href="https://devopsinterview.cloud/ebooks/complete-devops-mastery-bundle" rel="noopener noreferrer"&gt;DevOps Interview Mastery bundle&lt;/a&gt;, and every purchase includes the free Interview-Day Playbook with a 7-day and 24-hour prep plan and a 30-second answer framework.&lt;/p&gt;

&lt;p&gt;Prefer to watch or listen? Answers like these are walked through on &lt;a href="https://www.youtube.com/@DevOpsInterviewCloud" rel="noopener noreferrer"&gt;YouTube&lt;/a&gt; and on the podcast, on &lt;a href="https://open.spotify.com/show/033txr35oY2Rcn6H4CVZ4S" rel="noopener noreferrer"&gt;Spotify&lt;/a&gt; and &lt;a href="https://music.amazon.com/podcasts/49554d87-91cb-4816-9420-cdac8a06ac75/devops-cloud-interview-prep-real-scenarios-answers" rel="noopener noreferrer"&gt;Amazon Music&lt;/a&gt;.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://devopsinterview.cloud/blog/terraform-state-interview-questions" rel="noopener noreferrer"&gt;devopsinterview.cloud&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>devops</category>
      <category>infrastructure</category>
      <category>interview</category>
      <category>terraform</category>
    </item>
    <item>
      <title>Kubernetes GPU Sharing: MIG, Time-Slicing, MPS and DRA</title>
      <dc:creator>DevOpsInterview.Cloud</dc:creator>
      <pubDate>Sun, 08 Nov 2026 09:20:02 +0000</pubDate>
      <link>https://dev.to/devopsinterview_cloud/kubernetes-gpu-sharing-mig-time-slicing-mps-and-dra-141b</link>
      <guid>https://dev.to/devopsinterview_cloud/kubernetes-gpu-sharing-mig-time-slicing-mps-and-dra-141b</guid>
      <description>&lt;p&gt;Kubernetes GPU sharing usually comes up in an interview as a cost problem: a small model holds a whole card, and the card sits mostly idle. The interviewer then wants to know how a pod gets a GPU at all, what MIG, time-slicing and MPS each isolate, where Dynamic Resource Allocation fits, and how you would prove the savings. These questions follow that order. They assume NVIDIA hardware, since that's where most of the tooling and most of the interview questions are.&lt;/p&gt;

&lt;h2&gt;
  
  
  How does Kubernetes schedule a GPU by default?
&lt;/h2&gt;

&lt;h3&gt;
  
  
  1. What does the NVIDIA device plugin actually do?
&lt;/h3&gt;

&lt;p&gt;It runs as a DaemonSet, finds the GPUs on each node and advertises them to the kubelet as an extended resource, &lt;code&gt;nvidia.com/gpu&lt;/code&gt;. From then on the scheduler treats GPUs as a count, and a pod asks for them in its limits:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;resources&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;limits&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;nvidia.com/gpu&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;1&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three follow-ups come straight from that design:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Why can't a pod ask for half a GPU?&lt;/strong&gt; The API server &lt;a href="https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#consuming-extended-resources" rel="noopener noreferrer"&gt;restricts extended resources to whole numbers&lt;/a&gt;, so &lt;code&gt;0.5&lt;/code&gt; is rejected. Every sharing method has to work around that.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Why limits and not requests?&lt;/strong&gt; &lt;a href="https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/" rel="noopener noreferrer"&gt;GPUs are only specified in limits&lt;/a&gt;. The limit becomes the request, and if you set both they must match, because extended resources can't be overcommitted.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;What can't the scheduler see?&lt;/strong&gt; Anything other than the count. Memory size, model and how busy the card is aren't part of the decision unless you add node labels and selectors.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The &lt;a href="https://github.com/NVIDIA/k8s-device-plugin#running-gpu-jobs" rel="noopener noreferrer"&gt;plugin's README&lt;/a&gt; also warns about containers that don't request a GPU. Depending on the node's runtime configuration they can still see all of them, because &lt;a href="https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/docker-specialized.html" rel="noopener noreferrer"&gt;base CUDA images default &lt;code&gt;NVIDIA_VISIBLE_DEVICES&lt;/code&gt; to &lt;code&gt;all&lt;/code&gt;&lt;/a&gt;. That's a good reason to keep general workloads off these nodes.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. What does the GPU Operator add on top?
&lt;/h3&gt;

&lt;p&gt;The device plugin assumes the node already has an NVIDIA driver and container runtime support. The &lt;a href="https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/overview.html" rel="noopener noreferrer"&gt;GPU Operator&lt;/a&gt; manages that whole stack: driver, container toolkit, device plugin, GPU Feature Discovery labels, the DCGM exporter and the MIG Manager. If your node image already ships the driver, install with &lt;code&gt;driver.enabled=false&lt;/code&gt; and let the Operator handle the rest.&lt;/p&gt;

&lt;h2&gt;
  
  
  Kubernetes GPU sharing: MIG vs time-slicing vs MPS
&lt;/h2&gt;

&lt;p&gt;They differ mainly in what they isolate.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Time-slicing&lt;/th&gt;
&lt;th&gt;MPS&lt;/th&gt;
&lt;th&gt;MIG&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;How it shares&lt;/td&gt;
&lt;td&gt;Processes take turns on the GPU&lt;/td&gt;
&lt;td&gt;Kernels from different processes run concurrently&lt;/td&gt;
&lt;td&gt;Fixed hardware partitions&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Memory&lt;/td&gt;
&lt;td&gt;Shared, no per-pod limit&lt;/td&gt;
&lt;td&gt;Separate address spaces, per-client limit&lt;/td&gt;
&lt;td&gt;Dedicated to each instance&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Compute&lt;/td&gt;
&lt;td&gt;Equal time per process, no guarantee&lt;/td&gt;
&lt;td&gt;Limited per client&lt;/td&gt;
&lt;td&gt;Dedicated to each instance&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Fault isolation&lt;/td&gt;
&lt;td&gt;None&lt;/td&gt;
&lt;td&gt;None between clients on the same GPU&lt;/td&gt;
&lt;td&gt;Yes, between instances&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Hardware&lt;/td&gt;
&lt;td&gt;NVIDIA GPUs generally&lt;/td&gt;
&lt;td&gt;Whole GPUs only in the device plugin&lt;/td&gt;
&lt;td&gt;Ampere and newer data center GPUs&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h3&gt;
  
  
  3. What does time-slicing isolate?
&lt;/h3&gt;

&lt;p&gt;Nothing. The device plugin advertises several replicas of each GPU and hands them to whoever asks. This config makes one GPU look like four:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;version&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;v1&lt;/span&gt;
&lt;span class="na"&gt;flags&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;migStrategy&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;none&lt;/span&gt;
&lt;span class="na"&gt;sharing&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;timeSlicing&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;failRequestsGreaterThanOne&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
    &lt;span class="na"&gt;resources&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;nvidia.com/gpu&lt;/span&gt;
      &lt;span class="na"&gt;replicas&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;4&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Replicas aren't units of capacity. &lt;a href="https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-sharing.html" rel="noopener noreferrer"&gt;NVIDIA's docs&lt;/a&gt; are explicit: with four replicas, a pod requesting two gets the same share of compute as a pod requesting one. Setting &lt;code&gt;failRequestsGreaterThanOne: true&lt;/code&gt; makes that request fail with &lt;code&gt;UnexpectedAdmissionError&lt;/code&gt; instead of quietly misleading someone. It defaults to &lt;code&gt;false&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Memory is the bigger risk. Each workload can reach all of the GPU's memory, so one pod can use up what its neighbour needed. The workloads also share a fault domain. The README's wording is blunt: "if one workload crashes, they all do."&lt;/p&gt;

&lt;p&gt;Time-slicing suits dev environments, notebooks and bursty batch jobs from teams who trust each other, and not much else.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. What does MIG isolate, and what does it cost you?
&lt;/h3&gt;

&lt;p&gt;&lt;a href="https://docs.nvidia.com/datacenter/tesla/mig-user-guide/introduction.html" rel="noopener noreferrer"&gt;MIG&lt;/a&gt; splits one GPU into up to seven instances, with separate paths through the memory system, partitioned compute and fault isolation between them. One tenant's heavy load doesn't take another's cache allocation or memory bandwidth, so performance stays predictable.&lt;/p&gt;

&lt;p&gt;Profile names give you the slice and the memory. On an H100 80GB, &lt;a href="https://docs.nvidia.com/datacenter/tesla/mig-user-guide/latest/supported-mig-profiles.html" rel="noopener noreferrer"&gt;&lt;code&gt;1g.10gb&lt;/code&gt;&lt;/a&gt; gets 1/7 of the compute and 1/8 of the memory, and you can fit seven. On an A100 40GB the same slot is &lt;code&gt;1g.5gb&lt;/code&gt;. MIG starts at the Ampere generation, and &lt;a href="https://docs.nvidia.com/datacenter/tesla/mig-user-guide/supported-gpus.html" rel="noopener noreferrer"&gt;the supported list&lt;/a&gt; doesn't include T4, L4 or L40S, so it won't help on those cards.&lt;/p&gt;

&lt;p&gt;With the GPU Operator you pick a layout through a node label:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl label nodes gpu-node-1 nvidia.com/mig.config&lt;span class="o"&gt;=&lt;/span&gt;all-1g.10gb &lt;span class="nt"&gt;--overwrite&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;a href="https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-mig.html" rel="noopener noreferrer"&gt;&lt;code&gt;single&lt;/code&gt; strategy&lt;/a&gt; exposes every slice as &lt;code&gt;nvidia.com/gpu&lt;/code&gt; and needs the same profile on all the node's GPUs. The &lt;code&gt;mixed&lt;/code&gt; strategy gives each profile its own resource, such as &lt;code&gt;nvidia.com/mig-1g.10gb&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The costs:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Reconfiguring is disruptive.&lt;/strong&gt; MIG Manager requires that no user workloads are running on the GPUs being changed, and some clouds need a reboot. A layout is a capacity-planning decision, not something a pod chooses.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Slices don't lend to each other.&lt;/strong&gt; Whatever a tenant leaves idle inside its slice stays idle.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Multi-GPU work suffers.&lt;/strong&gt; &lt;a href="https://docs.nvidia.com/datacenter/tesla/mig-user-guide/latest/deployment-considerations.html" rel="noopener noreferrer"&gt;Peer-to-peer between MIG instances on different GPUs isn't supported&lt;/a&gt;, so large distributed training belongs on whole cards.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Can you combine them? In the device plugin, yes for time-slicing on top of MIG resources from the &lt;code&gt;mixed&lt;/code&gt; strategy, and no for MPS, which isn't supported on MIG-enabled devices there.&lt;/p&gt;

&lt;h3&gt;
  
  
  5. What does MPS isolate?
&lt;/h3&gt;

&lt;p&gt;Memory and compute, but not faults. A control daemon manages access to the GPU, and &lt;a href="https://docs.nvidia.com/deploy/mps/when-to-use-mps.html" rel="noopener noreferrer"&gt;kernels from different processes can run concurrently&lt;/a&gt;, which helps when no single process has enough work to saturate the card. Clients get isolated address spaces. The device plugin limits each replica to an equal fraction of device memory and also limits its compute.&lt;/p&gt;

&lt;p&gt;It isn't full isolation. A fatal fault from one client reaches every client on that GPU, and it &lt;a href="https://docs.nvidia.com/deploy/mps/when-to-use-mps.html#memory-protection-and-error-containment" rel="noopener noreferrer"&gt;spreads further&lt;/a&gt; when one of those clients also uses another GPU. The error doesn't say which client caused it. The device plugin also labels its MPS support experimental, and time-slicing and MPS are mutually exclusive there.&lt;/p&gt;

&lt;h3&gt;
  
  
  6. Which one would you pick?
&lt;/h3&gt;

&lt;p&gt;Start with who shares the card:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Tenants who shouldn't affect each other, or inference with a latency target:&lt;/strong&gt; MIG, or a whole GPU if the model doesn't fit a slice.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Dev, notebooks and trusted bursty jobs:&lt;/strong&gt; time-slicing.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Small trusted processes that don't saturate a GPU and need to run at the same time:&lt;/strong&gt; MPS, after you've tested it on your own workload.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Training or a large model that fills the card:&lt;/strong&gt; don't share it.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What does Dynamic Resource Allocation change for GPUs?
&lt;/h2&gt;

&lt;h3&gt;
  
  
  7. What problem does DRA solve?
&lt;/h3&gt;

&lt;p&gt;The count. &lt;a href="https://kubernetes.io/blog/2025/09/01/kubernetes-v1-34-dra-updates/" rel="noopener noreferrer"&gt;DRA graduated to GA in Kubernetes 1.34&lt;/a&gt;, and from 1.35 its feature gate is locked on. Instead of asking for a number of an extended resource, a pod references a claim. The driver publishes its devices with their attributes and capacity, and the claim selects from them with a CEL expression, so the scheduler can match on things it could never see before:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;resource.k8s.io/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ResourceClaimTemplate&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;large-gpu&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;devices&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;requests&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;gpu&lt;/span&gt;
        &lt;span class="na"&gt;exactly&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;deviceClassName&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;gpu.nvidia.com&lt;/span&gt;
          &lt;span class="na"&gt;selectors&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
          &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;cel&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
              &lt;span class="na"&gt;expression&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;device.capacity['gpu.nvidia.com'].memory.compareTo(quantity('40Gi')) &amp;gt;= &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;
&lt;span class="nn"&gt;---&lt;/span&gt;
&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Pod&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;inference&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;resourceClaims&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;gpu&lt;/span&gt;
    &lt;span class="na"&gt;resourceClaimTemplateName&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;large-gpu&lt;/span&gt;
  &lt;span class="na"&gt;containers&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;server&lt;/span&gt;
    &lt;span class="na"&gt;image&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;registry.example/server:v1&lt;/span&gt;
    &lt;span class="na"&gt;resources&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;claims&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;gpu&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A template gives each pod its own claim, and that claim is deleted with the pod. A standalone &lt;code&gt;ResourceClaim&lt;/code&gt; referenced by several pods shares one allocation.&lt;/p&gt;

&lt;p&gt;Know which parts are mature, because this is moving quickly. In the &lt;a href="https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/dra-intro-install.html" rel="noopener noreferrer"&gt;NVIDIA DRA driver&lt;/a&gt;, full GPUs and existing MIG slices are GA. MPS, custom time-slice intervals, dynamic MIG creation and consumable shares are all alpha gates, off by default. Consumable shares build on &lt;a href="https://kubernetes.io/docs/concepts/resource-management/dynamic-resource-allocation/dra-features/#consumable-capacity" rel="noopener noreferrer"&gt;consumable capacity&lt;/a&gt;, beta in Kubernetes since 1.36, which lets independent claims share one device while the scheduler tracks how much is used. Device plugin replicas never gave you that accounting.&lt;/p&gt;

&lt;h3&gt;
  
  
  8. Would you move to DRA today?
&lt;/h3&gt;

&lt;p&gt;On a new cluster where you want attribute-based selection, it's a reasonable default. For a fleet where the device plugin works, plan it rather than rush it, because of these catches:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Migration.&lt;/strong&gt; With the GPU Operator, a cluster has either a &lt;code&gt;GPUCluster&lt;/code&gt; for DRA or a &lt;code&gt;ClusterPolicy&lt;/code&gt; for the device plugin, not both, and moving in place isn't supported. That workflow means a new cluster.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Preemption.&lt;/strong&gt; &lt;a href="https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/" rel="noopener noreferrer"&gt;The scheduler doesn't preempt for DRA resources&lt;/a&gt;, so a high-priority pod waits until a device frees up. If your design relies on &lt;a href="https://devopsinterview.cloud/questions/pod-priority-and-preemption-who-gets-evicted" rel="noopener noreferrer"&gt;pod priority and preemption&lt;/a&gt; to evict batch work for inference, that's a real gap.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Autoscaling.&lt;/strong&gt; &lt;a href="https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/docs/dra_support.md" rel="noopener noreferrer"&gt;Cluster Autoscaler&lt;/a&gt; can scale a DRA node group up from zero only if the cloud provider implements it for that driver. Otherwise you keep at least one node.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;There's a bridge for old manifests: a DeviceClass can set &lt;a href="https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#extended-resources-allocation-by-dra" rel="noopener noreferrer"&gt;&lt;code&gt;extendedResourceName&lt;/code&gt;&lt;/a&gt;, so pods still asking for an extended resource can get devices through DRA. It went stable in 1.37.&lt;/p&gt;

&lt;h2&gt;
  
  
  How do you keep GPU nodes for GPU workloads?
&lt;/h2&gt;

&lt;h3&gt;
  
  
  9. How do taints and node pools fit in?
&lt;/h3&gt;

&lt;p&gt;Give GPUs their own node pools and use two mechanisms together, because they do different jobs. A taint keeps other pods off, and a label selector or affinity pulls GPU pods on. A toleration alone only permits a pod to land there. The free &lt;a href="https://devopsinterview.cloud/questions/taints-tolerations-and-node-affinity-quick-guide" rel="noopener noreferrer"&gt;taints and node affinity question&lt;/a&gt; covers the general distinction.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;tolerations&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;key&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;nvidia.com/gpu&lt;/span&gt;
    &lt;span class="na"&gt;operator&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Exists&lt;/span&gt;
    &lt;span class="na"&gt;effect&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;NoSchedule&lt;/span&gt;
  &lt;span class="na"&gt;nodeSelector&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;nvidia.com/gpu.product&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;A100-SXM4-40GB&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Tainting with the resource name lets the &lt;a href="https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#extendedresourcetoleration" rel="noopener noreferrer"&gt;&lt;code&gt;ExtendedResourceToleration&lt;/code&gt;&lt;/a&gt; admission controller add that toleration for you, though it isn't on by default. GKE adds the &lt;a href="https://docs.cloud.google.com/kubernetes-engine/docs/how-to/gpus" rel="noopener noreferrer"&gt;&lt;code&gt;nvidia.com/gpu=present:NoSchedule&lt;/code&gt; taint&lt;/a&gt; itself when you add a GPU pool to a cluster that already has a non-GPU pool, so check what your provider does.&lt;/p&gt;

&lt;p&gt;Separate the pools by GPU model and by sharing mode too, since the plugin applies one sharing method to all the GPUs on a node. On shared clusters, a &lt;a href="https://kubernetes.io/docs/concepts/policy/resource-quotas/" rel="noopener noreferrer"&gt;ResourceQuota on &lt;code&gt;requests.nvidia.com/gpu&lt;/code&gt;&lt;/a&gt; caps each team, and the &lt;a href="https://devopsinterview.cloud/blog/kubernetes-multi-tenancy-shared-cluster-design" rel="noopener noreferrer"&gt;multi-tenant cluster design&lt;/a&gt; post covers the rest of that layout. Time-slicing replicas count as GPUs in that quota, so a limit of four could mean a single physical card.&lt;/p&gt;

&lt;h2&gt;
  
  
  How do you measure GPU utilization and cost?
&lt;/h2&gt;

&lt;h3&gt;
  
  
  10. How do you prove that sharing saves money?
&lt;/h3&gt;

&lt;p&gt;Keep allocation and real use apart. A pod holding &lt;code&gt;nvidia.com/gpu: 1&lt;/code&gt; blocks the card whether it's working or not. &lt;a href="https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md" rel="noopener noreferrer"&gt;Cluster Autoscaler&lt;/a&gt; makes the same assumption: &lt;code&gt;scale-down-gpu-utilization-threshold&lt;/code&gt; (default 0.5) compares GPU requests with allocatable GPUs and ignores CPU and memory. A node full of idle GPU pods never looks underused to it.&lt;/p&gt;

&lt;p&gt;For real use, the DCGM exporter gives you hardware numbers:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;DCGM_FI_PROF_GR_ENGINE_ACTIVE&lt;/code&gt;: the fraction of time the graphics or compute engines were active.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;DCGM_FI_PROF_SM_ACTIVE&lt;/code&gt;: closer to useful work, though commented out in the exporter's default counters. &lt;a href="https://docs.nvidia.com/datacenter/dcgm/latest/user-guide/feature-overview.html" rel="noopener noreferrer"&gt;NVIDIA's guidance&lt;/a&gt; is that 0.8 or above is necessary but not sufficient for effective use, and below 0.5 likely means ineffective use.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;DCGM_FI_DEV_FB_USED&lt;/code&gt;: GPU memory in use on the device. Measure a workload's peak on a dedicated card, add headroom, and you know which slice it fits.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This query averages engine activity per GPU, and per MIG instance where one exists:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;avg by (UUID, GPU_I_ID) (avg_over_time(DCGM_FI_PROF_GR_ENGINE_ACTIVE[7d]))
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Monitoring has two gaps here. The exporter &lt;a href="https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-sharing.html" rel="noopener noreferrer"&gt;can't associate metrics with containers when time-slicing is on&lt;/a&gt;, so per-team chargeback gets harder. And on Ampere GPUs, &lt;a href="https://docs.nvidia.com/datacenter/tesla/mig-user-guide/getting-started-with-mig.html" rel="noopener noreferrer"&gt;NVML and &lt;code&gt;nvidia-smi&lt;/code&gt; can't attribute utilization to MIG devices&lt;/a&gt;, so use DCGM there.&lt;/p&gt;

&lt;p&gt;Then report a unit cost: cost per thousand requests for inference, or per run for training. Measure throughput on the slice instead of assuming it. Sharing has only saved money if that unit cost falls and the latency target still holds. The rest of the toolkit is the usual one, covered in &lt;a href="https://devopsinterview.cloud/blog/kubernetes-cost-optimization-200-node-cluster" rel="noopener noreferrer"&gt;Kubernetes cost optimization for a 200-node cluster&lt;/a&gt;: scale dev GPU pools to zero outside working hours, and run checkpointed training on spot. The wider round is on the &lt;a href="https://devopsinterview.cloud/ai-infrastructure-interview-questions" rel="noopener noreferrer"&gt;AI infrastructure interview questions page&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  GPU sharing scenario-based interview questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  11. You turned on time-slicing and pods stopped scheduling. Why?
&lt;/h3&gt;

&lt;p&gt;Sharing changes what the node advertises, in one of two ways:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;With &lt;code&gt;renameByDefault: true&lt;/code&gt;, the resource becomes &lt;code&gt;nvidia.com/gpu.shared&lt;/code&gt;, so pods asking for &lt;code&gt;nvidia.com/gpu&lt;/code&gt; no longer fit.&lt;/li&gt;
&lt;li&gt;With the default of &lt;code&gt;false&lt;/code&gt;, the resource name stays but the product label gains a suffix, such as &lt;code&gt;A100-SXM4-40GB-SHARED&lt;/code&gt;, so an exact &lt;code&gt;nodeSelector&lt;/code&gt; on the old value stops matching.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The Operator also doesn't watch time-slicing ConfigMaps, so after editing one you restart the device plugin pods yourself.&lt;/p&gt;

&lt;h3&gt;
  
  
  12. Two pods share a GPU and one keeps hitting CUDA out-of-memory. What do you do?
&lt;/h3&gt;

&lt;p&gt;Find out what's holding the memory. &lt;code&gt;nvidia-smi&lt;/code&gt; on the node lists GPU processes and their memory use, while &lt;code&gt;DCGM_FI_DEV_FB_USED&lt;/code&gt; only gives the device total. On a time-sliced GPU nothing stops a neighbour from taking it all. A framework cap such as PyTorch's &lt;a href="https://docs.pytorch.org/docs/2.15/generated/torch.cuda.memory.set_per_process_memory_fraction.html" rel="noopener noreferrer"&gt;&lt;code&gt;set_per_process_memory_fraction&lt;/code&gt;&lt;/a&gt; helps, but it limits that process's caching allocator and depends on every team setting it. The real fix is a boundary the platform enforces: MIG slices, MPS limits, or separate cards.&lt;/p&gt;

&lt;h3&gt;
  
  
  13. A GPU pod is Pending but the cluster has free GPUs. Where do you look?
&lt;/h3&gt;

&lt;p&gt;&lt;code&gt;kubectl describe pod&lt;/code&gt; first, then compare the event's reason with what the nodes advertise:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl describe node gpu-node-1 | &lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="nt"&gt;-E&lt;/span&gt; &lt;span class="s1"&gt;'nvidia.com/(gpu|mig)'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The usual causes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The plugin or driver isn't up, so the node advertises zero.&lt;/li&gt;
&lt;li&gt;The MIG layout doesn't offer the profile the pod requested.&lt;/li&gt;
&lt;li&gt;A taint has no matching toleration, or a label doesn't match.&lt;/li&gt;
&lt;li&gt;The free GPUs are spread out: a pod needs two on one node, and two nodes each have one free.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If the autoscaler is involved, its &lt;a href="https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md" rel="noopener noreferrer"&gt;&lt;code&gt;resourceUnready&lt;/code&gt;&lt;/a&gt; count points at nodes whose GPU resource never registered.&lt;/p&gt;

&lt;h2&gt;
  
  
  Interview questions this covers
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;How does Kubernetes schedule a GPU, and why can't a pod request half of one?&lt;/li&gt;
&lt;li&gt;MIG vs time-slicing vs MPS: what does each one isolate?&lt;/li&gt;
&lt;li&gt;What does Dynamic Resource Allocation change compared with the device plugin?&lt;/li&gt;
&lt;li&gt;How do you keep general workloads off GPU nodes?&lt;/li&gt;
&lt;li&gt;How do you measure GPU utilization and prove that sharing saved money?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For more practice, browse &lt;a href="https://devopsinterview.cloud/ebooks" rel="noopener noreferrer"&gt;all the DevOps interview ebooks&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;None of the books covers AI infrastructure end to end, but they cover the platform underneath it. &lt;a href="https://devopsinterview.cloud/ebooks/container-orchestration-journey" rel="noopener noreferrer"&gt;Containers &amp;amp; Orchestration: Kubernetes Interview Questions&lt;/a&gt; works through 52 senior-level questions with worked answers, including how you would design a custom scheduler for GPU ML workloads that prefers spot instances and how you would build an MLOps platform on Kubernetes. It is one of the five books in the &lt;a href="https://devopsinterview.cloud/ebooks/complete-devops-mastery-bundle" rel="noopener noreferrer"&gt;DevOps Interview Mastery bundle&lt;/a&gt;, and every purchase includes the free Interview-Day Playbook with a 7-day and 24-hour prep plan and a 30-second answer framework. If a full AI infrastructure book goes ahead, the &lt;a href="https://devopsinterview.cloud/ai-infrastructure-interview-questions" rel="noopener noreferrer"&gt;AI infrastructure interview questions page&lt;/a&gt; is where to get the one email about it.&lt;/p&gt;

&lt;p&gt;Prefer to watch or listen? Answers like these are walked through on &lt;a href="https://www.youtube.com/@DevOpsInterviewCloud" rel="noopener noreferrer"&gt;YouTube&lt;/a&gt; and on the podcast, on &lt;a href="https://open.spotify.com/show/033txr35oY2Rcn6H4CVZ4S" rel="noopener noreferrer"&gt;Spotify&lt;/a&gt; and &lt;a href="https://music.amazon.com/podcasts/49554d87-91cb-4816-9420-cdac8a06ac75/devops-cloud-interview-prep-real-scenarios-answers" rel="noopener noreferrer"&gt;Amazon Music&lt;/a&gt;.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://devopsinterview.cloud/blog/kubernetes-gpu-sharing-mig-time-slicing-mps" rel="noopener noreferrer"&gt;devopsinterview.cloud&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>architecture</category>
      <category>infrastructure</category>
      <category>kubernetes</category>
    </item>
    <item>
      <title>Kubernetes Networking Interview Questions and Answers</title>
      <dc:creator>DevOpsInterview.Cloud</dc:creator>
      <pubDate>Sat, 07 Nov 2026 09:20:03 +0000</pubDate>
      <link>https://dev.to/devopsinterview_cloud/kubernetes-networking-interview-questions-and-answers-2ocp</link>
      <guid>https://dev.to/devopsinterview_cloud/kubernetes-networking-interview-questions-and-answers-2ocp</guid>
      <description>&lt;p&gt;Kubernetes networking interview questions start simple and end at the packet level. "How does a Service work?" becomes "so where does the ClusterIP live?", then "what changes if you remove kube-proxy?", then "why does DNS only time out under load?". The questions below follow that path, from the pod network through Services, NetworkPolicy, Gateway API and DNS to a debugging scenario.&lt;/p&gt;

&lt;h2&gt;
  
  
  Kubernetes networking interview questions on the pod network
&lt;/h2&gt;

&lt;h3&gt;
  
  
  1. What does the Kubernetes network model require, and who implements it?
&lt;/h3&gt;

&lt;p&gt;Every pod gets its own cluster-wide IP address, shared by all of its containers, which reach each other over localhost. Pods reach pods on any node directly, without proxies or NAT, and agents on a node such as the kubelet can reach every pod on that node.&lt;/p&gt;

&lt;p&gt;Kubernetes implements very little of this itself. The container runtime sets up each pod's network namespace, and the pod network comes from a network plugin; on Linux, most container runtimes talk to it through CNI. kube-proxy is the default Service implementation, but some plugins ship their own. NetworkPolicy is enforced by the plugin or not at all: on a plugin without NetworkPolicy support, the API still accepts your policies and they have no effect.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. How do you choose a CNI plugin?
&lt;/h3&gt;

&lt;p&gt;Start from the decisions the plugins make differently, not from a brand name:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Decision&lt;/th&gt;
&lt;th&gt;Options&lt;/th&gt;
&lt;th&gt;What it costs&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;How pod packets cross the network&lt;/td&gt;
&lt;td&gt;Overlay (VXLAN and similar) or native routing (BGP, or addresses from the cloud VPC)&lt;/td&gt;
&lt;td&gt;Overlay adds encapsulation overhead; native routing needs the network underneath to route pod CIDRs&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;NetworkPolicy enforcement&lt;/td&gt;
&lt;td&gt;Calico and Cilium enforce it; flannel on its own does not&lt;/td&gt;
&lt;td&gt;Without it, policies are accepted and ignored&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Service data plane&lt;/td&gt;
&lt;td&gt;kube-proxy, or the plugin's own, such as Cilium's kube-proxy replacement&lt;/td&gt;
&lt;td&gt;A replacement moves Service debugging to the plugin's tools&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;On EKS, the Amazon VPC CNI gives every pod a private IP from the VPC, so two limits show up early: network interface limits cap pods per node (prefix delegation raises them), and every pod consumes a subnet address (a secondary CIDR for pods, or IPv6, relieves that). The &lt;a href="https://devopsinterview.cloud/blog/aws-vpc-interview-questions" rel="noopener noreferrer"&gt;AWS VPC interview questions&lt;/a&gt; post covers the subnet-sizing side.&lt;/p&gt;

&lt;h2&gt;
  
  
  Services, kube-proxy and eBPF
&lt;/h2&gt;

&lt;h3&gt;
  
  
  3. How does traffic to a ClusterIP Service reach a pod?
&lt;/h3&gt;

&lt;p&gt;The Service gets a virtual IP, and the EndpointSlice controller tracks the pods matching its selector, with each endpoint's readiness. kube-proxy on every node watches Services and EndpointSlices and programs the kernel to rewrite traffic for the ClusterIP and port to one backend pod. In iptables and nftables modes the backend is picked at random by default, or by session affinity if set, and no interface holds the ClusterIP: the packet is rewritten on the client's node before it leaves.&lt;/p&gt;

&lt;p&gt;The backend is picked per connection, not per request. Connection tracking carries every later packet of that connection to the same pod, so long-lived HTTP/2 or gRPC connections stay pinned to whichever pods they first reached, and pods added by a scale-up get none of that traffic until clients reconnect. Balancing those needs client-side load balancing or a proxy that understands requests.&lt;/p&gt;

&lt;p&gt;To check which mode kube-proxy runs in, ask it on the node:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl &lt;span class="nt"&gt;-s&lt;/span&gt; http://localhost:10249/proxyMode
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  4. iptables, IPVS, nftables or eBPF: which Service data plane would you run?
&lt;/h3&gt;

&lt;p&gt;Know where each one stands in 2026:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;iptables&lt;/strong&gt; is still the default on Linux. kube-proxy writes a few rules per Service and per endpoint, and with tens of thousands of Pods and Services, updating them after a change can take a long time.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;IPVS&lt;/strong&gt; is deprecated since Kubernetes 1.35, off by default from 1.40 and removed in 1.43. It never handled every edge case of the Services API, so skip it for new clusters.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;nftables&lt;/strong&gt; replaces both, and the docs say a future release will make it the default. It needs kernel 5.13 or later and processes endpoint changes faster than iptables mode. Migration gotchas: NodePorts listen only on the node's primary IPs by default, and it adds no firewall accept rules for them.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;eBPF&lt;/strong&gt;, through Cilium's kube-proxy replacement, handles ClusterIP, NodePort, LoadBalancer and externalIPs Services itself. Its socket-level load balancing picks a backend when a pod calls &lt;code&gt;connect()&lt;/code&gt;, so the socket connects straight to the backend with no per-packet NAT. The costs: kube-proxy has to go (removing it on a live cluster breaks existing Service connections), and you debug Services with Cilium's tools.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Interviewers want a reason tied to the cluster in front of you, not a favourite tool. The free question on &lt;a href="https://devopsinterview.cloud/questions/cilium-vs-kube-proxy-why-ebpf-wins" rel="noopener noreferrer"&gt;Cilium vs kube-proxy&lt;/a&gt; covers the eBPF case in more depth.&lt;/p&gt;

&lt;h2&gt;
  
  
  NetworkPolicy and default deny
&lt;/h2&gt;

&lt;h3&gt;
  
  
  5. What happens when the first NetworkPolicy lands in a namespace?
&lt;/h3&gt;

&lt;p&gt;By default, pods are non-isolated: all traffic in and out is allowed. A pod becomes isolated for ingress once any policy that selects it lists &lt;code&gt;Ingress&lt;/code&gt; in &lt;code&gt;policyTypes&lt;/code&gt;, and the same applies to egress. Once the plugin enforces it, new connections need an allow from some applicable policy, apart from reply traffic and ingress from the pod's own node; whether existing connections are cut is up to the implementation. Policies are additive, so the allowed set is the union of every policy that applies, order does not matter, and the core API has no deny rule. A connection needs both the source pod's egress and the destination pod's ingress to allow it.&lt;/p&gt;

&lt;p&gt;A common baseline per namespace is default deny in both directions, plus a DNS exception:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;networking.k8s.io/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;NetworkPolicy&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;default-deny&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;shop&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;podSelector&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;{}&lt;/span&gt;
  &lt;span class="na"&gt;policyTypes&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Ingress"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Egress"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
&lt;span class="nn"&gt;---&lt;/span&gt;
&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;networking.k8s.io/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;NetworkPolicy&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;allow-dns&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;shop&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;podSelector&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;{}&lt;/span&gt;
  &lt;span class="na"&gt;policyTypes&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Egress"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
  &lt;span class="na"&gt;egress&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;to&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;namespaceSelector&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;matchLabels&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;kubernetes.io/metadata.name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;kube-system&lt;/span&gt;
      &lt;span class="na"&gt;podSelector&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;matchLabels&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;k8s-app&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;kube-dns&lt;/span&gt;
    &lt;span class="na"&gt;ports&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;protocol&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;UDP&lt;/span&gt;
      &lt;span class="na"&gt;port&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;53&lt;/span&gt;
    &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;protocol&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;TCP&lt;/span&gt;
      &lt;span class="na"&gt;port&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;53&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Default-deny egress also blocks queries to CoreDNS, so without the second policy, lookups through the kube-dns Service fail across the namespace. With NodeLocal DNSCache (question 9), the DNS rule needs adjusting. The control plane sets the &lt;code&gt;kubernetes.io/metadata.name&lt;/code&gt; label on every namespace and it cannot be changed, which makes it the safe way to select a namespace by name. This &lt;a href="https://devopsinterview.cloud/blog/kubernetes-multi-tenancy-shared-cluster-design" rel="noopener noreferrer"&gt;multi-tenant cluster design&lt;/a&gt; shows the same pattern across many teams.&lt;/p&gt;

&lt;h3&gt;
  
  
  6. Which NetworkPolicy mistakes show up in production?
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The AND/OR YAML trap.&lt;/strong&gt; A &lt;code&gt;namespaceSelector&lt;/code&gt; and a &lt;code&gt;podSelector&lt;/code&gt; in the same &lt;code&gt;from&lt;/code&gt; entry mean "pods with this label in these namespaces". Put a dash in front of the &lt;code&gt;podSelector&lt;/code&gt; and they become two entries: any pod in those namespaces, or matching pods in the policy's own namespace. &lt;code&gt;kubectl describe networkpolicy&lt;/code&gt; shows how Kubernetes parsed it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Expecting to target a Service.&lt;/strong&gt; Policies select pods and namespaces by label, never Services by name, so changing the labels in a pod template quietly changes who can reach those pods.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Assuming features the API lacks.&lt;/strong&gt; Core NetworkPolicy has no logging of dropped connections, no explicit deny and no cluster-wide default. Calico and Cilium add their own policy resources for those.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The free question on &lt;a href="https://devopsinterview.cloud/questions/network-policies-stop-lateral-movement-in-3-rules" rel="noopener noreferrer"&gt;network policies that stop lateral movement&lt;/a&gt; walks through the security reasoning.&lt;/p&gt;

&lt;h2&gt;
  
  
  Ingress vs Gateway API
&lt;/h2&gt;

&lt;h3&gt;
  
  
  7. Should a new cluster use Ingress or Gateway API?
&lt;/h3&gt;

&lt;p&gt;Gateway API. The Kubernetes docs recommend Gateway over Ingress and describe Ingress as frozen: it stays GA with no plans for removal, but gets no further changes. Ingress models host and path routing plus TLS. Anything more, such as header matching, traffic weighting or rewrites, lives in controller-specific annotations, which is what makes Ingress manifests hard to move between controllers.&lt;/p&gt;

&lt;p&gt;Gateway API puts those features in typed fields and splits ownership across resources:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;GatewayClass&lt;/strong&gt; names the controller that implements gateways, owned by the infrastructure or platform team.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Gateway&lt;/strong&gt; defines listeners (ports, hostnames, TLS certificates) and which namespaces may attach routes, owned by the cluster operator.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;HTTPRoute&lt;/strong&gt; and &lt;strong&gt;GRPCRoute&lt;/strong&gt; hold routing rules in the application team's namespace and attach to a Gateway through &lt;code&gt;parentRefs&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That split is also a security boundary. A Gateway only accepts routes from other namespaces if its owner allows them on the listener, and a route pointing at a Service in another namespace needs a ReferenceGrant in the target namespace. Gateway API ships as CRDs, so install them first, directly or through your implementation.&lt;/p&gt;

&lt;p&gt;The 2026 follow-up is ingress-nginx. In November 2025, SIG Network and the Security Response Committee announced that the community ingress-nginx controller would get only best-effort maintenance until March 2026, and no releases, bug fixes or security fixes after that. Existing installs keep running, which is exactly the risk: an unpatched edge proxy. A credible plan inventories the annotations on every Ingress, maps each to a Gateway API field or an implementation policy, runs the new gateway alongside the old one, then moves DNS. The kubernetes-sigs &lt;code&gt;ingress2gateway&lt;/code&gt; tool converts manifests as a starting point.&lt;/p&gt;

&lt;h2&gt;
  
  
  Kubernetes DNS failures
&lt;/h2&gt;

&lt;h3&gt;
  
  
  8. How does a pod resolve a Service name?
&lt;/h3&gt;

&lt;p&gt;With the default &lt;code&gt;ClusterFirst&lt;/code&gt; DNS policy, the kubelet writes each pod's &lt;code&gt;/etc/resolv.conf&lt;/code&gt; with the cluster DNS Service as the nameserver, a search list, and &lt;code&gt;ndots:5&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;nameserver 10.96.0.10
search shop.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;CoreDNS answers behind a Service still named &lt;code&gt;kube-dns&lt;/code&gt; for backward compatibility. A bare &lt;code&gt;payments&lt;/code&gt; resolves only within the pod's own namespace; from another namespace you need &lt;code&gt;payments.shop&lt;/code&gt; or the full &lt;code&gt;payments.shop.svc.cluster.local&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;ndots:5&lt;/code&gt; means any name with fewer than five dots is tried with each search domain before it is tried as written. For a glibc-based client, one lookup of &lt;code&gt;api.example.com&lt;/code&gt; can become eight queries: A and AAAA for three search-domain names that return NXDOMAIN, then the real one. Two fixes: write external names with a trailing dot (&lt;code&gt;api.example.com.&lt;/code&gt;), or lower &lt;code&gt;ndots&lt;/code&gt; for pods that mostly call external hosts:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;dnsConfig&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;options&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ndots&lt;/span&gt;
      &lt;span class="na"&gt;value&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;2"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;payments.shop&lt;/code&gt; still uses the search list with &lt;code&gt;ndots:2&lt;/code&gt;, but names with two or more dots are now tried as written first, so test anything that relies on the search list before rolling it out.&lt;/p&gt;

&lt;h3&gt;
  
  
  9. DNS lookups sometimes take five seconds. What is going on?
&lt;/h3&gt;

&lt;p&gt;Five seconds is the glibc resolver's default timeout, so the number says a query went unanswered and the client waited before retrying. The usual causes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The conntrack race.&lt;/strong&gt; The long-running Kubernetes issue "DNS intermittent delays of 5s" traced back to races in Linux connection tracking: a client sends its A and AAAA queries from the same UDP socket at the same moment, through the DNAT to the DNS Service, and one gets dropped. Kernel fixes narrowed the race but did not remove every variant.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A full conntrack table.&lt;/strong&gt; UDP entries linger until they time out (30 seconds by default), and a busy node logs &lt;code&gt;nf_conntrack: table full, dropping packet&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;CoreDNS under-provisioned.&lt;/strong&gt; Too few replicas for the query rate, which &lt;code&gt;ndots:5&lt;/code&gt; multiplies, CPU throttling from tight limits, or slow upstream resolvers. CoreDNS's own Prometheus metrics show request rates, latency and errors.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For the conntrack causes, the Kubernetes docs point to NodeLocal DNSCache, a caching DaemonSet on every node. Pods query the cache on their own node, which skips DNAT and connection tracking on that hop, and the cache can send its misses to CoreDNS over TCP, whose conntrack entries disappear when the connection closes. The cache also takes load off CoreDNS, but capacity and upstream problems still need their own fixes. One catch: queries now go to an address on the node rather than to the CoreDNS pods, so check that egress policies still allow them. The free question on &lt;a href="https://devopsinterview.cloud/questions/kubernetes-dns-not-resolving-troubleshoot-coredns" rel="noopener noreferrer"&gt;Kubernetes DNS not resolving&lt;/a&gt; covers the CoreDNS checks.&lt;/p&gt;

&lt;h2&gt;
  
  
  Kubernetes networking scenario based interview questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  10. Calls from checkout to the payments Service time out. Walk me through it.
&lt;/h3&gt;

&lt;p&gt;Start with the error. A resolution failure points at DNS or a missing namespace in the name. Connection refused means something answered with a reset, often a &lt;code&gt;targetPort&lt;/code&gt; no container listens on. A timeout means packets are dropped somewhere: by policy, by the pod network, or at backends that never answer.&lt;/p&gt;

&lt;p&gt;Then test from the caller itself. A throwaway pod has different labels, so NetworkPolicy treats it differently; an ephemeral container shares the checkout pod's network namespace and sees what it sees:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Service IP, ports and selector; then each endpoint's conditions.ready&lt;/span&gt;
kubectl &lt;span class="nt"&gt;-n&lt;/span&gt; shop get svc payments &lt;span class="nt"&gt;-o&lt;/span&gt; yaml
kubectl &lt;span class="nt"&gt;-n&lt;/span&gt; shop get endpointslices &lt;span class="nt"&gt;-l&lt;/span&gt; kubernetes.io/service-name&lt;span class="o"&gt;=&lt;/span&gt;payments &lt;span class="nt"&gt;-o&lt;/span&gt; yaml

&lt;span class="c"&gt;# From inside the checkout pod: the ClusterIP first, then one pod IP directly&lt;/span&gt;
kubectl &lt;span class="nt"&gt;-n&lt;/span&gt; web debug &lt;span class="nt"&gt;-it&lt;/span&gt; checkout-7d9f8c6b5-x2x4q &lt;span class="nt"&gt;--image&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;busybox:1.36 &lt;span class="nt"&gt;--&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  wget &lt;span class="nt"&gt;-qO-&lt;/span&gt; &lt;span class="nt"&gt;-T&lt;/span&gt; 3 http://10.96.45.12:8080/healthz
kubectl &lt;span class="nt"&gt;-n&lt;/span&gt; web debug &lt;span class="nt"&gt;-it&lt;/span&gt; checkout-7d9f8c6b5-x2x4q &lt;span class="nt"&gt;--image&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;busybox:1.36 &lt;span class="nt"&gt;--&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  wget &lt;span class="nt"&gt;-qO-&lt;/span&gt; &lt;span class="nt"&gt;-T&lt;/span&gt; 3 http://10.0.42.17:8080/healthz
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Read the results as a decision tree:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;No ready endpoints.&lt;/strong&gt; The selector doesn't match the pod labels, or readiness probes are failing. Find out why before going further.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The pod IP works and the ClusterIP doesn't.&lt;/strong&gt; Check that the Service's &lt;code&gt;port&lt;/code&gt; and &lt;code&gt;targetPort&lt;/code&gt; map to the port the container listens on. If they do, suspect the Service data plane on the caller's node: kube-proxy or its replacement, and its logs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Neither works from checkout, both work from a pod in &lt;code&gt;shop&lt;/code&gt;.&lt;/strong&gt; Check NetworkPolicy first, on checkout's egress and payments' ingress, then compare which nodes each test ran from.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It fails only when the two pods are on different nodes.&lt;/strong&gt; The pod network between nodes: overlay traffic blocked by a cloud firewall or security group, or an MTU mismatch where small requests succeed and large responses hang.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;For ephemeral containers on images with no shell, see this &lt;a href="https://devopsinterview.cloud/blog/kubernetes-production-debugging-walkthrough" rel="noopener noreferrer"&gt;Kubernetes production debugging walkthrough&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Interview questions this covers
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;How does a Kubernetes Service route traffic to pods, and what does kube-proxy actually do?&lt;/li&gt;
&lt;li&gt;When would you move kube-proxy to nftables mode, or replace it with Cilium's eBPF data plane?&lt;/li&gt;
&lt;li&gt;How do you roll out a default-deny NetworkPolicy without breaking DNS?&lt;/li&gt;
&lt;li&gt;Should a new cluster use Ingress or Gateway API, and what replaces ingress-nginx?&lt;/li&gt;
&lt;li&gt;Calls to a Service in another namespace time out. How do you troubleshoot it?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For more practice, browse &lt;a href="https://devopsinterview.cloud/ebooks" rel="noopener noreferrer"&gt;all the DevOps interview ebooks&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Platform interviews go further: Calico tenant isolation that still allows shared services, intermittent ClusterIP failures after enabling IPv6, and when a service mesh replaces an API gateway. &lt;a href="https://devopsinterview.cloud/ebooks/container-orchestration-journey" rel="noopener noreferrer"&gt;Containers &amp;amp; Orchestration: Kubernetes Interview Questions&lt;/a&gt; works through 52 senior-level questions with worked answers, including a chapter on networking and service mesh. It is one of the five books in the &lt;a href="https://devopsinterview.cloud/ebooks/complete-devops-mastery-bundle" rel="noopener noreferrer"&gt;DevOps Interview Mastery bundle&lt;/a&gt;, and every purchase includes the free Interview-Day Playbook with a 7-day and 24-hour prep plan and a 30-second answer framework.&lt;/p&gt;

&lt;p&gt;Prefer to watch or listen? Answers like these are walked through on &lt;a href="https://www.youtube.com/@DevOpsInterviewCloud" rel="noopener noreferrer"&gt;YouTube&lt;/a&gt; and on the podcast, on &lt;a href="https://open.spotify.com/show/033txr35oY2Rcn6H4CVZ4S" rel="noopener noreferrer"&gt;Spotify&lt;/a&gt; and &lt;a href="https://music.amazon.com/podcasts/49554d87-91cb-4816-9420-cdac8a06ac75/devops-cloud-interview-prep-real-scenarios-answers" rel="noopener noreferrer"&gt;Amazon Music&lt;/a&gt;.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://devopsinterview.cloud/blog/kubernetes-networking-interview-questions" rel="noopener noreferrer"&gt;devopsinterview.cloud&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>interview</category>
      <category>kubernetes</category>
      <category>networking</category>
    </item>
    <item>
      <title>MLOps Interview Questions for DevOps and Platform Engineers</title>
      <dc:creator>DevOpsInterview.Cloud</dc:creator>
      <pubDate>Fri, 06 Nov 2026 09:20:03 +0000</pubDate>
      <link>https://dev.to/devopsinterview_cloud/mlops-interview-questions-for-devops-and-platform-engineers-43nl</link>
      <guid>https://dev.to/devopsinterview_cloud/mlops-interview-questions-for-devops-and-platform-engineers-43nl</guid>
      <description>&lt;p&gt;MLOps interview questions for DevOps and platform engineers usually start from what you already know. You have built CI/CD pipelines, so the interviewer asks how an ML pipeline is different, then keeps pushing until the answer stops sounding like application delivery with a GPU attached. The questions below follow that path, from the pipeline itself through registries, feature stores and drift to gating a model release.&lt;/p&gt;

&lt;h2&gt;
  
  
  MLOps interview questions on ML pipelines vs app CI/CD
&lt;/h2&gt;

&lt;h3&gt;
  
  
  1. What changes when the artifact is a model instead of a binary?
&lt;/h3&gt;

&lt;p&gt;Three things. The artifact is the product of code, data and configuration together, so the same commit trained on a newer data snapshot gives a different model. The tests are statistical: a model gets a score, and someone has to decide which score is good enough. And a model can get worse in production while nothing about it changes, because the world moves away from the data it learned from.&lt;/p&gt;

&lt;p&gt;Google's &lt;a href="https://docs.cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning" rel="noopener noreferrer"&gt;MLOps architecture guide&lt;/a&gt; spells out the consequences: CI also tests and validates data, data schemas and models; CD ships a training pipeline that deploys another service; and continuous training (CT), retraining and serving models automatically, is unique to ML systems.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;App CI/CD&lt;/th&gt;
&lt;th&gt;ML pipeline&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;What gets versioned&lt;/td&gt;
&lt;td&gt;Code and dependencies&lt;/td&gt;
&lt;td&gt;Code, data snapshot, config, hyperparameters&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;What starts a run&lt;/td&gt;
&lt;td&gt;A commit&lt;/td&gt;
&lt;td&gt;A commit, new data, a schedule or a monitoring alert&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;What the tests check&lt;/td&gt;
&lt;td&gt;Behaviour is correct&lt;/td&gt;
&lt;td&gt;Data is valid and model scores clear a bar&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;What gets deployed&lt;/td&gt;
&lt;td&gt;A service&lt;/td&gt;
&lt;td&gt;A model, and often the pipeline that retrains it&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;How it fails in production&lt;/td&gt;
&lt;td&gt;Errors and latency&lt;/td&gt;
&lt;td&gt;Errors and latency, plus quiet accuracy loss&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The free &lt;a href="https://devopsinterview.cloud/questions/mlops-vs-devops-what-s-different" rel="noopener noreferrer"&gt;MLOps vs DevOps question&lt;/a&gt; is the short version of this answer.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. What does reproducible training actually require?
&lt;/h3&gt;

&lt;p&gt;Being able to rebuild a model from its record. That means pinning the code commit, an immutable data snapshot (a DVC-tracked file, a lakeFS commit, a Delta Lake or Iceberg table version), the container image by digest, the hyperparameters and the random seeds. A tracking server such as MLflow logs parameters, metrics and artifacts per run, so the registry entry can point back to the exact run that produced it.&lt;/p&gt;

&lt;p&gt;The caveat: bit-for-bit identical weights are often out of reach. The PyTorch docs say completely reproducible results are not guaranteed across releases, individual commits or platforms, so the realistic goal is a rebuilt model that scores within tolerance of the original.&lt;/p&gt;

&lt;h2&gt;
  
  
  Training vs serving
&lt;/h2&gt;

&lt;h3&gt;
  
  
  3. Why do training and serving run as separate systems?
&lt;/h3&gt;

&lt;p&gt;Because they want opposite things. Training is a batch job that wants throughput and large GPUs, and with checkpoints it survives interruption, so it can run on spot capacity. Serving is a latency-bound service that runs all the time, scales with traffic, and often fits on smaller accelerators or CPU.&lt;/p&gt;

&lt;p&gt;On Kubernetes, training runs as jobs, often through Kubeflow Trainer or a RayJob, with Kueue in front: it manages quotas and decides when a job waits, is admitted or is preempted. Its timeout-based all-or-nothing mode evicts and requeues a distributed job whose pods don't all start, instead of letting it sit on half the GPUs it asked for. Serving runs as Deployments or a KServe InferenceService behind an autoscaler. Share one node pool without quotas and a large training job can starve serving. GPU scheduling and sharing have their own set of questions on the &lt;a href="https://devopsinterview.cloud/ai-infrastructure-interview-questions" rel="noopener noreferrer"&gt;AI infrastructure interview questions page&lt;/a&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. What is training-serving skew, and how do you prevent it?
&lt;/h3&gt;

&lt;p&gt;Skew is the model seeing features at serving time computed differently from the ones it trained on. The classic cause is two implementations: a Spark or SQL job computes "orders in the last 30 days" for training, and the online service recomputes it in application code with a different time zone, window boundary or null handling. Nothing errors. The model just gets worse.&lt;/p&gt;

&lt;p&gt;Ways to prevent it, cheapest first:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Log the exact feature values the service used for each prediction, and train the next model on those logs.&lt;/li&gt;
&lt;li&gt;Define each feature once and serve that definition to both training and inference, which is the job of a feature store.&lt;/li&gt;
&lt;li&gt;Compare each feature's training distribution with what serving sees.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A related trap is leakage through time: a training row for a prediction made on 1 March may only use feature values known on 1 March. Get it wrong and offline scores look better than production ever will.&lt;/p&gt;

&lt;h2&gt;
  
  
  Model registry and feature store questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  5. What does a model registry give you that an S3 bucket doesn't?
&lt;/h3&gt;

&lt;p&gt;A bucket stores files. A registry stores versions with meaning attached: the run that produced each one, its evaluation results, its input and output signature, who approved it, and which version is live.&lt;/p&gt;

&lt;p&gt;In MLflow, an alias is a mutable, named reference to one model version. Serving loads &lt;code&gt;models:/fraud-scorer@champion&lt;/code&gt; instead of a hard-coded version number. The alias resolves when the model loads, so promotion means moving the alias and then rolling or reloading the servers, and rollback is the same in reverse. MLflow deprecated the older Staging and Production stages in favour of aliases and tags. Whatever registry you use, versions should be immutable: a new model is a new version, never an overwritten file.&lt;/p&gt;

&lt;h3&gt;
  
  
  6. When is a feature store worth it, and when is it overkill?
&lt;/h3&gt;

&lt;p&gt;A feature store has two halves. The offline store holds historical values for building training sets; the online store holds only the latest value per entity for low-latency lookups at serving time. In Feast, &lt;code&gt;feast materialize&lt;/code&gt; loads values from offline into online, and &lt;code&gt;get_historical_features&lt;/code&gt; builds a training set with a point-in-time join: each row gets the latest feature value whose event timestamp isn't later than the row's, which keeps future values out. Backfills and corrections can still leak in, because they carry old event timestamps, so reproducing what serving actually knew also needs the time each value was written.&lt;/p&gt;

&lt;p&gt;It earns its place when several models share features, when online serving needs the same features training used, and when you have to say which feature definitions a model trained on. It's overkill for a single model scored in batch from one warehouse query: there's no online path to skew against. The free &lt;a href="https://devopsinterview.cloud/questions/feature-stores-3-reasons-your-ml-pipeline-needs-one" rel="noopener noreferrer"&gt;feature store question&lt;/a&gt; covers the build-or-buy follow-up.&lt;/p&gt;

&lt;h2&gt;
  
  
  Drift detection and retraining questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  7. Data drift, concept drift and prediction drift: what's the difference?
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Data drift&lt;/strong&gt; (covariate shift): the distribution of inputs changes. A new customer segment arrives, or an upstream team changes a unit.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Concept drift&lt;/strong&gt;: the relationship between inputs and the correct answer changes. Fraudsters adapt to your model, so the same transaction pattern now means something else.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Prediction drift&lt;/strong&gt;: the distribution of the model's outputs changes, which is often the first thing you can see.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Data and prediction drift only need the inputs and outputs you already log. Concept drift needs ground-truth labels, which often arrive late: a chargeback can land weeks after the transaction. NannyML estimates performance before labels arrive, using the model's confidence scores for classifiers, but its docs say those methods don't work under concept drift, the case where you most need labels.&lt;/p&gt;

&lt;h3&gt;
  
  
  8. How do you detect drift without drowning in false alarms?
&lt;/h3&gt;

&lt;p&gt;Compare each feature between a reference window, such as the training data, and a recent window. Kolmogorov-Smirnov for numeric features and chi-squared for categorical ones are the textbook tests. The trap is sample size: with millions of rows, a p-value test flags differences far too small to matter. &lt;a href="https://docs.evidentlyai.com/metrics/explainer_drift" rel="noopener noreferrer"&gt;Evidently's defaults&lt;/a&gt; reflect this: when the reference dataset has up to 1,000 observations it uses statistical tests such as K-S and chi-squared, and above that it switches to Wasserstein distance and Jensen-Shannon divergence, with a threshold on the distance instead of a p-value. Beyond the choice of test:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Weight alerts by feature importance. Drift in a feature the model barely uses rarely matters.&lt;/li&gt;
&lt;li&gt;Alert on drift that persists across several windows, not one noisy hour.&lt;/li&gt;
&lt;li&gt;Check data quality first. A lot of "drift" is a broken pipeline: a column of nulls, a renamed category, a changed unit.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  9. Should a model retrain on a schedule, on drift, or on performance?
&lt;/h3&gt;

&lt;p&gt;Each has a failure mode. A schedule retrains when nothing changed and waits when something did. A drift trigger reacts to changed inputs, but drift doesn't always hurt accuracy. A performance trigger is the best signal and the slowest, because it waits for labels.&lt;/p&gt;

&lt;p&gt;A defensible design keeps a schedule as the baseline, lets drift and performance alerts start the pipeline early, and sends every retrained model through the same release gate. Retraining is not deploying. Watch for models that shape the data they are later trained on. A recommender only collects clicks on what it chose to show, so its training data needs some exploration traffic or a holdout group.&lt;/p&gt;

&lt;h2&gt;
  
  
  Gating a model release: scenario-based questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  10. How do you gate a model release the way you gate code?
&lt;/h3&gt;

&lt;p&gt;With checks that run in the pipeline and fail closed, roughly in this order:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Data validation before training&lt;/strong&gt;: schema, null rates, value ranges and row counts against the previous snapshot. Great Expectations and TensorFlow Data Validation are common tools.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Artifact safety, before anything loads the model&lt;/strong&gt;. Pickle-based files can run code when they load, which is why PyTorch 2.6 made &lt;code&gt;weights_only=True&lt;/code&gt; the default for &lt;code&gt;torch.load&lt;/code&gt;. Prefer safetensors, and load pickle files from outside the pipeline only in an isolated sandbox: a scanner catches known patterns, not every payload.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Offline evaluation&lt;/strong&gt; on a fixed, versioned holdout set, compared with the current production model on the same set.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Slice metrics&lt;/strong&gt;. An overall improvement can hide a regression for one region, device type or customer segment.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Serving checks&lt;/strong&gt;: the signature matches the serving contract, the artifact loads in the serving image, and latency and memory fit the target hardware.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Human approval&lt;/strong&gt; where the model makes decisions with legal or financial weight.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;It is the same principle as the gates in a &lt;a href="https://devopsinterview.cloud/blog/devsecops-pipeline-from-scratch" rel="noopener noreferrer"&gt;DevSecOps pipeline built from scratch&lt;/a&gt;: the pipeline decides, and nobody promotes around it by hand. If both runs log their score on the same versioned holdout set, the comparison step is a short script against the registry:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;mlflow&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;MlflowClient&lt;/span&gt;

&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;MlflowClient&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;NAME&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;METRIC&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;fraud-scorer&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;holdout_auc&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="n"&gt;TOLERANCE&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mf"&gt;0.002&lt;/span&gt;  &lt;span class="c1"&gt;# agreed with the model owner
&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;score&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;alias&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;version&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get_model_version_by_alias&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;NAME&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;alias&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;value&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get_run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;version&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;run_id&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;metrics&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;METRIC&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="mf"&gt;0.0&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="mf"&gt;1.0&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;  &lt;span class="c1"&gt;# also rejects NaN
&lt;/span&gt;        &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;SystemExit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;alias&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; v&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;version&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;version&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;: no valid &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;METRIC&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;version&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt;

&lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;champion_score&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;score&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;champion&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;challenger&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;challenger_score&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;score&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;challenger&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;challenger_score&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;champion_score&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;TOLERANCE&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;SystemExit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;METRIC&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; regressed; challenger v&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;challenger&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;version&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; blocked&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set_registered_model_alias&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;NAME&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;canary&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;challenger&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;version&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Passing only moves the canary alias. The deploy step that follows rolls the canary servers onto that version, and it becomes champion after it survives live traffic.&lt;/p&gt;

&lt;h3&gt;
  
  
  11. Shadow, canary or A/B test: which do you use for a new model?
&lt;/h3&gt;

&lt;p&gt;Often all three, in that order, because they answer different questions.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Shadow&lt;/strong&gt;: copy live requests to the new model and discard its responses. You see its latency, errors and how its predictions differ from the current model's, with no user exposure, but nothing about outcomes. Istio's traffic mirroring works this way: mirrored requests are fire and forget. Make sure the shadow path has no side effects, such as writing decisions to a database.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Canary&lt;/strong&gt;: send a small share of real traffic to the new model and roll back on errors, latency and whatever quality signal you have. KServe splits the traffic through &lt;code&gt;canaryTrafficPercent&lt;/code&gt; on an InferenceService (&lt;a href="https://kserve.github.io/website/docs/model-serving/predictive-inference/rollout-strategies/canary-example" rel="noopener noreferrer"&gt;only in serverless deployment mode&lt;/a&gt;), but promotion and rollback are changes you apply, so the automatic part has to come from your pipeline or a controller watching the metrics. Without KServe, Argo Rollouts (a Rollout resource with an analysis template, in place of the Deployment or referencing it) or Flagger (a Canary resource that targets the Deployment) handles both the split and the metric-driven rollback.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A/B test&lt;/strong&gt;: a randomized experiment with enough traffic and time to measure a business metric with statistical confidence. A canary can be run that way too, but a traffic percentage on its own shows the model is safe to run, not that it is better for the business.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The general mechanics are in the free &lt;a href="https://devopsinterview.cloud/questions/canary-deployments-test-in-production-safely" rel="noopener noreferrer"&gt;canary deployment question&lt;/a&gt;. What's specific to ML is that "healthy" includes prediction quality, which HTTP metrics won't show.&lt;/p&gt;

&lt;h3&gt;
  
  
  12. A new model is live and conversion drops. How do you roll back?
&lt;/h3&gt;

&lt;p&gt;Rollback should be a pointer change: point serving back at the previous version, which is still in the registry and ideally still warm. In KServe's canary flow, setting &lt;code&gt;canaryTrafficPercent&lt;/code&gt; to 0 sends all traffic back to the previous revision.&lt;/p&gt;

&lt;p&gt;Then come the parts that make ML rollback harder than app rollback:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Features.&lt;/strong&gt; If the new model needed a changed feature definition, the old model needs the old one. Add new features alongside old ones instead of changing them in place.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Predictions already written.&lt;/strong&gt; Batch scores and stored decisions came from the bad model. Log the model version with every prediction so you can find and recompute them.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The retraining loop.&lt;/strong&gt; If the bad model's outputs feed the next training set, exclude that window, or the next retrain learns from them.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  How would you run an MLOps platform on Kubernetes?
&lt;/h2&gt;

&lt;p&gt;A reasonable design, layer by layer:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Layer&lt;/th&gt;
&lt;th&gt;Common choice&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Pipeline orchestration&lt;/td&gt;
&lt;td&gt;Kubeflow Pipelines or Argo Workflows&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Training jobs and GPU quota&lt;/td&gt;
&lt;td&gt;Kubeflow Trainer or RayJob, queued through Kueue&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Experiment tracking and registry&lt;/td&gt;
&lt;td&gt;MLflow, with a database and object storage behind it&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Feature serving&lt;/td&gt;
&lt;td&gt;Feast, or a managed feature store&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Model serving&lt;/td&gt;
&lt;td&gt;KServe, or a model server behind a Deployment&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Drift and quality monitoring&lt;/td&gt;
&lt;td&gt;Evidently or NannyML jobs, alerting through the existing stack&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The interviewer cares less about the logos than the boundaries: separate node pools and quotas for training and serving, data scientists submitting work through a pipeline instead of writing manifests, and model promotions recorded like code changes. The Kubernetes basics are in &lt;a href="https://devopsinterview.cloud/blog/kubernetes-interview-questions-2026" rel="noopener noreferrer"&gt;25 Kubernetes interview questions for 2026&lt;/a&gt;, and sharing one cluster between teams is covered in &lt;a href="https://devopsinterview.cloud/blog/kubernetes-multi-tenancy-shared-cluster-design" rel="noopener noreferrer"&gt;designing a multi-tenant Kubernetes cluster&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Interview questions this covers
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;How is an ML pipeline different from a normal CI/CD pipeline?&lt;/li&gt;
&lt;li&gt;What is training-serving skew, and how would you prevent it?&lt;/li&gt;
&lt;li&gt;How do you detect model drift, and when should it trigger retraining?&lt;/li&gt;
&lt;li&gt;How would you gate a model release before it reaches users?&lt;/li&gt;
&lt;li&gt;A new model hurts a business metric after release. How do you roll it back?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For more practice, browse &lt;a href="https://devopsinterview.cloud/ebooks" rel="noopener noreferrer"&gt;all the DevOps interview ebooks&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;None of the books covers AI infrastructure end to end, but they cover the platform underneath it. &lt;a href="https://devopsinterview.cloud/ebooks/container-orchestration-journey" rel="noopener noreferrer"&gt;Containers &amp;amp; Orchestration: Kubernetes Interview Questions&lt;/a&gt; works through 52 senior-level questions with worked answers, including how you would build an MLOps platform on Kubernetes so data scientists can train and serve models without writing YAML. It is one of the five books in the &lt;a href="https://devopsinterview.cloud/ebooks/complete-devops-mastery-bundle" rel="noopener noreferrer"&gt;DevOps Interview Mastery bundle&lt;/a&gt;, which also includes CI/CD &amp;amp; GitOps and its question on building a pipeline that trains, validates and ships a model and gates a model release the way you gate code. Every purchase includes the free Interview-Day Playbook with a 7-day and 24-hour prep plan and a 30-second answer framework. If a full AI infrastructure book goes ahead, the &lt;a href="https://devopsinterview.cloud/ai-infrastructure-interview-questions" rel="noopener noreferrer"&gt;AI infrastructure interview questions page&lt;/a&gt; is where to get the one email about it.&lt;/p&gt;

&lt;p&gt;Prefer to watch or listen? Answers like these are walked through on &lt;a href="https://www.youtube.com/@DevOpsInterviewCloud" rel="noopener noreferrer"&gt;YouTube&lt;/a&gt; and on the podcast, on &lt;a href="https://open.spotify.com/show/033txr35oY2Rcn6H4CVZ4S" rel="noopener noreferrer"&gt;Spotify&lt;/a&gt; and &lt;a href="https://music.amazon.com/podcasts/49554d87-91cb-4816-9420-cdac8a06ac75/devops-cloud-interview-prep-real-scenarios-answers" rel="noopener noreferrer"&gt;Amazon Music&lt;/a&gt;.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://devopsinterview.cloud/blog/mlops-interview-questions" rel="noopener noreferrer"&gt;devopsinterview.cloud&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>career</category>
      <category>devops</category>
      <category>interview</category>
      <category>machinelearning</category>
    </item>
    <item>
      <title>AWS VPC Interview Questions for Experienced Engineers</title>
      <dc:creator>DevOpsInterview.Cloud</dc:creator>
      <pubDate>Thu, 05 Nov 2026 09:20:03 +0000</pubDate>
      <link>https://dev.to/devopsinterview_cloud/aws-vpc-interview-questions-for-experienced-engineers-pc0</link>
      <guid>https://dev.to/devopsinterview_cloud/aws-vpc-interview-questions-for-experienced-engineers-pc0</guid>
      <description>&lt;p&gt;AWS VPC interview questions at the senior level rarely stop at "what is a subnet". The interviewer wants to know whether you can trace a packet: which route table it hits, which gateway it leaves through, which firewall layer drops it, and what the bill looks like when it gets there. The questions below run from addressing and routing through egress, endpoints and hybrid connectivity to a private-subnet troubleshooting scenario.&lt;/p&gt;

&lt;h2&gt;
  
  
  AWS VPC interview questions on subnets and routing
&lt;/h2&gt;

&lt;h3&gt;
  
  
  1. How do you choose CIDR ranges for a new VPC?
&lt;/h3&gt;

&lt;p&gt;Start from the organisation, not the VPC. Any range may one day need routing to another VPC or a data centre, and neither peering nor a plain Transit Gateway route connects overlapping ranges. So begin with an address plan kept in one central record, such as Amazon VPC IPAM.&lt;/p&gt;

&lt;p&gt;Then size the subnets. An IPv4 subnet can be anywhere from /28 to /16, and AWS reserves five addresses in each: the first four (network, VPC router, DNS, future use) and the last. A /28 leaves 11 usable addresses. The follow-up to expect is EKS: with the default Amazon VPC CNI, every pod that does not use host networking takes its own subnet address, so a cluster can exhaust a /24 long before the nodes look busy.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. What actually makes a subnet public?
&lt;/h3&gt;

&lt;p&gt;Its route table. A subnet is public when its associated route table has a route to an internet gateway, usually for 0.0.0.0/0. For IPv4, an instance needs both the route and a public IPv4 address or Elastic IP, so an instance with only a private IPv4 address cannot reach the internet directly even in a public subnet. For IPv6, the instance needs its IPv6 address and a ::/0 route to the internet gateway.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. A subnet has no explicit route table association. What does it use?
&lt;/h3&gt;

&lt;p&gt;The main route table, which makes it a trap. If someone adds an internet gateway route to it, every subnet created later without an explicit association becomes public by accident. Leave the main route table with only the local route and associate every subnet explicitly.&lt;/p&gt;

&lt;p&gt;When several routes match, the VPC picks the most specific (longest prefix match). That is how a gateway endpoint coexists with a default route: the S3 prefix list is more specific than 0.0.0.0/0, so S3 traffic for that Region goes to the endpoint.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. Security groups or network ACLs: which one does the real work?
&lt;/h3&gt;

&lt;p&gt;Security groups, almost always. They are stateful, contain only allow rules, and can reference other security groups, so "the app tier may reach the database on 5432" is one rule that survives instances coming and going. Network ACLs are stateless, apply to a whole subnet, support allow and deny, and are evaluated in rule-number order, lowest first.&lt;/p&gt;

&lt;p&gt;Statelessness is what catches people. When an instance makes an outbound call, the NACL must also allow the reply to the client's ephemeral port. Many Linux kernels use 32768-61000, while NAT gateways, Elastic Load Balancing and Lambda use 1024-65535, which is why return-traffic rules usually open 1024-65535. Use NACLs as coarse guardrails and leave application policy to security groups.&lt;/p&gt;

&lt;h2&gt;
  
  
  NAT and internet egress
&lt;/h2&gt;

&lt;h3&gt;
  
  
  5. How do you design NAT for a VPC that spans three Availability Zones?
&lt;/h3&gt;

&lt;p&gt;For zonal NAT gateways the classic answer holds: one NAT gateway per AZ, each in a public subnet, with each private route table pointing 0.0.0.0/0 at the gateway in its own AZ. A single shared gateway saves money until its AZ fails and the healthy AZs lose outbound access too, and meanwhile you pay cross-AZ data transfer to reach it. The free &lt;a href="https://devopsinterview.cloud/questions/aws-vpc-design-subnets-nat-and-security-groups" rel="noopener noreferrer"&gt;VPC design question on subnets, NAT and security groups&lt;/a&gt; covers that baseline.&lt;/p&gt;

&lt;p&gt;The 2026 answer adds regional NAT gateways, launched in November 2025. One regional NAT gateway serves the whole VPC, needs no public subnet, and lets every private route table point at the same ID. In automatic mode it expands into an AZ when workloads appear there; in manual mode you manage the AZs and addresses yourself. Name the caveats: automatic expansion into a new AZ can take up to 60 minutes, during which that traffic is processed in another zone, and regional mode does not support private NAT.&lt;/p&gt;

&lt;h3&gt;
  
  
  6. Your NAT gateway costs more than the instances behind it. Where do you look?
&lt;/h3&gt;

&lt;p&gt;Split the bill first. NAT gateways charge for the hours they are provisioned and per gigabyte processed, so idle capacity costs money too, but when the processing line dominates, bytes are the problem. The usual suspects are AWS services rather than the internet: S3, DynamoDB, ECR image pulls and CloudWatch logs. Flow logs summed in both directions and grouped by the remote address (&lt;code&gt;pkt-dstaddr&lt;/code&gt; outbound, &lt;code&gt;pkt-srcaddr&lt;/code&gt; for replies) show the top talkers.&lt;/p&gt;

&lt;p&gt;The fixes are endpoints. S3 and DynamoDB gateway endpoints carry no additional charge and take that traffic off the NAT path. ECR pulls need interface endpoints for the ECR API and Docker registry, plus the S3 gateway endpoint, because ECR stores image layers in S3.&lt;/p&gt;

&lt;h3&gt;
  
  
  7. What does ErrorPortAllocation on a NAT gateway tell you?
&lt;/h3&gt;

&lt;p&gt;That the gateway could not allocate a source port because too many connections are open. Each NAT gateway IPv4 address supports up to 55,000 simultaneous connections to each unique destination (IP, port and protocol). A fleet opening a fresh connection per request to one API hits that early. Add IP addresses to the gateway (up to 8 on a zonal gateway, 32 per AZ on a regional one) or, usually better, make the client reuse connections.&lt;/p&gt;

&lt;h2&gt;
  
  
  VPC endpoints
&lt;/h2&gt;

&lt;h3&gt;
  
  
  8. Gateway endpoint or interface endpoint: how do you choose?
&lt;/h3&gt;

&lt;p&gt;Know where each one stops working:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Gateway endpoint&lt;/th&gt;
&lt;th&gt;Interface endpoint&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Services&lt;/td&gt;
&lt;td&gt;S3 and DynamoDB only&lt;/td&gt;
&lt;td&gt;Most AWS services, plus PrivateLink services from other accounts&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;How traffic gets there&lt;/td&gt;
&lt;td&gt;Route table entry using the service's prefix list&lt;/td&gt;
&lt;td&gt;DNS resolves to private IPs of endpoint network interfaces&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Cost&lt;/td&gt;
&lt;td&gt;No additional charge&lt;/td&gt;
&lt;td&gt;Per hour per AZ, plus per GB processed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Usable from on-premises, peered VPCs or a transit gateway&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;Yes, with DNS set up for it&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Access controls&lt;/td&gt;
&lt;td&gt;Endpoint policy, route table associations&lt;/td&gt;
&lt;td&gt;Security groups, plus endpoint policies where the service supports them&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;For services that support private DNS, enabling it on the endpoint (with &lt;code&gt;enableDnsSupport&lt;/code&gt; and &lt;code&gt;enableDnsHostnames&lt;/code&gt; on the VPC) makes AWS create a hidden private hosted zone, so the service's usual hostname resolves to the endpoint and existing SDK calls need no changes. Without it, the hostname resolves to public addresses and the route table decides the path, which is usually NAT. S3 can scope private DNS to queries from an inbound Resolver endpoint, so on-premises clients use the interface endpoint while VPC traffic stays on the free gateway endpoint. Endpoints are also what let &lt;a href="https://devopsinterview.cloud/questions/aws-ssm-vs-ssh-stop-opening-port-22" rel="noopener noreferrer"&gt;Session Manager replace SSH&lt;/a&gt; for private instances with no NAT.&lt;/p&gt;

&lt;h3&gt;
  
  
  9. You added an S3 gateway endpoint and something broke. What happened?
&lt;/h3&gt;

&lt;p&gt;Several things change at once:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Bucket policies that rely on &lt;code&gt;aws:SourceIp&lt;/code&gt; break, because that key is not available for requests through a VPC endpoint. Restrict by endpoint or VPC with &lt;code&gt;aws:SourceVpce&lt;/code&gt; or &lt;code&gt;aws:SourceVpc&lt;/code&gt;, and if client IPs matter, pair &lt;code&gt;aws:VpcSourceIp&lt;/code&gt; with one of them.&lt;/li&gt;
&lt;li&gt;A restrictive endpoint policy can block buckets nobody thought about, such as those AWS services use for agents or drivers.&lt;/li&gt;
&lt;li&gt;A gateway endpoint only covers buckets in its own Region; other Regions still go through NAT.&lt;/li&gt;
&lt;li&gt;Creating or modifying the endpoint drops open TCP connections to S3, so long transfers fail once and need to retry.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Connecting VPCs and on-premises networks
&lt;/h2&gt;

&lt;h3&gt;
  
  
  10. VPC peering, Transit Gateway or PrivateLink: how do you decide?
&lt;/h3&gt;

&lt;p&gt;They solve different problems:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;VPC peering&lt;/strong&gt; is a one-to-one link with no transit. If A peers with B and with C, B cannot reach C through A, nor use A's internet gateway, NAT gateway, VPN or gateway endpoint. CIDRs cannot overlap. A full mesh of n VPCs needs n(n-1)/2 connections, each with routes on both sides, which is why larger networks move to a hub.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Transit Gateway&lt;/strong&gt; is a regional router. VPCs, VPNs and Direct Connect gateways attach to it, its route tables decide who reaches whom, and it bills per attachment and per gigabyte. It routes on destination prefixes too, so two VPCs with the same CIDR need NAT between them.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;PrivateLink&lt;/strong&gt; exposes one service, not a network. The provider fronts the service with a Network Load Balancer and an endpoint service; consumers create an interface endpoint in their own VPC. Only the consumer initiates connections, and overlapping CIDRs between the two VPCs do not matter.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Real networks mix them: Transit Gateway for layer-3 connectivity, PrivateLink for a shared internal API, peering where it is genuinely simpler. For organising the Terraform across those accounts, see this &lt;a href="https://devopsinterview.cloud/blog/terraform-mono-repo-20-aws-accounts" rel="noopener noreferrer"&gt;Terraform layout for 20 AWS accounts&lt;/a&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  11. How do you stop dev VPCs from reaching prod over a shared Transit Gateway?
&lt;/h3&gt;

&lt;p&gt;With Transit Gateway route tables. An attachment is associated with at most one TGW route table, which decides where traffic entering through it can go, and separately can propagate its routes into any number of tables, or none, which decides who can reach it. A common layout:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A prod table for prod attachments, with propagations from prod, shared services and on-premises.&lt;/li&gt;
&lt;li&gt;A dev table built the same way for dev.&lt;/li&gt;
&lt;li&gt;A shared-services table that receives propagations from both.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Prod and dev never propagate into each other's tables, so there is no route, and blackhole routes stop a broad static route leaking through. Two follow-ups: each VPC still needs routes in its own subnet route tables pointing at the transit gateway, and an attachment for a stateful inspection appliance needs appliance mode so both directions of a flow stay in the same AZ.&lt;/p&gt;

&lt;h3&gt;
  
  
  12. How do you connect a data centre to AWS, and what happens when a link fails?
&lt;/h3&gt;

&lt;p&gt;Site-to-Site VPN is the fast start. Each connection has two tunnels in different AZs; configure both, because a tunnel can be down for maintenance. A standard tunnel carries up to 1.25 Gbps; Large Bandwidth Tunnels reach 5 Gbps on VPNs attached to a transit gateway or Cloud WAN. Direct Connect gives a dedicated private path, but it does not encrypt traffic by default: add MACsec where the connection supports it, or run IPsec VPN over it. A common pattern is Direct Connect primary and VPN backup, with BGP choosing between them and a scheduled failover test.&lt;/p&gt;

&lt;p&gt;DNS is the part people forget. The Route 53 Resolver cannot be queried from outside the VPC, so hybrid DNS uses Resolver endpoints: inbound for on-premises clients resolving private names, outbound with forwarding rules for VPC workloads resolving on-premises domains.&lt;/p&gt;

&lt;h2&gt;
  
  
  AWS VPC scenario based interview questions: debugging a broken path
&lt;/h2&gt;

&lt;h3&gt;
  
  
  13. An instance in a private subnet can't reach an external API. Walk me through it.
&lt;/h3&gt;

&lt;p&gt;Name the failure first. A DNS error points at resolution, although reaching a custom resolver is itself a network path. A refused connection means a packet reached something that answered with a reset. A connect timeout means the SYN got no answer: dropped, never routed, or sent to a host too overloaded to reply, and the VPC layers are the first suspects. The free question on &lt;a href="https://devopsinterview.cloud/questions/connection-refused-vs-timeout-different-fixes" rel="noopener noreferrer"&gt;connection refused vs timeout&lt;/a&gt; covers that split. It is the same discipline as reading the exit code first in this &lt;a href="https://devopsinterview.cloud/blog/kubernetes-production-debugging-walkthrough" rel="noopener noreferrer"&gt;Kubernetes production debugging walkthrough&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Then check each layer:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The subnet's route table, explicit or main. Does 0.0.0.0/0 point at a NAT gateway that serves this AZ? A route whose NAT gateway was deleted shows a &lt;code&gt;blackhole&lt;/code&gt; state.&lt;/li&gt;
&lt;li&gt;The NAT gateway. Is it &lt;code&gt;available&lt;/code&gt;? For a zonal gateway, does its own subnet route 0.0.0.0/0 to an internet gateway?&lt;/li&gt;
&lt;li&gt;The instance's security group outbound rules. Replies are allowed automatically.&lt;/li&gt;
&lt;li&gt;The network ACLs on the instance's subnet and, for a zonal gateway, the NAT gateway's subnet, in both directions, including inbound ephemeral ports.&lt;/li&gt;
&lt;li&gt;The far end. Third-party APIs often allowlist source IPs, and the request arrives from the NAT gateway's public address.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;These commands cover step 1 and the gateway's state without the console:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Routes in the subnet's explicitly associated route table&lt;/span&gt;
aws ec2 describe-route-tables &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--filters&lt;/span&gt; &lt;span class="nv"&gt;Name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;association.subnet-id,Values&lt;span class="o"&gt;=&lt;/span&gt;subnet-0abc1234 &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--query&lt;/span&gt; &lt;span class="s1"&gt;'RouteTables[].Routes[]'&lt;/span&gt;

&lt;span class="c"&gt;# Empty output above means the subnet uses the main route table&lt;/span&gt;
aws ec2 describe-route-tables &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--filters&lt;/span&gt; &lt;span class="nv"&gt;Name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;vpc-id,Values&lt;span class="o"&gt;=&lt;/span&gt;vpc-0123abcd &lt;span class="nv"&gt;Name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;association.main,Values&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nb"&gt;true&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--query&lt;/span&gt; &lt;span class="s1"&gt;'RouteTables[].Routes[]'&lt;/span&gt;

aws ec2 describe-nat-gateways &lt;span class="nt"&gt;--nat-gateway-ids&lt;/span&gt; nat-0def5678 &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--query&lt;/span&gt; &lt;span class="s1"&gt;'NatGateways[].[State,SubnetId]'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For a zonal gateway, run the route table query again with the NAT gateway's subnet ID to confirm its route to the internet gateway.&lt;/p&gt;

&lt;p&gt;Reachability Analyzer models your configuration rather than sending packets, and when the path is blocked it names the component responsible:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;aws ec2 create-network-insights-path &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--source&lt;/span&gt; i-0abc1234 &lt;span class="nt"&gt;--destination&lt;/span&gt; igw-0def5678 &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--protocol&lt;/span&gt; tcp &lt;span class="nt"&gt;--destination-port&lt;/span&gt; 443

aws ec2 start-network-insights-analysis &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--network-insights-path-id&lt;/span&gt; nip-0123abcd

aws ec2 describe-network-insights-analyses &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--network-insights-analysis-ids&lt;/span&gt; nia-0456efgh &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--query&lt;/span&gt; &lt;span class="s1"&gt;'NetworkInsightsAnalyses[].[NetworkPathFound,Explanations]'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Flow logs show what actually happened. The pattern worth recognising is an ACCEPT for the outbound request and a REJECT for the reply. Security groups allow replies on a tracked connection, so a network ACL is the first suspect; confirm it against the NACL rules, because packets arriving after a connection has closed can also be rejected. In the default format:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;2 123456789012 eni-0aaa1111 10.0.12.34 203.0.113.10 41822 443 6 3 180 1761470000 1761470060 ACCEPT OK
2 123456789012 eni-0aaa1111 203.0.113.10 10.0.12.34 443 41822 6 3 180 1761470000 1761470060 REJECT OK
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two caveats. Flow logs do not record traffic to the Amazon-provided DNS server or instance metadata, and they never contain query names, so use Route 53 Resolver query logging for DNS problems. And if the dropping NACL is on the NAT gateway's subnet, the REJECT appears on the NAT gateway's interface, so check both.&lt;/p&gt;

&lt;h2&gt;
  
  
  VPC design checklist
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;One central address plan, with no overlaps between VPCs or with on-premises.&lt;/li&gt;
&lt;li&gt;Main route table with the local route only; every subnet associated explicitly.&lt;/li&gt;
&lt;li&gt;NAT per AZ, or a regional NAT gateway, so steady-state traffic does not cross zones to reach NAT.&lt;/li&gt;
&lt;li&gt;S3 and DynamoDB gateway endpoints everywhere; interface endpoints where volume justifies them.&lt;/li&gt;
&lt;li&gt;Transit Gateway route tables per environment, appliance mode on inspection attachments.&lt;/li&gt;
&lt;li&gt;Hybrid DNS through Resolver endpoints, and a tested failover for on-premises links.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Interview questions this covers
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;How would you design a VPC with public and private subnets across three Availability Zones?&lt;/li&gt;
&lt;li&gt;What is the difference between a gateway endpoint and an interface endpoint, and when would you pay for one?&lt;/li&gt;
&lt;li&gt;VPC peering, Transit Gateway or PrivateLink: which would you use to connect 50 VPCs and a data centre?&lt;/li&gt;
&lt;li&gt;An instance in a private subnet can't reach the internet. How do you troubleshoot it?&lt;/li&gt;
&lt;li&gt;Why does a NAT gateway bill keep growing, and how do you bring it down?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For more practice, browse &lt;a href="https://devopsinterview.cloud/ebooks" rel="noopener noreferrer"&gt;all the DevOps interview ebooks&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Cloud networking questions in real interviews go further: a network that scales to hundreds of VPCs, resilient connectivity between a data centre and the cloud, and Lambda functions reaching private APIs without NAT gateways. &lt;a href="https://devopsinterview.cloud/ebooks/cloud-interview-mastery" rel="noopener noreferrer"&gt;Cloud Platforms: AWS, Azure &amp;amp; GCP Interview Questions&lt;/a&gt; works through 50 senior-level questions with worked answers, including a chapter on cloud networking and security. It is one of the five books in the &lt;a href="https://devopsinterview.cloud/ebooks/complete-devops-mastery-bundle" rel="noopener noreferrer"&gt;DevOps Interview Mastery bundle&lt;/a&gt;, and every purchase includes the free Interview-Day Playbook with a 7-day and 24-hour prep plan and a 30-second answer framework.&lt;/p&gt;

&lt;p&gt;Prefer to watch or listen? Answers like these are walked through on &lt;a href="https://www.youtube.com/@DevOpsInterviewCloud" rel="noopener noreferrer"&gt;YouTube&lt;/a&gt; and on the podcast, on &lt;a href="https://open.spotify.com/show/033txr35oY2Rcn6H4CVZ4S" rel="noopener noreferrer"&gt;Spotify&lt;/a&gt; and &lt;a href="https://music.amazon.com/podcasts/49554d87-91cb-4816-9420-cdac8a06ac75/devops-cloud-interview-prep-real-scenarios-answers" rel="noopener noreferrer"&gt;Amazon Music&lt;/a&gt;.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://devopsinterview.cloud/blog/aws-vpc-interview-questions" rel="noopener noreferrer"&gt;devopsinterview.cloud&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>architecture</category>
      <category>aws</category>
      <category>interview</category>
      <category>networking</category>
    </item>
    <item>
      <title>Kubernetes Multi-Tenancy: Designing a Shared Cluster</title>
      <dc:creator>DevOpsInterview.Cloud</dc:creator>
      <pubDate>Wed, 04 Nov 2026 09:20:04 +0000</pubDate>
      <link>https://dev.to/devopsinterview_cloud/kubernetes-multi-tenancy-designing-a-shared-cluster-hkp</link>
      <guid>https://dev.to/devopsinterview_cloud/kubernetes-multi-tenancy-designing-a-shared-cluster-hkp</guid>
      <description>&lt;p&gt;"We have twenty engineering teams and we want them all on one shared Kubernetes cluster. How do you design it so teams are isolated, cannot affect each other, and ops does not become a bottleneck?" That is the usual shape of the Kubernetes multi-tenancy question in platform engineering interviews, and it tends to arrive with no further constraints. Part of a good answer is asking what the interviewer left out, starting with how much the tenants trust each other. The rest is a layered design: namespaces and RBAC for API access, NetworkPolicies for traffic, quotas for capacity, admission policy to enforce it all, and automation so it stays that way.&lt;/p&gt;

&lt;p&gt;The video version covers the same design if you prefer to watch it:&lt;/p&gt;

&lt;p&gt;  &lt;iframe src="https://www.youtube.com/embed/5yIN10vSIJM" width="710" height="399"&gt;
  &lt;/iframe&gt;
&lt;/p&gt;

&lt;h2&gt;
  
  
  Soft vs hard Kubernetes multi-tenancy: which model do you need?
&lt;/h2&gt;

&lt;p&gt;Pin down the tenancy model first, because every later decision depends on it. Soft multi-tenancy assumes tenants are cooperative, like teams in one company, and relies on logical separation: namespaces, RBAC and NetworkPolicies on shared nodes and a shared control plane. Hard multi-tenancy assumes tenants may be hostile to each other and needs much stronger isolation, which in practice means sandboxing or dedicated infrastructure on top of those controls. It is a spectrum rather than two settings, and different tenants on one platform can sit at different points.&lt;/p&gt;

&lt;p&gt;What decides it is trust between tenants. Internal teams who share an employer and a security team mostly need protection from accidents: a runaway deployment, a pod with no memory limit. A SaaS product running customer-supplied code needs protection from attackers, and soft tenancy alone is not enough there. Every container on a node shares that node's Linux kernel, so a container escape exposes every tenant scheduled on it.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Infrastructure choice&lt;/th&gt;
&lt;th&gt;What it adds&lt;/th&gt;
&lt;th&gt;Cost and ops burden&lt;/th&gt;
&lt;th&gt;Fits&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Shared nodes&lt;/td&gt;
&lt;td&gt;Namespaces, RBAC, NetworkPolicy, quotas&lt;/td&gt;
&lt;td&gt;Lowest: one cluster to upgrade and monitor&lt;/td&gt;
&lt;td&gt;Internal teams who trust each other&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Dedicated node pools&lt;/td&gt;
&lt;td&gt;Node-level separation via taints and node selectors&lt;/td&gt;
&lt;td&gt;Moderate: more pools, worse bin-packing&lt;/td&gt;
&lt;td&gt;Noisy or sensitive internal tenants&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Virtual control plane&lt;/td&gt;
&lt;td&gt;A per-tenant API server, for example vCluster&lt;/td&gt;
&lt;td&gt;Moderate: one more control plane per tenant&lt;/td&gt;
&lt;td&gt;Tenants who need their own CRDs&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Separate clusters&lt;/td&gt;
&lt;td&gt;Everything&lt;/td&gt;
&lt;td&gt;Highest: every cluster to patch and observe&lt;/td&gt;
&lt;td&gt;Untrusted code, regulatory separation&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;For the twenty-team question, the defensible answer is soft tenancy with strong defaults and a path to move specific tenants onto dedicated nodes or clusters when their threat model demands it.&lt;/p&gt;

&lt;h2&gt;
  
  
  How should you lay out namespaces and RBAC for each team?
&lt;/h2&gt;

&lt;p&gt;Namespaces are the primary isolation unit. A common layout is one namespace per team per environment: &lt;code&gt;payments-dev&lt;/code&gt;, &lt;code&gt;payments-staging&lt;/code&gt; and &lt;code&gt;payments-prod&lt;/code&gt; rather than one &lt;code&gt;payments&lt;/code&gt; namespace holding everything, so bindings, quotas and audit entries are scoped to something meaningful.&lt;/p&gt;

&lt;p&gt;The platform team defines ClusterRoles as permission templates and grants them per namespace with RoleBindings. A RoleBinding that references a ClusterRole grants its permissions only inside the binding's namespace, so one definition serves every team:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;rbac.authorization.k8s.io/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ClusterRole&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;app-developer&lt;/span&gt;
&lt;span class="na"&gt;rules&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
&lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;apiGroups&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;apps"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
  &lt;span class="na"&gt;resources&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;deployments"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
  &lt;span class="na"&gt;verbs&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;get"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;list"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;watch"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;create"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;update"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;patch"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
&lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;apiGroups&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;apps"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
  &lt;span class="na"&gt;resources&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;deployments/scale"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
  &lt;span class="na"&gt;verbs&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;get"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;update"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;patch"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
&lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;apiGroups&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;apps"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
  &lt;span class="na"&gt;resources&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;replicasets"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
  &lt;span class="na"&gt;verbs&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;get"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;list"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;watch"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
&lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;apiGroups&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
  &lt;span class="na"&gt;resources&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;services"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;configmaps"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
  &lt;span class="na"&gt;verbs&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;get"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;list"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;watch"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;create"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;update"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;patch"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
&lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;apiGroups&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
  &lt;span class="na"&gt;resources&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;pods"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
  &lt;span class="na"&gt;verbs&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;get"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;list"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;watch"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
&lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;apiGroups&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
  &lt;span class="na"&gt;resources&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;pods/log"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
  &lt;span class="na"&gt;verbs&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;get"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
&lt;span class="nn"&gt;---&lt;/span&gt;
&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;rbac.authorization.k8s.io/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;RoleBinding&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;payments-developers&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;payments-prod&lt;/span&gt;
&lt;span class="na"&gt;roleRef&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;apiGroup&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;rbac.authorization.k8s.io&lt;/span&gt;
  &lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ClusterRole&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;app-developer&lt;/span&gt;
&lt;span class="na"&gt;subjects&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
&lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Group&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;payments-team&lt;/span&gt;
  &lt;span class="na"&gt;apiGroup&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;rbac.authorization.k8s.io&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;kubectl scale&lt;/code&gt; needs the &lt;code&gt;deployments/scale&lt;/code&gt; rule: it goes through the scale subresource, which RBAC treats as a separate resource, so patch rights on Deployments are not enough. Extend the template with what your teams actually deploy, one resource at a time.&lt;/p&gt;

&lt;p&gt;Tenants never get cluster-scoped permissions or &lt;code&gt;cluster-admin&lt;/code&gt;. RBAC's escalation rules already stop a tenant from binding a role with permissions they lack unless they hold &lt;code&gt;bind&lt;/code&gt; on it, so the realistic risk is a platform engineer granting &lt;code&gt;cluster-admin&lt;/code&gt; "temporarily" during an incident and never removing it. An admission policy that rejects creating or updating any RoleBinding or ClusterRoleBinding that references &lt;code&gt;cluster-admin&lt;/code&gt;, unless the request comes from the platform's deploy identity, stops anyone handing one out directly, including by adding a subject to an existing binding; protected-branch review on the repository that identity deploys from is what puts the grant through Git review.&lt;/p&gt;

&lt;p&gt;Deployment rights are a privilege path of their own that the &lt;code&gt;bind&lt;/code&gt; checks do not cover. A workload can run as any ServiceAccount in its namespace and mount any Secret there, so keep privileged ServiceAccounts and shared credentials out of tenant namespaces. The &lt;a href="https://devopsinterview.cloud/blog/kubernetes-rbac-least-privilege-lockdown" rel="noopener noreferrer"&gt;Kubernetes RBAC lockdown guide&lt;/a&gt; covers escalation verbs, ServiceAccounts and audit queries in depth.&lt;/p&gt;

&lt;h2&gt;
  
  
  How do you isolate tenant network traffic with NetworkPolicies?
&lt;/h2&gt;

&lt;p&gt;By default, any pod in a cluster can reach any other pod. The multi-tenant baseline is a default-deny NetworkPolicy in every namespace from the moment it exists, applied by the provisioning pipeline so no team has to remember it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;networking.k8s.io/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;NetworkPolicy&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;default-deny-all&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;payments-prod&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;podSelector&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;{}&lt;/span&gt;
  &lt;span class="na"&gt;policyTypes&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s"&gt;Ingress&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s"&gt;Egress&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Policies are additive, so allow rules go on top. Pods in the same namespace need one, an ingress and egress rule whose peer is an empty &lt;code&gt;podSelector&lt;/code&gt;. DNS needs its own rule: with egress denied, every service lookup fails until you allow it explicitly.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;networking.k8s.io/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;NetworkPolicy&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;allow-dns&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;payments-prod&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;podSelector&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;{}&lt;/span&gt;
  &lt;span class="na"&gt;policyTypes&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s"&gt;Egress&lt;/span&gt;
  &lt;span class="na"&gt;egress&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;to&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;namespaceSelector&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;matchLabels&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;kubernetes.io/metadata.name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;kube-system&lt;/span&gt;
      &lt;span class="na"&gt;podSelector&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;matchLabels&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;k8s-app&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;kube-dns&lt;/span&gt;
    &lt;span class="na"&gt;ports&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;protocol&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;UDP&lt;/span&gt;
      &lt;span class="na"&gt;port&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;53&lt;/span&gt;
    &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;protocol&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;TCP&lt;/span&gt;
      &lt;span class="na"&gt;port&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;53&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This assumes pods query the CoreDNS pods directly, and &lt;code&gt;k8s-app: kube-dns&lt;/code&gt; is a common label but not universal. Check the labels and a pod's &lt;code&gt;/etc/resolv.conf&lt;/code&gt; first: with NodeLocal DNSCache or another resolver path you need a different rule, often an &lt;code&gt;ipBlock&lt;/code&gt; for the node-local address. Then add small allows for platform services, such as ingress from the ingress controller and from Prometheus, selecting those namespaces by &lt;code&gt;kubernetes.io/metadata.name&lt;/code&gt;, an immutable label the control plane sets to each namespace's name.&lt;/p&gt;

&lt;p&gt;NetworkPolicy is enforced by the network plugin, and on a plugin without support a policy silently does nothing. Calico and Cilium enforce it; Flannel on its own does not.&lt;/p&gt;

&lt;p&gt;Tenants also must not be able to undo the baseline. The built-in &lt;code&gt;admin&lt;/code&gt; and &lt;code&gt;edit&lt;/code&gt; ClusterRoles include NetworkPolicy writes, and restoring a deleted default-deny is not enough, because a tenant who can write policies can add an allow-all next to it. Keep NetworkPolicy writes out of tenant roles. If teams need to manage their own rules, constrain what those rules may allow at admission, or put the cross-tenant boundary in cluster-scoped policies tenants cannot edit; Calico and Cilium both offer cluster-scoped policies that can enforce a deny a namespace-level allow cannot override.&lt;/p&gt;

&lt;p&gt;Cilium's own CiliumNetworkPolicy adds L7 rules, so you can restrict a shared API by HTTP method and path as well as port, at the price of a resource that only works on Cilium.&lt;/p&gt;

&lt;h2&gt;
  
  
  How do ResourceQuota and LimitRange prevent noisy neighbours?
&lt;/h2&gt;

&lt;p&gt;Without resource controls, one team's bad deployment can starve everyone else on the same nodes. A ResourceQuota caps a namespace's total requests, limits and object counts; a LimitRange sets per-container defaults and maximums. They work as a pair:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ResourceQuota&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;team-quota&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;payments-prod&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;hard&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;requests.cpu&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;8"&lt;/span&gt;
    &lt;span class="na"&gt;requests.memory&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;16Gi&lt;/span&gt;
    &lt;span class="na"&gt;limits.cpu&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;16"&lt;/span&gt;
    &lt;span class="na"&gt;limits.memory&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;32Gi&lt;/span&gt;
    &lt;span class="na"&gt;count/pods&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;50"&lt;/span&gt;
&lt;span class="nn"&gt;---&lt;/span&gt;
&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;LimitRange&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;default-limits&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;payments-prod&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;limits&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Container&lt;/span&gt;
    &lt;span class="na"&gt;default&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;cpu&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;250m&lt;/span&gt;
      &lt;span class="na"&gt;memory&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;256Mi&lt;/span&gt;
    &lt;span class="na"&gt;defaultRequest&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;cpu&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;100m&lt;/span&gt;
      &lt;span class="na"&gt;memory&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;128Mi&lt;/span&gt;
    &lt;span class="na"&gt;max&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;cpu&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;4"&lt;/span&gt;
      &lt;span class="na"&gt;memory&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;4Gi&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Once a namespace has a CPU or memory quota, every new pod must declare requests or limits for those resources or the API server can reject it. Without a LimitRange, that surfaces as ReplicaSets stuck on &lt;code&gt;FailedCreate&lt;/code&gt; events. The LimitRange fills in defaults for containers that declare nothing, so their pods are admitted and still counted. The numbers are placeholders; size them from real usage, as the &lt;a href="https://devopsinterview.cloud/blog/kubernetes-cost-optimization-200-node-cluster" rel="noopener noreferrer"&gt;Kubernetes cost optimization walkthrough&lt;/a&gt; describes.&lt;/p&gt;

&lt;p&gt;PriorityClasses do less against eviction than is often assumed. Priority decides which pods the scheduler preempts when the cluster is full, and it is one factor in the kubelet's eviction order under node pressure, ranked after whether a pod is using more than it requested. The multi-tenancy risk runs the other way: a tenant can set &lt;code&gt;priorityClassName&lt;/code&gt; to any PriorityClass you have created, including the high one meant for platform agents. Kubernetes can restrict that with ResourceQuota scope selectors plus an API server admission configuration file. On managed clusters, where you cannot pass API server flags, a policy that rejects high priority classes outside platform namespaces does the same job.&lt;/p&gt;

&lt;h2&gt;
  
  
  Which admission policies does a multi-tenant cluster need?
&lt;/h2&gt;

&lt;p&gt;Admission control checks every object against your standards before it is stored. Three layers are worth naming in an interview.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Pod Security Admission&lt;/strong&gt; is built in. Labelling tenant namespaces with &lt;code&gt;pod-security.kubernetes.io/enforce: restricted&lt;/code&gt; rejects privileged containers, host namespaces, hostPath volumes and containers allowed to run as root, which closes the easiest paths from a container to its node. Tenants cannot remove the label while their roles have no write access to Namespace objects.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A policy engine&lt;/strong&gt; covers what Pod Security does not: approved registries, required labels, the &lt;code&gt;cluster-admin&lt;/code&gt; rule above, and a ban on Service &lt;code&gt;externalIPs&lt;/code&gt;, which CVE-2020-8554 showed a tenant who can create Services could use to intercept other traffic. Kyverno and OPA Gatekeeper are the usual choices. Gatekeeper templates have traditionally been written in Rego; Kyverno policies are Kubernetes resources, and its current policy types use CEL, the language of the built-in ValidatingAdmissionPolicy. A registry restriction looks like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;policies.kyverno.io/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ValidatingPolicy&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;restrict-image-registries&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;validationActions&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s"&gt;Deny&lt;/span&gt;
  &lt;span class="na"&gt;matchConstraints&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;resourceRules&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;apiGroups&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
      &lt;span class="na"&gt;apiVersions&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;v1"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
      &lt;span class="na"&gt;operations&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;CREATE"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;UPDATE"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
      &lt;span class="na"&gt;resources&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;pods"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;pods/ephemeralcontainers"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
  &lt;span class="na"&gt;validations&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;message&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Images&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;must&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;come&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;from&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;registry.example.com."&lt;/span&gt;
    &lt;span class="na"&gt;expression&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;&amp;gt;-&lt;/span&gt;
      &lt;span class="s"&gt;object.spec.containers.all(c, c.image.startsWith('registry.example.com/')) &amp;amp;&amp;amp;&lt;/span&gt;
      &lt;span class="s"&gt;object.spec.?initContainers.orValue([]).all(c, c.image.startsWith('registry.example.com/')) &amp;amp;&amp;amp;&lt;/span&gt;
      &lt;span class="s"&gt;object.spec.?ephemeralContainers.orValue([]).all(c, c.image.startsWith('registry.example.com/'))&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Matching the &lt;code&gt;pods/ephemeralcontainers&lt;/code&gt; subresource matters, because &lt;code&gt;kubectl debug&lt;/code&gt; adds containers through it, and a rule on &lt;code&gt;pods&lt;/code&gt; alone never sees them. Since the policy matches Pods, a bad image fails when the ReplicaSet creates the pod rather than at &lt;code&gt;kubectl apply&lt;/code&gt;; Kyverno's autogen option generates policies for Deployments and other controllers to move the failure earlier. Older Kyverno releases use the &lt;code&gt;kyverno.io/v1&lt;/code&gt; ClusterPolicy, which current docs mark deprecated, so check what your version supports.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Policy as code.&lt;/strong&gt; Keep policies in Git and apply them through the same GitOps review as the rest of the platform, rolling new rules out with &lt;code&gt;validationActions: [Audit]&lt;/code&gt; before switching to &lt;code&gt;Deny&lt;/code&gt;. Mutating policies that inject default labels or security contexts help too, if kept small and documented.&lt;/p&gt;

&lt;h2&gt;
  
  
  When do tenants need dedicated nodes or separate clusters?
&lt;/h2&gt;

&lt;p&gt;Soft tenancy shares the kernel. For internal teams worried about accidents, that is usually acceptable with Pod Security enforced. When it is not, there are several steps up.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Dedicated node pools.&lt;/strong&gt; Taint the tenant's nodes and give the tenant's pods a matching toleration plus a node selector or affinity. That keeps noisy workloads off other tenants' nodes, and a container escape lands on a node running only that tenant's pods. It does not make an escape harmless: the node still holds kubelet credentials and runs platform DaemonSets, so a compromised node remains a foothold into the shared cluster. Taints also steer scheduling but enforce nothing. A tenant can add the toleration and selector to their own pod spec and land on another tenant's pool, so the assignment has to be enforced at admission, by a policy engine, a tenant controller such as Capsule, or the PodNodeSelector and PodTolerationRestriction plugins where you control the API server.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Sandboxed runtimes.&lt;/strong&gt; A RuntimeClass pointing at gVisor or Kata Containers puts a user-space kernel or a lightweight VM between the pod and the host kernel, at some cost in performance and compatibility.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Virtual control planes.&lt;/strong&gt; vCluster runs a per-tenant API server inside a namespace of the host cluster, so tenants can install CRDs and cluster-scoped objects without touching the host's. In its default mode their pods still run on the host's nodes, so it isolates the control plane, not the kernel.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Separate clusters&lt;/strong&gt;, for untrusted code or regulatory separation. Cluster API makes creating and upgrading them declarative, but each cluster still has its own control plane and its own upgrades to run.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why does manual namespace provisioning break multi-tenancy?
&lt;/h2&gt;

&lt;p&gt;A common way shared clusters decay is manual namespace provisioning. Someone runs &lt;code&gt;kubectl create namespace&lt;/code&gt;, applies some RBAC, forgets the NetworkPolicy and the quota, and moves on. Months later nobody can say which namespaces have which controls.&lt;/p&gt;

&lt;p&gt;Treat namespace creation like any other infrastructure change. A team opens a pull request against a platform repository, and after review a GitOps controller creates the namespace with its full baseline:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;the Namespace, with Pod Security labels and an owning-team label&lt;/li&gt;
&lt;li&gt;ResourceQuota and LimitRange&lt;/li&gt;
&lt;li&gt;the default-deny NetworkPolicy plus the DNS and same-namespace allows&lt;/li&gt;
&lt;li&gt;RoleBindings for the team's identity-provider groups&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;With self-healing or drift correction enabled, a baseline object someone deletes by hand comes back on the next sync.&lt;/p&gt;

&lt;p&gt;The other slow failure is RBAC drift. People are added to groups for one-off access and never removed, and bindings outlive their projects, because RBAC expires nothing on its own. Bindings in Git and group-based access help; a regular access review, scheduled from day one, catches the rest.&lt;/p&gt;

&lt;h2&gt;
  
  
  Should you use Capsule or build your own tenant layer?
&lt;/h2&gt;

&lt;p&gt;Building the namespace controller, policy distribution and quota management yourself is a lot of undifferentiated work, so it is reasonable to say you would evaluate an existing project first.&lt;/p&gt;

&lt;p&gt;Capsule provides an existing tenant layer. Its cluster-scoped Tenant resource groups namespaces under a set of owners and lets those owners create namespaces themselves, which plain RBAC cannot safely allow. Constraints set on the tenant apply to every namespace it owns:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;capsule.clastix.io/v1beta2&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Tenant&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;payments&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;owners&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;payments-team&lt;/span&gt;
    &lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Group&lt;/span&gt;
  &lt;span class="na"&gt;namespaceOptions&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;quota&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;5&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;namespaceOptions.quota&lt;/code&gt; caps how many namespaces the tenant can create. Capsule can also apply quotas, registry restrictions, node selectors and network policies per tenant, but that part of its API is changing: the &lt;a href="https://projectcapsule.dev/docs/tenants/" rel="noopener noreferrer"&gt;Capsule tenant docs&lt;/a&gt; mark several older Tenant fields as legacy in favour of a rules-based model, so write those against the version you run.&lt;/p&gt;

&lt;p&gt;The Hierarchical Namespace Controller (HNC), with RBAC and policies propagating from parent to child namespaces, used to be the other standard answer. Its upstream repository is now archived, so treat it as a design idea rather than something to deploy on a new platform.&lt;/p&gt;

&lt;h2&gt;
  
  
  A checklist for the shared-cluster answer
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Ask how much tenants trust each other, and pick namespace, node or cluster isolation from the answer.&lt;/li&gt;
&lt;li&gt;A namespace per team per environment, ClusterRole templates bound per namespace, no &lt;code&gt;cluster-admin&lt;/code&gt; for tenants.&lt;/li&gt;
&lt;li&gt;Default-deny NetworkPolicy plus DNS and platform allows, on a CNI that enforces it.&lt;/li&gt;
&lt;li&gt;ResourceQuota with LimitRange everywhere, and restricted access to high PriorityClasses.&lt;/li&gt;
&lt;li&gt;Pod Security &lt;code&gt;restricted&lt;/code&gt;, a policy engine for the rest, and policies in Git.&lt;/li&gt;
&lt;li&gt;Dedicated nodes enforced at admission, then sandboxes, virtual control planes or separate clusters as needed.&lt;/li&gt;
&lt;li&gt;Namespaces provisioned from Git, and access reviewed on a schedule.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Interview questions this covers
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;How would you design a shared Kubernetes cluster for twenty teams so they cannot affect each other?&lt;/li&gt;
&lt;li&gt;What is the difference between soft and hard multi-tenancy, and what decides which one you need?&lt;/li&gt;
&lt;li&gt;Why does a default-deny egress NetworkPolicy break DNS, and how do you fix it?&lt;/li&gt;
&lt;li&gt;Why does a namespace with a ResourceQuota also need a LimitRange?&lt;/li&gt;
&lt;li&gt;How do you stop a tenant from scheduling onto another tenant's dedicated nodes?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The &lt;a href="https://devopsinterview.cloud/blog/kubernetes-interview-questions-2026" rel="noopener noreferrer"&gt;25 Kubernetes interview questions for 2026&lt;/a&gt; cover more of the scheduling, networking and debugging ground.&lt;/p&gt;

&lt;p&gt;Multi-tenancy questions rarely stop at namespaces. Interviewers follow up into hostile tenants, container escapes, and how a SaaS tenancy model keeps unit economics sane. &lt;a href="https://devopsinterview.cloud/ebooks/container-orchestration-journey" rel="noopener noreferrer"&gt;Containers &amp;amp; Orchestration&lt;/a&gt; has a chapter on identity, mTLS and multi-tenancy among its 52 worked questions, with follow-up chains and production war stories. It is one of the five books in the &lt;a href="https://devopsinterview.cloud/ebooks/complete-devops-mastery-bundle" rel="noopener noreferrer"&gt;DevOps Interview Mastery bundle&lt;/a&gt;, and every purchase includes the free Interview-Day Playbook with a day-of checklist and behavioural answer frameworks.&lt;/p&gt;

&lt;p&gt;Prefer to watch or listen? Answers like these are walked through on &lt;a href="https://www.youtube.com/@DevOpsInterviewCloud" rel="noopener noreferrer"&gt;YouTube&lt;/a&gt; and on the podcast, on &lt;a href="https://open.spotify.com/show/033txr35oY2Rcn6H4CVZ4S" rel="noopener noreferrer"&gt;Spotify&lt;/a&gt; and &lt;a href="https://music.amazon.com/podcasts/49554d87-91cb-4816-9420-cdac8a06ac75/devops-cloud-interview-prep-real-scenarios-answers" rel="noopener noreferrer"&gt;Amazon Music&lt;/a&gt;.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://devopsinterview.cloud/blog/kubernetes-multi-tenancy-shared-cluster-design" rel="noopener noreferrer"&gt;devopsinterview.cloud&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>architecture</category>
      <category>devops</category>
      <category>infrastructure</category>
      <category>kubernetes</category>
    </item>
    <item>
      <title>Kubernetes Cost Optimization for a 200-Node EKS Cluster</title>
      <dc:creator>DevOpsInterview.Cloud</dc:creator>
      <pubDate>Tue, 03 Nov 2026 09:20:03 +0000</pubDate>
      <link>https://dev.to/devopsinterview_cloud/kubernetes-cost-optimization-for-a-200-node-eks-cluster-1i7m</link>
      <guid>https://dev.to/devopsinterview_cloud/kubernetes-cost-optimization-for-a-200-node-eks-cluster-1i7m</guid>
      <description>&lt;p&gt;"We run a 200-node EKS cluster with about 400 microservices. The bill is out of control, and the VP of Engineering wants it cut by 40 percent without hurting reliability. Walk me through your approach." This is a common senior Kubernetes cost optimization interview question, and the weak answer is "right-size the pods", which is too vague to act on. Interviewers want an order of operations: measure first, take the low-risk wins, then make the architectural changes, and explain the trade-off at each step.&lt;/p&gt;

&lt;p&gt;The video version covers the same scenario if you prefer to watch it:&lt;/p&gt;

&lt;p&gt;  &lt;iframe src="https://www.youtube.com/embed/VYPjNDu_wmE" width="710" height="399"&gt;
  &lt;/iframe&gt;
&lt;/p&gt;

&lt;h2&gt;
  
  
  Why do resource requests drive Kubernetes costs?
&lt;/h2&gt;

&lt;p&gt;The scheduler places a pod by comparing the requests already on a node against that node's allocatable capacity. What the pods actually use plays no part in the decision. If every service copied a template that requests 2 CPU and 4Gi of memory, nodes fill up on paper long before they do in reality, and new pods need new nodes while existing ones sit mostly idle.&lt;/p&gt;

&lt;p&gt;Picture a node with 32 vCPU allocatable, 31 vCPU requested and 3 vCPU in use. To the scheduler it has one CPU left, so the next 2-CPU pod cannot land there. The autoscaler is just as blind: Cluster Autoscaler judges whether a node is underutilized by the sum of its pods' requests, not by measured usage, so over-requesting can keep a mostly idle node above the scale-down threshold.&lt;/p&gt;

&lt;p&gt;Worth saying out loud in an interview: if a container sets a limit but no request, and no LimitRange supplies a default, Kubernetes copies the limit into the request, so generous limits quietly become generous requests.&lt;/p&gt;

&lt;h2&gt;
  
  
  How do you find where the money goes?
&lt;/h2&gt;

&lt;p&gt;Before editing a single manifest, get cost allocation. &lt;a href="https://opencost.io/docs/specification" rel="noopener noreferrer"&gt;OpenCost&lt;/a&gt; is the CNCF project that Kubecost originally built; it breaks cluster cost down by namespace, controller, pod or label. The Helm chart assumes you already run Prometheus, so point it at yours in &lt;code&gt;values.yaml&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;helm repo add opencost-charts https://opencost.github.io/opencost-helm-chart
helm repo update
helm &lt;span class="nb"&gt;install &lt;/span&gt;opencost opencost-charts/opencost &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--namespace&lt;/span&gt; opencost &lt;span class="nt"&gt;--create-namespace&lt;/span&gt; &lt;span class="nt"&gt;-f&lt;/span&gt; values.yaml
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The API listens on port 9003. This pulls the ten most expensive namespaces over the last week:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# in a second terminal: kubectl -n opencost port-forward deployment/opencost 9003&lt;/span&gt;
curl &lt;span class="nt"&gt;-sG&lt;/span&gt; http://localhost:9003/allocation &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-d&lt;/span&gt; &lt;span class="nv"&gt;window&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;7d &lt;span class="nt"&gt;-d&lt;/span&gt; &lt;span class="nv"&gt;aggregate&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;namespace &lt;span class="se"&gt;\&lt;/span&gt;
  | jq &lt;span class="s1"&gt;'.data[0] | to_entries | sort_by(-.value.totalCost) | .[:10]
        | map({namespace: .key, totalCost: .value.totalCost})'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;How OpenCost assigns cost separates two different problems. Its specification charges each workload for the larger of what it requested and what it used. A container that requests 2 CPU and uses 200m is billed for 2 CPU, so over-requesting shows up on the owning team's line. Capacity that no workload requested or used lands in a separate &lt;code&gt;__idle__&lt;/code&gt; entry, which the API returns when you add &lt;code&gt;includeIdle=true&lt;/code&gt;.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;High allocated cost, low usage:&lt;/strong&gt; pods ask for far more than they use. The fix is right-sizing.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Large idle cost:&lt;/strong&gt; nodes are running that nothing has claimed. The fix is better bin-packing and scale-down.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Mixing them up is how optimization projects stall: autoscaler tuning cannot fix over-requested pods, and right-sizing cannot remove nodes the autoscaler keeps. Before quoting numbers to a VP, check how OpenCost prices your nodes (list prices versus your discounted or spot rates); treat the first report as a ranking, not an invoice.&lt;/p&gt;

&lt;h2&gt;
  
  
  How do you right-size requests without breaking production?
&lt;/h2&gt;

&lt;p&gt;Once you know which namespaces cost the most, fix their requests. &lt;a href="https://goldilocks.docs.fairwinds.com/installation/" rel="noopener noreferrer"&gt;Goldilocks&lt;/a&gt; from Fairwinds creates a Vertical Pod Autoscaler object in recommendation-only mode for each workload in a namespace you label, then shows the recommendations per container in a dashboard. It needs only the VPA recommender, not the VPA updater or admission controller, so nothing gets changed automatically. Install the recommender first, or set &lt;code&gt;vpa.enabled=true&lt;/code&gt; to have the Goldilocks chart install it as a sub-chart.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;helm repo add fairwinds-stable https://charts.fairwinds.com/stable
helm &lt;span class="nb"&gt;install &lt;/span&gt;goldilocks fairwinds-stable/goldilocks &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--namespace&lt;/span&gt; goldilocks &lt;span class="nt"&gt;--create-namespace&lt;/span&gt;

kubectl label namespace payments goldilocks.fairwinds.com/enabled&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nb"&gt;true
&lt;/span&gt;kubectl &lt;span class="nt"&gt;-n&lt;/span&gt; goldilocks port-forward svc/goldilocks-dashboard 8080:80
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Let the recommender observe a full business cycle, including peak traffic and scheduled jobs, before trusting it. Then move the numbers into your Helm values or GitOps manifests and roll them out service by service. The code does not change, but requests do more than steer the scheduler, so watch three things:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;CPU:&lt;/strong&gt; a lower request means a smaller guaranteed share when the node is contended. Throttling comes from CPU limits, not requests.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Memory:&lt;/strong&gt; be conservative. When a node runs short of memory, the kubelet's eviction ranking puts pods using more than they requested ahead of the rest, and a limit set too low gets the container OOMKilled. The &lt;a href="https://devopsinterview.cloud/blog/kubernetes-production-debugging-walkthrough" rel="noopener noreferrer"&gt;production debugging walkthrough&lt;/a&gt; covers what that failure looks like.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;HPA targets:&lt;/strong&gt; an HPA computes CPU utilization against the request. Lower the request and the same usage reads as a higher percentage, so a service scaled on a CPU target can add replicas and claw back the saving. Revisit the HPA target in the same change.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Should you let VPA apply changes automatically? Its &lt;code&gt;Recreate&lt;/code&gt; mode evicts pods so their replacements start with new values. &lt;code&gt;Auto&lt;/code&gt; was deprecated in VPA 1.5.0 and is currently an alias for &lt;code&gt;Recreate&lt;/code&gt;. Newer releases add &lt;code&gt;InPlaceOrRecreate&lt;/code&gt;, which resizes pods in place when it can and falls back to eviction when it cannot. Any of these needs solid PodDisruptionBudgets and readiness probes first, and the VPA project warns against having VPA and an HPA both act on CPU or memory for the same workload.&lt;/p&gt;

&lt;h2&gt;
  
  
  How do you run spot instances on Kubernetes safely?
&lt;/h2&gt;

&lt;p&gt;Right-sizing reduces how many nodes you need. Spot reduces what each node costs. AWS advertises spot at &lt;a href="https://aws.amazon.com/ec2/spot/" rel="noopener noreferrer"&gt;up to 90 percent off On-Demand&lt;/a&gt;; the real discount varies by instance type, region and time. The catch: EC2 can reclaim a spot instance with two minutes' notice.&lt;/p&gt;

&lt;p&gt;Do it deliberately:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Separate the capacity.&lt;/strong&gt; Taint spot nodes so only workloads validated for interruption can land there. A toleration only permits scheduling on a tainted node; it does not attract pods to it. Add a node selector as well, using the label your provisioner sets: &lt;code&gt;karpenter.sh/capacity-type: spot&lt;/code&gt; with Karpenter, or &lt;code&gt;eks.amazonaws.com/capacityType: SPOT&lt;/code&gt; on EKS managed node groups.
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="c1"&gt;# Pod template of a deployment validated for spot interruption&lt;/span&gt;
&lt;span class="na"&gt;nodeSelector&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;karpenter.sh/capacity-type&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;spot&lt;/span&gt;
&lt;span class="na"&gt;tolerations&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;key&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;spot&lt;/span&gt;
    &lt;span class="na"&gt;operator&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Equal&lt;/span&gt;
    &lt;span class="na"&gt;value&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;true"&lt;/span&gt;
    &lt;span class="na"&gt;effect&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;NoSchedule&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Diversify instance types and zones.&lt;/strong&gt; Spot capacity is priced and reclaimed per pool, so more pools means fewer simultaneous losses. With Cluster Autoscaler, every instance type in one node group must have the same vCPU and memory (m5.xlarge, m5a.xlarge and m6i.xlarge all have 4 vCPU and 16 GiB), so add more node groups for more shapes. Karpenter can choose from a broad list in one NodePool.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Handle the notice.&lt;/strong&gt; Something must cordon and drain the node when the warning arrives: Karpenter's interruption queue (SQS plus EventBridge rules) does this, as does AWS Node Termination Handler on self-managed node groups. Pods should shut down cleanly well inside two minutes.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Know what PodDisruptionBudgets do here.&lt;/strong&gt; A PDB limits voluntary evictions such as drains, consolidation and scale-down. It cannot stop AWS reclaiming the instance; when the two minutes are up, the node is gone either way. Reliability on spot comes from replicas spread across many pools and zones, with the PDB keeping planned disruption orderly.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Keep stateful systems, singletons and long batch jobs without checkpointing on on-demand capacity.&lt;/p&gt;

&lt;h2&gt;
  
  
  Karpenter vs Cluster Autoscaler: which one saves more?
&lt;/h2&gt;

&lt;p&gt;Cluster Autoscaler resizes node groups you defined in advance (Auto Scaling groups on AWS). It scales up when pods are unschedulable and, by default, marks a node for removal when the requests on it fall below 50 percent of allocatable and stay there for 10 minutes, provided its pods can move elsewhere. It can only add the shape a group defines, so if the only group runs 16-CPU nodes, a pending pod that needs 3 CPU gets a new 16-CPU node.&lt;/p&gt;

&lt;p&gt;Karpenter works from the pending pods instead. It picks instance types from the NodePool's requirements to fit what is waiting, launches them directly, and runs consolidation: deleting empty nodes and replacing or merging underutilized ones. Its core lives in kubernetes-sigs under SIG Autoscaling (not, as often claimed, a CNCF project), with providers for AWS and Azure.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;karpenter.sh/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;NodePool&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;spot-general&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;template&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;nodeClassRef&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;group&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;karpenter.k8s.aws&lt;/span&gt;
        &lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;EC2NodeClass&lt;/span&gt;
        &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;default&lt;/span&gt;
      &lt;span class="na"&gt;taints&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
        &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;key&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;spot&lt;/span&gt;
          &lt;span class="na"&gt;value&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;true"&lt;/span&gt;
          &lt;span class="na"&gt;effect&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;NoSchedule&lt;/span&gt;
      &lt;span class="na"&gt;requirements&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
        &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;key&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;karpenter.sh/capacity-type&lt;/span&gt;
          &lt;span class="na"&gt;operator&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;In&lt;/span&gt;
          &lt;span class="na"&gt;values&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;spot"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
        &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;key&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;karpenter.k8s.aws/instance-category&lt;/span&gt;
          &lt;span class="na"&gt;operator&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;In&lt;/span&gt;
          &lt;span class="na"&gt;values&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;c"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;m"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;r"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
  &lt;span class="na"&gt;disruption&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;consolidationPolicy&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;WhenEmptyOrUnderutilized&lt;/span&gt;
    &lt;span class="na"&gt;consolidateAfter&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;5m&lt;/span&gt;
    &lt;span class="na"&gt;budgets&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;nodes&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;10%"&lt;/span&gt;
  &lt;span class="na"&gt;limits&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;cpu&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;1000"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The trade-offs interviewers probe:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Churn.&lt;/strong&gt; Consolidating occupied nodes moves pods. PDBs, the &lt;code&gt;karpenter.sh/do-not-disrupt&lt;/code&gt; annotation and NodePool disruption budgets (10 percent of nodes by default) limit how much moves at once, but your monitoring needs to tell routine restarts apart from real regressions.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Spot replacement.&lt;/strong&gt; Karpenter deletes empty or redundant spot nodes by default, but replacing one spot node with a cheaper one requires the &lt;code&gt;SpotToSpotConsolidation&lt;/code&gt; feature flag and, for single-node replacement, at least 15 candidate instance types.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The &lt;a href="https://karpenter.sh/docs/getting-started/migrating-from-cas/" rel="noopener noreferrer"&gt;Karpenter migration guide&lt;/a&gt; does not run both autoscalers side by side: it pins Karpenter's controller to an existing node group, scales Cluster Autoscaler to zero, then shrinks the old node groups while Karpenter provisions replacements. On EKS, lowering a managed node group's desired size terminates instances without honoring PDBs, and the Auto Scaling group chooses which ones go. So empty the nodes before shrinking anything: keep the controller on a node group you are not retiring, cordon the retiring nodes, drain them in small batches with &lt;code&gt;kubectl drain&lt;/code&gt; (eviction respects PDBs), and shrink or delete those groups only once they run nothing but DaemonSet pods.&lt;/p&gt;

&lt;h2&gt;
  
  
  Can topology spread constraints inflate your node count?
&lt;/h2&gt;

&lt;p&gt;Yes. A chart written years ago might say "spread evenly across zones, never schedule anything that breaks the spread":&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="c1"&gt;# Strict: a pod stays Pending rather than exceed the skew&lt;/span&gt;
&lt;span class="na"&gt;topologySpreadConstraints&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;maxSkew&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;1&lt;/span&gt;
    &lt;span class="na"&gt;topologyKey&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;topology.kubernetes.io/zone&lt;/span&gt;
    &lt;span class="na"&gt;whenUnsatisfiable&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;DoNotSchedule&lt;/span&gt;
    &lt;span class="na"&gt;labelSelector&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;matchLabels&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;app&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;internal-reports&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;With &lt;code&gt;DoNotSchedule&lt;/code&gt;, if the zone that would keep the skew in bounds has no room, the pod stays Pending, and the only way to place it is a new node in that specific zone, even when other zones have plenty of free capacity. The same rules work against scale-down: Cluster Autoscaler will not remove a node whose pods have nowhere legal to go, and Karpenter's consolidation runs the same check. Required pod anti-affinity on &lt;code&gt;kubernetes.io/hostname&lt;/code&gt; (one replica per node) has the same effect at higher replica counts.&lt;/p&gt;

&lt;p&gt;Audit these rules per service. Keep strict zone spreading where a service genuinely needs to survive a zone failure. For low-traffic internal services, a soft constraint lets the scheduler prefer an even spread without blocking placement:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="c1"&gt;# Soft: spread is a scoring preference, not a hard rule&lt;/span&gt;
&lt;span class="na"&gt;topologySpreadConstraints&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;maxSkew&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;2&lt;/span&gt;
    &lt;span class="na"&gt;topologyKey&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;topology.kubernetes.io/zone&lt;/span&gt;
    &lt;span class="na"&gt;whenUnsatisfiable&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ScheduleAnyway&lt;/span&gt;
    &lt;span class="na"&gt;labelSelector&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;matchLabels&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;app&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;internal-reports&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The price of &lt;code&gt;ScheduleAnyway&lt;/code&gt; is that the spread is no longer guaranteed. Pods that define no constraints of their own and belong to a Service, ReplicaSet or StatefulSet already get the scheduler's built-in soft defaults (&lt;code&gt;maxSkew: 3&lt;/code&gt; per hostname and &lt;code&gt;maxSkew: 5&lt;/code&gt; per zone, both &lt;code&gt;ScheduleAnyway&lt;/code&gt;).&lt;/p&gt;

&lt;h2&gt;
  
  
  Which metric shows a cluster is over-provisioned?
&lt;/h2&gt;

&lt;p&gt;Track requested-to-allocatable utilization, for CPU and memory separately: the share of each node's allocatable capacity that pods have requested. It maps directly to node count. A quick per-node view:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl describe nodes | &lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="nt"&gt;-A&lt;/span&gt; 8 &lt;span class="s2"&gt;"Allocated resources"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For a dashboard, kube-state-metrics exposes both sides:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;sum(
  kube_pod_container_resource_requests{resource="cpu"}
  * on (namespace, pod) group_left ()
  (kube_pod_status_phase{phase="Running"} == 1)
)
/
sum(kube_node_status_allocatable{resource="cpu"})
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Treat this as an approximation: it ignores init containers, pod overhead and pods still starting. Where you can scrape kube-scheduler, its &lt;code&gt;kube_pod_resource_request&lt;/code&gt; metric on the &lt;code&gt;/metrics/resources&lt;/code&gt; endpoint reports the effective request the scheduler actually uses. Either way, read it alongside actual usage against requests. A low requested ratio points at idle capacity the autoscaler is not removing. A high requested ratio with low real usage points at over-requesting, the right-sizing problem. There is no universal target; the right number depends on how much headroom you keep for traffic surges, zone loss and autoscaler lag. For reference, Cluster Autoscaler's default scale-down threshold treats a node under 50 percent requested as removable.&lt;/p&gt;

&lt;h2&gt;
  
  
  In what order should you apply Kubernetes cost optimization changes?
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Step&lt;/th&gt;
&lt;th&gt;What it fixes&lt;/th&gt;
&lt;th&gt;Risk&lt;/th&gt;
&lt;th&gt;Effort&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1. OpenCost visibility&lt;/td&gt;
&lt;td&gt;Shows who drives cost; separates over-requesting from idle&lt;/td&gt;
&lt;td&gt;None (read-only)&lt;/td&gt;
&lt;td&gt;Low&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2. Right-size requests&lt;/td&gt;
&lt;td&gt;Over-requested pods inflating node count&lt;/td&gt;
&lt;td&gt;Low with memory headroom and gradual rollout&lt;/td&gt;
&lt;td&gt;Medium&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;3. Spot for tolerant workloads&lt;/td&gt;
&lt;td&gt;Price per node&lt;/td&gt;
&lt;td&gt;Medium&lt;/td&gt;
&lt;td&gt;Medium&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4. Karpenter or a tuned Cluster Autoscaler&lt;/td&gt;
&lt;td&gt;Idle capacity and poor bin-packing&lt;/td&gt;
&lt;td&gt;Medium (pod churn)&lt;/td&gt;
&lt;td&gt;High&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;5. Spread constraint audit&lt;/td&gt;
&lt;td&gt;Nodes held up by strict placement rules&lt;/td&gt;
&lt;td&gt;Low to medium&lt;/td&gt;
&lt;td&gt;Low&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Measure after each step so you can show which change saved what. Once right-sizing has settled the steady baseline, cover the always-on on-demand capacity with Savings Plans or Reserved Instances; committing before right-sizing locks in the waste. And stop the problem returning: a LimitRange gives containers that omit requests a default, and a ResourceQuota caps what each namespace can request. On a shared cluster, quotas pair with the per-team permissions in the &lt;a href="https://devopsinterview.cloud/blog/kubernetes-rbac-least-privilege-lockdown" rel="noopener noreferrer"&gt;RBAC lockdown guide&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;A strong close sounds like this: "I would start with visibility and right-sizing because they are low risk and show where the rest of the money is, move tolerant workloads to spot once interruption handling is proven, and treat the autoscaler change as the biggest project because it touches every workload."&lt;/p&gt;

&lt;h2&gt;
  
  
  Interview questions this covers
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Why does the Kubernetes scheduler use requests rather than actual usage, and what does that mean for cost?&lt;/li&gt;
&lt;li&gt;How would you cut the bill of a 200-node cluster by 40 percent without hurting reliability?&lt;/li&gt;
&lt;li&gt;Karpenter vs Cluster Autoscaler: how does each decide when to add and remove nodes?&lt;/li&gt;
&lt;li&gt;What does a PodDisruptionBudget protect against on spot capacity, and what can it not prevent?&lt;/li&gt;
&lt;li&gt;How can topology spread constraints increase your node count?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The &lt;a href="https://devopsinterview.cloud/blog/kubernetes-interview-questions-2026" rel="noopener noreferrer"&gt;25 Kubernetes interview questions for 2026&lt;/a&gt; cover more of the scheduling, networking and debugging ground.&lt;/p&gt;

&lt;p&gt;Cost questions lead straight into the rest of the Kubernetes interview: how you set requests and limits across a multi-tenant cluster, what happens when a node runs out, and how a tenancy model keeps unit economics sane. &lt;a href="https://devopsinterview.cloud/ebooks/container-orchestration-journey" rel="noopener noreferrer"&gt;Containers &amp;amp; Orchestration&lt;/a&gt; covers that arc with 52 worked questions, follow-up chains and production war stories. It is one of the five books in the &lt;a href="https://devopsinterview.cloud/ebooks/complete-devops-mastery-bundle" rel="noopener noreferrer"&gt;DevOps Interview Mastery bundle&lt;/a&gt;, and every purchase includes the free Interview-Day Playbook with a day-of checklist and behavioural answer frameworks.&lt;/p&gt;

&lt;p&gt;Prefer to watch or listen? Answers like these are walked through on &lt;a href="https://www.youtube.com/@DevOpsInterviewCloud" rel="noopener noreferrer"&gt;YouTube&lt;/a&gt; and on the podcast, on &lt;a href="https://open.spotify.com/show/033txr35oY2Rcn6H4CVZ4S" rel="noopener noreferrer"&gt;Spotify&lt;/a&gt; and &lt;a href="https://music.amazon.com/podcasts/49554d87-91cb-4816-9420-cdac8a06ac75/devops-cloud-interview-prep-real-scenarios-answers" rel="noopener noreferrer"&gt;Amazon Music&lt;/a&gt;.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://devopsinterview.cloud/blog/kubernetes-cost-optimization-200-node-cluster" rel="noopener noreferrer"&gt;devopsinterview.cloud&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>aws</category>
      <category>devops</category>
      <category>infrastructure</category>
      <category>kubernetes</category>
    </item>
    <item>
      <title>DevSecOps Pipeline from Scratch: Gates That Block Bad Code</title>
      <dc:creator>DevOpsInterview.Cloud</dc:creator>
      <pubDate>Mon, 02 Nov 2026 09:20:03 +0000</pubDate>
      <link>https://dev.to/devopsinterview_cloud/devsecops-pipeline-from-scratch-gates-that-block-bad-code-3jkf</link>
      <guid>https://dev.to/devopsinterview_cloud/devsecops-pipeline-from-scratch-gates-that-block-bad-code-3jkf</guid>
      <description>&lt;p&gt;"We want to shift security left. Walk me through how you would build a DevSecOps pipeline from scratch for a Kubernetes-based microservices stack." Naming tools does not answer it. The interviewer wants to know why each layer of the pipeline exists, what it catches, what it misses, and where it blocks a change instead of warning about it. This post builds that answer layer by layer, with configuration you can adapt.&lt;/p&gt;

&lt;p&gt;The video version covers the same pipeline if you prefer to watch it:&lt;/p&gt;

&lt;p&gt;  &lt;iframe src="https://www.youtube.com/embed/njqv0sqVie0" width="710" height="399"&gt;
  &lt;/iframe&gt;
&lt;/p&gt;

&lt;h2&gt;
  
  
  Why do security scans fail to block releases?
&lt;/h2&gt;

&lt;p&gt;A familiar failure goes like this. A team runs a dependency scanner in CI for months. It reports hundreds of findings, none of them fail a build, and the results go out as a weekly report that developers never open. Then a known critical CVE in a transitive dependency gets exploited, and the scanner had been flagging it for weeks.&lt;/p&gt;

&lt;p&gt;Scanning without enforcement is expensive logging. What an interviewer wants to hear is what actually stops a known-bad change from reaching production. Each layer below has a job, a set of blind spots and a specific enforcement point.&lt;/p&gt;

&lt;h2&gt;
  
  
  What are the layers of a DevSecOps pipeline?
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Layer&lt;/th&gt;
&lt;th&gt;Where it runs&lt;/th&gt;
&lt;th&gt;What it catches&lt;/th&gt;
&lt;th&gt;Enforcement point&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Secrets detection&lt;/td&gt;
&lt;td&gt;Pre-commit hook, then CI&lt;/td&gt;
&lt;td&gt;Keys, tokens and passwords in commits&lt;/td&gt;
&lt;td&gt;Required CI check&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SAST&lt;/td&gt;
&lt;td&gt;Every pull request&lt;/td&gt;
&lt;td&gt;Injection flaws and insecure patterns in your own code&lt;/td&gt;
&lt;td&gt;Required CI check&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SCA&lt;/td&gt;
&lt;td&gt;Every pull request&lt;/td&gt;
&lt;td&gt;Known CVEs in direct and transitive dependencies&lt;/td&gt;
&lt;td&gt;Required CI check&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Image scanning and signing&lt;/td&gt;
&lt;td&gt;After the build&lt;/td&gt;
&lt;td&gt;CVEs in OS packages and app dependencies inside the image&lt;/td&gt;
&lt;td&gt;Build fails; only passing images get signed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Admission control&lt;/td&gt;
&lt;td&gt;Kubernetes API server&lt;/td&gt;
&lt;td&gt;Unsigned images, privileged or root pods&lt;/td&gt;
&lt;td&gt;The API server rejects the pod&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The layers overlap on purpose. SAST reads the code you wrote, including how it calls libraries, but it does not check those libraries for known CVEs. SCA knows the libraries but not your logic. Image scanning sees the OS packages in the base image, which neither of the first two looks at. Admission control is the only layer that still applies when someone skips CI and runs &lt;code&gt;kubectl apply&lt;/code&gt; by hand.&lt;/p&gt;

&lt;h2&gt;
  
  
  How do you add secrets scanning and SAST to pull requests?
&lt;/h2&gt;

&lt;p&gt;Secrets detection starts on the developer's laptop with a pre-commit hook; gitleaks ships one (&lt;code&gt;id: gitleaks&lt;/code&gt; in &lt;code&gt;.pre-commit-config.yaml&lt;/code&gt;). Treat the hook as a convenience. Anyone can skip it with &lt;code&gt;git commit --no-verify&lt;/code&gt;, so the enforcing copy of the scan runs in CI.&lt;/p&gt;

&lt;p&gt;This workflow runs Semgrep and gitleaks as parallel jobs on every pull request:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;pr-security&lt;/span&gt;
&lt;span class="na"&gt;on&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;pull_request&lt;/span&gt;
&lt;span class="na"&gt;permissions&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;contents&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;read&lt;/span&gt;
&lt;span class="na"&gt;jobs&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;sast&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;runs-on&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ubuntu-latest&lt;/span&gt;
    &lt;span class="na"&gt;steps&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;uses&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;actions/checkout@v7&lt;/span&gt;
        &lt;span class="na"&gt;with&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;fetch-depth&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Semgrep (new findings only)&lt;/span&gt;
        &lt;span class="na"&gt;env&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;BASE_SHA&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;${{ github.event.pull_request.base.sha }}&lt;/span&gt;
        &lt;span class="na"&gt;run&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;|&lt;/span&gt;
          &lt;span class="s"&gt;pipx install semgrep==1.181.0&lt;/span&gt;
          &lt;span class="s"&gt;semgrep scan --config p/owasp-top-ten --error --baseline-commit "$BASE_SHA"&lt;/span&gt;
  &lt;span class="na"&gt;secrets&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;runs-on&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ubuntu-latest&lt;/span&gt;
    &lt;span class="na"&gt;steps&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;uses&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;actions/checkout@v7&lt;/span&gt;
        &lt;span class="na"&gt;with&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;fetch-depth&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Gitleaks (commits in this PR)&lt;/span&gt;
        &lt;span class="na"&gt;env&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;BASE_SHA&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;${{ github.event.pull_request.base.sha }}&lt;/span&gt;
          &lt;span class="na"&gt;HEAD_SHA&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;${{ github.event.pull_request.head.sha }}&lt;/span&gt;
        &lt;span class="na"&gt;run&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;|&lt;/span&gt;
          &lt;span class="s"&gt;curl -sSfLO https://github.com/gitleaks/gitleaks/releases/download/v8.30.1/gitleaks_8.30.1_linux_x64.tar.gz&lt;/span&gt;
          &lt;span class="s"&gt;echo "551f6fc83ea457d62a0d98237cbad105af8d557003051f41f3e7ca7b3f2470eb  gitleaks_8.30.1_linux_x64.tar.gz" | sha256sum -c -&lt;/span&gt;
          &lt;span class="s"&gt;tar -xzf gitleaks_8.30.1_linux_x64.tar.gz gitleaks&lt;/span&gt;
          &lt;span class="s"&gt;./gitleaks git --redact --log-opts="$BASE_SHA..$HEAD_SHA" .&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;What to point out when you walk through it:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;A red check only blocks if it is required.&lt;/strong&gt; Mark both jobs as required status checks in branch protection or a ruleset. Otherwise the failure is a warning label and you are back to expensive logging.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;--error&lt;/code&gt; is what makes Semgrep a gate.&lt;/strong&gt; &lt;code&gt;semgrep scan&lt;/code&gt; exits 0 on findings by default; &lt;code&gt;--error&lt;/code&gt; makes it exit 1. The old &lt;code&gt;returntocorp/semgrep-action&lt;/code&gt; wrapper is deprecated, so the job installs the CLI directly.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;--baseline-commit&lt;/code&gt; reports only new findings.&lt;/strong&gt; A team adopting Semgrep on an old codebase is not blocked by years of existing debt. That debt still needs tracking, just not in the merge gate.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;fetch-depth: 0&lt;/code&gt; is the simplest reliable choice.&lt;/strong&gt; The checkout default of a single commit has neither the base commit Semgrep compares against nor the commit range gitleaks walks.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Gitleaks scans each commit, not only the final files.&lt;/strong&gt; &lt;code&gt;gitleaks git&lt;/code&gt; with &lt;code&gt;--log-opts&lt;/code&gt; reads every commit in the range, so a key added in one commit and deleted in the next is still caught, and it exits 1 on a leak. The CLI needs no license key or repository secrets, so it also runs on pull requests from forks. If a secret does reach a shared branch, rotate it; rewriting history does not un-leak it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Pin what the gate runs.&lt;/strong&gt; The workflow pins the Semgrep version and checks the gitleaks download against a known SHA-256. In production, also pin actions to full commit SHAs, and keep in mind that a registry ruleset like &lt;code&gt;p/owasp-top-ten&lt;/code&gt; can change between runs unless you vendor it.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These two jobs cover mistakes developers introduce. Vulnerable libraries are the next layer.&lt;/p&gt;

&lt;h2&gt;
  
  
  How do you scan dependencies and container images with Trivy?
&lt;/h2&gt;

&lt;p&gt;Trivy handles both SCA and image scanning. Pointed at the repository, &lt;code&gt;trivy fs&lt;/code&gt; reads lockfiles, so transitive dependencies pinned there are covered. It needs no build, so it can run in parallel with SAST:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;trivy fs &lt;span class="nt"&gt;--scanners&lt;/span&gt; vuln &lt;span class="nt"&gt;--severity&lt;/span&gt; CRITICAL &lt;span class="nt"&gt;--exit-code&lt;/span&gt; 1 &lt;span class="nt"&gt;--ignore-unfixed&lt;/span&gt; &lt;span class="nb"&gt;.&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;After the build, &lt;code&gt;trivy image&lt;/code&gt; scans what actually ships: the OS packages from the base image plus the application dependencies inside it. The order of the next steps matters. Cosign signs an image that already exists in a registry, so the push comes before signing, and you sign the immutable digest rather than a tag that can be moved. &lt;code&gt;set -euo pipefail&lt;/code&gt; makes the script stop at the first failing command, so a failed scan never reaches the push:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;set&lt;/span&gt; &lt;span class="nt"&gt;-euo&lt;/span&gt; pipefail
&lt;span class="nv"&gt;IMAGE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"registry.example.com/myapp:&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;GIT_SHA&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
docker build &lt;span class="nt"&gt;-t&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$IMAGE&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nb"&gt;.&lt;/span&gt;

&lt;span class="c"&gt;# Gate: fail on CRITICAL CVEs that have a fixed version&lt;/span&gt;
trivy image &lt;span class="nt"&gt;--exit-code&lt;/span&gt; 1 &lt;span class="nt"&gt;--severity&lt;/span&gt; CRITICAL &lt;span class="nt"&gt;--ignore-unfixed&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$IMAGE&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;

&lt;span class="c"&gt;# Push only after the scan passes, then sign the pushed digest&lt;/span&gt;
docker push &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$IMAGE&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
&lt;span class="nv"&gt;DIGEST&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;docker inspect &lt;span class="nt"&gt;--format&lt;/span&gt; &lt;span class="s1"&gt;'{{index .RepoDigests 0}}'&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$IMAGE&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;
cosign sign &lt;span class="nt"&gt;--yes&lt;/span&gt; &lt;span class="nt"&gt;--key&lt;/span&gt; &lt;span class="nb"&gt;env&lt;/span&gt;://COSIGN_PRIVATE_KEY &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$DIGEST&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;cosign generate-key-pair&lt;/code&gt; encrypts the private key, so the job also needs &lt;code&gt;COSIGN_PASSWORD&lt;/code&gt; in its environment; store both as protected CI secrets. &lt;code&gt;--yes&lt;/code&gt; only skips the confirmation prompt.&lt;/p&gt;

&lt;p&gt;Each Trivy flag needs a reason you can defend:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;--exit-code 1&lt;/code&gt; is the gate. Without it, Trivy exits 0 even when it finds vulnerabilities, and without &lt;code&gt;set -e&lt;/code&gt; the script carries on regardless.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;--ignore-unfixed&lt;/code&gt; reports only vulnerabilities that have a fixed version, so the build only fails on problems an upgrade can solve. That keeps the gate actionable, but the unfixed findings still need an owner. Run a second, non-blocking scan that writes a full JSON report and track those findings outside the gate.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Image scanning covers userspace packages. Containers share the host's kernel, so kernel vulnerabilities get fixed by patching nodes, not by rebuilding images.&lt;/p&gt;

&lt;p&gt;The base image decides how much there is to scan in the first place. Non-debug distroless images contain no shell and no package manager, which removes a lot of attack surface and makes scan reports much shorter. &lt;code&gt;gcr.io/distroless/static-debian13&lt;/code&gt; only suits statically linked binaries, such as a Go service built with &lt;code&gt;CGO_ENABLED=0&lt;/code&gt;; interpreted languages need the matching distroless runtime image, pinned by digest. The &lt;a href="https://devopsinterview.cloud/blog/docker-image-optimization-multi-stage-builds" rel="noopener noreferrer"&gt;multi-stage build and distroless guide&lt;/a&gt; covers those trade-offs.&lt;/p&gt;

&lt;h2&gt;
  
  
  How do you block unsigned images at Kubernetes admission?
&lt;/h2&gt;

&lt;p&gt;Signing is only useful if something checks the signature. The admission layer rejects pods whose images from your registry lack a valid signature from the pipeline key, which also catches an image someone built on a laptop and pushed by hand.&lt;/p&gt;

&lt;p&gt;Be precise about what a passing check proves. A Cosign signature shows that the holder of the private key signed that exact digest. It only means "this image passed the scan" if the key is available exclusively to the protected CI job that runs the gate. If you need the scan result itself to be verifiable, publish it as a signed attestation and verify that too.&lt;/p&gt;

&lt;p&gt;As of Kyverno 1.19, &lt;code&gt;ClusterPolicy&lt;/code&gt; is deprecated, and &lt;code&gt;ImageValidatingPolicy&lt;/code&gt; replaces its &lt;code&gt;verifyImages&lt;/code&gt; rules:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;policies.kyverno.io/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ImageValidatingPolicy&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;require-signed-images&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;validationActions&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="nv"&gt;Deny&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
  &lt;span class="na"&gt;failurePolicy&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Fail&lt;/span&gt;
  &lt;span class="na"&gt;matchConstraints&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;resourceRules&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;apiGroups&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'&lt;/span&gt;&lt;span class="s"&gt;'&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
        &lt;span class="na"&gt;apiVersions&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'&lt;/span&gt;&lt;span class="s"&gt;v1'&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
        &lt;span class="na"&gt;operations&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'&lt;/span&gt;&lt;span class="s"&gt;CREATE'&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'&lt;/span&gt;&lt;span class="s"&gt;UPDATE'&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
        &lt;span class="na"&gt;resources&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'&lt;/span&gt;&lt;span class="s"&gt;pods'&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
  &lt;span class="na"&gt;matchImageReferences&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;glob&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'&lt;/span&gt;&lt;span class="s"&gt;registry.example.com/*'&lt;/span&gt;
  &lt;span class="na"&gt;attestors&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;cosign&lt;/span&gt;
      &lt;span class="na"&gt;cosign&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;key&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;data&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;|&lt;/span&gt;
            &lt;span class="s"&gt;-----BEGIN PUBLIC KEY-----&lt;/span&gt;
            &lt;span class="s"&gt;(contents of cosign.pub)&lt;/span&gt;
            &lt;span class="s"&gt;-----END PUBLIC KEY-----&lt;/span&gt;
  &lt;span class="na"&gt;validations&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;expression&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;&amp;gt;-&lt;/span&gt;
        &lt;span class="s"&gt;(images.containers + images.?initContainers.orValue([]))&lt;/span&gt;
        &lt;span class="s"&gt;.map(image, verifyImageSignatures(image, [attestors.cosign]))&lt;/span&gt;
        &lt;span class="s"&gt;.all(e, e &amp;gt; 0)&lt;/span&gt;
      &lt;span class="na"&gt;message&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;image is not signed with the pipeline key&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The details that decide whether this is a real gate:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;validationActions: [Deny]&lt;/code&gt; is the point.&lt;/strong&gt; With &lt;code&gt;[Audit]&lt;/code&gt; the pod is admitted and the violation only shows up in a report, which is the scanning-without-enforcement problem moved into the cluster. Starting in Audit to find what would break is a sensible rollout, as long as there is a date for switching to Deny.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Init containers are verified too.&lt;/strong&gt; The expression runs the signature check on init container images as well as &lt;code&gt;images.containers&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Tags get pinned to digests.&lt;/strong&gt; By default (&lt;code&gt;mutateDigest&lt;/code&gt;), Kyverno rewrites a matched image tag to the digest it resolved, so the node runs the digest that was checked even if someone moves the tag afterwards.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Other registries are not checked at all.&lt;/strong&gt; &lt;code&gt;matchImageReferences&lt;/code&gt; limits the policy to images under your registry, so an unsigned image from Docker Hub never reaches the signature check. Pair this with a policy that only allows images from your registry.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Some namespaces are exempt.&lt;/strong&gt; The default Helm install excludes &lt;code&gt;kube-system&lt;/code&gt; and Kyverno's own namespace from its webhooks, so pods created there are never checked. Keep pod creation in those namespaces limited to cluster admins.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Ephemeral containers take a separate path.&lt;/strong&gt; Containers added with &lt;code&gt;kubectl debug&lt;/code&gt; arrive through the &lt;code&gt;pods/ephemeralcontainers&lt;/code&gt; subresource, which a &lt;code&gt;pods&lt;/code&gt; match does not cover. Restrict who can use that subresource with RBAC.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Test it against your real registry.&lt;/strong&gt; Kyverno reads signatures from the registry itself, so a private registry needs a &lt;code&gt;credentials&lt;/code&gt; block: a supported cloud credential provider, or a registry secret in the Kyverno namespace. Cosign 3 also writes signatures in the newer Sigstore bundle format by default, so confirm your Kyverno version verifies what your Cosign version writes. Then prove the gate: deploy an unsigned image and watch it get rejected.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Signature checks say nothing about how a pod runs. For non-root containers, no privileged mode, no privilege escalation and dropped capabilities, the built-in Pod Security Admission &lt;code&gt;restricted&lt;/code&gt; profile is the simplest gate:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl label namespace production pod-security.kubernetes.io/enforce&lt;span class="o"&gt;=&lt;/span&gt;restricted
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;restricted&lt;/code&gt; does not require a read-only root filesystem. If you want that, add a Kyverno or OPA Gatekeeper rule for &lt;code&gt;readOnlyRootFilesystem: true&lt;/code&gt;. Gatekeeper handles pod and registry rules like these well, but it does not verify Cosign signatures on its own; that needs an external data provider such as Ratify. Admission policy works alongside access control, which the &lt;a href="https://devopsinterview.cloud/blog/kubernetes-rbac-least-privilege-lockdown" rel="noopener noreferrer"&gt;Kubernetes RBAC least-privilege guide&lt;/a&gt; covers, and the &lt;a href="https://devopsinterview.cloud/blog/kubernetes-multi-tenancy-shared-cluster-design" rel="noopener noreferrer"&gt;shared-cluster multi-tenancy design&lt;/a&gt; shows how these policies fit a cluster with many teams.&lt;/p&gt;

&lt;h2&gt;
  
  
  How do you keep security scans from slowing down CI?
&lt;/h2&gt;

&lt;p&gt;Every layer adds time to a pull request, and a slow pipeline gets bypassed. The main levers:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Run independent layers in parallel.&lt;/strong&gt; SAST, secrets scanning and &lt;code&gt;trivy fs&lt;/code&gt; need nothing but the source. Only the image scan waits for the build.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cache the Trivy database.&lt;/strong&gt; It is rebuilt every six hours, so downloading it on every run is wasted time. Persist a directory between runs and point Trivy at it with &lt;code&gt;--cache-dir&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Keep SAST diff-aware.&lt;/strong&gt; A gate that fails on findings the pull request did not introduce trains developers to ignore it. &lt;code&gt;--baseline-commit&lt;/code&gt; keeps the report to new findings.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Agree the severity threshold as a team.&lt;/strong&gt; Blocking only on CRITICAL lets serious HIGH findings through. Blocking on HIGH and above creates friction, because plenty of HIGH CVEs sit in transitive dependencies with no reachable exploit path. Decide it together, with an exceptions process, rather than leaving it to whoever writes the workflow.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What does a DevSecOps pipeline miss after deployment?
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Images that were clean when they shipped.&lt;/strong&gt; A scan is a point-in-time result. New CVEs are disclosed against packages your image already contains, and the image never picks up upstream base image patches until you rebuild it. If the base is pinned by digest, rebuilding alone changes nothing: bump the digest when upstream publishes a patched build (a dependency bot such as Renovate can open those pull requests), then rebuild. With a mutable tag, &lt;code&gt;docker build --pull&lt;/code&gt; fetches the current base. Either way, keep scanning what is actually running. Trivy Operator runs in the cluster, scans the images your workloads use and stores the results as &lt;code&gt;VulnerabilityReport&lt;/code&gt; resources. Reports expire after a TTL (24 hours by default) and are regenerated, which is how newly disclosed CVEs show up:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl get vulnerabilityreports &lt;span class="nt"&gt;-n&lt;/span&gt; production &lt;span class="nt"&gt;-o&lt;/span&gt; wide
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Secrets that never touch the repository.&lt;/strong&gt; The secrets and SAST jobs above only see what is in the repository. A database password in a Helm values file kept outside the repo, or pasted into a ConfigMap, is invisible to them. That needs a different control: keep secrets in Vault or a cloud secrets manager, sync them into Kubernetes Secrets with the External Secrets Operator, and enable encryption at rest for Secrets in etcd.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Gates nobody has tested.&lt;/strong&gt; Make each gate fail at least once on purpose: open a pull request containing a dummy credential that matches a gitleaks rule, push an unsigned image and try to run it. A gate you have never seen block anything may still be in Audit mode.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why generate an SBOM for every image?
&lt;/h2&gt;

&lt;p&gt;A software bill of materials lists the packages in an image, as far as the generator can detect them; vendored or statically compiled code can slip past. Syft generates one in CycloneDX or SPDX format, and Cosign attaches it to the image as a signed attestation. The older &lt;code&gt;cosign attach sbom&lt;/code&gt; route is deprecated.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;syft &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$DIGEST&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;-o&lt;/span&gt; cyclonedx-json &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; sbom.cdx.json
cosign attest &lt;span class="nt"&gt;--yes&lt;/span&gt; &lt;span class="nt"&gt;--key&lt;/span&gt; &lt;span class="nb"&gt;env&lt;/span&gt;://COSIGN_PRIVATE_KEY &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--type&lt;/span&gt; cyclonedx &lt;span class="nt"&gt;--predicate&lt;/span&gt; sbom.cdx.json &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$DIGEST&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The payoff comes when the next Log4Shell-class vulnerability is disclosed. Instead of searching every repository for log4j, you ask the stored SBOMs which images contain &lt;code&gt;log4j-core&lt;/code&gt;, and at which versions:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;jq &lt;span class="nt"&gt;-r&lt;/span&gt; &lt;span class="s1"&gt;'.components[] | select(.name == "log4j-core") | .version'&lt;/span&gt; sbom.cdx.json

&lt;span class="c"&gt;# Match an SBOM against current vulnerability data without pulling the image&lt;/span&gt;
grype sbom:./sbom.cdx.json
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;One file at a time does not scale, so store SBOMs centrally. OWASP Dependency-Track is built for this, and it answers the question across every image at once.&lt;/p&gt;

&lt;h2&gt;
  
  
  DevSecOps pipeline checklist
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Gitleaks as a pre-commit hook for convenience and as a required CI check for enforcement.&lt;/li&gt;
&lt;li&gt;Diff-aware Semgrep with &lt;code&gt;--error&lt;/code&gt; on every pull request, as a required check.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;trivy fs&lt;/code&gt; on dependencies, in parallel with SAST.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;trivy image --exit-code 1&lt;/code&gt; after the build, then push and sign the digest with Cosign.&lt;/li&gt;
&lt;li&gt;An &lt;code&gt;ImageValidatingPolicy&lt;/code&gt; in Deny mode, an allowed-registries policy, and Pod Security Admission &lt;code&gt;restricted&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Base image updates that trigger rebuilds, plus in-cluster rescanning of running images.&lt;/li&gt;
&lt;li&gt;An SBOM attestation for every image, stored where you can query it.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Interview questions this covers
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Walk me through a DevSecOps pipeline you would build from scratch for Kubernetes microservices.&lt;/li&gt;
&lt;li&gt;What is the difference between SAST, SCA and image scanning, and what does each one miss?&lt;/li&gt;
&lt;li&gt;How do you stop unsigned images from running in a cluster, and what does a valid signature actually prove?&lt;/li&gt;
&lt;li&gt;An image passed its scan three weeks ago. Is it still safe to run?&lt;/li&gt;
&lt;li&gt;A critical CVE is disclosed in a popular library. How do you find out which production images are affected?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For more practice at this depth, try the &lt;a href="https://devopsinterview.cloud/blog/kubernetes-interview-questions-2026" rel="noopener noreferrer"&gt;Kubernetes interview questions for 2026&lt;/a&gt; or browse &lt;a href="https://devopsinterview.cloud/ebooks" rel="noopener noreferrer"&gt;all the DevOps interview ebooks&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Pipeline security questions in real interviews go several follow-ups deeper: the supply chain threat model, gating releases on SBOM scans, and rolling DevSecOps out without making the security team the bottleneck. &lt;a href="https://devopsinterview.cloud/ebooks/modern-cicd-gitops" rel="noopener noreferrer"&gt;CI/CD &amp;amp; GitOps&lt;/a&gt; works through 50 questions with worked answers, including chapters on pipeline and supply-chain security (SBOM, Sigstore and SLSA) and on DevSecOps and compliance as code. It is one of the five books in the &lt;a href="https://devopsinterview.cloud/ebooks/complete-devops-mastery-bundle" rel="noopener noreferrer"&gt;DevOps Interview Mastery bundle&lt;/a&gt;, and every purchase includes the free Interview-Day Playbook with a day-of checklist and behavioural answer frameworks.&lt;/p&gt;

&lt;p&gt;Prefer to watch or listen? Answers like these are walked through on &lt;a href="https://www.youtube.com/@DevOpsInterviewCloud" rel="noopener noreferrer"&gt;YouTube&lt;/a&gt; and on the podcast, on &lt;a href="https://open.spotify.com/show/033txr35oY2Rcn6H4CVZ4S" rel="noopener noreferrer"&gt;Spotify&lt;/a&gt; and &lt;a href="https://music.amazon.com/podcasts/49554d87-91cb-4816-9420-cdac8a06ac75/devops-cloud-interview-prep-real-scenarios-answers" rel="noopener noreferrer"&gt;Amazon Music&lt;/a&gt;.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://devopsinterview.cloud/blog/devsecops-pipeline-from-scratch" rel="noopener noreferrer"&gt;devopsinterview.cloud&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>cicd</category>
      <category>devops</category>
      <category>kubernetes</category>
      <category>security</category>
    </item>
    <item>
      <title>Terraform Mono-Repo Design for 20 AWS Accounts</title>
      <dc:creator>DevOpsInterview.Cloud</dc:creator>
      <pubDate>Sun, 01 Nov 2026 09:20:02 +0000</pubDate>
      <link>https://dev.to/devopsinterview_cloud/terraform-mono-repo-design-for-20-aws-accounts-1aae</link>
      <guid>https://dev.to/devopsinterview_cloud/terraform-mono-repo-design-for-20-aws-accounts-1aae</guid>
      <description>&lt;p&gt;"Your platform team manages twenty AWS accounts: dev, staging, prod and a handful of product-team sandboxes. All of the infrastructure is Terraform. Structure a single repository so that one engineer can safely deploy to one account without touching another. Walk me through the design." The prompt is deliberately open-ended. A strong answer is not about knowing a CLI flag; it shows how you reason about isolation, blast radius and team safety at a scale where a copy-paste mistake reaches production. There is also a tempting first answer, "use workspaces, one per account", and the follow-up questions probe whether you know where that holds and where it breaks: for deployments that need separate credentials and access controls, CLI workspaces alone are unsuitable.&lt;/p&gt;

&lt;p&gt;The video version walks the same design if you prefer to watch it:&lt;/p&gt;

&lt;p&gt;  &lt;iframe src="https://www.youtube.com/embed/UkupNeFXGAo" width="710" height="399"&gt;
  &lt;/iframe&gt;
&lt;/p&gt;

&lt;h2&gt;
  
  
  Why "one folder, twenty workspaces" fails
&lt;/h2&gt;

&lt;p&gt;Be precise about what workspaces share, because the imprecise version ("they share state") is common and wrong. Each CLI workspace does get its own state file. With the S3 backend the default workspace is stored at the configured &lt;code&gt;key&lt;/code&gt;, and every other workspace is stored in the same bucket under the &lt;code&gt;env:/&lt;/code&gt; prefix, followed by the workspace name and then the key. A corrupted &lt;code&gt;sandbox&lt;/code&gt; state does not corrupt &lt;code&gt;prod&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;What every workspace shares is everything around the state: the same backend configuration and bucket, the same provider blocks, the same variable declarations, and whatever credentials that shared configuration resolves to. You can vary values with &lt;code&gt;terraform.workspace&lt;/code&gt; conditionals, but those conditionals live in the same root module, so a mistake in the shared parts reaches every environment. HashiCorp's own documentation says it plainly: workspaces within a working directory use the same backend, so they are not a suitable isolation mechanism when deployments need separate credentials and access controls. Add the fact that &lt;code&gt;terraform workspace select&lt;/code&gt; is mutable state in the operator's shell, easy to forget in a second terminal during an incident, and you have the real problem.&lt;/p&gt;

&lt;p&gt;The design therefore has to deliver three things:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;State isolation per account.&lt;/strong&gt; A locked, corrupted or mid-migration state in account A must not block or affect account B.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Independent authentication per account.&lt;/strong&gt; Each account's Terraform runs with credentials scoped to that account, obtained by role assumption rather than long-lived keys.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;CI that scopes itself.&lt;/strong&gt; A pull request that touches one account plans and applies against that account, not all twenty.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Repo layout: one root module per account
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;infra-repo/
├── modules/
│   ├── vpc/
│   ├── eks/
│   └── iam-baseline/
├── accounts/
│   ├── prod-111111111111/
│   │   ├── backend.tf
│   │   ├── providers.tf
│   │   ├── main.tf
│   │   └── terraform.tfvars
│   ├── staging-222222222222/
│   └── sandbox-eng-333333333333/
└── .github/workflows/terraform.yml
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two top-level directories carry the design. &lt;code&gt;modules/&lt;/code&gt; holds reusable child modules: the VPC, the EKS cluster, the IAM baseline every account receives. They declare resources, variables, outputs and their &lt;code&gt;required_providers&lt;/code&gt;, but they contain no &lt;code&gt;backend&lt;/code&gt; block and no &lt;code&gt;provider&lt;/code&gt; configuration blocks; the calling root module supplies those. &lt;code&gt;accounts/&lt;/code&gt; holds one subdirectory per AWS account, named with a human-readable label plus the account ID so nobody has to remember which twelve-digit number is prod. Each account directory is a complete, independent Terraform root module with its own backend, its own provider configuration, its own tfvars and its own calls into the shared modules.&lt;/p&gt;

&lt;p&gt;One detail to say out loud, because a common variant of this layout gets it wrong: there is no shared &lt;code&gt;globals/variables.tf&lt;/code&gt;. Terraform only loads the &lt;code&gt;.tf&lt;/code&gt; files in the root module's own directory, so a variable declared in a sibling directory is never read. Values that genuinely apply to every account belong in a small module that returns them as outputs, or in a common tfvars file that each pipeline passes with &lt;code&gt;-var-file&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Backend and provider per account
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight hcl"&gt;&lt;code&gt;&lt;span class="c1"&gt;# accounts/prod-111111111111/backend.tf&lt;/span&gt;
&lt;span class="nx"&gt;terraform&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;required_version&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"&amp;gt;= 1.10, &amp;lt; 2.0"&lt;/span&gt;   &lt;span class="c1"&gt;# S3 native locking arrived in 1.10&lt;/span&gt;

  &lt;span class="nx"&gt;backend&lt;/span&gt; &lt;span class="s2"&gt;"s3"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nx"&gt;bucket&lt;/span&gt;       &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"tfstate-prod-111111111111"&lt;/span&gt;
    &lt;span class="nx"&gt;key&lt;/span&gt;          &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"prod/terraform.tfstate"&lt;/span&gt;
    &lt;span class="nx"&gt;region&lt;/span&gt;       &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"us-east-1"&lt;/span&gt;
    &lt;span class="nx"&gt;encrypt&lt;/span&gt;      &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
    &lt;span class="nx"&gt;use_lockfile&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
    &lt;span class="c1"&gt;# assume_role is supplied at init time from one of the two files below&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="c1"&gt;# accounts/prod-111111111111/backend.plan.hcl   (read state, hold the lock)&lt;/span&gt;
&lt;span class="nx"&gt;assume_role&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;role_arn&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"arn:aws:iam::111111111111:role/TerraformPlanRole"&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="c1"&gt;# accounts/prod-111111111111/backend.apply.hcl  (read and write state)&lt;/span&gt;
&lt;span class="nx"&gt;assume_role&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;role_arn&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"arn:aws:iam::111111111111:role/TerraformDeployRole"&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="c1"&gt;# accounts/prod-111111111111/providers.tf&lt;/span&gt;
&lt;span class="nx"&gt;variable&lt;/span&gt; &lt;span class="s2"&gt;"terraform_role_name"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;type&lt;/span&gt;        &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;string&lt;/span&gt;
  &lt;span class="nx"&gt;description&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"TerraformPlanRole for pull-request plans, TerraformDeployRole for applies from main"&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="nx"&gt;provider&lt;/span&gt; &lt;span class="s2"&gt;"aws"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;region&lt;/span&gt;              &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"us-east-1"&lt;/span&gt;
  &lt;span class="nx"&gt;allowed_account_ids&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"111111111111"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

  &lt;span class="nx"&gt;assume_role&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nx"&gt;role_arn&lt;/span&gt;     &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"arn:aws:iam::111111111111:role/${var.terraform_role_name}"&lt;/span&gt;
    &lt;span class="nx"&gt;session_name&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"terraform"&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The credential model is one chain with two rungs, and pull-request plans and applies from &lt;code&gt;main&lt;/code&gt; use different roles at every rung. In CI, the workflow authenticates via OpenID Connect to one of two roles in a central tooling account: &lt;code&gt;GitHubActionsPlan&lt;/code&gt; for pull-request runs and &lt;code&gt;GitHubActionsApply&lt;/code&gt; for applies from &lt;code&gt;main&lt;/code&gt;. Both trust policies restrict the token to this repository, and the apply role's additionally to the protected environment or branch its job runs under. Each account then carries a matching pair. &lt;code&gt;TerraformPlanRole&lt;/code&gt; trusts the central plan role and has read-only infrastructure access plus read and lock access to state; &lt;code&gt;TerraformDeployRole&lt;/code&gt; trusts only the central apply role and can change infrastructure and write state. The backend and the provider each assume the account role for the run, because the backend inherits nothing from the provider: without its own &lt;code&gt;assume_role&lt;/code&gt; (an object attribute in the backend, unlike the nested block in the provider, and supplied here through a &lt;code&gt;-backend-config&lt;/code&gt; file because backend blocks cannot read variables), &lt;code&gt;terraform init&lt;/code&gt; would try to read the state bucket with the caller's ambient credentials, which have no rights in the target account. Humans do not get a third path. Applies are CI-only, and an engineer who needs a local plan uses an IAM Identity Center permission set that each &lt;code&gt;TerraformPlanRole&lt;/code&gt; trust policy lists alongside the central plan role, so a laptop can plan but never apply.&lt;/p&gt;

&lt;p&gt;The split exists because &lt;code&gt;terraform init&lt;/code&gt; and &lt;code&gt;terraform plan&lt;/code&gt; execute whatever HCL the pull request contains, including the providers it downloads, so pull-request code inherits the plan job's credentials. Those credentials must be unable to change infrastructure or overwrite state. What a plan still needs, and therefore what pull-request code can do, is read state and hold the lock; if the repository accepts pull requests from people who must not read state, gate the plan job behind an approval as well. Four more points here are interview material.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The state lives inside the account it describes.&lt;/strong&gt; The prod bucket is owned by the prod account, so IAM policies in the prod account, together with the bucket policy, decide who can read or write the prod state. Nothing outside the account is granted S3 access directly; the central roles only get &lt;code&gt;sts:AssumeRole&lt;/code&gt; into each account.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Locking no longer needs DynamoDB.&lt;/strong&gt; &lt;code&gt;use_lockfile = true&lt;/code&gt; uses S3 conditional writes to hold a &lt;code&gt;.tflock&lt;/code&gt; object next to the state. Terraform 1.10 introduced it and 1.11 made it generally available; DynamoDB-based locking via &lt;code&gt;dynamodb_table&lt;/code&gt; is deprecated. The S3 permissions are small: &lt;code&gt;s3:ListBucket&lt;/code&gt; on the bucket, &lt;code&gt;s3:GetObject&lt;/code&gt; and &lt;code&gt;s3:PutObject&lt;/code&gt; on the state object, and &lt;code&gt;s3:GetObject&lt;/code&gt;, &lt;code&gt;s3:PutObject&lt;/code&gt; and &lt;code&gt;s3:DeleteObject&lt;/code&gt; on the &lt;code&gt;.tflock&lt;/code&gt; object. Terraform never deletes the state file, so the plan role gets that whole set minus &lt;code&gt;s3:PutObject&lt;/code&gt; on the state object. If you mention DynamoDB locking in an interview, present it as the previous approach.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;allowed_account_ids&lt;/code&gt; is the wrong-account guard.&lt;/strong&gt; The provider refuses to run if the credentials resolve to any other account. It costs one line and catches the pasted-from-staging role ARN before a plan runs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Be honest about what a directory is.&lt;/strong&gt; The directory is the configuration and state boundary: it makes the target explicit and removes the hidden workspace-selection step. It is not a security boundary. A pull request can contain any HCL its author likes, including a provider block that names another account's role, so the ceiling on what a run can do is set by IAM: which roles the central identity is trusted to assume, and what those roles are allowed to do.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  CI: plan and apply only the accounts a PR touched
&lt;/h2&gt;

&lt;p&gt;Running plans against all twenty accounts on every pull request is slow and noisy, and a plan failure in an account the PR never touched is a false red build. Changed-path detection fixes that. A naive version has three bugs, so the excerpt below is longer than the one you would sketch on a whiteboard.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="c1"&gt;# .github/workflows/terraform.yml (excerpt)&lt;/span&gt;
&lt;span class="na"&gt;on&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;pull_request&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;paths&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;accounts/**"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;modules/**"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;

&lt;span class="na"&gt;permissions&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;id-token&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;write&lt;/span&gt;   &lt;span class="c1"&gt;# required for OIDC&lt;/span&gt;
  &lt;span class="na"&gt;contents&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;read&lt;/span&gt;

&lt;span class="na"&gt;jobs&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;detect-changes&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;runs-on&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ubuntu-latest&lt;/span&gt;
    &lt;span class="na"&gt;outputs&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;accounts&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;${{ steps.changed.outputs.accounts }}&lt;/span&gt;
    &lt;span class="na"&gt;steps&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;uses&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;actions/checkout@v4&lt;/span&gt;
        &lt;span class="na"&gt;with&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;fetch-depth&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt;   &lt;span class="c1"&gt;# the default depth of 1 has no base branch to diff against&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;id&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;changed&lt;/span&gt;
        &lt;span class="na"&gt;shell&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;bash&lt;/span&gt;
        &lt;span class="na"&gt;run&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;|&lt;/span&gt;
          &lt;span class="s"&gt;CHANGED=$(git diff --name-only "origin/${{ github.base_ref }}...HEAD")&lt;/span&gt;
          &lt;span class="s"&gt;if grep -q '^modules/' &amp;lt;&amp;lt;&amp;lt;"$CHANGED"; then&lt;/span&gt;
            &lt;span class="s"&gt;# a shared module changed: every account that consumes it needs a plan&lt;/span&gt;
            &lt;span class="s"&gt;ACCOUNTS=$(ls -d accounts/*/ | cut -d/ -f2)&lt;/span&gt;
          &lt;span class="s"&gt;else&lt;/span&gt;
            &lt;span class="s"&gt;ACCOUNTS=$({ grep -E '^accounts/[^/]+/' &amp;lt;&amp;lt;&amp;lt;"$CHANGED" || true; } | cut -d/ -f2 | sort -u)&lt;/span&gt;
          &lt;span class="s"&gt;fi&lt;/span&gt;
          &lt;span class="s"&gt;# only roots that still exist can be planned; removing an account is a&lt;/span&gt;
          &lt;span class="s"&gt;# separate teardown workflow, not a directory deletion&lt;/span&gt;
          &lt;span class="s"&gt;ACCOUNTS=$(for a in $ACCOUNTS; do if [ -d "accounts/$a" ]; then echo "$a"; fi; done)&lt;/span&gt;
          &lt;span class="s"&gt;JSON=$(printf '%s\n' $ACCOUNTS | jq -R . | jq -sc 'map(select(length &amp;gt; 0))')&lt;/span&gt;
          &lt;span class="s"&gt;echo "accounts=$JSON" &amp;gt;&amp;gt; "$GITHUB_OUTPUT"&lt;/span&gt;

  &lt;span class="na"&gt;plan&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;needs&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;detect-changes&lt;/span&gt;
    &lt;span class="na"&gt;if&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;needs.detect-changes.outputs.accounts != '[]'&lt;/span&gt;   &lt;span class="c1"&gt;# an empty matrix fails the job&lt;/span&gt;
    &lt;span class="na"&gt;runs-on&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ubuntu-latest&lt;/span&gt;
    &lt;span class="na"&gt;strategy&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;fail-fast&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;   &lt;span class="c1"&gt;# one account's failed plan must not cancel the others&lt;/span&gt;
      &lt;span class="na"&gt;matrix&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;account&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;${{ fromJson(needs.detect-changes.outputs.accounts) }}&lt;/span&gt;
    &lt;span class="na"&gt;env&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;TF_VAR_terraform_role_name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;TerraformPlanRole&lt;/span&gt;   &lt;span class="c1"&gt;# read-only chain for PR code&lt;/span&gt;
    &lt;span class="na"&gt;steps&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;uses&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;actions/checkout@v4&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;uses&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;hashicorp/setup-terraform@v3&lt;/span&gt;
        &lt;span class="na"&gt;with&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;terraform_version&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;1.16.2&lt;/span&gt;   &lt;span class="c1"&gt;# the default is latest; pin and bump deliberately&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;uses&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;aws-actions/configure-aws-credentials@v4&lt;/span&gt;
        &lt;span class="na"&gt;with&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;role-to-assume&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;arn:aws:iam::999999999999:role/GitHubActionsPlan&lt;/span&gt;
          &lt;span class="na"&gt;aws-region&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;us-east-1&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;run&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;terraform init -input=false -backend-config=backend.plan.hcl &amp;amp;&amp;amp; terraform plan -input=false&lt;/span&gt;
        &lt;span class="na"&gt;working-directory&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;accounts/${{ matrix.account }}&lt;/span&gt;

  &lt;span class="c1"&gt;# Apply lives in a separate workflow triggered on push to main. It runs inside a&lt;/span&gt;
  &lt;span class="c1"&gt;# protected environment, assumes GitHubActionsApply, sets TerraformDeployRole and&lt;/span&gt;
  &lt;span class="c1"&gt;# inits with backend.apply.hcl.&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The three fixes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Shallow checkouts have nothing to diff against.&lt;/strong&gt; &lt;code&gt;actions/checkout&lt;/code&gt; fetches a single commit by default, so a diff against the base branch fails. Fetch the history and diff against the PR's base ref rather than a hardcoded &lt;code&gt;main&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A module change touches no account directory.&lt;/strong&gt; With relative module sources, editing &lt;code&gt;modules/vpc&lt;/code&gt; changes files only under &lt;code&gt;modules/&lt;/code&gt;, so naive path filtering plans nothing and the breaking change surfaces on the next unrelated PR to each account. Fan out to every account when &lt;code&gt;modules/&lt;/code&gt; changes, or maintain an explicit module-to-consumers map. Atlantis handles this with per-project &lt;code&gt;when_modified&lt;/code&gt; globs that can reference module paths.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;An empty matrix is a failed job.&lt;/strong&gt; GitHub Actions refuses to expand a matrix whose vector has no values, so a PR that only touched a stray file under &lt;code&gt;accounts/&lt;/code&gt; would fail the plan job without the &lt;code&gt;if&lt;/code&gt; guard.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Two smaller reproducibility points belong in the same breath: pin the Terraform version in CI, since &lt;code&gt;setup-terraform&lt;/code&gt; installs the latest release when none is given, and commit each root's &lt;code&gt;.terraform.lock.hcl&lt;/code&gt; with provider version constraints so a plan on Tuesday uses the same provider builds as the apply on Wednesday.&lt;/p&gt;

&lt;p&gt;Approval tiers are configuration, not code. Put the apply job behind GitHub Environments: no reviewers for sandbox accounts, required reviewers plus a deployment-branch rule restricting deploys to &lt;code&gt;main&lt;/code&gt; for staging and production. Check what your GitHub plan offers for private repositories before promising this in an interview, since required reviewers are not available on every plan. Atlantis covers the same ground with per-project &lt;code&gt;apply_requirements&lt;/code&gt; such as &lt;code&gt;approved&lt;/code&gt; and &lt;code&gt;mergeable&lt;/code&gt;, and it plans only the projects whose files a PR modified, which is the changed-path logic above without the shell script.&lt;/p&gt;

&lt;h2&gt;
  
  
  Versioning shared modules inside the same repo
&lt;/h2&gt;

&lt;p&gt;A natural follow-up: the modules live in the repo, so how do twenty accounts adopt a module change at twenty different speeds?&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight hcl"&gt;&lt;code&gt;&lt;span class="c1"&gt;# Option A: relative path. Every account tracks the module at the current commit.&lt;/span&gt;
&lt;span class="nx"&gt;module&lt;/span&gt; &lt;span class="s2"&gt;"vpc"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;source&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"../../modules/vpc"&lt;/span&gt;
  &lt;span class="nx"&gt;cidr&lt;/span&gt;   &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"10.0.0.0/16"&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="c1"&gt;# Option B: pinned tag. Each account upgrades when its owners choose to.&lt;/span&gt;
&lt;span class="nx"&gt;module&lt;/span&gt; &lt;span class="s2"&gt;"vpc"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;source&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"git::https://github.com/org/infra-repo.git//modules/vpc?ref=vpc-v1.4.2"&lt;/span&gt;
  &lt;span class="nx"&gt;cidr&lt;/span&gt;   &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"10.0.0.0/16"&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Relative paths are simple and refactor-friendly, and they are the right default while the module set is young. Their cost is coupling: a breaking change to the VPC module becomes part of every account's configuration the moment it merges, and the CI fan-out above is what turns that from a surprise into twenty visible plans. Pinned tags decouple accounts from module changes: sandbox can move to the new tag the day it is cut while prod waits. Renovate's Terraform manager understands git-tag module sources and can open the upgrade PRs, with one caveat: all modules in one repo share a single tag namespace, so per-module prefixes such as &lt;code&gt;vpc-v1.4.2&lt;/code&gt; and &lt;code&gt;eks-v2.0.0&lt;/code&gt; need a Renovate rule that maps each module to its own tag pattern, or you version the whole module set together, or you publish modules to a registry. The costs are real too: &lt;code&gt;terraform init&lt;/code&gt; now clones the repository once per pinned module instead of reading a local directory, the runner needs Git credentials for a repo it already has checked out, and someone owns tagging and release notes. Pinned tags for prod-tier accounts and relative paths for the sandboxes is a defensible middle ground.&lt;/p&gt;

&lt;h2&gt;
  
  
  The pushback: "why not just workspaces?"
&lt;/h2&gt;

&lt;p&gt;If this lands after you have explained the layout, answer in terms of failure modes rather than taste. Workspaces share one backend configuration and one set of provider blocks; per-workspace conditionals can vary the values, but they are not a credential or access-control boundary, and a mistake in the shared parts reaches every environment. Workspace selection is mutable state in the operator's shell (or the &lt;code&gt;TF_WORKSPACE&lt;/code&gt; variable), which is exactly the kind of thing that goes wrong under pressure. Directories remove that hidden step: the configuration in front of you names the account it targets, and &lt;code&gt;allowed_account_ids&lt;/code&gt; plus IAM stop a wrong-account run from succeeding.&lt;/p&gt;

&lt;p&gt;Then give workspaces their due. Short-lived, near-identical environments inside a single account, such as a per-branch preview stack torn down when the PR closes, are what workspaces are good at. Account-level isolation across twenty accounts with different credentials is not.&lt;/p&gt;

&lt;h2&gt;
  
  
  The gotcha: drift across twenty accounts goes unnoticed
&lt;/h2&gt;

&lt;p&gt;With this many accounts, someone will change something in the console. A security engineer tightens a security group during an incident; a developer attaches an IAM policy to unblock a deploy. State and reality now disagree, and Terraform will not tell you until an unrelated PR triggers a plan for that account and the diff includes changes nobody proposed. The CLI does not schedule drift checks for you (HCP Terraform's health assessments do), so with plain Terraform you schedule them.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/usr/bin/env bash&lt;/span&gt;
&lt;span class="c"&gt;# nightly-drift-check.sh: report tracked objects whose live values differ from state&lt;/span&gt;
&lt;span class="nb"&gt;set&lt;/span&gt; &lt;span class="nt"&gt;-u&lt;/span&gt;
&lt;span class="nb"&gt;export &lt;/span&gt;&lt;span class="nv"&gt;TF_VAR_terraform_role_name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;TerraformPlanRole   &lt;span class="c"&gt;# read-only chain is enough&lt;/span&gt;
&lt;span class="nv"&gt;ROOT&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;pwd&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;
&lt;span class="nb"&gt;mkdir&lt;/span&gt; &lt;span class="nt"&gt;-p&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$ROOT&lt;/span&gt;&lt;span class="s2"&gt;/drift-logs"&lt;/span&gt;
&lt;span class="nv"&gt;drifted&lt;/span&gt;&lt;span class="o"&gt;=()&lt;/span&gt;
&lt;span class="nv"&gt;errors&lt;/span&gt;&lt;span class="o"&gt;=()&lt;/span&gt;
&lt;span class="k"&gt;for &lt;/span&gt;&lt;span class="nb"&gt;dir &lt;/span&gt;&lt;span class="k"&gt;in &lt;/span&gt;accounts/&lt;span class="k"&gt;*&lt;/span&gt;/&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do
  &lt;/span&gt;&lt;span class="nv"&gt;account&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;basename&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$dir&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;
  &lt;span class="o"&gt;(&lt;/span&gt;
    &lt;span class="nb"&gt;cd&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$dir&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nb"&gt;exit &lt;/span&gt;1
    terraform init &lt;span class="nt"&gt;-input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nb"&gt;false&lt;/span&gt; &lt;span class="nt"&gt;-no-color&lt;/span&gt; &lt;span class="nt"&gt;-backend-config&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;backend.plan.hcl &lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;
      terraform plan &lt;span class="nt"&gt;-refresh-only&lt;/span&gt; &lt;span class="nt"&gt;-detailed-exitcode&lt;/span&gt; &lt;span class="nt"&gt;-input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nb"&gt;false&lt;/span&gt; &lt;span class="nt"&gt;-no-color&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
        &lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$ROOT&lt;/span&gt;&lt;span class="s2"&gt;/drift-logs/&lt;/span&gt;&lt;span class="nv"&gt;$account&lt;/span&gt;&lt;span class="s2"&gt;.log"&lt;/span&gt; 2&amp;gt;&amp;amp;1
  &lt;span class="o"&gt;)&lt;/span&gt;
  &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="nv"&gt;$?&lt;/span&gt; &lt;span class="k"&gt;in
    &lt;/span&gt;0&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"clean   &lt;/span&gt;&lt;span class="nv"&gt;$account&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="p"&gt;;;&lt;/span&gt;
    2&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"DRIFT   &lt;/span&gt;&lt;span class="nv"&gt;$account&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; drifted+&lt;span class="o"&gt;=(&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$account&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="p"&gt;;;&lt;/span&gt;
    &lt;span class="k"&gt;*&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"ERROR   &lt;/span&gt;&lt;span class="nv"&gt;$account&lt;/span&gt;&lt;span class="s2"&gt; (init or plan failed)"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; errors+&lt;span class="o"&gt;=(&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$account&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="p"&gt;;;&lt;/span&gt;
  &lt;span class="k"&gt;esac&lt;/span&gt;
&lt;span class="k"&gt;done&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="k"&gt;${#&lt;/span&gt;&lt;span class="nv"&gt;drifted&lt;/span&gt;&lt;span class="p"&gt;[@]&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt; &lt;span class="nt"&gt;-eq&lt;/span&gt; 0 &lt;span class="o"&gt;]&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nb"&gt;printf&lt;/span&gt; &lt;span class="s1"&gt;'drifted: %s\n'&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;drifted&lt;/span&gt;&lt;span class="p"&gt;[*]&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="k"&gt;${#&lt;/span&gt;&lt;span class="nv"&gt;errors&lt;/span&gt;&lt;span class="p"&gt;[@]&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt; &lt;span class="nt"&gt;-eq&lt;/span&gt; 0 &lt;span class="o"&gt;]&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt; &lt;span class="nb"&gt;printf&lt;/span&gt; &lt;span class="s1"&gt;'failed: %s\n'&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;errors&lt;/span&gt;&lt;span class="p"&gt;[*]&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nb"&gt;exit &lt;/span&gt;1&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="k"&gt;${#&lt;/span&gt;&lt;span class="nv"&gt;drifted&lt;/span&gt;&lt;span class="p"&gt;[@]&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt; &lt;span class="nt"&gt;-eq&lt;/span&gt; 0 &lt;span class="o"&gt;]&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nb"&gt;exit &lt;/span&gt;2
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three details matter. First, &lt;code&gt;-detailed-exitcode&lt;/code&gt; returns 0 for no changes, 1 for an error and 2 for changes present. On a plain &lt;code&gt;terraform plan&lt;/code&gt;, exit 2 also fires for configuration that merged but was never applied, which is not drift; &lt;code&gt;-refresh-only&lt;/code&gt; restricts the comparison to state versus what the providers currently report. Read the log rather than paging on the exit code alone, since output-only changes and provider normalisation can show up there too. Second, be clear about the limit: a refresh compares objects Terraform already tracks, so a resource somebody created by hand outside Terraform is invisible to it. Third, a failed &lt;code&gt;cd&lt;/code&gt;, init or plan must be reported as an error rather than counted as clean, which is what the chaining and the separate error list are for. And because this is a read-only path, it runs on the plan chain; the deploy roles are never involved.&lt;/p&gt;

&lt;h2&gt;
  
  
  Bootstrapping account twenty-one
&lt;/h2&gt;

&lt;p&gt;The last operational question: how does a brand-new account get its first Terraform run? The state bucket has to exist before &lt;code&gt;terraform init&lt;/code&gt; can use it, and the account roles have to exist before anything can assume them, and you want both created by code. There are two workable approaches.&lt;/p&gt;

&lt;p&gt;The organisation-level answer: a new member account already contains a role the management account can assume, &lt;code&gt;OrganizationAccountAccessRole&lt;/code&gt; for accounts created through AWS Organizations and &lt;code&gt;AWSControlTowerExecution&lt;/code&gt; for accounts enrolled through Control Tower. A CloudFormation StackSet with service-managed permissions, or a Control Tower customization, then creates the state bucket and the Terraform role in every new account uniformly. Terraform never has to solve its own chicken-and-egg problem because the bootstrap happens before Terraform arrives.&lt;/p&gt;

&lt;p&gt;The Terraform-only answer: a tiny bootstrap root module inside the account directory, run once with the local backend using the organization access role, that creates the bucket and the two roles, and is then migrated into that bucket under its own state key so it never collides with the account root's state.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight hcl"&gt;&lt;code&gt;&lt;span class="c1"&gt;# accounts/sandbox-data-444444444444/bootstrap/main.tf&lt;/span&gt;
&lt;span class="c1"&gt;# Run once with the local backend. Then replace it with the s3 backend using&lt;/span&gt;
&lt;span class="c1"&gt;# key = "bootstrap/terraform.tfstate" and run:&lt;/span&gt;
&lt;span class="c1"&gt;#   terraform init -migrate-state -backend-config=backend.bootstrap.hcl&lt;/span&gt;
&lt;span class="c1"&gt;# where that file sets assume_role to the same OrganizationAccountAccessRole,&lt;/span&gt;
&lt;span class="c1"&gt;# because the backend authenticates separately from the provider below.&lt;/span&gt;
&lt;span class="nx"&gt;terraform&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;backend&lt;/span&gt; &lt;span class="s2"&gt;"local"&lt;/span&gt; &lt;span class="p"&gt;{}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="nx"&gt;provider&lt;/span&gt; &lt;span class="s2"&gt;"aws"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;region&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"us-east-1"&lt;/span&gt;
  &lt;span class="nx"&gt;assume_role&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nx"&gt;role_arn&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"arn:aws:iam::444444444444:role/OrganizationAccountAccessRole"&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="nx"&gt;resource&lt;/span&gt; &lt;span class="s2"&gt;"aws_s3_bucket"&lt;/span&gt; &lt;span class="s2"&gt;"tfstate"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;bucket&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"tfstate-sandbox-data-444444444444"&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="nx"&gt;resource&lt;/span&gt; &lt;span class="s2"&gt;"aws_s3_bucket_versioning"&lt;/span&gt; &lt;span class="s2"&gt;"tfstate"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;bucket&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;aws_s3_bucket&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;tfstate&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;
  &lt;span class="nx"&gt;versioning_configuration&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nx"&gt;status&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"Enabled"&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="nx"&gt;resource&lt;/span&gt; &lt;span class="s2"&gt;"aws_s3_bucket_public_access_block"&lt;/span&gt; &lt;span class="s2"&gt;"tfstate"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;bucket&lt;/span&gt;                  &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;aws_s3_bucket&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;tfstate&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;
  &lt;span class="nx"&gt;block_public_acls&lt;/span&gt;       &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
  &lt;span class="nx"&gt;block_public_policy&lt;/span&gt;     &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
  &lt;span class="nx"&gt;ignore_public_acls&lt;/span&gt;      &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
  &lt;span class="nx"&gt;restrict_public_buckets&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="c1"&gt;# Plus aws_iam_role and policy resources for TerraformPlanRole and&lt;/span&gt;
&lt;span class="c1"&gt;# TerraformDeployRole, each trusting its central CI role, so the account&lt;/span&gt;
&lt;span class="c1"&gt;# root can run without this access role.&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Enable versioning on every state bucket: HashiCorp recommends it because it lets you recover a state object that was deleted or corrupted, although it does not roll back the infrastructure an apply changed. With S3 native locking there is no DynamoDB table to create, which is one fewer thing to end up subtly different between account seventeen and account eighteen. Whichever route you pick, write it down in the repo. The failure mode is not the bootstrap itself but discovering a year later that three accounts were bootstrapped by hand and nobody remembers how.&lt;/p&gt;

&lt;h2&gt;
  
  
  The answer in one pass
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Directories, not workspaces: one root module per account with its own backend, provider configuration and hardcoded role ARN, guarded by &lt;code&gt;allowed_account_ids&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;State lives in a bucket inside the account it describes, locked with &lt;code&gt;use_lockfile&lt;/code&gt;, and the backend assumes the per-account role itself: the plan role for pull requests, the deploy role for applies.&lt;/li&gt;
&lt;li&gt;CI authenticates via OIDC to a central plan or apply role and assumes the matching per-account role, so pull-request code can read infrastructure and state but change neither; it pins Terraform, diffs against the PR base with full history, fans out when &lt;code&gt;modules/&lt;/code&gt; changes, guards against an empty matrix, and gates applies with environment protection or Atlantis apply requirements.&lt;/li&gt;
&lt;li&gt;Shared modules are pinned by tag where accounts need to upgrade on their own schedule, with Renovate opening the PRs.&lt;/li&gt;
&lt;li&gt;Drift detection is a scheduled &lt;code&gt;plan -refresh-only -detailed-exitcode&lt;/code&gt; across every account, alerting on exit 2 and treating errors separately.&lt;/li&gt;
&lt;li&gt;New accounts are bootstrapped uniformly, by a StackSet or Control Tower customization, or by a documented local-backend bootstrap root followed by a state migration.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Land that and the natural follow-up is Terragrunt, which exists to remove the repeated backend and provider configuration this design puts in twenty folders and adds dependency ordering between roots; be ready to explain what it buys and what it costs.&lt;/p&gt;

&lt;p&gt;Multi-account layout is one of the design questions Terraform interviews go deep on, alongside state internals, module design, drift and day-2 operations. &lt;a href="https://devopsinterview.cloud/ebooks/infrastructure-automation-mastery" rel="noopener noreferrer"&gt;Infrastructure as Code: Terraform &amp;amp; OpenTofu&lt;/a&gt; works through all of it with 50 senior-level interview questions, worked answers and the production reasoning behind them. It is one of the five books in the &lt;a href="https://devopsinterview.cloud/ebooks/complete-devops-mastery-bundle" rel="noopener noreferrer"&gt;DevOps Interview Mastery bundle&lt;/a&gt;, and every purchase includes the free Interview-Day Playbook with a day-of checklist and behavioural answer frameworks.&lt;/p&gt;

&lt;p&gt;Prefer to watch or listen? Answers like these are walked through on &lt;a href="https://www.youtube.com/@DevOpsInterviewCloud" rel="noopener noreferrer"&gt;YouTube&lt;/a&gt; and on the podcast, on &lt;a href="https://open.spotify.com/show/033txr35oY2Rcn6H4CVZ4S" rel="noopener noreferrer"&gt;Spotify&lt;/a&gt; and &lt;a href="https://music.amazon.com/podcasts/49554d87-91cb-4816-9420-cdac8a06ac75/devops-cloud-interview-prep-real-scenarios-answers" rel="noopener noreferrer"&gt;Amazon Music&lt;/a&gt;.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://devopsinterview.cloud/blog/terraform-mono-repo-20-aws-accounts" rel="noopener noreferrer"&gt;devopsinterview.cloud&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>architecture</category>
      <category>aws</category>
      <category>devops</category>
      <category>terraform</category>
    </item>
    <item>
      <title>Locking Down Kubernetes RBAC in a Multi-Tenant Cluster</title>
      <dc:creator>DevOpsInterview.Cloud</dc:creator>
      <pubDate>Sat, 31 Oct 2026 09:20:03 +0000</pubDate>
      <link>https://dev.to/devopsinterview_cloud/locking-down-kubernetes-rbac-in-a-multi-tenant-cluster-6i4</link>
      <guid>https://dev.to/devopsinterview_cloud/locking-down-kubernetes-rbac-in-a-multi-tenant-cluster-6i4</guid>
      <description>&lt;p&gt;A new product team is onboarding onto your shared cluster. Their developers need to deploy to their own namespace, their CI pipeline needs to roll deployments and run database migrations, and the security auditor wants proof that no tenant identity can touch another team's workloads. Design it. This scenario shows up in interviews for the same reason it shows up in real platform work: it cannot be answered by reciting object names. It tests whether you can turn four RBAC primitives into a working permission model, and whether you know the handful of places where a copy-pasted ClusterRoleBinding hands someone the whole cluster.&lt;/p&gt;

&lt;p&gt;The video version walks the same design if you prefer to watch it:&lt;/p&gt;

&lt;p&gt;  &lt;iframe src="https://www.youtube.com/embed/G5rsrDLoESE" width="710" height="399"&gt;
  &lt;/iframe&gt;
&lt;/p&gt;

&lt;h2&gt;
  
  
  The four objects, and where identities actually come from
&lt;/h2&gt;

&lt;p&gt;RBAC in Kubernetes is four kinds of object. A &lt;strong&gt;Role&lt;/strong&gt; is a list of allowed verbs on resources, scoped to one namespace. A &lt;strong&gt;ClusterRole&lt;/strong&gt; is a cluster-scoped object with the same rule format; its rules can target cluster-scoped resources like nodes and PersistentVolumes as well as namespaced ones, and where those rules apply depends on how it is bound. A &lt;strong&gt;RoleBinding&lt;/strong&gt; grants a Role (or, importantly, a ClusterRole) to subjects, with effect limited to the RoleBinding's namespace; the subjects need not live in that namespace (users and groups are not namespaced at all, and a ServiceAccount subject can be referenced from any namespace). A &lt;strong&gt;ClusterRoleBinding&lt;/strong&gt; grants a ClusterRole to subjects across the entire cluster.&lt;/p&gt;

&lt;p&gt;The subjects are users, groups, and ServiceAccounts, and the distinction matters more than most candidates realise. Users and groups are not Kubernetes objects at all: an authenticator maps each presented credential to a username and group memberships, from the subject fields of a client certificate or from the claims in an OIDC token issued by your identity provider. ServiceAccounts are the identities Kubernetes itself creates and manages, and they are what pods and CI pipelines use to talk to the API server.&lt;/p&gt;

&lt;h2&gt;
  
  
  The pattern most candidates miss: ClusterRole plus RoleBinding
&lt;/h2&gt;

&lt;p&gt;A RoleBinding can reference a ClusterRole, which grants that ClusterRole's permissions only inside the RoleBinding's namespace. That turns ClusterRoles into reusable permission templates. You define a developer role once and stamp it into each team's namespace with a per-namespace binding, so there is no duplicated YAML drifting apart between teams:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;rbac.authorization.k8s.io/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ClusterRole&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;namespace-developer&lt;/span&gt;
&lt;span class="na"&gt;rules&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
&lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;apiGroups&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;apps"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
  &lt;span class="na"&gt;resources&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;deployments"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
  &lt;span class="na"&gt;verbs&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;get"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;list"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;watch"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;create"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;patch"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;update"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
&lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;apiGroups&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
  &lt;span class="na"&gt;resources&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;pods"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
  &lt;span class="na"&gt;verbs&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;get"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;list"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;watch"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
&lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;apiGroups&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
  &lt;span class="na"&gt;resources&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;pods/log"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
  &lt;span class="na"&gt;verbs&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;get"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The discipline that goes with the pattern: a ClusterRoleBinding only for identities that genuinely need cluster-wide access, such as cluster operators and a small set of controllers. Binding the built-in &lt;code&gt;cluster-admin&lt;/code&gt; ClusterRole cluster-wide bypasses everything else in this post, and every one of those bindings is something you will have to explain to an auditor.&lt;/p&gt;

&lt;h2&gt;
  
  
  Wiring up the team namespace
&lt;/h2&gt;

&lt;p&gt;For the new team, create their namespace and one RoleBinding in it that points the reusable ClusterRole at their group:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;rbac.authorization.k8s.io/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;RoleBinding&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;payments-dev-binding&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;team-payments&lt;/span&gt;
&lt;span class="na"&gt;roleRef&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;apiGroup&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;rbac.authorization.k8s.io&lt;/span&gt;
  &lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ClusterRole&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;namespace-developer&lt;/span&gt;
&lt;span class="na"&gt;subjects&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
&lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Group&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;payments-devs&lt;/span&gt;
  &lt;span class="na"&gt;apiGroup&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;rbac.authorization.k8s.io&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The group name matches whatever group claim your identity provider puts in the token. Joiners and leavers are then handled in the IdP, with no Kubernetes changes at all, and the binding itself is a small, reviewable file in Git. Everyone in &lt;code&gt;payments-devs&lt;/code&gt; can deploy and read pods in &lt;code&gt;team-payments&lt;/code&gt;, and nothing in this binding grants anything anywhere else. RBAC is additive across bindings, though: a second binding somewhere else can widen what the group can do, which is why the audit section below exists.&lt;/p&gt;

&lt;h2&gt;
  
  
  ServiceAccounts and the default-account trap
&lt;/h2&gt;

&lt;p&gt;Every pod that does not specify a ServiceAccount runs as the &lt;code&gt;default&lt;/code&gt; one in its namespace, and by default that account's token is mounted into the pod whether the workload needs API access or not. Under RBAC the &lt;code&gt;default&lt;/code&gt; ServiceAccount starts with nothing beyond the basic discovery permissions granted to all authenticated principals, but the moment someone grants it more (usually to unblock one workload in a hurry), every pod in the namespace silently inherits the grant. Two rules prevent that:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Every workload that needs API access gets its own dedicated ServiceAccount.&lt;/li&gt;
&lt;li&gt;The &lt;code&gt;default&lt;/code&gt; ServiceAccount gets &lt;code&gt;automountServiceAccountToken: false&lt;/code&gt;, so pods that never asked for API access do not carry a credential. A pod can still set the same field on its own spec, which takes precedence, so this is a default rather than an enforcement mechanism; enforcement of what pods may do comes from the admission layer covered below.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For the team's CI pipeline, that looks like:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ServiceAccount&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;payments-ci&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;team-payments&lt;/span&gt;
&lt;span class="nn"&gt;---&lt;/span&gt;
&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;rbac.authorization.k8s.io/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Role&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ci-deployer&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;team-payments&lt;/span&gt;
&lt;span class="na"&gt;rules&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
&lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;apiGroups&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;apps"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
  &lt;span class="na"&gt;resources&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;deployments"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
  &lt;span class="na"&gt;verbs&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;get"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;patch"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;update"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
&lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;apiGroups&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;batch"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
  &lt;span class="na"&gt;resources&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;jobs"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
  &lt;span class="na"&gt;verbs&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;create"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;get"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;watch"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
&lt;span class="nn"&gt;---&lt;/span&gt;
&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;rbac.authorization.k8s.io/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;RoleBinding&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;payments-ci-binding&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;team-payments&lt;/span&gt;
&lt;span class="na"&gt;roleRef&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;apiGroup&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;rbac.authorization.k8s.io&lt;/span&gt;
  &lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Role&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ci-deployer&lt;/span&gt;
&lt;span class="na"&gt;subjects&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
&lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ServiceAccount&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;payments-ci&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;team-payments&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The Job rule exists because this pipeline runs its database migrations as Kubernetes Jobs; if yours does not, drop it. Also give the pipeline short-lived credentials, either time-limited TokenRequest tokens (&lt;code&gt;kubectl create token&lt;/code&gt;) or a federated external identity, rather than a long-lived ServiceAccount token Secret, which Kubernetes itself discourages.&lt;/p&gt;

&lt;p&gt;Assume the CI credential leaks eventually, and be precise about the blast radius. Patching deployments and creating Jobs are both workload creation, which Kubernetes documents as a privilege-escalation path in its own right: the stolen credential can run an arbitrary image, mount any Secret in the namespace, and run pods as any ServiceAccount in the namespace. The credential's direct API permissions stop at &lt;code&gt;team-payments&lt;/code&gt;, but what a malicious workload reaches from there depends on the other layers: pod networking, cloud metadata endpoints and workload identity, node hardening, and whatever the namespace itself holds. Containment therefore rests on keeping high-privilege ServiceAccounts and shared secrets out of tenant namespaces, on the network and admission controls covered below, and on admission policies if you need to restrict which images or ServiceAccounts a Job may use.&lt;/p&gt;

&lt;h2&gt;
  
  
  Auditing a cluster you inherited
&lt;/h2&gt;

&lt;p&gt;The built-in tooling answers most audit questions. &lt;code&gt;kubectl auth can-i&lt;/code&gt; checks a single permission, &lt;code&gt;--list&lt;/code&gt; dumps everything the current identity can do (with the caveat that the list can be incomplete depending on the authorizers in play, which is worth stating rather than trusting it blindly), and &lt;code&gt;--as&lt;/code&gt; impersonates another identity, provided your own identity has impersonation rights:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Everything the CI ServiceAccount can do in its namespace&lt;/span&gt;
&lt;span class="nv"&gt;$ &lt;/span&gt;kubectl auth can-i &lt;span class="nt"&gt;--list&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--as&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;system:serviceaccount:team-payments:payments-ci &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-n&lt;/span&gt; team-payments

&lt;span class="c"&gt;# Spot-check cross-namespace access: should come back "no"&lt;/span&gt;
&lt;span class="nv"&gt;$ &lt;/span&gt;kubectl auth can-i get pods &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--as&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;system:serviceaccount:team-payments:payments-ci &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-n&lt;/span&gt; team-checkout

&lt;span class="c"&gt;# Every subject bound to cluster-admin&lt;/span&gt;
&lt;span class="nv"&gt;$ &lt;/span&gt;kubectl get clusterrolebindings &lt;span class="nt"&gt;-o&lt;/span&gt; json | jq &lt;span class="s1"&gt;'.items[]
  | select(.roleRef.name=="cluster-admin")
  | {name: .metadata.name, subjects: .subjects}'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two caveats before treating those answers as proof. First, &lt;code&gt;--as&lt;/code&gt; with a ServiceAccount name does not automatically include the groups that account would carry in real requests, such as &lt;code&gt;system:serviceaccounts&lt;/code&gt; and &lt;code&gt;system:serviceaccounts:team-payments&lt;/code&gt;; permissions granted to those groups only show up if you add them explicitly with &lt;code&gt;--as-group&lt;/code&gt;. Second, a &lt;code&gt;no&lt;/code&gt; from &lt;code&gt;can-i&lt;/code&gt; proves exactly one verb-resource-namespace combination. Stronger RBAC evidence for an auditor is an enumeration of every binding that references the identity and its groups, with &lt;code&gt;can-i&lt;/code&gt; spot checks on top; even that is a point-in-time view of RBAC alone, so it sits alongside the authorizer configuration, the real group claims coming from the IdP, and the audit log rather than standing in for them.&lt;/p&gt;

&lt;p&gt;The cluster-admin query is usually the fastest way to find trouble in an inherited cluster. Expect the built-in binding for the &lt;code&gt;system:masters&lt;/code&gt; group plus a small, explainable set of operator or break-glass identities; a long list of humans and pipeline accounts is a finding. For a fuller picture, &lt;a href="https://github.com/corneliusweig/rakkess" rel="noopener noreferrer"&gt;rakkess&lt;/a&gt; renders an access matrix per subject, and &lt;a href="https://github.com/alcideio/rbac-tool" rel="noopener noreferrer"&gt;rbac-tool&lt;/a&gt; (from Alcide, now part of Rapid7) can visualise bindings and flag risky grants across the cluster.&lt;/p&gt;

&lt;h2&gt;
  
  
  The bind and escalate verbs
&lt;/h2&gt;

&lt;p&gt;Kubernetes has escalation-prevention rules built into the RBAC API: you can only create or update a role binding if you already hold every permission in the referenced role at the binding's scope, or if you hold the &lt;code&gt;bind&lt;/code&gt; verb on that role. Likewise, creating or updating a role that contains permissions you do not yourself hold requires the &lt;code&gt;escalate&lt;/code&gt; verb. Both verbs target the &lt;code&gt;roles&lt;/code&gt; and &lt;code&gt;clusterroles&lt;/code&gt; resources, and both are deliberate bypasses of the safety rails.&lt;/p&gt;

&lt;p&gt;That makes them a quiet escalation path when combined with write access to the corresponding objects. A subject who can create ClusterRoleBindings and holds unrestricted &lt;code&gt;bind&lt;/code&gt; on ClusterRoles can grant &lt;code&gt;cluster-admin&lt;/code&gt; cluster-wide to any identity, including their own; the same combination with namespaced RoleBindings hands over everything in that namespace. A subject with &lt;code&gt;escalate&lt;/code&gt; plus write access on roles can author permissions they were never granted. There is a legitimate use: &lt;code&gt;bind&lt;/code&gt; constrained with &lt;code&gt;resourceNames&lt;/code&gt; to a specific, limited role is how you delegate that one role safely. Unrestricted &lt;code&gt;bind&lt;/code&gt; and &lt;code&gt;escalate&lt;/code&gt;, and wildcards generally, are the red flags, so scan for them and then read what the scan finds instead of treating every hit as an incident:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# ClusterRoles with wildcard resources&lt;/span&gt;
&lt;span class="nv"&gt;$ &lt;/span&gt;kubectl get clusterroles &lt;span class="nt"&gt;-o&lt;/span&gt; json | jq &lt;span class="s1"&gt;'.items[]
  | select(.rules[]?.resources[]? == "*")
  | .metadata.name'&lt;/span&gt;

&lt;span class="c"&gt;# ClusterRoles granting bind or escalate&lt;/span&gt;
&lt;span class="nv"&gt;$ &lt;/span&gt;kubectl get clusterroles &lt;span class="nt"&gt;-o&lt;/span&gt; json | jq &lt;span class="s1"&gt;'.items[]
  | select(.rules[]?.verbs[]? == "bind" or .rules[]?.verbs[]? == "escalate")
  | .metadata.name'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;These queries cover ClusterRoles only, and the same problems hide in namespaced Roles, in wildcard verbs and apiGroups, and in non-resource URLs, so a real audit repeats the pattern across those too.&lt;/p&gt;

&lt;p&gt;Aggregation is the other way permissions grow without anyone editing a role. The rules of the built-in &lt;code&gt;admin&lt;/code&gt;, &lt;code&gt;edit&lt;/code&gt; and &lt;code&gt;view&lt;/code&gt; ClusterRoles are aggregation-controlled: a controller populates them from every ClusterRole carrying a label like &lt;code&gt;rbac.authorization.k8s.io/aggregate-to-edit: "true"&lt;/code&gt;. Operators and CRD-based add-ons routinely ship ClusterRoles with those labels, which means installing one can silently expand what &lt;code&gt;edit&lt;/code&gt; grants in every namespace that binds it. After installing an operator, diff the contents of the aggregated roles against what they held before.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where RBAC stops
&lt;/h2&gt;

&lt;p&gt;RBAC governs the Kubernetes API server and nothing else. It does not restrict traffic between pods, it does not stop a privileged container from reaching the node, and it has no opinion about what runs inside a container. For this scenario's threat model, tenant isolation rests on three layers: RBAC for API access, NetworkPolicies for pod-to-pod traffic, and Pod Security Admission (or a policy engine like OPA Gatekeeper or Kyverno) to stop privileged pods from being created in the first place. Stricter tenancy requirements add more: resource quotas, storage isolation, node isolation or sandboxed runtimes, and at the far end dedicated clusters per tenant. The mistake to avoid is treating RBAC as the whole security story and skipping the rest.&lt;/p&gt;

&lt;h2&gt;
  
  
  Making least privilege the default
&lt;/h2&gt;

&lt;p&gt;Three habits turn all of the above from a one-off cleanup into a steady state:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Bake it into provisioning.&lt;/strong&gt; Namespace creation should apply the default-ServiceAccount lockdown automatically, so new workloads start safe and opt in to API access:
&lt;/li&gt;
&lt;/ol&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;kubectl patch serviceaccount default &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-n&lt;/span&gt; team-payments &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-p&lt;/span&gt; &lt;span class="s1"&gt;'{"automountServiceAccountToken": false}'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ol&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Keep RBAC in Git.&lt;/strong&gt; Applying bindings through a reviewed pipeline means every permission change has a commit and an approver behind it. Be careful not to oversell this in an interview: a GitOps reconciler manages the resources it is told about, and an out-of-band ClusterRoleBinding that was never in Git will not be automatically reverted, and may not even be reported, unless you have configured drift or orphan detection to cover it. Git gives you provenance; catching out-of-band changes is the next habit's job.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Alert on ClusterRoleBinding writes.&lt;/strong&gt; Watch updates and patches as well as creations, since adding a subject to an existing binding grants just as much as creating a new one. This requires API server auditing to be configured with a policy that captures RBAC writes; managed providers can expose the audit stream through their logging services (often behind an explicit control-plane logging toggle), and self-managed API servers log nothing until you set a policy. Once the events flow, tools such as Falco can consume them via the Kubernetes audit event source; note that Falco's shipped rule set covers specific cases like bindings to &lt;code&gt;cluster-admin&lt;/code&gt;, so alerting on all ClusterRoleBinding writes means a small custom rule. These writes should be rare, deliberate and reviewed; an unexpected one is either a mistake or an attack, and both deserve a page.&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  The model to present
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;ClusterRoles as reusable permission templates, granted per namespace via RoleBindings.&lt;/li&gt;
&lt;li&gt;Subjects are IdP groups for humans, dedicated ServiceAccounts for workloads; joiners and leavers are an IdP change, not a cluster change.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;automountServiceAccountToken: false&lt;/code&gt; on default ServiceAccounts; every API-consuming workload gets its own account with minimal verbs.&lt;/li&gt;
&lt;li&gt;Audit by enumerating bindings per subject, with &lt;code&gt;kubectl auth can-i --list&lt;/code&gt; and impersonation (including &lt;code&gt;--as-group&lt;/code&gt;) as spot checks; enumerate &lt;code&gt;cluster-admin&lt;/code&gt; bindings and hunt for unrestricted &lt;code&gt;bind&lt;/code&gt;, &lt;code&gt;escalate&lt;/code&gt; and wildcards in both ClusterRoles and Roles.&lt;/li&gt;
&lt;li&gt;Watch aggregated ClusterRoles after installing operators.&lt;/li&gt;
&lt;li&gt;RBAC plus NetworkPolicy plus Pod Security Admission, since each layer covers what the others cannot. And because workload creation is itself an escalation path, keep high-privilege ServiceAccounts and shared secrets out of tenant namespaces.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Walking those six points through the multi-tenant scenario covers the design question, the audit question and the trade-off question in one pass, and it names the residual risks explicitly.&lt;/p&gt;

&lt;p&gt;RBAC is one chapter of the Kubernetes story real interviews go deep on, alongside networking, scheduling, debugging and scaling. &lt;a href="https://devopsinterview.cloud/ebooks/container-orchestration-journey" rel="noopener noreferrer"&gt;Containers &amp;amp; Orchestration&lt;/a&gt; covers that full arc with 52 worked questions, follow-up chains and production war stories. It is one of the five books in the &lt;a href="https://devopsinterview.cloud/ebooks/complete-devops-mastery-bundle" rel="noopener noreferrer"&gt;DevOps Interview Mastery bundle&lt;/a&gt;, and every purchase includes the free Interview-Day Playbook with a day-of checklist and behavioural answer frameworks.&lt;/p&gt;

&lt;p&gt;Prefer to watch or listen? Answers like these are walked through on &lt;a href="https://www.youtube.com/@DevOpsInterviewCloud" rel="noopener noreferrer"&gt;YouTube&lt;/a&gt; and on the podcast, on &lt;a href="https://open.spotify.com/show/033txr35oY2Rcn6H4CVZ4S" rel="noopener noreferrer"&gt;Spotify&lt;/a&gt; and &lt;a href="https://music.amazon.com/podcasts/49554d87-91cb-4816-9420-cdac8a06ac75/devops-cloud-interview-prep-real-scenarios-answers" rel="noopener noreferrer"&gt;Amazon Music&lt;/a&gt;.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://devopsinterview.cloud/blog/kubernetes-rbac-least-privilege-lockdown" rel="noopener noreferrer"&gt;devopsinterview.cloud&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>devops</category>
      <category>kubernetes</category>
      <category>architecture</category>
      <category>security</category>
    </item>
  </channel>
</rss>
