DEV Community: Devam Parikh

GitOps Without Surprises: Generate, Review, Then Reconcile

Devam Parikh — Tue, 14 Jul 2026 14:29:59 +0000

GitOps is supposed to make production changes boring: review a pull request, merge it, and let the controller reconcile the desired state.

But there is a subtle gap in many Kubernetes delivery flows.

When a repository stores Helm charts, Helm values, Kustomize bases, or overlays, Git often contains the inputs to the desired state. The actual Kubernetes objects are created later, usually inside Argo CD at sync time.

That is convenient. It is also where surprises hide.

The problem: Git may not show what Kubernetes will receive

Helm and Kustomize are excellent authoring tools. They reduce duplication, let teams share common patterns, and make multi-environment delivery manageable.

The problem is not the tools. The problem is where rendering happens.

If rendering happens only inside the GitOps controller, reviewers may approve a change without seeing the exact final YAML. A one-line values change can modify a Deployment, Service, ConfigMap, HPA, RBAC rule, or annotation in ways that are not obvious from the pull request.

That hurts in a few places:

Reviews focus on abstraction changes, not final Kubernetes objects.
Audits require re-rendering with matching tool versions and inputs.
Rollbacks are harder to reason about.
Incident debugging starts with “what did Argo CD actually apply?”
Policy and API deprecation checks happen later than they should.

For platform teams, this becomes painful as more teams and environments onboard onto the same GitOps platform.

The pattern: generate, review, then reconcile

A safer GitOps flow looks like this:

Developers keep authoring with Helm, Helmfile, or Kustomize.
CI renders the final Kubernetes manifests.
CI commits the rendered output back to the pull request branch.
Reviewers inspect the exact YAML diff.
Policy, security, and API deprecation checks run against rendered objects.
Argo CD reconciles from generated-manifest/.

In other words:

Author with abstractions, but reconcile reviewed YAML.

This is usually called the rendered manifests pattern or hydrated manifests pattern.

Why this works well with Argo CD

Argo CD ultimately applies Kubernetes manifests. Even when you point Argo CD at a Helm chart, Argo CD renders that chart and applies the resulting YAML.

The rendered-manifest pattern moves that rendering step earlier so humans and automation can inspect the result before reconciliation.

A deployment repository can look like this:

k8s/
  demo/
    demo-api/
      Chart.yaml
      values.yaml
      staging-nam-values.yaml
      production-nam-values.yaml

generated-manifest/
  staging/
    nam/
      demo/
        demo-api/
          manifest.lock.yaml
  production/
    nam/
      demo/
        demo-api/
          manifest.lock.yaml

Argo CD then points to the rendered path:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: demo-api-staging
spec:
  project: default
  source:
    repoURL: https://github.com/your-org/your-deploy-repo.git
    targetRevision: main
    path: generated-manifest/staging/nam/demo/demo-api
  destination:
    server: https://kubernetes.default.svc
    namespace: demo

Now a pull request does not only show a values file change. It also shows the resulting manifest.lock.yaml change.

That is the part reviewers actually need.

A reusable GitHub Action for this flow

I created a reusable GitHub Action for this pattern:

https://github.com/Devamparikh/gitops-render-manifests-action

It supports:

helm
helmfile
kustomize
none for repositories that already store raw YAML
optional shared-directory detection
optional commit-back to the pull request branch
optional Kubernetes API deprecation checks with Pluto

A basic caller workflow looks like this:

name: Render Kubernetes manifests

on:
  pull_request:
    paths:
      - "k8s/**"
      - "common-values/**"
      - ".github/workflows/render-manifests.yaml"

permissions:
  contents: write
  pull-requests: write

jobs:
  render:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout pull request branch
        uses: actions/checkout@v4
        with:
          ref: ${{ github.event.pull_request.head.ref }}
          token: ${{ secrets.GITHUB_TOKEN }}
          fetch-depth: 0

      - name: Render manifests
        uses: Devamparikh/gitops-render-manifests-action@v0
        with:
          package_manager: helm
          shared_dirs: common-values
          target_k8s_version: v1.33.0

For Kustomize:

- name: Render manifests
  uses: Devamparikh/gitops-render-manifests-action@v0
  with:
    package_manager: kustomize
    kustomize_load_restrictor_none: "true"

For Helmfile:

- name: Render manifests
  uses: Devamparikh/gitops-render-manifests-action@v0
  with:
    package_manager: helmfile
    shared_dirs: common-values,charts
    parallel: "4"

The review experience changes

Before rendered manifests, a reviewer might see:

replicaCount: 3

After rendered manifests, the reviewer also sees:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-api
spec:
  replicas: 3

That sounds small, but it changes the conversation.

Reviewers can ask:

Did this change the expected namespace?
Did the image tag move?
Did RBAC expand?
Did resource requests change?
Did labels, annotations, selectors, or topology rules change?
Will this API still work on the target Kubernetes version?

Those are production questions. They deserve to be answered before Argo CD syncs.

Trade-offs

This pattern is not free.

It adds generated files to Git. It can make pull requests larger. It requires CI to have the right rendering tools. It also requires a clear repository contract so teams know where source inputs live and where generated manifests go.

You should also be careful with secrets. Rendered manifests should not commit plaintext secrets. Use External Secrets Operator, Sealed Secrets, SOPS, secret references, or another secret-management pattern that fits your organization.

The pattern is worth considering when:

multiple teams deploy through a shared platform,
reviewers need stronger auditability,
regulated environments need explicit change records,
Argo CD sync-time rendering has caused surprises,
you want policy and deprecation checks before sync,
you want ApplicationSets to reconcile plain generated YAML.

For very small teams or simple services, direct Helm or Kustomize in Argo CD may still be enough.

Where this fits with Source Hydrator and other tools

Argo CD Source Hydrator is making hydrated manifests a more first-class workflow inside Argo CD itself. Kargo and OCI-based promotion patterns can also solve parts of this problem, especially around environment promotion.

The action in this post is intentionally simple: it gives teams a CI-based way to try the pattern without changing their entire platform.

Start small. Render one service. Review the manifest diff. Point Argo CD at the generated path. Then decide whether the pattern should become part of your platform contract.

Final thought

GitOps is not just “put YAML in Git.”

The stronger promise is this:

The desired state that humans review is the same desired state the controller reconciles.

That is the heart of GitOps without surprises.

When the Agent Is Wrong: Surface Real Kubernetes Errors, Not Model Guesses

Devam Parikh — Tue, 14 Jul 2026 10:05:02 +0000

AI agents are good at producing explanations.

That is useful until the explanation is wrong.

In Kubernetes operations, a failed tool call often already contains the truth:

Error from server (Forbidden): cannot patch resource "deployments/scale"

But if the agent summarizes too aggressively, it may say something like:

This may have been blocked by a policy engine or GitOps drift prevention.

That might sound plausible. It might also be completely wrong.

For operational agents, tool output must be treated as evidence. The model's explanation is secondary.

The source of truth

When a Kubernetes tool fails, preserve the real error:

command or tool name
target cluster
namespace
resource
Kubernetes API error
stderr
exit code
admission response if present
RBAC denial if present

The agent can summarize this, but it should not replace it with a guess.

Good:

The scale operation failed because Kubernetes RBAC denied patch access to deployments/scale for the tool service account.

Bad:

The scale operation likely failed because GitOps reverted it.

Unless the tool output proves GitOps was involved, that is speculation.

Common Kubernetes failures agents misread

RBAC denial

cannot patch resource "deployments/scale"

This is not an application failure. It is not a model failure. It is not necessarily a policy failure.

It means the service account does not have the requested permission.

Missing binary in a container

executable file not found in $PATH

This often happens with distroless or minimal images.

The exec path may be working perfectly. The target container just does not have sh, bash, curl, or ps.

Namespace or tool guardrail

Some Kubernetes tools intentionally block sensitive namespaces.

If a tool refuses to operate in kube-system, that may be a built-in guardrail rather than an RBAC issue.

Resource not found

deployments.apps "api" not found

This might mean:

wrong namespace
wrong cluster
stale incident context
typo
resource renamed

The agent should verify before inventing a root cause.

Show evidence in the response

For incident workflows, I prefer this response shape:

Result: Failed
Operation: scale deployment
Target: namespace/app
Reason: RBAC denied
Evidence: service account cannot patch deployments/scale
Next step: grant scoped RBAC if this operation should be allowed

This is much better than a paragraph of uncertain explanation.

It gives the human a clear path:

fix RBAC
choose a different action
reject the operation
escalate to someone with permissions

Distinguish diagnosis from speculation

Agents should use language carefully.

Evidence-backed:

The API server denied this request because...

Speculative:

This could be related to...

Speculation is allowed, but it must be labeled as speculation.

In production operations, confident wrong answers are worse than cautious incomplete answers.

Use tool traces for debugging the agent itself

When testing an operational agent, inspect the tool trace:

Did it choose the right tool?
Did it choose the right cluster?
Did it pass the right namespace?
Did it ask for approval when required?
Did it preserve stderr?
Did it summarize the actual error?

If the model explanation does not match the tool trace, fix the prompt, tool description, or response formatter.

Do not train users to trust polished guesses.

Improve prompts, but do not rely only on prompts

Prompts can help:

When a tool call fails, quote the concrete error category and do not invent causes that are not present in the tool output.

But prompts are not enough.

The platform should structure tool results so the model receives:

status
error category
raw error
target metadata
suggested remediation

Structured evidence gives the agent less room to improvise.

Final thought

The best Kubernetes agents are not the ones that sound most confident.

They are the ones that preserve operational truth.

Let the model explain, summarize, and guide. But keep Kubernetes errors visible, structured, and auditable.

When the system fails, the human should be able to answer:

What exactly happened, and what evidence do we have?

Exposing Internal Kubernetes Agent UIs: Lessons from VPC Lattice, DNS, and Private Access

Devam Parikh — Tue, 14 Jul 2026 10:05:01 +0000

A Kubernetes UI can be healthy and still unreachable from your browser.

That sounds contradictory until you debug private networking.

This post covers a common pattern: exposing an internal Kubernetes UI through AWS VPC Lattice, Route53, and private network access.

The examples are generic, but the lesson is practical:

A working backend does not guarantee a reachable human URL.

The symptom

The UI is deployed. The pod is running. The Kubernetes Service has endpoints. The route is accepted. Health checks are passing.

But the browser times out.

At this point, it is tempting to blame the application.

In many private-network cases, the application is fine. The problem is the access path.

Trace from the browser inward

Debug in this order:

DNS resolution
local route table
VPN/private network routing
Lattice or ingress listener
route attachment
target group health
Kubernetes Service endpoints
pod readiness

Do not start at the pod if the user cannot reach the hostname.

Start with:

dig internal-ui.example.com

Then check what kind of IP the hostname resolves to.

Link-local addresses are a clue

If a private UI hostname resolves to something like:

169.254.x.x
fd00:...

your laptop may not be able to route it, even if you are on VPN.

Many VPN clients route VPC CIDRs such as:

10.0.0.0/8
172.16.0.0/12
192.168.0.0/16

They may not route Lattice link-local addresses from your machine into AWS.

So the backend can be reachable from inside the VPC, while the browser on your laptop cannot reach the same DNS target.

Compare with a known working service

The fastest way to find the difference is to compare with another internal UI that already works.

Check:

What does its Route53 record point to?
Does it resolve to VPC-routable private IPs?
Does it use HTTPS?
Does it have an ACM certificate attached?
Does it point to a VPC endpoint association DNS name?
Does the non-working service point directly to a raw service DNS name?

In one common working pattern, Route53 points to a VPC endpoint association DNS name:

vpce-...-snsa-....vpc-lattice-svcs.region.on.aws

That may resolve to private VPC IPs reachable over VPN.

A broken pattern may point directly to a Lattice service-assigned domain:

service-name-....vpc-lattice-svcs.region.on.aws

That can resolve to link-local addresses that are not reachable from a user's laptop.

The fix is usually DNS target ownership

For human browser access, Route53 should point at the DNS target that matches the intended access path.

If users connect over VPN, the hostname should resolve to addresses their VPN routes.

That often means using the VPC endpoint association path rather than the raw service domain.

The exact implementation can be owned by Terraform, Crossplane, or another platform controller. The important part is not the tool. The important part is the contract:

Human UI hostname -> private routable endpoint -> Lattice/ingress -> Kubernetes Service

Not:

Human UI hostname -> link-local service address unreachable from laptop

Keep service-to-service and human access separate

The best endpoint for service-to-service traffic is not always the best endpoint for browser access.

For example:

MCP or internal service traffic may use one Lattice path.
Human UI access may need a different private DNS and endpoint association path.
Public access may be forbidden entirely.

That is fine.

Platform teams should make the access modes explicit:

agent-to-tool traffic: internal service path
human UI traffic: private browser-accessible path

Verification checklist

Before declaring the UI broken, verify:

Pod is running.
Service has endpoints.
Route is accepted.
Lattice target is healthy.
DNS resolves to the expected target.
Resolved IPs are routable from the client network.
HTTPS listener and certificate match the hostname.
Security groups and network policy allow the path.

If steps 1-4 pass and step 6 fails, the application is not the problem.

Final thought

Private access bugs are often naming bugs in disguise.

The hostname exists. The service exists. The route exists. But the DNS target points to the wrong kind of network address for the client.

For internal Kubernetes UIs, always ask:

Does this hostname resolve to something the user can actually route to?

That question saves a surprising amount of time.

Read-Only First: A Safer Adoption Model for Agentic Platform Engineering

Devam Parikh — Tue, 14 Jul 2026 10:04:27 +0000

The fastest way to make people nervous about AI agents is to start with:

Let the agent fix production.

For platform teams, a better starting point is:

Let the agent inspect production safely and help humans make better decisions faster.

That is the read-only-first model.

Why read-only first works

Most incidents do not begin with a fix. They begin with questions:

What changed?
Which pods are failing?
Are events showing scheduling issues?
Did a rollout just happen?
Are logs showing one error or many?
Is the issue isolated to one namespace or many?
Is the service routing to ready endpoints?

These are inspection tasks.

An agent that answers them quickly is already useful, even if it cannot mutate anything.

Read-only is not useless

A read-only Kubernetes agent can:

list pods, deployments, services, ingresses, gateways, and routes
summarize events
inspect rollout state
fetch logs
compare resources across clusters
identify missing endpoints
explain likely failure modes
generate a human-readable incident summary

That can reduce time-to-context without increasing blast radius.

The first adoption goal should be:

Can the agent help a human understand the system faster?

Not:

Can the agent autonomously remediate production?

Make read-only tools boring

Read-only tools should be boring in the best way:

declarative
versioned
reviewed
scoped by RBAC
observable
easy to disable
attached explicitly to agents

Avoid giving the agent a giant toolbox and hoping the prompt keeps it safe.

Prefer explicit tool lists:

toolNames:
  - k8s_get_resources
  - k8s_describe_resource
  - k8s_get_events
  - k8s_get_logs

The smaller the first tool surface, the easier it is to reason about.

Add governance before mutation

Before adding write tools, answer these questions:

Who can invoke the agent?
Where can it be invoked from?
Which clusters can it reach?
Which namespaces are allowed?
Which tools require approval?
How are approvals logged?
What counts as a high-risk request?
Can the agent access secrets indirectly?
How are failed tool calls surfaced?

If those answers are unclear, write access is premature.

A staged rollout model

I like this progression:

Stage 1: Read-only single-cluster inspection
Stage 2: Read-only multi-cluster inspection
Stage 3: ChatOps integration with approved channels
Stage 4: Approval-gated diagnostic exec
Stage 5: Approval-gated low-risk remediation
Stage 6: Pull-request based config fixes
Stage 7: Policy-backed automated remediation

Each stage earns the next one.

The platform should prove reliability, auditability, and user trust before gaining more power.

Read-only still needs guardrails

Read-only does not mean zero risk.

Logs can contain sensitive data. Resource names can reveal architecture. Events can expose internal details. Cross-cluster access can broaden visibility.

So even read-only agents need:

authentication
authorization
audit logs
namespace scoping
rate limits
clear data handling rules
private or gateway-controlled model options

The difference is that read-only failures usually have a smaller blast radius than write failures.

When to introduce write tools

Add write tools when:

users trust read-only responses
tool traces are observable
RBAC is scoped
approval gates work
risk classification is understandable
the team has tested failure modes

Start with reversible operations:

rollout restart
scale within safe bounds
patch a known annotation
run a narrow diagnostic exec

Avoid starting with:

delete arbitrary resources
apply arbitrary manifests
broad shell execution
secret inspection
cluster-wide mutation

The best first demo

The best first demo is not deleting a pod.

A better demo:

Ask the agent why a deployment is unhealthy.
Agent gathers pods, events, rollout, and logs.
Agent summarizes the likely cause.
Agent proposes a safe next step.
Human decides.

That demonstrates value without pretending the system is ready for autonomy.

Final thought

Read-only-first is not a timid approach. It is a platform engineering approach.

It lets teams introduce agentic workflows while preserving the trust model that production systems need.

The goal is not to make the agent powerless.

The goal is to make each new capability earned, observable, and reversible.

From Tool Discovery to Real Execution: Verifying a Multi-Cluster MCP Path

Devam Parikh — Tue, 14 Jul 2026 10:04:26 +0000

Seeing tools in an agent UI is not the same as proving the platform works.

In a multi-cluster MCP setup, there are many layers between a user prompt and a Kubernetes API response:

Prompt
  -> agent
  -> MCP client
  -> gateway route
  -> federated backend
  -> remote MCP server
  -> Kubernetes API
  -> tool result
  -> agent response

If any layer is misconfigured, the final experience may fail in confusing ways.

This post is a verification checklist for multi-cluster MCP paths.

Start with the backend tools

Before involving the agent, verify the application-cluster MCP server directly.

Questions to answer:

Is the MCP server running?
Is it reachable over the expected network path?
Can it list tools?
Can it execute a simple read-only tool?
Does its Kubernetes service account have the intended RBAC?

Do not start debugging the agent until the backend tool server works independently.

Verify the gateway backend

If Agentgateway or another gateway is federating MCP servers, verify the backend object next.

For each target:

is the host correct?
is the port correct?
is the MCP path correct?
is the protocol correct?
is the target name stable?

Target names often become tool prefixes.

If the target is named cluster-a, a backend tool like:

k8s_get_resources

may appear to the agent as:

cluster-a_k8s_get_resources

That prefix is part of the operator experience. Choose names humans can recognize.

Verify routing

Next, confirm the route that publishes the federated MCP endpoint.

Check:

the route is accepted
backend references are resolved
the path matches the URL configured in the agent
the gateway listener is attached
network policy or service mesh rules are not blocking traffic

This is where many failures hide. The backend may be healthy, but the route may point at the wrong path or namespace.

List tools from the federated endpoint

Now connect directly to the federated MCP endpoint and list tools.

You want to see tools from every expected cluster with the right prefixes:

cluster-a_k8s_get_resources
cluster-a_k8s_describe_resource
cluster-b_k8s_get_resources
cluster-b_k8s_describe_resource

If tools are missing, the problem is likely gateway federation, backend reachability, or tool server discovery.

If tools are present, you have proven discovery, but not execution.

Execute a safe read-only tool

Pick a boring command:

list namespaces
list pods in a non-sensitive namespace
describe a known deployment

The goal is not to show off. The goal is to prove:

the tool call reaches the correct cluster
RBAC allows the action
the response returns through the gateway
the MCP client receives structured output

Keep the first test read-only.

Attach the tool to the agent

Only after direct MCP verification should you attach the tools to the agent.

In the agent definition, explicitly list allowed tools:

toolNames:
  - cluster-a_k8s_get_resources
  - cluster-a_k8s_describe_resource
  - cluster-b_k8s_get_resources
  - cluster-b_k8s_describe_resource

Explicit tool lists are easier to review than broad access.

They also make it obvious which clusters and actions the agent can use.

Test the agent with specific prompts

Avoid vague prompts at first.

Use prompts that force the intended route:

List pods in Cluster A in the default namespace.

Describe the frontend deployment in Cluster B.

Compare pod readiness between Cluster A and Cluster B for this namespace.

The agent should select the correctly prefixed tool.

If it chooses the wrong cluster, fix the system prompt or tool descriptions before adding more tools.

Watch for false success

An agent can produce a confident answer even when the tool failed.

For verification, inspect the actual tool calls and tool outputs.

Look for:

wrong tool name
wrong namespace
RBAC denial
network timeout
empty result interpreted as success
model explanation that does not match tool stderr

The agent's natural-language answer is not enough. The tool trace is the source of truth.

Add write access last

Once read-only federation works, write tools can be added carefully:

separate service account
smaller RBAC surface
explicit tool list
human approval
risk classification
audit logging
namespace restrictions

Do not mix early read-only validation with write-tool debugging. You want a clean baseline first.

Final checklist

Use this rollout order:

Direct MCP server health.
Direct read-only tool call.
Gateway backend accepted.
Route accepted.
Federated endpoint lists tools.
Federated endpoint executes a read-only tool.
Agent discovers tools.
Agent selects the correct prefixed tool.
Agent response matches real tool output.
Approval-gated write tools added later.

The important idea:

Tool discovery proves the menu exists. Real execution proves the kitchen works.

Designing ChatOps Sessions for Kubernetes Agents

Devam Parikh — Tue, 14 Jul 2026 10:04:24 +0000

Running a Kubernetes agent from Slack or Teams sounds simple:

Mention the bot and ask it to inspect the cluster.

The hard part is not sending a message to an agent. The hard part is deciding what context that message belongs to.

In incident response, context boundaries matter.

If two unrelated incident threads share the same agent memory, the agent may mix assumptions. If a bot can be invoked from any channel or personal chat, sensitive operations can drift outside approved spaces.

This post covers a practical model for ChatOps sessions with Kubernetes agents.

The core rule

For incident workflows, the safest default is:

one chat thread = one agent context

That means:

Incident thread A -> agent session A
Incident thread B -> agent session B
Personal chat      -> blocked or separate session

This keeps debugging context, approvals, tool results, and follow-up questions scoped to one operational conversation.

Chat threads are not always obvious to bots

Humans see a thread as a clear conversation.

Bot frameworks may expose several identifiers:

team ID
channel ID
conversation ID
message ID
reply/root thread ID

Depending on the platform, multiple channel threads may share a conversation ID and differ only by reply metadata.

If your bot stores agent state only by conversation.id, two threads may accidentally share an agent session.

A better session key includes the thread root:

session_key = channel_id + thread_root_id

The exact fields vary by provider, but the intent is stable:

bind the agent session to the incident thread, not only to the channel.

The bot may not see old thread history

Another subtle issue: if an agent is mentioned halfway through a long thread, it may not automatically receive the previous messages.

The bot might only receive:

@agent can you inspect this?

It may not receive:

Earlier: deployment failed after rollout
Earlier: image pull errors were seen
Earlier: only one namespace is affected

There are three ways to handle this:

Ask users to include context when mentioning the agent.
Use the chat provider API to fetch thread history.
Summarize thread history before sending it to the agent.

For production usage, fetching thread context requires careful permissions and data handling. Do not add broad chat-read permissions casually.

Control where the agent can be used

There are two layers of access control:

Where the app can be installed.
Where the runtime will actually respond.

The first layer is app configuration. For example, you might remove personal chat or group chat support and allow only team/channel usage.

The second layer is runtime enforcement.

A Kubernetes operations bot should support allowlists such as:

ALLOWED_TEAM_IDS
ALLOWED_CHANNEL_IDS
ALLOWED_CONVERSATION_IDS

The bot should reject normal messages and approval actions outside approved locations.

This matters because an approval click is operational authority. It should be constrained to the same trusted space as the request.

Approval state belongs to the session

If the agent asks to run a protected tool, the approval decision should be tied to:

user
request
tool
target cluster/namespace/resource
chat thread
timestamp

Avoid approval flows that can be replayed or detached from their context.

An approval should mean:

I approve this specific tool call for this specific target in this specific thread.

Not:

I approve the agent to do something generally.

Make the agent summarize risk before approval

A useful ChatOps flow looks like:

User:
  Check this workload and restart it if needed.

Agent:
  I found failing readiness probes.
  Restarting the deployment is a Yellow operation because it mutates live workload state.

Approval card:
  Tool: restart deployment
  Target: namespace/app
  Risk: Yellow
  Approve / Reject

The agent explains. The platform enforces.

That separation keeps the UX friendly without relying on the model as the policy boundary.

Final thought

Chat is a great interface for Kubernetes incident response because it is where people already coordinate.

But a chat interface is not automatically safe.

Design the session model deliberately:

one incident thread maps to one agent context
approved channels are enforced at runtime
old thread history is handled intentionally
approval actions are bound to a specific request
sensitive Kubernetes tools require real gates

The agent should feel easy to use, but the boundaries should be boringly explicit.

Why Kubernetes Exec Is Secret-Adjacent in Agentic Operations

Devam Parikh — Tue, 14 Jul 2026 10:04:23 +0000

When teams connect AI agents to Kubernetes, they often start the security review with one question:

Can this agent read Kubernetes Secrets?

That is a good question, but it is not sufficient.

An agent may be blocked from the Kubernetes Secrets API and still be able to expose sensitive data indirectly through exec.

That makes Kubernetes exec a secret-adjacent capability.

The misleading comfort of blocked Secret API access

Imagine a Kubernetes tool service account with this posture:

get/list/watch secrets: no
create pods/exec: yes

At the API layer, this looks reassuring. The agent cannot run:

kubectl get secrets
kubectl describe secret

But if the same agent can exec into a running pod, it may be able to inspect what the container can see at runtime.

That can include:

mounted Secret volumes
service account tokens
environment variables
application config files
credentials cached on disk
cloud identity tokens
files written by init containers or sidecars

So the real posture is:

Kubernetes Secret API access: blocked
Runtime secret exposure through pod exec: possible

That distinction matters a lot when the user interface is natural language.

Why exec is powerful

Exec is not one permission. It is a door into a process environment.

Even harmless-looking prompts can become risky:

Can you check the environment of this pod?

Can you cat the config directory?

Can you print the token file?

The Kubernetes API may not expose Secrets to the agent, but the application container might.

This is why pods/exec should not be treated like a normal read-only permission. It is interactive runtime access.

Practical risk classification

For agentic Kubernetes workflows, I like to classify exec requests separately from ordinary reads.

Green:  Pure Kubernetes API reads
Yellow: Exec with safe diagnostic intent
Red:    Exec that may expose secrets, credentials, tokens, or broad filesystem state

Examples:

Request	Suggested Risk	Reason
List pods	Green	Kubernetes API read
Describe deployment	Green	Kubernetes API read
Get pod logs	Green/Yellow	Logs may contain sensitive data
Exec `uname -a`	Yellow	Runtime access, but narrow output
Exec `ps aux`	Yellow	Process details may be sensitive
Exec `printenv`	Red	Environment variables often contain secrets
Exec `cat /var/run/secrets/...`	Red	Explicit token/secret access
Exec `find / -type f`	Red	Broad filesystem inspection

This model is intentionally simple. It is designed to make risk visible before execution.

Use approval gates, not just polite prompts

An LLM asking "Do you approve?" is not an enforcement mechanism.

For sensitive tools, the platform should pause the actual tool call and require a real approval decision before execution.

A better flow:

User:
  Run uname in this pod.

Agent:
  Risk: Yellow
  Reason: pod exec enters a live container, but the requested command is narrow.

Approval gate:
  Approve / Reject

Tool:
  Runs only after approval.

For Red requests, the platform can either block them outright or require a stronger approval path.

Add command-level guardrails

Start with obvious patterns:

secret
token
password
credential
kubeconfig
printenv
/var/run/secrets

This will not catch everything, but it catches the requests that should never slip through casually.

Then add context:

Is the namespace sensitive?
Is the pod part of the platform control plane?
Is the user asking for broad filesystem access?
Is the command trying to dump environment variables?
Is the command wrapped in a shell?

An exec request is not just a command. It is a command plus a target plus a user plus an operational context.

RBAC still matters

Guardrails do not replace Kubernetes RBAC.

RBAC should still be scoped tightly:

avoid broad cluster-wide exec permissions
prefer namespace scoping where possible
separate read-only tools from write/exec tools
use different service accounts for different risk tiers
keep destructive tools out of default agents

The agent should not have more power than the platform team is willing to expose through a normal operational workflow.

Final thought

Agentic Kubernetes platforms should treat exec with respect.

It is not the same as get pods.

It is closer to temporary access into the runtime boundary of an application. Sometimes that access is exactly what an incident needs, but it should be visible, approved, logged, and risk-classified.

The safest rule is simple:

If an agent can exec into a pod, assume it may be able to see what that pod can see.

Guardrails Before Write Access: Building Agentic Kubernetes Operations with Human Approval

Devam Parikh — Tue, 14 Jul 2026 10:04:22 +0000

Agentic operations for Kubernetes sound exciting until the agent asks to run a production command.

That is the moment the architecture stops being a demo and starts becoming a platform problem.

It is not enough to connect an LLM to Kubernetes tools. A real platform needs boundaries:

What can the agent read?
What can it change?
Who approves risky actions?
Can the agent retrieve secrets?
How do we stop a chat interface from becoming a cluster-admin shortcut?
How do we prove the request actually went through the intended path?

This post walks through a practical pattern for building a guarded Kubernetes agent workflow using chat-based invocation, MCP tools, risk classification, RBAC, and human approval.

The examples are intentionally anonymized and use generic names, but the lessons come from real implementation work.

The platform shape

The architecture had three layers:

Chat surface
  |
  | user asks for inspection/remediation
  v
Agent runtime
  |
  | calls approved tools through MCP
  v
Kubernetes tool server
  |
  | uses a scoped Kubernetes service account
  v
Application cluster

The chat surface can be Slack, Microsoft Teams, or another collaboration tool. The important part is that the agent is invoked where incident response already happens.

The agent runtime is responsible for interpreting the request, selecting tools, summarizing results, and asking for approval when needed.

The Kubernetes tool server is the dangerous part. It is where natural language becomes Kubernetes API access, so it must be designed as a platform boundary rather than a convenience script.

Start with read-only operations

The safest starting point is read-only inspection:

list pods
describe deployments
read events
fetch logs
inspect services, ingresses, gateways, and routes
compare resource state across namespaces or clusters

This already supports many incident workflows.

During an incident, a useful agent does not need to delete a pod immediately. It first needs to gather context quickly:

What changed?
Which pods are unhealthy?
Are there recent restarts?
Are events showing image pull errors, scheduling failures, or failed probes?
Is the service routing to ready endpoints?
Is this isolated to one namespace, one cluster, or many clusters?

Read-only tools let the team gain speed without handing the agent mutation rights on day one.

Add write tools behind approval

Eventually, teams will want remediation actions:

restart a deployment
scale a workload
patch a resource
execute a command in a container
apply a manifest
delete a stuck pod

These tools should not be exposed as ordinary chat commands. They should be protected tools with explicit approval.

A good flow looks like this:

User:
  Run a diagnostic command in this pod.

Agent:
  Risk: Yellow
  Operation: pod exec
  Target: namespace/app-pod
  Reason: exec is read-only in intent, but the target namespace is sensitive.

Approval gate:
  Approve / Reject

Agent:
  Runs the tool only after approval and returns the result.

The agent should explain the risk before the tool runs, but the actual enforcement should live in the platform approval mechanism.

That distinction matters. A model politely asking "Do you approve?" is not the same as an approval gate that pauses the tool call.

Use simple risk classes

You do not need a perfect policy engine on the first day. A simple risk model is already useful:

Green   Low-risk read-only inspection
Yellow  Sensitive read-only action or limited operational change
Red     Destructive, secret-adjacent, broad, or hard-to-reverse action

Example classification:

Request	Risk	Why
List pods in a namespace	Green	Read-only inspection
Get logs from an application pod	Green/Yellow	Depends on log sensitivity
Exec `uname -a` in a pod	Yellow	Exec is powerful even if command is harmless
Scale a deployment	Yellow	Mutates live workload capacity
Delete a pod	Red or Yellow	Depends on namespace, controller, and blast radius
Print environment variables	Red	Secret-adjacent
Read mounted service account token	Red	Secret exposure

The goal is not to make the agent "smart" in a vague way. The goal is to make risk visible before execution.

Treat exec as secret-adjacent

One of the most important lessons: blocking Kubernetes Secret API access is not enough.

A service account may have:

get/list/watch secrets: no
create pods/exec: yes

At first glance, that sounds safe. The agent cannot call kubectl get secret.

But exec can still enter a running container. From inside that container, a user might read:

mounted secret files
service account tokens
environment variables
app config files
credentials written to disk

So the real security posture is:

Kubernetes Secret API access: blocked
Secret exposure through pod runtime/exec: still possible

That does not mean exec must be forbidden everywhere. It means exec deserves special handling:

require approval
classify suspicious commands as Red
block obvious secret-reading patterns
scope RBAC carefully
prefer diagnostic commands with predictable output
avoid exec into sensitive namespaces unless there is a clear reason

Good guardrails understand that Kubernetes permissions and runtime access are related but not identical.

RBAC is part of the product experience

When an agent fails to execute a command, the model may guess why.

For example, it might say:

The operation may have been blocked by policy or GitOps drift prevention.

But the actual reason could simply be Kubernetes RBAC:

cannot patch resource "deployments/scale"

That distinction matters. If the agent guesses, users lose trust.

The platform should capture and surface the real tool error:

RBAC denied
resource not found
container binary not found
namespace blocked by tool guardrail
admission denied
network timeout
invalid command arguments

The agent can summarize, but it should not invent the root cause when the tool already returned one.

Distroless containers change your demo plan

Another practical lesson: many containers do not have a shell.

Commands like these may fail:

sh -c "..."
bash
ps
curl

That does not mean exec is broken. It may just mean the target image is distroless or minimal.

For a reliable demo or diagnostic workflow:

pick commands that exist in the target image
do not assume a shell is available
test against known-good pods
treat "executable file not found" as a container/runtime fact, not an agent failure

This is mundane, but it saves a lot of confusion.

Chat permissions need their own boundary

If an agent is available from a collaboration tool, the chat surface also needs access control.

There are two separate layers:

Where the app can be installed or invoked.
Which teams, channels, or conversations the bot will actually serve.

The first layer is app configuration. The second layer is runtime enforcement.

For sensitive operational agents, runtime allowlists are useful:

ALLOWED_TEAM_IDS
ALLOWED_CHANNEL_IDS
ALLOWED_CONVERSATION_IDS

The agent should reject messages and approval actions from unapproved locations.

This matters because an approval button is also an operational action. It should not be usable from a random personal chat or copied into an uncontrolled place.

Thread context is subtle

Chat threads look obvious to humans, but bot frameworks may not automatically provide the full thread history to the bot.

If the agent is mentioned halfway through a long incident thread, it may only receive the message where it was tagged, not every previous reply.

That means the platform has to choose:

require users to include relevant context in the prompt
fetch thread history through the chat provider API
summarize thread history before passing it to the agent
map each chat thread to a separate agent context

For incident response, the safest model is:

one chat thread = one agent context

That prevents unrelated incidents from accidentally sharing memory.

What I would build first

For an early production version, I would keep the first iteration intentionally conservative:

read-only Kubernetes tools enabled by default
write tools disabled until RBAC and approvals are ready
exec treated as approval-required
secret-looking commands classified as Red
chat channel allowlists enabled
one chat thread mapped to one agent context
real Kubernetes errors surfaced clearly
all tool calls logged with actor, target, risk, and decision

Then I would add automation gradually:

Read-only triage.
Approved exec for diagnostic commands.
Approved rollout restart or scale operations.
Policy-backed remediation.
Pull-request based fixes for configuration changes.

Final thought

Agentic Kubernetes operations should not start with "let the AI fix production."

They should start with:

Can we help humans gather context faster, classify risk clearly, and execute approved actions through a controlled platform path?

That framing is less flashy, but it is much closer to how platform teams can actually adopt agentic systems.

The interesting work is not just connecting an LLM to Kubernetes.

The interesting work is building the boundaries around it.

One Agent, Many Clusters: Federating Kubernetes MCP Tools with Kagent and Agentgateway

Devam Parikh — Tue, 14 Jul 2026 10:04:21 +0000

Platform teams rarely operate one Kubernetes cluster.

A more realistic environment has a management or agent cluster, several application clusters, and a growing set of operational tools that need to work across all of them.

That creates an interesting question for Kubernetes-native agents:

How can one Kagent agent inspect multiple clusters without embedding every cluster connection directly into the agent?

This post walks through a practical pattern: run Kagent in an agent cluster, expose Kubernetes MCP servers from application clusters, and federate those MCP servers through Agentgateway Virtual MCP.

The result is a single MCP endpoint where tools are namespaced by cluster, such as:

cluster-a_k8s_get_resources
cluster-a_k8s_describe_resource
cluster-b_k8s_get_resources
cluster-b_k8s_describe_resource

The cluster names above are examples. In your own environment, the prefixes should be boring, obvious identifiers: prod_eu, stage_us, payments, search, or whatever matches your operating model.

Why this pattern matters

Kagent is Kubernetes-native: agents, model configuration, and tool connections are represented as Kubernetes resources.

That is powerful because platform teams can manage agents the same way they manage the rest of their platform: declaratively, with GitOps, RBAC, reviews, and repeatable rollouts.

MCP gives agents a standard way to discover and call tools. Agentgateway adds the routing layer that makes MCP manageable in a platform setting.

Instead of every agent talking directly to every MCP server, Agentgateway can expose a stable endpoint, route to one or more MCP backends, and make the resulting tool surface easier to govern.

For multi-cluster operations, the important bit is Virtual MCP: multiple MCP servers can be federated behind a single gateway endpoint. Each backend contributes tools, and the gateway prefixes those tools so clients can tell which backend they are calling.

That gives you a useful separation of concerns:

Application clusters own their local Kubernetes MCP servers.
The agent cluster owns Kagent, the RemoteMCPServer, and the agent definitions.
Agentgateway owns the federated MCP endpoint that joins those worlds.
Agents choose the correct cluster by selecting the correctly prefixed tool.

The architecture

At a high level, the setup looks like this:

Kagent Agent
  |
  | RemoteMCPServer
  v
Agentgateway endpoint
  |
  | Virtual MCP targets
  +--> Cluster A Kubernetes MCP server
  |
  +--> Cluster B Kubernetes MCP server

Each application cluster exposes a Kubernetes MCP server over HTTP. The agent cluster does not need to run all of those MCP servers locally. It only needs network reachability to their MCP endpoints.

Agentgateway is configured with an AgentgatewayBackend that contains one target per application cluster.

In this example, the gateway uses static targets because the MCP servers live behind cross-cluster addresses rather than ordinary in-cluster Services:

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
  name: application-clusters-k8s-readonly
  namespace: platform-mcps
spec:
  mcp:
    failureMode: FailOpen
    sessionRouting: Stateless
    targets:
      - name: cluster-a
        static:
          host: cluster-a-mcp.example.internal
          port: 80
          path: /mcp
          protocol: StreamableHTTP
      - name: cluster-b
        static:
          host: cluster-b-mcp.example.internal
          port: 80
          path: /mcp
          protocol: StreamableHTTP

The target names matter. They become part of the tool names that the agent sees.

If cluster-a exposes a tool named k8s_get_resources, the federated endpoint exposes it as cluster-a_k8s_get_resources.

An HTTPRoute then publishes the federated backend on a stable path:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: application-clusters-k8s-readonly
  namespace: platform-mcps
spec:
  parentRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: shared-mcps
      namespace: platform-mcps
  rules:
    - backendRefs:
        - group: agentgateway.dev
          kind: AgentgatewayBackend
          name: application-clusters-k8s-readonly
          namespace: platform-mcps
          weight: 1
      matches:
        - path:
            type: PathPrefix
            value: /mcps/application-clusters-k8s-readonly

From Kagent's point of view, this is just a remote MCP server:

apiVersion: kagent.dev/v1alpha2
kind: RemoteMCPServer
metadata:
  name: application-clusters-k8s-readonly
  namespace: kagent
spec:
  description: Federated read-only Kubernetes MCP tools for application clusters
  protocol: STREAMABLE_HTTP
  terminateOnClose: true
  url: http://shared-mcps.platform-mcps.svc.cluster.local:80/mcps/application-clusters-k8s-readonly/mcp

Finally, the agent can be configured to use the federated MCP server and explicitly list the tools it is allowed to call:

apiVersion: kagent.dev/v1alpha2
kind: Agent
metadata:
  name: application-clusters-mcp-test-agent
  namespace: kagent
spec:
  description: Test agent for federated application-cluster Kubernetes MCP access
  type: Declarative
  declarative:
    modelConfig: default-model-config
    runtime: python
    systemMessage: |
      You are a read-only Kubernetes assistant for application clusters.
      Use cluster-prefixed MCP tools to inspect live resources.
      Use cluster-a_* tools for Cluster A and cluster-b_* tools for Cluster B.
      Do not suggest or perform write operations.
    tools:
      - type: McpServer
        mcpServer:
          apiGroup: kagent.dev
          kind: RemoteMCPServer
          name: application-clusters-k8s-readonly
          toolNames:
            - cluster-a_k8s_describe_resource
            - cluster-a_k8s_get_resources
            - cluster-a_k8s_get_resource_yaml
            - cluster-b_k8s_describe_resource
            - cluster-b_k8s_get_resources
            - cluster-b_k8s_get_resource_yaml

What worked well

The cleanest part of the design is that the agent only needs one MCP connection. Tool selection becomes a routing decision expressed in the tool name.

For example:

"List pods in Cluster A" maps to cluster-a_k8s_get_resources.
"Show events in Cluster B" maps to cluster-b_k8s_get_events.
"Describe this deployment in Cluster A" maps to cluster-a_k8s_describe_resource.

That makes the agent prompt simpler and makes operational boundaries easier to see.

You can also keep the tools read-only, which is a good default for early production experiments. The agent can inspect resources, summarize state, compare clusters, and gather troubleshooting context without being able to mutate the environment.

failureMode: FailOpen is also useful for multi-cluster setups. If one target is unavailable, the gateway can continue serving tools from healthy targets instead of failing the entire federated session. That is often the behavior you want when clusters are managed independently.

A practical lesson: static targets vs. selectors

Agentgateway supports selector-based MCP federation, which is a nice fit when the MCP servers are ordinary Kubernetes Services in the same cluster.

In cross-cluster setups, though, the MCP endpoints may be represented by external DNS names, service network addresses, or other forms of cross-cluster connectivity. In that case, static targets are often the more predictable model:

The target name is explicit.
The host, port, path, and protocol are explicit.
Tool prefixes are stable.
The agent-facing endpoint stays the same as clusters are added or removed.

Selectors are still worth considering when your MCP servers are discoverable as regular in-cluster Services with the expected labels and protocol metadata.

For cross-cluster endpoints, start with static targets and automate their generation later.

Verification checklist

A good rollout should prove each layer independently:

Confirm each application-cluster MCP endpoint responds directly.
Confirm the AgentgatewayBackend is accepted.
Confirm the HTTPRoute is accepted and resolves the backend reference.
Connect to the federated MCP endpoint and list tools.
Verify that tools from every cluster are present with the expected prefixes.
Create or update the Kagent RemoteMCPServer.
Confirm Kagent discovers the federated tools.
Invoke the agent and ask it to query both clusters.

The most important test is the last one.

A tool list proves discovery. An agent invocation proves the complete path: prompt, tool selection, MCP call, gateway routing, backend MCP server, and response synthesis.

The screenshot below shows the kind of tool discovery output you want to see: prefixed Kubernetes tools from more than one application cluster available through a single federated MCP server.

Design principles for production use

This pattern is intentionally simple, but it benefits from a few production-minded constraints:

Start read-only. Inspection tools are enough for a lot of platform and SRE workflows.
Use clear cluster prefixes. Humans should be able to predict the right tool name.
Keep the agent prompt explicit. Tell the agent how prefixes map to clusters.
Treat the gateway endpoint as a platform API. Review it, version it, and observe it.
Prefer least privilege. The MCP server in each cluster should only have the permissions it needs.
Add write tools later, behind approvals and stronger policy.
Verify with real agent invocations, not only resource status.

Where this goes next

Once the basic pattern works, the next step is automation.

Platform teams can generate Agentgateway targets from a cluster registry, cloud service discovery, GitOps inventory, or another source of truth. They can also attach policies for tool access, rate limits, audit logging, and identity-aware authorization.

The bigger idea is not just "one agent can see many clusters." It is that MCP tools can become a governed platform surface.

Kagent provides the Kubernetes-native agent model, application clusters provide local operational tools, and Agentgateway gives the platform a single place to federate, route, and control access.

That is a practical foundation for multi-cluster agentic operations: one agent-facing endpoint, many cluster-local tools, and a clear path from read-only inspection toward safer automated remediation.

This post is based on a KAgent website contribution draft: https://github.com/kagent-dev/website/pull/401

Testing Application Resilience: How to Stop Amazon ElastiCache Cluster and Manage Traffic

Devam Parikh — Fri, 13 Oct 2023 06:02:29 +0000

Introduction

As developers, it is crucial to test the resiliency of our applications and understand how they handle failures or disruptions. In this blog post, we will explore a scenario where we need to stop an Amazon ElastiCache cluster to see how our application behaves when Redis is unavailable. Although ElastiCache clusters cannot be stopped, we will discuss alternative approaches to achieve our testing objective.

Understanding Amazon ElastiCache

Amazon ElastiCache for Redis is a powerful in-memory data structure service that provides real-time performance for modern applications. It serves as a cache or a data store, delivering high-speed access to data. ElastiCache uses a synchronous replication mechanism to maintain data consistency across its nodes.

Challenges with Stopping ElastiCache Cluster

Stopping an ElastiCache cluster is not possible due to the synchronous replication mechanism. If we stop a node, the cluster's redundancy is compromised, potentially leading to instability or complete failure. However, we can explore other methods to create scenarios where our application experiences Redis unavailability.

Blocking Incoming Traffic using Security Groups

To simulate Redis unavailability, we can block incoming traffic to the ElastiCache cluster. Security groups act as virtual firewalls, controlling inbound and outbound traffic. By removing all the inbound rules for the ElastiCache cluster, we can prevent any incoming requests from reaching it.

However, it is essential to understand that security groups are stateful[1]. This means that existing connections are not interrupted when security group rules are changed. Thus, our application may still be connected to the ElastiCache cluster.

Addressing the Issue

Two methods can be used to tackle this issue:

1. Restarting the Application: By restarting the application, existing connections will be terminated, forcing the application to establish new connections. This can validate the application's ability to handle Redis unavailability.

2. Using Network ACLs: Network Access Control Lists (ACLs)[2] operate at the subnet level and allow or deny specific inbound or outbound traffic. Unlike security groups, network ACLs are stateless, meaning they don't automatically allow response traffic. Introducing a network ACL that blocks traffic in either direction can break existing connections.

Network ACL in Depth

You can either use the default VPC network ACL or create a custom one with rules similar to security groups for extra VPC security at no extra cost.

The following diagram depicts a VPC with two subnets, each having its network ACL. When traffic enters the VPC (such as from a peered VPC, VPN connection, or the internet) the router directs it to its destination.

Network ACL A controls which traffic can enter subnet 1 and leaves it to destination outside subnet 1. Similarly, network ACL B regulates traffic entering and leaving subnet 2.

Creating a Custom Network ACL

As illustrated in the figure below, this is how I've configured the denial of incoming traffic from my application to the ElastiCache cluster.

A network ACL comprises both inbound and outbound rules, each capable of allowing or denying traffic. These rules are numbered from 1 to 32766.

When determining whether to allow or deny traffic, we evaluate the rules sequentially, starting with the lowest numbered rule. If a rule matches the traffic, it is applied, and no further rules are assessed.

Conclusion

Testing application resilience is essential to ensure smooth operation in challenging scenarios. While stopping an ElastiCache cluster is not feasible due to its replication mechanism, alternative approaches such as blocking incoming traffic using security groups or employing network ACLs can help simulate Redis unavailability. By understanding the statefulness of security groups and the statelessness of network ACLs, we can effectively test our application's behaviour when critical resources are not available.

In summary, remember these key points:

ElastiCache clusters cannot be stopped and rely on synchronous replication for real-time performance.
Security groups are stateful, meaning existing connections persist when rules are modified.
Network ACLs are stateless and can be used to block traffic, potentially breaking existing connections.

Reference:
[1] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/security-group-connection-tracking.html
[2] https://docs.aws.amazon.com/vpc/latest/userguide/vpc-network-acls.html

Demystifying DDoS Attacks and CloudFront: A DevSecOps Guide

Devam Parikh — Fri, 21 Jul 2023 10:49:11 +0000

The IT industry is currently facing the worst crisis in its history, with numerous factors affecting its stability. One significant factor that demands attention is the rising number of DDoS attacks. According to the reports available to me, there was a 150% increase in DDoS attacks in 2022 compared to 2021. Moreover, experts predict a continued upward trend in the upcoming years.

Attacks such as DDoS can have a significant impact on organizations, causing service disruptions, financial losses, reputation damage, data loss, increased vulnerability to other attacks, and more.

In today's digital landscape, understanding and mitigating DDoS attacks is crucial for DevSecOps professionals. In this blog post, we'll explore the intricacies of DDoS attacks and how CloudFront, Amazon Web Services' (AWS) content delivery network (CDN) service, can be a valuable tool in protecting your applications and infrastructure. By answering key questions along the way, we'll unravel the mysteries surrounding DDoS attacks and equip you with the knowledge to fortify your systems.

What do you need to know about DDoS attackt as DevSecOps?

A DDoS attack, or distributed denial-of-service attack, is a cyber-attack that targets a website or server by flooding it with so much traffic that it becomes unavailable to legitimate users. The goal of a DDoS attack is to disrupt the targeted website or server's services, making it inaccessible to users.

There are two main types of DDoS attacks: volume-based attacks and application-layer attacks. Volume-based attacks flood the target with a large amount of traffic, overwhelming its resources and making it unable to handle legitimate requests. Application-layer attacks exploit vulnerabilities in the targeted website or server's applications, causing them to crash or malfunction.

To launch a DDoS attack, the attacker first identifies the target website or server. Then, they gather a large number of infected computers, called "zombies," to flood the target with traffic. This overwhelms the website or server, making it unable to handle legitimate requests. As a result, users are unable to access the website or server.

How can CloudFront help with hosting and DDoS attackts?

CloudFront is a content delivery network (CDN) service that speeds up the delivery of your content to users all over the world by caching your content in edge locations. These edge locations are servers that are located close to your users. When a user requests your content, CloudFront delivers it from the edge location that is closest to them, which reduces latency and improves performance.

As most requests are answered by the edge location (from the cache), this significantly reduces the load on the origin server. The edge location acts as an absorbing agent, preventing requests from reaching the origin server directly, as they only need to pass through the edge location.

Why do you need to have WAF protection enabled for CloudFront distribution?

As discussed above, DDoS attacks rely on "zombies" to attack. Therefore, blocking the requests from IP addresses of zombie machines can help us to stop DDoS attacks.

A web application firewall (WAF) serves as a protective layer in front of web applications, shielding them from common web exploits. WAFs function by inspecting HTTP requests and blocking those that match known attack patterns.

CloudFront has a built-in integration with WAF, which implies that if WAF is enabled for any CloudFront distribution, AWS will automatically create and handle the WAF Rules for you. These rules include the following:

Protect against the most common vulnerabilities found in web applications.
Protect against malicious actors discovering application vulnerabilities.
Block IP addresses from potential threats based on Amazon internal threat intelligence.

After enabling WAF for a CloudFront distribution, it would appear similar to the below image.

How to setup logging for CloudFront?

Before delving into the process of setting up logging, let's first understand why logging is crucial. Logging plays a pivotal role in enabling you to identify the source of an attack, mitigate the attack, and prevent future attacks.

You can easily set up logging with just one click to send your logs to CloudWatch, S3, or Kinesis Data Firehose. Among these options, S3 is the most cost-effective choice.

Bonus

Utilize Athena to query data from any S3 bucket. It is recommended to use separate S3 buckets for different environments as this will enhance Athena's performance.

Conclusion

By delving into the above subtopics, we've gained a comprehensive understanding of DDoS attacks and how CloudFront can fortify your DevSecOps practices. Armed with this knowledge, you're better equipped to protect your applications and infrastructure against potential threats. Stay vigilant, stay informed, and leverage the power of CloudFront to bolster your defenses.