DEV Community: Krishan Thisera

Designing for Assumed Compromise: Securing Autonomous AI Agents on Kubernetes

Krishan Thisera — Fri, 18 Sep 2026 08:44:06 +0000

A Defense-in-Depth Blueprint from Orchestration to Identity, Credentials, and Scale

An agent sandbox is a runtime instance, a container, microVM, or Kubernetes pod, built to run an autonomous AI agent's generated code and tool calls behind an isolation boundary. That boundary is meant to keep a compromised session from reaching the host machine or other tenants. How effectively it does that depends on which isolation technique backs it. Autonomous coding and browsing agents built on large language models now routinely generate and execute code, install packages, browse the live web, and handle credentials. Each of those actions carries only probabilistic guarantees about what the model will do, and that uncertainty is the problem an agent sandbox exists to solve.

That combination compounds two risks: arbitrary code execution and instructions an attacker can influence through prompt injection. Together, they have pushed the infrastructure conversation away from asking whether a container is hardened enough, toward assuming the container will eventually be compromised and designing every layer around that assumption.

This article covers that security posture, together with what it costs and how you govern a fleet of sessions rather than one, once you run it at scale.

Chapter 1: The Kubernetes-Native Orchestration Primitive

Kubernetes offers two dominant workload abstractions. Deployments manage stateless, interchangeable pods, and StatefulSets manage numbered fleets with generic identities. Neither fits an agent session, a singleton needing a stable hostname, persistent storage, and a lifecycle that pauses and resumes rather than only starting and stopping. Teams hand-assembled a StatefulSet of one, a Service, and a persistent volume claim (PVC). That combination had no shared lifecycle controller.

The Sandbox custom resource definition (CRD), maintained by kubernetes-sigs/agent-sandbox under Kubernetes SIG Apps, closes that gap with a declarative API for one stateful pod with a stable identity. Three CRDs extend it:

SandboxTemplate enforces security defaults.
SandboxWarmPool pre-provisions instances to cut cold-start latency.
SandboxClaim lets a framework request an instance without managing the template.

Chapter 2: Choosing an Isolation Technique

The Sandbox CRD manages a pod under the hood, so it inherits the standard Kubernetes runtimeClassName field for selecting the runtime, the same field any pod-based workload can set. runc, gVisor, and Kata Containers are the common runtime options it selects between, and seccomp layers a syscall filter on top of whichever one is chosen. Together the four trade isolation strength against cost differently. runc is the default low-level runtime that Docker and containerd use to create every container. It offers the least isolation of the four. It isolates a process using the host kernel's own namespaces and control groups (cgroups), so every container on a node shares a single kernel. That is fine for trusted code, but risky once an autonomous agent might run attacker-influenced instructions inside it.

seccomp (secure computing mode) is a Linux kernel facility that lets an operator allowlist or denylist the syscalls a container may invoke, rejecting anything outside that list. It adds no new kernel boundary. The host kernel still handles every allowed syscall directly, which keeps it cheap enough to layer onto almost any container. According to NVIDIA, it is one item in a defense-in-depth stack, suited to trusted internal automation rather than genuinely untrusted, agent-generated code.

gVisor adds the kernel boundary that seccomp does not. It interposes a user-space component called the Sentry between a container's processes and the real kernel. The Sentry intercepts every system call and handles it through a restricted reimplementation of the kernel surface. That shrinks the hundreds of syscalls a workload can reach down to a minimal, vetted subset. gVisor's overhead is commonly cited as 10 to 30 percent on I/O-heavy workloads, with little impact on compute-heavy work such as model inference. According to gVisor's performance guide, though, overhead varies considerably by platform and workload, and small, syscall-heavy operations can exceed that range. That makes gVisor a strong fit for CPU-bound tasks and a weaker one for constant file or network I/O.

Kata Containers provides the most isolation of the four, running a pod inside hardware-assisted virtualisation instead of a shared kernel. A hypervisor such as Firecracker creates a genuinely separate virtual machine with its own guest kernel, using the processor's virtualisation features rather than the host kernel's namespaces. Firecracker is a purpose-built hypervisor developed at AWS to back Lambda and Fargate, with reported boot times clustering around 100 to 125 milliseconds and memory overhead under 5 megabytes per microVM. Firecracker speaks its own API rather than Kubernetes', so Kata Containers closes that gap by implementing the standard Container Runtime Interface (CRI). A Kata-backed pod then schedules through the same kubectl apply workflow, with only a runtimeClassName field marking the difference. Kata supports Cloud Hypervisor as its default backend, with Firecracker and QEMU as alternatives, and boots in roughly 150 to 300 milliseconds. Because the isolation boundary is an entire separate kernel, an attacker who compromises the workload still has to escape both the guest kernel and the hypervisor to reach the host. That closes off namespace abuse and kernel-level exploits regardless of what the agent inside does.

None of this comes free. Even at 100 to 300 milliseconds, microVM boot time is measurable overhead once multiplied across every agent turn in a busy pipeline. Running Kata in production also means managing nested virtualisation support and a larger per-pod memory footprint than a bare container carries. seccomp and gVisor trade some of that isolation strength back for lower cost and higher throughput, which is exactly why the RuntimeClass mechanism matters. A single cluster can run a short-lived code interpreter session under gVisor, tolerating little latency budget but running cheaply at high volume, alongside a long-running, browser-equipped agent under Kata or Firecracker, through the same API.

Docker-in-Docker, isolated at the hypervisor level

Letting an agent build or run its own containers has conventionally meant Docker-in-Docker (DinD), a nested daemon in privileged mode, or the host's Docker socket mounted directly into the container. Both are well-known escape vectors, acceptable for trusted CI/CD pipelines but not for an autonomous agent whose build steps could be influenced by a prompt injected through a compromised dependency.

Docker Sandboxes removes both vectors instead of trying to harden them. Each agent runs inside its own rootless microVM with its own private Docker daemon, isolated at the hypervisor level rather than sharing the host's, with full privileges granted only inside that guest. In its "direct" mode, file changes still sync back to the host filesystem, so any git hooks, install scripts, or task configuration the agent touched need review before a human executes them locally. Closing this gap requires review discipline, not additional infrastructure. It means diffing hooks and checking scripts after each session.

Chapter 3: Threat Modelling for Assumed Compromise

The central threat behind every isolation decision is indirect prompt injection. Direct prompt injection, a user typing malicious instructions into a chat box, is comparatively well understood. Indirect prompt injection hides those instructions inside content the agent is expected to read as normal work. Vectors include a compromised file in a repository, a pull request description, a configuration file such as .cursorrules or CLAUDE.md, and a response from an MCP (Model Context Protocol) server. According to NVIDIA's AI Red Team guidance on sandboxing agentic workflows, an agent cannot reliably tell an operator's instructions apart from text inside a document it was asked to process. That means anyone who can put content in front of it gains a channel for influencing its actions.

Filtering the prompt cannot close this gap. Agentic tools execute arbitrary code by design. Once an action passes into a subprocess, the application has no visibility into or control over what that subprocess does. A compromised agent can also route around an allowlist by calling a restricted tool indirectly through one that is already approved. According to NVIDIA's guidance, only isolation with a kernel boundary of its own can reliably contain the risk, because containment then does not depend on the model behaving as instructed.

NVIDIA's guidance replaces a binary trusted-or-not judgement with four escalating rules:

Non-overridable enterprise denylist. An absolute floor of blocked operations, such as reads or writes to credential files or hooks, that neither a user nor the agent can approve away.
Workspace-scoped, allow-by-default access. Read and write permission inside the agent's active project directory, granted without approval for every action, since constant friction on routine work undermines the practice of approval altogether.
Narrow allowlisted exceptions. Specific operations outside the workspace that the agent's job still requires, such as a named SSH key for a legitimate git operation, approved individually.
Default-deny. Fresh manual approval required for everything else, never cached. A cached "yes" from an earlier legitimate action can be silently reused by an attacker-influenced action that looks similar later in the same session.

The same principle underlies all four rules. Each applies least privilege continuously, rather than only once at startup. A kernel boundary limits what a compromised process can touch. A network allowlist limits where it can send data. A credential proxy, covered in Chapter 4, limits what it can ever possess. Each is the same idea, enforced at a different layer.

Chapter 4: Network, Credential, and Identity Boundaries

Chapter 3 closed by naming three layers where least privilege gets enforced: a kernel boundary, a network allowlist, and a credential proxy. This chapter covers the latter two, network and credential boundaries, and adds a third: identity management. Together, egress control, credential handling, and identity management govern what a compromised agent can still do once it is talking to the outside world. That is where most of the actual damage from indirect prompt injection gets carried out.

Egress filtering is not content-level safety

Network egress control matters independently of kernel isolation, because even a sandbox that holds does not stop a compromised agent from sending data out through a connection it was always permitted to make. According to NVIDIA's guidance, egress filtering is a mandatory baseline control, not optional hardening. It calls for blocking outbound connections to unknown destinations by default. That boundary should be enforced through enterprise denylists, HTTP proxies, and DNS-level restriction, rather than trusting the agent's own code to self-police.

Domain-level allowlisting is not the same thing as content-level safety. Permitting a broad domain such as github.com allows access to any content hosted there. An attacker who can post data to an already-allowed gist or issue comment therefore has an exfiltration channel a domain filter will never flag.

Production implementations span a real spectrum rather than one fixed policy. At one end, a narrow allowlist permits only a model API endpoint and a package registry. At the other, full air-gapping removes all internet access for high-risk batch jobs with no legitimate need to reach it.

The same logic applies inside an organisation's network, not only at its external edge: a destination being reachable doesn't mean it's safe to trust. Zero-trust segmentation is what enforces that internally. An agent sandbox that can reach a production database or a deployment credential store has simply moved the exfiltration risk internally rather than removed it. In multi-tenant designs, that internal segmentation needs to be mandatory, not optional, on equal footing with external egress control.

Credentials that never enter the sandbox

The conventional pattern for handling secrets is to inject an API key into a sandbox as an environment variable through a Kubernetes secretKeyRef. That pattern has an irreducible weakness. The agent process itself can still read those environment variables at runtime.

Three failure modes follow from that fact alone, regardless of how well the surrounding sandbox is isolated:

The agent inadvertently includes the secret in a generated response or log.
A prompt-injected agent deliberately exfiltrates it through an otherwise-legitimate outbound call.
An adversarial input tricks the agent into printing or transmitting its own environment variables.

The credential-proxy pattern, implemented independently by projects including Infisical's Agent Vault, inverts this. The sandbox holds no secret at all. It routes outbound HTTPS traffic to a proxy, typically through the standard HTTPS_PROXY variable. That proxy terminates TLS and strips the placeholder credential the agent's request carried. It then injects the real credential from an encrypted store and re-establishes the connection to the actual upstream service. A session-level agent token issued at proxy handshake ties every request to a specific agent. That is what stops one sandbox from riding another's credentials, and it lets the proxy log, rate-limit, and revoke access mid-session without touching the sandbox itself. Because the sandbox never holds the real secret, none of the three failure modes above can occur. There is nothing for the agent to leak, exfiltrate, or be tricked into revealing.

Three approaches implement this pattern today, and they differ in what infrastructure they assume and how mature each one is. A team already running Istio can build it directly on EnvoyFilter and ext_authz. That path requires Istio's full service mesh to already be in place, and it inherits the fragility that Istio's documentation attributes to that escape-hatch mechanism. A purpose-built AI gateway requires no service mesh at all. agentgateway and Envoy AI Gateway both run as a standalone binary. Infisical's Agent Proxy is the third option, a vendor-managed proxy. Agent Proxy also requires no service mesh, but delegates secret storage to Infisical's own service rather than an operator-controlled vault. All three remain a single point of failure if the proxy itself is compromised.

No single mechanism covers identity completely

Three largely separate mechanisms address what identity an agent, or a sub-agent it spawns, actually carries, and none covers the problem completely. Bearer tokens and long-lived API keys assume a human authenticates once and performs a bounded set of actions. Handing an autonomous agent one broad token instead creates ambient authority, letting it exercise everything the token permits at any time, with no link between a specific tool call and a specific authorisation decision.

The problem compounds with sub-agents. Whichever of three common patterns a team picks carries its own risk:

Passing the parent's token down to a sub-agent erases the audit trail's ability to distinguish parent from child.
Issuing a new static credential per sub-agent multiplies the credentials that can leak.
Running a sub-agent unauthenticated is a compliance failure outright.

SPIFFE and its SPIRE implementation issue a short-lived, cryptographically verifiable identity based on runtime attestation rather than a static secret. SPIRE, however, expects every workload variant pre-registered ahead of time and delivers identity through a pull-based call, both of which sit awkwardly against a sandbox whose useful life might be seconds.

Cloud workload identity federation authenticates a pod to its cloud provider, but only coarsely. Google Kubernetes Engine (GKE) implements this through Workload Identity Federation, which makes every pod sharing one Kubernetes ServiceAccount cryptographically indistinguishable to Google Cloud IAM. Amazon Elastic Kubernetes Service (EKS) offers a choice between two mechanisms, each with a different tradeoff. IAM Roles for Service Accounts (IRSA) refreshes credentials without a pod restart, at the cost of AWS Security Token Service (STS) quota. EKS Pod Identity avoids that quota cost but can cache credentials for up to six hours before a permission change takes effect. Delegation-aware authorisation systems such as OpenFGA and SpiceDB fill the remaining gap, checking each individual tool call against an explicit graph of user and sub-agent permissions rather than authorising a session once at the start.

At the time of writing, stitching these three layers together remains real engineering work rather than a documented, turnkey integration.

Chapter 5: Scaling and Scheduling Agent Workloads

Chapters 2 through 4 covered how to secure a single sandbox. Running thousands of them at once introduces a different problem. It means avoiding a multi-second wait every time an agent starts a new turn, and stopping one tenant's burst of activity from crowding out every other tenant on the same infrastructure.

Balancing statefulness and disposability

An agent sandbox has to satisfy two requirements that pull in opposite directions. Many agent workloads are genuinely stateful. A persistent notebook kernel needs to preserve variables across cells, and a coding-agent workspace needs its file changes to survive across many separate tool calls. Both want a stable hostname and durable storage. At the same time, the premise of a sandbox, as distinct from a persistent server, is that it should be cheap to start per task and safe to tear down immediately afterward.

This differs from the ephemeral containers used in continuous integration and delivery (CI/CD). A CI/CD container starts, runs a predetermined pipeline of steps, and is discarded, with no expectation of resuming mid-pipeline. An agent sandbox typically needs to persist across many discrete tool calls over an extended, possibly paused, conversation. It might sit idle for minutes while a person reviews its output, then resume with the same files, the same running processes, and the same hostname. Chapter 1 covered the interface built for this, the Sandbox custom resource's declarative API for a single stateful pod with a stable identity. It exists because a hand-rolled StatefulSet, Service, and persistent volume claim had no shared lifecycle controller to reproduce that fidelity.

Warm pools and checkpoint-restore

Chapter 1 also introduced SandboxWarmPool, a pool of pre-booted sandboxes that lets a new session claim one instead of creating it from scratch. This brings allocation down to the millisecond range. A fresh pod with a hardware-isolated kernel boot (Chapter 2) costs seconds instead, an order of magnitude slower.

GKE adds a further refinement. Pod Snapshots is a checkpoint-and-restore capability that captures a running pod's full state and cuts startup time from minutes to seconds, including for GPU-backed workloads that are expensive to cold-boot. A companion capability, snapshot-pausing, reclaims an idle sandbox's compute without discarding its state the way a hard termination would.

EKS has no equivalent yet. AWS does support checkpoint and restore, built on Checkpoint/Restore In Userspace (CRIU) through the Kubernetes kubelet checkpoint API. Its purpose is different. It exists for forensic evidence preservation, capturing a suspicious container's state without killing it so an investigator can examine it later. This forensic checkpointing is not a substitute for GKE's cold-start capability.

One project outside AWS closes part of this gap today. Agent Substrate, a joint effort between Solo.io and Google, runs its own cross-cloud snapshot layer, independent of any single vendor's kubelet feature, coordinating full-state snapshots across an agent sandbox's suspend and resume cycles.

Quotas, tokens, and rate limits for a sandbox fleet

Kubernetes' standard ResourceQuota object sets a namespace-level ceiling on aggregate CPU, memory, and object counts, the same primitive any multi-tenant cluster already relies on. LimitRange complements it by bounding what any single pod or container inside that namespace can request, so one workload cannot claim the entire quota for itself. Neither one governs what a workload does over the network, which the egress allowlists from Chapter 4 control separately. Many of the threat categories catalogued by OWASP for agentic systems operate entirely within otherwise-permitted resource and network boundaries. That means an attack can satisfy both checks at once and still succeed. A compromised agent exfiltrating data through an already-allowed API call, for instance, breaches no quota at all. Quota and network policy alone cannot catch that, which is why agent workloads need a further, behavioural layer constraining the pattern of an agent's actions rather than only their volume.

This behavioural layer governs token consumption most directly, typically enforced at the same AI gateway introduced in Chapter 4. Envoy AI Gateway's QuotaPolicy and agentgateway's token budgets both implement it as a three-level hierarchy. An overall platform budget subdivides into per-tenant budgets, which further subdivide into optional per-agent budgets. The gateway rejects a request once its level's budget runs out.

Rate limiting for agent fleets operates at two distinct layers, and mixing them up causes real problems. One governs how many outbound calls an agent may make against an external service. The other governs how many sandboxes, and how much aggregate compute, a tenant may have running at once. Confusing them produces either over-restrictive throttling or under-enforcement. Over-restrictive throttling kicks in when compute quota looks tight but the real bottleneck is API calls. Under-enforcement lets a tenant within its compute quota keep overloading an external API for everyone sharing that connection.

The first layer sits naturally at the network and proxy layer. The same egress point that injects credentials in Chapter 4 typically enforces it too, so a platform already routing traffic through that proxy for credentials gets per-tenant throttling as an adjacent policy. The second layer sits at the scheduler and quota layer described above.

Agent workloads are bursty. A single user task can fan out into several rapid tool calls, or spawn sub-agents that act concurrently. A per-tenant limit checked independently by each agent cannot see the other agents' concurrent draws against the same shared budget. An external rate-limit service closes this gap. Envoy AI Gateway and agentgateway both call one to share a budget across replicas. The service checks and decrements one shared counter atomically, so concurrent agents see a consistent, current balance rather than a stale per-replica count.

What isolation actually costs

Chapter 2 covered isolation's overhead in compute and latency terms. What it costs in dollars is harder to establish. No source reviewed for this article publishes a like-for-like comparison of what a single provider charges for the same workload under runc, seccomp, gVisor, and Kata Containers. You cannot currently look up what stronger isolation costs while holding everything else constant. You have to benchmark it yourself.

Published pricing does make one part of the spectrum concrete, the deployment model. At a reference scale of 200 concurrently running sandboxes, a managed, shared platform lists monthly pricing around US\$7,200. A comparable footprint under a bring-your-own-cloud model costs about US\$2,060 (more than a threefold difference) for broadly similar isolation and capability. Most of that gap is operational and multi-tenancy overhead a fully managed platform absorbs on a customer's behalf, separate from the isolation technology itself.

Chapter 6: Benchmark Evidence and Attack Surfaces Beyond Code

Chapters 2 and 3 argued, from engineering principles, that hardware isolation (gVisor's kernel boundary, Kata's separate guest kernel) is necessary once an agent might execute attacker-influenced code. This chapter tests that argument against evidence, drawing on a 2026 benchmark measuring real escape attempts and an attack surface those technologies were never built to cover.

What SandboxEscapeBench found

SandboxEscapeBench tests whether a model with root access inside a container can find its way out to the host. It spans 18 scenarios across the orchestration, runtime, and kernel layers, each rated by difficulty. Frontier models reliably solved easy misconfigurations, such as privileged containers and exposed Docker sockets. On medium-difficulty exploits requiring genuine multi-step work, GPT-5 and Claude Opus 4.5 both scored around 0.40. On kernel-level exploits, every model tested scored zero.

The gap between model tiers was large. Claude Opus scored 0.49 overall against Haiku's 0.18, roughly threefold, and performance scaled with the compute a model was given. Plain Docker isolation is insufficient by default. Real deployments should be stress-tested rather than assumed safe. No novel vulnerabilities were used, only known misconfigurations and disclosed Common Vulnerabilities and Exposures (CVE) entries. This remains a single study, not yet independently replicated.

These results are worth reading together: escape success scaled with the compute a model was given, and every model scored zero on every kernel-level exploit. Together, they suggest the current zero-success ceiling reflects where model capability stands today, not a permanent limit. If that holds, three directions follow:

Hardware isolation stops being reserved for high-risk workloads and becomes the default for any untrusted one.
Credential proxying and egress filtering move from a feature teams opt into to one platforms ship by default.
Escape benchmarking becomes a routine part of due diligence, the way penetration testing did for cloud infrastructure.

The browser as a different kind of risk

Browsing agents introduce a second execution surface that Chapter 2's isolation technologies do not cover. The risk is semantic rather than technical. A malicious instruction hidden in a web page's rendered content does not need to escape a sandbox. It manipulates the agent's reasoning directly, the same indirect prompt injection risk from Chapter 3, with a page rather than a file as the vector. The stakes are higher because of what a browsing session carries, live authentication cookies and sometimes payment details, which a manipulated agent can leak through actions the browser was always permitted to take.

Browser isolation complements code-execution sandboxing rather than replacing it, typically through an HTTP-level policy layer sitting above the browser's existing process sandboxing. It remains younger and less standardised than code-execution isolation, with no equivalent yet of Chapter 1's RuntimeClass for an isolated browser session.

Chapter 7: Buy Versus Build

Everything from Chapter 1's orchestration primitive to Chapter 6's benchmark evidence describes a stack a team could build. Most teams do not build it themselves. They buy a hosted sandbox instead, from a provider such as E2B, Fly.io, Modal or Northflank. Each sells a scheduled, isolated, credential-brokered sandbox. None of them require the buyer to assemble the underlying cluster, isolation configuration, credential proxy and governance layer.

The isolation layer is not a differentiator among these vendors. Nearly every one uses microVM or gVisor isolation, the two strongest techniques Chapter 2 covered. The industry already treats both as sufficient for untrusted, LLM-generated code. Vendors compete instead on cold-start speed, session persistence, pricing and developer experience. That makes the buy decision simple for most teams. Vendors' published isolation stacks appear broadly comparable to what this article has described, at a fraction of the build cost. Building it yourself only becomes worthwhile at organisations already running Kubernetes at scale, or held to compliance requirements that rule out shared infrastructure.

Chapter 8: Conclusion

One fact underlies why agent sandboxes exist at all. A large language model can now execute code it wrote itself, in response to instructions nobody can fully verify. That same capability gives the agent its value and makes it dangerous.

Nothing this article has covered removes that risk. Every layer instead narrows it, each covered in an earlier chapter:

the orchestration primitive
the isolation technique
the threat model
network and credential boundaries
fleet scaling and governance
the benchmark evidence

None of these ideas were invented specifically for AI agents. Multi-tenant cloud security supplied the isolation techniques, and a decade of browser security research supplied the browser controls.

The result is defense-in-depth, a posture rather than a solution. It treats every layer as something that will eventually fail, and it builds each one so that failure does not cascade into whatever comes next.

Chapter 9: References

Chapter 1

Chapter 2

Chapter 4

Chapter 5

Chapter 6

Chapter 7

Monitoring Mixins, Observability Libraries, and the Signal Pattern: A Practitioner's Guide

Krishan Thisera — Thu, 13 Aug 2026 07:53:44 +0000

A few months ago, I completed a significant revamp of our metrics and monitoring implementation. That work meant taking a hard look at how we package and distribute Prometheus alerts and Grafana dashboards across services, and led me down a rabbit hole most DevOps and SRE folks eventually encounter: monitoring mixins, observability libraries, and Grafana's newer signal pattern.

This article is a write-up of what I found, centred on Grafana's own jsonnet-libs repository. If you're comfortable with Prometheus and Grafana but haven't yet explored how the ecosystem handles reusable, composable monitoring configuration, this is meant as a practical starting point.

Chapter 1: The Dashboard That Wouldn't Load

You've done this before. You find a highly-rated Kubernetes dashboard on the Grafana community site, import it into your own Grafana instance, and wait for the panels to populate. Instead, half of them show "No data." The PromQL queries underneath assume a job label or selector that doesn't match how you name things. The dashboard isn't broken. It's written for someone else's Prometheus.

This kind of failure is common enough to be a recognised pattern in the observability community. A dashboard built for one environment carries hidden assumptions about label names, selectors, and job structures that rarely survive contact with a different one.

The root cause is structural, not accidental. The engineers building a piece of infrastructure software are best positioned to know which metrics matter and how to interpret them. But they are rarely the ones running that software in someone else's production environment. It's like a car built by one team and serviced by another: the people who know the engine best rarely turn the wrench.

For years, the default response was to route around this gap manually: build dashboards from scratch, or copy one from the internet and hope. Neither scales. What changed the equation was a specific piece of tooling built to close that gap directly, and that's where the story of monitoring mixins begins.

Chapter 2: How Mixins Fixed Monitoring Distribution

A monitoring mixin is a Jsonnet program. When you evaluate it with your own configuration values, it produces three things at once: a Grafana dashboard, a set of Prometheus alerting rules, and a set of Prometheus recording rules, all built from the same label and selector logic. Think of it as a recipe rather than a finished meal: the mixin ships with instructions and defaults, and you supply the ingredients specific to your kitchen.

Tom Wilkie and Frederic Branczyk proposed this format in 2018, packaging the idea as a way to distribute Grafana dashboards and Prometheus rules together rather than as separate, disconnected artifacts. The premise, as Grafana Labs later described it on its own blog, was simple: system authors should not only emit metrics, they should also tell you how to use them. That shifts the burden of building good observability artifacts onto the open-source community, instead of leaving every team to solve the same problem alone.

What a mixin actually produces follows a community-maintained style guide. Alert names use camelCase with no whitespace (NodeFilesystemAlmostOutOfFiles is one example), and every alert carries a severity label: critical, warning, or info. Every alert also carries two mandatory annotations, summary and description, so the person paged at 2 a.m. knows what fired and why.

The other core capability is customisation: you inject or override configuration values, such as job selectors and thresholds, at build time, without touching the mixin's source. That's what lets an upstream author ship something useful without hard-coding your specific configurations.

But that flexibility has a ceiling. A generic dashboard stays generic no matter how many configuration knobs it exposes; deciding which panels matter to your team requires local knowledge no upstream author has. Understanding why requires understanding the language mixins are built on.

Chapter 3: The Jsonnet Engine Underneath

Every mixin is written in Jsonnet, a data-templating language developed at Google. Grafana Labs describes it as a powerful extension of JSON, one that adds variables, functions, conditionals, imports, and error handling on top of the plain JSON data model. Tom Wilkie has pointed out that this lineage isn't accidental: Kubernetes borrowed from Google's internal Borg, Prometheus borrowed from Borgmon, and Jsonnet followed the same pattern for monitoring configuration.

Before settling on Jsonnet, the mixin's authors ruled out several alternatives. YAML was judged insufficiently dynamic for injecting labels into templates. JSON, a strict subset of YAML, offered even less. String substitution was rejected because it doesn't understand the structure it's editing; a bad substitution can silently produce broken syntax. Templating engines like Jinja only allow structural changes at points the template author explicitly anticipated. And general-purpose languages such as Go or Python were set aside because, in Wilkie's words, writing full programs to produce configuration "feels too much like writing programs, and it's generally a bit of a pain."

A handful of Jsonnet features do the actual structural work. The + operator merges two objects, and when fields collide, the right-hand side wins. Fields declared with :: are hidden: they exist during evaluation but never appear in the final rendered JSON, which is how internal objects like _config stay invisible until a build step extracts them. The self keyword is a late-bound reference to the current object, letting a mixin refer to configuration you haven't supplied yet. super reaches into a base object during a merge, so you can extend rather than replace inherited behavior.

Jsonnet isn't the only answer to this problem. CUE, created by former Google engineer Marcel van Lohuizen, takes a validation-first approach instead of a templating-first one.

We'll return to that trade-off, and to the teams who've made the switch, in a later chapter. For now, these operators are what turned a single mixin into something composable, which is exactly what observability libraries build on next.

Chapter 4: From Mixins to Observability Libraries

A plain mixin (see legacy istio-mixin) has a ceiling: it's a self-contained bundle built for one piece of software. Two independent mixins generally can't merge cleanly into a single dashboard set; you end up reconciling conflicts by hand. Observability libraries, or observ-libs, exist to remove that ceiling. According to Grafana's own jsonnet-libs repository, an observ-lib is a flexible format for describing dashboards and alerts in a modular way, built specifically so libraries can import into one another, or into a mixin.

You'll recognise an observ-lib by its folder name: kafka-observ-lib, jvm-observ-lib, windows-observ-lib, snmp-observ-lib, process-observ-lib, golang-observ-lib. A shared common library applies consistent style across all of them.

The directory structure tells you what changed. A legacy plain mixin ships dashboards.libsonnet, alerts.libsonnet, and config.libsonnet. An observ-lib adds three things: a signals/ directory, a panels.libsonnet file for reusable panel definitions, and a rows.libsonnet file for reusable dashboard row groupings. It also splits its entry point in two: main.libsonnet for the full modern feature set, and mixin.libsonnet retained purely so older tooling built around the plain-mixin standard keeps working.

This is composition, not just organisation. A team building observability for a Java application can import the JVM library for garbage-collection and heap metrics, then layer application-specific dashboards on top, with no need to reimplement JVM monitoring from scratch. It's the difference between building with prefabricated modules and pouring a new foundation every time.

The payoff shows up in the dashboards themselves, which tend to follow a consistent shape across the ecosystem: an Overview section for health indicators, a Performance section for throughput and latency, a Resources section for CPU and memory, and a Details section for diagnostics.

That signals/ directory isn't decorative. It houses the ecosystem's most consequential idea yet, and it's where we're headed to see next.

Chapter 5: Signals: Define Once, Render Everywhere

Here's a failure mode every SRE has lived through: a dashboard panel and its corresponding alert quietly drift apart. Someone edits the panel's query to fix a bug. Nobody updates the alert. Months later, what you see on the dashboard during an incident no longer matches the condition that actually paged you. The signal pattern exists to make that drift structurally impossible.

A signal lets you declare a metric once and render it as a time series panel, a stat panel, a table column, or a Prometheus alert, all from that single declaration. The core abstraction lives in common-lib/common/signal, inside files like base.libsonnet, signal.libsonnet, and one file per supported type: counter.libsonnet, gauge.libsonnet, histogram.libsonnet, info.libsonnet, raw.libsonnet.

The type you choose does real work. Each signal type carries an automatic PromQL transformation, so you don't have to remember to apply it by hand:

Type	Transformation
`gauge`	Rendered as-is
`counter`	Wrapped in `rate(...[interval])`
`histogram`	Wrapped in `histogram_quantile(q, sum(rate(...[interval])) by (le, ...))`
`info`	Rendered as static values, not a time series
`raw`	No transformation, used verbatim

This table is arguably the single most consequential design choice in the whole pattern. Graphing a raw counter instead of its rate, or averaging a histogram's _bucket (i.e., request_duration_seconds_bucket, http_client_request_body_size_bytes_bucket) series without histogram_quantile, is one of the most common Prometheus mistakes there is. Tying the transform to the declared type, instead of to each place the metric gets used, removes that error class entirely.

Once declared, a signal exposes render functions: asTimeSeries(), asStat(), asGauge(), asTable() for panels, and asRuleExpression() for a Prometheus alert. A separate family of modification functions, including withTopK(), withQuantile(), withLegendFormat(), and withInterval(), lets you reshape the output without redefining the underlying query. You can declare signals in bulk, JSON-style, for simple cases, or through a lower-level imperative API when you need finer control over an individual metric.

Think of a signal as a single master recording that gets remixed for different formats: same source material, different output, guaranteed to stay in sync. Because asTimeSeries() and asRuleExpression() pull from the same underlying expression, the panel and the alert can never quietly disagree.

That guarantee comes with a caveat. The signal pattern is explicitly marked experimental by its own maintainers at the time of writing. Its render-function surface is large, with more than a dozen panel functions alone, and the interface isn't stabilised. Adopting it today means accepting some risk of backward-incompatible change in exchange for the correctness it buys you.

Understanding what a signal is only gets you so far. Next, you'll want to know how to actually run one: which tools you need installed, and what the workflow looks like from a fresh clone to a deployed dashboard.

Chapter 6: Getting Hands-On: The Toolchain and Workflow

Three tools cover most of what you need.

Jsonnet is the language runtime itself
jb (jsonnet-bundler) is the package manager that fetches mixins and libraries.
mixtool validates and scaffolds mixins against the standard schema.

Trying an existing mixin takes three commands:

git clone <mixin-repo-url>
cd <mixin-repo>/<mixin-dir>
jb install

Customising a mixin follows a different path. You don't edit the upstream source; you start a fresh project instead:

jb init
jb install github.com/kubernetes-monitoring/kubernetes-mixin

Then write a local mixin.libsonnet that imports the upstream mixin and overrides its _config:

local kubernetes = import "kubernetes-mixin/mixin.libsonnet";
kubernetes {
  _config+:: {
    kubeStateMetricsSelector: 'job="kube-state-metrics"',
  },
}

You can use the jsonnet CLI, or the Jsonnet Language Server extension for VS Code for inline evaluation, linting, and navigation.

From here, deployment branches based on your preference, for instance, through your own CI/CD, bundled with kube-prometheus for a full Kubernetes-native stack, or through Tanka.

This is the entire mixin/observ-lib/signal pattern in working form. It's also not the only way to solve this problem, and the next chapter looks squarely at the alternatives.

Chapter 7: The Other Side of the Fence: Alternatives Worth Knowing

The strongest critique of this entire ecosystem comes from inside Grafana Labs itself. Grafana's own documentation states plainly that Grafonnet, while useful for generating dashboard JSON, "requires knowing Jsonnet, so that can be unappealing to some users." That's not outside commentary. It's the tool's primary vendor naming its own barrier to adoption.

The Grafana Terraform provider is the most direct answer to that barrier. Its grafana_dashboard resource holds the dashboard model in a config_json field, populated inline with jsonencode(), loaded from a JSON file, or templated with Terraform's templatefile() function. Grafana documentation lists three more non-Jsonnet paths:

Grafana Ansible collection, for teams already standardised on Ansible playbooks.
Grafana Operator, a Kubernetes-native controller that fits naturally into ArgoCD-style GitOps.
Grafana Crossplane provider (experimental), exposing the same resource surface as native Kubernetes manifests.

Rule management has a parallel alternative, distinct from dashboards. A large share of the Kubernetes-native monitoring community manages Prometheus alerting and recording rules through the Prometheus Operator's PrometheusRule custom resource: native Kubernetes manifests reconciled by a controller, rather than YAML generated ahead of time from Jsonnet. The two approaches are often complementary rather than competing. Many teams use a mixin, or kube-prometheus (which bundles several mixins together), specifically to generate the PrometheusRule manifests that the Operator then deploys: Jsonnet handles authoring, and Kubernetes-native reconciliation handles runtime.

Terraform and the Operator both change how configuration gets deployed. CUE makes a different argument entirely. CUE is validation-first where Jsonnet is templating-first: its merge operation is associative, commutative, and idempotent, a guarantee that the same configuration fragments always produce the same result regardless of combination order. Jsonnet's simpler right-hand-side-wins merge doesn't make that promise. Mercari's observability platform team adopted CUE for parts of their workflow specifically to cut down on the number of configuration languages engineers needed to know.

None of this makes Jsonnet's approach wrong. It has the deepest community track record here: the mixin idea dates to 2018 and has years of production use. Nothing else discussed offers an equivalent to the signal pattern's single-definition, multi-render guarantee. The honest answer is contextual: teams mainly consuming community mixins get the most value staying in Jsonnet, since that's where the maintained content lives.

That answer holds for the tools available right now. Where Grafana Labs is taking the platform next is a separate question, and it's the one Chapter 8 takes on.

Chapter 8: Where It's Heading, And Why It's Worth Exploring

Grafana Labs' own roadmap tells you where this is going. Grafana 12 introduced the Foundation SDK, entering public preview with libraries for defining Grafana resources programmatically in Go, Java, PHP, Python, and TypeScript: mainstream languages, not Jsonnet. Alongside it came the App Platform APIs and Dashboard schema v2, both experimental, and grafanactl for validating and pushing resources. By 2026, the Grafana Terraform provider had already split its documentation between a legacy HTTP API for Grafana 12 and earlier, and a newer Kubernetes-style API for Grafana 13 and beyond. The resource model itself is mid-migration.

Don't read this as Jsonnet losing. The Foundation SDK solves a different problem than mixins do. It answers how do I define a Grafana resource in a language my team already knows. Mixins and observ-libs answer how do I package and distribute best-practice monitoring so other organisations can use and improve it. Grafonnet sits at the bridge between the two: it's a Jsonnet library, but it's generated straight from the Foundation SDK's OpenAPI specs. These are complementary tracks, not a fork in the road.

What stays constant is the idea, not the tooling: define your signal once, render it everywhere, and let configuration, not a fork, absorb the differences between environments. That's the part worth carrying forward regardless of which SDK wins.

If you're running Prometheus and Grafana already, you have a fast way to test this yourself. Clone the upstream mixin, run jb install, and generate its dashboards against your own cluster. You'll see the pattern this whole article has been describing, in about ten minutes.

References

CUE Configuration Use Case - cuelang.org
Everything You Need to Know About Monitoring Mixins - Grafana Blog
Getting Started with Monitoring Mixins - povilasv.me
Grafana as Code: A Complete Guide to Tools, Tips, and Tricks - Grafana Blog
Grafana Labs' Jsonnet Libraries - GitHub
Grafana Terraform Provider - grafana_dashboard Resource - Terraform Registry
grafana/jsonnet-libs Overview - DeepWiki
Grafonnet - GitHub Pages
How to Provision Dashboards as Code in Grafana - OneUptime
Jsonnet Tutorial - jsonnet.org
Monitoring Mixins Documentation - GitHub
Observability as Code in Grafana 12 - Grafana Blog
Observability-kit: Adventures of Using CUE at Scale - Mercari Engineering
Provision Grafana Cloud with Infrastructure as Code - Grafana Documentation

How We Solved ArgoCD Notifications' Single-Secret Limitation with Kyverno

Krishan Thisera — Sat, 04 Jul 2026 06:50:00 +0000

Note: I wrote this post a while back, before Kyverno introduced its new Mutating Policies (released in Kyverno 1.15). The Kyverno manifests here use the older approach, but the core concept remains the same.

If you run ArgoCD Notifications with multiple integrations (for example, GitLab + Grafana), you will eventually hit the single-secret constraint. This post walks through the options I evaluated, why most of them were not so elegant or risky, and the Kyverno policy pattern that worked cleanly in production.

TL;DR

ArgoCD Notifications is hardcoded to read from a single secret. When you need credentials for multiple services — each managed independently — that becomes a problem fast.

The constraint: ArgoCD Notifications only reads from argocd-notifications-secret
The complication: Two Vault Secret Operator (VSO) resources (static or dynamic) cannot safely co-manage the same Kubernetes Secret object — they’ll fight and flap
The fix: Use a Kyverno mutate policy to merge specific keys from individual source secrets into one target secret

A Bit of Context

We run a GitOps workflow using ArgoCD. App code lives in one repo, rendered manifests are committed to a deploy repo by GitLab CI, and ArgoCD reconciles those manifests across clusters.

After each deployment, ArgoCD Notifications sends a webhook back to the GitLab pipeline with the sync status. This webhook setup tells us whether the deployment actually landed successfully.

For secret management, we use Vault Secret Operator (VSO). It talks to HashiCorp Vault and provisions secrets directly into the cluster.

Below is a very high-level overview of VSO for the context. The internals of VSO are outside the scope of this post.

The Problem

This setup is very common in the industry, and ArgoCD Notifications is a great way to send deployment updates to external systems. It supports Slack, Microsoft Teams, Grafana, custom webhooks (which is how GitLab is integrated in this case), etc.

In our case, we needed two notifications firing after every deployment: a GitLab webhook and a Grafana annotation on our dashboards.

See the Docs: ArgoCD Grafana notification docs for details.

Grafana notification support is already built into ArgoCD — it just needs a Grafana API token and a few configuration changes on the ArgoCD application side (See ArgoCD Grafana notification docs).

However, there is one catch: all service credentials for the ArgoCD notification controller must live in a single Kubernetes Secret named argocd-notifications-secret. That is the only place the ArgoCD Notifications controller will look.

If you are to notify multiple services, all credentials need to be consolidated into that one secret.

At this point, the GitLab token was already being managed by VSO. Now we needed the Grafana token to live alongside it.

So it would look like this,

At least at the time of writing, there was no way to point ArgoCD Notifications to multiple secrets (for example, one for GitLab and one for Grafana), or to another secret name.

So it was one secret, both the tokens had to end up in the same secret, but still be managed separately.

Co-locating unrelated credentials in one place raises both operational and security concerns, worth avoiding.

At that point, a few options were left.

Options Considered

Option 1 — A separate secret for the Grafana token

We already established this that (at the time of writing) this was not possible. ArgoCD Notifications was hardcoded to use argocd-notifications-secret.

Option 2a — Combine both tokens into one VSO secret

Technically possible. Store both tokens in one Vault path (a single location in Vault that contains both keys), then let VSO create one Kubernetes Secret from it.

But that means co-locating unrelated credentials in the same Vault path — not ideal from a security standpoint, and it creates an operational headache too. Especially once you factor in dynamic secrets, where each service should own and rotate its own credentials independently.

Option 2b — Use two VSO resources to manage the same secret

This sounds cleaner: one VSO resource per token, both targeting argocd-notifications-secret.

The issue is that VSO manages the entire Secret resource, not individual fields. If two VSO resources claim the same Secret, they continuously reconcile against each other and overwrite each other’s updates. The result is a flapping Secret that does not stabilise.

Here is a sample secret managed by VSO. Check managedFields to see ownership.

Option 3 — Use an Istio EnvoyFilter to inject the token

Since we use Istio service mesh, an EnvoyFilter could intercept requests to Grafana and inject the API token as an HTTP header, without ArgoCD knowing about it.

A clever workaround, but it comes with trade-offs:

Wrong abstraction layer: token management becomes network behaviour.
Coupling risk: upgrades in Istio/Envoy can break filter behaviour.
Testing complexity: harder to validate compared to policy/controller logic.
Scope risk: Host-based matches can affect unintended traffic after routing changes.

Option 4 — Write a custom controller or use a templating controller

We already had internal controller tooling, so this was possible. We also considered kluctl as a templating-controller path.

Both are valid, but they add operational overhead.

Option 5 — Use Kyverno

Kyverno was already on our DevOps roadmap, just not yet adopted.

Kyverno is a Kubernetes-native policy engine that can mutate resources. Instead of writing a custom controller, we could use policy-driven mutation to handle secret composition.

So, I decided to go ahead with Kyverno.

The Solution

The plan was straightforward:

VSO creates two source secrets (one per token):
- argocd-notifications-secret-gitlab-merge
- argocd-notifications-secret-grafana-merge
A Kyverno mutate policy watches those source secrets and patches argocd-notifications-secret.

One important detail:

Kyverno cannot watch a Secret and generate another Secret from it — it cannot generate a resource of the same kind as the trigger. What it can do is mutate a resource of the same kind.

So the approach is: use a mutate rule that is triggered by the source secrets and patches argocd-notifications-secret directly. So this means argocd-notifications-secret needs to be provisioned first.

The easiest way is to use the ArgoCD chart’s built-in secret provisioning to create an empty placeholder Secret (resource shell only). Kyverno then updates its data fields when source secrets change.

As we do not want Kyverno mutating a resource that does not exist yet, we can use ArgoCD sync-waves to ensure the empty argocd-notifications-secret is created first (by setting its sync-wave to a value lower than the VSO source secrets).

This is the Kyverno policy that ties it all together. It watches both source secrets (Kubernetes) and patches their keys into argocd-notifications-secret whenever they change:

apiVersion: kyverno.io/v1
kind: Policy
metadata:
  annotations:
    policies.kyverno.io/subject: Secrets
    policies.kyverno.io/title: argocd-notifications-secret
  name: argocd-notifications-secret
  namespace: argocd
spec:
  rules:
  - name: mutate-secret-on-gitlab-secret-update
    match:
      any:
      - resources:
          kinds:
          - Secret
          names:
          - argocd-notifications-secret-gitlab-merge
          operations:
          - CREATE
          - UPDATE
    mutate:
      mutateExistingOnPolicyUpdate: true
      targets:
      - apiVersion: v1
        kind: Secret
        name: argocd-notifications-secret
      patchStrategicMerge:
        data:
          gitlab-token: '{{ request.object.data."gitlab-token" }}'
  - name: mutate-secret-on-grafana-secret-update
    match:
      any:
      - resources:
          kinds:
          - Secret
          names:
          - argocd-notifications-secret-grafana-merge
          operations:
          - CREATE
          - UPDATE
    mutate:
      mutateExistingOnPolicyUpdate: true
      targets:
      - apiVersion: v1
        kind: Secret
        name: argocd-notifications-secret
      patchStrategicMerge:
        data:
          grafana-api-key: '{{ request.object.data."grafana-api-key" }}'

A few useful notes:

Two rules, one policy: each rule updates a specific field from a specific source secret.
mutateExistingOnPolicyUpdate: true: reconciles immediately on policy apply/update.
patchStrategicMerge: merges specific keys instead of replacing the whole data map (Kyverno strategic merge patch docs).
match: Kyverno watches for CREATE and UPDATE events on the source secrets (argocd-notifications-secret-gitlab-merge and argocd-notifications-secret-grafana-merge). Any change to either of those triggers the corresponding rule.
targets: this tells Kyverno which resource to patch — in this case argocd-notifications-secret, the empty shell provisioned by the ArgoCD Helm chart.

And that is really it. VSO manages each source secret independently, Kyverno watches for changes and patches the right keys into argocd-notifications-secret, and ArgoCD Notifications picks it up without knowing any of this is happening.

How Kyverno Tracks Ownership — Managed Fields

One thing worth knowing is how Kubernetes handles field ownership here. Because multiple controllers are touching the same Secret, Kubernetes managed fields track which controller owns which keys — so nothing gets blindly overwritten.

In this setup:

ArgoCD chart owns the shell (empty secret resource)
Kyverno background controller owns the token fields that it writes

You can verify this directly:

kubectl get secrets argocd-notifications-secret \
  --show-managed-fields -n argocd -o yaml

The output includes a managedFields section showing which manager owns which keys.

Things to Keep in Mind When Running Kyverno in Production

These are some of the points that I make a note of:

Install all CRDs. Some Kyverno controllers can error if required CRDs are missing, even when those CRDs are not actively used. I observed this in the version we ran at that time, so check your version and release notes.
Give Kyverno enough resources. If the admission controller capacity is insufficient, resource creation can fail cluster-wide.
Test policies early. Two solid options: – Kyverno CLI tests (fast, no cluster required): docs – Chainsaw tests (integration style, with real cluster or mock API server): repo

My preference is Chainsaw for realism, while CLI tests are great for quick validation.

Wrapping Up

What started as a “just add one more token” request ended up being a neat Kyverno adoption point for us. Sometimes a constraint in one tool opens the door to doing something smarter in another.

If you are running ArgoCD with multiple notification integrations and have hit this exact wall, hopefully, this gives you a concrete path forward.

And if you have solved it differently — especially with the newer Kyverno mutation features — I would genuinely love to hear how you approached it.

GitOps for Devs - Part 03: CI/CD

Krishan Thisera — Tue, 09 Jan 2024 04:24:12 +0000

In the previous article, we discussed the codebase of the Album-App. This article will discuss how we can implement CI/CD for the Album-App.

CI/CD Workflow

There are many ways to implement CI/CD workflow. But in summary,

Testing and Readiness Check: Once the changes to the application code are thoroughly tested and verified for release, the process can proceed to the release phase.
Triggering the Release Pipeline: This phase can be initiated manually or through an automated process, depending on your workflow and requirements.
Building and Pushing Docker Images: The release process involves building the application code and pushing the resulting images to the designated Docker registry or container repository.
Updating Helm Chart Values: Post-image creation, the Helm chart values in the configuration repo need an update with the new Docker image tag.
ArgoCD Trigger and Sync Process: ArgoCD, being a continuous deployment tool, detects changes within the configuration repository. ArgoCD compares the desired state (defined in the Git repository) with the actual state of the cluster. Any disparities trigger the synchronization process to ensure alignment between the desired and actual state.
Sync and Deployment: Upon detecting changes, ArgoCD starts the sync process, pulling the updated configurations from the Git repository. ArgoCD applies these changes to the Kubernetes cluster, managing deployments, updates, or rollbacks as necessary to align the cluster with the desired state defined in the Helm charts.

The pipeline

The album-app 's release pipeline leverages GitHub actions. The pipeline is implemented in the application repository.

name: Build and Publish Docker Images

on:
  release:
    types:
      - created

env:
  REGISTRY: ghcr.io
  CONFIG_REPO: ${{ github.repository }}-config

jobs:
  build-and-publish:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Check tag name
        id: check_tag
        run: echo "::set-output name=tag_name::${GITHUB_REF#refs/tags/}"

      - name: Check app name
        id: check_app
        run: |
          app_name=$(echo ${{ steps.check_tag.outputs.tag_name }} | awk -F'-' '{print $1}')
          echo "::set-output name=app_name::$app_name"

      - name: Check Docker Image Version
        id: check_version
        run: |
          version=$(echo ${{ steps.check_tag.outputs.tag_name }} | awk -F'-' '{print $2}')
          echo "::set-output name=version::$version"

      - name: Login to GitHub Packages
        uses: docker/login-action@v1
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GHCR_TOKEN }}

      - name: Build and push frontend or backend image
        id: docker_build
        uses: docker/build-push-action@v2
        with:
          context: ${{ steps.check_app.outputs.app_name }}
          push: true
          tags: ghcr.io/${{ github.repository }}-${{ steps.check_app.outputs.app_name }}:${{ steps.check_version.outputs.version }}
          build-args: |
            VERSION=${{ steps.check_version.outputs.version }}
    outputs:
      version: ${{ steps.check_version.outputs.version }}
      app_name: ${{ steps.check_app.outputs.app_name }}

  deployment:
    needs: build-and-publish
    runs-on: ubuntu-latest
    permissions:
      contents: write
      packages: read
    # Checkout album-app-config repository
    steps:
      - name: Checkout album-app-config
        uses: actions/checkout@v2
        with:
          repository: ${{ env.CONFIG_REPO }} # Replace with the URL or name of the album-app-config repository
          ref: main
          token: ${{ secrets.CONFIG_REPO_TOKEN }}

        # Step to update album-app-config repository
      - name: Update album-app-config repository
        env:
          GH_TOKEN: ${{ secrets.CONFIG_REPO_TOKEN }}
        run: |
          # Get the release tag and app name
          version='${{ needs.build-and-publish.outputs.version }}'
          app_name='${{ needs.build-and-publish.outputs.app_name }}'

          # Set the release branch name
          release_branch=release/${app_name}_${version}

          # Modify the Helm value file with the new tag
          sed -i.bak "s/tag: \".*\"/tag: \"${version}\"/" ${app_name}/values.yaml

          # Create a new branch
          git config --global user.email ${{ github.actor }}@github.com
          git config --global user.name ${{ github.actor }}
          git checkout -b ${release_branch}

          # Commit changes
          git add ${app_name}/values.yaml
          git commit -m "release: ${app_name} ${version}"

          # Push the changes to the remote repository
          git push origin ${release_branch}

          # Body Content
          pr_body="${{ github.repository }} ${app_name} release ${version}. Please note that this is an automated PR."

          # Title Content
          pr_title="release: ${app_name} ${version}"

          # Open a pull request
          gh pr create --title "${pr_title}" --body "${pr_body}" --base main --head ${release_branch} --repo ${{ env.CONFIG_REPO }}

Triggers

Event: Triggered when a new release is created.

Environment Variables

Registry: The container registry used (GitHub Container Registry - ghcr.io).
Config Repository: Repository used for configuration.

Jobs

build-and-publish
- Steps:
  - Checkout: Fetches the repository code.
  - Check tag name: Extracts the tag name from the release.
  - Check app name: Retrieves the application name from the tag.
  - Check Docker Image Version: Determines the version of the Docker image.
  - Login to GitHub Packages: Authenticates to the GitHub Container Registry.
  - Build and push image: Uses Docker Build-Push Action to build and push the Docker image to the registry. It uses the extracted application name and version from earlier steps to tag the image appropriately.
deployment
- Dependencies: Depends on the completion of the 'build-and-publish' job.
- Steps:
  - Checkout album-app-config: Fetches the configuration repository (CONFIG_REPO) code.
  - Update album-app-config repository:
    - Retrieves the version and app name from the 'build-and-publish' job's outputs.
    - Creates a new release branch with the format release/${app_name}_${version}.
    - Modifies a Helm value file in the config repository with the new tag.
    - Commits the changes and pushes them to the remote repository.
    - Creates a pull request with automated content.

Once the pull request has been merged, ArgoCD will detect the changes/disparities and start the sync process.

Sync Policies

You can use ArgoCD sync policies to control the sync behaviour.

Note: By default, changes that are made to the live cluster will not trigger automated sync.

To enable self-healing, we might need to include below:

spec:
  syncPolicy:
    automated:
      selfHeal: true

By the time you read this article, the selfHeal option may or may not be included. See here

See the docs

Sync Options

Sync options specify how the synchronization should occur, providing additional configurations and parameters for the synchronization process itself.

When using auto-sync in Argo CD, it currently applies all objects in an application, causing delays and strain on the API server, but enabling selective sync will only sync resources that are out-of-sync, reducing time and server load for applications with many objects.

spec:
  syncPolicy:
    syncOptions:
    - ApplyOutOfSyncOnly=true

See the docs

Conclusion

Across these articles, we've gone from the basics to some advanced concepts in GitOps using ArgoCD. We started by setting up ArgoCD and deploying a sample app, giving a solid foundation. Then, we dive deep into the code intricacies. Lastly, we explored CI/CD implementation, an essential part of any developer's toolkit.

Hopefully, these pieces have been a practical guide, making GitOps more accessible and showing its potential to streamline development workflows. As we close this series, keep experimenting and leveraging GitOps—it's a game-changer for modern development.

GitOps for Devs - Part 2: Navigating the Code

Krishan Thisera — Sun, 31 Dec 2023 04:39:11 +0000

We're continuing our exploration of GitOps with ArgoCD. In the first part, we got a grasp of GitOps basics and set up an example app called Album-App. Now, let's explore the repositories that drive this application:

The Two Repositories

Application Code - This contains the actual code for both the Frontend and Backend applications.
- Backend: A simple REST API written in Go, accessible on port 8080.
- Frontend: A React application that communicates with the backend.
Config Repo - Hosts Helm charts defining the application's infrastructure.

Application Repository

The application repository contains the code for both Frontend and Backend applications.

Backend

The backend application itself is a simple REST API written in Go.

The application exposes port 8080.

You can test the application on your docker environment by,

docker run -p 8080:8080 ghcr.io/krishanthisera/album-app-backend

Once the container is up click here to see API documentation.

Frontend

The Frontend is a React application that talks to the backend.

To test the application locally,

docker run -p 3000:3000 ghcr.io/krishanthisera/album-app-frontend

Please note that the backend app should be running on port 8080.

Configuration Repository

The Config Repo contains Helm charts that define the application infrastructure.

Here we have four helm charts ⎈,

root-app: Acts as ArgoCD's entry point, managing and deploying the whole application stack.
apps: Contains ArgoCD applications.
frontend and backend: Helm charts for respective Kubernetes deployments.

Understanding Root App

As the name suggests, the root app is the entry point for ArgoCD to manage and deploy the entire application stack.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root-app
  namespace: argocd
  finalizers:
  - resources-finalizer.argocd.argoproj.io
spec:
  destination:
    namespace: album-app
    name: in-cluster
  project: default
  source:
    path: apps
    repoURL: https://github.com/krishanthisera/album-app-config
    targetRevision: HEAD

Here's a snippet:

Destination: Specifies where the application will be deployed (namespace and cluster).

Project: Defines the ArgoCD project the application belongs to. Projects offer a way to group and organize applications based on shared policies, permissions, and settings.

# Default ArgoCD Project
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: default  # Name of the project
  namespace: argocd  # Namespace where the project resides
spec:
  clusterResourceWhitelist:  # Whitelisted cluster resources
  - group: '*'  # All resource groups allowed
    kind: '*'   # All resource kinds allowed
  destinations:  # Deployment destinations
  - namespace: '*'  # Deploy to all namespaces
    server: '*'  # Deploy to all servers
  sourceRepos:  # Allowed source repositories
  - '*'  # Allow all repositories

Source: Indicates the source of the application's configuration (Git repository URL, path, and target revision).

Here the root-app points to the /apps directory in the config repo.

Apps Helm Chart

As mentioned in the previous article, here we follow the ArgoCD App-of-Apps pattern.

Above are ArgoCD application resources residing in the argocd namespace. They define their respective apps' deployment targets, projects, and source repositories.

# Backend Application
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: album-app-backend  # Name of the ArgoCD application
  namespace: argocd  # Namespace where the ArgoCD application resides
  finalizers:
    - resources-finalizer.argocd.argoproj.io  # Action to be taken when deleting the resource
spec:
  destination:
    namespace: album-app  # Deployment target namespace for the backend application
    server: https://kubernetes.default.svc  # Kubernetes server for deployment
  project: album-app  # Project to which the backend application belongs
  source:
    path: backend  # Directory containing backend code/configuration
    repoURL: https://github.com/krishanthisera/album-app-config  # Git repository URL
    targetRevision: HEAD  # Target revision of the repository

---
# Frontend Application
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: album-app-frontend  # Name of the ArgoCD application
  namespace: argocd  # Namespace where the ArgoCD application resides
  finalizers:
    - resources-finalizer.argocd.argoproj.io  # Action to be taken when deleting the resource
spec:
  destination:
    namespace: album-app  # Deployment target namespace for the frontend application
    server: https://kubernetes.default.svc  # Kubernetes server for deployment
  project: album-app  # Project to which the frontend application belongs
  source:
    path: frontend  # Directory containing frontend code/configuration
    repoURL: https://github.com/krishanthisera/album-app-config  # Git repository URL
    targetRevision: HEAD  # Target revision of the repository

If you follow the repo, the helm template creates the namespace and the ArgoCD project.

Note: If you want to ensure that child apps and all of their resources are deleted when the parent app is deleted, it's essential to add the appropriate finalizer to your Application definition. This finalizer action ensures proper cleanup and deletion of all related resources when the parent application is removed. See here

I have included all the ArgoCD applications this repo intended to deploy.

The frontend and backend ArgoCD Application resources point to their respective directory in the repository.

Frontend and Backend Helm Charts

These Helm charts define how the frontend and backend applications will be deployed on Kubernetes. They contain configuration details encapsulated within values.yaml. files.

replicaCount: 1
namespace: album-app
image:
  repository: ghcr.io/krishanthisera/album-app-backend
  pullPolicy: IfNotPresent
  tag: "v1.3.2" # Important: This tag signifies the version of the backend application.
  containerPort: 8080

service: 
  name: album-app-backend
  port: 8080

The value files are pretty straightforward. The only thing to note is the tag value. It is the image tag of the backend application.

During the release process of the application, the tag value will be replaced with the respective image tag.

Part 3 of the article series will cover the release process including CI/CD pipelines.

Stay tuned!

GitOps for Devs - Part 01: Installation

Krishan Thisera — Thu, 21 Dec 2023 12:09:27 +0000

GitOps is a hot topic in DevOps circles these days, and this article series aims to break down its core concepts starting from the basics. We'll kick things off by setting up ArgoCD a GitOps operator and deploying a sample application in this initial article. The following parts will dive deeper into configuration specifics.

GitOps 101

GitOps automates modern cloud infrastructure setup by treating configuration files like code—similar to application source code. It ensures consistent and replicable infrastructure deployments, aligning with DevOps practices for quick code deployment, efficient cloud resource management, and adaptation to rapid development cycles.

In the context of Kubernetes and microservices, GitOps serves as a model for managing infrastructure and application deployment.

Declarative Infrastructure Management: GitOps takes a declarative approach, defining the desired state of the Kubernetes cluster and its resources in configuration files (often YAML) stored in a Git repository.
Automated Synchronization: Changes to these configuration files trigger an automated synchronization process in GitOps, ensuring the Kubernetes cluster's actual state matches the defined state in the repository.
Version Control Using Git: Leveraging Git for version control allows teams to track changes, revert as needed, and collaborate on configurations.

Now that we've covered the basics of GitOps, let's jump into the example. We'll deploy a simple application on Kubernetes using minikube to illustrate these concepts.

CI/CD Workflow

There are primarily two workflows: pull-based and push-based.

Push-based GitOps Workflow

In a push-based GitOps workflow, the deployment and synchronization are initiated externally by a CI/CD system or an operator.

Application Repo: Contains the application source code, which is intended to package and later deploy.

Config Repo: The desired state of the infrastructure and applications is defined in this Git repository, often through declarative configuration files like YAML.

GitOps Operator: An external system, like a CI/CD tool, monitors the codebase and, upon changes, triggers the GitOps workflow.

Developers make changes to the Application Repo, either by creating pull requests or directly committing alterations to the codebase. As a result, a CI/CD pipeline is triggered, initiating the creation and pushing of new container images to the container registry.

Subsequently, these newly generated images or their tags are updated in the Config Repo. This step could be executed through an automated commit or by means of a pull request intended for manual merging.

The GitOps operator, an external system like a CI/CD tool, closely monitors the codebase for any changes. Once alterations to the Config Repo are detected, the GitOps operator pulls the most recent configuration from the repository. It then proceeds to deploy or update the infrastructure and applications based on this updated configuration. This process is continuous, as the operator consistently monitors the Config Repo for any further changes, ready to repeat the deployment cycle when new updates are detected.

Pull-based GitOps Workflow

In a pull-based GitOps workflow, the deployment and synchronization of the infrastructure and applications are triggered by changes in the Git repository.

Similar to the push-based workflow, there are two primary repositories: the Application Repo and the Config Repo.

GitOps Operator: The GitOps operator or tool continuously monitors the Git repository for changes. This operator usually sits closely with infrastructure, in this case, deployed in a Kubernetes cluster.

Developers, again, make changes to the Application Repo, either by creating pull requests or directly committing changes to the codebase. This triggers a CI/CD pipeline, resulting in the creation and pushing of new container images to the container registry.

Following this, the updated image or its tag is modified in the Config Repo, either through an automated commit or a pull request for manual merging.

However, the GitOps operator's role is distinct in this workflow. Instead of external initiation, the GitOps operator constantly monitors the Config Repo for any changes. As soon as changes are detected, the operator pulls the latest configuration from the repository and proceeds to deploy or update the infrastructure and applications based on this revised configuration.

Similar to the push-based workflow, this process remains continuous, with the GitOps operator continuously monitoring the Config Repo for any subsequent changes, ready to redeploy or update as needed.

Install minikube

If you haven't installed Minikube already you can follow the official documentation here

Or you can use below shell script to upgrade/install minikube if you are a Linux user.

#! /bin/sh

# Minikube update script file
# Ref: https://stackoverflow.com/questions/57821066/how-to-update-minikube-latest-version

minikube delete && \ 
sudo rm -rf /usr/local/bin/minikube && \ 
sudo curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 && \ 
sudo chmod +x minikube && \ 
sudo cp minikube /usr/local/bin/ && \ 
sudo rm minikube && \  
minikube start &&\

# Enabling addons: ingress, dashboard
minikube addons enable ingress && \
minikube addons enable dashboard && \
minikube addons enable metrics-server && \
# Showing enabled addons
echo '\n\n\033[4;33m Enabled Addons \033[0m' && \
minikube addons list | grep STATUS && minikube addons list | grep enabled && \

# Showing the current status of Minikube
echo '\n\n\033[4;33m Current status of Minikube \033[0m' && minikube status

To install kubectl and helm please follow the respective official documentation.

kubectl: https://kubernetes.io/docs/tasks/tools/#kubectl
helm: https://helm.sh/docs/intro/install/

Install Argo CD

Now if your minikube cluster is up and running you are ready to install the ArgoCD. GitOps operator in your cluster.

ArgoCD installation is straightforward.

Create the namespace for the ArgoCD operator:

kubectl create namespace argocd

Install the Operator and CRDs

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

You should expect to find similar resources within the argocd namespace.

NAME                                                    READY   STATUS    RESTARTS       AGE
pod/argocd-application-controller-0                     1/1     Running   0              137m
pod/argocd-applicationset-controller-6b67b96c9f-kcbxl   1/1     Running   0              137m
pod/argocd-dex-server-c9d4d46b5-phltz                   1/1     Running   2 (136m ago)   137m
pod/argocd-notifications-controller-6975bff68d-c5cdl    1/1     Running   0              137m
pod/argocd-redis-7d8d46cc7f-gjk25                       1/1     Running   0              137m
pod/argocd-repo-server-59f5479b7-xznv6                  1/1     Running   0              137m
pod/argocd-server-7d7fdcb49-st9vj                       1/1     Running   0              137m

NAME                                              TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
service/argocd-applicationset-controller          ClusterIP      10.102.75.218    <none>        7000/TCP,8080/TCP            137m
service/argocd-dex-server                         ClusterIP      10.98.223.112    <none>        5556/TCP,5557/TCP,5558/TCP   137m
service/argocd-metrics                            ClusterIP      10.107.209.210   <none>        8082/TCP                     137m
service/argocd-notifications-controller-metrics   ClusterIP      10.108.175.166   <none>        9001/TCP                     137m
service/argocd-redis                              ClusterIP      10.107.100.213   <none>        6379/TCP                     137m
service/argocd-repo-server                        ClusterIP      10.96.189.5      <none>        8081/TCP,8084/TCP            137m
service/argocd-server                             LoadBalancer   10.100.189.178   <pending>     80:31040/TCP,443:30770/TCP   137m
service/argocd-server-metrics                     ClusterIP      10.109.158.60    <none>        8083/TCP                     137m

NAME                                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/argocd-applicationset-controller   1/1     1            1           137m
deployment.apps/argocd-dex-server                  1/1     1            1           137m
deployment.apps/argocd-notifications-controller    1/1     1            1           137m
deployment.apps/argocd-redis                       1/1     1            1           137m
deployment.apps/argocd-repo-server                 1/1     1            1           137m
deployment.apps/argocd-server                      1/1     1            1           137m

NAME                                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/argocd-applicationset-controller-6b67b96c9f   1         1         1       137m
replicaset.apps/argocd-dex-server-c9d4d46b5                   1         1         1       137m
replicaset.apps/argocd-notifications-controller-6975bff68d    1         1         1       137m
replicaset.apps/argocd-redis-7d8d46cc7f                       1         1         1       137m
replicaset.apps/argocd-repo-server-59f5479b7                  1         1         1       137m
replicaset.apps/argocd-server-7d7fdcb49                       1         1         1       137m

NAME                                             READY   AGE
statefulset.apps/argocd-application-controller   1/1     137m

If you focus on the pods deployed in the namespace, there are three main pods, which are noteworthy.

pod/argocd-application-controller-0

The controller continuously monitors running applications, identifies and resolves inconsistencies in their states (OutOfSync), and executes user-defined hooks for synchronization lifecycle events.
pod/argocd-repo-server-59f5479b7-xznv6

The Repo server serves as an internal service supporting the application infrastructure by maintaining a local cache of Git repositories and generating Kubernetes manifests using repository-specific details.
pod/argocd-server-7d7fdcb49-st9vj

Argo CD server aka the API server acts as a central interface for various components, managing application operations, credentials, authentication, RBAC, and Git webhook events while serving as a gRPC/REST server for Web UI, CLI, and CI/CD systems.

As a part of the installation, following CRDs should be created in your cluster.

applications.argoproj.io     
applicationsets.argoproj.io   
appprojects.argoproj.io

Terminlogies

Assuming you're familiar with core concepts in Git, Docker, Kubernetes, Continuous Delivery, and GitOps, here are some specific Argo CD terminologies:

Application: A defined group of Kubernetes resources outlined in a manifest, treated as a Custom Resource Definition (CRD).
Target State: The intended state of an application, represented by files in a Git repository.
Live State: The current operational state of that application, including deployed pods and other components.
Sync Status: Indicates whether the live state matches the intended target state described in Git—essentially, is the deployed application aligned with what's defined in the repository?
Sync: The process of transitioning an application to its target state, often done by applying changes to a Kubernetes cluster.
Sync Operation Status: Indicates the success or failure of a sync process.
Refresh: The action of comparing the latest code in Git with the live state to identify any differences.
Health: Reflects the operational health of the application—whether it's functioning correctly and able to handle requests.

See here for further information

ArgoCD UI

ArgoCD ships with a very powerful User Interface. To access the UI,

Set the port forwarding
Grab the admin password

# Port Forwarding
kubectl port-forward svc/argocd-server -n argocd 8080:443

# Intial admin password
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

Now you should be able to access ArgoCD UI using https://127.0.0.1:8080 using the user name "admin".

ArgoCD has its own CLI as well, follow this article to install the argocli.

Install the Sample App - Album App

For this tutorial, I am using two repos:

Config: https://github.com/krishanthisera/album-app-config
Application: https://github.com/krishanthisera/album-app

As the names suggest, the config repo contains the infrastructure configuration, in this case, helm charts, while the application repo contains the code for frontend and backend.

From now on application is referred to ArgoCD Application objects inside the Kubernetes cluster.

To install the Album App,

Clone the config repo (I would recommend you to fork the repo and then clone the fork)
Install the helm chart

Here we are using ArgoCD App-of-Apps pattern. The root ArgoCD application object is defined in the root-app helm chart. The root application is responsible for creating ArgoCD application objects for Frontend and Backend.

# Clone the repo
git clone https://github.com/krishanthisera/album-app-config

# Create a namespace for the application
kubectl create ns album-app

# Install the root app
helm install root-app ./root-app -n album-app

# Check the installed application
kubectl get applications.argoproj.io -n argocd
NAME       SYNC STATUS   HEALTH STATUS
root-app   OutOfSync     Healthy

Once you install the helm chart, you should be able to see root-app under the applications in ArgoCD UI.

If you look closely, the root-app is out-sync; let's sync the root-app.

For now let's keep the default settings as it is.

Once the root-app is synced, both the Frontend and Backend ArgoCD application objects are created but they are yet to be synced.

Let's do the same for the Frontend and Backend application; let's get them synced.

Once both the Frontend and Backed apps are synced your ArgoCD applications should be look like this.

All the applications should be synced now 💫

kubectl get applications.argoproj.io -n argocd        
NAME                 SYNC STATUS   HEALTH STATUS
album-app-backend    Synced        Healthy
album-app-frontend   Synced        Healthy
root-app             Synced        Healthy

If you do a kubectl get all on the album-app namespace,

NAME                                               READY   STATUS    RESTARTS   AGE
pod/album-app-backend-backend-dcbbc657-stqmc       1/1     Running   0          11h
pod/album-app-frontend-frontend-789b5bbc67-9vd5x   1/1     Running   0          11h

NAME                                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/album-app-backend             ClusterIP   10.104.144.238   <none>        8080/TCP   11h
service/album-app-frontend-frontend   ClusterIP   10.108.6.129     <none>        80/TCP     11h

NAME                                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/album-app-backend-backend     1/1     1            1           11h
deployment.apps/album-app-frontend-frontend   1/1     1            1           11h

NAME                                                     DESIRED   CURRENT   READY   AGE
replicaset.apps/album-app-backend-backend-dcbbc657       1         1         1       11h
replicaset.apps/album-app-frontend-frontend-789b5bbc67   1         1         1       11h

Further, you can port-forward the album-app Frontend service.

kubectl port-forward -n album-app services/album-app-frontend-frontend 8090:80
Forwarding from 127.0.0.1:8090 -> 3000
Forwarding from [::1]:8090 -> 3000

Now that we have installed the Album App, let's dive deep into the ArgoCD-specific configuration.

Stay tuned. To be continued...

AWS Static Hosting - Part 02: CloudFront Edge Functions

Krishan Thisera — Wed, 23 Aug 2023 10:26:55 +0000

In the previous article we discussed how we can put together AWS CloudFront, S3 bucket and other associated services using Terraform to host our static website. We now need to make our site SEO-friendly, especially if it contains dynamic content.

Before we get started, let's discuss some theory.

What does it mean: Pre-rendering a website in the context of CSR

When a web crawler visits a website, it follows the same process as a regular user's browser: it sends request to the website's server, and in response, the server sends back the necessary files, such as HTML, CSS, JavaScript, and images. The web crawler uses this information to build an index of the website's content, which allows search engines to provide relevant results to users when they conduct searches.

Now, to speed up this indexing process and provide a more efficient experience for search engines. This is when pre-rendering comes into play. Pre-rendering involves sending a pre-rendered static HTML version of the webpage to the web crawler instead of just the server-side files. The pre-rendered version already contains all the essential content and is ready for the web crawler to process without the need for further rendering.

By providing a pre-rendered HTML version of the page to web crawlers, websites can ensure that search engines can quickly and accurately index their content. This can lead to better visibility in search engine results and an overall improved SEO (Search Engine Optimization) performance. It's a technique that benefits both the website owners and the search engines, as it allows for more efficient indexing and faster access to relevant content for users conducting searches.

How are we going to implement this?

As we are clear now what is Prerendering, we shall conquer what solution we are going to use.

There are more than hand full of solutions that we can use in this case, in fact, you should be able to come up with your own solution using your favorite programming language. But here we use, prerender.io as our solution.

You can host it on your own infrastructure if you wish, you can follow the document here.

For this article, I am going to use their cloud offering, which has a free tier.

Let's see what happen under the hood

First, before we dig deeper into the solution, let's clarify a couple of basics.

Request categories

Based on our context, we can categories requests to the web site based on client,

Browser client: This is a regular human user
Crawler: Web crawlers or bots who are visiting to website
Prerender crawlers: Crawlers from Prerender services

So how can we identify them?

Browser clients and crawlers can be easily identified by looking at the user-agent header. In terms of the Prerender service, Prerender service itself sends an x-Prerender header along with the request, so we can use that header.

CloudFront event and lambada at edge integration

In summary, by linking a CloudFront distribution with a Lambda@Edge function, CloudFront gains the ability to capture and handle requests and responses at its edge locations. This integration enables the execution of Lambda functions triggered by specific CloudFront events. These events encompass different stages in the request-response cycle:

Viewer request event: This occurs when CloudFront receives a request from a viewer, meaning a user or a client attempting to access content through CloudFront.
Origin request event: Before CloudFront forwards a request to the origin, this event takes place. The origin refers to the source server that holds the actual content being requested.
"Origin response" event: CloudFront triggers this event when it receives a response from the origin server. The response contains the requested content.
Viewer response event: This event happens just before CloudFront sends the response back to the viewer, ensuring any required modifications or customizations can be applied.

Refer this for further information.

Alright, back to the original solution discussion.

Ordinary Browser requests

Crawlers and Prerender request

Let's discuss what happens to the requests from crawlers.
Let's assume that the particular request has never been cached in Prerender.

Request from a crawler hit the CloudFront distribution
- CloudFront verify that it is a crawler, and it is not from Prerender
Then CloudFront talks to Prerender (I have a request from a crawler, please pass me the static/rendered webpage)
Then Prerender, hold the current request from CloudFront, and send a brand-new request to CloudFront
- Now, CloudFront validate that this request is from Prerender
CloudFront would serve this request from Prerender as a normal user request
Then Prerender render the content and respond to the CloudFront with static/rendered (HTML) web content
Finally, CloudFront respond to the Crawler with the static content from Prerender

Error Handling

Let's take a brief look at error handling. Suppose a request is made for a page that isn't available. Web servers typically return a "not found" page with or without 404 HTTP status code. Prerender should not cache these 404 pages, nor let search engines to index them; in such a case, we should inform the crawler that the request isn't valid.

To address this Prerender offers a very cool solution 💡. You can embed the error code into your HTML using meta tags, allowing Prerender to detect and relay it back. In other words, you can map a HTTP status code using HTML meta tags.

For more information, see here.

We will discuss this more with an example code later in this article.

Implementation

Before diving deep into the code, I'd like to touch upon the setup. In the subsequent sections, I'll reference our aws-edge-functions repository.

This repository to be used as a Terraform sub-module with the AWS Static Hosting module mention here. The primary focus of this module is to deploy our Lambda@Edge functions, facilitating Prerender integration with CloudFront.

The repository can be divided into two sections,

Terraform IaC dedicated to the edge function deployment
A monorepo for the edge functions and their development and build configuration

Terraform IaC for edge function deployment

Now, let's narrow our focus to the Terraform code, setting aside the Prerender and the theoretical aspects discussed earlier.

Our objective is to deploy a series of AWS Lambda functions. In this particular context,

first, we must build the code
then, we can package and upload it as a Lambda function (AKA deploy).

Assume all build configurations have been preset for us. All we'd need to do is run a specific build command, which will compile (transpile to be exact) the code.

The function below executes the build command each time we run the terraform apply command. Here, we've defined two null resources with some local provisioners:

To verify the presence of Node
To install dependencies and build the code.

# build.tf
resource "null_resource" "check_node_version" {

  triggers = {
    always_run = timestamp()
  }

  provisioner "local-exec" {
    command     = "npx yarn version  --non-interactive"
    working_dir = "${path.module}/${var.edge_function_path}"

    interpreter = ["bash", "-c"]

    on_failure = fail
  }
}

resource "null_resource" "build_edge_functions" {

  triggers = {
    always_run = timestamp()
  }

  provisioner "local-exec" {
    command     = "npx yarn install  --non-interactive && npx yarn build"
    working_dir = "${path.module}/${var.edge_function_path}"

    interpreter = ["bash", "-c"]

    on_failure = fail
  }

  depends_on = [null_resource.check_node_version]

}

A Quick Note: For this setup, I'm leveraging a pipeline to deploy the infrastructure. I've chosen Spacelift for this purpose. If you've been following along from the previous article, you might recall the GitHub repo associated with Article 01. Within that repo, you'll find a Dockerfile 🤔.

Why is this Dockerfile significant? Spacelift offers the capability to pair custom build environments with its runners. So, I've incorporated Node.js and npm into the runner's environment.

# https://github.com/krishanthisera/aws-static-hosting/blob/main/Dockerfile
FROM public.ecr.aws/spacelift/runner-terraform:latest

USER root

# Install node and npm
RUN apk add --update --no-cache nodejs npm

USER spacelift

Now, we need to create a role and associate it with the Lambda function. This step enables us to utilize our Lambda functions as Lambda@Edge functions:

# data.tf
data "aws_iam_policy_document" "lambda_edge_assume_role_policy" {
  statement {
    sid       = "LambdaEdgeExecution"
    effect    = "Allow"
    actions   = ["sts:AssumeRole"]
    principals {
      type        = "Service"
      identifiers = ["lambda.amazonaws.com", "edgelambda.amazonaws.com"]
    }
  }
}

# main.tf
resource "aws_iam_role" "lambda_edge_exec" {
  assume_role_policy = data.aws_iam_policy_document.lambda_edge_assume_role_policy.json
}

It's essential to specify both identifiers for Lambda@Edge functions. We will later associate this role with the function

Now, all that remains is to push the build artifacts to AWS. It's important to note that if we intend to associate these lambdas with CloudFront as Lambda@Edge functions, we must deploy them in the us-east-1 region.

We'll discuss more about the build process later. For the time being, let's assume our build, or as I prefer to term it, our "packed" artifacts, are stored in a designated location. We can utilize Terraform locals to represent them in a more comprehensible format.

We can utilize Terraform locals to display them in a more comprehensible manner.

 # main.tf
 locals {
  edge_functions = [
    {
      name = "prerender-proxy"
      path = "${var.edge_function_path}/packages/prerender-proxy/build/index.js"
      handler = "index.handler"
    },
    {
      name = "filter-function"
      path = "${var.edge_function_path}/packages/filter-function/build/index.js"
      handler = "index.handler"
    },
    {
      name = "response-handler"
      path = "${var.edge_function_path}/packages/response-handler/build/index.js"
      handler = "index.handler"
    }
  ]
}

Here, I have defined functions by their names, specify their locations (build), and indicate the handler.

We've defined three functions here. We'll dig deeper into the specifics of each function later. For now, our primary task is to package each of them into separate zip files for deployment.

Now, we just need to define the Lambda configuration. We can employ Terraform's count meta-argument to iterate over locals.

# main.tf
data "archive_file" "edge_function_archives" {
  count       = length(local.edge_functions)
  type        = "zip"
  source_file = "${path.module}/${local.edge_functions[count.index].path}"
  output_path = "${path.module}/${var.edge_function_path}/function_archives/${local.edge_functions[count.index].name}.zip"

  depends_on = [null_resource.build_edge_functions]
}

Here, for each function in local.edge_functions, an archive file will be created.

# main.tf
resource "aws_lambda_function" "edge_functions" {
  # If the file is not in the current working directory you will need to include a
  # path.module in the filename.
  count         = length(local.edge_functions)
  filename      = "${path.module}/${var.edge_function_path}/function_archives/${local.edge_functions[count.index].name}.zip"
  function_name = "${local.edge_functions[count.index].name}"
  handler       = local.edge_functions[count.index].handler
  publish       = true
  memory_size   = 128
  role          = aws_iam_role.lambda_edge_exec.arn

  source_code_hash = data.archive_file.edge_function_archives[count.index].output_base64sha256

  runtime = "nodejs16.x"

}

resource "aws_iam_role" "lambda_edge_exec" {
  assume_role_policy = data.aws_iam_policy_document.lambda_edge_assume_role_policy.json
}

While each line of code is self-explanatory, it's worth noting that we're associating the IAM role defined earlier.

Now, we need to think a couple of steps ahead, we can't solely rely on the lambda function names when associating them with CloudFront. We specifically need the ARN — more precisely, the ARN of a specific version. Since this will be a Terraform sub-module to be used in our static hosting module (as recalled from article 01 😉), accessing these ARNs programmatically is essential.

Hence, the ARNs will be output as follows:

# output.tf
output "function_arns" {
  value = {
    for function in aws_lambda_function.edge_functions :
    function.function_name => function.qualified_arn
  }
}

Static Hosting Stack

Let's discuss how we can associate our Lambda functions with CloudFront. For this segment, I'll be referencing the same repository as we see in the first article. You can find it here.

First we need to import our lambda at edge module.

# edge-functions.tf
module "edge-functions" {
   source = "github.com/krishanthisera/aws-edge-functions.git"
}

Then we can associate our Lambda functions with our CloudFront distribution by simply referencing their names. Neat, right?

resource "aws_cloudfront_distribution" "blog_distribution" {

  ...

  default_cache_behavior {
    allowed_methods  = ["GET", "HEAD"]
    cached_methods   = ["GET", "HEAD"]
    target_origin_id = "S3-${var.bucket_name}"

    lambda_function_association {
      event_type   = "origin-request"
      lambda_arn   = module.edge-functions.function_arns["prerender-proxy"]
    }

    lambda_function_association {
      event_type   = "origin-response"
      lambda_arn   = module.edge-functions.function_arns["response-handler"]
    }

    lambda_function_association {
      event_type   = "viewer-request"
      lambda_arn   = module.edge-functions.function_arns["filter-function"]
    }

    forwarded_values {
      headers      = ["x-request-prerender", "x-prerender-host"]
      query_string = false

      cookies {
        forward = "none"
      }
    }
  }

   ...
}

With this, we've primarily addressed the Terraform and infrastructure components. Next, we'll discuss the specifics of the edge functions, focusing especially on the business logic and it's implementation.

Edge functions

From this point on in our discussion, I'll be referencing the content inside the edge-functions directory of our aws-edge-functions repo.

This directory contains a monorepo for our edge functions. If you're unfamiliar with the term "monorepo", it stands for "monolithic repository". In a monorepo setup, multiple projects or components of a software application are stored within a single version control repository. So instead of managing distinct repositories for every project or component, everything is centralized.

In our setup, all our Lambda@Edge functions are located in the packages directory. These functions are written in TypeScript. When we build them, the TypeScript code for each edge function is transpiled into a single JavaScript package.

We manage dependencies using one main dependency/package management file (package.json) and have a single lock file (yarn.lock).

I've used esbuild for the build process. If you look inside each package, you'll find a file named esbuild.js. This file outlines how the application is built. Additionally, the package.json for each package contains the specified build script.

import { build } from "esbuild"
import fg from "fast-glob"


const define = {}

// For optional ENVs
for (const k in process.env) {
  define[`process.env.${k}`] = JSON.stringify(process.env[k])
}

export const buildNode = async ({ ...args }) => {
  await build({
    entryPoints: await fg("src/*.ts"),
    platform: "node",
    target: "node16",
    format: "cjs",
    outdir: "./build",
    sourcemap: false,
    logLevel: "info",
    bundle: true,
    define,
    ...args,
  })
}

await buildNode({})

I've also used turbo-repo to aid in the build process.

There are many tools out there to assist with this, but our main focus isn't to discuss the details of setting up a monorepo or how esbuild operates. We simply aim to transpile our TypeScript code to JavaScript so it can be used as a Lambda@Edge function.

Prerendering: The Business logic

Let's dive into the most crucial part of our discussion.

Imagine the scenario: our website has dynamic content, but search engines prefer static HTML for indexing. Our goal is to serve static HTML, rendered from our dynamic site, to these search engines.

To visualize our workflow, refer to the previously mentioned diagram:

The process kicks off when our CloudFront Distribution receives a request. We need to distinguish whether this request is from a search engine crawler or a regular user.

Step 1: Filtering Requests with Lambda@Edge | Filter Function

We'll employ a Lambda@Edge function to introduce a header, allowing us to later distinguish and appropriately handle requests during the CloudFront request flow.

Typescript handler implementation for this is not that complex 😄

// edge-functions/packages/filter-function/src/index.ts
export const handler = async (event: CloudFrontRequestEvent): Promise<CloudFrontRequest> => {
  const request = event.Records[0].cf.request

  // Check if the request:
  // 1. Does not match any of the recognized file extensions
  // 2. Is from a recognized bot user agent
  // 3. Does not already have an x-prerender header
  if (
    !IS_FILE.test(request.uri) &&
    IS_BOT.test(request.headers["user-agent"][0].value) &&
    !request.headers["x-prerender"]
  ) {
    // Set x-request-prerender header to inform origin-request Lambda function
    request.headers["x-request-prerender"] = [
      {
        key: "x-request-prerender",
        value: "true",
      },
    ]

    // Set x-prerender-host header to the host of the request
    request.headers["x-prerender-host"] = [
      {
        key: "X-Prerender-Host",
        value: request.headers.host[0].value,
      },
    ]
  }

  return request
}

Step 2: Requesting the Prerender Service | Prerender Proxy

Having filtered requests coming from crawlers, our next step is to ask the Prerender service to render the appropriate web page for us. This is a straightforward process: provide the webpage address and the Prerender-token from the prerender.io.

export const handler = async (event: CloudFrontRequestEvent): Promise<CloudFrontResponse | CloudFrontRequest> => {
  const request = event.Records[0].cf.request

  // If the request has the x-request-prerender header, it means the viewer-request function determined it should be prerendered
  if (request.headers["x-request-prerender"]) {
    // CloudFront alters requests for the root path to the default root object, /index.html.
    // However, when prerendering the homepage, this behavior is not desired.
    if (request.uri === `${PATH_PREFIX}/index.html`) {
      request.uri = `${PATH_PREFIX}/`
    }

    // Modify the request's origin to be the prerender service
    request.origin = {
      custom: {
        domainName: PRERENDER_URL,
        port: 443,
        protocol: "https",
        readTimeout: 60,
        keepaliveTimeout: 5,
        sslProtocols: ["TLSv1", "TLSv1.1", "TLSv1.2"],
        path: "/https%3A%2F%2F" + request.headers["x-prerender-host"][0].value,
        customHeaders: {
          "x-prerender-token": [
            {
              key: "x-prerender-token",
              value: PRERENDER_TOKEN,
            },
          ],
        },
      },
    }
  } else {
    if (request.uri.endsWith("/")) {
      request.uri += "index.html"
    } else if (!request.uri.includes(".")) {
      request.uri += "/index.html"
    }
  }

  return request
}

Once invoked, the Lambda@Edge function requests the Prerender service to render our website. Then Prerender service, in turn, sends a new request to our CloudFront distribution, captures the webpage (with the dynamic content), render it, and returns it as a static HTML page.

Handling Errors

What happens if the page isn't available? Typically, we would present a default 404 page (the HTTP status code could be 404 or 2XX). However, the Prerender service, allows us to specify a status code by embedding it within the error page's metadata:

<meta name="prerender-status-code" content="404">

You can read more about this here.

Step 3: Responding with Static HTML | Response Handler

Once the Prerender service completes its task and sends back the static HTML, our third Lambda@Edge function formats the response.

As part of this process, we also set cache control headers to dictate how the returned responses should be cached.

// edge-functions/packages/response-handler/src/index.ts

// Create an Axios client instance for HTTP requests.
// This instance is defined outside the Lambda function for reuse between calls.
const instance = axios.create({
  timeout: 1000, // Set request timeout
  maxRedirects: 0, // Disable following redirects
  validateStatus: (status) => status === 200, // Only consider HTTP 200 as a valid response
  httpsAgent: new https.Agent({ keepAlive: true }), // Use a keep-alive HTTPS agent
})

export const handler = async (event: CloudFrontResponseEvent): Promise<CloudFrontResponse> => {
  const response = event.Records[0].cf.response

  // If the x-Prerender-requestid header is present, set cache-control headers.
  if (response.headers[`${cacheKey}`]) {
    response.headers["Cache-Control"] = [
      {
        key: "Cache-Control",
        value: `max-age=${cacheMaxAge}`,
      },
    ]
  }
  // If the response status isn't 200 (OK), fetch and set a custom error page.
  else if (response.status !== "200") {
    try {
      const res = await instance.get(errorPageUrl)
      response.body = res.data
      response.headers["content-type"] = [
        {
          key: "Content-Type",
          value: "text/html",
        },
      ]
      // Remove any pre-existing content-length headers as they might contain values from the origin.
      delete response.headers["content-length"]
    } catch (error) {
      // If fetching the custom error page fails, return the original response.
      return response
    }
  }
  return response
}

It's crucial to understand that regardless of the origin/source (S3 or Prerender), if the response isn't a 200 status code, here we provide our own custom error page. This page is fetched on-the-fly by Axios (the web-client), which sends a request to our website's 404 page. If you are following along with my previous article, I have get rid of the custom error page configuration from the CloudFront. See here

Wrapping it UP

We've discussed how to use AWS CloudFront edge functions and S3 for our static hosting needs. Our main goal? Boosting our site's SEO prowess. We broke down how web crawlers work, comparing it to the usual browser requests. Digging deeper, we discuss the foundation of our solution. In short, this article offers a roadmap for those wanting to optimize their static sites using AWS tools.

AWS Static Hosting - Part 01: CloudFront, S3 and Terraform

Krishan Thisera — Sun, 30 Jul 2023 03:22:02 +0000

Depending on your requirement, There are many ways to host a website in a cloud environment. And tons of frameworks at you r disposal to get the website up and running. Here in this article, we will focus on how we can leverage AWS CloudFront and S3 to set up our static hosting infrastructure.

I am planning to discuss the scenario using two articles. In this article, we will focus primarily on infrastructure setup, and the second article would be dedicated to enhancing SEO, using Lambda at Edge functions.

The source code for this article is available here

Before you begin

I wanted to emphasize the following, If you come from an agile background (who ain't eh?) say these are our user stories

We want the website to host our static content
- The content should be securely stored in an S3 bucket (no public access to the bucket)
- The traffic to the site should be secured (HTTPS)
- Leverage CloudFront to deliver the static content from the bucket
- Leverage CloudFront edge functions to support multi-page routing
The website has its own repository and a release cycle
- Provision an IAM user to Deploy the content to the S3 bucket and invalidate the CloudFront cache

Understanding CSR and SSR

Before we jump into our Infrastructure setup, it's crucial to understand the two primary rendering methods for web applications:

Client-Side Rendering (CSR)
Server-Side Rendering (SSR).

Client-Side Rendering (CSR): In CSR, the browser is responsible for rendering the content. When a user accesses the website, they receive a minimal HTML file. The browser then fetches the JavaScript, which, when executed, populates the page with content. This method is popular with Single Page Applications (SPAs) and offers a smooth user experience, especially for dynamic content.

Server-Side Rendering (SSR): With SSR, the server processes the request, renders the full page, and then sends the complete HTML content to the browser. This method is beneficial for SEO as search engines can crawl the content directly without executing JavaScript. It also provides a faster initial page load.

Factors to Consider When Choosing Between CSR and SSR

SEO Needs: If SEO is a priority, SSR might be more suitable as it ensures that search engines can easily index your content. A teaser 💡: In our second article, we will discuss how we can address this very issue.
Performance: CSR might introduce a slight delay before the user sees the content, as the browser needs to download and execute the JavaScript. On the other hand, SSR provides content instantly but might be resource-intensive on the server side.
Development Complexity: CSR-based SPAs can be simpler to develop and deploy, especially when using modern frameworks. SSR might introduce additional complexities, especially when dealing with caching, state management, etc.
User Experience: For dynamic applications where content changes frequently based on user interactions, CSR can offer a more fluid experience.

Throughout this series, our primary focus is on applications that utilize Client-Side Rendering (CSR).

Setting things up

Before we start we shall set up our terraform environment. In Terraform it is vital to maintain your state file secure. I personally prefer Terraform Cloud as the configuration is straightforward and we don't need to worry about CI/CD (GitOps-like) setup.

First, you will need to a create Terraform cloud account here (its free). Then create a project and set up a workflow, at this point, you would need to have a GitHub account (or from any VCS provider) to maintain your infrastructure code. I am not going to show-how this, as this is very straightforward.

If you have set up your Terraform workspace correctly, you should be able to see a webhook configured in your GitHub account. If you are using a VCS other than GitHub you would need to set this webhook manually.

You can follow this to set up the backend.

If you refer to the GitHub repo, config.remote.tfbackend describe my remote backend configuration. In this case, I am using a CLI input to configure the backend.

terraform init -backend-config=config.remote.tfbackend

Let's have a quick peek into our provider's configuration.

# provider.tf
terraform {
  required_version = "~> 1.5.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.7"
    }
  }
  backend "remote" {}
}

provider "aws" {
  region = var.aws_region
}

As we have set up our remote backend and conquered the provider configuration we can start writing our code.

S3 Buckets

For this section please refer to the s3.tf.

# S3 bucket for website.
resource "aws_s3_bucket" "blog_assets" {
  bucket = var.bucket_name

  tags = var.common_tags
}

# S3 Bucket Policy Association
resource "aws_s3_bucket_policy" "assets_bucket_cloudfront_policy_association" {
  bucket = aws_s3_bucket.blog_assets.id
  policy = data.aws_iam_policy_document.s3_bucket_policy.json
}

# S3 bucket website configuration 
resource "aws_s3_bucket_website_configuration" "assets_bucket_website" {
  bucket = aws_s3_bucket.blog_assets.id

  index_document {
    suffix = "index.html"
  }

  error_document {
    key = "404.html"
  }
}

# S3 bucket ACL
resource "aws_s3_bucket_acl" "assets_bucket_acl" {
  bucket     = aws_s3_bucket.blog_assets.id
  acl        = "private"
  depends_on = [aws_s3_bucket_ownership_controls.assets_bucket_acl_ownership]
}

# S3 bucket CORS configuration
resource "aws_s3_bucket_cors_configuration" "assets_bucket_cors" {
  bucket = aws_s3_bucket.blog_assets.id
  cors_rule {
    allowed_headers = ["Authorization", "Content-Length"]
    allowed_methods = ["GET", "POST"]
    allowed_origins = ["https://www.${var.domain_name}", "https://${var.domain_name}"]
    max_age_seconds = 3000
  }
}

# Set Bucket Object ownership
resource "aws_s3_bucket_ownership_controls" "assets_bucket_acl_ownership" {
  bucket = aws_s3_bucket.blog_assets.id
  rule {
    object_ownership = "BucketOwnerPreferred"
  }
  depends_on = [aws_s3_bucket_public_access_block.assets_bucket_public_access]
}

# Block public access to the bucket
resource "aws_s3_bucket_public_access_block" "assets_bucket_public_access" {
  bucket                  = aws_s3_bucket.blog_assets.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

There are a couple of things to note here,

S3 bucket website configuration: here we set the paths to our index document and error document. You can even pull this path out for and defined them as a variable.
S3 bucket CORS configuration: CORS configuration is a pretty generic one for our scenario.
Block public access to the bucket: we don't want our bucket objects to be publicly available.
Set Bucket Object ownership: To avoid complications, we set the object ownership to bucket owner.
S3 bucket policy: here we are referring to the S3 bucket policy, which has been specified in data.tf.

Let's take a look at the policy document. See data.tf

# S3 Bucket Policy to Associate with the S3 Bucket
data "aws_iam_policy_document" "s3_bucket_policy" {

  # Deployer User access to S3 bucket
  statement {
    sid    = "DeployerUser"
    effect = "Allow"

    actions = [
      "s3:ListBucket"
    ]

    principals {
      type        = "AWS"
      identifiers = [aws_iam_user.pipeline_deployment_user.arn]
    }

    resources = [
      "arn:aws:s3:::${var.bucket_name}"
    ]
  }

  # CloudFront access to S3 bucket
  statement {
    sid    = "CloudFront"
    effect = "Allow"

    actions = [
      "s3:GetObject",
      "s3:ListBucket"
    ]

    principals {
      type        = "Service"
      identifiers = ["cloudfront.amazonaws.com"]
    }

    condition {
      test     = "StringEquals"
      variable = "aws:SourceArn"

      values = [
        "${aws_cloudfront_distribution.blog_distribution.arn}"
      ]
    }

    resources = [
      "arn:aws:s3:::${var.bucket_name}",
      "arn:aws:s3:::${var.bucket_name}/*"
    ]
  }
}

Here we are specifying two different statements.

To let the pipeline deployer user to list the bucket
CloudFront service to read the S3 bucket content

It is important to note that we will be using Origin Access Control (OAC) instead of legacy OAI (Origin Access Identity) to provide access to S3 bucket content to the CloudFront.

# cloudfront.tf
resource "aws_cloudfront_origin_access_control" "blog_distribution_origin_access" {
  name                              = "blog_distribution_origin_access"
  origin_access_control_origin_type = "s3"
  signing_behavior                  = "always"
  signing_protocol                  = "sigv4"
}

See here

CloudFront

For this article, we shall keep the CloudFront configuration to a minimum. Let's discover more when we configure the Lambda at edge functions.

aws_cloudfront_distribution

Here we specify the origin configurations, as most of them are self-explanatory I am not planning to go and explain each one of them. But I think we might need an explanation for SSL certification configuration and function association.

SSL configuration

If we refer to the repository, my certificate set up is sort of a "Bring your own certificate", but I have put together a configuration for Email or DNS challenge validation.

Please refer to acm.tf. For Email validation use the following.

# SSL Certificate
resource "aws_acm_certificate" "ssl_certificate" {
  provider                  = aws.acm_provider
  domain_name               = var.domain_name
  subject_alternative_names = ["*.${var.domain_name}"]
  validation_method         = "EMAIL"

  tags = var.common_tags

  lifecycle {
    create_before_destroy = true
  }
}

Function association

If you see src/astro.js, I have a little edge function there and it is self-explanatory. A little confession, my blog is using the Astro framework, so I grab the code to the edge function from the documentation here.

// src/astro.js
function handler(event) {
  var request = event.request;
  var uri = request.uri;

  // Check whether the URI is missing a file name.
  if (uri.endsWith("/")) {
    request.uri += "index.html";
  }
  // Check whether the URI is missing a file extension.
  else if (!uri.includes(".")) {
    request.uri += "/index.html";
  }

  return request;
}

# Edge Functions
resource "aws_cloudfront_function" "astro_default_edge_function" {
  name    = "default_edge_function"
  runtime = "cloudfront-js-1.0"
  comment = "CloudFront Functions for Astro"
  publish = true
  code    = file("src/astro.js")
}

IAM

During the deployment, the deployer user would need to:

Push build artifacts to the S3 bucket
Do the CloudFront invalidation

If you skim through the file, you would get a good understanding of how this has been set up.

# IAM Policy for put S3 objects
resource "aws_iam_policy" "allow_s3_put_policy" {
  name        = "allow_aws_s3_put"
  description = "Allow Pipeline Deployment to put objects in S3"
  policy      = data.aws_iam_policy_document.allow_aws_s3_put.json
}

# IAM policy for cloudfront to invalidate cache
resource "aws_iam_policy" "allow_cloudfront_invalidations_policy" {
  name        = "allow_cloudfront_invalidate"
  description = "Allow pipeline user to create CloudFront invalidation"
  policy      = data.aws_iam_policy_document.allow_cloudfront_invalidate.json
}

# IAM User group for Pipeline Deployment
resource "aws_iam_group" "pipeline_deployment_group" {
  name = "${var.domain_name}_deployment_group"
}

# IAM Policy attachment for Pipeline Deployment - S3 PUT
resource "aws_iam_group_policy_attachment" "s3_put_group_policy_attachment" {
  group      = aws_iam_group.pipeline_deployment_group.name
  policy_arn = aws_iam_policy.allow_s3_put_policy.arn
}

# IAM Policy attachment for Pipeline Deployment - CloudFront Invalidation
resource "aws_iam_group_policy_attachment" "cloudfront_invalidation_group_policy_attachment" {
  group      = aws_iam_group.pipeline_deployment_group.name
  policy_arn = aws_iam_policy.allow_cloudfront_invalidations_policy.arn
}

# IAM User for Pipeline Deployment
resource "aws_iam_user" "pipeline_deployment_user" {
  name = "${var.domain_name}_deployer"
}

# IAM User group membership for Pipeline Deployment
resource "aws_iam_group_membership" "deployment_group_membership" {
  name = "pipeline_deployment_group_membership"
  users = [
    aws_iam_user.pipeline_deployment_user.name
  ]
  group = aws_iam_group.pipeline_deployment_group.name
}

We shall have two policy documents for each use case mentioned earlier,

IAM policy for CloudFront to invalidate the cache

# data.tf: IAM policy for CloudFront to invalidate cache
data "aws_iam_policy_document" "allow_cloudfront_invalidate" {
  statement {
    sid    = "AllowCloudFrontInvalidation"
    effect = "Allow"

    actions = [
      "cloudfront:CreateInvalidation",
      "cloudfront:GetInvalidation",
      "cloudfront:ListInvalidations"
    ]

    resources = [
      "${aws_cloudfront_distribution.blog_distribution.arn}"
    ]
  }
}

IAM Policy for managing S3 objects

# data.tf: IAM Policy for put S3 objects
data "aws_iam_policy_document" "allow_aws_s3_put" {
  statement {
    sid    = "AllowS3Put"
    effect = "Allow"

    actions = [
      "s3:GetObject*",
      "s3:GetBucket*",
      "s3:List*",
      "s3:DeleteObject*",
      "s3:PutObject",
      "s3:PutObjectLegalHold",
      "s3:PutObjectRetention",
      "s3:PutObjectTagging",
      "s3:PutObjectVersionTagging",
      "s3:Abort*"
    ]

    resources = [
      "arn:aws:s3:::${var.bucket_name}/*"
    ]
  }
}

As they are pretty generic, I am not going dig deep, but here we are attaching those two policies, to the deployer group call {var.domain_name}_deployment_group and then create a user called ${var.domain_name}_deployer and add it to the group.

Variables

There are a couple of variables I have been using, In my case, I use terraform.tfvars file to set the bucket name and the domain name. As I am not super comfortable with sharing my AWS account ID, I have copied the certificate ARN from a previously created certificate and save it against TF_VAR_ssl_certificate_arn. You can configure these variables in Terraform Cloud.

⚠️ Don't forget to set your AWS access key pair.

Depending on your configuration, you may either use CLI to apply the Terraform plan or, execute the plan using Terraform cloud.

Once, you've deployed the environment, you may need to manually create the IAM key pair using AWS Console, and use it in your CI/CD pipeline

Conclusion

In this article we discussed setting up static hosting on AWS using CloudFront, S3, and Terraform. We covered essential steps from configuring S3 buckets to setting up CloudFront distributions and managing IAM roles for security. When you set up AWS static hosting, using this systematic guide will help make your infrastructure reliable, safe, and adaptable.

Why should you consider to adopt Service Mesh

Krishan Thisera — Sun, 30 Jul 2023 03:11:41 +0000

This is a brief overview of why you should consider adopting Service Mesh with your existing Kubernetes cluster.
I'm not trying to demystify the concepts regards to Service Mesh but here I am trying to pitch some basic considerations you may take into account.
I should mention that if you are already an expert on the domain you can skip this. Especially, for this article, I will refer ISTIO.

Service Mesh, in terms of Kubernetes, is a relatively new concept in which I found it as a set of tools that enhance the capabilities of the Kubernetes ecosystem.

In this case, what are the enhancement that can be done to your existing Kubernetes architecture? Here I am not referring to your blog which is hosted in a Kubernetes
cluster with a couple of containers but how about an infrastructure where it runs dozens of containers encapsulated in dozens of Kubernetes PODs or let's call them microservices?

How do you manage the logs and traces?
How do you manage the routes?
How do you do the Load Balancing?
How to make those microservices resilient?
How do you handle the security and etc.

Likewise, I could list down more than a handful of concerns regards to the matter which means that Kubernetes does not solve all the problems. I am not trying to emphasise that,
Service Mesh is the silver bullet to all problems but depending on the context your traditional Kubernetes implementation may no longer accommodate your requirement.

Let me briefly go through the standard architecture of Service Mesh.

Please note that this is a very high-level diagram. The Control plane may contain more than one component.

Generally, those sets of components in the service mesh can be categorized into control plane components and data plane components. As it is obvious, the control plane manages the service mesh while the data plane carries the actual workload of our microservices.

It is important to note that, Service Mesh is a combination of technologies. For example, for ISTIO Service Mesh, the envoy from "Lyft" has been used as their native sidecar proxy and you may use Kiali as your Service Dashboard. Further, Prometheus is a dependency for Kiali and other tools, which are meant to achieve observability.

In the context of the Service Mesh, microservices communicate to their subordinate microservices via a proxy. Especially, the proxy may run as a side container parallel to the original workload. The key point here is that, by having a sidecar proxy, Service Mesh addresses most of the concerns previously mentioned.

Telemetry

One of the main reasons to associate Service Mesh with your Kubernetes cluster is to enhance observability.

Monitoring the services at a more concrete level
Trace the traffic flows in a distributed manner
Identify the services which cause the delays
Oversees the service connectivity
As mentioned, if you are using a service mesh, traffic between your microservice should flow through the respective sidecar proxy. Hence, it enables the control plane to collect the metrics regards to the traffic from the proxy.

In terms of ISTIO, ISTIO uses Envoy as its native sidecar proxy. More importantly, the level of abstraction provided by ISTIO in which you are not required to directly configure those proxies.
ISTIO provides a handy set of Customer Resource Definition(CRD) to configure them.

Kiali: can be used to inspect the connectivity between the services, and It's powerful that you can tune the traffic flow.

Jaeger: I would recall as my second favorite tool can be used to inspect the traces.

Prometheus: One of the most popular metric scraper
Grafana: As your dashboard

Routing

I will summarise a couple of routing features that I have been using in production,

Canaries: Canary releases or weighted routings are one of the most popular routing scenarios in the context of microservices.
Assume that you have a service with multiple versions, If you ought to split the traffic between those versions depending on the percentage, those are canaries.

Circuit Breakers: Say that, you have service A, B, and C, where A depend on B and B, depending on C, in a scenario where service C take too much time to respond,
both A and service B will be affected. In this case, you may configure a Circuit breaker with a time-out that returns an acceptable response.
ISTIO uses Circuit Breaking and Outlier Detection interchangeably.

Hidden Release: Have you ever tried testing in production? Say that you have a service with two versions. Version-1 is the production version and Version-2 is
the newer version which is in the testing phase. You may deploy both versions on your mesh and use custom HTTP headers to direct the traffic to Version-2.

mTLS: The bonus feature which comes with ISTIO and other service meshes. You are no longer required to maintain SSL connectivities at the source code level.
Envoy proxy automatically does that for you. Say that you have 5-nodes Amazon EKS (Elastic Kubernetes Service). If you have multiple services (interconnect with each other),
your services may not be in the same node and you will never know the underlying physical connectivity between the nodes. Sometimes the traffic may flow through
different physical equipment (Switches. Routers, Servers) in the AWS datacentre.
By using SSL connectivity between proxies, service mesh secure your traffic without giving you any burden.
Fault Injection: If you are required to test your application with a scenario where a service got crashed or was poorly reachable,
you can assign the service a delay response at the proxy level and observe the behaviour. This feature is much useful when you are executing chaos testing.

Traffic Mirroring: You can easily mirror the traffic from a particular service to another service.

Create your own ISTIO Playground

Lastly, for the time being, ISTIO document has some vague areas where those sections are not that clear. For example, ISTIO ingress with SSL certificates.
The following GIT repository will walk you through ISTIO implementation with Cert-Manager.

ISTIO with cert-manager

Note that, this is a POC that I have designed and only describes the steps that you may follow to implement ISTIO with Cert-Manager.
It is heavily recommended to follow the official documentation for both ISTIO and Cert-Manger for better understanding.