DEV Community: reoring

Submariner Lighthouse: Multi-Cluster Service Discovery for Kubernetes

reoring — Tue, 15 Apr 2025 11:09:37 +0000

Background: The Multi-Cluster DNS Challenge

Kubernetes clusters handle service name resolution internally via DNS (CoreDNS or kube-dns). However, Kubernetes does not natively support cross-cluster service discovery – a service name is only visible within its own cluster. In a hybrid or multi-cloud environment with multiple clusters, this becomes a challenge: how can services in different clusters find and communicate with each other by name? The Submariner project addresses multi-cluster networking, and its Lighthouse component specifically tackles cross-cluster DNS-based service discovery (Multicluster Service Discovery in OpenShift (Part 1)). Lighthouse provides a way for pods in any connected cluster to resolve DNS names of services from other clusters as if they were local, enabling seamless multi-cluster service connectivity (Multicluster Service Discovery in OpenShift (Part 1)) (Multicluster Service Discovery in OpenShift (Part 1)).

Lighthouse was created before a Kubernetes standard for multi-cluster services existed, but it aligns with the emerging Kubernetes Multi-Cluster Services (MCS) API. In fact, Lighthouse now implements the MCS API definitions (like ServiceExport and ServiceImport) defined by the Kubernetes community (Service Discovery :: Submariner k8s project documentation website). This means it uses standard resource types and a special DNS domain (clusterset.local) to make services discoverable across clusters (Submariner Lighthouseにおける組み込み型CoreDNS設計の深掘り #kubernetes - Qiita). When configured, a service exported from one cluster becomes accessible via a DNS name of the form <service>.<namespace>.svc.clusterset.local in other clusters (Submariner Lighthouseにおける組み込み型CoreDNS設計の深掘り #kubernetes - Qiita).

What is Submariner’s Lighthouse?

Submariner is an open-source project (a CNCF Sandbox project) that creates secure IP tunnels between Kubernetes clusters, essentially “flattening” the network so that pods and services in different clusters can communicate directly. Lighthouse is a component of Submariner that builds on this connectivity to provide multi-cluster service discovery via DNS (Multicluster Service Discovery in OpenShift (Part 1)). It works in conjunction with the network tunnels: Submariner ensures cross-cluster networking, and Lighthouse ensures that a service in cluster A can be reached by name from cluster B without manual configuration.

In simpler terms, Lighthouse lets you expose a Kubernetes Service from one cluster and access it by the same DNS name from other clusters. It accomplishes this by distributing service information between clusters and leveraging DNS resolution. Lighthouse’s solution is compatible with any Kubernetes CNI plugin or environment (cloud or on-prem), because it operates at the DNS level and is agnostic to the underlying network provider (GitHub - submariner-io/lighthouse: DNS service discovery across connected Kubernetes clusters.).

Some key features of Lighthouse include:

Opt-in Service Export: Only services that you explicitly mark for export are shared across clusters. This is done by creating a ServiceExport resource with the same name and namespace as the Service you want to export (Multicluster Service Discovery in OpenShift (Part 1)). If a service isn’t exported, Lighthouse will ignore it (and other clusters won’t be able to resolve it).
Multi-Cluster DNS Resolution: Lighthouse introduces a special DNS domain (default clusterset.local) that is authoritative for multi-cluster service names. It runs a DNS server in each cluster to handle queries for this domain (Multicluster Service Discovery in OpenShift (Part 1)). The existing cluster DNS (CoreDNS) is configured to forward any queries for *.clusterset.local to the Lighthouse DNS server (Multicluster Service Discovery in OpenShift (Part 1)).
Service Discovery Across Clusters: When a Service is exported, Lighthouse shares its details (like cluster IP and ports) with other connected clusters. Those clusters can then resolve the service’s <svc>.<ns>.svc.clusterset.local name to the appropriate IP address, even if the actual service only exists in a different cluster (Submariner Lighthouseにおける組み込み型CoreDNS設計の深掘り #kubernetes - Qiita). This works for both A (address) records and SRV (service/port) records, meaning you can resolve the service’s IP and also perform service discovery for specific ports if needed (Service Discovery :: Submariner k8s project documentation website).

Lighthouse Architecture and Components

(Service Discovery :: Submariner k8s project documentation website) Lighthouse architecture: Lighthouse Agents in each cluster synchronize service metadata via a central broker, and each cluster runs a Lighthouse DNS (CoreDNS) server. The cluster DNS (CoreDNS) forwarder sends clusterset.local queries to the Lighthouse DNS server for resolution.

At a high level, Lighthouse’s architecture consists of a Broker, a Lighthouse Agent in each cluster, and a Lighthouse DNS server (based on CoreDNS) in each cluster (Service Discovery :: Submariner k8s project documentation website) (Multicluster Service Discovery in OpenShift (Part 1)). The Broker is a Kubernetes control plane (which can be a dedicated cluster or one of the participating clusters) that acts as a central exchange for service information. Each cluster that joins the Lighthouse cluster set does so by connecting to the Broker.

Let’s break down the components:

Lighthouse Agent (Controller)

The Lighthouse Agent runs in every member cluster. Its job is to watch for services being exported and share that information through the Broker to other clusters (Service Discovery :: Submariner k8s project documentation website). It also listens for services exported from other clusters and imports those into the local cluster’s records. In practice, Lighthouse leverages Kubernetes CRDs (Custom Resource Definitions) for this:

When you create a ServiceExport in a cluster, the Lighthouse Agent detects it. It then creates a corresponding ServiceImport resource (and related Endpoint data) in the Broker, representing that service in the multicluster context (Service Discovery :: Submariner k8s project documentation website). The ServiceImport contains information like the service’s cluster IP and which cluster it came from.
The Broker’s Kubernetes API acts as a hub. Agents from other clusters will see the new ServiceImport in the Broker and will import it into their local cluster (Service Discovery :: Submariner k8s project documentation website). Concretely, each cluster ends up with a copy of the ServiceImport for the service, which it can use to answer DNS queries.
This process happens for every exported service (and conversely, if a ServiceExport is removed, Lighthouse will withdraw that service’s info from the Broker and other clusters).

This architecture means service metadata is synchronized across clusters. The Lighthouse Agent essentially implements the controller logic of the multi-cluster service API. The result is that all clusters have a consistent view (via ServiceImport objects) of which services are available in the multi-cluster environment.

Lighthouse DNS Server (Multi-Cluster DNS)

The Lighthouse DNS server runs in each cluster as a CoreDNS instance with a custom plugin. It serves as the DNS authority for the special multi-cluster domain (by default, clusterset.local) (Multicluster Service Discovery in OpenShift (Part 1)). The cluster’s own DNS (typically CoreDNS) is configured to forward any queries for *.clusterset.local to this Lighthouse DNS service (Multicluster Service Discovery in OpenShift (Part 1)). In effect, Lighthouse inserts itself into the DNS resolution chain only for the multi-cluster service names.

Inside the Lighthouse DNS server, the custom lighthouse CoreDNS plugin uses the ServiceImports (and associated Endpoint data) that the Agent has distributed to build an in-memory cache of records (Service Discovery :: Submariner k8s project documentation website). When it receives a query, it will check this cache:

If a matching ServiceImport exists (meaning the service is exported by some cluster), the Lighthouse DNS will return an answer – an A record pointing to the service’s cluster IP address in one of the clusters that provide the service (Submariner Lighthouseにおける組み込み型CoreDNS設計の深掘り #kubernetes - Qiita). If multiple clusters have exported the same service, Lighthouse can return multiple IPs (one per cluster) or choose one. Notably, if the service also exists in the local cluster, Lighthouse favors the local service IP first, then falls back to remote cluster IPs in a round-robin fashion for load-balancing (Service Discovery :: Submariner k8s project documentation website).
If no ServiceImport is found for that name (meaning no cluster has exported such a service), the Lighthouse DNS responds with NXDOMAIN (non-existent domain) (Submariner Lighthouseにおける組み込み型CoreDNS設計の深掘り #kubernetes - Qiita).
Lighthouse DNS also supports SRV record queries for services, allowing discovery of service ports across clusters (Service Discovery :: Submariner k8s project documentation website). An SRV query like _http._tcp.myservice.myns.svc.clusterset.local can return the hostname(s) and port for that service if available.

The workflow for a cross-cluster DNS query is as follows (Service Discovery :: Submariner k8s project documentation website):

A pod in Cluster A tries to resolve myservice.myns.svc.clusterset.local. This DNS request is intercepted by Cluster A’s CoreDNS.
CoreDNS sees that the query is for the clusterset.local domain, which it is not authoritative for. According to its configuration, it forwards the query to the Lighthouse DNS service running in Cluster A (Service Discovery :: Submariner k8s project documentation website).
The Lighthouse DNS server in Cluster A checks its cache of ServiceImports (which includes information about services exported from Cluster B, Cluster C, etc.). If myservice.myns has been exported by another cluster, Lighthouse finds the corresponding record (e.g., an IP from Cluster B) and returns it in a DNS response (Submariner Lighthouseにおける組み込み型CoreDNS設計の深掘り #kubernetes - Qiita). If it doesn’t find anything, it returns NXDOMAIN.
CoreDNS receives the answer from Lighthouse and sends the response back to the requesting pod, which can now connect to the service’s IP as if it were a local service.

This approach allows seamless cross-cluster service name resolution – pods continue to use DNS to connect to services, and whether the service is local or in another cluster is transparent to them. The entire lookup happens behind the scenes via Lighthouse’s coordination.

Lighthouse’s Embedded DNS vs. CoreDNS Plugin Approach

One notable design choice of Lighthouse is how it implements the DNS server. Instead of simply adding a plugin to the existing CoreDNS deployment in the cluster, Lighthouse runs its own CoreDNS-based DNS server as a separate component. In essence, Lighthouse embeds a CoreDNS process within its deployment (hence an “embedded DNS server”) rather than modifying the cluster’s CoreDNS directly (Submariner Lighthouseにおける組み込み型CoreDNS設計の深掘り #kubernetes - Qiita).

To clarify, the Lighthouse project actually includes a CoreDNS plugin named “lighthouse” which can handle multi-cluster DNS logic. Initially, one might expect to install this plugin into the cluster’s CoreDNS. However, Submariner’s Lighthouse opts for a different architecture: it packages CoreDNS (with the Lighthouse plugin) into a dedicated Deployment (submariner-lighthouse-coredns) and just configures the cluster’s DNS to forward queries to it (Submariner Lighthouseにおける組み込み型CoreDNS設計の深掘り #kubernetes - Qiita) (Submariner Lighthouseにおける組み込み型CoreDNS設計の深掘り #kubernetes - Qiita). This design has several advantages:

Isolation of Cross-Cluster DNS Logic: The multi-cluster domain resolution is kept separate from the cluster’s normal DNS service. This means any issues or bugs in Lighthouse’s DNS handling will not directly affect the regular cluster.local DNS resolution for in-cluster services (Submariner Lighthouseにおける組み込み型CoreDNS設計の深掘り #kubernetes - Qiita). The core Kubernetes DNS remains untouched except for one forwarding rule. This isolation improves reliability – if Lighthouse were to crash, your normal in-cluster DNS still works fine.
Ease of Installation & Compatibility: If Lighthouse had to be integrated as a plugin, one might need to custom-build a CoreDNS image with the plugin, or modify the CoreDNS ConfigMap extensively to load the external plugin. This can be challenging, especially on managed Kubernetes platforms (like OpenShift, GKE, or AKS) where the CoreDNS configuration is managed by the platform and custom plugins may not be supported. By running a separate DNS server, Lighthouse avoids incompatibilities – you simply add a stub-domain or forward entry to existing DNS. For example, on OpenShift, the Submariner operator adds a DNS forwarding rule for supercluster.local (or clusterset.local) via the DNS Operator (Submariner Lighthouseにおける組み込み型CoreDNS設計の深掘り #kubernetes - Qiita), and on AKS it updates the coredns-custom ConfigMap to achieve the same (Submariner Lighthouseにおける組み込み型CoreDNS設計の深掘り #kubernetes - Qiita). No custom CoreDNS binaries are required.
Future Flexibility (Standard API alignment): The multi-cluster services API is still evolving. Lighthouse initially implemented its own CRDs (like ServiceImport) before the Kubernetes SIG Multicluster standardized them (Multicluster Service Discovery in OpenShift (Part 1)). By keeping the Lighthouse DNS as a separate process under its control, the project can evolve its implementation easily – for instance, switching to upstream API versions or changing internal logic – without affecting the cluster’s core DNS. As long as the interface (forwarding queries to Lighthouse) remains the same, the internals can be updated to adapt to new versions of the MCS API (Submariner Lighthouseにおける組み込み型CoreDNS設計の深掘り #kubernetes - Qiita). If it were a built-in plugin, updating it could be more involved and tied to the cluster DNS version.
Works with Any Cluster DNS/CNI: This design is very flexible – it doesn’t even require that the cluster DNS be CoreDNS. Even if a cluster used another DNS implementation, as long as it can forward a domain to an external server, Lighthouse will work (Submariner Lighthouseにおける組み込み型CoreDNS設計の深掘り #kubernetes - Qiita). Similarly, it’s independent of the network plugin or service mesh; Lighthouse only assumes that clusters can route traffic to each other (which Submariner provides) and that DNS queries for the special domain can be forwarded. This aligns with the project’s goal to be CNI-agnostic and environment-agnostic (GitHub - submariner-io/lighthouse: DNS service discovery across connected Kubernetes clusters.).

In summary, the Lighthouse team embedded a CoreDNS server within Lighthouse to provide a drop-in DNS solution for multi-cluster services. This “batteries-included” DNS server approach means each cluster runs a Lighthouse DNS pod (CoreDNS with the lighthouse plugin) that cooperates with the cluster’s main DNS. The clusterset DNS zone is handled by Lighthouse’s pods, and the main CoreDNS just delegates those queries. This was a deliberate design choice to maximize compatibility and minimize disruption to existing clusters (Submariner Lighthouseにおける組み込み型CoreDNS設計の深掘り #kubernetes - Qiita) (Submariner Lighthouseにおける組み込み型CoreDNS設計の深掘り #kubernetes - Qiita).

Configuring and Using Lighthouse

Setting up Lighthouse is straightforward, especially with the Submariner toolkit. Typically, you would:

Deploy a Broker and Join Clusters: One cluster (or a separate cluster) is designated as the broker. Using Submariner’s CLI subctl, you deploy the broker and then join each cluster to it. For example, subctl deploy-broker on the broker cluster, then on each member cluster subctl join --broker <broker-info> --clusterid <cluster-name> --enable-discovery (in newer versions this flag might be --enable-service-discovery) to deploy Submariner components including Lighthouse (Multicluster Service Discovery in OpenShift (Part 2)). The Submariner operator on each cluster will install the Lighthouse Agent and Lighthouse CoreDNS pods automatically, as well as set up the CoreDNS forwarding for clusterset.local domain.
Export a Service: By default, no service is exported (to avoid accidentally exposing everything). To export a service, you can use kubectl to create a ServiceExport object in the same namespace with the same name as the Service. For convenience, Submariner’s CLI offers subctl export service --namespace <ns> <service-name> which creates the ServiceExport for you (Multicluster Service Discovery in OpenShift (Part 2)). For example, if you have a Deployment and Service called “nginx” in the “default” namespace of Cluster B, you would run subctl export service --namespace default nginx on Cluster B to export it.
Discover from Other Clusters: Once exported, the service becomes discoverable to other clusters in the cluster set. From a pod in Cluster A (or even using kubectl run to create a temporary pod for testing), you should be able to resolve and reach nginx.default.svc.clusterset.local. For instance, you might exec into a test pod and run:

   $ kubectl exec -n default -ti <some-pod> -- /bin/sh
   $ nslookup nginx.default.svc.clusterset.local
   Server:    10.96.0.10
   Address:   10.96.0.10#53

   Name:   nginx.default.svc.clusterset.local
   Address: 172.31.173.226

In this example, 172.31.173.226 is the Service’s cluster IP from the remote cluster that exported “nginx” (Multicluster Service Discovery in OpenShift (Part 2)). The DNS resolution is handled by Lighthouse behind the scenes. If you then attempt to actually curl nginx.default.svc.clusterset.local:8080 (assuming the service listens on port 8080), the request will be routed over the Submariner tunnels to the cluster that hosts the service, and you should get a response.

If the DNS lookup fails (you get an NXDOMAIN or no answer), it likely means either the service wasn’t exported or the local CoreDNS isn’t correctly forwarding the clusterset.local zone. Ensure that the ServiceExport exists and is in an Exported state (you can check by kubectl get serviceexport <name> and seeing its status). Also verify that your cluster’s CoreDNS ConfigMap has an entry to forward clusterset.local to the Lighthouse DNS service IP (Troubleshooting :: Submariner k8s project documentation website) (Troubleshooting :: Submariner k8s project documentation website). This is usually done for you by Submariner, but in some cases (or custom setups) you might need to add a stanza like below to CoreDNS config:

clusterset.local:53 {
    forward . <IP-of-submariner-lighthouse-coredns-service>
}

This tells CoreDNS to forward queries for the clusterset.local domain to Lighthouse’s CoreDNS. The IP to use is the ClusterIP of the submariner-lighthouse-coredns service (which you can find with kubectl -n submariner-operator get svc submariner-lighthouse-coredns (Multicluster Service Discovery in OpenShift (Part 2))).

Keep in mind that Lighthouse only answers for services that have been exported. For example, if you try to resolve a service that wasn’t exported, it will not be found. In a quick test, if you attempt to curl a service’s clusterset URL before exporting it, it will fail to resolve. After you create the ServiceExport (and give it a few seconds to propagate), the same curl should succeed (Multicluster Service Discovery in OpenShift (Part 2)) (Multicluster Service Discovery in OpenShift (Part 2)).

Verifying Multi-Cluster Service Discovery

You can verify that Lighthouse is working using standard DNS tools. As shown above, using nslookup or dig inside a cluster to query a <svc>.<ns>.svc.clusterset.local name will indicate whether the DNS is resolving to an IP. You should see an IP address in the ANSWER section which should match one of the service’s cluster IPs from a remote cluster. Here’s an example using a demo service called “nginx-hello” in namespace “demo” (from an AWS blog example):

$ kubectl exec -it client-hello -n demo -- nslookup nginx-hello.demo.svc.clusterset.local
Name:   nginx-hello.demo.svc.clusterset.local
Address: 172.20.80.119

As expected, we got an IP address for the multi-cluster service (Kubernetes Multi-Cluster Service Discovery using Open Source AWS Cloud Map MCS Controller | AWS Open Source Blog). If that service were available in multiple clusters, multiple IPs could be returned (and by default, Lighthouse would give the IP local to the querying cluster first, if the service exists locally).

Lighthouse also supports round-robin DNS responses when a service is backed by multiple clusters. For instance, if cluster1 and cluster2 both export a “nginx-demo” service, queries in cluster1 might sometimes get the local cluster1 IP and sometimes the remote cluster2 IP (if configured to do round-robin). This can be observed by making repeated queries – the returned address may cycle through the available endpoints in different clusters (Kubernetes Multi-Cluster Service Discovery using Open Source AWS Cloud Map MCS Controller | AWS Open Source Blog).

Recent Features: ClusterSet IPs and Headless Services

The Submariner community has been actively improving Lighthouse. Two interesting features in recent versions:

Cluster-Set Virtual IP: Lighthouse can allocate a virtual IP that is shared across all clusters for an exported service. This is sometimes called a ClusterSet IP. Normally, when you resolve a service via Lighthouse, you get the actual ClusterIP of one of the Kubernetes Services. With the ClusterSet IP feature enabled (by adding an annotation lighthouse.submariner.io/use-clusterset-ip on the ServiceExport, or enabling it for all services when deploying the broker), Lighthouse will instead assign a global virtual IP to represent that service (User Guide :: Submariner k8s project documentation website). All clusters will see the same virtual IP for the service, and Lighthouse DNS will return that IP for queries (User Guide :: Submariner k8s project documentation website). (This is useful in scenarios where you might want a consistent IP for a service across clusters, perhaps for external DNS or other integration.) Note that Submariner itself does not route traffic to this virtual IP – you’d need an external mechanism (e.g., routers or an external DNS) to handle it, so use this feature with that in mind (User Guide :: Submariner k8s project documentation website).
Headless Service & StatefulSet Support: Initially, Lighthouse focused on cluster IP services. Newer updates added support for Headless Services, particularly to enable stateful workloads across clusters. Kubernetes headless services (with no cluster IP) allow pods in a StatefulSet to have stable DNS names within a cluster (e.g., pod-0.myservice.myns.svc.cluster.local). Lighthouse extends this concept across clusters by incorporating the cluster ID into the DNS name. For example, if you have a StatefulSet “web” with a headless service “nginx-ss” and you export it from multiple clusters, you could address a specific pod in a specific cluster via a DNS name like web-0.<cluster-id>.nginx-ss.nginx-test.svc.clusterset.local (User Guide :: Submariner k8s project documentation website). This feature is advanced but demonstrates Lighthouse’s ability to handle even complex DNS scenarios in multi-cluster setups (ensuring uniqueness by cluster). It requires that the cluster IDs are DNS-compatible labels (User Guide :: Submariner k8s project documentation website) (since they appear in the DNS names).

For most users, the core functionality of Lighthouse – exporting services and resolving them across clusters – will be the main attraction. These additional features provide flexibility for more complex use cases.

Conclusion

Submariner’s Lighthouse component greatly simplifies multi-cluster Kubernetes deployments by providing a built-in, DNS-based service discovery mechanism. By exporting a service from one cluster, you can make it available to all connected clusters via the familiar Kubernetes DNS naming conventions (with a clusterset.local twist). Under the hood, Lighthouse handles distribution of service endpoints and integrates with CoreDNS, so that your applications don’t need any special logic – they can just use DNS and Kubernetes Services as they normally would.

We’ve seen how Lighthouse’s architecture (with Lighthouse Agents and an embedded CoreDNS server per cluster) makes it robust and easy to adopt without altering existing cluster DNS servers. We also discussed how to configure and use Lighthouse, from setting up the broker and agents to exporting services and verifying cross-cluster connectivity.

If you are implementing multi-cluster Kubernetes for high availability, disaster recovery, or hybrid cloud workloads, Lighthouse is a powerful ally. It works in tandem with Submariner’s network connectivity to truly blur the lines between clusters when it comes to service access. Developers and DevOps teams can deploy multi-cluster services with minimal changes to their workflows.

Further Resources: The Submariner project’s official documentation and GitHub have more details. The [Submariner.io Service Discovery docs】 (Service Discovery :: Submariner k8s project documentation website) provide a deep dive into the architecture, and the project is open source on GitHub (submariner-io/lighthouse). The Kubernetes MCS (Multi-Cluster Services) API is also worth reading up on, as Lighthouse’s design aligns with it. With Lighthouse in place, you can turn a collection of Kubernetes clusters into a unified, service-discovery-friendly ClusterSet – making multi-cluster Kubernetes feel a lot more like a single extended cluster. (Multicluster Service Discovery in OpenShift (Part 1)) (Multicluster Service Discovery in OpenShift (Part 1))

Envoy Gateway 1.3.0 – Overview of the New “Rate Limiting with Cost” Feature

reoring — Tue, 04 Feb 2025 02:06:46 +0000

Envoy Gateway v1.3.0 introduces an important enhancement to its rate limiting capabilities: Rate Limiting with Cost. This feature allows each request to consume a configurable “cost” from the rate limit budget, rather than counting every request as a single hit. In practice, this enables usage-based rate limiting, where different requests can deduct different amounts from the allowed quota. This overview will explain the feature’s details, verify them against official sources, and provide an English translation of the key points from the Japanese article, with accurate technical context. The target audience is assumed to be familiar with Envoy Gateway, Kubernetes, and Envoy’s rate limiting concepts.

Background: Envoy Gateway Rate Limiting

Envoy Gateway supports rate limiting to control traffic to services. In prior versions, rate limits were primarily count-based – each request counted as “1” towards a fixed limit (e.g. 100 requests per minute). Envoy Gateway implements global rate limiting (using an external rate limit service with Redis) as well as local (per-instance) rate limits. Global rate limiting ensures the limit is enforced across all Envoy replicas (e.g. 10 req/sec globally means 5 req/sec on one proxy + 5 req/sec on another would hit the limit). By default, when a rate limit is exceeded, Envoy Gateway returns an HTTP 429 (Too Many Requests) response.

Why “cost”-based rate limiting? In some scenarios, not all requests are equal. For example, an API might want to allocate more budget to expensive operations (like complex queries or large data transfers) or charge users based on usage (e.g. bandwidth or computational cost). A key use case is Generative AI APIs – one request might generate a response with thousands of tokens, consuming significant compute resources. Counting each request as 1 doesn’t reflect the actual load or cost imposed by that request. The community raised this in GitHub issues such as “Usage based Rate Limiting (Counting from response header values)” (Issue #4756) and “Generative AI support” (Issue #4748), prompting the need for a more flexible rate limiting mechanism.

Envoy Gateway 1.3.0 addresses this with Rate Limiting with Cost, which lets you assign a variable cost per request (and response) when decremented from the rate limit counters, instead of a fixed 1 per request.

New Feature Overview: Rate Limiting with Cost

In Envoy Gateway v1.3.0, the rate limit API (part of the BackendTrafficPolicy CRD or RateLimitFilter CRD) now supports a cost specifier for each rate limiting rule. This is implemented by adding a cost field to the rate limit configuration. The official release notes highlight “Rate Limiting with Cost: Added support for cost specifier in the rate limit BackendTrafficPolicy CRD.”. In practice, this means you can configure how much each request will count against the limit, and even split that into a request-phase cost and a response-phase cost.

Cost Configuration in RateLimit Rules

Each rate limit rule can now include an optional cost setting, which has two sub-fields: request and response. If cost is omitted, the behavior remains the same as previous versions: each request consumes 1 count at request time, and nothing is consumed at response time. By default, every incoming request will decrement the remaining quota by 1, and the response has no effect on the quota.

When cost is specified, you have fine-grained control:

Request Cost (cost.request): This defines how much to deduct from the rate limit counter when a request is received (before it’s forwarded to the backend). If you set this to a number >1, each request will consume that many “credits.” If the remaining quota is less than this cost, the request will be rate-limited (Envoy will respond with 429 immediately). You can also set this to 0, meaning do not deduct any quota at request time – effectively just perform a check without consumption. A 0 request cost can be useful in scenarios where you only want to enforce limits based on the response (as described below).
Response Cost (cost.response): This defines how much to deduct from the rate limit counter after the response is sent back to the client (when the request/response stream completes). This is particularly useful for usage-based limits where the “cost” of a request can only be determined after processing the request – for instance, after generating a response (e.g., counting AI tokens or data size). The crucial point is that the response cost is applied after the request has been processed, so it does not retroactively affect the current request’s admission. Instead, it will reduce the available quota for subsequent requests. In other words, even if a response has a high cost, the current request will always be allowed to complete once started; the cost will be accounted against future requests. If cost.response is not specified, no deduction occurs on response (so responses don’t affect the quota by default).

Both cost.request and cost.response are defined as Cost Specifiers, which means you can decide how the cost value is obtained:

Fixed Number: You can specify a fixed integer. In the CRD, this is done by setting from: Number and providing a number value. For example, cost.request.from: Number with cost.request.number: 5 means each request will consume 5 units from the quota. A fixed number is straightforward for static costs. (If you set the number to 0, as mentioned, Envoy will only perform a limit check without consuming tokens – effectively allowing you to gate the request on the current budget without deducting at that moment.)
Dynamic Metadata: More powerfully, you can have the cost be determined dynamically from the request’s metadata. In this mode, you set from: Metadata and specify a metadata source with a namespace and key. Envoy will retrieve a numeric value from the per-request dynamic metadata under that namespace/key and use it as the cost. This requires that some part of the request processing (e.g., an external processing filter or a WASM extension) has injected the usage value into Envoy’s dynamic metadata. For instance, an external gRPC service (using Envoy’s External Processing filter) could calculate the number of tokens used by an AI model and return that in dynamic metadata; Envoy Gateway can then pick up that value and deduct it from the rate limit budget. This design was precisely intended for generative AI use cases, where the cost (number of tokens) is determined at runtime. The Envoy Gateway API reference confirms that valid sources for cost are “Number” or “Metadata”, and that the metadata source requires specifying which dynamic metadata namespace and key to read.

Supported Scope: It’s important to note that as of v1.3.0, cost-based rate limiting is supported only for HTTP global rate limits. Global rate limiting is the variant that uses the external rate limit service (with Redis) to coordinate counts across Envoy instances. The cost specifier currently works in that context. If you configure a BackendTrafficPolicy or RateLimitFilter with type: Local (per-proxy rate limiting), the cost fields are not applied in this release. Likewise, the cost mechanism is oriented toward HTTP traffic. (The release notes and docs do not mention TCP or gRPC usage for cost-based limits in this version.)

Example Configuration

Let’s illustrate how one would configure Rate Limiting with Cost in Envoy Gateway 1.3.0. Assume we want to limit each client to 1000 “tokens” per minute on a certain API route, where the token count of each request is determined by an external processing step (for example, the number of GPT-4 tokens generated in the response). We want to allow each request through initially (as long as some budget remains), and deduct the exact tokens used after the response is ready.

We could define a BackendTrafficPolicy (or a RateLimitFilter) with a rule like:

apiVersion: gateway.envoyproxy.io/v1alpha2
kind: BackendTrafficPolicy
metadata:
  name: ai-api-rate-limit
spec:
  rateLimit:
    type: Global
    global:
      rules:
      - limit:
          requests: 1000            # 1000 tokens 
          unit: Minute              # per minute window
        cost:
          request:
            from: Number
            number: 0              # don't consume at request, just check
          response:
            from: Metadata
            metadata:
              namespace: ext_proc  # the dynamic metadata namespace used by ext processor
              key: token_count     # the key where the token count is stored

In this example, when a request comes in matching this rule, Envoy will check the global counter (say, via Redis) to see if at least 1 token is available (because request cost is 0, it doesn’t immediately deduct, but presumably Envoy still ensures the limit is not already fully exhausted – effectively allowing one request even if at exactly 0 tokens left, since cost 0 means it wouldn’t block on count? In practice, one might set a minimal request cost of 1 to ensure a check. However, using 0 means “just check but not consume” as a special case). The request is allowed to proceed to the AI service. The AI service (through an external processing filter or another mechanism) returns, for example, that token_count = 150 for this response. Envoy then, after sending the response, will deduct 150 from the rate limit counter in Redis. The net effect is that the client has consumed 150 out of 1000 tokens in that minute. If another request comes in and the remaining budget is less than the needed tokens, it will be handled accordingly. Future requests will be rejected with 429 only once the accumulated token usage exceeds 1000 in the minute window.

Note: The ability to use number: 0 for request cost is a deliberate feature to support this pattern of “check then deduct later.” The Envoy Gateway docs state that using zero as the cost allows you to “only check the rate limit counters without reducing them”. This is ideal for scenarios where the exact cost isn’t known until later – you ensure there was budget when starting the request (or at least that the limit wasn’t already completely exhausted), and then finalize the accounting at the end.

Internal Implementation and Related Issues

This feature was implemented in the Envoy Gateway codebase via two key pull requests: PR #4957 (which defined the API changes adding the cost fields to the CRD) and PR #5035 (which implemented the translation logic that turns this API configuration into Envoy’s configuration). The maintainers discussed naming (initially calling it “hits_addend” before settling on a clearer cost terminology) and ensured that Envoy Proxy support was in place (Envoy Proxy added support for adjusting rate limit counters via dynamic metadata in a relatively recent version, which Envoy Gateway now leverages). In fact, the translator implementation notes that response cost requires Envoy proxy version >= 1.33.0 to work properly.

The driving use-cases for Rate Limiting with Cost are captured in the GitHub issues mentioned earlier. Issue #4756 outlined usage-based rate limiting, for example by counting values from a response header. Issue #4748 is related to Generative AI support – the idea of integrating Envoy Gateway with AI workloads. In tandem, an Envoy AI Gateway effort (see the separate repository envoyproxy/ai-gateway) introduced an AIGatewayRoute resource that can calculate token usage. In fact, a corresponding update in the AI Gateway project added a RequestCost field to the AI-specific route definition so that rate limiting based on “token usage” is possible. This demonstrates the synergy: Envoy Gateway’s core now supports the generic mechanism (cost-based limits), and the AI Gateway integration can supply the actual usage values (tokens) to the rate limiter. The commit message explicitly states this allows limiting based on calculated token usage.

Conclusion

“Rate Limiting with Cost” in Envoy Gateway 1.3.0 is a powerful new feature that enhances the flexibility of API rate limiting. It enables use cases like per-user consumption quotas, tiered API usage plans, and AI-inference usage limits that were difficult to enforce with simple request counting. By verifying against official sources, we confirm that the implementation details are as described: the BackendTrafficPolicy (or RateLimitFilter) CRD now accepts a cost specification with per-request and per-response cost values, which can be fixed numbers or dynamically fetched from metadata. The default behavior remains one request = one count (to maintain backward compatibility). This cost-based mechanism currently applies to global HTTP rate limits, working in conjunction with Envoy’s global rate limit service (Redis).

For organizations and developers, this means more granular control over how clients consume their APIs. You can now impose limits not just on the number of requests, but on the “cost” of those requests – whether defined by data size, CPU time, tokens generated, or any custom metric you can feed into Envoy’s metadata. Envoy Gateway 1.3.0’s documentation and release notes solidify the accuracy of this feature description, and the translated content above should faithfully reflect the original article’s intent with added clarity and verified technical correctness.

Sources:

Envoy Gateway v1.3.0 Release Notes
Envoy Gateway API Reference – RateLimit cost specification
Envoy Gateway Pull Request #4957 (API changes for cost specifier)
Envoy Gateway Pull Request #5035 (implementation of rate limit cost in translator)
GitHub Issue #4756 – “Usage based Rate Limiting (Counting from response header values)” (Motivation for cost feature)
GitHub Issue #4748 – “Generative AI support” (Related to dynamic cost usage for AI scenarios)

Is Manual Memory Management Really Necessary? A Look at Zig and Rust

reoring — Fri, 17 Jan 2025 08:28:23 +0000

In recent years, two languages have gained traction in the realm of systems programming: Zig and Rust. Both languages are often mentioned as potential alternatives to C/C++ in low-level development, yet they have surprisingly different design philosophies.

In this post, we'll compare Zig and Rust through the lens of "manual memory management." I should note that while I have plenty of experience with Rust, I'm relatively new to Zig. If you spot any mistakes or misinterpretations on the Zig side, please let me know—I'm still learning!

How Rust Guarantees Memory Safety

Let's start with Rust. Rust uses the concepts of ownership and borrowing, which the compiler enforces at compile time through what's commonly known as the borrow checker. Thanks to this mechanism, Rust can eliminate memory errors like double frees and dangling pointers before the program ever runs.

The Upsides

High level of safety: Memory bugs such as double frees or dangling pointers are caught at compile time
Concurrent contexts: Rust significantly reduces risks like data races or memory corruption in multithreaded scenarios
Great for large-scale development: Because the compiler ensures memory correctness across the entire codebase, even big projects benefit from a reduced risk of subtle memory issues

The Downsides

Learning curve: Concepts like ownership and lifetimes can be tricky at first
Not everything is "managed automatically": Sometimes, you really do need fine-grained control. Rust has a mechanism for that, but it comes with its own complexity

Rust's unsafe Blocks

For use cases where Rust's safe abstractions aren't quite enough, there's unsafe. This special block lets you perform low-level operations that bypass the borrow checker's usual rules. Common examples include:

Directly accessing physical addresses in OS kernels or drivers
Interfacing with C/C++ libraries, manipulating raw pointers
Controlling data structure layouts and alignment precisely

Of course, "unsafe" comes with a warning: if you misuse pointers or free something twice, the compiler won't stop you anymore. That's why Rust encourages limiting your unsafe blocks to the smallest possible areas—usually just where you do hardware or specialized memory operations. This way, the rest of your code can remain safe and take advantage of Rust's protections.

Zig's Philosophy: "Do Everything by Hand"—But with Optional Safety Checks

Now, on to Zig. Please remember I'm still new to Zig, so if there's a point that needs correction, I'd really appreciate any feedback!

Unlike Rust, Zig doesn't have an ownership system or garbage collection. Instead, you manage allocations and deallocations manually, reminiscent of C. For example, to dynamically allocate an array in Zig, you explicitly call the allocator and then free the memory yourself:

const std = @import("std");

pub fn main() !void {
    var allocator = std.heap.page_allocator;
    const array = try allocator.alloc(u8, 100);
    // Use the array here
    allocator.free(array);
}

That's conceptually similar to malloc/free in C. If you mess up and cause a memory overrun, Zig won't automatically protect you. However, Zig does include safety modes—both at compile time and runtime. In runtime safety mode, the language will perform checks (such as array bounds checking) and trigger a panic if something goes out of range. These safety behaviors depend on default settings and build options, so it's not correct to say "Zig has absolutely zero checks." Instead, it's more accurate to say that Zig is flexible, giving you the option to disable or enable these checks depending on your performance and safety requirements.

The Upsides

No runtime or GC (by default): This can make binaries smaller and potentially more predictable
Powerful compile-time execution (CTFE): Metaprogramming in Zig can be done at compile time in a relatively straightforward manner
Easy cross-compilation: The Zig compiler itself supports many platforms, making cross-compilation simpler
Optional safety checks: Zig can detect out-of-bounds array access in runtime safety mode, helping catch certain errors

The Downsides

Manual memory management: If you disable safety checks, memory errors will be your responsibility alone
No built-in ownership or lifetime checks: The risk of memory bugs can be higher, especially in large codebases that aren't leveraging Zig's optional safety features

When Do You Actually Need Manual Memory Management?

1. Real-Time and High-Performance Scenarios

In game engines or embedded systems, for instance, a garbage collector might cause unpredictable pauses (frame drops or missed deadlines). Manually managing memory lets you control exactly when and where allocations or deallocations happen, helping avoid unexpected performance dips.

2. Operating Systems, Kernels, and Drivers

When working at extremely low level (like OS kernels), you can't rely on a language runtime. There's no GC and often no standard library in the early stages. Directly interfacing with physical addresses is common, so manual memory management is a necessity.

3. Memory Layout Optimization

In large simulations or performance-critical applications, laying out data to optimize cache usage can be crucial. Sometimes you need to place data structures in a specific way to avoid false sharing or improve cache locality, and that often entails custom allocators or memory pools.

A Quick Look at False Sharing

While discussing performance, let's highlight false sharing—a subtle pitfall in multithreaded programming. False sharing occurs when different threads update different variables that happen to lie on the same CPU cache line.

For example, consider this Rust code:

struct Counter {
    value1: u64, // accessed by thread 1
    value2: u64, // accessed by thread 2
}

value1 and value2 look independent, but if they share the same 64-byte cache line, updating one can invalidate the line for the other, triggering expensive cache coherence operations.

Why Performance Suffers

Thread 1 updates value1, invalidating the cache line on other cores
Thread 2 tries to access value2, causing a cache line reload
This invalidation-reload cycle repeats, degrading performance

Potential Solutions

In Rust, you can address this by adding alignment or padding:

use std::sync::atomic::AtomicU64;

#[repr(align(128))]
struct Counter {
    value1: AtomicU64,
    _pad: [u8; 120], // extra padding
    value2: AtomicU64,
}

You can do something similar in Zig, too:

const std = @import("std");

const Counter = struct {
    value1: std.atomic.Value(u64),
    _pad: [120]u8 = undefined,
    value2: std.atomic.Value(u64),

    pub fn init() Counter {
        return .{
            .value1 = std.atomic.Value(u64).init(0),
            .value2 = std.atomic.Value(u64).init(0),
        };
    }
};

Remember: Only optimize after confirming (through profiling) that false sharing is a real bottleneck. Over-optimizing prematurely can harm code readability without tangible performance gains.

Rust's unsafe vs. Zig: Which Is Better?

Some say Rust's unsafe blocks are comparable to Zig's "always-manual" approach, and in a sense, that's correct. In an unsafe block, you can do everything C or Zig can do with pointers. However, there's a key difference:

Rust: You're mostly protected by the compiler's borrowing and lifetime rules. You only step outside that safety net in unsafe blocks, ideally isolating low-level code to specific places
Zig: The entire language philosophy centers around manual control, but with optional safety checks you can turn on or off. While it's simpler at heart, mistakes can be punishing if you don't use those checks properly

Conclusion: Pick What Suits Your Project

Zig is an excellent choice if you want maximum low-level control, minimal runtime overhead, and a straightforward cross-compilation experience. It's also appealing if you prefer a C-like style and don't mind being fully responsible for memory safety—though safety modes can help catch some errors if you choose to enable them
Personal note: Even though I'm new to Zig, I love how clean and simple its core design is
Rust is fantastic for team projects, larger codebases, or situations where the compiler's ownership system can significantly reduce bugs. When you do need manual control for performance or hardware-level tasks, you can isolate those parts in unsafe blocks. It strikes a balance between performance and safety that many find invaluable

Ultimately, there's no one-size-fits-all. Think about your performance requirements, development environment, and team preferences, then choose the language that best meets those needs. If you're working in a domain that demands tight memory control with minimal runtime footprint, Zig might be your best bet. If safety and productivity are paramount, Rust is an excellent choice.

References & Further Reading

The Zig Programming Language
Rust Book
Zig vs. Rust discussions on official forums and community resources

I hope you found this post helpful! If you have any comments, corrections (especially regarding Zig, since I'm still learning), or personal experiences to share, please leave a comment. Your input is always appreciated!

Can't DNAT After DNAT?

reoring — Wed, 15 Jan 2025 08:14:00 +0000

When dealing with Kubernetes and network configurations, we often encounter DNAT (Destination NAT) mechanisms. Whether it's "rewriting Service Virtual IP to Pod IP" or "forwarding external access to specific nodes or Pods" - DNAT is working behind the scenes in these cases.

However, this raises an interesting question: "Can you rewrite a DNAT-ed destination with another DNAT?" Many people encounter this question during various cases and troubleshooting. To cut to the chase, the answer is generally "no" or "it's difficult." This article explains the mechanism and reasons behind this.

1. What is DNAT?

Let's first review what DNAT is. DNAT (Destination NAT) is a technology that rewrites destination information (IP address and port) of incoming packets. In Linux's netfilter/iptables, it's primarily configured in the PREROUTING chain of the nat table.

Example: "Forward external traffic coming to Node's port 80 to Pod's port 8080"
- In PREROUTING, specify -j DNAT --to-destination :8080 for packets matching -d --dport 80

In Kubernetes, this DNAT is automatically configured on each node, implementing a mechanism to distribute Service (ClusterIP) destinations to Pod IPs.

2. Why Can't You "DNAT After DNAT"?

In conclusion, double DNAT is basically impossible with iptables. The main reasons are:

iptables NAT assumes one transformation per connection
- iptables NAT works with connection tracking (conntrack) to perform NAT on a per-packet-flow basis. Once DNAT is applied to the initial packet, subsequent packets in the same connection are treated as "already transformed" and generally won't undergo DNAT again.
Routing occurs after DNAT, so packets don't return to PREROUTING
- DNAT is typically applied in PREROUTING. After the destination IP is rewritten here, the Linux kernel's routing table determines the packet's destination. Packets flowing in the same direction won't re-enter PREROUTING, making it impossible to "DNAT after DNAT."
Double DNAT is possible with special configurations, but it's not common
- Theoretically, you can perform another DNAT after packets are delivered to a different host. However, multi-stage configurations like "PREROUTING → DNAT → DNAT again on the same host" aren't intended in standard iptables.

3. DNAT in Kubernetes Services

In Kubernetes, packets destined for Services (ClusterIP) are forwarded by DNATing them to Pod IPs. Specifically, the following steps are executed in KUBE-SERVICES and KUBE-POSTROUTING chains created on the Node:

DNAT in PREROUTING
- Packets addressed to Service ClusterIP are rewritten to the selected Pod IP:Port.
Routing table determines Pod destination
- After rewriting, routing occurs either internally to the Node where the Pod is running or to another Node.

Naturally, once the destination is determined by one DNAT, further DNAT operations aren't expected. This directly relates to why "DNAT after DNAT" is difficult.

4. Common Questions and Troubleshooting

4.1 "External Load Balancer → NodePort → Pod, More DNAT?"

In Kubernetes, when external load balancers route traffic to NodePort and then to Pods, DNAT occurs. However, this is essentially one flow of "external LB → NodePort → DNAT → Pod" tracked as a single continuous connection, not double DNAT.

4.2 "Is DNAT Performed Again When Forwarding from One Service to Another?"

When requesting from Service A to Service B on the same Node, DNAT might occur again as the packets are treated as a separate connection. However, this is just DNATing "as a completely separate connection."

5. Hints for When You Still Need Double DNAT

If you need to implement "DNAT after DNAT" for some requirements, consider these approaches:

Insert a different host or device
- After DNAT is applied, you can apply new DNAT when packets are delivered to a different host.
Separate NAT in OUTPUT or POSTROUTING chains
- You might create complex configurations like DNAT in one direction and SNAT in the return direction. However, this is hard to manage and not recommended.

Generally, the disadvantages of network complexity outweigh the benefits, so it's recommended to avoid "multi-stage DNAT" during design if possible.

Summary

Reasons why you can't DNAT after DNAT
1. iptables NAT works with connection tracking (conntrack) and "won't transform an already transformed connection"
2. After DNAT, routing occurs, so the same packet doesn't return to PREROUTING
Kubernetes Services and NodePort work the same way
- Both finish after one DNAT rewrites the destination to Pod IP
If double DNAT is absolutely necessary
- Special configurations like routing through different hosts are needed but aren't common

Understanding iptables' design principle of "one NAT per connection" helps with troubleshooting Kubernetes and network issues. When designing systems, it's recommended to keep flows simple to avoid complications from double DNAT.

Simplifying RBAC Extension in Kubernetes: Leveraging Aggregate ClusterRoles

reoring — Mon, 09 Dec 2024 07:24:49 +0000

When operating Kubernetes, Role-Based Access Control (RBAC) is an unavoidable aspect of cluster management. While ClusterRoles allow you to assign resource operation permissions to users and ServiceAccounts across the entire cluster, manually updating existing Roles and ClusterRoles becomes increasingly challenging as clusters evolve and new APIs and Custom Resource Definitions (CRDs) are added.

This is where Aggregate ClusterRoles come into play. In this article, we'll explore the benefits and basic concepts of Aggregate ClusterRoles, along with practical examples using sample YAML configurations.

What are Aggregate ClusterRoles?

Aggregate ClusterRoles, introduced in Kubernetes 1.9, are a mechanism that automatically aggregates (combines) one ClusterRole into another through specified annotations.

This feature offers several benefits:

Dynamic Permission Extension: When adding new Custom Resource Definitions (CRDs), you can define operation permissions as independent ClusterRoles and simply add the aggregate-to-* annotation to automatically extend standard roles like admin and view.
Reduced Operational Overhead: You no longer need to manually edit existing ClusterRoles each time you introduce multiple CRDs, reducing the need for RBAC policy redefinition.
Improved Readability and Scalability: While roles can be managed in smaller, discrete units, they can be automatically aggregated, making permission management more scalable.

How It Works

Kubernetes automatically integrates permissions when a ClusterRole is given specific annotations (rbac.authorization.k8s.io/aggregate-to-: "true") into the existing . Common target roles include:

aggregate-to-admin: "true" → Adds to existing admin role
aggregate-to-edit: "true" → Adds to existing edit role
aggregate-to-view: "true" → Adds to existing view role

Using these annotations, if you want to add operation permissions for a newly added CRD to the "edit" role, you simply need to create a ClusterRole with aggregate-to-edit: "true".

Sample Configuration

Let's consider a scenario where we've introduced a CRD called MyCustomResource, which has mycustomresources resources in the mygroup.example.com/v1 group.

1. CRD Definition (Example)

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: mycustomresources.mygroup.example.com
spec:
  group: mygroup.example.com
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
  scope: Namespaced
  names:
    plural: mycustomresources
    singular: mycustomresource
    kind: MyCustomResource
    shortNames:
      - mcr

Once this CRD is deployed, a new resource called mycustomresources is added to the Kubernetes API.

2. Creating ClusterRole for the CRD

We'll define a new ClusterRole that can operate on mycustomresources. By adding aggregate-to-edit: "true", users with the edit role automatically gain edit permissions for this new resource.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: mycustomresource-edit
  labels:
    rbac.authorization.k8s.io/aggregate-to-edit: "true"
rules:
  - apiGroups: ["mygroup.example.com"]
    resources: ["mycustomresources"]
    verbs: ["create", "update", "patch", "delete", "get", "list", "watch"]

The key point is the rbac.authorization.k8s.io/aggregate-to-edit: "true" label under metadata.labels (Note: While historically referred to as an annotation, it's actually implemented as a label in Kubernetes. There may be some terminology variations in Kubernetes documentation across versions, but labels are currently used for specification). This label ensures that this mycustomresource-edit ClusterRole is automatically aggregated into the edit role.

3. Verifying the Effect

When you check the edit role using kubectl get clusterroles edit -o yaml, you'll see that rules for mycustomresources have been automatically added. Now, users with the existing edit role can operate on mycustomresources without any additional configuration.

Operational Tips

Extending Beyond Standard Roles: While aggregation to admin, edit, and view roles is common by default, you can create your own Aggregate ClusterRole mechanism. In such cases, create your own base role and set up similar labels.
Permission Organization: In environments with growing numbers of CRDs, create multiple fine-grained ClusterRoles and use the Aggregate ClusterRole feature to combine them systematically. This allows explicit management of "which CRDs are aggregated into which roles."

Conclusion

By utilizing Aggregate ClusterRoles, Kubernetes RBAC management becomes more flexible and extensible. When granting permissions for new resources and add-ons, this mechanism automates and simplifies what traditionally required manual editing of existing roles.

For those planning to actively use custom resources or experiment with various add-ons, implementing Aggregate ClusterRoles can significantly improve the operational efficiency of your Kubernetes environment.

How to Process Strings in Kubernetes Secrets Using Kyverno

reoring — Mon, 09 Dec 2024 06:56:05 +0000

I'll create an English version of the article while maintaining the same technical depth and structure:

How to Process Secret Strings with Kyverno (Advanced Guide)

When operating Kubernetes clusters, you often need to store and reuse Secret values after applying specific string processing. For example, you might want to convert strings in Secrets injected by external systems to uppercase or replace certain patterns. This article explains two approaches to processing strings in Secrets at application time using Kyverno, a Kubernetes-native policy engine.

What is Kyverno?

Kyverno is a powerful policy engine that allows you to control and apply policies to Kubernetes resources using YAML-based definitions. Operating as an admission controller, Kyverno provides the following features:

Validate: Verify conditions before resource application
Mutate: Modify resources by applying patches during resource creation
Generate: Automatically create new resources triggered by other resource events

String Processing Scenario

Consider a scenario where you want to take a Secret named "source-secret" when it's created, convert its data.username to uppercase, and reflect it in a new Secret called "target-secret". The initial approach might be to use Kyverno's generate feature to automatically create target-secret when source-secret is created.

However, Kyverno's generate feature has a constraint: "it may not always be possible to generate resources of the same Kind simultaneously". This can be particularly challenging when generating Secrets, as certain patterns might be difficult to implement due to the generate mechanism's limitations.

Constraints of Generate Rules

While Kyverno's generate feature works well for scenarios like creating a ConfigMap when another ConfigMap is created, attempting to "generate a Secret when a Secret arrives" will result in an error stating "generation kind and match resource kind should not be the same". This restriction appears to be in place to prevent an infinite loop where generated resources would trigger the generation of additional resources of the same kind.

In such situations, using mutate offers an alternative approach.

The rest of the article remains the same, as this change only affects this specific section while maintaining consistency with the overall flow and technical accuracy of the content.

Implementing Data Transformation Within the Same Secret Using Mutate

Using mutate rules, you can modify the Secret resource itself just before source-secret is applied. This means you can uppercase data.username in the same Secret resource before storing it in the cluster, avoiding the same-Kind generation issue.

String Processing Functions Available in Kyverno

Kyverno provides JMESPath extension functions with many useful string manipulation functions:

to_upper(string): Convert string to uppercase
to_lower(string): Convert string to lowercase
replace(old, new, string), replaceAll(old, new, string): String replacement
regex_replaceAll(pattern, replace, string): Regular expression replacement
base64_decode(string) / base64_encode(string): Base64 encoding/decoding
Trimming functions (trim, trim_prefix, trim_suffix) and more

Since Secret data values are Base64 encoded, processing data.username requires first using base64_decode() to restore the original string, then applying to_upper() for uppercase conversion, and finally using base64_encode() to re-encode the result.

Example of a Mutate Rule

Here's a policy example that uppercases data.username when source-secret is created in the default namespace:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: mutate-secret-data
spec:
  rules:
    - name: mutate-secret-username
      match:
        resources:
          kinds:
            - Secret
          namespaces:
            - default
          names:
            - source-secret
      mutate:
        patchStrategicMerge:
          apiVersion: v1
          kind: Secret
          metadata:
            name: "{{ request.object.metadata.name }}"
            namespace: "{{ request.object.metadata.namespace }}"
          data:
            username: "{{ base64_encode(to_upper(base64_decode(request.object.data.username))) }}"

Verifying Policy Operation

Apply the above policy using kubectl apply -f
Apply the following Secret using kubectl apply -f:

apiVersion: v1
kind: Secret
metadata:
  name: source-secret
  namespace: default
type: Opaque
data:
  username: dXNlck5hbWU=  # Base64 encoded value of "userName"

After applying, check the contents using kubectl get secret source-secret -o yaml, and you'll see that the username value has been replaced with the Base64 encoded value of the uppercase "USERNAME".

This confirms that string transformation occurred within the same Secret during admission.

Summary

While we initially considered using the generate feature to "generate a new Secret with string processing triggered by another Secret," Kyverno's generate feature has limitations with same-Kind generation. Instead, when data transformation is needed within the same resource, using mutate allows us to transform the resource during admission.

With mutate, you can process data with any string manipulation just before the Secret is stored, and by combining Kyverno extension functions like base64_decode(), to_upper(), and base64_encode(), you can achieve complex string transformations.

Kyverno is a valuable tool for controlling and extending Kubernetes resources through policies. Consider the appropriate use of generate and mutate while managing Secrets, ConfigMaps, and other Kubernetes resources.

Making Git Workflows Incredibly Convenient with GitButler

reoring — Wed, 14 Feb 2024 10:43:41 +0000

tldr

The UI of GitButler looks like this.

What's good about it?

A key feature is the ability to work with git flexibly using virtual branches without creating physical branches. For example, you can drag changes from the current modification to multiple virtual branches to organize the changes.

It also automatically suggests commit messages.

Even the names of the virtual branches are automatically decided.

Why GitButler?

The Need for a New Git Client

In modern software development, GitHub has transformed collaboration among developers. However, using GitHub, GitLab, or Bitbucket still requires the use of cumbersome and error-prone command-line tools, which are not optimized for the workflows and processes used by today's developers.

The Innovation of GitButler

GitButler rethinks the entire software development process from coding in the editor to sharing code on GitHub. Why should we be troubled by unclear commands and complicated procedures? Source code management (SCM) should record everything for you, provide more meaningful commit messages, and make it easier to provide context about the code your team has written. It should also allow seamless transition of work between different devices.

What GitButler Aims For

GitButler is not just a new kind of Git client, but proposes an entirely new way of thinking about code management in software development. It acts as a "code concierge" to assist developers at every step of the software development process. The goal is to provide the necessary support and context for each line of code without developers losing a moment of their work.

Summary

Source code management can transcend the concepts of 40 years ago to become smarter and more advanced. GitButler aims not just to offer a new tool but to transform the very method of source code management. This allows developers to work more efficiently and, above all, more creatively, without being troubled by the complexities of traditional management tools.

Key Features of GitButler

1. Concurrent Branch Management:

GitButler introduces an innovative approach to handling multiple branches, allowing developers to work on several branches simultaneously, a function not directly supported by Git itself. This feature significantly reduces the complexity and time required for switching and managing multiple branches.

2. Virtual Branches

GitButler's Virtual Branch feature enables working on multiple branches simultaneously, committing and stashing each independently. Developers can handle multiple virtual branches at once, dragging each of three different changes in a file to different virtual branch lanes, committing and pushing them independently. Unlike Git's normal branch operations, virtual branches are maintained in vertical lanes, allowing each file or difference to be dragged between lanes like a kanban board. For more details, refer to GitButler's documentation.

3. Automatic Versioning and Commit Drafting:

GitButler not only creates commits but drafts commit messages while directly integrating automatic versioning into the Git working directory. This feature allows developers to focus more on coding, knowing that versioning is handled automatically.

4. Integration with Existing Workflows:

Designed to seamlessly integrate with existing Git workflows, GitButler enhances, rather than interrupts, current practices. Features like bookmarking crucial moments in the development timeline add layers of convenience and efficiency.

5. Butler Flow:

A lightweight, branch-based workflow facilitated by GitButler's virtual branch feature. It ensures that virtual branches continue to be applied locally until they are merged upstream, reducing confusion and overhead.

Why Choose GitButler?

GitButler is more than just a tool. It was created to address the subtle problems and challenges daily developers face. By managing source code more intelligently and providing support for every line of code, GitButler aims to be a step ahead in the software development process.

Getting Started with GitButler

Starting with GitButler is easy. The platform is available for download, and an open beta version lets you glimpse some of its features. The development team also fosters a community on Discord, where users can join, share experiences, and receive support.

Deploy Prometheus + Grafana to Kubernetes by Helm 3

reoring — Thu, 09 Jan 2020 08:09:57 +0000

Make your own Prometheus + Grafana in kubernetes cluster and start monitoring it in some minutes😌

Add repository of stable charts

Helm3 has not default repository.

helm repo add stable https://kubernetes-charts.storage.googleapis.com

Install prometheus-operator

helm install my-prometheus-operator stable/prometheus-operator

Show pods

kubectl --namespace default get pods -l "release=my-prometheus-operator"

Show Grafana UI

kubectl port-forward $(kubectl get pods --selector=app=grafana --output=jsonpath="{.items..metadata.name}") 3000

Open it http://localhost:3000/

Use Storybook with Vuejs

reoring — Wed, 15 Nov 2017 05:52:11 +0000

Storybook Since 3.2, support for Vuejs has been added, so I will try using it at once.

Storybook is a tool that makes it easy to create catalogs of components, cataloging self-made components in the project and how to use it.

Installation of storybook / cli

npm i - g @ storybook / cli

Prepared by existing Vuejs project

cd Directory where vuejs project is located

Installing the storybook

getstorybook

Start storybook server

yarn run storybook

In this state, opening http://localhost:6006/ opens the default setting screen.

Add component

To add a component to a storybook, add a definition to index.js in thestories directory created with getstorybook.

You can change this stories directory by editing.storybook / config.js.

Reference material

Generate CakePHP Migration class from existing class definition

reoring — Wed, 15 Nov 2017 05:32:53 +0000

Generate CakePHP Migration class from existing class definition
We created a generator that generates CakePHP 3 migration from existing class definition.

I think that I can use it when making a class with many properties and then making a draft for a migration class.

Generator class


<?php

class MigratinoClassGenerator
{
  public function generate(string $className)
  {
    $ref = new ReflectionClass($className);
    $properties = $ref->getProperties();

    $migrationClass = new MigrationClass();

    foreach ($properties as $property) {
      $migrationClass->addColumn($property->getName());
    }

    return $migrationClass->generate($ref->getName());
  }
}

class MigrationClass
{
  private $columns = [];

  public function addColumn(string $name, $type = 'string')
  {
    $this->columns[] = [
      'name' => $name,
      'type' => $type,
    ];
  }

  public function generate(string $className)
  {
    $buf = [];

    $buf[] = '<?php';
    $buf[] = '';
    $buf[] = 'use Migrations\AbstractMigration;';
    $buf[] = '';
    $buf[] = 'class CreateProduct extends AbstractMigration';
    $buf[] = '{';
    $buf[] = '    public function up()';
    $buf[] = '    {';

    $tableName = strtolower($className);
    $buf[] = sprintf('        $this->table(\'%s\')', $tableName);

    foreach ($this->columns as $column) {
      $buf[] = $this->generateColumn($column);
    }

    $buf[] = '            ->create();';
    $buf[] = '    }';
    $buf[] = '';
    $buf[] = '    public function down()';
    $buf[] = '    {';
    $buf[] = '        $this->dropTable(\''. $tableName . '\');';
    $buf[] = '    }';
    $buf[] = '}';

    return join("\n", $buf);
  }

  private function generateColumn(array $column)
  {
    $template = <<<TPL
            ->addColumn('%s', 'string', [
                'default' => null,
                'limit'   => null,
                'null'    => false,
            ])
TPL;

    return sprintf($template, $this->camelCase($column['name']));
  }

  private function camelCase(string $input) {
    preg_match_all('!([A-Z][A-Z0-9]*(?=$|[A-Z][a-z0-9])|[A-Za-z][a-z0-9]+)!',
                   $input,
                   $matches);

    $ret = $matches[0];
    foreach ($ret as &$match) {
      $match = $match == strtoupper($match) ? strtolower($match) : lcfirst($match);
    }

    return implode('_', $ret);
  }
}

Execution method

<?php

$g = new MigratinoClassGenerator();
echo $g->generate('Product'); // クラス名を指定する

Output result

<?php

use Migrations\AbstractMigration;

class CreateProduct extends AbstractMigration
{
    public function up()
    {
        $this->table('product')
            ->addColumn('id', 'string', [
                'default' => null,
                'limit'   => null,
                'null'    => false,
            ])
            ->addColumn('code', 'string', [
                'default' => null,
                'limit'   => null,
                'null'    => false,
            ])
            ->create();
    }

    public function down()
    {
        $this->dropTable('product');
    }
}