DEV Community

Ricardo Katz
Ricardo Katz

Posted on

Istio Pilot - Debugging and Development workflow

#istio #kubernetes #gatewayapi #go

As I've been touching more on Istio Pilot codebase recently, I've found very difficult to debug it without a DLP/debugger connected to it.

Usually I would say "just add some fmt.Prints and spew.Sdump" around the code, but given I need to understand step by step how Istio converts some Gateway API resources into its own model, attaching a debugger became a required step.

Additionally, the "compile/push/debug/repeat" loop became essential, so I kind of did my own workflow, that works for me.

All of my development workflow is done using my Gateway API fast provisioning script on kind.

Note: This is my development workflow, and may not work for you, so take with with a grain of salt ;)

Compile/Push repeat

This is my copy/paste/be happy script:

export TAG=istio-$(date +%s)
export HUB=localdev.tld/istio
export DEBUG=1
make docker.pilot && kind load docker-image ${HUB}/pilot:${TAG} &&  kubectl set image -n istio-system deployments/istiod discovery=${HUB}/pilot:${TAG}
Enter fullscreen mode Exit fullscreen mode

Remote debugging

First, why remote debugging? I wanted to check exactly what's going on my live code. I didn't wanted to run it locally.

I will update this article with a better approach, but for now:

  • Create and apply a patch on Dockerfile.pilot AND REMEMBER TO NOT COMMIT IT!

This patch gets delve from the Go official images, and copy to the final Istio image. It also changes the entrypoint to be delve starting pilot.

curl -o dockerfile.patch https://gist.githubusercontent.com/rikatz/f5a07945038f705c9d10625eceb99932/raw/c56c9f0fca2545fffbe87d4de55ccb5d474f173b/dockerfile.patch
git apply dockerfile.patch
Enter fullscreen mode Exit fullscreen mode

Note: right now bumping the used go image should be done manually .

  • Change your Pilot deployment to allow writing to filesystem and to add SYS_PTRACE capability

These changes are required for Delve to work properly

 kubectl patch deployment istiod -n istio-system --type='json' -p='[
    {"op": "replace", "path": "/spec/template/spec/containers/0/securityContext/allowPrivilegeEscalation", "value": true},
    {"op": "replace", "path": "/spec/template/spec/containers/0/securityContext/readOnlyRootFilesystem", "value": false},
    {"op": "add", "path": "/spec/template/spec/containers/0/securityContext/capabilities", "value": {"add": ["SYS_PTRACE"]}}
]'
Enter fullscreen mode Exit fullscreen mode
  • Do a port-forward to delve port

This is the port where you will connect your remote-debugger later (on localhost:40000):

kubectl port-forward -n istio-system deploy/istiod 40000
Enter fullscreen mode Exit fullscreen mode
  • Attach to remote process Here is my vscode file: 
{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Remote Debug Pilot in K8s",
            "type": "go",
            "request": "attach",
            "mode": "remote",
            "remotePath": "",
            "port": 40000,
            "host": "localhost",
            "showLog": true,
            "trace": "verbose",
            "substitutePath": [
                {
                    "from": "${workspaceFolder}",
                    "to": "/work"
                }
            ]
        }
    ]
}
Enter fullscreen mode Exit fullscreen mode

Running integration tests

Finally, this is my last step. Once I am writing some integration tests, I may want to run them locally before posting my PR. Here is usually how I do:

INTEGRATION_TEST_FLAGS="-run ^TestGatewayAPIResourcesOnly$"   ./prow/integ-suite-kind.sh --topology MULTICLUSTER  --skip-cleanup   test.integration.pilot.resourcefilter.kube
Enter fullscreen mode Exit fullscreen mode

This one runs a single integration test called TestGatewayAPIResourcesOnly. The topology flag is optional and can be used if a specific topology integration test is failing.

The final name test.integration.pilot.resourcefilter.kube can be translated to "tests/integration/pilot/somesubfolder".

So, in case you want to run a test that is directly on "tests/integration/pilot" you can replace it for test.integration.pilot.kube.

Top comments (0)