Setting Up AgentGateway on SAP BTP Kyma

A practical guide to deploying a production-grade MCP gateway — JWT authentication, CEL tool authorization, Istio mTLS, live backend registration, and Jaeger tracing — with the lessons that cost us the most time.

---

What this is

AgentGateway is an open-source, AI-first data plane that provides connectivity for agents, tools, and models in any environment. It handles agent-to-agent (A2A) communication, Model Context Protocol (MCP) tool serving, REST APIs exposed as agent-native tools, and AI routing to cloud and local LLMs (including the Gateway API Inference Extension). Across all of these it adds what the underlying protocols deliberately leave out: identity verification, fine-grained authorization, observability, and centralized routing.

This guide focuses on one of those use cases: running AgentGateway as an MCP gateway — in front of a fleet of MCP servers — and making it a genuinely enforced control point rather than an advisory one. The same infrastructure (Istio mTLS, XSUAA JWT auth, CEL authorization) applies to the other backend types, but everything below is written around MCP.

On SAP BTP, the natural runtime for this is Kyma — SAP's managed Kubernetes + Istio environment. Kyma gives you something most gateway setups don't: the ability to make the gateway genuinely enforced rather than advisory. Backend MCP servers get no public route. They exist only as ClusterIP Services. There is no hostname to call from outside the cluster, and inside the cluster an AuthorizationPolicy means only the gateway's workload identity can reach them. On Cloud Foundry, where every inter-app call goes over a public internet URL, we proved you could walk around the gateway entirely with a direct unauthenticated POST. On Kyma, there is no back door to walk around.

That's the architectural win this setup is designed to deliver. Here's how to build it.

---

What you'll end up with

A running deployment where:

• MCP backends are ClusterIP-only — no external hostname exists; bypass is structurally impossible
• The AgentGateway proxy validates XSUAA JWT tokens on every inbound request
• A CEL expression on each backend strips tools from tools/list unless the token carries the right scope
• Adding a new MCP server is kubectl apply of two manifests — no restart, no UI interaction
• All intra-cluster traffic is Istio mTLS (STRICT namespace-wide) with an AuthorizationPolicy that whitelists only the proxy's SPIFFE identity to reach backends
• Tool calls appear in structured access logs and as spans in Jaeger

                 ┌─────────────────────────────────────────────────────────┐
   MCP client    │  XSUAA (BTP service)                                     │
   (curl /       │  • issues client_credentials token                       │
   Inspector) ──▶│  • serves JWKS at /token_keys (HTTPS)                    │
        │        └─────────────────────────────────────────────────────────┘
        │ 1. get token                            ▲ 3. fetch JWKS (HTTPS)
        │ 2. call /mcp with Bearer                │
        ▼                                         │
 ┌───────────────┐   plaintext    ┌───────────────────────────────┐
 │ K8s Load-     │──────HTTP─────▶│  agentgateway-proxy   (2/2)   │
 │ Balancer Svc  │  (PERMISSIVE   │  agentgateway + Envoy sidecar │
 └───────────────┘   inbound)     │  • validates JWT (XSUAA)      │
                                  │  • CEL tool authorization     │
                                  └──┬───────────┬─────────┬──────┘
              xDS gRPC :9978          │           │ Istio   │ OTLP
              (DNS-verified TLS,      │           │ mTLS    │ gRPC
              sidecar excluded)       │           │ STRICT  │ :4317
                                      ▼           ▼         ▼
                          ┌────────────────┐ ┌──────────┐ ┌───────────┐
                          │ agentgateway   │ │ utility  │ │ jaeger-   │
                          │ control plane  │ │ -mcp 2/2 │ │ collector │
                          │  (1/1, NO      │ ├──────────┤ └─────┬─────┘
                          │   sidecar)     │ │ time-mcp │       │
                          └────────────────┘ │ 2/2      │       ▼
                                              └──────────┘ ┌──────────┐
                          AuthorizationPolicy: backends     │ jaeger   │
                          accept ONLY the proxy's SPIFFE   │ query UI │
                          identity, over mTLS              │(HTTPRoute│
                                                            └──────────┘

Everything except the proxy's LoadBalancer and the Jaeger UI HTTPRoute is ClusterIP. No other external addresses.

Kyma Gateway API note: AgentGateway installs its own GatewayClass named agentgateway. Kyma's own API Gateway module (kyma-system/kyma-gateway) is a separate implementation and is not required here. Both can coexist in the same cluster without conflict — each Gateway resource references its own gatewayClassName.

---

Prerequisites

• Kyma environment enabled on your subaccount; kubeconfig downloaded and kubectl context set
• Kyma modules enabled: Istio, SAP BTP Operator (services.cloud.sap.com/v1). The Kyma API Gateway module is not required.
• helm CLI
• Container registry reachable from the cluster for your MCP backend images. AgentGateway's own images pull from cr.agentgateway.dev — verify it's reachable:

STATUS=$(curl -o /dev/null -s -w "%{http_code}" https://cr.agentgateway.dev/v2/)
[[ "$STATUS" =~ ^(200|401)$ ]] && echo "reachable (HTTP $STATUS)" || echo "UNREACHABLE — check egress allowlist"

A 401 is normal — it's the OCI registry's auth challenge. Both 200 and 401 mean the registry is reachable.

---

How AgentGateway works on Kubernetes

There is no config.yaml and the Admin UI is read-only — it mirrors the control plane's xDS state for debugging. You cannot add backends or toggle policies through the UI. Every change is a kubectl apply. The control plane watches standard Kubernetes and Gateway API resources and pushes configuration to the data plane over xDS:

Resource	Purpose
AgentgatewayBackend (CRD)	MCP target registration — where is the server, what protocol
AgentgatewayPolicy (CRD)	Auth, authz, tracing, access-log configuration
HTTPRoute (Gateway API)	Routing rules — which path goes to which backend
Gateway (Gateway API)	Listener definition — AgentGateway's Deployer auto-creates the proxy Deployment and LoadBalancer Service

"Live-add an MCP server" means: apply a Deployment + Service, then apply an updated AgentgatewayBackend with the new target added. The control plane reconciles over xDS immediately. No restart, no UI interaction.

![AgentGateway Admin UI, Gateway Overview — a read-only banner reads "Configuration is managed by XDS. This view reflects the active runtime dump; editing is disabled." with counts of active Listeners, Routes, and Policies.]

The Admin UI in read-only mode: "Configuration is managed by XDS. This view reflects the active runtime dump; editing is disabled." It surfaces the live Listeners, Routes, and Policies (plus a CEL Playground for testing expressions) — but you can't add or change anything here. Every change is a kubectl apply.

---

Step 1 — Install AgentGateway

Label the namespace before creating any pods, so Istio injects sidecars from the start:

kubectl create namespace agentgateway-system
kubectl label namespace agentgateway-system istio-injection=enabled

Install the CRDs (includes Gateway API CRDs), then the control plane:

helm upgrade -i agentgateway-crds \
  oci://cr.agentgateway.dev/charts/agentgateway-crds \
  --version 0.0.0-latest-dev \
  -n agentgateway-system

helm upgrade -i agentgateway \
  oci://cr.agentgateway.dev/charts/agentgateway \
  --version 0.0.0-latest-dev \
  -n agentgateway-system

Then apply a GatewayClass (name: agentgateway) and a Gateway resource. The Deployer watches the Gateway and auto-creates the proxy Deployment and LoadBalancer Service.

Exit criteria — don't proceed until all three hold:

kubectl get gatewayclass agentgateway         # ACCEPTED: True
kubectl get gateway -n agentgateway-system    # PROGRAMMED: True
kubectl get svc -n agentgateway-system        # LoadBalancer IP/hostname assigned

---

Step 2 — The Istio sidecar split (the most important thing to get right)

This is the architecture decision that everything else depends on. You need three different sidecar configurations for three different workload roles:

Control plane — no sidecar at all (1/1)

AgentGateway's control plane must be sidecar-free. The proxy connects to it over gRPC TLS on port 9978 using standard DNS hostname verification. If Istio injects a sidecar, it intercepts that port and presents a SPIFFE cert (URI SAN only, no DNS SAN). The proxy's TLS client fails the handshake. Annotate the control plane deployment's pod template with sidecar.istio.io/inject: "false" so Istio skips it entirely.

On the proxy side, exclude port 9978 from Envoy's outbound interception:

# AgentgatewayParameters (applied to the agentgateway-proxy gateway)
spec:
  deployment:
    spec:
      template:
        metadata:
          annotations:
            traffic.sidecar.istio.io/excludeOutboundPorts: "9978"

Proxy — sidecar present, inbound PERMISSIVE (2/2)

The proxy needs an Envoy sidecar so it can originate mTLS toward the backends. The sidecar transparently upgrades the proxy's outbound plain HTTP to Istio mutual TLS — no backend TLS config needed anywhere.

The proxy's inbound traffic comes from the external LoadBalancer over plain HTTP (from real MCP clients). STRICT mode would reject this. Scope a workload-level PeerAuthentication: PERMISSIVE to the proxy only:

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: agentgateway-proxy-permissive
  namespace: agentgateway-system
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: agentgateway-proxy
  mtls:
    mode: PERMISSIVE

Backends — sidecar present, STRICT (2/2)

All MCP backend pods get a normal sidecar and stay under the namespace-wide STRICT policy. Their inbound only ever comes from the proxy's Envoy (mTLS) and they should never accept anything else.

The asymmetry is intentional and load-bearing: control plane: 1/1 sidecar-free; proxy: 2/2 with PERMISSIVE inbound; backends: 2/2 with STRICT.

---

Step 3 — Lock down the mesh

Apply a namespace-wide STRICT PeerAuthentication and an AuthorizationPolicy that restricts backend access to the proxy's SPIFFE identity:

# PeerAuthentication — namespace-wide STRICT
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: agentgateway-system
spec:
  mtls:
    mode: STRICT

# AuthorizationPolicy — only the proxy's service account reaches backends
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: backends-gateway-only
  namespace: agentgateway-system
spec:
  selector:
    matchLabels:
      role: mcp-backend     # applied to all backend pods via this label
  action: ALLOW
  rules:
    - from:
        - source:
            principals:
              - "cluster.local/ns/agentgateway-system/sa/agentgateway-proxy"

Every backend pod must carry role: mcp-backend in its template labels for this selector to pick it up.

---

Step 4 — Deploy MCP backends (ClusterIP only)

Deploy each MCP server with a ClusterIP Service — no type: LoadBalancer, no Ingress, no external hostname. The backend literally cannot be reached from outside the cluster.

apiVersion: v1
kind: Service
metadata:
  name: poc-utility-mcp
  namespace: agentgateway-system
spec:
  selector:
    app: poc-utility-mcp
  ports:
    - port: 8080
      targetPort: 8080
  # no type: LoadBalancer — ClusterIP is the default and what we want

Register all MCP targets in a single AgentgatewayBackend under spec.mcp.targets[]:

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
  name: all-mcp
  namespace: agentgateway-system
spec:
  mcp:
    targets:
      - name: utility
        static:
          host: poc-utility-mcp.agentgateway-system.svc.cluster.local
          port: 8080
          protocol: StreamableHTTP
      - name: time
        static:
          host: poc-time-mcp.agentgateway-system.svc.cluster.local
          port: 8080
          protocol: StreamableHTTP

Critical: all targets must be in one AgentgatewayBackend, not separate ones. If you create two AgentgatewayBackend resources with two HTTPRoute resources both matching /mcp, the proxy routes all traffic to only one — the other backend's tools never appear in tools/list. The same problem occurs with multiple backendRefs in a single HTTPRoute rule. One backend CRD, one HTTPRoute. Tool names are automatically prefixed with the target name (utility_uuid_generate, time_time_current) to avoid collisions.

---

Step 5 — XSUAA JWT authentication

Create the XSUAA service instance and binding via BTP Operator. The instance definition needs to grant the custom scope to the app's own OAuth client — otherwise a client_credentials token carries only uaa.resource and every tool gets stripped:

apiVersion: services.cloud.sap.com/v1
kind: ServiceInstance
metadata:
  name: poc-agentgateway-xsuaa
  namespace: agentgateway-system
spec:
  serviceOfferingName: xsuaa
  servicePlanName: application
  parameters:
    xsappname: poc-agentgateway
    tenant-mode: dedicated
    scopes:
      - name: "$XSAPPNAME.mcp.utility"
        description: "Access to utility MCP tools"
    authorities:
      - "$XSAPPNAME.mcp.utility"   # grant to own OAuth client — required

Once the binding is Ready, extract the values for the auth policy:

XSUAA_URL=$(kubectl get secret poc-agentgateway-xsuaa -n agentgateway-system \
  -o jsonpath='{.data.url}' | base64 -d)
CLIENT_ID=$(kubectl get secret poc-agentgateway-xsuaa -n agentgateway-system \
  -o jsonpath='{.data.clientid}' | base64 -d)

Configure JWT authentication on the Gateway:

traffic:
  jwtAuthentication:
    mode: Strict
    providers:
      - issuer: "<XSUAA_URL>/oauth/token"   # note: /oauth/token suffix required
        audiences:
          - "<CLIENT_ID>"                    # the sb-poc-agentgateway!t… value
        jwks:
          remote:
            url: "<XSUAA_URL>/token_keys"   # explicit https:// — use url, not backendRef

Set mode: Permissive initially (returns tools even without a token) to verify the backend wiring, then switch to Strict once routing is confirmed working.

---

Step 6 — CEL tool authorization

Apply an AgentgatewayPolicy targeting your AgentgatewayBackend:

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
  name: mcp-tool-authz
  namespace: agentgateway-system
spec:
  targetRefs:
    - group: agentgateway.dev
      kind: AgentgatewayBackend
      name: all-mcp
  backend:
    mcp:
      authorization:
        action: Allow
        policy:
          matchExpressions:
            - 'jwt.scope.exists(s, s.endsWith(".mcp.utility"))'

The endsWith pattern is deliberate: XSUAA prefixes scopes with the full application ID including a subaccount tenant index — e.g. poc-agentgateway!t123456.mcp.utility rather than the bare poc-agentgateway.mcp.utility. Matching literally on the full name will break across subaccounts. Matching by suffix works everywhere.

---

Step 7 — Access logs and Jaeger tracing

Enable structured access logging for MCP calls:

spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: agentgateway-proxy
  frontend:
    accessLog:
      # Only log the interesting lines: actual tool calls, plus non-MCP requests.
      filter: 'mcp.methodName == "tools/call" || !has(mcp.methodName)'
      attributes:
        add:
          - name: tool_args
            expression: mcp.tool.arguments
          - name: tool_result
            expression: mcp.tool.result
          - name: tool_error
            expression: mcp.tool.error

For Jaeger, deploy it as a single Deployment with a multi-port ClusterIP Service (ports 4317 for OTLP gRPC, 16686 for the query UI). Then configure the tracing policy using url — not backendRef:

spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: agentgateway-proxy
  frontend:
    tracing:
      url: "http://jaeger-collector.agentgateway-system.svc.cluster.local:4317"
      protocol: GRPC
      randomSampling: "true"

The reason url is required is explained in the lessons section. Point the Jaeger UI to an HTTPRoute at a path like /jaeger with a URL-rewrite filter.

![Jaeger UI showing 5 traces for the agentgateway service — tools/call, tools/list, initialize, and GET /mcp/* — each with span count, duration, and timestamp.]

AgentGateway emits a named span per MCP method. Every tools/call, tools/list, and initialize appears as a distinct trace. Click into any trace to see the full span waterfall with `gen_ai.` attributes.*

---

Live backend registration

Once everything is running, adding a new MCP server is two kubectl apply calls:

# Apply the new server's Deployment + ClusterIP Service
kubectl apply -f k8s/70-time-mcp.yaml

# Apply the updated all-mcp backend with the new target added
kubectl apply -f k8s/71-time-backend.yaml

The control plane sees the updated AgentgatewayBackend CRD and pushes the new xDS configuration to the proxy over the 9978 channel. The new server's tools appear in tools/list within seconds. Nothing restarts.

---

Isolation proof

The single most compelling moment in the demo is confirming the back door genuinely does not exist. From inside the cluster, with the wrong identity:

kubectl run probe --image=curlimages/curl --rm -it -n agentgateway-system -- \
  curl -s http://poc-utility-mcp.agentgateway-system.svc.cluster.local:8080/mcp
# → RBAC: access denied

![Terminal output showing RBAC: access denied when curling the backend ClusterIP directly from inside the cluster with the wrong workload identity.]

Correct in-cluster DNS name, correct port — still RBAC: access denied. The AuthorizationPolicy rejects any caller that isn't the proxy's SPIFFE identity.

From outside the cluster there is no poc-utility-mcp.<domain> to hit. That's the structural guarantee CF couldn't provide.

---

Lessons learned

These are the issues that cost real time during the initial setup. Each one has a non-obvious root cause. Reading this section before you start will save you hours.

1. The proxy won't go Ready — xDS TLS fails against its own control plane

Symptom: the proxy pod starts but never reaches the Ready state. Logs show UnknownIssuer or certificate not valid for name on the xDS gRPC channel.

Root cause: Istio injected a sidecar into the AgentGateway control plane (the namespace is istio-injection=enabled, so it injects into everything). That sidecar intercepts port 9978 and presents an Istio SPIFFE cert — URI SAN only, no DNS SAN. The proxy's xDS TLS client uses standard DNS hostname verification. The cert fails verification.

Fix: annotate the control plane deployment's pod template sidecar.istio.io/inject: "false", and exclude port 9978 from the proxy's outbound interception (traffic.sidecar.istio.io/excludeOutboundPorts: "9978", set via the proxy's AgentgatewayParameters). The proxy then reaches the control plane's own DNS-SAN cert and the handshake passes.

2. Every MCP call fails with filter_chain_not_found

Symptom: proxy is Ready, initialize returns Connection reset by peer, the backend's Envoy sidecar logs NR (no route) and filter_chain_not_found. The backend process never sees the request.

Root cause: the cluster has a STRICT PeerAuthentication (common on Kyma — the mesh policy may pre-date your deployment). The proxy was sending plain HTTP; STRICT means the backend's inbound listener has no plaintext filter chain. Connection refused before it reaches the app.

Dead end to avoid: the obvious instinct is to configure the proxy's built-in mesh integration (spec.istio.enabled: true) so it originates mTLS natively. On a sidecar-mode cluster that only gives a plaintext egress, and the SPIFFE-based backend TLS requires a Workload API the sidecar-free proxy doesn't have. Don't go down this path.

Fix: give the proxy an Envoy sidecar. The sidecar transparently upgrades the proxy's outbound connections to Istio mTLS — no backend TLS config needed. The proxy's inbound (from the LoadBalancer, plain HTTP) needs a workload-level PeerAuthentication: PERMISSIVE so STRICT doesn't reject it. Backends stay STRICT. The sidecar split in Step 2 above encodes this directly.

3. JWKS fetch fails — AgentGateway sends plain HTTP to a TLS port

Symptom: proxy is ready and auth is enabled, but every token is rejected because the JWKS keyset never loads. Logs show a connection error fetching the JWKS endpoint.

Root cause: jwks.remote.backendRef pointing at an ExternalName K8s Service causes AgentGateway to construct http://<host>:443/token_keys — plain HTTP to a TLS port. XSUAA only speaks HTTPS.

Fix: use jwks.remote.url with an explicit https:// scheme. When the scheme is https, the proxy originates backend TLS automatically. backendRef + ExternalName is the wrong tool for this.

4. Error(InvalidIssuer) — the token's iss doesn't match

Symptom: JWKS loads, signature verifies, but every request still returns 401 authentication failure … Error(InvalidIssuer).

Root cause: the ServiceBinding url field is https://<host>. But an XSUAA token's iss claim is https://<host>/oauth/token. The strings must match exactly.

Fix: issuer: <xsuaa-url>/oauth/token. Note: reaching this error is progress — it proves the JWKS fetch and signature verification are already working.

5. tools/list returns empty even with a valid token

Symptom: auth passes (200 response), but tools/list returns [].

Two root causes can stack here:

• Scope not in token: client_credentials tokens carry only what's explicitly granted to the OAuth client. If you define the scope in the ServiceInstance but omit authorities, the token carries only uaa.resource. Fix: add authorities: ["$XSAPPNAME.mcp.utility"] to the ServiceInstance parameters.
• CEL rule matching the wrong string: XSUAA prefixes scopes with the full application ID including a tenant index: poc-agentgateway!t123456.mcp.utility. A literal match on poc-agentgateway.mcp.utility fails. Fix: jwt.scope.exists(s, s.endsWith(".mcp.utility")) — matches by suffix, works across subaccounts.

6. Multi-backend routing conflict — only one server's tools appear

Symptom: tools/list returns tools from one MCP server but not the other, even after both are deployed and registered.

Root cause: two AgentgatewayBackend resources each with their own HTTPRoute both matching PathPrefix: /mcp. Gateway API tie-breaking sends all requests to one backend. The same problem occurs with multiple backendRefs in a single HTTPRoute rule — the proxy still routes to only the first.

Fix: put all MCP targets in a single AgentgatewayBackend under spec.mcp.targets[], with one HTTPRoute pointing to it. AgentGateway aggregates tools/list across all targets in the same backend. Tool names get prefixed with the target name to prevent collisions (utility_uuid_generate, time_time_current).

7. Jaeger OTLP export fails — connection reset

Symptom: traces never appear in Jaeger. Proxy Envoy logs show upstream_cluster: "PassthroughCluster" and response_flags: "UR" (upstream reset) on port 4317.

Root cause: AgentgatewayPolicy.spec.frontend.tracing.backendRef causes AgentGateway to resolve the K8s Service to pod IPs via the Endpoints API (standard Gateway API load-balancing behavior). When Envoy intercepts an outbound connection to a pod IP — not a ClusterIP — it has no named service cluster to match it against. It falls back to PassthroughCluster (plain TCP, no mTLS). Under STRICT PeerAuthentication, Jaeger's inbound Envoy has no plaintext filter chain: filter_chain_not_found, connection reset.

MCP backends work correctly because their AgentgatewayBackend.spec.mcp.targets[].static.host is set to the DNS hostname. AgentGateway connects by hostname → Envoy resolves to ClusterIP → matches the outbound|8080||... mTLS cluster → TLS handshake succeeds.

Fix: use url instead of backendRef in the tracing policy. url: "http://jaeger-collector.agentgateway-system.svc.cluster.local:4317" makes AgentGateway pass the DNS name to the TCP connection. Envoy resolves it to the ClusterIP and routes through the proper outbound|4317||jaeger-collector... mTLS cluster. response_code changes from reset to 200.

8. Jaeger UI returns 503 via the gateway HTTPRoute

Symptom: GET /jaeger/ returns 503 from the proxy with Connection reset by peer.

Root cause: same PassthroughCluster mechanism as lesson 7, but for port 16686. The HTTPRoute's backendRef resolves to the Jaeger pod IP, Envoy uses PassthroughCluster, Jaeger's inbound Envoy finds no plaintext filter chain under STRICT. Unlike the OTLP case, HTTPRoute has no url alternative in the Gateway API spec — you can't avoid the backendRef here.

Fix: add a workload-scoped PeerAuthentication: PERMISSIVE for Jaeger, overriding the namespace-wide STRICT only for the Jaeger pod:

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: jaeger-permissive
  namespace: agentgateway-system
spec:
  selector:
    matchLabels:
      app: jaeger
  mtls:
    mode: PERMISSIVE

This adds a plaintext filter chain alongside the mTLS one on Jaeger's inbound. The proxy's PassthroughCluster connection is accepted. All other workloads remain STRICT. The OTLP export (now using the correct mTLS path via url) is unaffected.

---

What the result demonstrates

Once everything above is in place, you can walk through these in order to show the system working:

1. Baseline — tools/list without a token returns all tools (auth in Permissive mode)
2. Isolation — curl from a random pod inside the cluster to a backend's ClusterIP gets RBAC: access denied
3. Live add — deploy a second MCP server with two kubectl applys; new tools appear in tools/list within seconds, nothing restarts
4. Auth on — apply the Strict auth policy; requests without a token return 401, requests with a valid token return tools
5. Scope authz — with the CEL rule applied, a token carrying *.mcp.utility sees the utility tools; a token without that scope gets []
6. Traces — tool calls appear as spans in Jaeger's UI with operation names like tools/call utility_uuid_generate; access logs on the proxy carry tool_args, tool_result, and tool_error for each tools/call

---

Key things to remember

• The sidecar split is load-bearing. Control plane sidecar-free, proxy with sidecar + PERMISSIVE inbound, backends with sidecar + STRICT. Getting this wrong is the most common source of connectivity failures.
• One AgentgatewayBackend, all targets. Separate backends with separate HTTPRoutes at the same path means only one backend wins. Consolidate under spec.mcp.targets[].
• XSUAA scopes have a tenant suffix. $XSAPPNAME.mcp.utility becomes poc-agentgateway!t123456.mcp.utility in a real token. Match with endsWith, not literal equality.
• Use url, not backendRef, for OTLP tracing. backendRef resolves to pod IPs, which Envoy routes through PassthroughCluster (no mTLS). url with a DNS hostname goes through the correct mTLS outbound cluster.
• The Admin UI is read-only. It's a debug view of the xDS-pushed state. Changes are kubectl apply.