DEV Community

Cover image for One Gateway, Multiple Applications: Building Kubernetes Ingress with MetalLB and Envoy Gateway
Abraham Naiborhu
Abraham Naiborhu

Posted on Edited on

One Gateway, Multiple Applications: Building Kubernetes Ingress with MetalLB and Envoy Gateway

#architecture #devops #kubernetes #networking

Hi! In the previous Kubernetes article, I discussed why seeing every node in Ready was only the first milestone. The cluster was running, but that did not mean it was ready to serve applications the way I wanted.

At that point, I could deploy a frontend and backend, give them ClusterIP Services, and communicate with them from inside the cluster.

That worked, but it only solved communication inside the cluster. It got me thinking:

How do I expose multiple applications without giving every application its own public endpoint or maintaining a collection of NodePorts?

I was building a small, non-production Kubernetes platform on existing hardware. I didn't want a new ingress design every time I added an application.

For this implementation, I chose MetalLB in Layer 2 mode, Envoy Gateway, Gateway API, and eventually Cloudflare Tunnel for external access.

But those components solve different problems. Installing all of them does not automatically produce a working request path. I still needed to prove that a request could reach the correct application, one layer at a time.

This article is about why I separated those responsibilities, how I configured the shared entry point, how I validated the routing, and a hostname decision that became much more important once TLS entered the picture.

About the examples: I tried to make everything as generic as possible. 192.0.2.50 is a documentation-only placeholder from a reserved address block, not an address to use on a real LAN. Any commands using it must be adapted before execution.

1. Start with an application that works internally

Before exposing anything, I used a simple frontend and mock backend to prove that Kubernetes Services worked independently of the ingress stack.

The initial design was straightforward:

Kubernetes cluster

frontend Pod(s)                backend Pod(s)
      |                               |
frontend Service              backend Service
  ClusterIP                      ClusterIP
      |                               |
      +----------- Pod network ------+

No external application endpoint yet.
Enter fullscreen mode Exit fullscreen mode

The frontend would serve a basic page. The backend had a predictable /api response. This gave me something small enough to debug without bringing in database credentials, authentication, or the actual business application yet.

I also deliberately kept these services as ClusterIP:

apiVersion: v1
kind: Service
metadata:
  name: frontend
  namespace: demo-staging
spec:
  type: ClusterIP
  selector:
    app: frontend
  ports:
    - name: http
      port: 80
      targetPort: 80
Enter fullscreen mode Exit fullscreen mode

The backend followed the same idea, with its own Service and port.

Why not create a LoadBalancer Service for every application?

Because I wanted the platform to own the incoming traffic, not each application.

For this platform, an application would normally need a Deployment, a ClusterIP Service, and an HTTPRoute. I would add NetworkPolicies and other resources based on what the application actually needed. But it should not require a new VIP, router change, or tunnel every time we deploy something else.

The responsibility split became:

Component What I wanted it to own
Calico Pod networking between workloads
Kubernetes Service Stable internal destination for application Pods
MetalLB LAN-side advertisement of a LoadBalancer Service address
Envoy Gateway Controller that programs Envoy proxies from Gateway API resources
Envoy proxy HTTP(S) traffic handling and forwarding
Gateway API Gateway, listener, hostname, and routing intent
Cloudflare Tunnel A public-to-private entry path without inbound router port forwarding

I think this distinction is more useful than starting with an installation command.

2. First, give Kubernetes a stable address on the LAN

In a managed cloud, a LoadBalancer Service can usually ask the provider's integration to provision a load balancer. And boy, that is convenient compared with managing the LAN myself.

This cluster didn't have that integration.

If I created a LoadBalancer Service without a load-balancer implementation, the external address would not be provisioned automatically.

MetalLB filled that gap.

Why Layer 2 instead of BGP?

MetalLB supports several advertisement approaches. I selected Layer 2 because the cluster was small, its nodes were on a suitable LAN, and I didn't want to integrate BGP with the surrounding network just for this platform. The network was already complicated enough. I didn't want to introduce BGP without a good reason.

The trade-off is important: in Layer 2 mode, one eligible node announces a particular VIP at a time. It is not an ECMP load-balanced set of nodes all announcing the same address simultaneously. If the elected node becomes unavailable, another eligible node can take over, but failover and traffic forwarding still depend on the rest of the cluster and network behaving correctly.

MetalLB also does not need to assign the VIP as a conventional IP address on the node's Ethernet interface. In L2 mode, the speaker responds to ARP for that VIP on the LAN.

Conceptually, the ARP exchange looks like this:

LAN client: "Who has the shared VIP?"
                    |
                    v
          MetalLB speaker on an
           elected eligible node
                    |
              ARP response
                    |
                    v
       Client sends traffic toward
            that node's MAC
                    |
                    v
          Kubernetes service path
Enter fullscreen mode Exit fullscreen mode

The exact packet forwarding after that point depends on the Service configuration and Kubernetes dataplane. MetalLB handles advertisement and address allocation; it is not the HTTP router.

Separate IP allocation from advertisement

I used two MetalLB resources for different purposes: IPAddressPool and L2Advertisement.

kubectl get ipaddresspools,l2advertisements -n metallb-system -o yaml
Enter fullscreen mode Exit fullscreen mode

A generalized example looks like this:

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: edge-pool
  namespace: metallb-system
spec:
  addresses:
    - 192.0.2.50-192.0.2.55 # DOCUMENTATION ONLY - replace
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: edge-l2
  namespace: metallb-system
spec:
  ipAddressPools:
    - edge-pool
  nodeSelectors:
    - matchLabels:
        edge-advertiser: "true"
Enter fullscreen mode Exit fullscreen mode

The range must be genuinely available and excluded from DHCP allocation in a real deployment. Do not copy a documentation subnet into a live cluster.

I also didn't want every node to be eligible to advertise that address. Some machines had multiple active interfaces during the initial setup. I kept the control plane out of the edge-advertisement set and selected appropriate wired workers instead.

For example, eligible nodes can be labeled:

# Replace this value with the node you verified as eligible.
WORKER_NODE='REPLACE_WITH_ELIGIBLE_WORKER'

kubectl label node "${WORKER_NODE}" \
  edge-advertiser=true --overwrite
Enter fullscreen mode Exit fullscreen mode

That label controls who may advertise the VIP through the MetalLB resource. It does not control where ordinary application Pods are scheduled.

One more detail: I didn't use one hardcoded network-interface name in the L2 advertisement. The machines were not identical, so an interface name on one machine was not necessarily present on another. Selecting eligible nodes based on their verified LAN connectivity was a more useful initial abstraction in this environment.

That choice limits which nodes can announce the VIP, not necessarily which interfaces they use. MetalLB can announce on multiple interfaces by default. In this initial design I selected wired workers, but if I later allow a multi-interface worker into the eligible set, I need to validate its ARP behavior and possibly add explicit interface restrictions.

3. MetalLB solves the address. It does not decide /api

A stable IP was the first milestone, but the problem wasn't solved yet.

Suppose I had two services:

frontend:80
backend:8080
Enter fullscreen mode Exit fullscreen mode

A client reaching a VIP still needs to be directed to one of those services. MetalLB does not inspect the HTTP path and decide that /api goes to a backend while / goes to a frontend.

That is where Envoy Gateway entered the design.

I chose Envoy Gateway because I wanted to express the shared entry point and application routing using the Kubernetes Gateway API, instead of giving application teams ownership over the load-balancing infrastructure.

The resulting relationship looks like this:

LAN client
    |
    v
Shared LAN VIP
    |
    | MetalLB advertises the VIP
    v
Envoy proxy Service (LoadBalancer)
    |
    v
Envoy proxy
    |
    | Gateway API routing configuration
    +-----------------------+
    |                       |
    v                       v
frontend Service       backend Service
    |                       |
frontend Pods           backend Pods
Enter fullscreen mode Exit fullscreen mode

The design is not "MetalLB versus Envoy." They complement each other.

One Gateway, application-owned HTTPRoutes

I kept the shared Gateway in a platform namespace. Applications received their own namespace and HTTPRoute.

A reduced example of the shared Gateway:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: shared-gateway
  namespace: platform-system
spec:
  gatewayClassName: envoy-gateway
  addresses:
    - type: IPAddress
      value: 192.0.2.50 # Example
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: Selector
          selector:
            matchLabels:
              environment: staging
Enter fullscreen mode Exit fullscreen mode

In the original implementation, the GatewayClass identified the Envoy Gateway controller. Envoy Gateway then managed the Envoy proxy resources for that Gateway, including its LoadBalancer Service.

The requested Gateway address and the MetalLB-managed Service address must agree. I wanted that relationship declared rather than manually patching controller-generated resources.

The allowedRoutes setting mattered too. I didn't want a Route from an arbitrary namespace attaching to the shared staging listener simply because someone knew the Gateway name.

In this example, namespaces require:

metadata:
  labels:
    environment: staging
Enter fullscreen mode Exit fullscreen mode

This limits which namespaces may attach Routes to that listener. It is not the same thing as a NetworkPolicy, RBAC policy, or a complete security boundary for the applications themselves.

There is also a permission assumption here: whoever can edit the namespace labels may be able to make that namespace eligible. So I would control who can change those labels instead of treating environment: staging as a strong security boundary by itself.

Route / and /api independently

The application's Route then references the platform Gateway:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: demo-route
  namespace: demo-staging
spec:
  parentRefs:
    - name: shared-gateway
      namespace: platform-system
      sectionName: http
  hostnames:
    - demo-staging.example.com
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /api
      backendRefs:
        - name: backend
          port: 8080
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: frontend
          port: 80
Enter fullscreen mode Exit fullscreen mode

The Route and its backend Services are in the same application namespace. A ReferenceGrant is not required just because the Route attaches to a Gateway in another namespace; the cross-namespace attachment is controlled by the Gateway's allowedRoutes.

There is another subtle point: /api is a more specific path match than /. With this example, a request for /api or /api/orders should go to the backend, while a request for / should go to the frontend. Gateway API PathPrefix matching is based on path segments, so /apix is not the same as /api.

I also gave the Route an explicit hostname. Without that, a catch-all / rule can be a bad surprise when more applications attach to the same shared listener.

I deliberately tested those paths separately. If the frontend worked while /api failed, that was useful information. It meant I could start by checking the Route, backend reference, and backend Service rather than debugging MetalLB from scratch.

4. The validation order mattered more than the installation order

I didn't want to begin testing from the public internet and then guess which of six different layers was broken.

The validation sequence in my runbooks progressed from internal Services to LAN ingress, and then to TLS and the public edge.

Test 1: Does the application Service actually have endpoints? 

kubectl get svc -n demo-staging

kubectl get endpointslice -n demo-staging \
  -l kubernetes.io/service-name=backend
Enter fullscreen mode Exit fullscreen mode

If the backend has no ready endpoints, Envoy cannot manufacture a healthy application for it. First inspect Pod readiness, selectors, and the Service's target port.

Test 2: Is there a LoadBalancer address? 

kubectl get svc -n envoy-gateway-system
kubectl get gateway -n platform-system
Enter fullscreen mode Exit fullscreen mode

I want the Envoy-managed LoadBalancer Service to receive the intended LAN VIP, and the Gateway to report useful status conditions.

However, an address displayed by the Kubernetes API is not the same thing as an ARP response from a real machine on the LAN. I need to check from outside the cluster as well.

Test 3: Did the Gateway accept the Route? 

kubectl get gateway shared-gateway \
  -n platform-system -o yaml

kubectl get httproute demo-route \
  -n demo-staging -o yaml
Enter fullscreen mode Exit fullscreen mode

I inspect conditions such as Accepted, Programmed, and ResolvedRefs, depending on the resource and controller.

Explanation:

  • Accepted=False on the Route can point toward listener/namespace/hostname attachment problems.
  • ResolvedRefs=False can indicate an invalid backend Service or port reference.
  • A programmed Gateway can still forward to an unhealthy backend.

The exact reason and message in the condition matter more than memorizing a green status.

Test 4: Do the hostname and paths reach the intended Services?

The example HTTPRoute uses demo-staging.example.com. If I curl the VIP by itself, the request will not match that hostname-specific Route. So I need to test both the hostname and the path.

From a machine on the appropriate LAN, using the real reserved VIP:

VIP='REPLACE_WITH_REAL_RESERVED_LAN_VIP'

curl -i --resolve "demo-staging.example.com:80:${VIP}" \
  "http://demo-staging.example.com/"

curl -i --resolve "demo-staging.example.com:80:${VIP}" \
  "http://demo-staging.example.com/api"
Enter fullscreen mode Exit fullscreen mode

This makes curl connect to the VIP while sending the correct HTTP Host header. The IP alone identifies the Gateway; the hostname and path identify the application route.

For my mock application, the expected behavior was a frontend response for / and an API response for /api. These are validation commands and expected behaviors, not pasted console results from the private environment.

I also wanted a negative test. An absent or invalid backend should not be described as working just because the Gateway reports that it has accepted a Route.

After validating the basic routing path, I could add more application hostnames without changing their internal Services.

5. Hostnames became part of the platform contract

A single shared VIP becomes more useful when it can host multiple applications.

For example:

app-a-staging.example.com  ----+
                               |
app-b-staging.example.com  ----+--> same shared Gateway VIP
                               |           |
app-c-staging.example.com  ----+           +--> separate HTTPRoutes
Enter fullscreen mode Exit fullscreen mode

Each application can own its hostname and Route while continuing to use ordinary ClusterIP Services.

This is also where I started caring more about the distinction between an IP address and the HTTP Host header.

A Gateway can receive requests on the same address and make different routing decisions based on the requested hostname. The VIP alone does not identify the application.

One application Route can be expressed like this:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: app-a
  namespace: app-a-staging
spec:
  parentRefs:
    - name: shared-gateway
      namespace: platform-system
      sectionName: https
  hostnames:
    - app-a-staging.example.com
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: frontend
          port: 80
Enter fullscreen mode Exit fullscreen mode

This is a later HTTPS-state example. The earlier Gateway manifest only declared an HTTP listener, so the sectionName: https reference will not work until I also add an HTTPS listener with a matching hostname and TLS certificate. The namespace must still have the allowed label, and the certificate must cover the application hostname.

Validate hostname routing without relying on public DNS

I used the idea of testing the origin first, before involving the external DNS or Cloudflare layers.

Once HTTPS and the certificate are configured:

VIP='REPLACE_WITH_REAL_RESERVED_LAN_VIP'

curl -i \
  --resolve "app-a-staging.example.com:443:${VIP}" \
  "https://app-a-staging.example.com/"
Enter fullscreen mode Exit fullscreen mode

If the origin uses an internal CA that my client does not trust, I should configure the correct CA bundle (for example with --cacert) instead of turning off verification.

--resolve makes curl connect to the specified address while still using the hostname in the URL. That helps test the correct TLS hostname and HTTP routing at the origin.

Do not add -k just to hide certificate errors. If the origin certificate is not trusted or its SAN does not cover the requested name, that is part of what needs investigation.

The documentation-only names and addresses in this article will not resolve or work unchanged. Use values that match a test environment and a certificate actually issued for that environment.

6. A naming preference turned into a TLS problem

This was one of the more interesting decisions from the platform notes.

My initial staging hostname convention had a hierarchy like:

app.staging.example.com
Enter fullscreen mode Exit fullscreen mode

I liked the structure. Applications would sit under the staging subdomain. From an organizational standpoint, it looked neat.

However, once I introduced Cloudflare at the public edge, the certificate coverage became important.

On a Cloudflare full DNS setup, Universal SSL normally covers the zone apex and first-level subdomains, such as:

example.com
app-staging.example.com
Enter fullscreen mode Exit fullscreen mode

It does not automatically cover a deeper hostname such as:

app.staging.example.com
Enter fullscreen mode Exit fullscreen mode

A wildcard like *.example.com covers one label below the apex; it does not cover app.staging.example.com. So yes, Cloudflare can support the deeper hostname. But in this full DNS setup, it meant adding different certificate coverage, with additional cost for the usual managed options. I didn't think that was worth it for staging.

I could have used additional Cloudflare certificate coverage for the deeper name. Instead, I changed the platform's naming convention to:

<application>-<environment>.example.com
Enter fullscreen mode Exit fullscreen mode

For example:

app-a-staging.example.com
app-b-staging.example.com
Enter fullscreen mode Exit fullscreen mode

That preserved the environment distinction without requiring a deeper subdomain hierarchy for the public hostname.

I think this is an example of a small-looking convention that actually matters. The application code and Service ports did not need to change. The affected resources were at the edge: certificates, Gateway listener hostnames, HTTPRoutes, and the Cloudflare published application configuration.

Important distinction: This was a problem with the public Cloudflare edge certificate coverage, not evidence that Gateway API cannot route deeper hostnames. I could still issue a separately appropriate origin certificate for deeper names using cert-manager DNS-01. The edge and origin certificates solve different TLS sessions.

Cloudflare's own Universal SSL documentation describes the first-level limitation for full DNS setups, as well as alternatives for deeper hostnames.

7. The internet path was a separate problem

So far, the design had a stable LAN-side gateway and hostname-aware routing.

That still didn't make it reachable from the internet.

I didn't want to open inbound router ports into the network just to expose staging applications. For this non-production environment, I selected Cloudflare Tunnel as the public entry path.

The intended flow was:

Browser
   |
   | HTTPS to Cloudflare edge
   v
Cloudflare Edge
   |
   | Encrypted tunnel transport
   v
cloudflared connector
   |
   | HTTPS to the Kubernetes origin
   v
Envoy Gateway (shared LAN VIP)
   |
   | HTTPRoute: hostname + path
   v
ClusterIP Service
   |
   v
Application Pods
Enter fullscreen mode Exit fullscreen mode

In this model, Cloudflare's public edge handles the browser-facing certificate. The origin Envoy Gateway presents its own certificate to cloudflared over HTTPS. The browser-to-edge and connector-to-origin TLS negotiations should not be confused with one another.

The origin's certificate name must still match

The origin can be reached by a LAN IP, but the certificate on the Envoy listener is normally issued for a hostname, not for that IP.

So when cloudflared connects to an origin such as:

https://<YOUR_REAL_LAN_VIP>:443
Enter fullscreen mode Exit fullscreen mode

I need to make sure it uses an appropriate server name for TLS verification and that the application hostname reaches Envoy for HTTPRoute matching. Cloudflare supports origin TLS settings such as originServerName and SNI-related options for this situation.

I deliberately did not want to work around a mismatch with noTLSVerify: true. Disabling verification can make the symptom disappear without fixing the trust and hostname problem.

The migration report records successful direct HTTPS responses after the hostname changes, including the frontend and /api path. However, the example direct-origin checks in that report use curl -vk. They show the TLS connection and HTTP responses, not that certificate-chain validation passed, because -k disables that verification. They also record a Cloudflare Tunnel 502 while the tunnel-to-origin configuration was being adjusted. One of the missing settings identified in that investigation involved matching SNI to the intended origin hostname.

Those observations prove different things:

Observation What it supports What it does not prove
Gateway Programmed=True The controller accepted and programmed the Gateway Application availability
HTTPRoute Accepted=True and ResolvedRefs=True Routing attachment and backend references were accepted Successful application responses
Direct HTTPS response from the origin using curl -vk The origin responded over HTTPS and routing returned the expected content Validated certificate trust or public Cloudflare Tunnel success
Tunnel returned 502 The connector-to-origin path still needed investigation That MetalLB or the application Service had failed

I don't want to call an end-to-end public path complete based on origin-only tests. The migration report explicitly treated public Tunnel acceptance as a separate checkpoint.

That separation between layers is one of the practical lessons from this implementation.

8. What I intentionally didn't give each application

Once the shared gateway was established, the application onboarding contract became simpler.

A normal application needed a Deployment, ClusterIP Service, and an HTTPRoute. Depending on the application's purpose, it would also need Secrets, ConfigMaps, health probes, resource settings, and NetworkPolicies for ingress or egress, plus resource quotas where appropriate. Yes, applications still need other configuration. But the point is that they no longer have to own the shared ingress infrastructure.

What it did not need was its own:

  • LAN VIP or LoadBalancer Service;
  • public NodePort;
  • inbound office-router forwarding rule;
  • standalone reverse proxy;
  • separate Cloudflare Tunnel just because the application was new.

That is what I was trying to achieve with the shared platform.

However, I would not describe it as highly available merely because the design has several workers. Layer 2 VIP leadership uses one announcer at a time. The gateway, tunnel connectors, DNS, control plane, underlying LAN, and applications each have their own failure modes. A shared entry point also introduces a shared dependency that must be operated carefully.

Security is another separate concern. allowedRoutes controls route attachment, but it doesn't stop arbitrary Pod-to-Pod ingress by itself. I added NetworkPolicy work as a later, independent validation step rather than treating a working Gateway as proof that applications were isolated.

And one more limitation: using a tunnel avoids inbound router port forwarding; it doesn't mean the service is automatically private to authorized users. Public access policy and authentication still matter.

9. What I'd validate before calling the public route ready

My checklist for any new application would be:

  1. The application is healthy behind its own ClusterIP Service.
  2. EndpointSlices contain ready endpoints for the intended backend.
  3. The application's namespace is allowed to attach to the shared Gateway.
  4. The HTTPRoute reports valid attachment and backend references.
  5. The shared Gateway is programmed and the intended LAN VIP is available.
  6. The hostname and path give the expected response directly at Envoy.
  7. The certificate's SAN and trust chain match the hostname being tested.
  8. The Cloudflare published hostname points to the intended tunnel/origin.
  9. The connector validates the origin TLS endpoint without disabling certificate checks.
  10. The full public hostname returns the intended application response.
  11. Where NetworkPolicies apply, negative tests confirm the traffic paths that are meant to be blocked.

Some of those checks belong to other parts of the platform and deserve separate articles. I don't want to pretend that successfully installing Envoy proves all of them.

10. What I learned

For me, the main lesson wasn't how to install MetalLB or Envoy Gateway.

It was learning to separate the problems properly and understand the complete request path, instead of guessing which component was responsible.

MetalLB made an address reachable on the LAN. Envoy Gateway gave that address a way to route HTTP traffic. Gateway API separated shared platform ownership from application routing. Cloudflare Tunnel provided another path into the private environment without router port forwarding.

And even with that design, I still had to think about certificate coverage, hostname conventions, and what each test actually proved.

I could have given every application its own external Service and moved on. For a small platform with more than one application, I'd rather make the shared boundary deliberate first.

The next part of this series will move from how traffic gets in to how changes get deployed automatically: separating application CI from the GitOps controller that owns Kubernetes desired state.

References

Top comments (0)