Traefik 3.7's service middlewares target one branch of a weighted service

I copied the YAML straight out of Traefik's own announcement post for 3.7's new service middlewares, pointed it at a container running the version that post describes, and Traefik refused to start the route at all.

field not found, node: middlewares

That sent me down a longer path than I expected, because the error wasn't about my setup. It was about where the blog puts the middlewares: key.

Service middlewares are the headline feature of Traefik 3.7 ("Langres"). Until this release, a middleware could only attach to a router or run at the entry point, which meant it applied before Traefik had picked which backend would actually serve the request. If you had a weighted service split between a stable backend and a canary one, there was no way to put a middleware on just the canary branch. Whatever you attached ran for every request through that router, regardless of which child of the weighted service ended up handling it. Traefik 3.7 lets you attach a middleware directly to a service, including one child of a composite service, so it runs after the load balancer has already chosen a backend.

I wanted to see whether that actually holds up, not just read the blog post, so I built a weighted service with two backends and tried to get a header onto one branch and not the other.

The example in the announcement doesn't parse

Traefik's blog post shows this as the way to attach a middleware to a service:

http:
  services:
    cashew:
      loadBalancer:
        servers:
          - url: "https://api.cashew.io"
        middlewares:
          - cashew-version
          - cashew-path

middlewares sits inside loadBalancer, next to servers. I built a minimal version of that, pointed a real Traefik 3.7.14 container at it with the file provider, and got the error above on every restart:

time="2026-10-12T05:22:44Z" level=error msg="Error while building configuration (for the first time)"
error="/etc/traefik/dynamic.yml: field not found, node: middlewares" providerName=file

That's not a typo on my side. I pulled the actual Go source for that version and checked the struct:

• Service has a field Middlewares []string.
• ServersLoadBalancer does not have that field at all.

Middlewares lives on the service itself, as a sibling of loadBalancer, not nested inside it. The blog's own example puts it one level too deep. The working version looks like this:

http:
  services:
    canary-svc:
      loadBalancer:
        servers:
          - url: "http://127.0.0.1:9002"
      middlewares:
        - canary-header

Moving middlewares up one level, out of loadBalancer and onto canary-svc directly, is the whole fix. With that change the same container started cleanly.

Testing the actual claim: does it stay on one branch?

With the syntax fixed, I built a weighted service with a 9:1 split between two backends, each a small Python server that echoes back whatever headers it received:

services:
  myservice:
    weighted:
      services:
        - name: stable-svc
          weight: 9
        - name: canary-svc
          weight: 1

  stable-svc:
    loadBalancer:
      servers:
        - url: "http://127.0.0.1:9001"

  canary-svc:
    loadBalancer:
      servers:
        - url: "http://127.0.0.1:9002"
    middlewares:
      - canary-header

canary-header just sets X-Canary: true. I fired 200 sequential requests through the router and checked which backend answered each one and whether it carried that header.

	requests	carried X-Canary
stable-svc	180 (90.0%)	0 / 180
canary-svc	20 (10.0%)	20 / 20

The split matched the 9:1 weight exactly, and the header landed on every single canary request and none of the stable ones. I repeated it with 500 requests fired 50 at a time with xargs -P 50, to rule out a race between the load balancer's branch selection and the middleware chain under concurrency:

	requests	carried X-Canary
stable-svc	450 (90.0%)	0 / 450
canary-svc	50 (10.0%)	50 / 50

Same split, same isolation, no leaks either direction. Watching the raw sequence of which backend answered each request, the weighting isn't random sampling — it's deterministic, roughly one canary request in every ten in a fixed pattern (SSSSSSSSCSSSSSSSSSCSSSSSSSSSSC...). That matters if you're testing a canary by hand with a handful of requests: you might get unlucky and never hit it, or you might assume one in ten is a probability when it's actually a fixed slot in a cycle.

Checking the order the docs claim

The announcement says router middlewares run before service middlewares. I could check this directly, because the canary branch had two middlewares layered on it: a router-level one setting X-Trace: router, and the service-level one on the canary branch setting X-Trace: service. If service middlewares really run after router middlewares, the later one wins and the backend should see service.

branch	middlewares applied	X-Trace received by backend
stable-svc	router-trace only	router
canary-svc	router-trace, then service-trace	service

That held for every one of the 500 requests. The service middleware overwrote the router middleware's header every time on the canary branch, and the stable branch only ever saw the router's value, since it has no service middleware of its own. The documented order is correct, at least for this case.

What the limitation in the docs actually does

The announcement also says a weighted or mirror child can't carry its middlewares inline — you have to pull it out into a standalone service, which is what I did above with canary-svc. I wanted to know what happens if you ignore that and try it inline anyway:

services:
  myservice:
    weighted:
      services:
        - name: stable-svc
          weight: 9
        - name: canary-svc
          weight: 1
          middlewares:
            - canary-header

I expected this to be silently dropped. It isn't. It's the identical field not found, node: middlewares error as before, because the struct for a weighted child (WRRService) has no Middlewares field either — only Name, Weight, Headers, Status, and GRPCStatus. The practical effect is that one misplaced field doesn't just fail to apply the middleware to that one branch. It fails the entire file provider, which means every router that provider manages stops updating. In a config with dozens of routers across many files, a single stray middlewares: line on a weighted child takes all of them down, not just the one you were editing.

The version gate

I ran the exact same dynamic config, unmodified, against a traefik:v3.6 container to see what happens on the version before this existed.

Version:      3.6.25
Codename:     ramequin

error="field not found, node: middlewares" providerName=file

Same error, same refusal, no partial support. If you're running 3.6 and copy a 3.7 config written for a shared service, Traefik won't ignore the field and start anyway — the whole file provider configuration fails to load, same as the misplaced-field case above.

Watching it in production: the API disagrees with itself

Traefik exposes /api/http/middlewares, which is normally how you'd check what's actually using a given middleware before you delete it. I queried it while the canary header was demonstrably live and modifying real traffic:

canary-header@file -> usedBy: None
router-trace@file  -> usedBy: ['testrouter@file']
service-trace@file -> usedBy: None

canary-header and service-trace are both attached to canary-svc and both are working, as the header test above proves. But the usedBy field on the global middleware listing only tracks router attachments; it shows None for both. If you went looking for orphaned middlewares to clean up using that endpoint, you'd delete two that are live. The per-service endpoint gets it right — /api/http/services/canary-svc@file lists "middlewares": ["canary-header", "service-trace"] correctly — but that only helps if you already know to check it per service rather than trusting the middleware-level view.

What I got wrong on the way

My first reaction to the parse error was to assume the feature just wasn't in this build yet, since beta features sometimes ship behind flags. I spent time looking for an experimental: flag to enable it before checking whether my YAML even matched the Go struct. The fix needed no flag at all — it needed the field moved up one level, which is in the Traefik docs reference pages, just not in the blog post that introduces the feature. I should have diffed the blog's YAML against the reference docs before assuming anything was disabled.

Run it yourself

This needs Docker and nothing else. Two backend servers, each echoing headers:

# backend.py
import sys, json
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

NAME, PORT = sys.argv[1], int(sys.argv[2])

class H(BaseHTTPRequestHandler):
    def do_GET(self):
        body = json.dumps({"served_by": NAME, "headers": dict(self.headers)}).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)
    def log_message(self, *a): pass

ThreadingHTTPServer(("127.0.0.1", PORT), H).serve_forever()

python3 backend.py stable 9001 &
python3 backend.py canary 9002 &

dynamic.yml:

http:
  routers:
    testrouter:
      rule: "PathPrefix(`/`)"
      entryPoints: ["web"]
      service: myservice
      middlewares: ["router-trace"]

  services:
    myservice:
      weighted:
        services:
          - { name: stable-svc, weight: 9 }
          - { name: canary-svc, weight: 1 }
    stable-svc:
      loadBalancer:
        servers: [{ url: "http://127.0.0.1:9001" }]
    canary-svc:
      loadBalancer:
        servers: [{ url: "http://127.0.0.1:9002" }]
      middlewares: ["canary-header", "service-trace"]

  middlewares:
    canary-header:
      headers: { customRequestHeaders: { X-Canary: "true" } }
    router-trace:
      headers: { customRequestHeaders: { X-Trace: "router" } }
    service-trace:
      headers: { customRequestHeaders: { X-Trace: "service" } }

traefik.yml:

entryPoints:
  web:
    address: ":8085"
api:
  insecure: true
providers:
  file:
    filename: /etc/traefik/dynamic.yml

Run it and fire requests:

docker run -d --name traefik37 --network host \
  -v "$(pwd)/traefik.yml:/etc/traefik/traefik.yml:ro" \
  -v "$(pwd)/dynamic.yml:/etc/traefik/dynamic.yml:ro" \
  traefik:v3.7

for i in $(seq 1 20); do curl -s http://127.0.0.1:8085/; echo; done

If you're on Traefik 3.7 and already use weighted services for canary releases, check your dynamic config against the real struct, not just the blog post, before you trust that a service middleware is scoped to the branch you think it is. And if you're auditing middlewares before deleting one, check the service it's attached to directly rather than trusting the global middleware list's usedBy field.