DEV Community: Fei Y

Secure Microservice using Istio w/ JWT

Fei Y — Fri, 22 Jul 2022 20:33:00 +0000

Istio security pillar provides a comprehensive solution to secure your workloads with strong identity (with optional integration with SPIRE since v1.14), authentication, and authorization (with optional integration with OPA). Istio sidecar (service proxy) and ingressgateway (edge proxy) serve as the PEP for the entire mesh.

According to NIST SP 800-207 that no resource is inherently trusted that 1) every asset (service) MUST have its security posture evaluated via a PEP before a request is granted ... 2) and the evaluation should be continual for as long as the session lasts. Istio typically operates at L7. And in order to implement Defense-in-Depth strategy we also need to complement it with L3/4 enforcement through NetworkPolicy. Please check out additional security considerations, and we won't talk about it. Instead, let me walk you through how to protect the workload using Istio with JWT.

You can acquire a free JWT token through auth0. We'll use it for the demo.

Obviously, we need to enforce mTLS using PeerAuthentication. By default, Istio enableAutoMtls is set to true but it was in PERMISSIVE mode.

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system
spec:
  mtls:
    mode: STRICT

We need to make sure the request MUST be denied at the edge (istio-ingressgateway) either without JWT token or invalid JWT token with the combination of RequestAuthentication and AuthorizationPolicy.

RequesetAuthentication CRD will make sure JWT token adheres to the jwtRules specified.

apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: jwt-authn-gw
  namespace: istio-gateway
spec:
  selector:
    matchLabels:
      istio: ingressgateway
  jwtRules:
  - issuer: "https://dev-wl5b26zy.us.auth0.com/"
    jwksUri: "https://dev-wl5b26zy.us.auth0.com/.well-known/jwks.json"
    audiences:
    - "https://httpbin/api"
    forwardOriginalToken: true

JWT Token typically uses RS256(RSA Signature with SHA-256) as the asymmetric signing algorithm. Istio will make sure the token is indeed valid and tamper-proof by verifying the digital signature through jwksUri. When a request comes in it goes through various HTTP filters, and one of them is envoy.filters.http.jwt_authn. To be more efficient the JWKS will be "cached" in localJwks to improve the performance instead of every time having outbound call to the jwksUri endpoint.

...
{
    "name": "envoy.filters.http.jwt_authn",
    "typedConfig": {
        "@type": "type.googleapis.com/envoy.extensions.filters.http.jwt_authn.v3.JwtAuthentication",
        "providers": {
            "origins-0": {
                "issuer": "https://dev-wl5b26zy.us.auth0.com/",
                "audiences": [
                    "https://httpbin/api"
                ],
                "localJwks": {
                    "inlineString": "..."
                },
                "forward": true,
                "payloadInMetadata": "https://dev-wl5b26zy.us.auth0.com/"
            }
        },
...
}
...

istio_authn filter is right after the jwt_authn filter. And it tries to verify the issuer as well:

{
    "name": "istio_authn",
    "typedConfig": {
        "@type": "type.googleapis.com/istio.envoy.config.filter.http.authn.v2alpha1.FilterConfig",
        "policy": {
            "origins": [
                {
                    "jwt": {
                        "issuer": "https://dev-wl5b26zy.us.auth0.com/"
                    }
                }
            ],
            "originIsOptional": true,
            "principalBinding": "USE_ORIGIN"
        },
        "skipValidateTrustDomain": true
    }
}

Just with RequesetAuthentication will not be enough since a request without any authentication credentials will be accepted but will not have any authenticated identity. That's why it's quintessential to have AuthorizationPolicy in place to enforce.

First, it stipulates that the request will be denied at istio-ingressgateway if it doesn't have the RequestPrincipal.

apiVersion: "security.istio.io/v1beta1"
kind: AuthorizationPolicy
metadata:
  name: deny-jwt-gw
  namespace: istio-gateway
spec:
  selector:
    matchLabels:
      istio: ingressgateway
  action: DENY
  rules:
  - from:
    - source:
        notRequestPrincipals: ["*"]

Second, at the service level it has to satisfy the claims that the workload expects. In order to make it work it's important to have forwardOriginalToken: true to allow JWT token to be forwarded from the istio-ingressgateway to the workload.

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: require-jwt-httpbin
  namespace: default
spec:
  selector:
    matchLabels:
      app: httpbin
  action: ALLOW
  rules:
  - from:
    - source:
        requestPrincipals: ["https://dev-wl5b26zy.us.auth0.com//8qPXVf5npNa4yXmeyHhnGh5GDgrDK3B5@clients"]
    when:
    - key: request.auth.claims[scope]
      values: ["read:messages"]
    - key: request.auth.claims[aud]
      values: ["https://httpbin/api"]

These enforcement have been "encoded" into workload's sidecar EnvoyProxy HTTP Filter called envoy.filters.http.rbac. Any violation will cause 403 Forbidden with the error RBAC: access denied.

{
    "name": "envoy.filters.http.rbac",
    "typedConfig": {
        "@type": "type.googleapis.com/envoy.extensions.filters.http.rbac.v3.RBAC",
        "rules": {
            "policies": {
                "ns[default]-policy[require-jwt-httpbin]-rule[0]": {
                    "permissions": [
                        {
                            "andRules": {
                                "rules": [
                                    {
                                        "any": true
                                    }
                                ]
                            }
                        }
                    ],
                    "principals": [
                        {
                            "andIds": {
                                "ids": [
                                    {
                                        "orIds": {
                                            "ids": [
                                                {
                                                    "metadata": {
                                                        "filter": "istio_authn",
                                                        "path": [
                                                            {
                                                                "key": "request.auth.principal"
                                                            }
                                                        ],
                                                        "value": {
                                                            "stringMatch": {
                                                                "exact": "https://dev-wl5b26zy.us.auth0.com//8qPXVf5npNa4yXmeyHhnGh5GDgrDK3B5@clients"
                                                            }
                                                        }
                                                    }
                                                }
                                            ]
                                        }
                                    },
...
        },
        "shadowRulesStatPrefix": "istio_dry_run_allow_"
    }
}

Thanks for reading, and learning with me. If you want to run it yourself here is the source code and instructions.

Voila!

Gatekeeper with Istio

Fei Y — Tue, 19 Jul 2022 00:40:00 +0000

For those who have been using Istio to manage microservices you shouldn't be too surprised about its superpower. While working with various teams and organizations within the Enterprise I have to admit that Istio sometimes can be very complicated. Fortunately, Google, Solo, and many others continue to drive the innovation in this space so you can eventually benefit as the end user.

According to ZTA we should Never Trust, Always Verify.

In this article I want to touch upon how to use PaC (aka. Policy-As-Code) to enforce the correct implementation of the Istio (to be clear that there is no absolute right or wrong, but by following the best practices you achieve the correctness for the time being), for example Protocol Selection. By default, Istio can automatically detect HTTP(/2) traffic otherwise it will be treated as plain TCP traffic. As the Zen of Python teaches us that Explicit is better than implicit. We should always strive for code's readability and maintainability. So, let's enforce this rule at design-time and run-time.

This article is not meant for a deep dive on Rego which I will leave it to the readers.

According to Istio Explicit Protocol Selection:

This can be configured in two ways:

By the name of the port: name: <protocol>[-<suffix>].
In Kubernetes 1.18+, by the appProtocol field: appProtocol: <protocol>.

Let's dissect that.

port: name

You can definitely use the build-in function re_match, but it might complicate the Gatekeeper's ConstraintTemplate which we will talked about it in a moment. Instead, we can just split the port_name by -, and check its membership.

protocol := split(port_name, "-")[0]
protocol in protocols

port: appProtocol

The are a couple of ways to check the membership:

#1 Unification, which is different from `:=` and `==` in Rego.

protocol = protocols[_]

#2 `in`, which requires to `import future.keywords`

protocol in protocols

#3 Set

protocol_set := { p | p := input.parameters.protocols[_] }
protocol_set[protocol]

By following Styra's Rego Style Guide the option #2 will be the preferred way.

There is one little subtlety which we haven't talked about yet. When Service has both port name and port appProtocol the latter takes precedence by Istio. So, how do we express that in Rego?

In Rego within the Rule the AND condition implies which means all Rule body needs to be True to make this Rule True. The OR condition is achieved by multiple Rules with the same Rule name.

# when port.appProtocl exists just use it and ignore port name altogether.
_is_valid(port, protocols) {
  port.appProtocol

  _match_app_protocol(port.appProtocol, protocols)
}

# when port.appProtocol doesn't exit port name has to exist and match the protocols we specified.
_is_valid(port, protocols) {
  not port.appProtocol
  port.name

  _match_port_name(port.name, protocols)
}

Let's put it all together:

package istio.security.protocolselection

import future.keywords

violation[{"msg": msg}] {
  protocols := input.parameters.protocols

  some port in input.review.object.spec.ports
  not _is_valid(port, protocols)

  msg := sprintf("port: %v name or appProtocol is invalid", [port])
}

_is_valid(port, protocols) {
  port.appProtocol

  _match_app_protocol(port.appProtocol, protocols)
}

_is_valid(port, protocols) {
  not port.appProtocol
  port.name

  _match_port_name(port.name, protocols)
}

_match_app_protocol(protocol, protocols) {
  protocol in protocols
}

_match_port_name(port_name, protocols) {
  protocol := split(port_name, "-")[0]

  protocol in protocols
}

Now, we have the hardest part resolved and let's turn our attention to the OPA Gatekeeper. Gatekeeper uses the OPA Constraint Framework to describe and enforce policy. Right now there are mainly 3 parts we should pay attention:

ContraintTemplate: describe both Rego that enforces the constraint and the schema of the contraint.
Constraint: describe what ContraintTemplate needs to be enforced, and how.
Config: describe behaviors for certain processes.

1: ContraintTemplate

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  annotations:
    description: Explicit protocol selection either by name or appProtocol
  name: istioexplicitprotocolselection
spec:
  crd:
    spec:
      names:
        kind: IstioExplicitProtocolSelection
      validation:
        openAPIV3Schema:
          type: object
          properties:
            prefixes:
              type: string
            protocols:
              type: array
              items:
                type: string
  targets:
  - target: admission.k8s.gatekeeper.sh
    rego: |-
      package istio.security.protocolselection

      import future.keywords

      violation[{"msg": msg}] {
        protocols := input.parameters.protocols

        some port in input.review.object.spec.ports
        not _is_valid(port, protocols)

        msg := sprintf("port: %v name or appProtocol is invalid", [port])
      }

      _is_valid(port, protocols) {
        port.appProtocol

        _match_app_protocol(port.appProtocol, protocols)
      }

      _is_valid(port, protocols) {
        not port.appProtocol
        port.name

        _match_port_name(port.name, protocols)
      }

      _match_app_protocol(protocol, protocols) {
        protocol in protocols
      }

      _match_port_name(port_name, protocols) {
        protocol := split(port_name, "-")[0]

        protocol in protocols
      }

2: Constraint

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: IstioExplicitProtocolSelection
metadata:
  name: explicitprotocolselection
spec:
  enforcementAction: deny
  match:
    kinds:
    - apiGroups:
      - ""
      kinds:
      - Service
  parameters:
    protocols:
    - http
    - https
    - http2
    - grpc
    - grpc-web
    - tcp
    - tls

3: Config

We will use it to ignore all namespaces that we don't want to enforce the policy.

apiVersion: config.gatekeeper.sh/v1alpha1
kind: Config
metadata:
  name: config
  namespace: "gatekeeper-system"
spec:
  match:
    - excludedNamespaces: ["kube-*", "istio-*"]
      processes: ["*"]

Voila!

OPA Gatekeeper has more than I just showed you here. I'll leave it to you to explore: Gator, Mutation, Audit, Replication, etc.

Ref: https://github.com/feiyao/gatekeeper-istio