DEV Community: Max Espinoza

Cloud Cost Pondering Part 1

Max Espinoza — Fri, 05 Sep 2025 00:53:35 +0000

It's been a while since I've posted here. I've been meaning to use this space to pose questions to myself and anyone else reading. This is meant to be a series of posts as I learn a more about GCP billing so if you know of any interesting resources, please do share!

I've been doing cost comparisons between providers. I'm familiar with one of the major providers' billing pages, but I'm still new when it comes to GCP. That said, GCP provides a handy Price Calculator to assist in getting a rough estimate. Almost all cloud vendors have some version of a price calculator. One thing I never asked myself until recently was: how accurate are these price calculators, especially those that do some behind-the-scenes math, like the Cloud Run calculator based on 'Traffic Shape'? And how should one best use these calculators before making large financial commitments?

I imagine there's already a wealth of material out there, including podcasts about this sort of thing. This series of posts isn't meant to add to the body of knowledge out there, but I hope it can help others sift through the material and get to the heart of these two questions¹:

How accurate are the online price calculators published by the big cloud vendors?
What is the best way to estimate cloud costs using these calculators?

Cover art generated with perchance.

That or at the very least be fodder for latest LLMs. ↩

Istio Authentication & Authorization

Max Espinoza — Fri, 16 Jul 2021 18:27:55 +0000

Istio Authorization Policies

I recently had to debug Istio authorization policies and I learned what a pain it can be understand how and when policies are applied to protect service on your mesh. This post is my attempt to explain the important bits of how to set this up.

With that said, checkout the Istio official docs for more detail.

The Tools

Istio

Istio is one of the leading service mesh implementations out there. It's used by a lot of companies to manage the growing overhead of having services running on Kubernetes. This is especially true when dealing with microservice architectures, which yield dozens (if not hundreds) services which need to talk to each other securely and reliably.

Using a service mesh, like Istio, handles a lot of this complexity for you. For example app developers do not need implement the following features and can instead use what's configurable in the mesh:

mTLS between services (encrypted both ways)
telemetry (common network metrics exposed for Prometheus)
rate limiting & fault tolerance
authentication (proving who you are)
authorization (can you do that?)

There is more the service mesh can do for your, but we're going to focus on the last two in this post.

The JWT

(pronounced 'jot')

Authentication is proving who are you. Using Istio, you can request clients calling services on the mesh prove who they are using JWTs (Javascript Web Token). This functionality is configured with a RequestAuthenication CRD (Custom Resource Definition). Once this functionality is configured you can use AuthorizationPolicies CRDs to enforce RBAC based on provided JWTs.

But to that, you need to understand what the JWT is, and how it's used in conjunction with Istio to enforce RBAC for services.

Above: Anatomy of a JWT

The JWT is the standard token used in OpenIDConnect (OIDC) spec, and widely used as result. It has three pieces to it, the header, payload, and signature. We are focused on the payload. Within the JSON payload one can put data (note: keys in the payload are called claims). That said, there are certain keys (claims) that are reserved by the JWT spec. Non-reserved claims are called custom claims, of which the OIDC spec defines a handful of. These claims are often grouped into scopes. Where a scopes can be thought of as groupings of related permissions. Again, unsurprisingly the OIDC specifies certain scopes.

Now why does this matter?

Because, having a JWT can act as proof of who you are, and claims within the JWT capture what you are allowed to do. Clients usually obtain a JWT by requesting one from a JWT issuer along with some credentials to prove identity (user/pass/2FA). The JWT is then usually provided by the client to the app in the Authorization: Bearer <token> request header. There's a whole slew of tooling around doing this "handshake", from both client and server-side (Istio included). The neat thing about using a service mesh is that Istio can handle this interaction transparently to services. You only need configure the RequestAuthenication and AuthorizationPolicy objects.

Above: JWT 'handshake'

The `RequestAuthentication` Object

In the RequestAuthentication object you can specify which workloads require a JWT from which trusted issuers. Note you need to provide jwksUri so that Istio knows where to grab the certs used in the validation of the tokens (aka the JSON Web Key Set). Your issuer will have an endpoint for this (sometimes linked from the well-known endpoint ).

Example Request Authentication

In this example, we want our service httpbin in the foo namespace to require a valid JWT issues from google.

apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: foo-req-auth
  namespace: foo
spec:
  selector:
    matchLabels:
      app: httpbin
  jwtRules:
  - issuer: "https://accounts.google.com" # Whose JWTs do you trust?
    jwksUri: "https://www.googleapis.com/oauth2/v3/certs" # Certs to verify JWTs

This alone does not however enforce that others cannot hit your endpoint publicly. You need to this this in with Authorization Policies.

The `AuthorizationPolicy` Object

Now here is the meat of what you will be configuring when using Istio enforce RBAC for your services. I won't go in depth here (there is a lot you can do) but I will briefly touch on the .

There are 3 kind of authorization policies:
1. allow policies
2. deny policies
3. custom policies
There is an ordering in how access is determined with these three policies
These policies have rules, which define if certain criteria is met, what level of access is allowed.

Example Authorization Policy

In this example, we allow access to our service httpbin in namespace foo from any JWT (regardless of the principle) to use the GET method.

apiVersion: "security.istio.io/v1beta1"
kind: "AuthorizationPolicy"
metadata:
  name: "allow-reads"
  namespace: foo
spec:
  selector:
    matchLabels:
      app: httpbin
  rules:
  - from:
    - source:
        principals: ["*"]
    to:
    - operation:
        methods: ["GET"]

Example tying together claims with Authorization Policies

🚧 I will fill out this section soon. I working on more examples and diagrams to tie these concepts together!
I highly recommend reading Istio By Example for more info!

Access Flow with Auth Policies

There is some logic behind how authorization is determined given defined AuthorizationPolicies. Below is the flow as taken directly from the Istio documentation.

1 - If there are any CUSTOM policies that match the request, evaluate and deny the request if the evaluation result is deny.
2 - If there are any DENY policies that match the request, deny the request.
3 - If there are no ALLOW policies for the workload, allow the request.
4 - If any of the ALLOW policies match the request, allow the request.
5 - Deny the request.
source

That said, as a visual learner, I need something tangible to keep this model in my head. So I think model the auth flow a person in a cooperate lobby trying to get enter an office.

In this analogy:

the Custom auth policies can be thought of C-level execs
- They can decided many things, including if your allowed in or not!
the Deny policies can be thought of a office security
- They are on the look out for features, if you catch their eye they will kick out
the Allow policies can thought of as employees of the office your are visiting
- They can swipe you into the office if they happen to be expecting you

Above: You enter the lobby and you notice the custom policies. If they are expecting (match) you, they will decide if you should be allowed in the office or not. They make the first call in regards to access.

Above: Here, the execs are not expecting you, you're barely noticed by them. But if you catch the eye (match) one of the deny policies, they will escort you out. So try not to look suspicious!

Above: In this case, no one wants you out and there are no Allow policies defined. It's assumed you are allowed to enter the office

Above: But, if any Allow policies are defined, you're going to need to have one of them expecting you (match) to allow you into the office.

Above: If no one is expecting you, but no one kicks you out explicitly, you are not allowed linger in the lobby. You will be kicked out.

In Summary

Using Istio you are just a few configuration files away from handling auth for your services. While the configuration of these files might be a little obtuse, thinking about how they fit and how they get applied shouldn't have to be!

Above: The main actors in this play and the steps that happen. Note in step 5, that we apply the flow discussed earlier in this post.

Rancher, Prometheus, and Alert Manager in 5 minutes

Max Espinoza — Thu, 29 Apr 2021 18:05:34 +0000

I recently had take a look at how to setup monitoring and alerting for a Kubernetes cluster. I knew that Rancher offered monitoring that via Prometheus and Alert Manager, but never had the chance to sit down do its.

This is my attempt to explain the important bits to anyone looking to implement monitoring and alerting (without just throwing the docs back at you).

First, let's talk about the main players.

The Tooling Chain

Prometheus

Simply put Prometheus can be thought of as a cronjob with a database.

It will scrape endpoints for metrics at set intervals of time
It will run queries against itself and save those results for later use at set intervals of time.

Alert Manager

Alert Manager is a watcher of Prometheus alerts and a notifer with support for tools like xMatters, Slack, PagerDuty, Etc.

It will look at alerts and notify based on how you configure it
You can slience triggered alerts for periods of time
For more advance use cases you can de-duplicate alerts from multiple sources

Rancher

It's what Ubuntu is to Linux. For this post, you can think of it as the thing that has already installed Alert Manager and Prometheus on your cluster. Also, it's got a nice UI for you configure those tools without writing the underlying Kubernetes CRDs.

What's neat about Rancher is that when monitoring is enabled, then Prometheus is already scraping your cluster for a wide variety of metrics you can set notifications on. However, if you are not using Rancher then be sure configure Prometheus to scrape targets for the metrics you'd like to notify on. I won't talk about this here, but a Google search will turn out a lot of resources on how configure Prometheus targets.

Potentially unfamiliar terms:

Scraping: Polling and collecting data exposed at endpoint

Targets: The endpoints Prometheus is scraping

CRD: Custom Resource Definitions -- Custom Kuberenetes API Objects that is used to configure something on the cluster

Prometheus hands-off alerting

Prometheus offers alerting, in much the same way that books offer knowledge. It's there but it's on you read and act on it. As mentioned before Prometheus can be configured to query against itself (at set intervals of time) and record those results. Now, that's as far as Prometheus goes in regards to alerting. It's just making notes of what alerts were triggered.

Above: How Prometheus operates glossing over some details

How exactly do you tell Prometheus what to make note of then? PrometheusRules, that's how. PrometheusRules are the CRD that hold the logic of what [with Alert Manager's help] to alert on.

Above: Basic pieces of PrometheusRule.

alert: ApiAppPodCrashing
annotations:
  message: Container {{ $labels.container }} in pod {{ $labels.pod }} in {{$labels.namespace}} is crashing perpetually.
  recipient: Infrastructure Team
expr: sum by(namespace) (kube_pod_container_status_last_terminated_reason{reason="CrashLoopBackoff", namespace!~"cattle-.*system|fleet-system|node-logging|cis-operator-system|ingress-nginx|kube-system"}) > 0
for: 0s
labels:
  priority: critical
  team: infra

Above: More detailed view of the alert YAML with a PromQL expression to app crashing.

Alert Manager doing the actual alerting

Alert Manager is the tooling that reads Promethus recorded alerts and pushes those alerts to other notification tools like Slack/xMatters.

What you need to know about Alert Manager is that it's configuration is done through creation of Routes and Receivers. Receivers can be thought of as endpoints to send alerts to. These can be slack rooms, xMatters integration, and a bunch more. Routes house the logic of "if you see an alert with these properties, then send these alert records to these recievers."

Above: How routes, receivers, and alerts relate to one another

Whats important to note here is that we can use labels to route the alerts to different places. In our setup, we use this to route alerts to various slack rooms (depending on app) and to trigger xMatters notifications on priority: critical.

Configuration all the way down

Monitoring and alerting is configuration files all the way down. These are the configurations you'd likely be working with.

There is more than one way to actually update/create these configurations. You can directly apply the PrometheusRule CRDs to the cluster and use the Alert Manager CLI to make the Routes/Receivers. But if you have Rancher, life gets a lot easier.

If you go the Cluster Explorer > Monitoring tab you'll find these configuration can all be setup from within the Rancher UI.

Above: View of monitoring page in Rancher.

In short

I've scratch the surface of what you can do and glossed over a lot of detail that you'll probably find better docs for, but in essence if you have the right tooling setup (read: Rancher Monitoring), you're only 3 configuration files away from having monitoring/alerting on services in your Kubernetes cluster.