DEV Community: Josef Polar

We're all hybrid now

Josef Polar — Wed, 20 May 2026 12:51:27 +0000

For about a decade, "cloud-native" was a kind of identity, a signal a choice had been made about how software would be built and where it would run. It meant containers and Kubernetes and managed services and twelve-factor everything, with greenfield as the default starting position. The architecture diagram had no on-prem box in the corner, and the absence of one was treated as a feature rather than an oversight.

This identity is quietly dying, even if the conference talks haven't caught up yet. Most teams running at any real scale now operate a mix of containers, VMs, and physical servers spread across two or three clouds and at least one data center nobody wants to touch, which means the label "cloud-native" has become a description of a workload rather than a description of a company. Once teams accept this, a surprising amount of received wisdom stops holding up under any kind of weight.

Service mesh is the most obvious example of an idea built for a world which never arrived. The sidecar-per-pod model assumed every workload would live in Kubernetes, and the considerable operational cost was worth paying because the architecture was uniform enough to justify it. Neither assumption survives contact with a real enterprise, where the mesh tax is real but the more serious problem is geometric: the mesh doesn't extend to the legacy payments system running on bare metal or the analytics cluster sitting on VMs. Teams end up running a mesh for the containerized half of the estate and something else entirely for the other half, and the boundary between the two becomes the actual hard problem to solve.

North-south and east-west are the same problem in a hybrid estate, even though the mesh treated them as different because it could only ever see one of them clearly. The moment traffic crosses an environment edge, and in any real hybrid setup it crosses one constantly, teams are back to the boundary problem the mesh was supposed to abstract away in the first place.
The industry built abstractions for a world which didn't arrive, and the tools assuming otherwise are now generating most of the operational friction platform teams feel day to day. We were promised greenfield Kubernetes everywhere and we got a patchwork instead, which has been the actual condition of enterprise infrastructure for longer than most vendors are willing to admit publicly.
Something less ideologically clean is emerging in response, and a few vendors have started naming it directly. HAProxy has been talking about Universal Mesh, the idea being a single data plane spanning Kubernetes and VMs and bare metal alike, managed by one control plane sitting at the boundary of each environment rather than embedded inside every pod. The framing matters because it concedes the thing most vendors have spent years dancing around, namely the assumption a uniform substrate was ever realistic in the first place. The same policies apply whether traffic is hitting the public internet, moving between clouds, or routing between internal services, which isn't as architecturally elegant as "the mesh handles everything" but has the considerable advantage of matching the territory people are actually operating in.

There's a generational shift underneath the technical shift too, and it's worth naming directly. Engineers who came up during the cloud-native peak treated Kubernetes as the substrate and everything else as a migration target to be eventually retired, whereas engineers running platforms today treat Kubernetes as one substrate among several and don't expect the others to disappear on any reasonable timeline. They're not nostalgic about VMs and they're not waiting for the legacy systems to be rewritten, because they've accepted the rewrite isn't coming, the on-prem data center is staying, and the job is to make a heterogeneous estate behave like a coherent system regardless of where any particular workload happens to live.
That's a more mature stance than the one which preceded it, and it's the stance the next decade of platform work will be built on whether the vendors notice or not.

The interesting tooling now isn't the kind insisting on a particular substrate as a precondition for operating, it's the kind substrate-agnostic by design, running the same way on a bare-metal box, a VM, and a Kubernetes node, giving operators one place to manage all of it without pretending the others don't exist.

Cloud-native served a real purpose as a forcing function, pushing teams toward better operational habits and cleaner deployment models and real observability practices most of them wouldn't have adopted otherwise. As an identity it has outlived its usefulness, and the teams doing the best work right now aren't cloud-native or on-prem or hybrid in any meaningful self-description, they're just running infrastructure wherever it happens to live and treating the seams between environments as the actual product they're shipping.

Kubernetes policy engine — a buyer's guide

Josef Polar — Mon, 18 May 2026 04:00:00 +0000

Four options for enforcing policy as code in a Kubernetes cluster, evaluated on learning curve, feature breadth, Kubernetes-native fit, community health, and how much ongoing maintenance you are taking on.

OPA / Gatekeeper

Criterion	Score
Learning curve	4 / 10
Feature breadth	8 / 10
Kubernetes-native fit	6 / 10
Community health	8 / 10
Ops overhead	5 / 10

OPA is a general-purpose policy engine originally built for service mesh, API gateways, and CI/CD; Gatekeeper is the Kubernetes admission controller wrapper on top of it
Policies are written in Rego, a purpose-built declarative language that has a steep learning curve and is genuinely difficult to debug — most engineers need dedicated training to write non-trivial rules
Gatekeeper handles validation only; mutation support was added later and remains limited compared to other options
No native resource generation — you cannot use Gatekeeper to create dependent resources in response to policy events
Two separate projects to track, update, and understand (OPA and Gatekeeper), which adds operational surface
OPA itself is a CNCF graduated project with strong backing; Gatekeeper is actively maintained by a healthy community
The Rego investment pays off if you need unified policy enforcement across Kubernetes and non-Kubernetes systems (APIs, microservices, Terraform) from a single language

Stay on OPA/Gatekeeper if you have existing Rego expertise or if cross-stack policy unification is a hard requirement.

Kubewarden

Criterion	Score
Learning curve	5 / 10
Feature breadth	7 / 10
Kubernetes-native fit	7 / 10
Community health	6 / 10
Ops overhead	6 / 10

Policies compile to WebAssembly (WASM) modules, which means you can write them in Rust, Go, or any language with a WASM target — a genuine differentiator for teams with existing language preferences
The WASM compilation and module distribution workflow adds a build step that most policy workflows do not need and most platform teams do not want to own
Supports validate and mutate; resource generation support is limited
PolicyReports are produced in the standard Kubernetes format, which integrates cleanly with dashboards
Smaller community and slower release cadence than Kyverno or OPA/Gatekeeper; fewer production case studies
Depends on a controller manager plus an audit scanner plus a policy server, which is more moving parts than the problem warrants for most clusters

Worth considering for teams that already invest in WASM tooling or want to distribute policies as OCI artifacts without learning a policy-specific language.

jsPolicy

Criterion	Score
Learning curve	6 / 10
Feature breadth	5 / 10
Kubernetes-native fit	5 / 10
Community health	2 / 10
Ops overhead	5 / 10

Policies are written in JavaScript or TypeScript and executed in a V8 sandbox — execution is fast and the language is familiar to a wide audience
The project is maintained by Loft Labs and has seen no meaningful releases or community activity since mid-2024; it should be treated as effectively unmaintained
No path to CNCF graduation; not on any active roadmap; the GitHub repository has stalled issue backlogs and no active maintainer engagement
Supports validation and mutation but not resource generation or image signature verification
Running an unmaintained admission controller in a production cluster is a security liability — admission webhooks intercept every API request, and an unpatched webhook is a significant attack surface

Do not adopt jsPolicy for new deployments. Teams running it should be planning a migration.

Kyverno ✦ recommended

Criterion	Score
Learning curve	9 / 10
Feature breadth	10 / 10
Kubernetes-native fit	10 / 10
Community health	9 / 10
Ops overhead	8 / 10

Policies are Kubernetes resources — ClusterPolicy and Policy objects written in YAML, managed with kubectl, versioned in Git, and applied with Kustomize or Helm like any other manifest
No new language to learn; engineers already fluent in Kubernetes YAML can write functional policies on day one
Supports the full policy lifecycle: validate, mutate, generate, clean up, and verify image signatures — all from a single engine with a consistent resource model
Native image verification against Cosign and Notary signatures, with ImageValidatingPolicy as a dedicated CRD (as of 1.14), covering software supply chain security without an additional tool
Resource generation allows policies to create dependent resources automatically — default NetworkPolicies on namespace creation, for example — which no other tool in this group handles as cleanly
Automatically generates rules for Pod controllers (Deployment, StatefulSet, DaemonSet) from a single Pod-level policy, removing a whole class of coverage gaps
PolicyReports are generated automatically and integrate with Policy Reporter for dashboards and alerting
CNCF incubating project with active development: 1.15 shipped in August 2025, 1.16 in November 2025; over 850 merged changes across 70+ contributors in the 1.15 cycle alone
CLI supports offline policy testing in CI/CD pipelines against local manifests before anything reaches a cluster
The new CEL-based policy types introduced in 1.14–1.16 align with native Kubernetes ValidatingAdmissionPolicy and MutatingAdmissionPolicy, giving a migration path toward in-server validation for performance-sensitive rules while keeping the Kyverno authoring experience

The only meaningful limitation is that Kyverno is purpose-built for Kubernetes and cannot enforce policy across non-Kubernetes systems — if cross-stack Rego reuse is genuinely required, OPA remains the answer.

Criteria comparison

	OPA / Gatekeeper	Kubewarden	jsPolicy	Kyverno
Policy language	Rego	Rust / Go / WASM	JavaScript / TypeScript	YAML (CEL in newer types)
Validate	✓	✓	✓	✓
Mutate	Limited	✓	✓	✓
Generate resources	✗	✗	✗	✓
Image verification	✗	✗	✗	✓ (Cosign + Notary)
Auto-gen for Pod controllers	✗	✗	✗	✓
PolicyReports	✓	✓	✗	✓
CNCF status	Graduated (OPA)	Sandbox	None — unmaintained	Incubating
Non-K8s policy use	✓	✗	✗	Limited

For teams starting fresh, Kyverno is the clear choice — it covers the full policy surface, requires no new language, and has the most active community in this group. Teams with cross-stack Rego investment should evaluate whether that investment justifies staying on OPA/Gatekeeper. Everyone running jsPolicy should migrate.

Buyer's Guide - S3 compatible storage in Kubernetes

Josef Polar — Wed, 13 May 2026 12:59:20 +0000

Five options for teams running object storage inside a cluster, evaluated on Kubernetes integration, resource footprint, multi-site replication, license terms, and operational overhead.

MinIO

Criterion	Score
K8s integration	8 / 10
Multi-site replication	4 / 10
Resource footprint	4 / 10
Ops overhead	5 / 10
License clarity	4 / 10

Kubernetes operator exists and handles day-two operations like expansion and upgrades
Documentation is thorough; large community with most operational questions already answered
In 2024 MinIO relicensed from Apache 2 to BUSL-1.1 (Business Source License), which prohibits competing commercial use and converts to AGPL after four years — the project is effectively dead for most self-hosted use cases where the license previously made it attractive
The last Apache 2 release (AGPL-era builds) is still in use at many sites but receives no upstream security fixes
Multi-site replication was added after the fact and shows it — works, but feels bolted on
Resource appetite makes it a poor fit when storage nodes are shared with application workloads

The right call when throughput is the primary concern and you have dedicated storage infrastructure with real IOPS behind it.

Ceph / Rook

Criterion	Score
K8s integration	7 / 10
Multi-site replication	8 / 10
Resource footprint	2 / 10
Ops overhead	2 / 10
License clarity	8 / 10

Rook operator wraps Ceph well enough that direct Ceph configuration is no longer required
Unified block, filesystem, and object storage from a single cluster is a genuine advantage when you need all three
Minimum viable three-node cluster consumes upward of 8 GB RAM before any workloads run
Failure modes are numerous and tend to surface as subtle performance problems rather than obvious errors
Deep Ceph expertise is effectively a prerequisite for running this in production without surprises

Stay on Rook if you already have a Ceph investment and need block and filesystem access alongside object storage; hard to justify for teams that only need S3.

SeaweedFS

Criterion	Score
K8s integration	5 / 10
Multi-site replication	5 / 10
Resource footprint	7 / 10
Ops overhead	5 / 10
License clarity	7 / 10

Handles high file counts and small objects particularly well
Apache 2 license is clean and unambiguous for commercial use
Kubernetes support relies on third-party Helm charts with inconsistent maintenance across versions
Multi-datacenter replication model is more involved to configure than the problem warrants
Less operational tooling and community support than MinIO or Ceph at equivalent scale

Worth evaluating for workloads dominated by small files, but the Kubernetes story needs more investment before it belongs in a production cluster alongside maintained operators.

Zenko / Cloudserver

Criterion	Score
K8s integration	6 / 10
Multi-site replication	8 / 10
Resource footprint	4 / 10
Ops overhead	4 / 10
License clarity	5 / 10

Works well as an abstraction layer when you need a single S3 endpoint over multiple backends or cloud providers
Multi-site story is strong for the gateway use case specifically
Production deployment adds hard dependencies on Kafka and MongoDB — two more systems to operate and monitor
Mixed licensing across components adds ambiguity that the Apache 2 core alone does not resolve
Overhead is difficult to justify unless the multi-cloud gateway is the actual requirement

A reasonable choice if the specific problem is abstracting over multiple existing backends; not the right fit if you are building a storage tier from scratch.

Garage ✦ recommended

Criterion	Score
K8s integration	8 / 10
Multi-site replication	9 / 10
Resource footprint	9 / 10
Ops overhead	8 / 10
License clarity	9 / 10

Single static binary with no runtime dependencies — nothing to install beyond the process itself
Zone-aware placement is built into the cluster layout model, not added as an afterthought
Idle footprint on a three-node cluster sits around 128 MB RAM, making it viable on edge nodes and shared hardware
AGPL license with no commercial carve-outs, BSL clauses, or enterprise-tier feature gates
Helm chart is actively maintained and the configuration model stays legible under operational pressure
Failure modes are narrow enough that a single engineer can own the storage layer without treating it as a full-time job

The recommendation for most Kubernetes deployments, particularly those spanning more than one physical location or running on hardware where memory is shared with application workloads. It does exactly what it promises with very little ceremony.

Criteria comparison

	MinIO	Ceph/Rook	SeaweedFS	Zenko	Garage
K8s integration	Operator exists	Rook operator	Helm, patchy	Helm, complex	Clean Helm
Multi-site	Added late	Native	Manual setup	Gateway model	Built into layout
Min. footprint	~2 GB RAM	8+ GB RAM	~512 MB RAM	Needs Kafka/Mongo	~128 MB RAM
License	BUSL-1.1 (relicensed 2024)	LGPL / Apache	Apache 2	Apache 2 core, mixed	AGPL, consistent
Ops model	Moderate	High — deep Ceph knowledge	Moderate	Moderate plus deps	Low — clear failure modes

For teams with an existing Ceph investment or a hard throughput requirement and dedicated storage nodes, MinIO and Rook remain legitimate choices. For everyone else, Garage earns its recommendation.

Kubernetes felt overwhelming until I understood the layers

Josef Polar — Thu, 07 May 2026 20:40:00 +0000

When I first looked at Kubernetes, I saw a wall of YAML and acronyms. PVC, PV, StatefulSet, Endpoint — none of it connected. What helped was stepping back and mapping out what each piece actually does before trying to use any of it.

Here's how I think about it now.

A pod is the smallest unit

A Pod is the smallest thing Kubernetes runs. It's one or more containers running together on the same node, sharing the same network and storage.

apiVersion: v1
kind: Pod
metadata:
  name: my-app
  labels:
    app: my-app
spec:
  containers:
    - name: my-app
      image: nginx:latest
      ports:
        - containerPort: 80

In practice I rarely create Pods directly. They're disposable. If a Pod dies, nothing brings it back on its own.

A deployment keeps pods running

A Deployment manages Pods for me. I tell it what to run and how many replicas I want, and it keeps that many alive.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
        - name: my-app
          image: nginx:latest
          ports:
            - containerPort: 80

If a Pod crashes, the Deployment notices and spins up a replacement. It also handles rolling updates, swapping old pods for new ones without downtime.

Deployments are for stateless apps. If the app doesn't need to remember anything between restarts, a Deployment is the right call.

A StatefulSet is for apps with memory

StatefulSets are for apps that need stable identity: databases, queues, anything that writes to disk and needs to know who it is across restarts.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: my-db
spec:
  serviceName: my-db
  replicas: 3
  selector:
    matchLabels:
      app: my-db
  template:
    metadata:
      labels:
        app: my-db
    spec:
      containers:
        - name: my-db
          image: postgres:15
          ports:
            - containerPort: 5432

Unlike a Deployment, each pod in a StatefulSet gets a stable name: my-db-0, my-db-1, my-db-2. They start in order, and each one gets its own persistent storage that follows it around.

A service gives pods a stable address

Pods come and go. Their IP addresses change every time they restart. A Service gives me a stable endpoint that always points at the right pods.

apiVersion: v1
kind: Service
metadata:
  name: my-app-service
spec:
  selector:
    app: my-app
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: ClusterIP

The selector is what connects it to the pods. It matches the labels on my Deployment, and that's how the Service knows where to send traffic.

There are a few Service types. ClusterIP is only reachable inside the cluster and is the default. NodePort exposes a port on every node. LoadBalancer provisions an external load balancer, usually through a cloud provider.

Endpoints are what actually back a service

This one confused me for a while. An Endpoint is the list of pod IPs that a Service routes to. Kubernetes manages them automatically based on which pods match the selector and are healthy.

I don't create Endpoints manually, but I do inspect them:

kubectl get endpoints my-app-service

When a Service wasn't routing traffic, this was the first thing I checked. It told me immediately whether Kubernetes had found any pods to send traffic to.

PersistentVolume is the actual storage

A PersistentVolume is storage that exists independently of any pod. It could be a disk on the node, an NFS share, or a cloud disk. Kubernetes abstracts all of that behind a common interface.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: /data/my-app

PersistentVolumeClaim is how a pod requests it

A PVC is how a pod asks for storage. Instead of pointing directly at a PV, it says "I need 5Gi with ReadWriteOnce access" and Kubernetes finds a PV that fits.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi

Then I reference the PVC in my pod:

volumes:
  - name: my-storage
    persistentVolumeClaim:
      claimName: my-pvc

The split between PV and PVC is intentional. Whoever runs the cluster provisions the storage. Whoever deploys the app requests it. Neither needs to know the details of what the other did.

How it stacks up

Ingress
  └── Service  (stable network endpoint)
        └── Deployment / StatefulSet  (manages pod lifecycle)
              └── Pod  (runs the container)
                    └── PVC → PV  (persistent storage, if needed)

Endpoints sit between the Service and the Pods, managed automatically by Kubernetes. Once this hierarchy clicked, everything else started to feel like detail work rather than a new concept to learn.

Adding Ingress with HAProxy

Josef Polar — Wed, 29 Apr 2026 21:29:42 +0000

After getting the Service running, the next thing I needed was a way to route external HTTP traffic to it. That's where Ingress comes in — and I went with the HAProxy ingress controller.

Installing the HAProxy ingress controller

helm repo add haproxytech https://haproxytech.github.io/helm-charts
helm repo update
helm install haproxy-ingress haproxytech/kubernetes-ingress \
  --namespace ingress-controller \
  --create-namespace

Then I checked it came up:

kubectl get pods -n ingress-controller

Writing the Ingress resource

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-app-ingress
  annotations:
    kubernetes.io/ingress.class: haproxy
spec:
  rules:
    - host: my-app.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: my-app-service
                port:
                  number: 80

The host field is the domain I want to route. The backend.service.name matches the Service I created earlier — that chain of connections (Deployment → Service → Ingress) is the pattern that keeps repeating in Kubernetes.

Applying it

kubectl apply -f ingress.yaml
kubectl get ingress

Testing it locally

If I don't have a real domain yet, I can fake it by adding an entry to /etc/hosts:

127.0.0.1 my-app.example.com

Then hitting http://my-app.example.com routes traffic through HAProxy into the service and down to the pods.

Getting Started with Kubernetes: Creating a Basic Service

Josef Polar — Wed, 29 Apr 2026 17:31:11 +0000

Run an app in a container, then expose it so traffic could actually reach it. Two resources handle this: a Deployment and a Service.

Step 1: Writing the Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
        - name: my-app
          image: echo-server:latest
          ports:
            - containerPort: 80

This tells Kubernetes to run 2 copies of echo-server and label them app: my-app. The label part matters — I'll need it in a second.

Step 2: Writing the Service

apiVersion: v1
kind: Service
metadata:
  name: my-app-service
spec:
  selector:
    app: my-app
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: ClusterIP

The selector matches the label I set in the Deployment. That's how Kubernetes knows which pods to send traffic to. Once I understood that connection, the rest started clicking.

Step 3: Applying both files

kubectl apply -f deployment.yaml
kubectl apply -f service.yaml

Then I checked that everything came up:

kubectl get pods
kubectl get services

Two pods in Running state, and the service listed with a cluster IP. Good sign.

Step 4: Testing it locally

kubectl port-forward service/my-app-service 8080:80

Opened http://localhost:8080 in the browser and got the echo-server return page. It worked.