Dynamic Local Persistent Volumes on Kubernetes via Open Service Broker

laoshanxi — Fri, 09 Jan 2026 00:59:05 +0000

Shared storage works well for many workloads, but once latency and IO consistency start to matter, local disks become very attractive.

Kubernetes supports Local Persistent Volumes (Local PVs), but with a big limitation:

Local PVs must be statically provisioned.

That makes them hard to use in dynamic environments where workloads are created on demand.

We ran into this problem while trying to expose local storage through an Open Service Broker interface.

Why Static Local PVs Are a Problem

With static provisioning:

PVs must exist before workloads request them
Capacity planning becomes manual
Automation pipelines break down

For service brokers and self-service platforms, this is a non-starter. Users expect storage to be provisioned dynamically.

The Approach We Took

Instead of fighting Kubernetes’ design, we worked around it.

The key idea was to separate:

Scheduling decisions (still done by Kubernetes)
Disk creation (done on the target node)

The Provisioning Flow

At a high level, our workflow looked like this:

A service broker receives a request for local storage
The broker submits a temporary “dummy” Kubernetes manifest with: resource requirements node affinity
Kubernetes schedules the workload to a specific node
Once the node is known, the broker: remotely creates the local disk generates the corresponding Local PV object
The real workload is deployed and bound to that PV
When the service is deleted, the local disk is cleaned up

This gave us something that felt like dynamic provisioning, even though Local PVs remain static under the hood.

Why This Worked

Kubernetes still decides placement
Disk creation happens only where needed
No pre-provisioning of unused capacity
Storage lifecycle is tied to the service instance

It’s not as elegant as a CSI driver, but for on-prem and hybrid clusters, it proved to be a practical solution.

Trade-offs and Lessons Learned

There are trade-offs:

Requires node-level access
Cleanup must be handled carefully
Failure paths need extra attention

But in exchange, we got predictable performance and a much better developer experience for stateful workloads.

When This Pattern Makes Sense

This approach works best when:

you control the cluster
IO performance matters
cloud block storage isn’t an option
service brokers are part of your platform

For many internal platforms, this turned out to be “good enough” — and far better than manual PV management.

Open Source Implementation

We documented and open-sourced this approach as part of a larger platform project:

👉 https://github.com/laoshanxi/app-mesh/blob/main/docs/source/success/open_service_broker_support_local_pv_for_K8S.md

If you’ve built dynamic storage workflows around Local PVs (or decided not to), I’d love to hear what worked — and what didn’t.

Running Native (Non-Container) Workloads on Kubernetes: A Practical Experiment

laoshanxi — Fri, 09 Jan 2026 00:51:17 +0000

Kubernetes is excellent at orchestrating containers. But every now and then, you run into workloads that simply don’t fit well into the container model.

In our case, we had several native binaries and host-level tools that needed to:

run on specific nodes
access host resources directly
integrate with existing CI/CD pipelines
follow Kubernetes-style retries and lifecycle management

Containerizing them felt forced. Privileged containers introduced security concerns, and tightly coupling containers to the host defeated the purpose of abstraction.

So we tried a different approach.

The Problem with “Just Containerize It”

In theory, everything can be containerized. In practice, that often means:

privileged mode
direct host mounts
fragile assumptions about the host environment
unclear ownership when jobs fail or restart

At that point, Kubernetes is mostly being used as a scheduler and lifecycle tracker, not as an isolation boundary.

We wanted to keep the good parts of Kubernetes — Jobs, retries, observability — without forcing native workloads into an unnatural container shape.

The Core Idea

Instead of running the workload inside the container, we flipped the model:

Kubernetes Jobs are still the scheduling primitive
The container acts as a thin command forwarder
The actual workload runs as a native OS process on the node
From Kubernetes’ perspective, nothing unusual is happening:
Jobs start
Jobs finish
Exit codes are recorded

Under the hood, the Job lifecycle is mapped to a host-level process.

How It Works (High-Level)

A lightweight agent runs on each node, exposing a local control interface
A Kubernetes Job starts a small container
That container forwards the command to the local agent
The agent launches and monitors the native process
Job success or failure reflects the process exit code

This keeps Kubernetes in control of when and where things run, while the host controls how they run.

What Worked Well

This approach gave us a few practical wins:

No privileged containers
Native tools run exactly as they expect
Kubernetes still provides retries, logs, and status
CI/CD pipelines remain unchanged

For legacy tooling or migration phases, this turned out to be surprisingly effective.

What Was Hard

The hardest part wasn’t execution — it was lifecycle correctness.

Node restarts
Job retries
Partial failures

All of these can leave orphaned processes behind if ownership isn’t carefully designed. We ended up treating Kubernetes Jobs as lifecycle signals, while enforcing stricter cleanup logic on the host side.

It’s not a perfect abstraction — but it’s an honest one.

When This Pattern Makes Sense

This isn’t a replacement for containers. It works best when:

workloads are hard to containerize
host-level access is unavoidable
you want Kubernetes semantics without container overhead

For fully cloud-native services, containers are still the right answer. For everything else, this can be a pragmatic bridge.

Open Source Implementation

We eventually open-sourced the tooling we built around this pattern, since it kept repeating across teams:

👉 https://github.com/laoshanxi/app-mesh/blob/main/docs/source/success/kubernetes_run_native_application.md

I’m curious how others approach native workloads in Kubernetes — especially in environments with frequent node churn.

DEV Community: laoshanxi

Dynamic Local Persistent Volumes on Kubernetes via Open Service Broker

Why Static Local PVs Are a Problem

The Approach We Took

The Provisioning Flow

Why This Worked

Trade-offs and Lessons Learned

When This Pattern Makes Sense

Running Native (Non-Container) Workloads on Kubernetes: A Practical Experiment

The Problem with “Just Containerize It”

The Core Idea

How It Works (High-Level)

What Worked Well

What Was Hard

When This Pattern Makes Sense

Open Source Implementation