DEV Community: kube-gopher

Introducing Hearth v0.3.0: Scale Long-Tail LLMs to Zero on Private Kubernetes

kube-gopher — Mon, 20 Jul 2026 12:58:26 +0000

** An open-source, composable control plane for teams that want to stop paying the always-on cost of rarely used models.
**
Hearth v0.3.0(https://github.com/hearthproject/hearth/releases/tag/v0.3.0) is now available.

Most conversations about LLM serving focus on high-traffic models

large fleets, complex routing, distributed inference, and maximum throughput. But many private Kubernetes clusters face a different problem.

They have a limited pool of GPUs or NPUs, a handful of frequently used models, and a much longer tail of models that may receive only occasional requests. Keeping every model warm wastes expensive accelerator capacity. Scaling them down sounds simple—until the next request arrives.

Setting a Deployment to zero replicas is easy. Preserving the first client request while Kubernetes schedules a Pod, pulls an image, loads model weights, passes readiness probes, and begins serving tokens is not. Safe scale-down is equally difficult when requests are still streaming.

This is the lifecycle problem Hearth is designed to solve.

What is Hearth?

Hearth (https://github.com/hearth-project/hearth) is a minimal, composable LLM serving control plane for private Kubernetes clusters.

It is not an inference engine and does not implement accelerator kernels. It does not install device plugins or replace Kubernetes schedulers. It is also not trying to become a fleet-level AI serving platform.

Instead, Hearth owns the Kubernetes lifecycle between a model declaration and a working, scale-to-zero inference endpoint.

** Application owners create a namespaced LLMService describing:
**

the model source;
the desired runtime;
accelerator, CPU, and memory requirements;
caching and prewarming;
scaling behavior; and
cold-start endpoint behavior.

** Cluster administrators publish reusable, cluster-scoped InferenceRuntime profiles describing:
**

the serving image and arguments;
the device-plugin resource name;
node selectors and tolerations;
optional scheduler and queue configuration;
health probes; and
graceful termination behavior.

This separation keeps serving intent portable while leaving hardware and cluster details under administrator control.

From these two resources, Hearth reconciles the serving backend, a stable gateway, Kubernetes Services, model-cache resources, an optional prewarm Job, and KEDA autoscaling objects.

** Scale-to-zero that understands cold starts
**
Hearth keeps a lightweight gateway available while the model backend scales between zero and the configured maximum number of replicas.

The lifecycle works as follows:

When idle, KEDA keeps the model backend at zero replicas.
A request arrives at the stable Hearth gateway.
The gateway records demand and either holds the request or returns 503 with Retry-After, according to policy.
KEDA activates the backend through the default metrics API or the opt-in external-push scaler.
The backend loads its model from cache and becomes Ready only after the serving runtime passes its health checks.
The gateway forwards the request and streams the model response.
Sustained queue demand can scale the backend from one replica to multiple replicas.
When demand disappears, graceful draining protects in-flight streams before the backend returns to zero.

The gateway also provides bounded admission. When the queue is full, excess requests receive 429 instead of silently exhausting memory or connections. Cold-start activation has a deadline, and streaming requests can receive SSE heartbeats while waiting for the model.

In v0.3.0, the new external-push mode removes KEDA’s polling interval from the initial activation path. An activation lease preserves demand even when reject mode immediately returns a 503 or when KEDA reconnects its scaler stream.

The default metrics-based mode remains available for compatibility.

** Caching is part of the scale-to-zero design
**
Releasing an accelerator is useful only if restarting the model remains practical.

Hearth supports HostPath and NodeLocalPVC caches, optional prewarming for Hugging Face and ModelScope models, and pvc:// sources for model weights that are already staged in the cluster.

Prewarm Jobs inherit the runtime’s scheduling constraints but do not request an accelerator. This lets a cluster download model weights before an inference Pod claims a GPU or NPU.

Hearth also exposes stable gateway and backend metrics, while monitoring remains independent. Teams can deploy kube-prometheus-stack, another Prometheus-compatible system, or no monitoring stack at all without coupling it to the core reconciler.

** Real hardware, not only rendered YAML
**
Hearth’s adapters and controllers are covered by automated tests, including a CPU-only vLLM stub and complete scale-to-zero suites on Kind. But rendered manifests are not treated as proof of hardware support.

The project records physical validation separately.

On two NVIDIA A10 GPUs, the v0.3.0 release candidate completed the full 0→1→2→0 lifecycle using vLLM v0.25.1 and KEDA external-push. The run covered real inference, cache prewarming and persistence, bounded admission, reject mode, graceful draining, Volcano placement, component replacement, and host reboot recovery.

Six pending requests drove the backend from one to two replicas, with the two Pods placed on different physical GPUs. A 105-client cold burst admitted 100 requests and rejected the remaining five with the expected 429 responses.

** Hearth and Kthena: hot models and the long tail
**
Hearth does not need to replace a broader serving platform.

The project includes a real-hardware demo in which Kthena keeps a frequently used model warm while Hearth manages a long-tail model on the same Kubernetes cluster. The Hearth backend starts at zero, activates when a request arrives, serves real tokens, and returns to zero afterward.

This suggests a practical division of responsibility:

keep high-traffic, latency-sensitive models on a platform designed for fleet-level serving; and
place occasional or long-tail models behind Hearth’s smaller scale-to-zero lifecycle.

Volcano can schedule both workloads, but it remains an external integration rather than a dependency embedded in Hearth.

Watch the Hearth and Kthena operational demo (https://github.com/hearth-project/hearth/blob/main/docs/demo.md).

** Where Hearth fits today
**
Hearth v0.3.0 is alpha software with a serving.hearth.dev/v1alpha1 API.

It is best suited to internal, development, laboratory, and staging environments where:

accelerator capacity is limited or expensive;
some models receive intermittent traffic;
cold-start latency is acceptable;
workloads run inside a trusted private boundary; and
teams want a small Kubernetes-native control plane rather than a complete serving platform.

Hearth is not yet intended for shared multi-tenant clusters or public customer-facing endpoints. The gateway does not currently provide built-in authentication, and external-push mode requires one gateway replica until demand aggregation is implemented.

Fractional accelerators, HAMi sharing, MIG, and general multi-node topology have not been established by the current whole-device validation.

GitHub: github.com/hearth-project/hearth (https://github.com/hearth-project/hearth)
Release: v0.3.0 (https://github.com/hearth-project/hearth/releases/tag/v0.3.0)
Roadmap: ROADMAP.md (https://github.com/hearth-project/hearth/blob/main/ROADMAP.md)
Contributing: CONTRIBUTING.md (https://github.com/hearth-project/hearth/blob/main/CONTRIBUTING.md)

Hearth: scale-to-zero LLM serving on Kubernetes — and you can hack on it without a GPU

kube-gopher — Sun, 07 Jun 2026 12:59:16 +0000

Repo:github.com/hearth-project/hearth · Apache-2.0 · v0.1.0, alpha.

I've been building Hearth, a Kubernetes operator that serves open-source LLMs (Qwen, DeepSeek, GLM, …) declaratively and scales them to zero when idle. It's at a point where the core works end-to-end on real GPUs, and I'm looking for people to build it with me. The thing I most want you to know up front: you can contribute without owning an accelerator. More on that below.

## The one interesting problem

Self-hosting an LLM on K8s is easy until you notice the GPU is burning money while nobody's using the model. The obvious fix — "scale to zero" — runs straight into a chicken-and-egg problem: a stock HPA can't scale up from zero, because zero replicas means zero metrics, which means it never wakes up.

Hearth puts a small gateway (an OpenAI-compatible reverse proxy) in front of each model. When a request arrives at a scaled-to-zero backend, the gateway accepts it, holds the connection open (SSE keepalive heartbeats so nothing times out), and bumps a pending counter exposed at /hearth/queue. KEDA polls that endpoint, sees pending > 0, and scales the backend 0 → 1. The pod loads weights from a warm cache, becomes Ready, and the gateway forwards the buffered request and streams tokens back. Idle again → KEDA scales it back to 0.

The whole thing is one manifest:

  apiVersion: serving.hearth.dev/v1alpha1
  kind: LLMService
  metadata: { name: qwen3-8b, namespace: ai }
  spec:
    model:
      source: { uri: modelscope://Qwen/Qwen3-8B-Instruct }   # or hf://
    runtime:
      selector: { vendor: [nvidia, ascend] }   # auto-pick a backend, in order
    resources: { accelerators: 1 }
    scaling: { min: 0, max: 3, metric: queueDepth, target: 10 }

  $ kubectl get llmservice -n ai
  NAME       PHASE          RUNTIME       REPLICAS   AGE
  qwen3-8b   ScaledToZero   vllm-nvidia   0          30s

It's deliberately vendor-neutral: backends (NVIDIA-vLLM, vLLM-Ascend, …) are described as data in a cluster-scoped InferenceRuntime CRD — image, args, the device-plugin resource name, probes, metrics paths. Adding a chip is a thin adapter that does K8s-layer adaptation only; it never re-implements vLLM or touches kernels. The same LLMService is meant to run unchanged on NVIDIA or Ascend.

Hearth deliberately stays in its lane: it's the K8s orchestration/lifecycle layer. The engine is vLLM; scheduling is device-plugins / HAMi / Volcano; datacenter-scale serving is KServe / llm-d Hearth is the few-GPU, scale-to-zero, private end of that spectrum.

Why you can contribute without a GPU

This is the part I'm proud of and the reason I'm posting. A vendor-neutral project is useless to contributors if every change needs a rack of hardware. So there's a full no-GPU test path: a CPU vllm-stub that fakes startup delay, streaming, and /metrics, plus a fake extended resource on the node. On a plain kind cluster, with no accelerator, one command —

make test-scale-e2e

— runs the entire 0 → 1 → N → 0 loop, including cold-start keepalive and graceful drain. A laptop is enough to develop and verify the core behavior.

Honest status

I won't oversell it. As of v0.1.0:

Works, verified end-to-end on real NVIDIA GPUs: multi-backend abstraction, model caching/prewarm, gateway + KEDA scale-to-zero, cold-start keepalive, graceful drain, 1→N autoscaling, Helm install, Grafana dashboard.
Scaffolded + golden-tested, not yet on real hardware: the Ascend backend renders correct manifests but hasn't been validated on real NPUs. This is the big v1 gap, blocked purely on hardware access.
Not there yet: auth, multi-tenancy. It's v1alpha1 and not production-ready — a strong fit today for internal/dev, latency-tolerant, cost-sensitive serving.

Where I'd love help

Got Ascend (or Cambricon) hardware? Validating the Ascend backend on a real NPU is the single most valuable thing right now.
No special hardware? Grab a good-first-issue (https://github.com/hearth-project/hearth/issues) — the no-GPU path above means you can build, test, and verify locally.
Just curious? Try the kind quickstart, poke holes, open an issue, or ⭐ and follow along.

If any of this resonates, the Welcome issue (#1)(https://github.com/hearth-project/hearth/issues/1) is the place to
say hi. Thanks for reading.

Your models, your hearth. 🔥

Idle GPUs also burn money — a Kubernetes Operator that can scale large models down to zero

kube-gopher — Thu, 04 Jun 2026 14:30:29 +0000

It's early — come build it with me

Hearth is moving fast and contributions are very welcome — especially validating the Ascend backend on real NPUs, plus the roadmap's P0/P1 items. There are good first issues waiting.

⭐ Star + follow along: github.com/hearth-project/hearth

Your idle GPUs are burning money. Here's a Kubernetes operator that fixes it

If you self-host open-source LLMs on Kubernetes, you've hit the same wall I did:

A GPU pinned to a model that gets traffic 3 hours a day still costs you 24 hours a day.
Every serving stack assumes NVIDIA-first, English-first — awkward if you're running Qwen, DeepSeek, or GLM, or deploying on Ascend / domestic chips.
"Just use KServe" means dragging in Knative + Istio to serve one model on one GPU.

🔥 Hearth — a vendor-neutral Kubernetes operator that turns "run Qwen on my private cluster" into a single LLMService manifest, with scale-to-zero built in.

One manifest. Scale-to-zero. Pick your chip.

apiVersion: serving.hearth.dev/v1alpha1
  kind: LLMService
  metadata:
    name: qwen3-8b
    namespace: ai
  spec:
    model:
      source:
        uri: modelscope://Qwen/Qwen3-8B-Instruct
    runtime:
      selector: { vendor: [nvidia, ascend] }   # auto-pick a backend
    resources:
      accelerators: 1
    scaling:
      min: 0        # 👈 scale-to-zero
      max: 3
      metric: queueDepth
      target: 10

$ kubectl apply -f qwen3-8b.yaml
$ kubectl get llmservice -n ai
NAME       PHASE          RUNTIME       REPLICAS   AGE
qwen3-8b   ScaledToZero   vllm-nvidia   0          30s

When a request arrives, Hearth's gateway buffers it, scales the model 0 → 1, holds the client connection alive with SSE heartbeats through the cold start, then streams tokens back. Idle again? Back to zero GPUs.

The same manifest runs on an Ascend cluster by making vllm-ascend the available runtime — no spec change. That portability is the whole point.

What makes it different

Hearth deliberately does not re-implement the things that already work:

Backends are described declaratively in a cluster-scoped InferenceRuntime (image, args, accelerator resource, probes,metrics). Adding a new chip is a thin adapter — not a rewrite.

What's actually working today

I'm being honest about maturity — this is pre-release v0.1.0 (alpha):

✅ NVIDIA backend + the full scale-to-zero path verified end-to-end on real A100s — cold-start keepalive, graceful drain (in-flight streams survive scale-down), model caching/prewarm, 1→N autoscaling, Grafana dashboard.
🧪 Ascend backend is scaffolded and golden-tested (renders correct manifests) — real-NPU validation is the v1 milestone.
⚠️ Not production-ready yet: no auth, no multi-tenancy. It's a strong fit today for internal / dev,latency-tolerant, cost-sensitive serving — scale-to-zero packs many idle models onto few GPUs.

Try it in 60 seconds — no GPU required

You can exercise the whole control plane on kind:

make install      # CRDs into your kube-context
make run          # run the operator
kubectl apply -f config/samples/serving_v1alpha1_inferenceruntime.yaml
kubectl apply -f config/samples/serving_v1alpha1_llmservice.yaml -n ai