Moshe Avdiel

Posted on Aug 13 • Originally published at github.com

Live Trace Sample Per Thousand for Energy — Mouth to Process in Seconds

The Aha: samplePerThousand is not a property file trophy. It is incident posture — and posture that waits for a jar is already late.

Black Friday does not accept 'we'll ship the knob tomorrow.' Neither should your control plane.

Domain: grid dispatch, demand response. This essay maps hub key samplePerThousand to trace sample per thousand so the lesson stays concrete for energy operators.

The problem: ceremony between judgment and effect

You already know the right number. Everyone in the war room knows the right number. What you do not have is a path from mouth → running process that is shorter than a release train.

When trace sample per thousand is frozen in YAML, every incident becomes a process argument. When it lives in a hub with clamps, the argument ends and the work begins.

Belief	Production
GitOps will handle it	Git is a ledger, not a pager for second-scale posture
Defaults are fine	Defaults become root causes under load
The mesh owns that	Mesh freezes share to another YAML dialect
We can SSH and edit	That is not an audit trail; that is folklore

The Aha: local read, live write

Kiponos.io holds the tree. The Python SDK keeps the latest value in memory, patched over WebSocket deltas. Hot path: local get — no per-request hub RTT.

examples/
  ops-energy-trace-sample/
    samplePerThousand: 10   # trace sample per thousand
    hardMax: compiled-in-app
    failClosed: true

policy = kiponos.path("examples", "ops-energy-trace-sample")
samplePerThousand = min(int(policy.get("samplePerThousand", 10)), HARD_MAX)
worker_pool.resize(samplePerThousand)  # or admission semaphore

Ops sets samplePerThousand in the dashboard (or automation writes the same path). The next evaluation uses the new value. Same jar. Same tests for structure.

What stays in the jar vs the hub

Jar (versioned)	Hub (live)
Code paths & clamps	Operational numbers
Hard maxima / allowlists	Current posture
Schema & types	Human judgment under pressure
Fail-closed defaults	Temporary incident overrides

Architecture

Dashboard / automation ──write──► Kiponos hub tree
                                      │ WebSocket delta
                                      ▼
                               SDK in-process cache
                                      │ local get
                                      ▼
                               Hot path decision (trace sample per thousand)

No sidecar tax on every request. No second product for "just this one dial."

Clone and learn the pattern

git clone https://github.com/kiponos-io/kiponos-io.git
# See examples/java/* for runnable Super Pattern / Aha modules
# Profile: ['app']['release']['env']['config'] — same shape as production

Getting started: GETTING-STARTED.md · Product: kiponos.io

Scenarios

Moment	Frozen YAML	Live hub
Incident	PR + pipeline	Seconds
Peak event	Over-provision	Dial down/up
Experiment	Long-lived branch	Same jar
Rollback	Redeploy previous	Revert hub value
Region skew	Copy three files	Per-folder values

When not to live-edit

Protocol or schema changes that need coordinated rollouts
Values that compliance requires code-reviewed only
Anything you cannot clamp or allowlist safely
Secrets (use a secret manager — never the ops posture tree)

Live knobs are for posture, not for inventing untested systems under fire.

Operational checklist

Name the hub path so humans find it under pressure (examples/ops-energy-trace-sample/samplePerThousand).
Default safely when the hub is unreachable (fail closed on money paths).
Allowlist writers (dashboard roles + automation identities).
Log the decision, not every get.
Rehearse the flip in staging with a sibling example module.
Document the one-line kill path (revert key).
Record from→to + reason code in the incident timeline.

Why this is not "just another flag"

Feature flags are often product gates. This essay is about ops posture on a hot path: trace sample per thousand for energy — numbers humans already change verbally in war rooms.

Kiponos makes that verbal decision executable without a second control plane tax on every request.

Observability you actually need

Ship counters with the key path baked in: decisions applied, rejects, and hub write events. Logging every local get teaches nothing; logging every change teaches ownership.

Where to enforce

Prefer the earliest durable hop that still knows identity. Edge hard-caps stay as seatbelts; app middleware reads live posture under that seatbelt. Do not invent a third control plane.

Failure budget vs this dial

Treat samplePerThousand as a slice of error budget, not a comfort blanket. Raise it when the dependency is healthy; lower it when the dependency is already sick. Write the number that survives a bad day, not the number that flatters a sunny demo.

A note on testing

Unit-test structure with fixed strings (no network). Integration-test the hub path against the public sandbox when you can.

Good tests:

Defaults when keys are missing
Clamps reject out-of-range values
Fail-closed behavior for money paths

Bad tests:

Hitting production hubs from CI
Asserting wall-clock times for WebSocket delivery

Closing note

Architecture diagrams do not absorb incidents. Steerable posture does — with audit, clamps, and a revert path written before you need it.

One-line runbook

Who may move this key under P1, what is the clamp, what is the revert? Write that sentence before you need it. Posture without a revert path is just another outage mode.

Moral

Isolation patterns without steerable thresholds are beautiful diagrams that lose to Tuesday.

Ship judgment. Leave the jar alone.

Series: Kiponos live ops posture · Pattern library: kiponos-io/docs · SDK examples: examples/java

DEV Community