ScaleScope — Watching Autoscaling Actually Happen, in Real Time

#node #webdev #hackathon #zerops

Built for a Zerops hackathon. Repo status: full backend built, syntax-checked,
and run end-to-end against real infrastructure. Here's what it does, why it
exists, and what it took to get it there.

The problem: autoscaling is a black box until it isn't

Every cloud platform promises "autoscaling." You set a CPU threshold, containers
appear, containers disappear, and you're told to trust the process. But as a
developer, you almost never see it happen. You check a dashboard five minutes
later, notice the container count went from 1 to 4, and have to reconstruct the
story after the fact from logs and graphs that don't line up in time.

I wanted to build something that made autoscaling watchable — a system
where you press a button, fire real load at a real service, and see the
container count climb and fall live, second by second, with the request
latency and throughput lines moving right alongside it. Not a synthetic demo.
Not a mocked graph. An actual service, actually scaling, on actual
infrastructure, with the causal chain visible the entire time.

That's ScaleScope — a live autoscaling proving ground built on and about
Zerops.

What it actually does

You open a console, pick a load profile (steady, ramp, or spike), set a
duration, and hit start run. From that moment:

A load fleet of worker containers fires requests at a target service
The target burns real CPU per request (tunable), and Zerops' own autoscaler decides when to add or remove containers — nothing about the scaling decision is faked or simulated
Every container mints a UUID at boot and stamps it on every response, so the actual live container count is measured from HTTP responses, not self-reported by a privileged API
A digital twin ("the oracle") watches the same telemetry and predicts the container count ~15 seconds ahead, so you can compare "what the twin expected" against "what Zerops actually did"
Every run is recorded as an append-only event log, so it can be replayed afterwards through the exact same rendering pipeline that shows live traffic — the frontend has no code path that knows whether it's watching now or three hours ago
A chaos module can kill, degrade, or partition the target mid-run, so you can watch the system recover in real time, not just scale up
A scheduler can run unattended experiment suites — sweep a load curve, search for the "knee" where latency breaks down, or run a regression check — turning one run into a whole family of runs with zero babysitting

Twelve services total. Every one is load-bearing; none of them exist to pad
the architecture diagram.

The architecture

web (Story / Console / Lab)
  │  SSE (live or replayed — same events, same reducer)
  ▼
gateway ──── PLAIN ────▶ worker (fleet, N containers)
  │ REST + barrier                    │ fires load, measures latency
  │                                   ▼
  │                                 target  ◀── Zerops scales this
  │                                   │  X-Instance-Id / X-Instance-Age
  │                                   ▼
  │                          worker measures, publishes samples
  │                                   │ QUEUE
  │                                   ▼
  │                              collector ──▶ ClickHouse (samples)
  │                                   │ merged tick frames
  │                                   ├──▶ oracle (prediction)
  │                                   └──▶ gateway ──▶ SSE broadcast
  ▼
Postgres (run registry) ◀── folded from the JetStream event log at finalisation

gateway — control plane: admission, a two-phase start barrier (PREPARE → workers warm up and ack READY → GO with an absolute T0), SSE fan-out, REST
collector — ingest: merges fleet samples into per-second tick frames, writes them to ClickHouse
worker — the load fleet: fires the chosen profile, or runs a latency-target autopilot (a PID controller that adjusts request rate to hold a target p95)
target — the service actually under test, the one Zerops scales
oracle — a digital twin doing online parameter estimation, not ML — it learns the autoscaler's real behavior from observed ticks and refuses to persist what it's learned from runs shorter than 20 ticks, so a handful of noisy samples can't corrupt a model future runs depend on
chaos — fault injection: kill / degrade / partition the target on command, authenticated with a timing-safe shared secret
scheduler — runs unattended experiment suites through the gateway's own public API, like any other client would
nats (JetStream) — every run is one append-only event log: created, armed, started, one tick per second, scaled, slo, chaos, prediction, completed
Postgres / ClickHouse / Valkey — three purpose-built projections of that one event log, not three independent sources of truth. Postgres is the run registry, ClickHouse holds every observed sample for aggregate queries, Valkey is the live materialized view (rolling container window, run locks, credit budget)

Replay works by re-emitting the same JetStream log at its original pace into
the identical SSE pipe that live traffic uses. That single design decision —
one event stream, one reducer, two sources (now or the log) — is what let the
whole story/console/lab frontend be built without a single "if this is a
replay" branch.

The two tricks that actually make the scaling demo work

1. Force horizontal scaling, not vertical. Zerops scales vertically first
— more CPU inside the container — and only adds containers once that ceiling
is hit. That's the right default for most workloads, but it's exactly wrong
for a demo whose whole point is to watch containers appear. The fix: cap
target's vertical ceiling low and set cpuMode: DEDICATED — Zerops' own
docs specify that the horizontal-trigger CPU thresholds (minFreeCpuPercent
and friends) only apply to dedicated CPU. Miss this one setting and the
system will vertically scale forever and never add a second container. (This
was a literal bug in an earlier draft of the import YAML — the kind of thing
that's silent until you're staring at a container count stuck at 1 during a
demo.)

2. Count containers without a privileged platform API. There's no
Zerops API call that hands you "how many containers are running right now"
from inside the app layer, and even if there were, depending on it would be
fragile and slow. Instead, every target container mints a UUID at boot and
returns it as an X-Instance-Id header on every response. Count the distinct
IDs seen in a rolling ten-second window, and you have the live container
count — measured from real traffic, not self-reported, and it needs zero
platform credentials. An X-Instance-Age header on top of that lets the
system reconstruct a full container lifecycle swimlane purely from HTTP
response headers.

Challenges along the way

Env vars cached before they're set. Every service logged with the tag
[svc] instead of its real name ([gateway], [worker], etc.) — completely
harmless for correctness, brutal for debugging eight interleaved service logs
during a live demo. The cause: the shared telemetry package cached
SCALESCOPE_SERVICE at module-load time, but in ES modules, imports resolve
before the entry point's own code runs — so by the time each service set
that env var, the logger had already cached the wrong value. Fixed by reading
the env var fresh on every log call instead of caching it once.

A Postgres type mismatch that would have broken every single run. An
update statement used one query parameter in two different type contexts —
as a bigint in one clause, implicitly cast to numeric inside
to_timestamp() in another — without an explicit cast. node-postgres
throws rather than guessing which type you meant, so the very first
POST /api/runs in a real end-to-end test failed outright. This is exactly
the kind of bug that reads as correct in review and only shows up when a
real driver refuses to guess.

A status panel that "fixed itself" into being wrong. The architecture
panel showed the gateway and NATS as unknown about ten seconds after every
page load. The cause was a heartbeat-reconciliation loop overwriting the
result of a direct health check with a heartbeat-table lookup — for two
services nothing ever populates a heartbeat for. Fixed by excluding gateway
and NATS from that reconciliation and re-asserting their direct checks on
every poll tick.

None of these three bugs would have been caught by re-reading the code
carefully. They only showed up by actually running the full pipeline against
real infrastructure — which is exactly why that run happened before writing
the test notes, not after.

Designing for partial infrastructure. ClickHouse wasn't available in the
local dev environment at all, so instead of skipping that path, I deliberately
left it unreachable and let the local end-to-end run happen anyway — to prove
that "ClickHouse is best-effort at boot" actually holds under a real failure,
not just in the source. It did: gateway and collector both logged a
warning and kept running, no crash, no cascading failure. That's the one
honest gap going into the live demo — the ClickHouse-backed read paths are
code-reviewed and syntax-checked, not yet exercised against a live ClickHouse
instance, and that's called out explicitly rather than papered over.

Key learnings

Run the thing before you trust the thing. All twelve services passed node --check — pure syntax validation — from very early on. That told me almost nothing. All three real bugs above were caught only by actually driving load through the full pipeline: barrier → load fleet → target → collector → event log → finalisation. Static checks and code review are necessary; they are nowhere near sufficient for a distributed system with a live event stream at its core.
One event log beats three sources of truth. Making Postgres, ClickHouse, and Valkey each a projection of the same JetStream log — rather than three services independently deciding what happened — is what made replay basically free. If they'd each tracked state independently, replay would have needed a special code path instead of just re-emitting the log slower.
A public "start run" button on a credit-billed backend needs a ceiling, by design, not as an afterthought. Runs are capped server-side (duration, cooldown, concurrency), and the gateway enforces an hourly run budget plus a single-active-run lock — both stored in Valkey so the limits hold even if the gateway itself scales to multiple containers. Building that in from the start was cheaper than retrofitting it after a demo got hammered.
Read the platform's own scaling docs before you build the demo that depends on them. The cpuMode: DEDICATED requirement wasn't something I could have guessed from first principles — it's a specific, documented Zerops behavior, and getting it wrong makes the entire premise of the project ("watch it scale") silently fail.

The hackathon experience

Building ScaleScope meant treating the infrastructure platform itself as
the feature, not just the thing an app happens to run on. That's a different
kind of build than "here's an app, deployed somewhere." It meant reading
Zerops' scaling documentation closely enough to find the dedicated-CPU
requirement, designing a container-counting mechanism that works with zero
platform privileges, and building failure tolerance (the ClickHouse
best-effort boot) into the architecture from day one rather than bolting it
on.

The most satisfying moment was watching the very first real end-to-end run
complete — worker warmed, run started, collector logging scaled 0 -> 1,
and the console's container tile actually moving — and realizing that moment
only existed because three real bugs had already been caught and fixed by
runs just like it, minutes earlier.

If you're building something similar: get infrastructure up and get one real
signal flowing through the whole pipeline as early as possible. Every hour
spent looking at code that hasn't actually run yet is an hour where a
type-cast bug or a module-load-order bug is sitting there, waiting for the
worst possible moment — a live demo — to show up.

ScaleScope is built on Zerops. If you want to see
autoscaling stop being a black box, that's the whole point of the project.