Backyard Endurance OS: Designing Zero-Loss Telemetry Ingestion for Athletes and Distributed Systems

Run_as_daemon — Sat, 12 Sep 2026 05:14:24 +0000

Problem Statement

At lastyard.space, we treat endurance telemetry like production infrastructure: if the runner is still moving, the system must still be receiving, ordering, persisting, and replaying data.

A recent live trail run entered the ingestion path with these facts:

Workout: Trail Run
Distance: 21.1 km
Duration: 115.0 min
Avg HR: 155 bpm
Max HR: 178 bpm
Elevation gain: 450 m
Cadence: 166 spm
Received: 2026-09-12T04:58:48.126630+00:00

That is not just fitness data. It is a distributed systems trace.

A 21.1 km effort over 115 minutes is an event stream under metabolic load. The athlete averages 155 bpm, peaks at 178 bpm, climbs 450 m, and still holds 166 spm cadence. The architecture equivalent is a service operating near threshold: high utilization, sustained pressure, burst spikes, and degradation risk if pacing is wrong.

Backyard Ultra gives us the governing principle: Fallen does not mean finished. In system terms, a failed edge collector, delayed watch upload, flaky LTE hop, or overloaded database writer must never mean telemetry loss. It means graceful degradation, replay, backpressure, and recovery.

Architectural Diagram

 Apple Health / Watch
        |
        | batched samples, workout summary, HR dynamics
        v
 Mobile Edge Collector
        |
        | WireGuard tunnel, mTLS, retry queue
        v
 Traefik Ingress
        |
        | HTTP/2 gRPC + SSE fanout
        v
 Ingestion Gateway
        |
        | idempotency key: athlete_id + workout_id + sample_ts
        v
 Durable Event Log
        |
        | append-only PostgreSQL partitioned table
        v
 Stream Processors
        |
        | HR drift, cadence variance, elevation-normalized pace
        v
 Timeseries Views + Live SSE
        |
        v
 Crew Dashboard / Race Control / Alerts

The core shift is simple: stop treating telemetry ingestion as CRUD. Treat it as high-availability timing infrastructure.

In-depth Technical Breakdown

The ingestion layer has four duties:

Accept delayed, duplicated, and out-of-order samples.
Persist before processing.
Preserve athlete-time semantics, not server-arrival semantics.
Surface live data without coupling the dashboard to the write path.

The mobile edge collector sends telemetry over WireGuard where possible. The public ingress is Traefik, terminating TLS and routing gRPC ingestion separately from SSE read streams.

http:
  routers:
    telemetry-grpc:
      rule: "Host(`ingest.lastyard.space`)"
      entryPoints: ["websecure"]
      service: telemetry-ingest
      tls: {}
    telemetry-live:
      rule: "Host(`live.lastyard.space`)"
      entryPoints: ["websecure"]
      service: telemetry-sse
      tls: {}

  services:
    telemetry-ingest:
      loadBalancer:
        servers:
          - url: "h2c://ingest-a:8080"
          - url: "h2c://ingest-b:8080"
    telemetry-sse:
      loadBalancer:
        servers:
          - url: "http://sse-a:8090"
          - url: "http://sse-b:8090"

For endurance telemetry, exactly-once delivery is an illusion at the network edge. The practical invariant is at-least-once delivery plus idempotent writes.

CREATE TABLE telemetry_events (
  athlete_id uuid NOT NULL,
  workout_id uuid NOT NULL,
  sample_ts timestamptz NOT NULL,
  received_at timestamptz NOT NULL DEFAULT now(),
  source text NOT NULL,
  metric text NOT NULL,
  value double precision NOT NULL,
  unit text NOT NULL,
  seq bigint,
  payload jsonb NOT NULL,
  PRIMARY KEY (athlete_id, workout_id, metric, sample_ts)
) PARTITION BY RANGE (sample_ts);

CREATE INDEX telemetry_events_workout_ts_idx
  ON telemetry_events (workout_id, sample_ts DESC);

The primary key is the antidote to duplicate mobile retries. If Apple Health resends a heart-rate sample after reconnect, it is merged instead of double-counted. If LTE stalls on a climb, sample_ts preserves the athlete's actual timeline while received_at exposes transport lag.

The writer must apply backpressure before PostgreSQL becomes the athlete equivalent of cardiac drift. Sustained overload should lengthen queues, reject low-priority derived metrics, and preserve raw samples.

func ingest(ctx context.Context, e Event) error {
    if rawQueue.Depth() > highWatermark {
        if e.Kind != "raw_sample" {
            return ErrBackpressure
        }
    }

    _, err := db.ExecContext(ctx, `
      INSERT INTO telemetry_events
        (athlete_id, workout_id, sample_ts, received_at, source, metric, value, unit, seq, payload)
      VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10)
      ON CONFLICT (athlete_id, workout_id, metric, sample_ts)
      DO UPDATE SET
        received_at = LEAST(telemetry_events.received_at, EXCLUDED.received_at),
        payload = telemetry_events.payload || EXCLUDED.payload
    `, e.AthleteID, e.WorkoutID, e.SampleTS, e.ReceivedAt, e.Source,
       e.Metric, e.Value, e.Unit, e.Seq, e.Payload)

    return err
}

This is the same discipline as pacing a long race. You do not sprint every hill because your max heart rate allows 178 bpm. You regulate output around sustainable threshold. Likewise, ingestion should not burn database connections to keep vanity dashboards smooth. Raw telemetry survives first. Derived views catch up.

eBPF and Runtime Observability

Heart-rate drift is when effort stays constant but physiological cost rises. Distributed systems have the same pattern: request rate is flat, but latency, retransmits, syscall time, or lock contention climbs.

We use eBPF probes to detect the infrastructure version of HR drift:

signals:
  tcp_retransmits_per_minute
  grpc_server_handling_seconds_bucket
  postgres_wal_fsync_seconds
  run_queue_latency_ms
  conntrack_drops_total
  sse_client_lag_seconds

If the athlete's HR rises from 155 toward 178 while cadence drops from 166 spm, the model flags fatigue. If the system's p95 ingest latency rises while throughput is flat, the platform flags architectural fatigue.

Both are threshold stories. Both require intervention before collapse.

Live SSE Without Endangering Writes

Dashboards consume Server-Sent Events from read models, not from the ingestion transaction. SSE is perfect for crew dashboards because it is simple, reconnectable, ordered enough, and cache-friendly when fronted correctly.

GET /workouts/{id}/stream
Accept: text/event-stream
Last-Event-ID: 2026-09-12T04:58:48.126630Z:hr

On reconnect, the server replays from the last event id using PostgreSQL:

SELECT metric, value, unit, sample_ts
FROM telemetry_events
WHERE workout_id = $1
  AND sample_ts > $2
ORDER BY sample_ts ASC
LIMIT 1000;

This gives the crew a live stream while keeping the durable event log authoritative.

Benchmark and Telemetry Results

For the 21.1 km trail run, the athlete generated a compact but meaningful telemetry envelope:

Duration: 6,900 seconds
Distance: 21,100 meters
Mean pace: 5:27 min/km
Vertical density: 21.3 m gain/km
Avg HR: 155 bpm
Max HR: 178 bpm
Cadence: 166 spm
Total estimated steps: 19,090

Production targets for this workload class:

raw ingest p50: < 25 ms
raw ingest p95: < 120 ms
raw ingest p99: < 300 ms
SSE fanout lag p95: < 2 s
PostgreSQL commit failure rate: < 0.01%
duplicate sample amplification: 1.00x after idempotency
accepted out-of-order window: 72 h
zero-loss recovery objective: all raw samples replayable

The important number is not max heart rate. It is control under stress. A max HR of 178 bpm is tolerable when the system returns to stable rhythm. A p99 spike is tolerable when queues drain, WAL remains healthy, and replay closes the gap.

Production Invariants

The architecture is allowed to degrade. It is not allowed to lie.

1. Raw telemetry is persisted before enrichment.
2. All writes are idempotent by athlete, workout, metric, and sample timestamp.
3. Server arrival time never replaces athlete sample time.
4. Dashboards read from replayable state, not volatile ingest memory.
5. Backpressure drops derived work before raw samples.
6. SSE clients reconnect with Last-Event-ID and receive gap replay.
7. WireGuard path is preferred, but ingestion survives public TLS fallback.
8. PostgreSQL partitions are managed by sample time with retention explicit.
9. eBPF observes network and kernel pressure outside application logs.
10. Failed collectors rejoin as delayed producers, not data-loss events.

Backyard Ultra has a brutal clarity: you are out only when you stop answering the bell. Distributed systems need the same philosophy.

A collector can fall behind. A pod can restart. A connection can flap. A dashboard can disconnect. Fallen is not finished if the event log is durable, the protocol is replayable, and the architecture knows the difference between fatigue and failure.

That is Backyard Endurance OS: physiological truth mapped onto production-grade telemetry resilience.

DEV Community: Run_as_daemon