Adaptive Bitrate Log Streaming in CI/CD

---
title: "Adaptive Bitrate Log Streaming for CI/CD Pipelines"
published: true
description: "How to architect real-time CI/CD log streaming that handles 10GB builds without OOM crashes — chunked encoding, SSE, backpressure, and virtual scrolling explained."
tags: devops, architecture, cloud, performance
canonical_url: https://blog.mvpfactory.co/adaptive-bitrate-log-streaming-cicd
---

## What We're Building

Let me show you a pattern I use in every project that involves real-time build output: an adaptive log streaming architecture that browses 10GB CI/CD build logs without your viewer ever exceeding 150MB of memory. We'll wire up Server-Sent Events, a ring buffer for bounded server memory, backpressure signaling, and client-side virtual scrolling with anchor-based seek.

By the end, you'll understand exactly why "just show me the logs" is a distributed systems problem — and how to solve it.

## Prerequisites

- Familiarity with HTTP streaming (chunked transfer encoding basics)
- A server runtime (Kotlin/JVM examples here, but the patterns are portable)
- A frontend that renders log output (TypeScript examples for the client)

## Step 1: Choose Your Delivery Protocol

Here's the gotcha that will save you hours: most teams jump straight to WebSockets without considering the operational cost. For log streaming — a unidirectional, append-only data flow — SSE is the correct default.

| Criteria | SSE | WebSocket |
|---|---|---|
| Auto-reconnect | Yes (built-in) | Manual |
| HTTP/2 multiplexing | Yes | No (upgrade required) |
| Load balancer support | Excellent | Requires sticky sessions |
| Memory overhead per conn | ~4KB | ~8KB |

You get automatic reconnection with `Last-Event-ID`, native HTTP/2 multiplexing, and zero load balancer headaches. Only introduce WebSockets if you have a proven bidirectional requirement.

## Step 2: Server-Side Ring Buffer

The docs don't mention this, but a typical Kotlin multiplatform CI build generates 50,000–200,000 lines of output. Monorepo Gradle builds push past 500,000 lines. Your log ingestion server should never hold the full log in memory. Here's the minimal setup to get this working:

kotlin
class LogRingBuffer(private val capacity: Int = 50_000) {
private val buffer = ArrayDeque(capacity)
private var globalOffset: Long = 0

@Synchronized
fun append(line: LogLine) {
    if (buffer.size >= capacity) {
        buffer.removeFirst()
        globalOffset++
    }
    buffer.addLast(line)
}

fun slice(from: Long, count: Int): List<LogLine> {
    val start = (from - globalOffset).coerceAtLeast(0).toInt()
    return buffer.drop(start).take(count)
}

}

The ring buffer keeps the last N lines in memory while the full log streams to object storage (S3, GCS) in compressed chunks. Clients connecting mid-build get the tail from the ring buffer and seek historically via range requests against stored chunks — similar to how adaptive bitrate video streaming works with segment-based seeking.

## Step 3: Backpressure Signaling

This is where most implementations quietly fall apart. Without backpressure, a fast build and a slow client become a cascading failure. The protocol:

1. Server tracks per-client send queue depth. If the queue exceeds 1,000 unsent lines, switch that client to **summary mode** — one compacted message per N lines.
2. Client signals readiness via SSE reconnection with a `Last-Event-ID` encoding the last consumed offset.
3. When the client falls behind, show a "streaming paused, click to catch up" indicator. Honesty beats invisible data loss.

This is the same pattern used in reactive streams (`Publisher`/`Subscriber` with demand signaling), adapted for HTTP.

## Step 4: Client-Side Virtual Scrolling

Rendering 500K DOM nodes is not viable. Virtual scrolling renders only the visible viewport (50–100 lines) plus a small overscan buffer:

typescript
interface LogViewport {
visibleStart: number;
visibleEnd: number;
anchorOffset: number | null; // jump target for failure line
totalLines: number;
}

The key addition: **anchor-based seek**. When a build fails, the server sends a failure-line offset and the client jumps directly to it. Client memory stays under 150MB regardless of total log size.

## Gotchas

- **Don't skip backpressure.** Without it, a slow consumer forces the server to buffer indefinitely, eventually OOM-killing the process. Budget 50K lines in the ring buffer and stream the rest to storage.
- **GitHub Actions truncates at ~500K lines** and renders the full DOM with no virtual scrolling. GitLab archives beyond ~1M lines. Buildkite handles ~10M lines with adaptive batching and virtual scrolling — their approach most closely mirrors this architecture.
- **Chunked Transfer Encoding lacks auto-reconnect.** SSE gives you that for free. Only use raw chunked encoding if you're building a download-style flow with no reconnection needs.
- **HTTP/2 multiplexing breaks with WebSockets.** Each WebSocket connection requires an upgrade, bypassing multiplexing entirely. With SSE, hundreds of log streams share a single TCP connection.

## Wrapping Up

Default to SSE. Implement explicit backpressure. Use virtual scrolling with anchor-based seek. These three decisions turn a "just show me the logs" feature from a distributed systems landmine into a reliable, memory-bounded streaming architecture. The sooner you treat log viewing as a backpressure problem rather than a rendering problem, the less time you'll spend debugging your debugging tools.