DEV Community: Alex Zhdankov

SQLite as an offline metrics buffer in a monitoring agent

Alex Zhdankov — Thu, 28 May 2026 14:53:32 +0000

Reliable monitoring becomes surprisingly hard once you assume the network will eventually fail.

Monitoring systems quietly depend on one dangerous assumption:

the monitoring server will always be reachable

In production, that assumption breaks constantly.

And once it breaks,
you discover that metrics pipelines are really distributed systems.

The problem

Our PostgreSQL monitoring agent periodically collects metrics and sends them to the Control Plane.

Under normal conditions:

Agent
   │
   └──► metrics ───► Control Plane ───► storage

Simple enough.

But then production networks happen:

VPN reconnects
firewall reloads
DNS failures
routing instability
Control Plane maintenance windows

Once connectivity drops,
the entire delivery pipeline becomes unreliable.

The obvious answer sounds simple:

“just retry later”

That idea turned out to be much harder than expected.

Questions immediately appeared:

how long should metrics stay in memory?
what happens during process restart?
how do we avoid unbounded memory growth?
how do we recover after long outages?
how do we know delivery is degraded?

At that point,
this stopped being “metrics collection”.

It became a buffering and reliability problem.

The incident that changed the design

One outage lasted almost six hours after a network partition between monitoring agents and the Control Plane.

The agents themselves stayed healthy.
PostgreSQL stayed healthy.

Only the network path disappeared.

The original implementation buffered metrics in memory.

At 4:12 AM, one agent restarted during a package update.

Six hours of buffered monitoring data vanished instantly.

The dashboards looked like this:
timeline ─────── gap ─────── timeline

No visibility.
No recovery.
No idea what happened during the outage.

That incident permanently killed the idea of in-memory buffering.

Metrics had to survive process restarts.

The architecture

The final design became intentionally simple:

                PostgreSQL
                     │
                     ▼
              metrics collector
                     │
                     ▼
          +----------------------+
          |      SQLite WAL      |
          |----------------------|
          | durable local queue  |
          | bounded retention    |
          +----------------------+
                     │
                     ▼
             batch delivery loop
                     │
                     ▼
               Control Plane

The key architectural shift:

the Control Plane is no longer in the critical path of collection

Metrics are always written locally first.

Network delivery became asynchronous.

That single separation changed the reliability model completely.

The network could fail for hours,
and the agent would continue operating normally.

Why SQLite

The interesting part here is not SQL.

It's what SQLite fundamentally is:

a single file
atomic writes
crash-safe persistence
zero infrastructure
included in Python itself

We evaluated alternatives.

In-memory queue

Failed because:

metrics disappear on restart
memory usage becomes unbounded during long outages

Plain append-only files

Failed because:

manual locking
manual serialization
corruption handling
cleanup complexity

Redis

We already use Redis elsewhere in the platform (for the psql terminal).

But for the metrics buffer, Redis was the wrong fit:

Redis is in-memory by default. Persistence (RDB/AOF) is optional and adds latency.
If Redis restarts, the buffer disappears — even if the agent is healthy.
Mixing low-latency terminal traffic with background batch writes would create contention.

SQLite gives us durability on disk, survives agent restarts, and has zero dependencies outside the agent process.

Redis is great for real-time coordination. SQLite is better for durable local buffering.

The schema is intentionally tiny

CREATE TABLE IF NOT EXISTS metrics_queue (
    id          INTEGER PRIMARY KEY AUTOINCREMENT,
    instance_id TEXT    NOT NULL,
    timestamp   TEXT    NOT NULL,
    metric_type TEXT    NOT NULL,
    data        TEXT    NOT NULL,
    created_at  TEXT    NOT NULL DEFAULT (datetime('now'))
)

One row per metric type per collection cycle.

Payloads are JSON-encoded.

This is not relational modeling.

This is durable buffering.

Concurrency: why SQLite didn't become a bottleneck

One obvious question:

“doesn't SQLite lock the whole database?”

Yes.
And that matters.

The reason it worked here:

exactly one writer exists
the collector loop owns all inserts
delivery workers only read/delete in batches

We also enabled WAL mode:

PRAGMA journal_mode=WAL;
PRAGMA synchronous=NORMAL;

WAL mode mattered for one specific reason:

the collector continuously appends metrics,
while the delivery loop simultaneously reads and deletes batches.

Without WAL,
those operations would frequently block each other.

WAL allows readers to continue while writes are happening.

We also use:

PRAGMA synchronous=NORMAL;

This reduces fsync frequency while still preserving crash consistency,
which was a better latency/durability tradeoff for monitoring data.

For our workload:

collection every 10–30 seconds
batches of 100–500 rows
occasional degraded-network bursts

SQLite contention was negligible.

If we needed hundreds of concurrent writers,
we would likely move to PostgreSQL instead.

For this system,
operational simplicity mattered more than horizontal scalability.

The buffer state machine

The buffer itself behaves like a bounded queue:

COLLECT
   │
   ├── buffer healthy ──► INSERT
   │
   └── limits exceeded
              │
              ▼
         DROP + WARN

Two limits control growth:

maximum file size
maximum record age

Those constraints turned out to be operationally critical.

Why age-based cleanup matters

Imagine a six-hour outage.

Without retention limits,
the moment connectivity returns,
the agent floods the Control Plane with six hours of historical metrics.

On dashboards, that looks like:

timeline ─── gap ─── MASSIVE SPIKE ─── normal

Operationally, that's worse than losing data.

The spike is misleading.

So we intentionally discard stale records.

If buffered metrics exceed max_age_hours,
they are silently removed during cleanup.

The result becomes:

timeline ─── gap ─── normal

The outage remains visible.
But the monitoring data stays trustworthy.

That tradeoff mattered more than perfect retention.

Batch delivery

The delivery loop periodically reconstructs batches from SQLite:

SELECT id, instance_id, metric_type, data
FROM metrics_queue
ORDER BY created_at ASC
LIMIT ?

Rows are grouped back into metric payloads and sent to the Control Plane.
Importantly:

rows are deleted only after successful delivery

if send_to_control_plane(batch):
    cursor.execute(
        f"""
        DELETE FROM metrics_queue
        WHERE id IN ({','.join('?' * len(ids))})
        """,
        ids,
    )
    conn.commit()

This creates a very simple delivery guarantee:
INSERT → SEND → DELETE
If the process crashes before DELETE,
the metrics remain in SQLite and are retried later.

That makes delivery effectively at-least-once.

This effectively turns SQLite into a durable local write-ahead queue.

The second production incident

Several months later,
another issue appeared.

Monitoring dashboards showed healthy agents,
but metrics were arriving hours late.

The agents were buffering correctly.

The problem was:
the SQLite files had silently grown to hundreds of megabytes after a prolonged outage.

Delivery throughput could not drain the backlog fast enough.

At that point we realized:

the buffer itself needed observability

Buffer observability

Every heartbeat sent to the Control Plane now includes buffer state:

{
    'buffer_size_bytes': size,
    'oldest_record_age': age,
    'buffer_utilization': utilization,
}

Operationally, the most useful metrics became:

Metric             | Meaning                  | Operational action
───────────────────+──────────────────────────+────────────────────────
buffer_size_bytes  | Current SQLite file size | Alert if >100 MB
oldest_record_age  | Delivery lag             | Alert if >1 hour
buffer_utilization | Current size / max size  | Scale retention policy
delivery_failures  | Consecutive failed sends | Investigate connectivity

Without these metrics,
network degradation looked identical to healthy operation.

The buffer itself became another production subsystem
that required monitoring.

Production numbers

Typical production values:

collection interval: 10–30 seconds
average batch size: 100–500 rows
SQLite buffer limit: 128 MB
longest outage survived: ~6 hours

Under normal degraded-network conditions,
the SQLite file rarely exceeded 10–15 MB.

CPU overhead was effectively negligible.

Failure modes we actually cared about

A. Agent restart during outage

Metrics survive because the buffer lives on disk.
Delivery resumes automatically after restart.

B. Control Plane recovery after long outage

Fresh metrics resume immediately.
Stale buffered records expire automatically.
No misleading spikes.

C. Disk pressure

Once the buffer exceeds the configured size limit:

new writes stop
warnings are emitted
collection pauses temporarily

Delivery resumes automatically once backlog drains.

D. SQLite corruption

Rare, but possible.

We intentionally kept handling simple:

log the error
recreate the buffer
continue operation

For a monitoring system,
temporary metric loss is acceptable.

Operational complexity was not worth stronger guarantees.

Bounded loss over unbounded growth

One important design decision:

the system intentionally prefers bounded data loss over unbounded resource growth

Old metrics eventually expire.

The SQLite file has a hard size limit.

That tradeoff was deliberate.

For operational monitoring,
partial visibility is better than:

a dead agent
an exhausted disk
or an overloaded recovery pipeline

Perfect durability was never the goal.

Operational survivability was.

The mental model

If you understand only one thing:

collect
   ▼
SQLite
   ▼
deliver
   ▼
delete

SQLite here is not acting as a database.

It's:

a durable queue
with bounded retention
bounded size
atomic writes
and zero infrastructure dependencies

That's the entire design.

Final thought

The interesting realization was this:

monitoring reliability problems are mostly buffering problems

Once network delivery became unreliable,
the architecture naturally evolved toward:

local durability
asynchronous delivery
bounded retention
explicit degradation handling

Not because SQLite is magical.

Because a single durable file solved the exact failure mode we actually had.

Why your SSH scripts will fail in production

Alex Zhdankov — Mon, 18 May 2026 15:41:40 +0000

Remote command execution looks trivial — until unstable networks, retries, long-running commands, and half-open connections turn it into a reliability problem.

We use Paramiko with a thin supervision layer on top.
The same operational problems apply to AsyncSSH, Fabric, or plain OpenSSH subprocesses.

At first, the implementation looked completely straightforward:

client = paramiko.SSHClient()
client.connect(hostname=host, username=user)

stdin, stdout, stderr = client.exec_command(
    "systemctl restart postgres"
)

output = stdout.read().decode()

In development, this worked perfectly.

Then production happened.

Hundreds of hosts.
Unstable networks.
Long-running commands.
Frozen sessions.
Half-open connections.
Retries.
Partial execution.

At that point this stopped being “SSH scripting”.

It became a distributed systems problem.

SSH is deceptively simple

Most developers intuitively model SSH like this:
local subprocess, but remote

But production SSH execution is actually:

network transport
+ stateful session
+ interactive channel
+ remote process lifecycle
+ unreliable infrastructure
+ partial execution visibility

And failures can happen independently at every layer.

Application
    ↓
SSH Client
    ↓
TCP transport        ← packets can vanish
    ↓
SSH session          ← can hang without closing
    ↓
Remote shell         ← can ignore commands
    ↓
Process execution    ← may continue after disconnect
    ↓
stdout/stderr        ← can block forever

This distinction changes everything.

Failure mode #1 — execution uncertainty

This was the first major production lesson.

If the SSH transport dies, you do not know whether the command:

succeeded
failed
partially executed
is still running remotely

That uncertainty completely changes retry semantics.

For example:

systemctl restart postgres

If the connection drops immediately after sending the command:

did restart begin?
is postgres still restarting?
did it already succeed?
is the service now dead?

You no longer have execution certainty.

This is not a “Paramiko problem”.

This is a distributed systems problem.

Retry is dangerous

Retries sound harmless until commands become stateful.

Some operations are naturally idempotent:

cat /proc/meminfo
ls -la /etc
systemctl status postgres

Others are not:

useradd deploy
rm -rf /some/path
systemctl restart postgres

A failed transport does not imply failed execution.

That means naive retry logic can create destructive side effects.

This forced us to separate failures into two categories:

transport uncertainty
command failure

Those are fundamentally different operational states.

Timeouts are not one thing

One of the most common mistakes in SSH automation is treating timeout as a single concept.

Production systems usually need several independent timeout layers:

TCP connect timeout
SSH handshake timeout
authentication timeout
command execution timeout
idle/read timeout

Each failure means something different operationally.

client.connect(
    hostname=host,
    username=username,
    timeout=10,
    banner_timeout=15,
    auth_timeout=15
)

But even that is insufficient.

A command may still hang forever while the socket technically remains alive.

That distinction matters a lot in production.

Half-open connections are nasty

This became one of the hardest reliability problems.

Sometimes:

TCP stays alive
SSH transport stays alive
but the remote process is effectively dead

Or:

packets silently disappear
the remote kernel freezes
stdout stops forever
but the socket never closes

From the application perspective:
everything looks connected

while the operation is permanently stalled.

This is the classic half-open connection problem.

Blocking reads break automation

This code looks innocent:

stdout.read()

But under real workloads it becomes dangerous.

If:

the command hangs
stdout stops producing data
the socket remains alive

then:
the thread blocks forever

We eventually moved to streaming execution instead of buffered reads.

Streaming changes the execution model

Long-running commands fundamentally change how remote execution must be handled.

Operations like:

pg_dump
VACUUM
package upgrades
log exports

can run for minutes or hours.

Buffering all output in memory is unreliable.
Blocking until completion destroys observability.

Instead we switched to chunked streaming:

while not channel.exit_status_ready():
    if channel.recv_ready():
        data = channel.recv(4096)
        callback(data)

This solved several production problems simultaneously:

realtime progress visibility
lower memory usage
cancellation support
dead session detection

Streaming ended up being much more operationally stable than buffered execution.

Security becomes infrastructure, not validation

Another important lesson:

SSH automation is remote code execution infrastructure.

That means command construction rules matter enormously.

This is catastrophic:

cmd = f"rm -rf {user_input}"

Because eventually someone passes:

/home/user; rm -rf /

We ended up treating all remote commands as infrastructure-sensitive operations.

Input validation alone was insufficient.

Every dynamic argument had to be:

validated
escaped
constrained

safe_value = shlex.quote(user_input)

Even simple automation eventually becomes security-critical.

Resource cleanup matters more than expected

SSH resources leak surprisingly easily.

Channels.
Sockets.
Transports.
PTY buffers.

Under load, forgotten cleanup accumulates fast.

We eventually standardized all operations around explicit lifecycle management:

with ssh_operation(...) as ssh:
    ssh.execute(...)

The important part was not aesthetics.

It was guaranteeing cleanup under:

exceptions
timeouts
partial failures
interrupted execution

Production automation lives or dies on cleanup guarantees.

The architecture we ended up with

Over time the system evolved into several independent layers:

Connection management
    ↓
Retry classification
    ↓
Execution supervision
    ↓
Streaming transport
    ↓
Resource cleanup
    ↓
Observability

The important realization was:

remote execution is not a helper function

It is infrastructure.

Final insight

The happy path is trivial.

Production architecture begins where execution certainty ends.

SSH automation fails when treated like scripting.

Because it is not scripting.

It is:

remote process orchestration
over unreliable transport
with partial execution visibility
inside a distributed system

And once you accept that,
the architecture changes completely.

We built a real psql terminal in the browser. Here’s what made it unexpectedly hard.

Alex Zhdankov — Wed, 13 May 2026 20:53:07 +0000

A PTY-backed PostgreSQL console running in the browser using reverse WebSockets, Redis Streams, and xterm.js — designed around centralized control-plane constraints and production failure modes.

We needed a real PostgreSQL terminal inside the browser.

Not a SQL editor.
Not a query API.
A real psql session with full terminal semantics.

That requirement immediately forced several architectural constraints:

a real PTY
a long-lived stateful process
bidirectional streaming
terminal resize handling
signal forwarding (Ctrl+C)
native psql behavior

And then the infrastructure constraints made things significantly more interesting:

agents live in internal networks
all traffic must go through the Control Plane
xterm.js only supports WebSocket transport
we could not emulate psql

At that point, this stopped being a “web feature”.
It became a distributed terminal runtime problem.

High-level architecture

This system only makes sense if you read it as a dataflow graph, not as isolated services.

Browser (xterm.js)
    │
    │ WebSocket (terminal I/O)
    ▼
Control Plane
    │
    │ session management + auth
    ▼
Redis Streams (output buffer)
    │
    │ coordination + async delivery
    ▼
Agent WebSocket channel
    │
    │ PTY stdin/stdout bridge
    ▼
PTY → real psql process

The critical architectural decision:

the browser never connects to the agent directly.

The Control Plane is the only public entrypoint in the entire system.
Everything flows through it.

Why the architecture looks “backwards”

The surprising part is that the agent initiates the terminal transport.

Not because NAT traversal was impossible.

But because the system was intentionally designed around a centralized Control Plane.

Agents sit in internal networks.
The browser has no direct visibility into them.

So instead of:

Browser → Agent

the architecture becomes:

Browser → Control Plane ← Agent

The Control Plane acts as:

session coordinator
auth boundary
transport router
lifecycle owner

Once that decision is made, reverse WebSockets become the natural transport model.

Session establishment

The session lifecycle happens in multiple stages.

Importantly:

the PTY process does not exist when the browser first connects

Only a logical session exists.

Step 1 — Browser creates a logical session

Browser
  │
  │ WebSocket connect
  ▼
Control Plane
  ├── creates session_id
  ├── registers browser handler
  └── starts auth timeout

At this point:

no PTY exists
no psql exists
no database connection exists

The Control Plane only knows:

“a browser wants a terminal session”

Step 2 — Control Plane signals the agent

The Control Plane sends a lightweight HTTP request:
POST /terminal?session_id=<uuid>

This is intentionally the only HTTP hop in the entire terminal lifecycle.

The request does not carry terminal traffic.

It only means:

“establish terminal transport for this session”

Step 3 — Agent opens reverse WebSocket

Agent
  │
  │ outbound WebSocket
  ▼
Control Plane

Now the system has two independent transport channels:

Browser WS → Control Plane
Agent WS → Control Plane

But they are still disconnected.

The system is in a half-connected state.

Session stitching

This is the moment where the architecture becomes interesting.

Browser Handler ───────┐
                       ├── session binding
Agent Handler ─────────┘

At this point:

the Control Plane stops being a transport endpoint and becomes a message router

It now forwards:

browser input → agent
agent output → browser

But critically:

not directly

All terminal output passes through an asynchronous buffering layer.

That layer ended up being one of the most important production decisions in the system.

PTY process creation

Once the session is fully initialized, the agent forks a real PTY:

(child_pid, fd) = pty.fork()

if child_pid == 0:
    subprocess.run([
        "psql",
        "-U", user,
        "-d", dbname
    ])

At this point the architecture fundamentally changes.

This is no longer “web infrastructure”.

It becomes:

PTY supervision
file descriptor management
process lifecycle handling
signal propagation
backpressure management

Most complexity appeared after this step.

Not before it.

The real data pipeline

This is the most important flow in the system.

Browser
  │
  │ keystroke
  ▼
Control Plane
  │
  ▼
Agent WS handler
  │
  │ write(fd)
  ▼
PTY → psql
  │
  │ stdout
  ▼
PTY reader thread
  │
  │ Redis XADD
  ▼
Redis Streams
  │
  │ async consumer
  ▼
Control Plane
  │
  │ WS push
  ▼
Browser

The most important line in the entire architecture is this:
PTY reader → Redis XADD → async consumer → WebSocket
That line is the system’s stability boundary.

Why Redis Streams became mandatory

The original implementation directly forwarded PTY output into WebSocket writes:
PTY → WebSocket

It worked in development.

It failed in production.
The issue was subtle:

PTY reads are synchronous
WebSocket writes can block
backpressure propagates backwards

The resulting failure mode was catastrophic for terminal UX:

slow network
    ↓
blocked WS writes
    ↓
frozen PTY reader
    ↓
terminal stalls

The terminal looked dead while psql was still running underneath.

Redis Streams solved this by introducing a decoupling boundary.

Now:

PTY reads stay non-blocking
network latency becomes isolated
consumers can temporarily lag
output survives reconnects

The additional latency was negligible.

The operational stability improvement was enormous.

The architecture is actually two independent loops

This is the part most terminal architectures hide.

Input loop
Browser → Control Plane → Agent → PTY

Output loop
PTY → Redis → Control Plane → Browser

These loops are intentionally independent.

That separation is what allows the system to survive partial failures.

Why we split browser and agent handlers

We intentionally kept browser-facing and agent-facing handlers separate.

Because they solve fundamentally different problems.

Browser Handler:

auth
user session ownership
browser disconnect semantics
user errors

Agent Handler:

PTY lifecycle
process supervision
reconnect semantics
infrastructure errors

Trying to merge them created tightly coupled failure modes and significantly more lifecycle complexity.

Separating them made the system dramatically easier to reason about.

Failure modes that mattered in production

The hardest problems were not PostgreSQL problems.

They were long-lived process problems.

A. Redis failure

Impact:

output pipeline breaks
PTY continues running

Mitigation:

memory limits
retention limits
monitoring
bounded stream lifetime

B. Agent disconnect

Impact:

transport disappears
PTY may still be alive

Mitigation:

reconnect window
session reattachment
delayed teardown

C. Process explosion

Impact:

memory exhaustion
PostgreSQL connection storms

Mitigation:
BoundedSemaphore(max_sessions=10)
This was one of the simplest and most effective safeguards in the system.

D. xterm resize storms
xterm.js emits resize events aggressively during browser resizing.

Impact:
Each resize triggers:
ioctl(TIOCSWINSZ)

Mitigation:

Without throttling, the PTY spent significant time processing resize events instead of actual terminal traffic.
Simple debounce logic completely fixed the issue.

Scaling reality

The system does not scale like a normal WebSocket service.

Each session includes:

a real psql process
a PTY
multiple threads
Redis streams
two WebSocket channels
a database connection

The scaling bottleneck is not Redis.

It is not CPU.

It is not WebSockets.

It is:

how many real PostgreSQL sessions the infrastructure can sustain

Why HTTP and SSE were rejected

We evaluated both.

HTTP
Failed because:

stateless
no streaming terminal semantics
no signal handling
no persistent shell state

SSE
Failed because:

one-directional transport
incompatible with terminal interaction patterns
xterm.js expects bidirectional communication

At the end, terminals naturally map onto WebSockets.

Trying to avoid that only complicates the architecture.

What this system actually is

If you remove all abstractions:

this is a distributed process supervisor for a PTY running psql

Everything else is transport, routing, buffering, and failure handling around that core idea.

Final architecture insight

The system is ultimately defined by three separations.

Connection separation
The Control Plane isolates browsers from agents.
Process separation
PTY isolates PostgreSQL from the web layer.
Flow separation
Redis isolates terminal I/O from network I/O.

Final mental model

If you understand only one thing, understand this:

Browser ↔ Control Plane ↔ Agent ↔ PTY ↔ psql
                     ↑
              Redis is the buffer

Everything else is lifecycle management around this chain.

Final thought

We did not build a “web UI for PostgreSQL”.

We built a distributed, fault-tolerant runtime for a stateful terminal process.

PostgreSQL just happened to be the process attached to it.