I got tired of SSHing into the box every time I shipped something, just to watch the logs come up. So I wanted a page in the admin panel where the lines scroll past as they happen, no dashboard, no Grafana, just the raw tail. The surprising part was how little I had to build for it, most of the pieces were already sitting on the server waiting for me to connect them.
Here's the whole idea. The app writes JSON to stdout, systemd grabs that stdout and drops every line into the journal, and journalctl can follow the journal and hand the lines back live. All three of those already exist on an Ubuntu box. So the "live log viewer" is really just me spawning journalctl on the server and piping its output to the browser over an EventSource, which is a lot less code than it sounds like.
journald is boring and well understood. SSE is boring and well understood (it's been in browsers since about 2011). Nobody gets excited about either one on its own. But snap the two together and you get a real-time log tail with no agent, no log shipper, no vendor, and nothing new to keep alive. Two defaults meeting each other and pretending to be a feature.
Systemd thing
People have opinions about systemd. Some of them are that you should avoid every part of it that you can, run your own supervisor, ship logs somewhere with your own daemon, and treat journald as a thing to route around. That's a fine hobby if you have the time for it. I don't. The box boots, systemd starts my unit, and when the unit writes to stdout the line ends up in the journal without me configuring anything. Being a purist here costs real hours and buys me a philosophy. Being pragmatic costs nothing and buys me a log tail.
So this is the pragmatic path. If you're on a distro where journald is the default (Ubuntu, Debian, Fedora, most of them now) the setup below is basically free.
Getting logs into the journal
Here is the part most people overcomplicate. You do not "set up journald". You do not open a socket to it or pull in a library. You write to standard output, and if your process is a systemd service, systemd is already reading your stdout and stderr and writing every line into the journal tagged with your unit name.
My unit is about as plain as it gets:
# /etc/systemd/system/dusanmalusev-dev.service
[Unit]
Description=dusanmalusev.dev
After=network.target
[Service]
ExecStart=/usr/bin/node /srv/dusanmalusev-dev/build/index.js
WorkingDirectory=/srv/dusanmalusev-dev
Restart=always
User=www-data
Environment=NODE_ENV=production
[Install]
WantedBy=multi-user.target
// src/lib/server/logger.ts
const stdout = pino.destination({ dest: 1, sync: false });
So each journal entry's `MESSAGE` field is a full JSON object, `{"level":30,"time":1720...,"msg":"request completed",...}`. That matters later, because it means I don't need journald's own timestamp or metadata. My structured fields are already inside the message.
Why bother with the journal at all instead of a file? A few reasons that add up. The journal is structured (every entry has fields you can query on, not just a blob of text), it rotates itself by size and age so I never touch logrotate, it survives across restarts, and I can already ask it questions from the shell without grep gymnastics:
bash
journalctl -u dusanmalusev-dev.service -f # follow live
journalctl -u dusanmalusev-dev.service --since "1 hour ago"
journalctl -u dusanmalusev-dev.service -p err # errors and worse
journalctl -u dusanmalusev-dev.service -o json # structured out
That last flag is the one that unlocks everything else. `-o json` (and its lighter sibling `-o cat`) turn the journal into something a program can parse, not just something a human can read.
## reading the journal back out
To follow the log live I spawn `journalctl` and read its stdout line by line. The flags that matter are `-f` (follow), `-n 0` (don't dump history, I only want new lines), and `-o cat` (print just the `MESSAGE` field). Since my MESSAGE is already pino JSON, `-o cat` hands me exactly the JSON I want with no journal wrapper to peel off.
typescript
// src/lib/server/admin/logs.ts (followLogs)
const child = spawn('journalctl', [
'-u', unit,
'-o', 'cat', // just the MESSAGE field, which is our pino JSON
'-f', // follow
'-n', '0', // no backlog, only new lines from now
'--no-pager'
]);
let buf = '';
child.stdout?.setEncoding('utf8');
child.stdout?.on('data', (chunk: string) => {
buf += chunk;
let idx;
while ((idx = buf.indexOf('\n')) !== -1) {
const line = buf.slice(0, idx);
buf = buf.slice(idx + 1);
if (!line) continue;
const parsed = parseLine(line);
if (!parsed) continue;
const ev = toStreamEvent(parsed);
if (passesFilter(ev, filter)) onEvent(ev);
}
});
The buffering there is the one thing you can't skip. `data` events don't arrive on line boundaries, a single chunk can be half a line, or three lines, or a line plus a fragment of the next. So I accumulate into `buf` and only cut on `\n`, keeping whatever partial line is left for the next chunk. Get this wrong and you'll spend an afternoon wondering why one log line in fifty is truncated JSON.
Parsing is forgiving on purpose. Most lines are pino JSON, but startup banners and stray stderr traces aren't, and I'd rather show those than drop them:
typescript
// src/lib/server/admin/logs.ts (parseLine)
function parseLine(line: string): LogLine | null {
try {
const obj = JSON.parse(line) as Record;
const time = typeof obj.time === 'number' ? obj.time : Date.now();
const level = typeof obj.level === 'number' ? obj.level : 30;
const msg = typeof obj.msg === 'string' ? obj.msg : '';
return { time, level, levelName: LEVEL_NAME[level] ?? String(level), msg, raw: obj };
} catch {
// Non-JSON line (a stderr trace or startup banner). Surface it as
// info-level with the raw text so the user still sees it.
const trimmed = line.trim();
if (!trimmed) return null;
return { time: Date.now(), level: 30, levelName: 'info', msg: trimmed, raw: { msg: trimmed } };
}
}
The filtering (`level`, `contains`) runs server-side before anything goes out on the wire. A client watching only errors shouldn't make the server serialize and send every debug line just to have the browser throw it away.
## the SSE endpoint
Server-Sent Events is the right transport here and WebSockets would be the wrong one. The data goes one direction (server to browser), it's text, and I want automatic reconnection without writing any of it. That's the exact shape SSE was designed for. The wire format is almost insultingly simple, you write `event: <name>` and `data: <json>` and a blank line, and the browser's `EventSource` does the rest.
In SvelteKit the endpoint is a `GET` that returns a `ReadableStream` with the `text/event-stream` content type. `followLogs` pushes each parsed line into the stream as a `log` event:
typescript
// src/routes/admin/logs/stream/+server.ts
export const GET: RequestHandler = async (event) => {
requireAdmin(event);
const encoder = new TextEncoder();
let stopFollow: (() => void) | null = null;
let heartbeat: ReturnType<typeof setInterval> | null = null;
const stream = new ReadableStream<Uint8Array>({
start(controller) {
const send = (name: string, payload: unknown) => {
try {
controller.enqueue(
encoder.encode(`event: ${name}\ndata: ${JSON.stringify(payload)}\n\n`)
);
} catch {
/* controller closed */
}
};
const handle = followLogs(
{ level, contains },
(ev) => send('log', ev),
(err) => send('error', { message: err.message })
);
stopFollow = handle.stop;
send('ready', { source: handle.source, available: handle.available });
// keeps idle-killing proxies (nginx / Cloudflare) from cutting us off
heartbeat = setInterval(() => {
try {
controller.enqueue(encoder.encode(`: ping\n\n`));
} catch {
/* noop */
}
}, 25_000);
},
cancel() {
stopFollow?.();
if (heartbeat) clearInterval(heartbeat);
}
});
return new Response(stream, {
headers: {
'content-type': 'text/event-stream; charset=utf-8',
'cache-control': 'no-store, no-transform',
'x-accel-buffering': 'no',
connection: 'keep-alive'
}
});
};
Three lines in there are the difference between "works on my laptop" and "works behind nginx", and I'll come back to them.
The `cancel()` on the stream is the part you can't forget. When the browser tab closes, the `ReadableStream` is cancelled, and if I don't kill the `journalctl` child there, every disconnect leaks a process. On a long-running box that's how you end up with two hundred orphaned `journalctl -f` processes and a confused future you. `stopFollow()` sends `SIGTERM` to the child, done.
## the browser side
The client is the smallest piece. `new EventSource(url)`, then listen for the named events. `EventSource` reconnects on its own if the connection drops, which is most of why I picked SSE over rolling something on `fetch`.
typescript
// src/routes/admin/logs/+page.svelte
let es: EventSource | null = null;
function connect() {
if (es) es.close();
es = new EventSource(streamUrl());
es.addEventListener('ready', (ev) => {
const payload = JSON.parse((ev as MessageEvent).data);
liveSource = payload.source;
connected = payload.available;
});
es.addEventListener('log', (ev) => {
if (paused) return;
const e = JSON.parse((ev as MessageEvent).data) as LogEvent;
events = [e, ...events].slice(0, linesCap); // newest first, capped
});
es.addEventListener('error', (ev) => {
const payload = JSON.parse((ev as MessageEvent).data ?? '{}');
if (payload.message) streamError = payload.message;
connected = false;
});
}
`events = [e, ...events].slice(0, linesCap)` prepends the newest line and caps the array, so the DOM never grows without bound on a chatty service. Everything else is just rendering. Level filter and search box rebuild the URL (`?level=warn&q=timeout`), close the old `EventSource`, and open a new one, so the filtering happens server-side and the browser only ever receives lines it's going to show.
## the parts that actually bite you
The happy path above is maybe forty lines. The three things that cost me real time were all environmental, not code.
**Permissions.** By default the app user can't read the journal. `journalctl` runs, returns nothing or errors, and you assume your spawn is broken when it isn't. The fix is one group, the user your service runs as needs to be in `systemd-journal`:
bash
usermod -aG systemd-journal www-data
I only figured this out fast because I catch the error and translate it instead of swallowing it, so the viewer says `Permission denied reading the journal, the app user needs to be in the systemd-journal group` instead of just going quiet. Past-me thanking past-me for that one.
**Proxy buffering.** nginx buffers proxied responses by default, which for a normal request is what you want and for an event stream means the browser sees nothing for thirty seconds and then a wall of lines. The `x-accel-buffering: no` header turns nginx buffering off for that one response, and `cache-control: no-transform` stops anything in the middle from trying to be clever. Without those two, SSE behind nginx just looks broken.
**Idle timeouts.** A quiet service might not emit a line for minutes, and proxies love to kill connections that go quiet. The `: ping\n\n` every 25 seconds is an SSE comment (any line starting with `:` is ignored by the client), it carries no data and exists only to keep the pipe warm. Cheap insurance.
None of those three are journald problems or SSE problems. They're deployment problems, the kind you only meet once the thing leaves your laptop.
## the short version
The whole path is four hops. The app writes JSON to stdout, systemd puts every line into the journal under your unit, `journalctl -f -o cat` follows that journal and hands the lines back, and an SSE endpoint pipes them to an `EventSource` in the browser. On the code side that's a `spawn`, a `ReadableStream`, and about ten lines of client. Everything expensive (structured logs, rotation by size and age, retention, querying) was already done for you by something that booted before your app did.
So if you're already on a systemd distro, this is close to free. Write to stdout, add the service user to `systemd-journal`, point `journalctl -f -o cat` at your unit, and pipe it out over SSE. That's the whole trick, and most of it isn't yours to build.
Top comments (0)