Building reqlog: a Go CLI for tracing request flows across logs (files, Docker, SSH)

In backend systems, one of the most common debugging workflows is also one of the most frustrating:

tracing a single request across multiple services using logs.

You usually end up jumping between:

• log files on different servers
• Docker containers
• SSH sessions into machines
• repeated grep commands
• and manually reconstructing what actually happened

grep is powerful, but it only gives you lines, not the story behind a request.

That gap is what led me to build reqlog.

Table of Contents

• What is reqlog
• Core usecases
• Architecture overview
• Performance decisions
• Transport layer abstraction
• Why this matters
• reqlog-ui (optional companion)
• Who is reqlog for
• What reqlog is NOT
• Closing thoughts
• Repo

What is reqlog?

reqlog is a Go-based CLI tool for tracing request flows across distributed logs.

It allows you to search logs using keys like request_id, trace_id, or any custom event key and reconstruct a chronological timeline of events.

It currently supports:

• File-based logs
• Docker container logs
• SSH-based remote logs (multiple hosts)

The goal is simple:

make distributed log debugging feel like using grep on a single machine.

Core usecases

reqlog is designed around real debugging workflows rather than abstract log querying.

1. Debugging completed requests

A common scenario:

A request fails in production, and you want to understand what happened across services.

Instead of manually grepping logs per service:

reqlog -d /path/to/logs req-1001

Or if you're tracing an event-based workflow:

reqlog -d /path/to/logs -k event_key --limit 100 order.created

reqlog scans logs across services and reconstructs a chronological timeline so you can quickly understand how a request moved through the system.

2. Live request tracing

Sometimes you’re debugging an ongoing request flow (for example, an API request or a telephony call in my case).

reqlog supports live log streaming so you can trace a request in real time as logs are generated:

reqlog -d /path/to/logs -f req-1001

Or for Docker-based services:

reqlog -S docker -f req-1001

This is especially useful when you want to observe a request while it is actively progressing through the system.

3. Debugging a single busy service

Interestingly, reqlog turned out to be useful even when debugging a single service.

In production systems handling dozens or hundreds of concurrent requests, logs become noisy very quickly.

Even when debugging a single API request or user action, unrelated logs are constantly interleaved, making it hard to follow the flow manually.

For example:

reqlog -d /path/to/logs req-1001

Instead of mentally filtering through unrelated log entries, reqlog isolates the request flow and shows only matching events in order.

This turned out to be one of the most practical day-to-day use cases.

4. Multi-service / container debugging

When services run inside containers, debugging often involves jumping across multiple docker logs outputs.

Instead of inspecting containers individually:

reqlog -S docker -s order-service,inventory-service req-1001

Or when tracing event-driven workflows:

reqlog -S docker -s order-service,inventory-service -k event_key --limit 100 order.created

reqlog merges logs from multiple services into a single chronological timeline.

5. Multi-host debugging (SSH logs)

In many setups, logs are distributed across multiple servers.

Instead of SSH-ing into each machine and grepping separately:

reqlog --hosts api1,api2,api3 -d /var/log/services req-1001

reqlog fetches logs remotely over SSH and combines them into a unified timeline.

Architecture overview

One of the main goals of reqlog was to keep the system extensible without duplicating logic across log sources.

Scanners (log sources)

reqlog currently uses a scanner-based model:

• FileScanner → reads log files
• DockerScanner → reads container logs

Each scanner implements a common interface so the rest of the pipeline remains unchanged.

LineProcessor (shared log pipeline)

At the core of reqlog is a shared processing pipeline:

• parsing log lines
• filtering by key/value
• extracting timestamps and metadata
• preparing structured output

This ensures all log sources behave consistently once data enters the pipeline.

Fast filtering optimization

A simple but important optimization:

Before fully parsing a log line, reqlog performs a quick check:

strings.Contains(line, searchValue)

This avoids unnecessary parsing work on large log files and significantly improves performance in real-world usage.

Performance decisions

Global --latest using a min-heap

One of the more interesting challenges was implementing:

“show the latest logs across multiple sources”

Sorting all logs in memory is expensive, especially for large files.

Instead, reqlog uses a bounded min-heap to track only the most recent entries.

This keeps memory usage under control while still returning accurate results.

Tradeoff:

• Lower memory usage
• Slightly higher CPU cost when many matches exist

In practice, this works best when combined with time filtering like --since.

Preserving timeline correctness with --limit

Another interesting problem was implementing --limit while keeping timelines accurate across services.

Since reqlog builds a chronological timeline, simply taking the first N matches globally would produce incorrect results.

For example, imagine searching logs across multiple services:

api-service
order-service
inventory-service

If reqlog stopped after finding N matches from the first service that produced results, one busy service could consume the entire limit while other services never even got the chance to be scanned, resulting in an incomplete timeline.

Instead, reqlog does something slightly different:

1. Fetch up to N matching logs from each source/service
2. Merge and sort them chronologically
3. Keep the final global N entries
4. Discard the rest

This ensures timeline ordering remains correct across distributed logs while still keeping memory usage bounded.

The tradeoff is that --limit is optimized for correctness first, not minimal scanning work, especially when many services contain matching entries, though this is still faster in most cases unless you're requesting something like --limit 500 across 50 services.

Transport layer abstraction

To support multiple log sources without duplicating logic, reqlog introduces a transport abstraction layer.

Interfaces

Two key abstractions:

• FileSystem
• Executor

Why this matters

This design allows reqlog to treat all sources uniformly:

• local file system
• SSH remote systems
• Docker containers

The core processing logic does not change regardless of where logs come from.

SSH implementation

For remote logs:

• reqlog uses SSH to execute remote commands
• and SFTP for file access

This keeps the same FileScanner logic reusable even for remote machines.

Docker implementation

For Docker logs:

Instead of reinventing container log handling, reqlog uses the Docker CLI directly via the Executor abstraction:

docker logs <container>

This avoids unnecessary complexity and keeps behavior consistent with standard tooling.

reqlog-ui (optional companion)

Alongside the CLI, there is a lightweight UI wrapper called reqlog-ui.

It allows users (especially QA or developers without SSH access) to:

• search logs from a browser
• view timelines visually
• avoid direct server access for simple debugging

This helps teams reduce friction when investigating issues without requiring full infrastructure access.

Who is reqlog for?

reqlog is designed for:

• backend engineers working with microservices
• devops / SRE engineers debugging production issues
• teams without full observability stacks
• systems where logs are still the primary debugging tool
• engineers who frequently trace request flows manually

It works best in environments where:

logs exist everywhere, but correlation is still manual.

What reqlog is NOT

To keep expectations clear:

reqlog is not:

• an observability platform
• a log storage system
• a monitoring or alerting tool
• a replacement for tools like Datadog, Loki, or ELK

It is intentionally lightweight and stateless.

Closing thoughts

reqlog started as a small internal tool to solve a recurring debugging problem: tracing requests across multiple services and log sources.

Over time, it evolved into a CLI that unifies file logs, Docker logs, and SSH-based logs into a single searchable timeline.

It is still evolving based on real usage and feedback.

This is also my first open-source tool after several years of backend engineering, and building it has been a genuinely enjoyable experience, especially seeing it used in real debugging workflows.

Repo

GitHub:
https://github.com/SagarMaheshwary/reqlog