Most applications know how to create logs.
Very few know how to understand them at scale.
Printing this:
2026-09-18 14:32:51 ERROR Payment failed for order 9812
is easy.
Printing ten thousand of those messages is still easy.
Printing ten million is where things become interesting.
And once your application is running across dozens, hundreds, or thousands of machines, logging stops being a simple console.log() problem.
It becomes a distributed systems problem.
You now have questions like:
- Where should logs be stored?
- How do we search them?
- How do we search billions of records quickly?
- How do we handle logs arriving out of order?
- How do we prevent one noisy service from destroying the cluster?
- How do we index arbitrary JSON?
- How do we aggregate logs by service, host, status code, or time?
- How do we survive machine failures?
- How do we expire old logs automatically?
- How do we make writes fast without making queries painfully slow?
This is roughly the territory occupied by systems such as Elasticsearch.
But building a smaller logging platform from scratch is one of the best ways to understand what these systems are actually doing.
Because the interesting part isn't the dashboard.
The interesting part is the machinery underneath it.
A logging platform is essentially a machine that turns an enormous stream of semi-structured events into something that can be searched, filtered, aggregated, retained, and distributed.
In this article, we're going to design one.
Not a toy logs table.
A real distributed architecture.
1. The Problem
Imagine we operate an e-commerce platform.
We have:
API Server
Payment Service
Order Service
Inventory Service
Notification Service
Worker Nodes
Database
Every component generates logs.
For example:
{
"timestamp": "2026-09-18T14:32:51Z",
"level": "ERROR",
"service": "payment",
"host": "payment-03",
"message": "Payment failed",
"order_id": "9812",
"status": 402
}
Another event might be:
{
"timestamp": "2026-09-18T14:32:52Z",
"level": "INFO",
"service": "inventory",
"message": "Stock updated",
"product_id": "P102",
"quantity": 48
}
We want developers to ask questions such as:
Find all ERROR logs from payment during the last hour.
Or:
Show all requests with status >= 500.
Or:
How many payment failures happened per minute?
Or:
Find every log containing "timeout".
Or:
Show the top 20 services producing errors.
This immediately creates two fundamentally different workloads.
Writes
Millions of log events arrive continuously.
Reads
Users want arbitrary searches over those events.
This is the first important insight:
A logging platform is simultaneously a streaming system and a search engine.
If we optimize only for writes, queries become slow.
If we optimize only for queries, ingestion becomes expensive.
We need an architecture that balances both.
2. The Architecture
A simplified architecture might look like this:
┌─────────────────────┐
│ Applications │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ Log Collectors │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ Ingestion Layer │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ Buffer / Queue │
└──────────┬──────────┘
│
┌─────────────┴─────────────┐
▼ ▼
┌────────────────┐ ┌────────────────┐
│ Indexing Nodes │ │ Storage Nodes │
└───────┬────────┘ └───────┬────────┘
│ │
└─────────────┬─────────────┘
▼
┌─────────────────────┐
│ Query Coordinator │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ API / UI │
└─────────────────────┘
Each layer has a specific responsibility.
The collector gathers logs.
The ingestion layer validates and normalizes them.
The queue absorbs bursts.
The indexing layer transforms logs into searchable structures.
The storage layer persists them.
The query coordinator distributes searches.
The API exposes everything to humans and applications.
That separation is what allows the system to scale.
3. Start With the Log Event
Before thinking about distributed nodes, we need to define our fundamental unit.
The log event.
A useful internal representation could be:
struct LogEvent {
timestamp: i64,
level: String,
service: String,
host: String,
message: String,
fields: Map<String, Value>,
}
The interesting field is:
fields
because modern logs are rarely flat.
Consider:
{
"user": {
"id": 42,
"country": "ZM"
},
"request": {
"method": "POST",
"path": "/api/payments"
}
}
Our platform should preserve that structure.
Eventually, users might search:
user.country = "ZM"
or:
request.method = "POST"
This leads to an important design decision.
Do we store logs as raw JSON?
Yes.
Should we search the raw JSON directly?
Probably not.
And this is where indexing begins.
4. Why a Database Table Isn't Enough
The obvious implementation is:
CREATE TABLE logs (
id BIGINT,
timestamp TIMESTAMP,
level VARCHAR,
service VARCHAR,
message TEXT,
data JSON
);
Then:
SELECT *
FROM logs
WHERE message LIKE '%timeout%';
It works.
Until it doesn't.
Imagine one billion records.
The database may have to inspect enormous amounts of text.
Searching:
timeout
is fundamentally different from searching:
service = payment
An equality query can use a conventional index.
Full-text search requires another strategy.
This is why search engines build specialized indexes.
The database isn't necessarily bad.
The problem is that we're asking one data structure to solve multiple problems.
A logging platform needs several indexes.
5. The Inverted Index
One of the most important ideas behind search engines is the inverted index.
Suppose we have these logs:
Log 1:
"payment failed timeout"
Log 2:
"payment succeeded"
Log 3:
"database timeout"
Instead of storing only:
Log -> words
we build:
Word -> logs
For example:
payment -> [1, 2]
failed -> [1]
timeout -> [1, 3]
database -> [3]
succeeded -> [2]
Now searching:
timeout
becomes:
[1, 3]
instead of scanning every document.
This is the core conceptual transformation:
We trade some write complexity and storage for dramatically faster search.
6. Tokenization
The first step is breaking text into searchable terms.
Given:
Payment failed because gateway timeout
we might produce:
payment
failed
because
gateway
timeout
We normalize them.
For example:
Payment
payment
PAYMENT
could all become:
payment
A simple tokenizer:
import re
def tokenize(text):
return re.findall(r"[a-zA-Z0-9_]+", text.lower())
Then:
tokenize("Payment failed: gateway timeout")
produces:
["payment", "failed", "gateway", "timeout"]
A production search engine does much more.
It may handle:
- stemming
- stop words
- Unicode
- language analysis
- synonyms
- phrase matching
- token positions
- analyzers
But the basic idea remains simple.
Turn text into searchable terms.
7. Posting Lists
Our inverted index can be represented as:
Dictionary<String, PostingList>
Where:
PostingList = [document IDs]
For example:
timeout -> [17, 21, 44, 51, 89]
If we search:
timeout AND payment
we retrieve:
timeout -> [17,21,44,51,89]
payment -> [17,21,30,44]
Then compute:
intersection(timeout, payment)
Result:
[17,44]
Boolean search suddenly becomes a set operation.
That is one of the beautiful things about search engines.
Complex-looking queries often reduce to carefully engineered operations over indexes.
8. Structured Fields Need Their Own Indexes
Full-text search isn't enough.
Suppose we have:
{
"service": "payment",
"status": 500
}
A query like:
status >= 500
shouldn't tokenize 500.
Numeric fields need numeric indexes.
Similarly:
timestamp BETWEEN A AND B
needs a time-oriented structure.
We might maintain:
service index
level index
status index
timestamp index
Conceptually:
service:
payment -> [1,2,8,10]
level:
ERROR -> [1,8]
INFO -> [2,10]
status:
500 -> [1,8]
200 -> [2,10]
This is why schemas matter even when your log system claims to support arbitrary JSON.
Flexible data does not mean structure doesn't matter.
It means the platform has to discover or maintain structure dynamically.
9. Time Is the Natural Partition
Logging systems have one enormous advantage.
Logs are temporal.
Most queries are temporal too.
Developers rarely ask:
Search every log since the beginning of civilization.
They ask:
What happened in the last 15 minutes?
Or:
Show yesterday's errors.
This suggests a powerful architectural decision:
Partition logs by time.
For example:
logs-2026-09-18
logs-2026-09-17
logs-2026-09-16
Or:
logs-2026-09-18-14
logs-2026-09-18-13
Now a query for:
14:00 - 15:00
doesn't touch unrelated data.
Time partitioning also makes retention easy.
If we want to keep logs for 30 days, we can delete old partitions.
Instead of deleting individual rows:
DELETE FROM logs WHERE timestamp < ...
we remove entire partitions.
That is much cheaper.
10. Segments
Now we get to another important concept.
We don't want to modify one giant index forever.
Instead, we can create immutable segments.
Imagine ingestion creates:
Segment A
Segment B
Segment C
Each contains:
documents
inverted index
field indexes
metadata
Once a segment is sealed, we stop modifying it.
This has huge advantages.
Immutable structures are easier to:
- read concurrently
- cache
- replicate
- move between machines
- compress
- recover
- validate
New logs go into new segments.
Old segments remain untouched.
11. Segment Merging
Of course, if we keep creating tiny segments, eventually we'll have thousands of them.
That creates overhead.
A search might need to ask:
Segment 1
Segment 2
Segment 3
...
Segment 1000
So we periodically merge segments.
For example:
A + B + C
becomes:
D
where:
D = merged(A,B,C)
This is called compaction or segment merging.
The general pattern is:
Write small immutable structures
↓
Accumulate
↓
Merge
↓
Produce larger immutable structures
This is a recurring pattern throughout modern storage engines.
You see versions of it in search engines, LSM trees, databases, and distributed storage systems.
12. Write-Ahead Logging
What happens if the indexing node crashes halfway through processing a batch?
We need durability.
Before acknowledging a log event, we can write it to a durable write-ahead log.
Conceptually:
Client
↓
WAL
↓
Memory Buffer
↓
Segment
The WAL contains enough information to reconstruct uncommitted events.
If the process crashes:
Restart
↓
Read WAL
↓
Replay events
↓
Rebuild in-memory structures
↓
Continue
This transforms crashes from catastrophic events into recovery events.
That distinction matters enormously in distributed systems.
13. Batching
We should not write every log individually to disk.
Suppose we receive:
1 log
1 log
1 log
1 log
...
Millions of small disk operations are expensive.
Instead:
log
log
log
log
↓
batch
↓
single write
For example:
Batch size = 5,000 events
or:
Flush every 100ms
The exact values depend on workload.
This creates a classic trade-off.
Larger batches:
- better throughput
- fewer I/O operations
- potentially higher latency
Smaller batches:
- lower latency
- more overhead
- lower throughput
Good infrastructure engineering is often the art of choosing where to sit on these curves.
14. The Ingestion Pipeline
Let's design ingestion.
POST /logs
The request enters the ingestion service.
Step 1:
Authenticate
Step 2:
Validate payload
Step 3:
Normalize fields
Step 4:
Assign event ID
Step 5:
Write to buffer/WAL
Step 6:
Return acknowledgement
The actual indexing can happen asynchronously.
This is important.
We don't necessarily want the API request to wait for:
tokenization
index construction
segment creation
replication
compression
That would make the logging API slow.
Instead:
Producer
↓
Durable queue
↓
Index workers
The queue becomes the shock absorber.
15. Why a Queue Matters
Imagine our application normally generates:
50,000 logs/sec
Then an incident happens.
Suddenly:
500,000 logs/sec
If our indexers can process only:
100,000 logs/sec
we have a problem.
Without buffering:
logs arrive
↓
indexers overloaded
↓
requests fail
With a queue:
logs arrive
↓
queue absorbs burst
↓
indexers process steadily
The queue turns instantaneous pressure into accumulated work.
But this creates another question.
How much backlog can we tolerate?
If the queue grows forever, the system is not healthy.
So queue depth becomes a critical operational metric.
16. Backpressure
A logging system must have backpressure.
Otherwise logs can consume the entire infrastructure.
Imagine a service producing:
5 million events/sec
because someone accidentally enabled debug logging in production.
If we accept everything indefinitely, our logging system may become the next outage.
Possible policies include:
drop DEBUG first
sample repetitive messages
rate-limit noisy clients
reject oversized events
prioritize ERROR and WARN
We can define levels:
CRITICAL
ERROR
WARN
INFO
DEBUG
TRACE
Under extreme pressure, perhaps:
preserve CRITICAL
preserve ERROR
preserve WARN
sample INFO
drop DEBUG
The exact policy is a product decision.
The engineering principle is universal:
A system must know what to do when demand exceeds capacity.
17. Distributed Sharding
One machine eventually becomes insufficient.
So we shard the data.
Suppose we have:
Shard 0
Shard 1
Shard 2
Shard 3
Each shard owns a subset of documents.
We need a routing function.
A simple option:
hash(document_id) % number_of_shards
But logs have timestamps.
Another approach is time-based shards.
For example:
2026-09-18-14
├── shard 0
├── shard 1
├── shard 2
└── shard 3
This gives us two dimensions:
time
+
hash
That can distribute a huge volume of logs while keeping queries time-local.
18. The Query Coordinator
Now suppose the user searches:
service = payment
AND level = ERROR
AND timestamp > now - 1h
The API shouldn't know where every document lives.
Instead:
Client
↓
Query Coordinator
↓
Shard 0
Shard 1
Shard 2
Shard 3
Each shard performs a local search.
Then:
Shard 0 → results
Shard 1 → results
Shard 2 → results
Shard 3 → results
The coordinator merges them.
This is scatter-gather.
19. Scatter-Gather
The coordinator scatters the query:
Q → S1
Q → S2
Q → S3
Q → S4
Then gathers:
R1
R2
R3
R4
and combines them.
For a simple search:
merge(R1,R2,R3,R4)
For a sorted query:
sort(all_results, timestamp DESC)
For an aggregation:
combine(local_buckets)
This sounds straightforward.
But now latency becomes:
query latency =
max(shard latency)
+
coordination overhead
One slow shard can slow down the entire query.
This is the tail-latency problem.
20. Searching Millions of Logs Isn't the Same as Returning Millions
Suppose the user asks:
Find errors from the last hour.
There may be:
3,000,000 matching logs
We shouldn't send all three million across the network.
Instead, each shard can return only the top N.
For example:
Shard 1 → top 100
Shard 2 → top 100
Shard 3 → top 100
Shard 4 → top 100
The coordinator merges:
400 candidates
to produce:
top 100
This drastically reduces network traffic.
The same principle applies to aggregations.
Push computation toward the data.
21. Distributed Aggregations
Suppose we ask:
Count errors by service.
Shard 1 might return:
payment: 100
inventory: 40
orders: 20
Shard 2:
payment: 70
inventory: 10
orders: 50
Shard 3:
payment: 30
inventory: 20
The coordinator merges:
payment: 200
inventory: 70
orders: 70
This is a beautiful distributed computation.
Each node calculates locally.
The coordinator combines partial results.
The general pattern is:
global aggregation
=
merge(local aggregations)
22. Replication
What happens when a storage node dies?
If each log exists on only one machine:
node dies
→ data unavailable
So we replicate.
For example:
Shard 1 Primary
├── Replica A
└── Replica B
Now one failure doesn't necessarily mean data loss.
But replication introduces another question:
When do we acknowledge a write?
Option A:
write primary
→ acknowledge
→ replicate later
Fast, but weaker durability.
Option B:
write primary
→ replicate
→ acknowledge
Slower, but stronger durability.
Option C:
quorum
For example:
3 replicas
2 acknowledgements required
Now we're entering distributed consistency design.
23. Logs Are Usually Append-Heavy
Logging has a useful characteristic.
Most data is written once and then read many times.
We rarely edit:
2026-09-18 ERROR payment failed
into:
2026-09-18 ERROR payment succeeded
This means we can optimize heavily around immutable data.
Our storage model becomes:
append
append
append
append
rather than:
update
update
delete
update
This makes:
- sequential writes attractive
- immutable segments attractive
- compression attractive
- replication simpler
- compaction manageable
The workload itself gives us architectural advantages.
24. Compression
Logs are repetitive.
Consider:
service=payment
service=payment
service=payment
service=payment
or:
status=500
status=500
status=500
Compression can significantly reduce storage.
We can compress:
segments
blocks
posting lists
field values
But compression costs CPU.
So again:
storage efficiency
vs
CPU consumption
vs
query latency
A useful design is to compress larger immutable blocks rather than individual records.
That allows efficient sequential reads.
25. Columnar Thinking
Search systems can also borrow ideas from analytical databases.
Suppose a query asks only:
timestamp
service
status
There is no reason to load:
large message
stack trace
request body
metadata
for every record.
We can store certain fields separately.
Conceptually:
timestamp column
service column
status column
message column
Then queries that need only metadata can avoid loading huge text fields.
This becomes especially useful for aggregations.
For example:
COUNT(*) GROUP BY service
doesn't need the full log message.
26. Bloom Filters
Imagine searching for:
"database_timeout"
A segment might contain no such term.
We don't want to fully inspect the segment just to discover that.
A Bloom filter can help answer:
Could this segment contain the term?
If the answer is:
NO
we can skip the segment.
If:
YES
we inspect it.
The key property is that Bloom filters can produce false positives but not false negatives.
So:
NO → definitely absent
YES → maybe present
This makes them excellent for eliminating unnecessary work.
27. Caching
Logging queries are often repetitive.
Developers might repeatedly search:
service=payment AND level=ERROR
during an incident.
Caching can help.
We can cache:
query → result
or lower-level structures such as:
segment metadata
posting lists
frequently accessed fields
But caching raw results has a problem.
New logs arrive continuously.
A result from:
14:00:00
may be stale at:
14:00:05
Therefore cache policy becomes important.
Historical segments are easier to cache because they are immutable.
This is another benefit of immutable architecture.
28. Retention
Logs can consume enormous amounts of storage.
Suppose we ingest:
100 GB/day
Then:
30 days = 3 TB
and:
1 year = 36.5 TB
before replication and overhead.
So retention must be designed from day one.
We might define:
Hot: 0–7 days
Warm: 8–30 days
Cold: 31–180 days
Delete: >180 days
Hot data stays on fast storage.
Warm data can use cheaper disks.
Cold data can move to object storage.
Eventually:
delete
This is lifecycle management.
29. Hot, Warm, and Cold Storage
A mature architecture might look like:
┌─────────────┐
│ Hot Storage │
└──────┬──────┘
│
aging policy
│
▼
┌─────────────┐
│Warm Storage │
└──────┬──────┘
│
aging policy
│
▼
┌─────────────┐
│Cold Storage │
└──────┬──────┘
│
▼
Deleted
The key insight is:
Not all data deserves the same storage price.
Recent logs are operationally valuable.
Old logs are usually searched less frequently.
Storage architecture should reflect that reality.
30. The Query Language
Now we need a way for users to express searches.
A simple query language might support:
service:payment
level:ERROR
status:500
service:payment AND level:ERROR
service:payment OR service:orders
message:"connection timeout"
status:[500 TO 599]
We can parse this into an abstract syntax tree.
For example:
service:payment AND level:ERROR
becomes:
AND
├── service = payment
└── level = ERROR
Then the query planner decides how to execute it.
31. Query Planning
A naive engine might evaluate:
service = payment
then:
level = ERROR
But which one should come first?
Suppose:
service=payment
matches:
40% of logs
while:
level=ERROR
matches:
1% of logs
It may be more efficient to start with:
level=ERROR
because it produces fewer candidates.
This is query optimization.
The query parser tells us:
what the user wants
The query planner determines:
how to get it efficiently
Those are different problems.
32. Query Execution
For:
service:payment AND level:ERROR
we might execute:
posting(service=payment)
↓
candidate documents
posting(level=ERROR)
↓
candidate documents
intersection
↓
results
For:
message:"database timeout"
we might use positional information in the inverted index.
For:
timestamp > X
we first eliminate irrelevant segments.
Then:
field index
→ candidate documents
→ filters
→ sorting
→ top N
The query engine becomes a pipeline.
33. Pagination Is Harder Than It Looks
Suppose we have:
10 million matching logs
and the user requests:
page 1000
Using:
OFFSET 100000
can become expensive.
The system may have to process and discard enormous amounts of data.
A better approach is cursor-based pagination.
For example:
timestamp
+
unique ID
The first page returns:
last_seen_timestamp
last_seen_id
The next query says:
timestamp < last_seen_timestamp
or, for ascending order:
timestamp > last_seen_timestamp
This lets the engine continue from a known position.
34. Exactly-Once Is Usually the Wrong Dream
Distributed systems fail.
Messages can be retried.
Connections can disappear.
Nodes can restart.
So the same log might arrive twice.
We need an event ID.
For example:
event_id = UUID
or:
hash(source + timestamp + sequence)
Then indexing can be idempotent.
If:
event 123
arrives twice:
first → store
second → recognize duplicate
This is usually more practical than trying to construct a perfectly exactly-once distributed pipeline.
At-least-once delivery plus deduplication is often a powerful architecture.
35. Ordering
Distributed logs don't necessarily arrive in order.
Imagine:
Application:
14:00:01
14:00:02
14:00:03
Network behavior could produce:
14:00:01
14:00:03
14:00:02
Our system needs to distinguish:
ingestion time
from:
event time
Both are useful.
Store:
event_timestamp
ingestion_timestamp
Then we can answer:
When did the event happen?
and:
When did our platform receive it?
This distinction becomes critical during distributed incident investigation.
36. Clock Skew
Even event timestamps can be unreliable.
Machine A might think it is:
14:00:00
while Machine B thinks it is:
13:59:54
Six seconds of clock skew can completely confuse event reconstruction.
So a mature platform should treat timestamps carefully.
Useful metadata includes:
event_time
ingestion_time
host
source
sequence
Don't assume the timestamp emitted by an application is absolute truth.
Distributed systems rarely give us perfect clocks.
37. Schema Management
If every developer can send:
{
"status": 500
}
and someone else sends:
{
"status": "ERROR"
}
we have a type conflict.
Now our indexing engine has to decide:
Is status numeric?
Is status text?
This becomes especially problematic when schemas evolve.
We can introduce mappings:
{
"status": "integer",
"service": "keyword",
"message": "text",
"timestamp": "datetime"
}
Unknown fields can either:
be dynamically indexed
or:
be stored but not indexed
Dynamic indexing is convenient.
Strict mappings provide predictability.
Again:
flexibility vs control
38. Cardinality
Some fields have low cardinality:
level
because there might be only:
INFO
WARN
ERROR
Other fields have enormous cardinality:
request_id
or:
UUID
Indexing every unique value can become expensive.
A logging system must understand cardinality.
For example:
service → low cardinality
country → low/medium
user_id → high
request_id → extremely high
High-cardinality fields can consume large amounts of index memory.
So not every field should automatically receive the same indexing strategy.
39. Security
Logs often contain sensitive information.
Developers accidentally log:
Authorization: Bearer ...
or:
password=...
or:
credit_card=...
A logging platform should therefore support:
field redaction
masking
access control
encryption
audit logs
For example:
password=supersecret
becomes:
password=[REDACTED]
before storage.
We can also define roles:
Admin
Developer
Support
Auditor
Viewer
A developer might see:
service
timestamp
message
while sensitive fields are hidden.
The logging system itself becomes security infrastructure.
40. Multi-Tenancy
If we're building this as a SaaS platform, different customers must not see each other's data.
Every event needs tenant context:
{
"tenant_id": "company_123",
...
}
Queries must automatically include:
tenant_id = current_tenant
But this should not rely solely on the client.
A malicious client shouldn't be able to submit:
tenant_id=company_456
and retrieve another customer's logs.
Authorization must exist below the API layer.
Security should be part of query execution.
41. Observability of the Logging Platform
Here's the funny part.
A logging platform needs logs.
Our own system produces:
ingestion failures
queue depth
indexing latency
segment counts
query latency
replication lag
disk usage
CPU usage
memory pressure
We need to monitor the system that monitors everything else.
Useful metrics include:
logs_ingested_total
logs_dropped_total
queue_depth
indexing_latency
query_latency
segment_count
storage_bytes
replication_lag
We should also expose health endpoints:
/health
/ready
/metrics
Infrastructure that cannot observe itself eventually becomes mysterious.
And mysterious infrastructure is expensive infrastructure.
42. Failure Handling
Let's imagine an indexing node crashes.
Our system detects:
heartbeat timeout
Then:
mark node unhealthy
The cluster manager determines:
which shards are affected
Then replicas can take over.
Eventually:
new replica
is created.
Recovery might look like:
Node failure
↓
Failure detection
↓
Replica promotion
↓
Traffic rerouting
↓
New replica allocation
This is where our logging system becomes a distributed systems laboratory.
We are no longer building a log viewer.
We are building a cluster.
43. Metadata and Cluster State
The cluster needs to know:
Which nodes exist?
Which shards exist?
Who owns each shard?
Which replicas are healthy?
Which indexes exist?
What mappings are configured?
We need a metadata layer.
Conceptually:
Cluster State
├── Nodes
├── Indexes
├── Shards
├── Replicas
├── Mappings
└── Routing
The metadata system must itself be highly available.
If every node has a different understanding of cluster state, chaos follows.
Distributed consensus mechanisms can be used to maintain authoritative cluster metadata.
44. Building the First Version
We don't need to build everything at once.
A practical implementation roadmap is:
Version 1
Single node.
HTTP API
↓
Memory buffer
↓
Local storage
↓
Basic inverted index
↓
Search API
Support:
ingest
search
filter
time range
Version 2
Add:
WAL
segments
compression
batching
Version 3
Add:
query parser
aggregations
pagination
Version 4
Add:
sharding
replication
query coordinator
Version 5
Add:
retention
hot/warm/cold storage
authentication
multi-tenancy
Version 6
Add:
cluster management
rebalancing
failure recovery
advanced query optimization
This incremental approach is important.
Trying to build the distributed version first is a good way to spend six months debugging distributed metadata before you even have search working.
45. A Possible Internal API
Our ingestion API could be:
POST /v1/logs
Content-Type: application/json
Body:
{
"timestamp": "2026-09-18T14:32:51Z",
"level": "ERROR",
"service": "payment",
"message": "Payment failed",
"fields": {
"order_id": "9812",
"status": 402
}
}
Search:
POST /v1/search
Body:
{
"query": {
"and": [
{"term": {"service": "payment"}},
{"term": {"level": "ERROR"}}
]
},
"time_range": {
"from": "2026-09-18T13:00:00Z",
"to": "2026-09-18T15:00:00Z"
},
"limit": 100
}
Aggregation:
{
"query": {
"term": {
"level": "ERROR"
}
},
"aggregations": {
"services": {
"terms": {
"field": "service"
}
}
}
}
The API doesn't need to expose the internal architecture.
That's one of the great things about good infrastructure.
The complexity stays behind the interface.
46. The Core Data Flow
The complete system now looks like this:
APPLICATIONS
│
▼
LOG COLLECTORS
│
▼
INGESTION API
│
validation
normalization
│
▼
DURABLE WAL
│
▼
QUEUE
│
▼
INDEX WORKERS
│
┌──────────┴──────────┐
▼ ▼
TOKENIZATION FIELD INDEXING
│ │
└──────────┬──────────┘
▼
SEGMENTS
│
replication
│
▼
DISTRIBUTED STORE
│
▼
QUERY COORDINATOR
│
scatter/gather
│
▼
QUERY RESULTS
│
▼
USER
This is the architecture hiding underneath what appears to be:
"Search my logs."
47. The Most Interesting Part: Logs Become Data Structures
This is where the entire subject becomes fascinating.
At the beginning we have:
strings
Then:
JSON documents
Then:
tokens
Then:
posting lists
Then:
segments
Then:
shards
Then:
replicas
Then:
distributed query plans
The logging platform is essentially transforming the same information repeatedly so that different operations become cheap.
The raw event is optimized for:
ingestion
The inverted index is optimized for:
search
The numeric index is optimized for:
ranges
The segment is optimized for:
storage and sequential access
The shard is optimized for:
distribution
The replica is optimized for:
availability
The query coordinator is optimized for:
global computation
This is architecture as transformation.
48. Why Elasticsearch Is Interesting
Systems like Elasticsearch can look almost magical from the outside.
You write:
service:payment AND level:error
and a moment later you receive results from an enormous dataset.
But the magic is mostly engineering.
Underneath the interface are ideas from:
- information retrieval
- databases
- distributed systems
- operating systems
- storage engines
- networking
- compression
- concurrency
- fault tolerance
- query optimization
The dashboard is only the visible layer.
The real product is the data engine.
49. What I Would Build in Rust
If I were implementing a serious educational version, Rust would be an interesting choice.
The core structures might look something like:
struct Document {
id: u64,
timestamp: i64,
fields: HashMap<String, Value>,
}
An inverted index:
struct InvertedIndex {
terms: HashMap<String, Vec<u64>>,
}
A segment:
struct Segment {
documents: Vec<Document>,
index: InvertedIndex,
}
A shard:
struct Shard {
segments: Vec<Segment>,
}
And a cluster:
struct Cluster {
shards: Vec<Shard>,
}
Of course, a production implementation would use much more sophisticated structures.
But these abstractions expose the architecture clearly.
50. Async Concurrency
The ingestion pipeline is naturally asynchronous.
We might have:
Producer
↓
Channel
↓
Indexer workers
Workers consume batches:
while let Some(batch) = queue.recv().await {
index(batch).await?;
}
Different workers can process independent shards concurrently.
Queries can also execute concurrently:
let futures = shards
.iter()
.map(|shard| shard.search(&query));
let results = join_all(futures).await;
Rust becomes especially interesting here because the architecture naturally contains:
parallel work
shared state
ownership
channels
backpressure
I/O
The programming language starts reflecting the architecture.
51. Don't Build the Dashboard First
This is a common mistake.
A developer builds:
beautiful dark UI
graphs
filters
search box
charts
before building the engine.
It looks impressive.
But the hard part is:
How fast can I ingest?
How durable are writes?
How do I index?
How do I query?
How do I shard?
How do I recover?
Build the engine first.
A command-line interface is enough:
logctl ingest logs.json
Then:
logctl search 'service:payment AND level:error'
Then:
logctl aggregate 'count by service'
Once the underlying engine works, the dashboard becomes a client.
That is a much healthier architecture.
52. Benchmarking the System
A logging platform needs numbers.
We should measure:
Ingestion throughput
events/sec
Query latency
p50
p95
p99
Storage efficiency
bytes/log
Indexing throughput
documents/sec
Compression ratio
raw bytes / stored bytes
Recovery time
time to restore after failure
Queue delay
ingestion → searchable
That last metric is particularly important.
A logging system can accept logs quickly while taking minutes to make them searchable.
So:
accepted
and:
searchable
are different states.
53. The Real Trade-Off
There is no perfect logging architecture.
You are constantly balancing:
throughput
latency
durability
storage cost
query flexibility
availability
consistency
operational complexity
For example:
More replicas:
+ availability
+ durability
- storage cost
- write cost
More aggressive indexing:
+ query performance
- ingestion performance
- storage
Longer retention:
+ historical visibility
- storage cost
Larger batches:
+ throughput
- real-time latency
More flexible schemas:
+ developer convenience
- predictability
- index management complexity
There is no architecture without trade-offs.
There is only architecture with explicit trade-offs.
54. What Makes a Logging Platform Difficult?
The first 20% is surprisingly easy.
You can build:
POST /logs
GET /search
in a weekend.
The next 80% is where infrastructure engineering begins.
You have to answer:
What happens at 1 million events/sec?
What happens when disk fills?
What happens when a node disappears?
What happens when the queue is full?
What happens when schemas conflict?
What happens when clocks disagree?
What happens when a query hits 500 shards?
What happens when one tenant becomes noisy?
What happens when a segment becomes corrupted?
What happens when replication falls behind?
What happens when a query takes 30 seconds?
What happens when everyone starts searching during an outage?
Those questions are the actual system.
55. The Deeper Lesson
Building a logging platform teaches something larger than logging.
It teaches that software architecture is often about changing the shape of information.
We begin with:
events
We transform them into:
documents
Then:
indexes
Then:
segments
Then:
shards
Then:
distributed results
Every transformation exists because a particular operation needs to become cheaper.
This is one of the deepest ideas in systems engineering.
You don't make every operation fast with one magical data structure.
You create representations optimized for different operations.
56. Final Architecture
Our final conceptual system looks like this:
┌──────────────────────┐
│ Applications │
└──────────┬───────────┘
│
▼
┌──────────────────────┐
│ Log Collectors │
└──────────┬───────────┘
│
▼
┌──────────────────────┐
│ Ingestion API │
└──────────┬───────────┘
│
validation
normalization
│
▼
┌──────────────────────┐
│ Durable WAL │
└──────────┬───────────┘
│
▼
┌──────────────────────┐
│ Message Queue │
└──────────┬───────────┘
│
▼
┌──────────────────────────────┐
│ Index Workers │
└──────────────┬───────────────┘
│
┌──────────────┴──────────────┐
▼ ▼
┌──────────────┐ ┌──────────────┐
│ Text Index │ │ Field Index │
└──────┬───────┘ └──────┬───────┘
│ │
└──────────────┬──────────────┘
▼
┌──────────────┐
│ Segments │
└──────┬───────┘
│
compaction/compression
│
▼
┌────────────────────┐
│ Distributed Shards │
└─────────┬──────────┘
│
replication
│
▼
┌────────────────────┐
│ Storage Tiers │
│ Hot / Warm / Cold │
└─────────┬──────────┘
│
▼
┌────────────────────┐
│ Query Coordinator │
└─────────┬──────────┘
│
scatter/gather
│
▼
┌────────────────────┐
│ Search Results │
└────────────────────┘
And that is the fundamental shape of the system.
Conclusion
A logging platform looks simple from the outside.
Logs go in.
Search results come out.
But underneath that simple interface is a fascinating collection of computer science.
We need:
Inverted indexes for text search.
Field indexes for structured queries.
Time partitioning for efficient temporal access.
Immutable segments for efficient storage.
WALs for crash recovery.
Queues for buffering and backpressure.
Sharding for horizontal scalability.
Replication for fault tolerance.
Scatter-gather execution for distributed queries.
Aggregation merging for global analytics.
Compression for storage efficiency.
Retention policies for lifecycle management.
Caching for frequently accessed data.
Schema management for predictable indexing.
Access control for security.
And eventually:
cluster coordination for keeping the whole machine coherent.
The most interesting part is that none of these ideas exists in isolation.
They interact.
Batching affects latency.
Indexing affects write throughput.
Replication affects durability.
Sharding affects query latency.
Retention affects storage cost.
Cardinality affects memory.
Compression affects CPU.
Caching affects consistency.
Everything touches everything.
That is what makes infrastructure engineering different from simply building another CRUD application.
You are no longer asking:
"How do I store this data?"
You are asking:
"What representation of this data makes the operations I care about cheap, reliable, and scalable?"
That question is much bigger.
And it is the question behind almost every great systems project.
A logging platform is therefore not really about logs.
It is about turning an ocean of events into a structure that a machine can reason about quickly.
That is the engineering challenge.
The dashboard is just the window.
The real product is the engine behind the glass.
Build the engine.
Then build the window.
Top comments (0)