Where Were the Logs? Building a Log Ingestion Engine That Never Silently Drops a Log

Three hours were lost in this example scenario. Two hours were spent on root cause analysis (where an engineer would log onto each server, open up various log files, and search manually). There was no error reported from any of the applications on the monitoring dashboard, meaning that no panels highlighted the service that was failing. There was also no clear indication of which service was failing the most. The compliance team wanted to see audit trails, but there were none to provide.

The brief is the running example I built this project from, and I later published the project as a repository.

---

TL;DR: Go service which receives batched JSON on /logs, validates at field level, enriches and routes to three SQLite destinations via RabbitMQ with a bounded in-memory fallback. Admission is durable via a write-ahead spool fsynced before 202; unacked spool entries are replayed on startup. Batch writer with bounded retries and rotating dead-letter file, idempotent writes per log_id. Load test sustained 1,000 logs/s acceptance with 12 pipeline workers, batch 100, on a laptop; default 4 workers do not sustain the target.

---

1. The problem

There are twelve services, each producing bespoke log files onto individual disks. There is no central search, no live dashboard, and no audit trail. It can take an hour to determine which service (or machine), or which file had the issue. By using grep on a laptop, it can be found in seconds. However, using grep on all twelve different formats takes an entire afternoon.

This resulted in three requirements:

• A central search of all log files.

• Signals in minutes about which service has an issue and how big the issue is.

• An audit trail that is persistent and no data is lost when restarting the service.

The log ingestion engine has been built to provide some of the searchable requirements (see section 7).

---

2. The goal and the ground rules

Overview

A centralised logging platform to receive logs using an API, validate and enrich log data, route by service, store it reliably, and provide signals so we can see issues occur before our customers do.

Constraints

The brief says we must limit our libraries. I made a few changes here:

• Used the rate-limiting library golang.org/x/time/rate rather than implementing my own token bucket. The interesting engineering was implementing per-IP keys, request shaping, and Retry-After.

• The frontend uses React with Vite, Tailwind CSS as a utility CSS framework, Space Grotesk from the Google Fonts CDN, and Chart.js for charts, instead of hand-written HTML/CSS.

Storage is kept within the brief:

• Three databases using SQLite.

• One file per destination.

The configuration is simple with only .env files, integer values for duration, and ROUTING_RULES. The cost of using SQLite on real logs is discussed in section 7.

---

3. Architecture at a glance

This diagram represents the architecture shipped with the project:

flowchart TB
    client(["Clients"])

    subgraph ingress["HTTP ingress"]
        direction TB
        rateLimit["Rate limiter<br/>per IP, in-memory"]
        logsEndpoint["POST /logs"]
        validate["Validate batch<br/>body size + per-entry errors"]
    end

    spool[("Write-ahead spool<br/>fsync before ack")]
    rawQueue[("raw_logs<br/>bounded channel")]

    subgraph processing["Pipeline worker pool"]
        direction TB
        enrich["Enrich<br/>received_at · source_ip · environment"]
        route["Route<br/>service → destination rules"]
    end

    subgraph publishing["Publisher"]
        direction TB
        rabbitPublisher["RabbitMQ publisher<br/>persistent + confirms<br/>5s publish timeout<br/>auto-reconnect with backoff"]
        fallback[("Bounded in-memory fallback<br/>used while disconnected")]
    end

    broker[("RabbitMQ<br/>exchange_service1..3<br/>queue_service1..3<br/>routing key: log.write")]

    subgraph delivery["Consumers and storage"]
        direction TB
        supervisor["Supervisor<br/>one connection<br/>declares topology, reconnects"]
        amqpConsumer["AMQP consumers<br/>one connection per queue<br/>ack after stored"]
        fallbackConsumer["Fallback consumer"]
        batchWriter["Batch writer<br/>per-destination buffers<br/>retry with backoff"]
        sqlite[("SQLite<br/>logs_service1.db<br/>logs_service2.db<br/>logs_service3.db")]
        deadLetters[("Dead-letter NDJSON<br/>one shared file<br/>rotated by size")]
        replay["cmd/replay"]
    end

    client --> rateLimit --> logsEndpoint --> validate --> spool --> rawQueue
    rawQueue --> enrich --> route --> rabbitPublisher
    rabbitPublisher -->|"connected"| broker
    rabbitPublisher -->|"publish fails"| fallback
    fallback -.->|"fallback full: dead letter"| deadLetters
    broker -->|"deliver"| supervisor
    supervisor --> amqpConsumer
    fallback -->|"stored directly, never re-published"| fallbackConsumer
    amqpConsumer --> batchWriter
    fallbackConsumer --> batchWriter
    batchWriter -->|"stored"| sqlite
    batchWriter -->|"retries exhausted"| deadLetters
    deadLetters -.-> replay
    replay -.->|"replay"| sqlite

There are 6 stages within the pipeline:

• Post

• Validate

• Buffer

• Enrich/Route

• Publish/Fallback

• Consume/Store
  

flowchart LR
    post["1 Post<br/>rate limit,<br/>accept the batch"]
    validate["2 Validate and spool<br/>per-entry errors,<br/>fsync admission"]
    buffer["3 Buffer<br/>bounded raw_logs<br/>channel (10,000)<br/>202 or 503"]
    enrichRoute["4 Enrich and route<br/>stamp fields,<br/>pick destination"]
    publish["5 Publish<br/>RabbitMQ, or the<br/>in-memory fallback"]
    store["6 Consume and store<br/>batched SQLite writes,<br/>dead letters on failure"]

    post --> validate --> buffer --> enrichRoute --> publish --> store

Each stage handles one specific type of failure/slowdown.

• The rate limiter handles abusive clients.

• raw_logs handles separating quick HTTP traffic from slow enrichment.

• The publisher fallback handles outages of the broker. The logs are each written to storage once, as persistence is idempotent on a per log_id basis. They are never ping-ponged between the broker and memory.

• The batch writer spreads the cost of inserting into SQLite.

• The dead letter file handles unrecoverable failure.

The fallback is consumed by a dedicated consumer and written directly to storage. No re-publishing occurs. As consumers only acknowledge once data has been stored, it provides at-least-once delivery. Persistence within the database is idempotent on a per log_id basis, so duplicate data can be handled.

---

4. The stages

4.1 Ingestion and validation

This will validate the Content-Type header, ensure the body is a certain size using http.MaxBytesReader, parse the JSON, and ensure that there are four fields for each log entry. These are:

• Timestamp (RFC3339)

• Service (at most 100 characters)

• Level (INFO/WARN/ERROR/DEBUG)

• Message (at most 10,000 characters)

Validation runs first. Valid entries are appended to the write-ahead spool and fsynced to disk. The handler then attempts to enqueue the batch onto the bounded raw_logs channel. If the enqueue succeeds within CHANNEL_SEND_TIMEOUT, the handler returns 202 with the batch ID and per-entry rejection details; if the channel is full or the send times out, the handler returns 503. The 202 therefore means the batch is durably admitted for replay and now being processed, not yet in SQLite. The spool guarantees the batch can be replayed on restart. In the 503 case, the spool entry is retained so it can be replayed later; the client may retry and create duplicates, which receive new log_ids and are therefore stored as separate rows (idempotency does not collapse them).

Design decision: I decided to accept partial batches. This will mean that if a log has an invalid time, it will only lose one log instead of one hundred. Each valid log entry will have a UUID added as the log_id field. This means they can be safely resent and played back.

Design decision: The spool fsync is performed inline on the request path to make a 202 durable. This breaks the “no blocking” ideal and adds latency on the critical path, but it guarantees admission durability. The fsync is small per batch, and the cost is accepted for durability; compaction of the spool log and .acked file occurs only when every spooled ID has been acknowledged.

func ValidateLogEntries(logEntries []model.LogEntry) (validEntries []model.LogEntry, rejectedEntries []model.RejectedEntry) {
 for entryIndex, logEntry := range logEntries {
  fieldErrors := validateLogEntry(logEntry)
  if len(fieldErrors) > 0 {
   rejectedEntries = append(rejectedEntries, model.RejectedEntry{
    Index:  entryIndex,
    Errors: fieldErrors,
   })
  } else {
   validEntries = append(validEntries, logEntry)
  }
 }
 return validEntries, rejectedEntries
}

Valid and invalid log entries will be separated by ValidateLogEntries whilst retaining their original index.

Every log entry needs to have a time, service, level, and message. POST /logs can return 202, 400, 413, 415, 503, or a 429 from the limiter.

4.2 Protecting the boundary

Rate limiting middleware using a lazy in-memory token bucket, keyed on the source IP address. Token buckets are purged explicitly when idle.
Rejected requests will not consume a token.

Design decision: The AllowWithoutConsumingCapacity method is used. This method checks how much delay is required. If the delay is positive, the reservation is cancelled, and the request is rejected before any work is done, returning 429 with a Retry-After header in whole seconds. The rejected request keeps its token.

reservation := entry.limiter.Reserve()
delay := reservation.Delay()
if delay > 0 {
 reservation.Cancel()
 return delay, false
}

• Rate limiting middleware uses a token bucket for each IP address. 429 status code is returned with a delay in whole seconds.

• No token is consumed on rejection. X-Forwarded-For is trusted only when a trusted proxy is configured; with no proxy list, the socket address is used for identity.

4.3 The raw_logs buffer

I have a bounded channel that defaults to 10,000 entries for RAW_LOGS_BUFFER. Validations happen quickly, but enrichments happen at their own pace.

Design decision: Producers wait for a bounded 100 ms rather than fail instantly. PushBatch rejects oversized batches immediately, then takes the producer lock and waits up to CHANNEL_SEND_TIMEOUT 100 ms for room for the entire batch. This is blocking with a bounded timeout, providing backpressure.

for len(rawLogs) > cap(rawLogsQueue.channel)-len(rawLogsQueue.channel) {
 select {
 case <-capacityCheck.C:
 case <-sendContext.Done():
  return ErrQueueOverloaded
 }
}

Writing straight to SQL would be on the slow path. Using the bounded channel means they are separated. I use back pressure by returning a 503.

4.4 Enrichment

Adds received_at, source_ip, and environment (defaulting to production) from config. Also adds log_id, batch_id, and retry_count (starting at 0) to every log. No I/O operation, so no failure case.

Design decision: All data is captured at the edge as it is based on request context. This means I only have access to the socket address, forwarded chain, and received time. These are stamped and then passed through. If any of these fields are missing, a sensible default is used.

return model.EnrichedLog{
 LogEntry:    rawLog.LogEntry,
 LogID:       logEnricher.logIDOrDefault(rawLog.LogID),
 BatchID:     rawLog.BatchID,
 ReceivedAt:  receivedAt,
 SourceIP:    sourceIP,
 Environment: logEnricher.applicationEnvironment,
 RetryCount:  0,
}

received_at uses the handler clock; source_ip uses the socket address or the first valid X-Forwarded-For header from behind a trusted proxy. All of these are captured at the edge.

4.5 Routing

The destination is stamped based on a validated mapping from the .env file using the ROUTING_RULES=claims=service2,billing=service3 format. If an unknown service is provided, it will default to service1. The config is loaded using only godotenv with no merging.

Design decision: This was a design decision to keep the routing rules in the .env file. I accept unknown services, as I can have new services logging and then add a new rule. All destination names are checked on startup and will fail if an unknown value is found. This means any typo will cause a crash on startup rather than the service being incorrectly routed.

destination, exists := serviceRouter.rules[enrichedLog.Service]
if !exists {
 slog.Warn("service has no routing rule, using default destination", "service", enrichedLog.Service, "default_destination", string(serviceRouter.defaultDestination))
 enrichedLog.Destination = serviceRouter.defaultDestination
}

A validated map of ROUTING_RULES has been created to route to either of the 3 destinations based on the service. The default destination has been set in the config and will show a warning if there is no rule to match.

4.6 Queuing with RabbitMQ

3 direct exchanges (exchange_service1..3) are bound to durable queues with a routing key log.write. Messages are published using amqp.Persistent, with a 5-second timeout (PUBLISH_TIMEOUT) and confirm mode. The publish will only return nil once the broker has acknowledged. The publisher will retry with backoff.

Design decision: Each destination uses an individual exchange and queue so that messages do not queue up in one queue, starving the others. We always get confirms and cannot allow fire-and-forget.

if publishError := publisher.publishOperation(enrichedLog); publishError != nil {
 publisher.markDisconnected()
 slog.Warn("publishing to rabbitmq failed, using in-memory fallback", "error", publishError, "destination", enrichedLog.Destination, "batch_id", enrichedLog.BatchID)
 return publisher.publishToFallbackAndRecord(enrichedLog)
}

publisher.recordPublishedMessage()
return nil

The broker is a persistent buffer, so it can handle bursts of activity and restarts. If it goes offline, messages will be sent to an in-memory store with a maximum size. These messages will be written to the store and will never be republished. The publisher will retry with backoff.

Timeouts on confirms do not necessarily mean the broker has failed. In ambiguous situations, the publisher will fall back to using the in-memory store. This means that a message that has been accepted by the broker may be stored twice. However, storage is idempotent based on the log_id, so duplicate entries will be merged into a single row.

4.7 Storage, retries and dead letters

A single batch writer for SQLite. Consumes:

• AMQP consumers

• Fallback consumer

• cmd/replay

Flush per destination at 100 logs WRITE_BATCH_SIZE or every 1 second WRITE_FLUSH_INTERVAL, one transaction per destination. Failed flush retries: retry one after 1s, retry two after 5s, retry three after 10s. Four attempts over ~16s. If all fail, the whole batch is handed to a dead letter file as NDJSON, rotated by size.

The dead letter file is appended to in NDJSON format. It will be rotated by closing and renaming the file. Each entry is written with a new line and flushed. If a crash occurs while writing, the last line may be truncated. cmd/replay reads line by line and stops with an error on the first unparsable line, so a truncated tail causes replay to fail.

As the same destination databases are opened by cmd/replay, it is safe to run during the service. SQLite will serialise writes to the database per transaction. Both processes will use WAL and a busy timeout. This means the replay process will wait for/obtain a write lock, and the file will never be corrupted.

Design decision: Two triggers have been used to flush the data. This provides a good balance between latency and efficiency. Persistence is idempotent, as each log_id will be persisted using ON CONFLICT (log_id) DO NOTHING on the unique index. Duplicate rows will not occur, as messages will not be duplicated when redelivered, and dead letters will not be duplicated when replayed. The dead letter file can safely be replayed twice.

deadLetterError := batchWriter.deadLetterBatch(enrichedLogs, storeError)

The writer will flush each destination once every 100 logs or 1s. If an error occurs, it will retry with a backoff. The retry_count will be stamped, and the message will be dead-lettered. Recovery can be done using cmd/replay, which will use the same idempotent StoreBatch.

The dead letter store contains NDJSON format for each batch. It contains the full enriched log, storage exception, and time of failure.

4.8 Observation

The in-memory registry contains the following metrics:

• Receipts

• Request counts per level/service

• Rate-limited requests

• Dead letters

• Write errors

• 1-minute moving average of throughput

Design decision: Metrics are collected by the pipeline. The limiter collects the number of times it rejected a request. The writer collects dead letters and write errors. The queue backlog can come from either the RabbitMQ management API or estimated from publish minus store if the API isn't available.

A single registry containing metrics from each stage. Backlog will be collected from local estimate if the management API isn't available. The metrics can be viewed as a JSON snapshot or via server-sent events.

---

5. Designing for failure

The table maps scenarios to outcomes.

Scenario	The client sees	Where the log ends up	Section and path
RabbitMQ unreachable	202 still, if memory capacity lasts	Stored directly by the fallback consumer, never re-published	4.6
RabbitMQ down and fallback full	202 still	Dead-letter NDJSON entry with the reason	4.6, publish error path to the sink
SQLite write failure	Nothing, by design	Retried at 1, 5, then 10 seconds, then the whole batch dead-lettered	4.7
Queue overloaded past the send timeout	503	Durably spooled, not yet enqueued; will be replayed on restart, client may retry and create duplicates handled by idempotency	4.3
Body or batch oversized	413	Not ingested, visibly rejected	4.1
Wrong content type	415	Not ingested, visibly rejected	4.1
Per-IP limit exceeded	429 + Retry-After in whole seconds	Not ingested, visibly rejected, token kept	4.2
Ungraceful crash	Transport error	In-memory buffer and writer batches lost; spooled batches survive on disk and are replayed on restart. What that costs is section 7	7

Two failure mechanisms.

• Closed/shutting down writer: Consumer nack's messages immediately, and the broker will redeliver shortly. This only happens if the writer is shutting down.

• SQLite flush failure: Message will be scheduled up to 16s before dead-lettering. This is spread out to reduce retries.

---

6. Proof

I have 172 individual tests across 15 packages that are CI'd with the race detector. Timings are simulated using clock injection as I am testing out how the retry system deals with exhaustion in milliseconds.

Integration tests under cmd/server/integration_test.go use real components. This includes end-to-end functionality, zero dead letters during an outage, flush functionality triggered both ways, and retry to dead letter when the store fails.

I also have a load that simulates a constant request rate with cmd/loadtest. It uses fifty IPs in the documentation range as sources, and will output statistics on how many requests have gone through per status and latency percentiles.

I ran this with 12 pipeline workers, batch size 100, and 8 senders. This gave me 300 requests, each with 100 entries. This gave me 30,000 logs accepted in total. I ran this on my laptop with RabbitMQ and SQLite, with spool fsync enabled.

Sustained target run	Result
Sustained ingestion	1000.0 logs/s, zero rejections
Response latency	mean 2.9 ms, p50 2.2 ms, p95 7.1 ms, p99 14.1 ms
End-to-end accounting	300 requests accepted, all 30,000 accepted logs stored when the queue drained 47 seconds later, no dead letters
Server footprint	~26 MB RSS, ~9 percent CPU

This proves:

• Sustained acceptance rate of 1,000 logs/s on one laptop with local broker and SQLite, confirmed by row-count reconciliation.

• End-to-end with 30 seconds of sending time and 47 seconds of draining. Ingestion accepted at 1,000 logs/s while storage lagged; the backlog sat in the RabbitMQ queues. 30,000 logs in 77 seconds, about 390 logs/s including tail. This end-to-end rate reflects the publisher confirm loop, not consumers or SQLite. The test was mapped to a single service, so a single SQLite writer did the work; spreading across destinations does not change the per‑log confirm latency.

Does not prove:

• Rate limiter saturation.

• Multiple instances.

NOTE: When using 12 workers, you can get a throughput of 1,000 logs per second, but if you use the default of 4, you won't. You should aim for around 100 logs per second per worker. The confirmation loop is the limiting factor.

---

7. Trade-offs and what I would change

• Log scale makes columnar storage ideal. Failure logic is contained within the code in SQLite. Batch writer means I can batch inserts. I would use ClickHouse in production. Two triggers mean the contract remains intact.

• I guarantee at-least-once delivery by having an idempotent storage per log_id. If a crash occurs between inserting a record and sending the ack message, then the records will be delivered again. The command and replay functionality means the dead letters are applied again. The log_id is set during ingestion, and using ON CONFLICT DO NOTHING ensures only one row will exist for both.

• Dead letters are applied at the batch level. After 16s of failed flushes, up to 99 logs that have done nothing wrong will be moved to a dead letter file. These are stored and can be restored. They are also grouped into batches.

• In-memory state will be lost if there is an ungraceful crash. The writer has a buffer of less than 1s per destination, and the fallback will be lost. The broker will have the confirmed but unacknowledged messages. Spooled batches survive on disk and are replayed on restart, so admission is durable even if the process crashes; the cost is potential duplicates on retry.

• The spool adds fsync latency on the request path and requires compaction of the spool log and .acked files on restart. This is a deliberate trade-off for durable admission.

• Having a single point of state makes scaling out difficult. I have a limiter, counters, and backlog estimation, which are held in memory. The second replica will have its own buckets; these will be reset when the service is restarted. Having a shared rate limiting store such as Redis and storing/persisting metrics allows scaling out.

• No search endpoint. Initially, this wasn't searchable. Everything is stored centrally and monitored. Logs can be viewed through SQLite and dead letter tools. Next step is to implement full-text search on the logs table in SQLite.

• No alerts. Counters are in place to trigger, but paging on a threshold with cooldown and dedupe is a monitoring-stack integration.

What I would improve:

• Search endpoint

• State for scaling out

• Alerts

Performance tip: The max we can publish per log is the publish-confirmed rate. If I were to publish the entire batch per confirmation, then this would increase the max per worker.

8. Lessons

• Defence of the ingest boundary. Inexpensive inline decisions, and things to bound and make visible. All components have capacity and timeout so we can never block or drop.

• Batching in async pipelines. We can be honest about our options. Either take the whole batch, or fail the whole request. The producer lock ensures we can do this under concurrency. When shutting down, we drain the channel, flush the writer, close the store, and lastly stop the dead letter writer.

• Designing for queue/store failures. Each failure point has a different decision to make. If the broker fails, we use a one-time fallback store. If SQLite fails, we will retry with a limit, using stamps and dead-lettering out. If we fully fallback, we get dead letters. We can chain these decisions. It makes it easier to have a stage that can't fail.

• Registry for event-to-action. The least appealing but most useful decision. We can pull from our registry. During integration testing, the broker failed, but the metrics endpoint still responded, as the fallback estimate was in the same registry. We can instrument what we are already doing. The signals will fail at the same time as the pipeline, meaning they are exactly when we need them.

Review Questions

#	Question	One-line answer
1	Which fields are required, and which status codes can POST /logs return?	timestamp (RFC 3339), service (100 characters), level (four values), message (10,000 characters); 202, 400, 413, 415, 503, plus 429 from the limiter.
2	How does the rate limiter work, and what happens when the limit is exceeded?	A per-IP token bucket in middleware before the handler; 429 with Retry-After in whole seconds, and the rejected request keeps its token.
3	Why an in-memory channel instead of writing straight to SQLite?	The channel keeps the slow path out of the request and turns queue pressure into 503 backpressure with a timed, all-or-nothing send.
4	Which enrichment fields are added, and where does each value come from?	received_at from the handler's clock, source_ip from the socket address or a trusted proxy's X-Forwarded-For, environment from configuration, all captured at the edge.
5	How are routing rules structured, and how is a database chosen?	ROUTING_RULES from .env maps service to one of three destinations, stamped on the log, with the configured default and a warning when no rule matches.
6	Why RabbitMQ, and what changes when it is unavailable?	The broker is the disk-backed buffer absorbing bursts; down means the bounded in-memory fallback stores directly, never re-publishes, and the publisher reconnects with backoff.
7	How does batch writing work, and what is the retry strategy?	One writer flushes per destination at 100 logs or one second inside a transaction, retries once per backoff delay stamping retry_count, then dead-letters the batch.
8	What reaches the dead-letter file, and how is it recovered?	Batch-level entries as NDJSON with the log, the error and the timestamp, rotated by size, restored by cmd/replay through the same idempotent StoreBatch.
9	Which metrics are tracked, and what does GET /metrics.json return?	Receipt, level and service counts, error rate, throughput, queue backlog, rate-limited requests, dead letters and write failures, as one atomic JSON snapshot.
10	Which alert conditions exist, and how is alert spam prevented?	None are built: the counters are ready to trigger against, and alerting with its cooldown design is a monitoring-stack integration named in section 7.