Short answer: the most beginner-friendly API is a small, standards-shaped ingestion interface that accepts structured log records, preserves a stable event identity, and makes the import run ID searchable from the first record to the final outcome. The important choice is not a fashionable endpoint. It is the boundary between an event that was emitted, an event that was accepted, and an event that was actually applied.
For scheduled imports, that boundary determines whether an operator can reconstruct a missing result at 02:10 or merely see that a worker was alive. This article treats centralized logging as an incident-reconstruction system, with the same discipline used for payment and ledger backends: explicit invariants, idempotent writes, and an audit trail that survives retries.
That is the trade-off.
Architecture decision record: what must the log API preserve?
The decision is to expose one ingestion contract around a structured record and to keep query semantics separate from ingestion. A beginner can call it with a JSON object, while the service can later add batching, authentication, or a different transport without changing the event model.
Four invariants define the contract:
- Every record has an
import_run_id, a source identifier, an event timestamp, and a severity. - A retry of the same logical event does not create a second audit fact.
- The accepted response means durable acceptance by the log boundary, not successful import completion.
- The final result is represented by a distinct terminal event, including zero-result runs and failures.
Those boundaries matter because a scheduler firing, a worker starting, and an import producing rows are three different facts. Collapsing them into one success message makes reconstruction ambiguous.
| Option | Beginner experience | Reconstruction quality | Main boundary |
|---|---|---|---|
| Plain text over a generic collector | Easy to emit | Weak unless fields are parsed later | Meaning is recovered after the incident |
| Structured records over a minimal HTTP API | Easy to inspect and test | Strong when identity and outcome are required | Transport durability still needs an explicit contract |
| Full telemetry SDK from the first line of code | More setup | Strong when the team already has conventions | SDK lifecycle and versioning become part of the first task |
The middle option is the practical default for a small application. It gives a new team a request body it can inspect with ordinary tools, while leaving room to map records into an OpenTelemetry-compatible logs pipeline. OpenTelemetry describes logs as a signal with fields that can be correlated with other signals, which is the useful standards direction here; it does not remove the need to define application-level identity.
The limitation is deliberate: this interface does not provide a complete incident platform by itself. It leaves retention, indexing, access control, and delivery guarantees to the receiving system, so a team choosing it must own those decisions instead of treating the endpoint as the whole observability design. That increases design work at the start, but keeps the failure boundary visible.
Why does a scheduled import need an event identity?
A timestamp is not an identity. A message such as import finished can be emitted twice after a timeout, and two legitimate runs can finish in the same second. Use an immutable key composed from the run identifier and a logical event sequence, or generate an equivalent idempotency key at the producer.
The key should remain stable across network retries. The payload can be enriched with a trace ID, host, or deployment revision, but those details must not silently become the identity: a retry may come from a different process. This is an exactly-once mindset applied where it is useful. The network may deliver a request more than once; the receiver makes the resulting audit fact appear once.
A useful record set for one run is small:
record := LogRecord{
EventID: "run-2026-10-25T02:10Z-001",
RunID: "run-2026-10-25T02:10Z",
Event: "import.completed",
OccurredAt: time.Date(2026, 10, 25, 2, 13, 42, 0, time.UTC),
Severity: "INFO",
Fields: map[string]any{"result_count": 1842},
}
The date above is an example event value, not a claim about a deployment. In a real system, the producer owns occurred_at; the collector records its own receipt time as a separate field. That distinction is valuable during clock skew, queue delay, and incident review.
The critical path: accept, deduplicate, and query by run
The smallest useful client should make the identity visible and should not pretend that a successful HTTP exchange proves the import succeeded. The Go example below models the application-facing portion of that path; persistence and authorization remain explicit interfaces.
type LogRecord struct {
EventID string `json:"event_id"`
RunID string `json:"import_run_id"`
Event string `json:"event"`
OccurredAt time.Time `json:"occurred_at"`
Severity string `json:"severity"`
Fields map[string]any `json:"fields,omitempty"`
}
type LogStore interface {
PutIfAbsent(ctx context.Context, record LogRecord) (accepted bool, err error)
}
func recordImportEvent(ctx context.Context, store LogStore, record LogRecord) error {
if record.EventID == "" || record.RunID == "" || record.Event == "" {
return errors.New("event_id, import_run_id, and event are required")
}
_, err := store.PutIfAbsent(ctx, record)
return err
}
PutIfAbsent is the critical contract. Returning accepted=false for a known event is not an error; it is a retry that has already been accounted for. The store must make the identity check and insert atomic. A process-local map is insufficient once two workers or two replicas can receive the same event.
For incident reconstruction, queries should begin with import_run_id, then order by occurred_at and a stable tie-breaker such as event_id. Alerting can be simple: if a scheduled run has a started record but no terminal record within the agreed window, raise an alert; if the terminal record says result_count: 0, treat that as an explicit business outcome rather than missing telemetry. The window is a policy decision, not a universal API default.
What should be rejected, and when is it still valid?
The rejected design is a text-only fire-and-forget logger. It is attractive because the first line is easy to write, but it moves parsing, correlation, and duplicate handling into the worst possible phase: after an incident. It also encourages an alert based on process liveness instead of import output.
That design is valid for local debugging, short-lived development commands, and diagnostic messages whose loss has no operational consequence. It is not sufficient as the system of record for scheduled imports. A second rejected extreme is requiring a complete telemetry SDK before a team has agreed on event names and terminal states. Instrumentation cannot settle an undefined business contract; it can only transport one.
The compliance boundary deserves equal attention. Logs may contain account identifiers, filenames, or payload fragments that fall under retention and access-control rules. Keep sensitive values out of free-form messages, define retention by data class, and record who can query the reconstruction trail. A log API that is pleasant to call but impossible to audit is a poor fit for payment-adjacent systems.
A decision rule for choosing the API
Choose the smallest interface that can answer five questions without joining unrelated data: which scheduled run was this, when did it occur, what did the worker attempt, what result was produced, and was this event already recorded? If an API cannot preserve those answers, a nicer client library will not fix the architecture.
Start with JSON records and a stable ingestion contract. Add batching only after measuring request overhead, add trace correlation when the surrounding system has trace context, and add richer schemas when operators can name the investigation they need to perform. This ordering keeps the first integration teachable while protecting the audit trail that matters during a failure.
Top comments (0)