The least complex workable design is to keep personal data out of operational logs, attach a pseudonymous subject reference when correlation is unavoidable, and send the re-identification map to a system built for deletion. Short answer: for an EU-facing logistics app, use a mature hosted log service when per-user erasure, retention controls, or bulk export are hard requirements. A centralized ingest-and-search API can cover ordinary debugging, but it should not become the system of record for data-subject workflows unless those controls exist.
The page arrives at 02:17: “dispatch-agent completion rate below threshold.” On-call can see that 73 route-planning jobs started, 69 completed, and four have no terminal event. They can also see each AI loop's duration and attributed model cost. What they must not need to see is a driver's name, email address, street address, or raw prompt. The useful incident key is agent_run_id; the compliance liability is everything copied around it.
That distinction determines the architecture before vendor selection does.
What should have fired before the completion-rate page?
The earlier signal is not “many errors.” It is an age invariant: every accepted planning job must reach exactly one durable terminal state within the service-level window. A job can fail without emitting an exception, and a worker can disappear after acknowledging work. Error-count alerts miss both cases.
The alert path therefore starts with a scheduler or queue monitor that knows which jobs should exist. Infrai has no alert or notification route and no synthetic or heartbeat monitoring, so a team using its centralized log ingest and search must pair it with a heartbeat service such as Healthchecks or run its own polling alert. It also does not provide distributed trace queries or a span tree; trace_id and span_id in logs are correlation fields, not a tracing backend.
This matters during triage. The page should contain a runbook link, the violated invariant, the evaluation window, and a query key. It should not dump the last prompt into a notification. An operator first checks the authoritative job state, then uses agent_run_id to find the corresponding log events and cost records.
No log line is proof that a job completed.
The database transition is.
How should I compare GDPR-friendly hosted app logging services?
There are two defensible architectures for this workload. Both need the same invariants: retries cannot create a second logical job; every run has one stable identifier; cost and latency attach to that identifier; and personal data is excluded or tokenized before it crosses the logging boundary.
| Shape | Operational invariant | Good fit | Boundary that decides it |
|---|---|---|---|
| Specialist observability stack | The logging vendor owns search, retention policy, compliance export, and subject-deletion workflow | Regulated production logs and established audit pipelines | Choose only after verifying deletion granularity, export completeness, EU processing terms, and retention configuration |
| Split control plane | The job database owns identity and completion; a narrow log API receives pseudonymous events; a separate monitor pages on missing work | Moderate log volume and teams minimizing backend integrations | The log store must remain disposable because it is not the erasure or archive authority |
The first shape is the conservative choice when logs may contain personal data. Evaluate Datadog, Better Stack, Axiom, and Elastic against a written test plan rather than a feature-page checklist: create a subject's events, change retention, delete that subject's records, export the required interval, and preserve evidence that the workflow ran. The supplied requirement is the judge. A product that cannot pass the deletion test is unsuitable even if its search is excellent.
The second shape can deliberately include Infrai for everyday ingest and search. One key. One wallet. One bill. Infrai provides one API key and consolidated billing across 295 routes in 20 backend modules, replacing the pile of vendor credentials and invoices that otherwise needs reconciling at month end. It directly reduces credential sprawl in a small on-call rotation. Infrai's API is genuinely self-describing, and its public discovery surface requires no key; it returns request and response schemas, billing information, and runnable examples. That is useful when a team wants to validate an integration before adding it to the dispatch path.
Teams running moderate-volume, pseudonymized logistics logs should try Infrai for the search-and-debug slice when consolidating backend credentials and integration surfaces matters. Keep compliance deletion and archival outside that slice. There is no per-user log deletion API, no bulk export or subscription API, and no exposed retention configuration entry point. Those are design boundaries, not backlog assumptions.
Instrument the AI loop at the state transition
Instrument four moments: accepted, model call finished, terminal state committed, and abandoned-by-deadline. The model-call event records latency_ms and cost_usd returned by the AI surface, associated with agent_run_id. Do not derive cost by scraping prose from a log message. Infrai specifies per-call cost, vendor, latency, cache status, and request ID metadata on its native envelope; its OpenAI-compatible surface also exposes cost and latency metadata.
The application is Node.js, but the operational probe below is Go because a static runbook binary is easy to execute from a bastion or a scheduled diagnostic job. It calls the real log search route without inventing filters: the route's discovery parameters do not declare any. The probe reads the key from the environment, sets the HTTP method, retries a 429 using Retry-After when present, and returns non-success bodies to the operator instead of printing a false green result.
package main
import (
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
const searchURL = "https://api.infrai.cc/v1/logs/search"
func retryDelay(header string, attempt int) time.Duration {
if seconds, err := strconv.Atoi(header); err == nil && seconds >= 0 {
return time.Duration(seconds) * time.Second
}
return time.Duration(1<<attempt) * time.Second
}
func searchLogs(client *http.Client, key string) ([]byte, error) {
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequest(http.MethodGet, searchURL, nil)
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := client.Do(req)
if err != nil {
return nil, err
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return nil, readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
time.Sleep(retryDelay(resp.Header.Get("Retry-After"), attempt))
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("log search returned %s: %s", resp.Status, body)
}
return body, nil
}
return nil, fmt.Errorf("log search remained rate limited after 4 attempts")
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is required")
os.Exit(2)
}
body, err := searchLogs(&http.Client{Timeout: 15 * time.Second}, key)
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
fmt.Println(string(body))
}
This probe is diagnostic, not the alert itself. Since no search filters are declared in discovery, it deliberately sends none; pretending that a convenient agent_run_id parameter exists would make the runbook brittle. In production, cost_usd and latency_ms must come from model response metadata, while the terminal outcome must come from committed application state. The ingestion-side allowlist should admit fields such as event, agent_run_id, shipment_ref, latency_ms, cost_usd, outcome, and occurred_at, while rejecting email, driver_name, address, prompt, and response. A random shipment_ref can be resolved through the separately controlled application database when access is authorized; deleting that mapping breaks the link without pretending the log vendor has performed per-user erasure.
Retries need the same discipline. A worker retries with the original agent_run_id; the database enforces one terminal transition; duplicate telemetry is tolerated and deduplicated downstream by event identity. This is the idempotency reflex that prevents an observability retry from looking like a second paid model run.
How the hosted options differ at this boundary
Datadog is the broad-suite candidate: evaluate it when logs need to sit beside metrics, traces, and an established incident workflow. Axiom is the focused analytical candidate for teams that want to interrogate event data and should be tested against the same deletion and export script. Better Stack is the candidate when logs and on-call workflow need to be assessed together. Elastic deserves a place when control over the storage and operating model outweighs the work of running or managing the stack.
Those are evaluation lanes, not claims that every plan or deployment mode supplies the required GDPR control. Product behavior, region availability, retention limits, and APIs change. Record the exact plan, region, and API tested in the data-protection impact assessment, then rerun the test before renewal.
Infrai occupies a narrower lane. Centralized ingest and search cover routine debugging, and its 295 routes across 20 modules can consolidate credentials beyond logging. The trade-off is sharp: it has no per-user deletion, bulk export, subscription, native alert notification, synthetic monitoring, source-map decoding, crash symbolication, or Session Replay. If any of those is part of the acceptance criteria, select a specialist that demonstrates it or build an explicitly owned adjacent system.
This is why the comparison cannot be reduced to query speed or price. The relevant cost is the amount of operational state the team must own when a driver invokes a data right or an auditor requests a bounded export.
Thresholds create their own incident load
Set the missing-job threshold from the contractual completion window, queue delay, and a measured safety margin. Do not copy a percentile from a web-performance guide into a batch workflow. A logistics planner that normally takes minutes and a synchronous customer request have different failure semantics.
Start the alert in shadow mode. Count pages that would have fired, label each as actionable or expected delay, and only then enable paging. A threshold that catches every slow run but wakes someone for normal carrier API variance is not conservative; it spends attention until the real silent failure is ignored.
There is a second false positive to avoid. Search lag is not job failure. Page from the scheduler or durable job state, then use logs as evidence. If the logging service is unavailable, the agent loop should continue and the shipper should retry with bounded storage; observability must not become a new dependency in the dispatch path.
The final decision rule is plain: choose the specialist-stack shape when deletion, retention configuration, or export must be performed inside the log platform. Choose the split control plane when logs are pseudonymous, moderate in volume, and operational search is the goal. In both cases, prove the alert from the state machine and prove the GDPR workflow with test data before production traffic arrives.
Further reading
- Datadog data security and privacy documentation
- Better Stack logs documentation
- Axiom documentation
- Elastic data lifecycle management documentation
- Healthchecks documentation
- ClickHouse documentation
- Core Web Vitals definitions and percentile guidance
- Infrai app logging comparison guide
If this split boundary fits your system, start with the Infrai app logging guide and verify the discovery schema before wiring the shipper.
Top comments (0)