Analytics-Service Performance Rewrite

Task: Analytics-Service Performance Rewrite (10s → <2s) + Fix Mislabeled "Software Request Trend"

Routes affected: /analytics?lang=en (Analytics Performance) and /analytics/booking?lang=en (Booking Analytics)

Engineering bar for this task: This is a rewrite, not a patch. Approach it as a senior backend engineer would — correct data model, correct query strategy for the actual data scale (billions of records in Prometheus), clean separation of concerns in analytics-service, and no shortcuts that will silently break at higher cardinality/volume than what's in front of you today.

---

Problem 1: 10-Second Response Time, Target <2 Seconds, at Billions-of-Records Scale

Root cause diagnosis (do this before writing any "optimization" code)

• [ ] Instrument (or check existing) timing breakdown of the current /analytics and /analytics/booking request lifecycle: how much time is spent in (a) the PromQL query itself, (b) network/HTTP round-trip to Prometheus, (c) response parsing/transform in analytics-service, (d) serialization back to the frontend. Don't optimize blind — confirm where the 10 seconds actually goes.
• [ ] Check whether each dashboard load fires one query per widget, sequentially (likely, given ~15+ distinct widgets across both pages) — serial round-trips to Prometheus, each with its own latency, is a common and easily-fixed source of compounding slowness.
• [ ] Check the PromQL queries themselves for expensive patterns at this data scale:
  • rate()/increase() over very large range vectors (e.g. [6mo] for the Software Request Trend) computed live, with no recording rule backing it — this forces Prometheus to scan and recompute across the full raw sample set on every dashboard load.
  • High-cardinality sum by (...) aggregations (e.g. grouping by customer or a raw request ID field) run against the full time range instead of pre-aggregated series.
  • Overly fine step parameters on query_range calls for long time windows (e.g. requesting per-second resolution over 6 months returns and processes far more data points than the chart can even render).

Required architectural fix (not just query tuning)

1. Prometheus Recording Rules — pre-aggregate at write time, not read time

This is the single highest-leverage fix at "billions of records" scale. Instead of computing expensive aggregations live on every dashboard load, define recording rules that Prometheus evaluates on a schedule (e.g. every 30s–1m) and stores as new, cheap-to-query time series:

# prometheus-rules.yml
groups:
  - name: analytics_aggregations
    interval: 30s
    rules:
      - record: job:api_requests:rate5m
        expr: sum by (route, status_code, service) (rate(api_requests_total[5m]))

      - record: job:bookings:rate1h_by_dispatch
        expr: sum by (dispatch_software) (rate(bookings_total[1h]))

      - record: job:latency:p95_5m
        expr: histogram_quantile(0.95, sum by (le, route) (rate(request_duration_seconds_bucket[5m])))

• [ ] Identify every "heavy" query currently powering a widget (Total Requests, Success Rate, Latency Percentiles, Request Volume Over Time, Software Request Trend, Dispatch Software Performance, etc.) and convert each into a recording rule.
• [ ] The dashboard's live queries should then read the pre-aggregated job:* metrics, not raw _total/_bucket metrics directly — this turns an expensive scan into a cheap lookup of already-computed values.
• [ ] Confirm recording rule evaluation interval matches an acceptable staleness window for the dashboard (e.g. 30s–1m staleness is normally fine for an analytics dashboard; document the chosen tradeoff).

2. Parallelize widget queries in analytics-service

• [ ] Audit the request handler for /analytics and /analytics/booking: if it awaits each PromQL query one at a time, refactor to fire all independent queries concurrently:

  const [totalRequests, successRate, latency, statusDist, ...] = await Promise.all([
    promClient.query(TOTAL_REQUESTS_QUERY, range),
    promClient.query(SUCCESS_RATE_QUERY, range),
    promClient.query(LATENCY_P95_QUERY, range),
    promClient.query(STATUS_DIST_QUERY, range),
    // ...
  ]);

• [ ] Use Prometheus's batch/multi-query capability where available, or at minimum a single HTTP client with connection pooling/keep-alive so N concurrent queries don't each pay a fresh connection-setup cost.

3. Add a caching layer in front of Prometheus for dashboard reads

• [ ] Introduce a short-TTL cache (Redis, in-memory LRU, or similar) keyed by (route, filters, time_range, granularity) in front of the Prometheus query layer.
  • TTL should be close to the recording-rule evaluation interval (e.g. 30–60s) — no value in re-querying Prometheus more often than the underlying aggregation actually updates.
  • This absorbs repeated identical dashboard loads (auto-refresh, multiple users viewing the same range) without hitting Prometheus at all on a cache hit.
• [ ] Cache invalidation: time-based TTL is sufficient here (no need for event-driven invalidation) since this is read-mostly aggregate data, not something requiring strict real-time consistency.

4. Consider a scheduled pre-computation job for the heaviest cross-cutting views (if recording rules + caching still aren't enough)

• [ ] For views that combine multiple metrics into one derived shape (e.g. the "Dispatch Software Performance" table, which needs bookings + forwarded + failed + pending + avg time per dispatch software in one row), consider a background job (cron/worker) that runs every N seconds/minutes, computes the full derived dashboard payload once, and writes it to a fast key-value store (Redis) or small materialized table — the API then just reads that precomputed blob instead of assembling it from multiple live PromQL calls per request. This is the standard pattern for "many widgets, high query volume, data changes slower than page views" dashboards.

5. Query-level hygiene

• [ ] Set an explicit, sane step on every query_range call based on the requested time range and chart width (e.g. don't request 1-second resolution for a 6-month chart that renders ~180 points) — reduces both Prometheus compute and payload size.
• [ ] Set query timeouts on the Prometheus HTTP client (e.g. 3–5s) with a clear error surfaced to the frontend on timeout, rather than letting a slow query hang the whole dashboard response indefinitely.
• [ ] Confirm label cardinality on hot query paths — the earlier cardinality warning (customer/vendor as raw labels) applies directly here; high-cardinality group by in a live query at billions-of-records scale is a direct cause of multi-second latency. Move any genuinely high-cardinality breakdowns (e.g. per-customer) to a separate, less frequently refreshed / paginated view rather than computing them inline with the main dashboard load.

Performance Acceptance Criteria

• [ ] /analytics?lang=en responds in <2 seconds under production-representative data volume (test against a realistic billions-of-records dataset or a representative sample, not an empty/small dev dataset).
• [ ] /analytics/booking?lang=en responds in <2 seconds under the same conditions, once its data pipeline is also fixed (see Problem 3).
• [ ] Recording rules are in place for every widget's underlying aggregation; no dashboard widget computes a live rate()/increase() over a multi-month raw range on each request.
• [ ] Widget queries execute in parallel, not serially.
• [ ] A cache layer exists in front of Prometheus reads, with a documented TTL tradeoff.
• [ ] Load test results (before/after) are documented in the PR — this is a performance task; it needs a number, not just "feels faster."

---

Problem 2: "Software Request Trend (6 Months)" Is Grouping by the Wrong Dimension

Current (incorrect) behavior

The chart currently groups by internal microservice name:

rate-service, obt-connection-service, vendor-connection-service,
user-service, auth-service, vendor-service

This is internal infrastructure, not the "dispatching software" the widget's title claims to show.

Correct behavior

"Software Request Trend" should track request volume over time, per actual dispatching software — i.e. Sixt, Gnet, Rentnet — matching the same three-vendor model already established and correctly used elsewhere on the Booking Analytics page (Dispatch Software Performance table, SLA definitions, Retry & Recovery Analysis).

Fix

• [ ] Identify the query currently powering this chart and confirm it's grouping by service (internal microservice label) instead of dispatch_software/receiverCode-resolved vendor.
• [ ] Change the group by dimension to the dispatch software field (dispatch_software label on the relevant metric — reuse the exact same field/label already used correctly in the "Dispatch Software Performance" table on the Booking Analytics page, don't introduce a second, inconsistent labeling scheme).
• [ ] Confirm the underlying metric being trended is meaningful per the earlier sync/async lifecycle work — e.g. if this chart is meant to show booking request volume (not raw HTTP request volume), it should source from bookings_total-style booking metrics grouped by dispatch_software, not from generic api_requests_total grouped by internal service. Clarify which of these it's supposed to represent before fixing the group-by, since fixing only the label without confirming the correct underlying metric would still produce a technically-relabeled-but-still-wrong chart.
• [ ] Rename the chart's legend/series to Sixt, Gnet, Rentnet once the correct grouping is in place.

Acceptance Criteria

• [ ] "Software Request Trend (6 Months)" shows exactly three series: Sixt, Gnet, Rentnet.
• [ ] The metric being trended is confirmed to represent actual dispatch/booking volume to each vendor, not generic internal service request counts.
• [ ] This chart's per-vendor numbers are broadly consistent with the "Dispatch Software Performance" table's totals for the same vendors over the same range (sanity cross-check, not necessarily exact due to different time bucketing).

---

Problem 3: /analytics/booking?lang=en Still Returns All Zeros / No Data

This is not a new bug — it's the same all-zero state investigated earlier in this project, still unresolved (Total Bookings, Successfully Forwarded, Failed Transaction, Forward Success Rate, Average/P95 Process Time, Retries, Retry Success Rate, In-Flight/Pending, Duplicate Attempt, Booking Transmission Funnel, Booking Transaction Status Distribution, Dispatch Software Performance rows, OBT → Vendor Matrix, Retry & Recovery Analysis, Booking Processing SLA, At-Risk/Stuck Transactions, Top OBT → Vendor Routes — everything on this page is zero or "No data").

Note: the footnote already present at the bottom of this dashboard —
"Rentnet Push Booking Status/TRNo is a receipt, not a confirmation. Sixt process time is request→response. Gnet/Rentnet process time is excluded from the blended average until create events can be joined to terminal callbacks..."
— confirms the correct business logic has already been documented and designed (this matches the sync/async lifecycle work from earlier in this project exactly). The problem here is purely that no real data is flowing through this correctly-designed model — this is a data pipeline / instrumentation gap, not a logic gap.

• [ ] Re-run the diagnostic checklist from the earlier "Booking Analytics all-zero" investigation: confirm booking creation events are actually being captured (Loki/Prometheus), confirm date-range defaults aren't excluding recent data, confirm lang=en isn't leaking into a data filter, confirm the query layer matches the label/field names actually being emitted.
• [ ] Given this page's queries will now also be reading from Prometheus (per Problem 1's architecture), confirm the required booking-lifecycle metrics (bookings_total, per-vendor terminal-state counters, retry counters) are actually instrumented and exposed via /metrics in the relevant services — if this instrumentation was never completed, this page cannot show real data no matter how the query/performance layer is fixed.
• [ ] This is a blocking prerequisite: fix data capture before or alongside the performance work, since there's no point optimizing a query pipeline that returns zero rows.

---

Code Quality Bar (applies to all of the above)

• [ ] TypeScript, strict typing on Prometheus query builders and response shapes — no any on query results.
• [ ] Single responsibility: separate the Prometheus client/query layer, the caching layer, and the widget-specific query definitions into distinct modules — don't inline raw PromQL strings scattered across route handlers.
• [ ] DRY: shared query-building utilities for common patterns (rate/increase over a parameterized range, percentile calculations, group-by-with-fallback-label) rather than duplicating PromQL string construction per widget.
• [ ] Config-driven: Prometheus URL, cache TTL, query timeout, and recording rule names should be environment-configurable, not hardcoded.
• [ ] Error handling: a failed/timed-out Prometheus query for one widget should not fail the entire dashboard response — return partial data with a clear per-widget error/loading state, consistent with the loading/error/empty-state requirements established earlier in this project.
• [ ] Tests: unit tests for query-building logic and cache-key generation; integration test confirming the full /analytics and /analytics/booking response shape against a test Prometheus instance with seeded data.

Deliverables

1. Prometheus recording rules file (prometheus-rules.yml or equivalent) covering every widget's aggregation.
2. Refactored analytics-service query layer: parallelized queries, caching layer, query-building utilities.
3. Fixed "Software Request Trend" query, grouped by dispatch_software (Sixt/Gnet/Rentnet), backed by the correct underlying booking metric.
4. Root-cause fix (or a blocking-dependency callout, if instrumentation is the actual gap) for /analytics/booking returning all zeros.
5. Before/after load test numbers demonstrating the <2s target is met at representative data scale.
6. Brief documentation of the caching TTL / recording-rule interval tradeoffs chosen.