The pipeline that was fine at a hundred thousand events a day starts failing in three specific ways at a few million an hour. Dashboards lag by hours instead of seconds. Duplicates appear in aggregates and nobody can say when they started. And reconciling usage against invoices turns into a manual exercise at the end of every period.
None of those are throughput problems exactly. They are consequences of a design that treated event tracking as logging rather than as a system whose output is money.
Short answer
High-volume event tracking stays accurate when raw events go to an append-only log first, ingestion is idempotent with windowed deduplication, stream processing handles out-of-order arrival with watermarks, hot aggregates live in a column store while raw data stays replayable, and consumer lag is monitored as a service level indicator rather than a debugging afterthought.
Analytics tooling and billing tooling are not the same layer
This matters before any tool selection, because the categories get conflated and the mismatch is expensive to discover late.
Product analytics platforms ingest very large event volumes and answer behavioural questions. Amplitude handles behavioural analysis and feature adoption at scale and connects to warehouses and SDKs. Mixpanel covers retention, funnels, and experiment analysis with dashboards a non-analyst can use. Heap autocaptures every click and pageview from a snippet, which removes upfront tagging work and creates an event hygiene problem in exchange. Countly is open source and self-hostable, which suits teams with privacy and compliance constraints, and it tracks count, sum, and duration metrics.
All four are good at what they do. None of them is a billing system. An analytics platform is allowed to sample, approximate, or drop a fraction of events under load, because a retention curve that is 99.7% accurate is still correct. An invoice that is 99.7% accurate is a dispute.
Metering and billing infrastructure has different guarantees: no double counting, no silent loss, and a path from any invoice line back to the events that produced it.
Running both is normal. Using one for the other is where the trouble starts.
Building the pipeline
1. Define the goal before the schema
Analytics, billing, fraud detection, and telemetry have different tolerances. Write down target event volume, acceptable end-to-end latency, required accuracy, retention period, and which compliance rules apply. Every later decision follows from those four numbers.
2. Design the events
Build a taxonomy and treat it as a contract. Use a schema format with an evolution story, such as Avro or Protobuf, and version it explicitly so a field rename does not silently break an aggregation downstream.
Every event needs a unique id and a timestamp, and a decision about how sensitive fields are handled before the event leaves the application.
{
"event_id": "evt_01J8Z2K4RQ",
"event_name": "inference_request",
"external_customer_id": "acct_2213",
"timestamp": "2026-09-11T09:14:33Z",
"properties": {
"schema_version": 3,
"model": "llama-3-70b",
"gpu_seconds": 4.21,
"region": "us-east-1"
}
}
3. Ingest through a backbone that absorbs spikes
SDKs and endpoints on the application side, Kafka or an equivalent log in the middle. Deduplicate at the entry point and apply backpressure rather than dropping when a consumer falls behind.
The emission path should be non-blocking. Batched delivery with automatic retry means the product request completes whether or not the metering service is healthy. Flexprice ships SDKs for Python, JavaScript, and Go that batch and retry on this model, and collectors for Kafka, CDP, and OpenTelemetry sources.
4. Process in the stream
Stream processors clean, enrich, deduplicate, and aggregate as events arrive. Two details decide accuracy here.
Windowed deduplication. Keep a bounded set of seen event ids per window rather than an unbounded one. The window has to be longer than the longest realistic retry delay.
Watermarks for late data. Events arrive out of order, sometimes hours late from a mobile client or a retried batch. A watermark defines how long a window stays open before it is finalised, and anything later goes to a correction path rather than silently into the wrong period.
Invalid events belong in a dead letter queue, never discarded.
5. Store hot and cold separately
Raw events stay in an immutable log or object storage so any window can be replayed. A column store such as ClickHouse serves real-time aggregation with sub-second reads. A warehouse handles deeper historical analysis.
Partition by time and by tenant. Tenant partitioning is what stops one large customer's traffic from degrading queries for everyone else.
6. Build the billing path deliberately
Define the billable units and the aggregation rule for each. Enforce idempotency keys on everything that produces a charge. Add credit wallets if the model is prepaid. Close each billing period with an audit log, and expose real-time usage to customers rather than only invoices.
This is the layer where Flexprice fits: event ingestion maps raw events to meters, aggregations turn them into billable metrics, and invoice calculation assembles line items that reference the underlying usage.
7. Instrument the pipeline itself
Track ingestion rate, duplicate rate, consumer lag, schema validation failures, and the gap between aggregated usage and billed totals. Alert on lag before it becomes a reconciliation problem. Most teams monitor the application and not the meter, which is why billing failures are usually reported by customers.
8. Design for failure and scale
Partition by tenant or project, scale consumers horizontally, keep two to three times headroom over peak, and treat consumer lag as a service level indicator with a target. Multi-region topics for disaster recovery, and load tests that actually reproduce a spike rather than a steady ramp.
9. Secure and comply
TLS or mTLS in transit, encryption for sensitive fields, and access control on both the event store and the pricing configuration. Keep an audit trail for every write and every change to a pricing rule, because "who changed this price and when" is a question that eventually gets asked under pressure.
10. Control cost
Egress, storage, and compute are the drivers. Tiered storage moves cold partitions to object storage. Materialised views and rollups cut query cost on the hot path. Track cost per tenant and per metric, otherwise the unit economics of the metering system itself become invisible.
11. Roll out in stages
Contract tests on the schema, load tests on the ingest path, and a shadow pipeline running alongside the current one before anything cuts over. Keep runbooks for the two incidents that will happen: an outage in the ingest path, and a billing mismatch discovered after invoices went out.
Migrating without downtime
The safe sequence is dual-write shadow mode. Send events to both the current system and the new one, and compare aggregates per tenant per period rather than spot-checking totals. Parity checks are the gate, not the calendar.
Cut over one tenant at a time at a period boundary, never mid-cycle, and keep the old path running for one full cycle as a fallback. Retire the legacy system only after a complete billing period has closed correctly on the new one.
Where the billing layer sits
Analytics tools stop at visualisation. The gap they leave is the path from a raw event to a priced, invoiced, auditable line item, and that path is what has to carry the accuracy guarantees.
Flexprice is enterprise-grade, open source usage based billing infrastructure for AI and SaaS companies. It can be deployed in your own VPC, on-prem, or on Flexprice's managed cloud, and all three run the same engine. The event pipeline uses Kafka for ingestion and ClickHouse for real-time aggregation, with idempotent ingestion and deduplication so an event counted twice upstream is charged once.
Compared with Lago, which is also open source and self-hostable, the difference is enterprise scale: Flexprice is built for real-time metering at high event volume, with deployment across any VPC and any geography. Because Flexprice is open source and self-hostable, usage and revenue data can stay entirely inside your own infrastructure and never reach a vendor's cloud.
Getting started
The architecture overview covers how the ingestion, aggregation, and rating stages are separated, which is the design question worth settling before you write any of this yourself. The source is on GitHub.
Top comments (0)