InfluxDB 3 (IOx) Deep Dive: Rust + DataFusion + Parquet + Object Storage

influxdb 3 is the ground-up rewrite that finally made time-series storage cloud-native — a full Rust reimplementation of the InfluxDB engine, built on Apache Arrow, Apache DataFusion, Apache Parquet, and durable object storage (S3 / GCS / Azure Blob) — and it is the single component senior data-platform engineers get asked about most often because "just keep running InfluxDB 2.x" stopped scaling the moment a single tag column tipped past a few million distinct values. Every metric your infrastructure emits — a per-container CPU sample, a per-request latency histogram, an IoT device heartbeat, a Kubernetes pod event, a per-user telemetry counter — has to land in a database that can absorb millions of points per second, keep years of history without falling over, and answer sub-second SQL against that history without the "series-file cliff" that made the old TSM engine unusable past a few million series per shard. The engineering trade-off does not live in "should we use a time-series database" — every observability, IoT, and industrial telemetry stack needs one — but in which engine survives at the cardinality, retention, and cost profile your workload actually has.

This guide is the senior-DE walkthrough you wished existed the first time an interviewer asked "walk me through the InfluxDB 3 IOx architecture and why the four services are separated," or "how does IOx avoid the cardinality cliff that killed TSM?", or "explain how a SELECT hits Parquet in S3 through DataFusion." It walks through the four IOx pillars — a Rust rewrite of the storage engine, datafusion parquet as the query engine, Apache Arrow as the in-memory format, and object storage as the durable base — the four separated services (router, ingester, compactor, querier) plus the Postgres catalog that ties them together, the LineProtocol + flight sql influxdb ingest paths, the SQL + InfluxQL query dialects, and the dual-write migration bridge from 2.x TSM/Flux to 3 IOx/SQL. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the SQL practice library →, rehearse aggregation on the aggregation practice library →, and sharpen the window-function axis with the window-functions practice library →.

---

On this page

• Why InfluxDB 3 (IOx) matters — the ground-up rewrite
• IOx architecture — separation of concerns
• Data model + writing — LineProtocol at scale
• Querying — SQL, InfluxQL, Flight SQL via DataFusion
• Migration + when InfluxDB 3 wins + interview signals
• Cheat sheet — InfluxDB 3 (IOx) recipes
• Frequently asked questions
• Practice on PipeCode

---

1. Why InfluxDB 3 (IOx) matters — the ground-up rewrite

Four pillars, one clean-slate rewrite — the choice binds every time-series workload for years

The one-sentence invariant: InfluxDB 3 (codenamed IOx while under development) is the complete Rust reimplementation of the InfluxDB engine — abandoning the Go + TSM storage stack in favour of Apache Arrow in memory, Apache DataFusion as the query engine, Apache Parquet on durable object storage, and a separated ingester / compactor / querier service topology — and every one of those four choices was made to eliminate a specific pain point (cardinality cliffs, single-node scalability, retention costs, dialect lock-in) that made the old engine untenable at 2026-scale telemetry. The engine you pick in month one becomes the engine you fight to migrate away from in year three, because every dashboard, alert rule, and downstream ETL job hard-codes assumptions about the query dialect, the ingest path, and the durability semantics — moving is a multi-quarter project.

The four pillars interviewers actually probe.

• Rust rewrite. The Go + TSM codebase (InfluxDB 1.x / 2.x) hit fundamental limits around cardinality (the in-memory series file capped hard past a few million series), GC pressure (Go's stop-the-world pauses hurt millisecond-scale query latency), and single-node coupling (the engine was one process holding one node's data). Rust gives predictable memory, zero-GC latency, and lets InfluxData ship a service-per-role architecture. Interviewers open here because it separates people who read the release notes from those who read the RFCs.
• Apache DataFusion. DataFusion is the Rust SQL/DataFrame engine (Arrow-native, extensible, optimizer-rich) that InfluxData adopted as the query planner. Instead of hand-writing a query engine, IOx delegates parsing → logical plan → physical plan → execution to DataFusion, then plugs in its Parquet reader and its InfluxQL-to-SQL translator on top. This is why IOx got SQL, window functions, joins, and predicate pushdown "for free" — DataFusion already had them.
• Apache Parquet on object storage. Instead of TSM's proprietary shard/series files bolted to local disk, IOx writes columnar Parquet files to S3 / GCS / Azure Blob. Object storage is cheap ($0.023/GB/month on S3 vs $0.10+/GB/month for gp3 EBS), durable (11 nines), and infinitely scalable. The trade-off: object storage is high-latency (~30 ms per GET), so IOx caches aggressively and reads in batches.
• Unlimited cardinality + unlimited retention. These are the two "user-visible" promises that fall out of the pillars above. Cardinality is unlimited because tags become Parquet columns (not entries in an in-memory series file). Retention is unlimited because Parquet on object storage costs pennies per GB per year. Both were structural constraints in TSM; both are gone in IOx.

The 2026 reality — three editions, one shared engine.

• InfluxDB 3 Core. Apache 2.0 OSS, single-node, no clustering. Ships as a single binary. Reasonable for dev, small production, and anyone who wants to poke at the engine. Free forever. Uses local disk or S3 as the storage tier; SQLite or Postgres as the catalog.
• InfluxDB 3 Enterprise. Self-hosted, licensed, clustered. Adds high-availability, RBAC, backup/restore tooling, and the separated ingester / compactor / querier deployment model that Cloud runs. Aimed at teams that must run on-prem or in their own VPC.
• InfluxDB 3 Cloud Serverless. Fully managed, multi-tenant, InfluxData-hosted. The pricing is usage-based (writes + storage + queries), which reflects the object-storage economics. The default answer when "we just want it to work" is the requirement.
• What breaks. 1.x / 2.x TSM engine data files are not readable by IOx — no in-place upgrade. Flux (the InfluxDB 2.x scripting language) is deprecated in favour of SQL; some Flux features have no direct SQL analogue, so a migration is a rewrite, not a translation.

What interviewers listen for.

• Do you name all four pillars (Rust, DataFusion, Parquet, object storage) without prompting? — senior signal.
• Do you say "cardinality is unlimited because tags are Parquet columns, not series-file entries" in the first sentence when cardinality comes up? — required answer.
• Do you push back on "just upgrade from 2.x" with the migration reality (no in-place upgrade, Flux rewrite required)? — senior signal.
• Do you name DataFusion as the SQL engine and describe predicate + partition pushdown as its key optimisation? — senior signal.
• Do you describe IOx as "a set of separated services sharing object storage and a catalog" rather than as "a single time-series database"? — required answer.

Worked example — the four-pillar comparison against InfluxDB 2.x

Detailed explanation. The single most useful artifact for an InfluxDB 3 interview is a memorised 4×2 comparison against the old 2.x engine. Every senior IOx discussion converges on this table within the first ten minutes; having it in your head is what separates a fluent answer from a stumbling one. Walk through building the table for a hypothetical Kubernetes observability workload that emits per-pod CPU + memory metrics.

• Workload. 50,000 pods × 6 metrics/pod × 15 s scrape interval = 20,000 points/sec, ~1.7 B points/day.
• Cardinality. Per-pod tags on pod_name, namespace, node, container — pods churn ~10× per day, so cardinality grows continuously; a 30-day window sees ~15 M distinct series.
• Retention. 90 days at full resolution, 2 years downsampled.
• Query pattern. Grafana dashboards asking rate(cpu[5m]) BY namespace on top of the raw stream, plus ad-hoc SQL from platform engineers.

Question. Build the four-pillar comparison between InfluxDB 2.x TSM and InfluxDB 3 IOx for the K8s observability workload, and identify which pillars matter most.

Input.

Pillar	InfluxDB 2.x (TSM)	InfluxDB 3 (IOx)
Language	Go	Rust
Storage engine	TSM (proprietary shard/series files on local disk)	Parquet columnar on object storage
Query engine	Custom Flux + InfluxQL runtime	Apache DataFusion (SQL-first)
Cardinality ceiling	~1-4 M series per shard (in-memory series file)	unlimited (tags as Parquet columns)
Retention ceiling	Bounded by local disk	Bounded by object-storage budget
Scale-out	Enterprise clustering (Anti-entropy)	Stateless services + shared object store

Code.

# InfluxDB 3 Core Docker skeleton — the entire stack in one binary
# Save as docker-compose.yml
services:
  influxdb3:
    image: influxdb:3-core
    ports:
      - "8181:8181"
    environment:
      INFLUXDB3_OBJECT_STORE: file
      INFLUXDB3_DB_DIR: /var/lib/influxdb3
      INFLUXDB3_NODE_IDENTIFIER_PREFIX: iox-core
    volumes:
      - influxdb3-data:/var/lib/influxdb3

  # In production, swap `file` for `s3` / `google` / `azure` and add a
  # Postgres catalog service; Core defaults to SQLite in the data dir.

volumes:
  influxdb3-data:

Step-by-step explanation.

1. The pillar that dominates the K8s observability workload is unlimited cardinality. At 15 M series over 30 days, InfluxDB 2.x TSM would either need aggressive sharding or would hit max-series-per-database and start dropping writes. IOx doesn't have that ceiling — tags become Parquet columns, and Parquet's dictionary encoding compresses repeated tag values efficiently.
2. The second pillar is object storage retention. 90 days of raw + 2 years of downsampled at 20 K pts/sec is ~150 GB raw + ~60 GB downsampled after compression. On EBS gp3 at $0.08/GB/month that's ~$17/month for raw; on S3 Standard at $0.023/GB/month it's ~$3.50/month. Order-of-magnitude cheaper retention with better durability.
3. The DataFusion pillar matters because platform engineers want SQL. TSM's InfluxQL and Flux are query DSLs that don't compose with the rest of the analytics stack (dbt, Superset, Metabase). SQL over Arrow via DataFusion means the same tools work against IOx and against your warehouse.
4. The Rust pillar shows up as tail-latency. TSM's Go GC pauses used to spike P99 query latency by 50-500 ms during compactions; IOx's Rust runtime has none. For a Grafana dashboard refreshing every 10 s across 30 panels, this is the difference between "snappy" and "laggy" perception.
5. The single-binary Core edition (shown in the compose file) is enough for a proof-of-concept. Production K8s observability at 20 K pts/sec would want the Enterprise or Cloud edition where router/ingester/compactor/querier scale independently.

Output.

Concern	Recommended engine	Why
K8s observability (15 M series, 90 d retention)	InfluxDB 3 IOx	cardinality + retention limits gone
Small edge / IoT (10 K series, 7 d retention)	InfluxDB 3 Core	single binary; free
Legacy Grafana dashboards on InfluxQL	InfluxDB 3 (dual dialect)	InfluxQL still supported
Ad-hoc SQL from analysts	InfluxDB 3 (DataFusion SQL)	first-class dialect
Multi-tenant hosted	InfluxDB 3 Cloud Serverless	usage-priced; no ops

Rule of thumb. Never pick a time-series engine based on "we already run InfluxDB." Pick it based on (cardinality × retention × query dialect × ops budget) — the four pillars. If any of those axes is above the TSM comfort zone, IOx is almost always the answer.

Worked example — what interviewers actually probe on the Rust rewrite

Detailed explanation. The senior data-engineering IOx interview has a predictable structure: the interviewer opens with an ambiguous question ("why did InfluxData rewrite the engine?"), then progressively narrows to test whether you understand the four pillars and their motivations. The candidates who name the pillars in sentence one score highest; the candidates who describe "it's faster now" score lowest. Walk through the interview grading rubric.

• Ambiguous opener. "Walk me through InfluxDB 3 — what's actually different from 2.x?" — invites you to name the four pillars.
• Follow-up 1. "Why Rust specifically, not just Go 1.21+ GC improvements?" — probes the tail-latency + memory-safety argument.
• Follow-up 2. "Why DataFusion instead of writing a new engine?" — probes the "SQL-for-free" argument.
• Follow-up 3. "What happens if S3 has a 10-second latency spike?" — probes the caching + ingester WAL story.
• Follow-up 4. "How does this compare to TimescaleDB / QuestDB / ClickHouse?" — probes the "when does IOx win" reasoning.

Question. Draft a 5-minute senior InfluxDB 3 answer that covers all four pillars without waiting to be asked.

Input.

Interview signal	Weak answer	Senior answer
Pillar named	"they rewrote it in Rust"	"Rust + DataFusion + Parquet on object storage — four pillars"
Rust motivation	"faster"	"deterministic memory, zero GC pauses, service-per-role decomposition"
DataFusion role	"the new query engine"	"Arrow-native SQL planner; IOx plugs its Parquet reader into DataFusion"
Storage cost	"S3 is cheap"	"$0.023/GB/mo vs $0.08+ for EBS; unlimited retention becomes tractable"
Cardinality argument	"no more limits"	"tags become Parquet columns; dictionary-encoded; scans skip via partition pruning"

Code.

Senior InfluxDB 3 answer template (5 minutes)
=============================================

Minute 1 — name the four pillars up front
  "InfluxDB 3 is a ground-up Rust rewrite that keeps the InfluxDB
   data model — measurements, tags, fields, timestamps — but replaces
   the storage layer with Parquet on object storage, adopts Apache
   DataFusion as the SQL engine, and decomposes the monolith into
   separate router / ingester / compactor / querier services."

Minute 2 — why Rust
  "Rust gives InfluxData deterministic memory management with zero
   GC pauses, which matters for query tail-latency; the borrow
   checker also lets them ship the service-per-role architecture
   without the concurrency footguns Go would introduce."

Minute 3 — why DataFusion + Parquet
  "DataFusion is the Arrow-native SQL planner they adopted rather
   than writing a new engine. IOx contributes the Parquet reader
   and the InfluxQL-to-SQL translator; DataFusion handles parsing,
   logical plans, physical plans, predicate pushdown, partition
   pruning, projection pushdown, and vectorised execution."

Minute 4 — why object storage
  "Parquet files land in S3 / GCS / Azure Blob. Storage cost is
   roughly a quarter of EBS, durability is 11 nines, and there's
   no ceiling on retention — you pay for what you keep. The
   trade-off is per-GET latency, which the querier absorbs with
   aggressive caching and the ingester absorbs with a WAL."

Minute 5 — cardinality + when IOx wins
  "Tags are Parquet columns, not entries in an in-memory series
   file, so cardinality is bounded only by object storage. IOx
   wins over TimescaleDB when cardinality is high and joins are
   thin; over QuestDB when horizontal scale matters; over
   ClickHouse when the workload is time-series-native rather than
   general analytics."

Step-by-step explanation.

1. Minute 1 is the crucial framing. Naming all four pillars — "Rust + DataFusion + Parquet + object storage" — signals you understand the architecture, not just the marketing. Weak candidates dive into features ("SQL support, unlimited cardinality…") without naming the substrate that makes those features possible.
2. Minute 2 addresses the Rust question with the two-part answer that interviewers listen for: tail latency (no GC pauses) and concurrency (borrow checker enables the separated services). "Faster" alone is not senior signal; naming the two structural motivations is.
3. Minute 3 handles DataFusion. The key phrase is "adopted rather than writing a new engine" — this signals you understand the engineering economics. Then naming the four optimisations (predicate pushdown, partition pruning, projection pushdown, vectorisation) demonstrates you've read a query plan, not just the README.
4. Minute 4 covers object storage. The cost differential (~4× cheaper than EBS) plus the durability argument (11 nines) plus the retention consequence (bounded only by budget) is the full argument. The latency trade-off must be named unprompted — that's the honest engineering.
5. Minute 5 covers cardinality and the competitive landscape. Naming when IOx wins against each competitor is more senior than blanket praise; every senior architect knows every DB has a niche.

Output.

Grading criterion	Weak score	Senior score
Names all four pillars in minute 1	rare	mandatory
Names GC + concurrency for Rust	rare	required
Names DataFusion + optimisations	occasional	senior signal
Names object-storage cost + latency trade-off	rare	mandatory
Names cardinality reasoning + competitor niches	rare	senior signal

Rule of thumb. The senior InfluxDB 3 answer is a 5-minute monologue that covers all four pillars, their motivations, and the competitive landscape without waiting for follow-ups. Rehearse it once; deploy it every time.

Worked example — the "pick the edition" decision tree

Detailed explanation. Given a new time-series workload, the senior architect runs a 4-question decision tree in their head to pick between Core, Enterprise, Cloud Serverless, and "not InfluxDB." Codifying the tree makes the interview answer reproducible: any interviewer can hand you a scenario and you can walk it out loud. Walk through the tree with three canonical scenarios: an IoT PoC, a K8s observability platform, and a hosted SaaS metrics service.

• Q1. Is the workload single-node dev / PoC / small edge? → yes = Core; no = go to Q2.
• Q2. Must it run on-prem or in your own VPC for compliance? → yes = Enterprise; no = go to Q3.
• Q3. Do you want to manage the storage tier + catalog yourself? → yes = Enterprise; no = Cloud Serverless.
• Q4 (parallel branch). Is cardinality bounded (< 100 K series) and is the query pattern relational-join-heavy? → yes = consider TimescaleDB instead.

Question. Walk the decision tree for the three scenarios and record the edition each ends up with.

Input.

Scenario	Q1 (single-node?)	Q2 (on-prem?)	Q3 (self-manage?)	Q4 (low card + joins?)
IoT PoC	yes	—	—	no
K8s observability platform	no	no	no	no
Hosted SaaS metrics service	no	yes (customer VPC)	yes	no

Code.

# Decision-tree helper (illustrative)
def pick_influxdb_edition(single_node: bool,
                           on_prem: bool,
                           self_manage: bool,
                           low_card_joins: bool) -> str:
    """Return the recommended InfluxDB 3 edition (or an alternative)."""
    if low_card_joins:
        return "consider TimescaleDB (relational + hypertables)"

    if single_node:
        return "influxdb 3 core"

    if on_prem:
        return "influxdb 3 enterprise"

    if self_manage:
        return "influxdb 3 enterprise"

    return "influxdb 3 cloud serverless"


# Walk the three scenarios
print(pick_influxdb_edition(True,  False, False, False))
# → 'influxdb 3 core'

print(pick_influxdb_edition(False, False, False, False))
# → 'influxdb 3 cloud serverless'

print(pick_influxdb_edition(False, True,  True,  False))
# → 'influxdb 3 enterprise'

Step-by-step explanation.

1. Scenario 1 — IoT PoC with a few thousand devices. Single binary, local disk, minimal ops. Q1 short-circuits to Core. This is the "download the Docker image and go" answer.
2. Scenario 2 — K8s observability platform serving multiple internal teams. Not single-node (needs scale-out and HA), not on-prem-locked (runs in cloud already), doesn't want to self-manage the storage tier. Q1 = no, Q2 = no, Q3 = no → Cloud Serverless. The team pays for writes + storage + queries and never touches a compactor.
3. Scenario 3 — a hosted SaaS metrics service that must run inside each customer's VPC for compliance. Q1 = no, Q2 = yes → Enterprise. The team runs router / ingester / compactor / querier themselves plus the Postgres catalog, and provides the object-storage bucket the customer already trusts.
4. The parallel Q4 branch pulls out workloads that don't belong in InfluxDB at all. If cardinality is bounded and the query pattern is relational-join-heavy (e.g. correlating a metric stream with a customer master table on user_id), TimescaleDB's hypertables on Postgres are usually a better fit — you get PostgreSQL joins for free.
5. The tree deliberately avoids "which is cheapest" as a first-order question. Cloud Serverless is often cheapest by TCO because you're not paying an SRE. Enterprise is often cheapest by list price because it's a flat licence. Do the TCO math after picking the technically appropriate edition, not before.

Output.

Scenario	Recommended edition	Notes
IoT PoC (few thousand devices)	InfluxDB 3 Core	single binary; free forever
K8s observability platform	InfluxDB 3 Cloud Serverless	usage-priced; no ops
SaaS in customer VPC	InfluxDB 3 Enterprise	run all four services yourself
Relational-join-heavy telemetry	TimescaleDB	Postgres hypertables win here

Rule of thumb. The 4-question decision tree is a whiteboard-friendly answer. Practice walking it end-to-end so an interviewer can hand you any scenario and get an edition name in under 60 seconds — including the "don't use InfluxDB" branch.

Senior interview question on InfluxDB 3 pillars

A senior interviewer often opens with: "You inherit a Kubernetes observability platform running InfluxDB 2.x that has hit max-series-per-database twice this quarter and now drops writes for the largest tenants. Walk me through migrating to InfluxDB 3, why the cardinality problem goes away, and the four pillars that make the new engine structurally different."

Solution Using InfluxDB 3 IOx with S3 storage, DataFusion SQL, and staged tenant cutover

# 1. InfluxDB 3 Enterprise deployment for a K8s observability platform
# helm values.yaml (abbreviated)
influxdb3:
  edition: enterprise
  license: ${env:INFLUXDB3_LICENSE}

  # Object storage — the durable base
  objectStore:
    type: s3
    bucket: acme-observability-lake
    region: us-east-1
    endpoint: https://s3.us-east-1.amazonaws.com

  # Catalog — Postgres in prod
  catalog:
    type: postgres
    dsn: postgres://iox_catalog:${env:CATALOG_PW}@catalog.internal:5432/iox

  # Separated services — scale each pool independently
  services:
    router:
      replicas: 3
      resources: { cpu: 2, memory: 4Gi }
    ingester:
      replicas: 6
      resources: { cpu: 4, memory: 16Gi }
      walDisk: 200Gi
    compactor:
      replicas: 2
      resources: { cpu: 8, memory: 32Gi }
    querier:
      replicas: 4
      resources: { cpu: 4, memory: 32Gi }
      objectStoreCache: 200Gi

-- 2. Create the database + retention on IOx
-- (via influxctl CLI or the HTTP API)
CREATE DATABASE observability
  WITH RETENTION_PERIOD = '90d';

-- No max-series-per-database! Cardinality is bounded only by object storage.

# 3. Staged tenant cutover — dual-write during the transition
#    Every write goes to both 2.x (existing dashboards) and 3 (validation)
#    until the query traffic can be cut over

import requests

INFLUX2_URL = "http://influx2.internal:8086/api/v2/write"
INFLUX3_URL = "http://influx3.internal:8181/api/v3/write_lp?db=observability"

def dual_write(line_protocol_batch: str) -> None:
    """Send LineProtocol to both 2.x (legacy) and 3 (target) during migration."""
    # Legacy 2.x
    r2 = requests.post(
        INFLUX2_URL,
        params={"org": "acme", "bucket": "observability", "precision": "ns"},
        headers={"Authorization": f"Token {LEGACY_TOKEN}"},
        data=line_protocol_batch,
        timeout=10,
    )
    r2.raise_for_status()

    # Target 3 (do not fail the pipeline on 3 errors during migration)
    try:
        r3 = requests.post(
            INFLUX3_URL,
            headers={"Authorization": f"Bearer {NEW_TOKEN}"},
            data=line_protocol_batch,
            timeout=10,
        )
        r3.raise_for_status()
    except requests.HTTPError as e:
        log_migration_error(e, line_protocol_batch)

-- 4. Sample IOx SQL — the query the old InfluxQL used to run
SELECT
    date_bin(INTERVAL '5 minutes', time) AS bucket,
    namespace,
    AVG(usage_pct)                        AS avg_cpu,
    MAX(usage_pct)                        AS peak_cpu
FROM   cpu
WHERE  time >= now() - INTERVAL '1 hour'
GROUP  BY 1, 2
ORDER  BY 1;

Step-by-step trace.

Step	Before (InfluxDB 2.x TSM)	After (InfluxDB 3 IOx)
Cardinality ceiling	~4 M series before drops	unlimited (Parquet columns)
Storage tier	local NVMe (paid per GB provisioned)	S3 (paid per GB stored)
Storage cost per TB / month	~$80 (gp3 EBS)	~$23 (S3 Standard)
Retention (90 d raw + 2 y downsampled)	requires aggressive downsampling	trivially fits
Query dialect for analysts	InfluxQL + Flux	SQL (via DataFusion) + InfluxQL
Service topology	1 monolithic process per node	router + ingester + compactor + querier
GC tail-latency (P99 query)	50-500 ms spikes	~none (Rust)
Migration model	in-place upgrade	dual-write bridge + cutover

After the migration, the platform stops dropping writes at high cardinality, storage bill drops ~4×, Grafana keeps working via the InfluxQL dialect while the platform team writes new alert rules in SQL, and Rust-side tail latency eliminates the "dashboard occasionally hangs during compaction" complaints that plagued 2.x.

Output:

Metric	Before (2.x)	After (3 IOx)
Max sustained cardinality	~4 M series	unlimited
Storage cost per TB / month	~$80	~$23
Retention at raw resolution	30 d (cost-bound)	90 d (comfortable)
P99 query latency	300-800 ms	80-150 ms
Write-drop incidents / quarter	2-4	0
Services to scale	1 monolith	4 (independent)

Why this works — concept by concept:

• Rust rewrite — the substrate under everything else. Deterministic memory, zero GC pauses, and the borrow checker's concurrency guarantees are what let InfluxData decompose the monolith into stateless services without racing themselves. This is the pillar the others depend on.
• DataFusion — the Arrow-native SQL planner IOx delegates parsing / logical planning / physical planning / execution to. IOx contributes the Parquet scan operator and the InfluxQL translator; DataFusion contributes optimiser rules, vectorised execution, and dialect compatibility. This is how IOx got SQL "for free."
• Parquet on object storage — columnar storage with dictionary encoding compresses repeated tag values dramatically; object storage puts durability, cost, and infinite retention on the same tier. The trade-off is per-GET latency, which the querier absorbs with caching.
• Separated services — router / ingester / compactor / querier each scale independently. Ingester needs disk (WAL) and RAM; querier needs CPU and RAM; compactor needs CPU and network bandwidth to object storage. Coupling them wastes hardware.
• Cost — one Postgres catalog, one object-storage bucket, four service pools sized independently, and no ceiling on cardinality or retention. Compared to 2.x's monolithic shard-per-node model, this is O(1) per write versus O(cardinality) per write. The eliminated cost is "we hit max-series again" incidents and the compression pattern (~4× cheaper storage) more than pays for the Postgres catalog.

SQL
Topic — sql
SQL problems on time-series and telemetry queries

Practice →

Aggregation Topic — aggregation Aggregation problems on time-bucketed metrics

Practice →

---

2. IOx architecture — separation of concerns

Router, ingester, compactor, querier + a Postgres catalog — four stateless services sharing one object store

The mental model in one line: IOx (InfluxDB's IOx = "IOx on Iceberg-ish object storage" architecture) decomposes the old monolithic InfluxDB into four stateless services — a router that accepts writes and shards them by partition, an ingester that owns the WAL + in-memory buffer and periodically flushes Parquet files to object storage, a compactor that merges small Parquet files into larger ones and enforces retention, and a querier that reads Parquet from object storage via DataFusion and executes SQL — all coordinated through a shared Postgres (or SQLite in Core) catalog that stores partition metadata, schema, and file locations. Every senior architecture review of IOx focuses on this decomposition because it is what makes the engine horizontally scalable on cloud infrastructure.

The four services in one paragraph each.

• Router. The write endpoint. Accepts LineProtocol via HTTP POST /api/v3/write_lp (and Flight SQL binary), parses it, groups points by partition key (measurement + tag hash + time bucket), and forwards each partition's batch to the owning ingester. Stateless — you scale router replicas horizontally behind a load balancer. Failure of a router loses only in-flight requests; the client retries.
• Ingester. The durability boundary. Owns a set of partitions; receives batches from routers; writes them first to a per-partition WAL on local disk (fsync'd before ack), then to an in-memory Arrow buffer. Every N seconds (or when the buffer exceeds a threshold), the ingester flushes the buffer as a Parquet file to object storage, records the file in the catalog, and truncates the WAL. Ingester crash before flush = replay from WAL on restart; crash after flush = catalog has the file, buffer was empty, no loss.
• Compactor. The maintenance service. Continuously scans the catalog for partitions with many small Parquet files (a natural outcome of frequent ingester flushes), merges them into fewer larger files, deduplicates rows by primary key + timestamp, and drops files whose retention window has expired. Compaction is CPU-heavy but never touches the write path; ingesters keep flushing while the compactor works.
• Querier. The read side. Receives a SQL/InfluxQL query (via HTTP, gRPC, or Flight SQL), consults the catalog for which Parquet files satisfy the query's time range + partition predicates, downloads those files from object storage (with an on-disk cache to avoid re-fetching), plans the query via DataFusion, executes it against Arrow batches read from Parquet, and streams the results back. Stateless — you scale querier replicas per query concurrency.

The catalog — the shared metadata that ties them together.

• What it holds. Databases, tables, schemas, retention periods, partition metadata (which Parquet files belong to which partition), file locations in object storage, WAL positions, compaction bookmarks.
• What it doesn't hold. Actual time-series data. The catalog is metadata-only; the data lives in Parquet files in object storage.
• Backend. Postgres in Enterprise / Cloud; SQLite (embedded) in Core. The catalog is the single stateful component; all four services are stateless around it.
• Consistency model. ACID transactions in Postgres serialise catalog mutations (new Parquet file added, old file dropped by compactor). Services read a catalog snapshot per operation, so a slow compactor doesn't stall the querier — it just sees a slightly older file set until the transaction commits.

Why separation of concerns wins on cloud infrastructure.

• Independent scaling. Ingester replicas grow with write throughput. Querier replicas grow with query concurrency. Compactor grows with compaction backlog. Router grows with connection count. Each dimension has a different unit economics; coupling them wastes hardware.
• Independent failure blast radius. A crashing querier doesn't affect writes. A hung compactor doesn't affect reads. An OOM'd ingester loses only in-flight WAL, and only for its partitions. In the 2.x monolith, any of these took the whole node down.
• Independent upgrade cadence. Roll a new querier binary without touching ingesters. Bump the compactor CPU limit without a maintenance window. Each service is a k8s Deployment or ECS Service; standard cloud-native operational patterns apply.
• Rented storage economics. The durable tier (object storage) is a pay-per-GB commodity. All ingester WAL and querier cache is ephemeral local disk — sized for velocity, not durability. This is the same shape as Snowflake, BigQuery, and every modern OLAP; IOx brings it to time series.

Common interview probes on the architecture.

• "Why separate the services?" — required answer: independent scaling, blast radius, upgrade cadence, and cloud storage economics.
• "What does the compactor guard against?" — Parquet file explosion, retention overrun, and query-side small-file overhead.
• "What if S3 is down?" — writes continue (WAL absorbs); reads fail if not cached; ingester flushes back up when S3 returns.
• "Where does the query engine live?" — the querier — DataFusion runs there, not in the catalog and not in the ingester.
• "How does the catalog stay consistent?" — Postgres ACID transactions; every state-changing operation is transactional.

Worked example — the write path end-to-end

Detailed explanation. The canonical write path in IOx: a Telegraf agent POSTs a batch of LineProtocol lines to a router, the router shards by partition, forwards each batch to the owning ingester, the ingester WAL-appends + buffers + acknowledges, and periodically flushes Parquet to S3 and updates the catalog. Walk through every hop with the corresponding latency budget.

• Client. Telegraf batches ~5,000 points per HTTP request; retries on 5xx with exponential backoff.
• Router. Parses LineProtocol, groups by partition key, forwards to ingesters.
• Ingester. WAL append (fsync) + in-memory Arrow buffer; ACKs once WAL is durable.
• Flush. Every 60 s or when buffer > 512 MB, write Parquet to S3, register in catalog, truncate WAL.

Question. Trace one LineProtocol batch of 5,000 points from the client through the router and ingester to the eventual Parquet flush, and identify the durability moment.

Input.

Component	Latency budget	State written
Client → Router HTTP	~5-20 ms	none
Router parse + shard	~1-5 ms	in-memory
Router → Ingester gRPC	~1-5 ms	in-memory
Ingester WAL fsync	~2-10 ms	durable on local disk
Ingester ACK → Router → Client	reverse hops	ack sent
Ingester → S3 Parquet flush	~500-2000 ms	durable on object store
Catalog Postgres UPDATE	~5-20 ms	durable in catalog
WAL truncate	~1 ms	free disk

Code.

# Client side — Telegraf-style batched LineProtocol POST
import requests
import time

INFLUX3_URL = "http://influx3-router.internal:8181/api/v3/write_lp"

def write_batch(db: str, lines: list[str]) -> None:
    """POST a batch of LineProtocol lines to the router."""
    payload = "\n".join(lines).encode("utf-8")

    for attempt in range(5):
        try:
            r = requests.post(
                INFLUX3_URL,
                params={"db": db, "precision": "ns"},
                headers={
                    "Authorization": f"Bearer {TOKEN}",
                    "Content-Type": "text/plain; charset=utf-8",
                },
                data=payload,
                timeout=15,
            )
            if r.status_code == 204:      # 204 No Content = accepted
                return
            if 500 <= r.status_code < 600:
                time.sleep(2 ** attempt)  # exponential backoff on 5xx
                continue
            r.raise_for_status()
        except requests.RequestException:
            time.sleep(2 ** attempt)

    raise RuntimeError("write failed after 5 attempts")


# Sample batch — Kubernetes pod CPU metrics
lines = [
    f"cpu,pod=web-{i:04d},namespace=prod usage_pct={40 + i % 20}.{i % 10},"
    f"load1={0.5 + (i % 100) / 100.0} {int(time.time_ns())}"
    for i in range(5000)
]
write_batch("observability", lines)

# Ingester config snippet — WAL + flush thresholds
ingester:
  wal:
    dir: /var/lib/iox/wal
    fsync_every_write: true            # durability > throughput
    max_size_per_partition: 1GB

  buffer:
    max_bytes_per_partition: 512MB      # flush trigger (size)
    max_age_seconds: 60                 # flush trigger (age)
    max_partitions_in_memory: 4096

  flush:
    concurrent_uploads: 8               # parallel Parquet writes to S3
    parquet_row_group_size: 500000
    parquet_compression: zstd

Step-by-step explanation.

1. The client sends 5,000 LineProtocol lines as one HTTP POST. Batching is essential — one HTTP round-trip per point would be crushing at 20 K pts/sec. The router accepts the batch and 204s once it has forwarded to the ingester and received an ACK.
2. The router parses each line, extracts the measurement + tag values, computes the partition key (typically hash(measurement, tags) mod N bucketed by time), and forwards each partition's slice of the batch to the owning ingester. Router does no durable state; it's a pure fan-out.
3. The ingester receives the batch, WAL-appends every point with an fsync (~2-10 ms depending on disk), then applies to the in-memory Arrow buffer. Only after the WAL fsync completes does the ingester ACK — this is the durability boundary. Router relays the ACK back to the client.
4. In the background, every 60 s (or when a partition's buffer exceeds 512 MB), the ingester serialises the in-memory Arrow batches to a Parquet file (row group 500 K, zstd compression), uploads to S3, then wraps a Postgres transaction: INSERT the new file into catalog.parquet_files, mark the WAL segment as flushable. Only after the transaction commits does the ingester truncate the WAL.
5. The durability chain is: client → WAL (durable on local disk, survives ingester crash) → Parquet on S3 (durable on object storage, survives cluster loss) → catalog (durable in Postgres, survives S3 metadata loss). The client sees only one ACK — after the WAL fsync — and IOx guarantees the point eventually reaches Parquet.

Output.

Stage	Cumulative wall-clock	Durability state
Client sends POST	0 ms	none
Router receives	~10 ms	in-memory (router)
Ingester receives	~15 ms	in-memory (ingester)
WAL fsync completes	~20 ms	durable on local disk
Client receives 204	~30 ms	client sees "written"
Ingester flushes to S3 (background)	+500-2000 ms	durable on object store
Catalog registers file	+5-20 ms	durable in catalog
WAL truncated	+1 ms	disk freed

Rule of thumb. The client-observed write latency is the WAL fsync time (~20-30 ms end-to-end). The Parquet flush latency is background — never on the write path. If an ingester crashes before flushing, WAL replay on restart re-creates the buffer and eventually flushes; no writes are lost after the ACK.

Worked example — the read path end-to-end

Detailed explanation. The canonical read path in IOx: a Grafana dashboard sends a SQL query over Flight SQL to the querier, the querier consults the catalog for matching Parquet files, downloads them (with local cache), DataFusion plans and executes, and Arrow batches stream back. Trace every hop.

• Client. Grafana sends SELECT date_bin('5m', time), namespace, AVG(usage_pct) FROM cpu WHERE time > now() - '1h' GROUP BY 1, 2 via Flight SQL.
• Querier. Consults catalog for Parquet files in the last hour; downloads (or cache-hits) each file.
• DataFusion. Predicate pushdown skips row groups outside the time range; projection pushdown reads only the time, namespace, usage_pct columns.
• Result. Arrow batches stream back to Grafana; Grafana renders.

Question. Trace one Grafana query from the dashboard through the querier and DataFusion back to the client, identifying every optimisation that fires.

Input.

Optimisation	Where it fires	What it does
Partition pruning	catalog lookup	skip Parquet files outside time predicate
Predicate pushdown	Parquet reader	skip row groups whose min/max miss the predicate
Projection pushdown	Parquet reader	read only referenced columns
Vectorised execution	DataFusion physical plan	process 8K-row Arrow batches per operator
Object-store cache	Querier local disk	skip re-download for hot files

Code.

# Client side — Flight SQL query using adbc_driver_flightsql
import adbc_driver_flightsql.dbapi as flight_sql

with flight_sql.connect(
    uri="grpc+tls://influx3-querier.internal:8443",
    db_kwargs={"adbc.flight.sql.authorization_header": f"Bearer {TOKEN}"},
) as conn:
    with conn.cursor() as cur:
        cur.execute("""
            SELECT date_bin(INTERVAL '5 minutes', time) AS bucket,
                   namespace,
                   AVG(usage_pct)                        AS avg_cpu,
                   MAX(usage_pct)                        AS peak_cpu
            FROM   cpu
            WHERE  time >= now() - INTERVAL '1 hour'
            GROUP  BY 1, 2
            ORDER  BY 1
        """)
        rows = cur.fetchall()
        for r in rows:
            print(r)

-- Verify DataFusion picked up predicate + projection pushdown
EXPLAIN
SELECT date_bin(INTERVAL '5 minutes', time) AS bucket,
       namespace,
       AVG(usage_pct) AS avg_cpu
FROM   cpu
WHERE  time >= now() - INTERVAL '1 hour'
GROUP  BY 1, 2;

-- Sample output (abbreviated):
-- SortExec: expr=[bucket ASC]
--   AggregateExec: mode=Final, gby=[bucket, namespace]
--     AggregateExec: mode=Partial, gby=[bucket, namespace]
--       ProjectionExec: expr=[date_bin(...) AS bucket, namespace, usage_pct]
--         RepartitionExec
--           ParquetExec:
--             file_groups={12 files},
--             projection=[time, namespace, usage_pct],   -- projection pushdown
--             predicate=time >= <lit>,                    -- predicate pushdown
--             pruning_predicate=time_max >= <lit>         -- row-group pruning

Step-by-step explanation.

1. Grafana sends the SQL query as a Flight SQL command over gRPC. Flight SQL carries the query as a string and returns results as Arrow record batches — no CSV / JSON serialisation tax. The querier authenticates the bearer token via the catalog.
2. The querier parses the SQL, extracts the time >= now() - '1h' predicate, and asks the catalog for all Parquet files whose partition time range overlaps the last hour. Partition pruning happens at catalog level — files outside the window are never downloaded.
3. For each surviving Parquet file, the querier checks its local disk cache. Cache hits (frequently-queried recent files) skip the S3 GET entirely. Cache misses trigger an async S3 download; the querier can start executing against files that are ready while others download.
4. DataFusion plans the query. The physical plan (visible via EXPLAIN) shows ParquetExec with projection=[time, namespace, usage_pct] (projection pushdown — only three columns read), predicate=time >= ... (pushed into the Parquet reader), and pruning_predicate (using per-row-group min/max stats to skip whole row groups whose time range misses the predicate). Aggregation is two-phase (partial per-node → final merge).
5. Execution processes Arrow batches (typically 8,192 rows each) through the operator pipeline. date_bin is a scalar UDF; AVG and MAX are vectorised aggregate kernels. Result batches stream back to Grafana as Arrow IPC over Flight; Grafana's Arrow-aware datasource renders directly.

Output.

Metric	Value
Parquet files considered (before pruning)	~1000 (24 h of history × 40 partitions)
Parquet files opened (after pruning)	~40 (1 h × 40 partitions)
Row groups scanned (after row-group pruning)	~120 (~3 per file)
Columns read (after projection pushdown)	3 of ~50
Bytes read from S3 (with cache)	~50 MB
Wall-clock end-to-end	~150 ms

Rule of thumb. IOx query latency is dominated by (a) object-store fetch time (mitigated by cache) and (b) DataFusion CPU time (mitigated by pushdowns and vectorisation). Always inspect EXPLAIN after the first slow query; the four pushdowns are the difference between 150 ms and 15 s.

Worked example — what happens when S3 is degraded

Detailed explanation. Object storage is durable but not always fast. Regional issues, throttling, or a client-side network hiccup can push S3 latency from 30 ms per GET to 5-30 s. Trace how each IOx service degrades — and what stays green — during a 10-minute S3 latency spike.

• Writes. Client → Router → Ingester → WAL are unaffected (WAL is local disk). Flushes back up in the ingester; WAL grows; ACKs still fast.
• Reads. Querier cache hits stay fast. Cache misses stall on S3 GETs; per-query latency spikes.
• Compactor. Backs off; retries with exponential backoff; catches up when S3 recovers.
• Catalog. Unaffected (Postgres, not S3).

Question. Trace each IOx service's behaviour during a 10-minute S3 latency spike and identify what the on-call runbook says.

Input.

Service	S3-independent path	S3-dependent path
Router	parse + shard + forward	none
Ingester	WAL append (fsync) + buffer	Parquet flush to S3
Compactor	catalog scan + plan	Parquet read + write to S3
Querier	cache-hit reads	cache-miss reads

Code.

# Ingester + querier resilience config
ingester:
  flush:
    upload_retries: 10
    upload_retry_backoff_seconds_max: 60
    max_wal_bytes_per_partition: 5GB   # WAL cap — drops writes if exceeded
    max_wal_age_seconds: 3600          # backpressure at 1 h

querier:
  object_store:
    request_timeout_seconds: 30
    retries: 5
    retry_backoff_ms_max: 5000
  cache:
    max_bytes: 200GB
    prefer_recent: true                # keep last N hours in cache

# Prometheus alerts to watch
# - iox_ingester_wal_bytes  > 2GB per partition  for 5m  →  S3 flush is stalled
# - iox_querier_object_store_error_rate > 5%     for 5m  →  S3 degraded
# - iox_compactor_backlog_files > 10000                  →  compaction is behind

# Runbook automation — check WAL depth per partition
import psycopg2

def wal_depth_per_partition(conn):
    """Query catalog for ingester WAL size to detect flush stalls."""
    with conn.cursor() as cur:
        cur.execute("""
            SELECT partition_id,
                   pg_size_pretty(wal_bytes) AS wal_size,
                   wal_bytes,
                   last_flush_at
            FROM   ingester_partition_state
            WHERE  wal_bytes > 1073741824    -- > 1 GB
            ORDER  BY wal_bytes DESC
            LIMIT  20
        """)
        for row in cur.fetchall():
            print(row)

Step-by-step explanation.

1. The router is completely unaffected — it only holds in-flight requests in memory and forwards to the ingester. As long as the ingester ACKs, the client sees no impact.
2. The ingester ACKs writes as soon as the WAL fsync completes, which is local-disk latency (~2-10 ms). S3 latency does not appear on the write path. What backs up is the flush — Parquet uploads that would normally take 500 ms now take 5-30 s. The ingester's WAL grows; if it grows past max_wal_bytes_per_partition (5 GB), the ingester starts NACKing new writes for that partition (backpressure).
3. The querier's cache hits stay fast — they read local disk. Cache misses hit S3 and stall for 5-30 s per file, which manifests as a Grafana panel that takes 30 s to render instead of 150 ms. The querier is not broken; it's slow.
4. The compactor sees S3 errors on its read/write cycle. It backs off with exponential retry and doesn't process new compaction batches. This is safe — the underlying Parquet files remain valid; compaction is a maintenance task, not a correctness one. When S3 recovers, the compactor catches up.
5. The catalog (Postgres) is unaffected because it doesn't touch S3. Metadata queries stay fast throughout the incident. This is why the catalog is not on S3 — a catalog outage would take everything down.

Output.

Service	Behaviour during S3 spike	Recovery behaviour
Router	unaffected	n/a
Ingester	WAL grows; ACKs stay fast until 5 GB cap	flushes drain when S3 returns
Compactor	backs off; no new batches	resumes; drains backlog
Querier	cache-hit fast; cache-miss slow	slow queries recover with S3
Catalog	unaffected	n/a

Rule of thumb. IOx's separation of concerns means an S3 hiccup slows reads and backs up flushes, but writes stay durable via WAL and metadata stays available via Postgres. The runbook is (a) alert on WAL depth, (b) alert on querier error rate, (c) don't panic — the engine is designed for this.

Senior interview question on IOx architecture

A senior interviewer might ask: "Design an IOx deployment for a multi-tenant SaaS metrics service that ingests 50 K points/sec across 200 tenants, keeps 30 days of raw + 1 year of downsampled data, and serves ~200 concurrent Grafana users. Cover the sizing of each service pool, the object-store bucket layout, the catalog choice, the WAL sizing, and the compactor backlog policy."

Solution Using a separated four-service deployment with S3, Postgres catalog, and per-tenant partition sharding

# 1. IOx Enterprise Helm values — production sizing
influxdb3:
  edition: enterprise
  license: ${env:INFLUXDB3_LICENSE}

  objectStore:
    type: s3
    bucket: acme-saas-iox
    region: us-east-1
    # Per-tenant prefix keeps IAM / cost accounting clean
    prefix_template: "{tenant_id}/"

  catalog:
    type: postgres
    dsn: postgres://iox_catalog:${env:PW}@catalog.rds.internal:5432/iox
    max_connections: 100

  # Router — 3 replicas behind an ALB
  router:
    replicas: 3
    resources: { cpu: 2, memory: 4Gi }

  # Ingester — 8 replicas, WAL on NVMe local disk
  ingester:
    replicas: 8
    resources: { cpu: 4, memory: 32Gi }
    storage:
      wal: 500Gi                        # NVMe
    flush:
      max_bytes_per_partition: 256MB
      max_age_seconds: 60
      max_wal_bytes_per_partition: 10GB

  # Compactor — 3 replicas, CPU-heavy
  compactor:
    replicas: 3
    resources: { cpu: 16, memory: 64Gi }
    backlog:
      max_files_per_partition: 500
      target_file_size: 128MB

  # Querier — 6 replicas, cache-heavy
  querier:
    replicas: 6
    resources: { cpu: 8, memory: 64Gi }
    storage:
      object_store_cache: 500Gi         # NVMe

-- 2. Per-tenant database + retention
-- (one database per tenant; keeps IAM + cost accounting per-tenant)

CREATE DATABASE tenant_1234
  WITH RETENTION_PERIOD = '30d';

-- Downsampling table for 1 y retention at coarser resolution
-- (via a scheduled dbt job that reads raw + writes downsampled)
CREATE TABLE cpu_5m (
    time         TIMESTAMP NOT NULL,
    namespace    STRING    NOT NULL,
    pod          STRING    NOT NULL,
    avg_usage    DOUBLE,
    peak_usage   DOUBLE
) WITH RETENTION_PERIOD = '365d';

# 3. Prometheus alerts — service-specific
groups:
- name: iox
  rules:
  - alert: IngesterWALDepth
    expr: iox_ingester_wal_bytes_per_partition > 5e9
    for: 5m
    annotations:
      summary: "Ingester WAL > 5 GB for a partition — S3 flush stalled?"
  - alert: QuerierErrorRate
    expr: rate(iox_querier_query_errors_total[5m]) > 0.05
    for: 10m
    annotations:
      summary: "Querier error rate > 5% — S3 or cache issue"
  - alert: CompactorBacklog
    expr: iox_compactor_pending_files > 20000
    for: 15m
    annotations:
      summary: "Compactor is > 20k files behind — scale up or investigate"
  - alert: CatalogSlowQueries
    expr: rate(pg_stat_activity_max_tx_duration{db="iox"}[5m]) > 30
    for: 5m
    annotations:
      summary: "Catalog Postgres has long-running transactions"

Step-by-step trace.

Concern	Config	Reasoning
Object store	S3 + per-tenant prefix	IAM + cost accounting per tenant
Catalog	Postgres RDS (100 conns)	ACID metadata; not on S3
Router replicas	3	stateless; scaled behind ALB
Ingester replicas	8 (WAL on NVMe 500 GB each)	50 K pts/sec / 8 = ~6 K pts/sec/ingester
Compactor replicas	3 (CPU-heavy)	keep small-file count bounded
Querier replicas	6 (500 GB cache each)	~200 concurrent users; cache heavy
Per-tenant DB	one per tenant	isolation; retention per tenant
Downsampling	dbt-scheduled 5m rollup	1 y retention affordable at coarser grain

After deployment, ingester throughput handles 50 K pts/sec headroom (each ingester at ~6 K sustained, room to spike 2×); querier cache serves most Grafana queries in < 200 ms; compactor keeps small-file count bounded at ~500 files per partition; per-tenant S3 prefix + database gives clean cost attribution.

Output:

Metric	Value
Ingest throughput headroom	50 K → 100 K pts/sec
P95 query latency (cached)	~150 ms
P95 query latency (uncached)	~1.5 s
Retention (raw)	30 d
Retention (5m downsampled)	365 d
S3 storage after compression	~2 TB raw + ~100 GB downsampled
S3 storage cost / month	~$50
Compactor throughput	200 M points / hour compacted

Why this works — concept by concept:

• Router replicas behind an ALB — the router is stateless, so horizontal scale is trivial. The ALB spreads client connections; router-side sharding decides which ingester owns each partition.
• Ingester on NVMe with 10 GB WAL cap — NVMe gives fast fsync (~2 ms); the 10 GB cap gives ~10 minutes of write buffering during an S3 outage before backpressure kicks in. Beyond the cap, writes are NACKed rather than lost.
• Compactor scaling on CPU — compaction is a merge-sort of Parquet row groups; CPU-heavy but I/O-bounded on read + write to S3. Three replicas at 16 vCPU each keep 50 K pts/sec ingest in bounded backlog.
• Querier cache on NVMe — 500 GB per querier covers ~5 days of hot data. Cache-hit queries are pure local-disk reads; cache-miss goes to S3 with parallel row-group fetching.
• Cost — 8 ingesters + 3 compactors + 6 queriers + 3 routers + 1 catalog Postgres + ~2 TB S3. Compared to InfluxDB 2.x Enterprise on the same workload (which would need ~15 TSM-monolith nodes and would drop writes past ~4 M cardinality), this is roughly half the compute cost with unlimited cardinality headroom. O(1) per write versus O(cardinality) per write.

Design
Topic — design
Design problems on distributed time-series systems

Practice →

SQL Topic — sql SQL problems on partitioned data and columnar reads

Practice →

---

3. Data model + writing — LineProtocol at scale

Measurements + tags + fields + timestamps — the InfluxDB data model that IOx kept while unlocking unlimited cardinality

The mental model in one line: InfluxDB 3 keeps the same wire-level data model that 1.x and 2.x used — a measurement (like a table name), a set of tags (indexed dimensions, always strings), a set of fields (values, typed as float / integer / boolean / string), and a nanosecond timestamp — but represents it under the hood as an Apache Arrow-batched, Parquet-serialised, column-per-tag layout that removes the in-memory series file entirely, so cardinality is bounded only by the object storage bill and no longer by the engine's RAM. This is the single most important compatibility decision InfluxData made: your LineProtocol writer, your Telegraf collector, your Kubernetes metrics agent — all keep working unchanged.

The four components of a point.

• Measurement. The logical table. Named like cpu, http_requests, sensor_temperature. In IOx this maps to a Parquet table with a schema derived from all tags + fields ever written to that measurement. Schema-on-write with additive evolution — new tags become new columns; existing rows carry NULL for them.
• Tags. The indexed dimensions. Always strings; always low-cardinality ideally (though IOx removes the hard ceiling). Tags become Parquet columns with dictionary encoding — 1 M rows with 20 K unique pod_name values encode as 20 K distinct dictionary entries + 1 M small integer references. Compact.
• Fields. The values. Typed: float64, int64, uint64, bool, string. One measurement can have many fields; a single point sets some or all. Fields are the "what the metric is worth" columns.
• Timestamp. Nanosecond precision (with millisecond and second precision available via the precision write parameter). Every point has exactly one timestamp; multiple points with identical (measurement, tag-set, timestamp) are deduplicated by the compactor with last-write-wins semantics.

Why unlimited cardinality is real in IOx (and was fictional in TSM).

• TSM's problem. The 2.x TSM engine kept an in-memory series file mapping every unique (measurement, tag-set) combination to a series ID. At ~50 bytes per series, 4 M series consumed 200 MB — but the lookup cost was O(series_count) for many queries. Past ~4 M series per shard, the engine would OOM, slow down catastrophically, or refuse new writes (max-series-per-database).
• IOx's fix. No series file. Tags are Parquet columns. A point with a new tag value writes a new dictionary entry in the Parquet file's tag column — a ~10-byte addition, not a 50-byte-plus-index entry. The only cost of high cardinality is (a) larger dictionaries per Parquet file and (b) more distinct row groups after grouping/aggregation — both O(unique_values), both cheap, both bounded by object-storage cost, not by RAM.
• The practical consequence. Deployments that struggled at 5 M series in TSM run comfortably at 500 M series in IOx. The engine simply doesn't care about cardinality growth in the same way.

LineProtocol — the same wire format, still the primary write path.

• The shape. measurement,tag1=v1,tag2=v2 field1=x,field2=y timestamp. Tags and fields are comma-separated key/value pairs. Whitespace is significant (one space between tags and fields, one between fields and timestamp).
• Batching. Send many lines per HTTP POST. 5,000 lines per batch is a good default; larger batches waste memory in the router; smaller batches waste network round-trips.
• Precision. The precision query parameter tells the router how to interpret the timestamp: ns (default), us, ms, s. Match your source clock's precision — mismatched precision is a common Telegraf configuration bug.
• HTTP endpoint. POST /api/v3/write_lp?db=<db>&precision=<precision> returns 204 No Content on success. Errors are 400 (bad LineProtocol), 401 (auth), 413 (batch too large), 429 (rate-limited), 503 (backpressure).

Flight SQL for high-throughput binary ingest.

• What it is. An Arrow-native binary protocol built on gRPC. Instead of serialising to LineProtocol text and re-parsing on the router, a producer can build an Arrow RecordBatch in memory and send it as raw Arrow IPC frames.
• When to use it. Producer throughput > 100 K pts/sec, or producer is already Arrow-native (Pandas, Polars, Spark). At those rates, LineProtocol parsing dominates client CPU; Flight SQL removes that tax.
• When not to use it. IoT edge devices, Telegraf, Prometheus remote-write — LineProtocol is fine and easier to observe with curl.

Retention — "unlimited" means "bounded by cost."

• Per-database retention. Set at CREATE DATABASE or altered later. RETENTION_PERIOD = '90d' keeps 90 days; 'INF' (or omit) keeps forever.
• Enforcement. The compactor deletes Parquet files whose partition-end time is older than the retention window. This is O(files-per-partition) rather than O(rows) — cheap.
• Downsampling. Not a first-class engine concept in IOx — instead, run a scheduled SQL job (dbt, Airflow, cron) that reads raw + writes a downsampled table with a longer retention. This is the standard "materialised rollup" pattern from warehousing.

Common interview probes on the data model.

• "How does IOx handle high cardinality?" — required answer: tags become dictionary-encoded Parquet columns; no series file.
• "What breaks if I add a new tag tomorrow?" — nothing; schema evolution is additive; existing rows are NULL for the new column.
• "How does LineProtocol precision work?" — the precision query parameter tells the router how to interpret integer timestamps.
• "When should I use Flight SQL for ingest?" — high-throughput Arrow-native producers; the parse tax on LineProtocol becomes the bottleneck.
• "How is retention enforced?" — the compactor drops Parquet files past the retention window; no scan-and-delete.

Worked example — a LineProtocol batch for Kubernetes pod metrics

Detailed explanation. The canonical write path for a K8s observability workload: a Telegraf DaemonSet collects /proc stats every 15 s, emits one cpu point per container plus one mem point, batches ~500 pods × 2 metrics = 1000 points, and POSTs them as LineProtocol every 15 s. Walk through the LineProtocol shape and the write.

• Measurements. cpu, mem.
• Tags. pod, namespace, node, container — enough to identify the source; low-to-medium cardinality individually but multiplicative.
• Fields. cpu: usage_pct, load1, iowait. mem: used_bytes, available_bytes, pct.
• Timestamp. Nanosecond precision at scrape time.

Question. Show the LineProtocol batch for one 15-second scrape and the Python client that POSTs it.

Input.

Component	Value
Measurement	cpu, mem
Tag cardinality per pod	pod × namespace × node × container
Fields per point	3 (cpu), 3 (mem)
Batch size	1000 lines (500 pods × 2 measurements)
Precision	ns

Code.

# Telegraf-style LineProtocol batcher for K8s pod metrics
import requests
import time

INFLUX3_URL = "http://influx3-router.internal:8181/api/v3/write_lp"
DB          = "observability"
TOKEN       = "..."

def build_lineprotocol(pod_metrics: list[dict]) -> str:
    """Convert a list of pod-metric dicts into LineProtocol lines."""
    ts_ns = int(time.time_ns())
    lines = []
    for m in pod_metrics:
        # cpu measurement
        lines.append(
            f"cpu,pod={m['pod']},namespace={m['namespace']},"
            f"node={m['node']},container={m['container']} "
            f"usage_pct={m['cpu_pct']},load1={m['load1']},iowait={m['iowait']} "
            f"{ts_ns}"
        )
        # mem measurement
        lines.append(
            f"mem,pod={m['pod']},namespace={m['namespace']},"
            f"node={m['node']},container={m['container']} "
            f"used_bytes={m['mem_used']}i,available_bytes={m['mem_avail']}i,"
            f"pct={m['mem_pct']} "
            f"{ts_ns}"
        )
    return "\n".join(lines)


def write_batch(lines: str) -> None:
    r = requests.post(
        INFLUX3_URL,
        params={"db": DB, "precision": "ns"},
        headers={
            "Authorization": f"Bearer {TOKEN}",
            "Content-Type": "text/plain; charset=utf-8",
        },
        data=lines.encode("utf-8"),
        timeout=15,
    )
    r.raise_for_status()


# Example — 500 pods × 2 measurements = 1000 lines
scrape = [
    {
        "pod": f"web-{i:04d}",
        "namespace": "prod" if i % 5 else "staging",
        "node": f"node-{i % 20}",
        "container": "app",
        "cpu_pct": 40.5 + (i % 30),
        "load1": 0.4 + (i % 100) / 100.0,
        "iowait": 0.1 + (i % 20) / 100.0,
        "mem_used": 500_000_000 + i * 1000,
        "mem_avail": 4_000_000_000,
        "pct": 12.5 + (i % 20),
    }
    for i in range(500)
]

lp = build_lineprotocol(scrape)
write_batch(lp)

Step-by-step explanation.

1. The build_lineprotocol helper renders each pod as two lines — one cpu, one mem — with identical tags but different fields. The trailing i after integer values (used_bytes=500000000i) tells the router this field is a int64, not a float64 — critical for storage and query efficiency.
2. Tag values must not contain , = or spaces without escaping. In practice, pod names, namespaces, and container names never contain these characters, so the raw f-string is safe. In LineProtocol libraries (like influxdb3-python), the escaping is handled automatically.
3. The batch of 1000 lines is ~150 KB uncompressed. HTTP POST /api/v3/write_lp returns 204 No Content on success — no body. If any line is malformed, the whole batch is rejected with 400 Bad Request and an error body indicating the offending line number.
4. The precision=ns query parameter tells the router that the trailing integer on each line is a nanosecond timestamp. Match this to your source's precision — if Telegraf writes second precision (integer seconds) and you send precision=ns, IOx will interpret 1720051200 as 1720051200 nanoseconds (1970-01-20) instead of 2024-07-04.
5. On the ingester side, this batch turns into ~1000 rows in the cpu measurement's Arrow buffer and ~1000 rows in the mem buffer, indexed by the (pod, namespace, node, container) tag combination. Repeated writes for the same pod at different timestamps append to the same partition; the compactor merges partitions into fewer, larger Parquet files over time.

Output.

Aspect	Value
Lines per batch	1000
Bytes per batch (raw)	~150 KB
Bytes per batch (gzip)	~15 KB
Distinct tag combinations added	~500 pods (if first scrape)
Ingester WAL bytes added	~200 KB (raw + framing)
Parquet bytes after compaction	~10-15 KB per scrape (dict + zstd)
End-to-end write latency	20-30 ms

Rule of thumb. Batch LineProtocol at 1000-10000 lines per POST for a good balance of network overhead vs router memory pressure. Always send an explicit precision — don't rely on defaults. If you're moving Pandas / Polars DataFrames, skip LineProtocol entirely and use Flight SQL.

Worked example — Flight SQL binary ingest for a Spark job

Detailed explanation. A Spark ETL job produces ~500 K rows/sec of aggregated metrics that need to land in IOx. LineProtocol serialisation (float → string → parse → Arrow) would eat 60% of the CPU. Flight SQL lets Spark send Arrow RecordBatches directly to the ingester. Walk through the Python client.

• Source. A PySpark DataFrame with columns time, pod, namespace, usage_pct.
• Target. The cpu measurement in IOx.
• Path. DataFrame → PyArrow RecordBatch → Flight SQL DoPut → ingester.

Question. Write the Python code that sends a PyArrow RecordBatch to IOx via Flight SQL, and quantify the CPU savings vs LineProtocol.

Input.

Aspect	LineProtocol path	Flight SQL path
Serialisation	float → string → LineProtocol	zero-copy Arrow RecordBatch
Router parse cost	O(bytes) parse	none
Client CPU	high (string formatting)	low (Arrow already built)
Network bytes (raw)	~150 bytes/point	~40 bytes/point (Arrow)
Throughput ceiling	~100 K pts/sec/client	~500 K pts/sec/client

Code.

# Flight SQL ingest for a Spark-produced Arrow batch
import pyarrow as pa
import adbc_driver_flightsql.dbapi as flight_sql

INFLUX3_FLIGHT_URI = "grpc+tls://influx3-flight.internal:8443"
TOKEN              = "..."

def build_arrow_batch(pandas_df) -> pa.RecordBatch:
    """Convert a pandas DataFrame into an Arrow RecordBatch with IOx-friendly schema.

    Column contract:
      - `time` must be timestamp[ns] (or ms / us with an appropriate cast).
      - Tag columns are string (dictionary-encoded is even better).
      - Field columns are typed (float64 / int64 / bool / string).
    """
    schema = pa.schema([
        ("time",       pa.timestamp("ns")),
        ("pod",        pa.string()),
        ("namespace",  pa.string()),
        ("usage_pct",  pa.float64()),
    ])
    return pa.RecordBatch.from_pandas(pandas_df, schema=schema)


def ingest_batch(rb: pa.RecordBatch, measurement: str, db: str) -> None:
    """Send an Arrow RecordBatch to IOx as an INSERT via Flight SQL."""
    with flight_sql.connect(
        uri=INFLUX3_FLIGHT_URI,
        db_kwargs={
            "adbc.flight.sql.authorization_header": f"Bearer {TOKEN}",
            "adbc.flight.sql.rpc.call_header.database": db,
        },
    ) as conn:
        with conn.cursor() as cur:
            # ADBC Flight SQL: adbc_ingest routes an Arrow table into a target
            # measurement via the ingester's Flight endpoint (zero-copy).
            cur.adbc_ingest(
                table_name=measurement,
                data=pa.Table.from_batches([rb]),
                mode="append",
            )


# Called from a Spark foreachPartition — one Flight connection per partition
def spark_partition_writer(rows_iter):
    import pandas as pd
    df = pd.DataFrame(rows_iter, columns=["time", "pod", "namespace", "usage_pct"])
    rb = build_arrow_batch(df)
    ingest_batch(rb, measurement="cpu", db="observability")

Step-by-step explanation.

1. The PySpark DataFrame is materialised as a Pandas DataFrame within each Spark partition via foreachPartition. This keeps memory bounded — one partition at a time, not the whole DataFrame in driver memory.
2. build_arrow_batch converts the Pandas DataFrame into a PyArrow RecordBatch with an explicit schema. The time column must be timestamp[ns] (or a precision IOx accepts); the tag columns are string; the field is float64. Explicit schema avoids Arrow's slower inference path.
3. adbc_ingest is the ADBC Flight SQL "bulk insert" call. It sends the Arrow batch as Arrow IPC frames over gRPC — the wire format is already the in-memory format, so there's no serialisation. The ingester receives the batch, WAL-appends, and buffers.
4. The client CPU savings are dramatic — for a 500 K rows/sec workload, LineProtocol serialisation was ~4-6 vCPU per Spark executor; Flight SQL is ~0.5-1 vCPU. The freed CPU turns into ~5× more headroom on the same cluster.
5. Network bytes drop too — Arrow's binary encoding is ~4× more compact than LineProtocol for numeric-heavy data (a float64 is 8 bytes in Arrow vs ~15-20 in LineProtocol text). At 500 K pts/sec, this is ~15 MB/sec instead of ~75 MB/sec — meaningful at cross-region rates.

Output.

Metric	LineProtocol path	Flight SQL path
Client CPU (500 K pts/sec)	~5 vCPU	~0.8 vCPU
Network bytes / sec	~75 MB/s	~15 MB/s
Ingester CPU (parse)	~2 vCPU	~0.2 vCPU
End-to-end p50 latency	~30 ms	~15 ms
Client library complexity	20 LOC	30 LOC

Rule of thumb. Flight SQL wins whenever the producer is already Arrow-native (Spark, Polars, Pandas, DuckDB) or when producer throughput passes ~100 K pts/sec/client. For low-rate producers (Telegraf, edge devices, small services), LineProtocol is fine and simpler to debug.

Worked example — proving unlimited cardinality with a 10 M series ingest

Detailed explanation. The most cited weakness of InfluxDB 2.x was the cardinality cliff. A concrete demonstration: ingest 10 million unique (host, region, service) combinations into a single measurement over one hour and observe that IOx stays flat while 2.x TSM would have OOM'd or dropped writes long before. Walk through the test.

• Workload. 10 M unique tag combinations, each writing one point every 60 s = ~166 K pts/sec.
• Duration. 1 hour → 600 M points, 10 M distinct series.
• Metric. Ingester WAL depth, catalog file count, memory usage — all should stay bounded.

Question. Write the ingest generator and record the metrics that demonstrate IOx handles the cardinality.

Input.

Metric	Expected in IOx	Would be in 2.x TSM
Series count	10 M (accepted)	OOM around 4 M
Ingester memory	~8 GB per node	N/A (crashed)
Parquet files / partition	60-100 (post-compaction)	N/A
Catalog rows	~10 M partitions	N/A
P99 write latency	~30 ms	N/A

Code.

# Cardinality stress test — 10 M unique tag combos
import requests
import time
import random

INFLUX3_URL = "http://influx3-router.internal:8181/api/v3/write_lp"
DB          = "cardinality_test"

def one_point(host_id: int, region: str, service: str, ts_ns: int) -> str:
    return (
        f"heartbeat,host=host-{host_id:07d},region={region},service={service} "
        f"beat=1i,latency_ms={random.random() * 100:.2f} "
        f"{ts_ns}"
    )

def batch(size: int, offset: int, ts_ns: int) -> str:
    lines = []
    for i in range(offset, offset + size):
        host_id = i % 10_000_000                       # 10 M unique hosts
        region  = ["us-east", "us-west", "eu", "apac"][i % 4]
        service = f"svc-{(i // 10_000_000) % 100:02d}" # additional dimension
        lines.append(one_point(host_id, region, service, ts_ns))
    return "\n".join(lines)

def run_stress_test():
    # Aim for ~166 K pts/sec sustained for 1 hour
    batch_size    = 5000
    batches_per_s = 33
    end_ts        = time.time() + 3600
    offset        = 0

    while time.time() < end_ts:
        ts_ns = time.time_ns()
        for _ in range(batches_per_s):
            payload = batch(batch_size, offset, ts_ns)
            r = requests.post(
                INFLUX3_URL,
                params={"db": DB, "precision": "ns"},
                data=payload.encode("utf-8"),
                timeout=10,
            )
            r.raise_for_status()
            offset += batch_size
        # Sleep to keep rate at ~166 K pts/sec
        time.sleep(max(0, 1.0 - (time.time_ns() - ts_ns) / 1e9))

if __name__ == "__main__":
    run_stress_test()

-- Verify cardinality growth is real
SELECT COUNT(DISTINCT host)     AS host_cardinality,
       COUNT(DISTINCT region)   AS region_cardinality,
       COUNT(DISTINCT service)  AS service_cardinality,
       COUNT(*)                 AS point_count
FROM   heartbeat
WHERE  time > now() - INTERVAL '1 hour';
-- Expected: host_cardinality ≈ 10_000_000

Step-by-step explanation.

1. The generator produces 5000-line batches, each with unique host values sampled from a 10 M-wide space plus a region + service combination that adds another 400× multiplicativity (though many combinations repeat). Sustained rate is ~166 K points/sec, matching a real-world large K8s cluster.
2. On the ingester side, each new (host, region, service) combination results in no series-file entry (there isn't one), only a new dictionary entry in the corresponding Parquet file's tag columns. Memory pressure is O(distinct-values-in-current-buffer), which stays bounded because the buffer flushes every 60 s.
3. The compactor eventually merges the many small Parquet files (one per flush) into fewer larger ones, adding dictionary entries for cross-partition cardinality without ever materialising a giant series-file. Query performance is O(row-groups-touched-by-predicate) — bounded by the query, not by the total series count.
4. Catalog growth is ~10 M distinct partition-key rows (one per unique host for a partition scheme that includes host), or much less if the partition key aggregates hosts into buckets. This is one place to tune — a per-host partition would explode the catalog; a bucketed partition keeps catalog rows bounded.
5. In 2.x TSM, this workload would OOM the process around 4 M distinct series (~15 minutes in) because every new series added a series-file entry and every query scanned the series file. In IOx, the same workload runs flat for the full hour.

Output.

Time	Series count observed	Ingester memory	P99 write latency	Notes
t + 10 min	~1 M	4 GB	25 ms	Baseline
t + 30 min	~4 M	6 GB	27 ms	TSM would have OOM'd here
t + 45 min	~7 M	7 GB	29 ms	Compactor is merging
t + 60 min	~10 M	8 GB	31 ms	Stable

Rule of thumb. IOx's cardinality behaviour is O(unique-values-per-buffer-flush), not O(total-cardinality). Design your tag schema for query patterns, not to appease a series-file cap that no longer exists. The only real cost of high cardinality is per-query row-group count, which the DataFusion pruning stats manage down to a few thousand row groups even at 100 M series.

Senior interview question on data model + writing

A senior interviewer might ask: "You need to ingest 300 K pts/sec of IoT sensor data with per-sensor tags (10 M sensors globally), 1-year retention, and a Grafana dashboard that plots avg(temp) BY region. Walk me through the LineProtocol schema, the batch sizes, the compactor tuning, the retention config, and how you'd prove the cardinality is not a problem."

Solution Using LineProtocol batching, a bucketed partition scheme, and a scheduled downsampling rollup

-- 1. Database with 30-day raw retention + a downsampled table for 1 y
CREATE DATABASE iot_telemetry
  WITH RETENTION_PERIOD = '30d';

-- Downsampled table — 5-minute averages, 1 y retention
CREATE TABLE sensor_5m (
    time           TIMESTAMP NOT NULL,
    region         STRING    NOT NULL,
    sensor_type    STRING    NOT NULL,
    avg_temp       DOUBLE,
    p95_temp       DOUBLE,
    sample_count   INT
) WITH RETENTION_PERIOD = '365d';

# 2. Ingest generator (production pattern; batched with backpressure handling)
import requests
import time
from itertools import islice

INFLUX3_URL   = "http://influx3-router.internal:8181/api/v3/write_lp"
DB            = "iot_telemetry"
TOKEN         = "..."
BATCH_SIZE    = 5000

def chunk(iterable, n):
    it = iter(iterable)
    while True:
        c = list(islice(it, n))
        if not c:
            return
        yield c

def sensor_lp(reading: dict, ts_ns: int) -> str:
    return (
        f"sensor,region={reading['region']},"
        f"sensor_type={reading['sensor_type']},"
        f"sensor_id={reading['sensor_id']} "
        f"temp={reading['temp']},humidity={reading['humidity']},"
        f"battery_pct={reading['battery']} "
        f"{ts_ns}"
    )

def write_readings(readings_iterable):
    """Batch, POST, and handle 429 backpressure with exponential retry."""
    for batch in chunk(readings_iterable, BATCH_SIZE):
        ts_ns = time.time_ns()
        payload = "\n".join(sensor_lp(r, ts_ns) for r in batch).encode("utf-8")

        for attempt in range(6):
            r = requests.post(
                INFLUX3_URL,
                params={"db": DB, "precision": "ns"},
                headers={"Authorization": f"Bearer {TOKEN}"},
                data=payload,
                timeout=15,
            )
            if r.status_code == 204:
                break
            if r.status_code in (429, 503):
                time.sleep(2 ** attempt)   # backpressure — back off
                continue
            r.raise_for_status()
        else:
            raise RuntimeError("write failed after 6 attempts")

-- 3. Downsampling rollup — scheduled every 5 minutes by Airflow / cron
INSERT INTO sensor_5m (time, region, sensor_type, avg_temp, p95_temp, sample_count)
SELECT date_bin(INTERVAL '5 minutes', time)      AS time,
       region,
       sensor_type,
       AVG(temp)                                  AS avg_temp,
       APPROX_PERCENTILE_CONT(temp, 0.95)         AS p95_temp,
       COUNT(*)                                   AS sample_count
FROM   sensor
WHERE  time >= now() - INTERVAL '10 minutes'
  AND  time <  now() - INTERVAL '5 minutes'
GROUP  BY 1, 2, 3;

# 4. Compactor tuning for the workload
compactor:
  target_file_size: 128MB
  max_files_per_partition: 500
  partition_scheme:
    # Bucket sensor_id into 1024 buckets so the catalog stays bounded
    # even as sensor cardinality grows unbounded
    - hash(sensor_id, 1024)
    - day(time)
  retention_enforcement:
    check_interval_minutes: 60

Step-by-step trace.

Layer	Choice	Reasoning
Batch size	5000 lines / POST	good HTTP overhead ratio; fits router memory
Precision	ns	matches sensor firmware timestamps
Tags	region, sensor_type, sensor_id	query-shape driven
Fields	temp, humidity, battery_pct	typed floats
Retention (raw)	30 d	dashboards look back 30 d
Retention (5m rollup)	365 d	year-over-year comparisons
Partition scheme	hash(sensor_id, 1024) × day(time)	bounds catalog at ~365 × 1024 rows
Rollup schedule	every 5 min via cron	matches rollup grain
Backpressure	retry on 429/503	ingester WAL cap surfaces here

After deployment, IOx handles 300 K pts/sec on 8 ingester replicas, the compactor keeps small-file count below 500 per partition, the 5-minute rollup runs in ~30 s each cycle, and the Grafana dashboard queries sensor_5m (365-day range) instead of sensor (30-day range) for long-range views.

Output:

Metric	Value
Sustained ingest rate	300 K pts/sec
Distinct series	~10 M sensors
Raw Parquet storage / day	~12 GB (zstd)
Rollup storage / day	~0.1 GB
Retention footprint (raw + rollup)	~380 GB total
S3 cost / month	~$9
P95 dashboard query latency	~200 ms

Why this works — concept by concept:

• LineProtocol batching — 5000 lines per POST amortises HTTP overhead and gives the router enough work to justify a network round-trip. Smaller batches waste round-trips; larger ones bloat router memory and hurt tail latency.
• Bucketed partition scheme — hash(sensor_id, 1024) × day(time) bounds the catalog at ~1024 × retention-days rows regardless of how many actual sensors exist. Without bucketing, a per-sensor partition would create ~10 M × retention-days catalog rows.
• DataFusion downsampling via scheduled SQL — no continuous-query engine; just a scheduled INSERT ... SELECT that reads the last 5-minute window from sensor and writes to sensor_5m. Standard warehousing pattern; runs anywhere (Airflow, Lambda, cron).
• Retention as partition dropping — the compactor drops Parquet files past the retention boundary. This is O(files) rather than O(rows) — cheap even at trillions of points.
• Cost — 300 K pts/sec on 8 ingesters + 3 compactors + 4 queriers + 1 Postgres catalog + ~380 GB S3. Compared to InfluxDB 2.x on the same workload (which would need ~30 TSM nodes and would drop writes past ~4 M sensors), this is roughly a third of the compute cost with unlimited sensor headroom. O(1) per write regardless of sensor cardinality.

Aggregation
Topic — aggregation
Aggregation problems on high-cardinality metrics

Practice →

SQL Topic — sql SQL problems on time-bucketed aggregations

Practice →

---

4. Querying — SQL, InfluxQL, Flight SQL via DataFusion

One engine, three dialects — SQL is first-class, InfluxQL stays for legacy dashboards, Flight SQL is the binary lane

The mental model in one line: InfluxDB 3 exposes three query dialects on top of a single DataFusion execution engine — influxdb sql (ANSI-flavoured SQL via DataFusion, with time-series functions like date_bin and date_bin_gapfill) is the primary interface, InfluxQL is preserved for backwards compatibility with 1.x / 2.x dashboards, and flight sql influxdb is the Arrow-native binary lane for high-throughput programmatic access — all three go through the same querier, the same DataFusion planner, the same Parquet reader, and the same pushdown-and-vectorised execution pipeline. Flux, the 2.x scripting language, is deprecated; do not build new pipelines on it.

The three dialects in one paragraph each.

• SQL (via DataFusion). Standard SQL with DataFusion's dialect extensions (comparable to Postgres in many ways, with Arrow-typed functions). Full window functions, joins, CTEs, subqueries. Time-series functions include date_bin (fixed-interval bucketing), date_bin_gapfill (buckets with gap-filling), first_value, last_value, lag, lead. This is the dialect for new pipelines, new dashboards, and any tool that speaks SQL (dbt, Superset, Metabase, DBeaver).
• InfluxQL. The legacy 1.x / 2.x query language. Kept intact so existing Grafana dashboards, alert rules, and Chronograf panels continue to work without rewrite. IOx translates InfluxQL to DataFusion logical plans behind the scenes. Not a good target for new dashboards; new dashboards should use SQL.
• Flight SQL. Not a dialect but a protocol — a gRPC + Arrow IPC-based way to send SQL queries and receive Arrow results without any text-based serialisation. Used by high-throughput BI tools (Grafana's InfluxDB v3 datasource uses it), programmatic ADBC clients (Python's adbc_driver_flightsql, Rust's arrow-flight), and DuckDB / Polars integrations.

Flux is deprecated — what that means in practice.

• Deprecation status. InfluxDB 3 Core does not ship Flux. Enterprise ships a Flux compatibility layer for 2.x migration but does not add new Flux features. Cloud has a documented sunset timeline.
• Migration story. Most Flux scripts translate cleanly to SQL — range(), filter(), aggregateWindow() map to WHERE, SELECT WHERE, GROUP BY date_bin(...). Complex Flux (multiple joins, custom map() transforms, pivot()) may need manual rewrite; simple ones can be scripted.
• What breaks. Anything that relied on Flux-specific features — Kapacitor-driven alerts, custom Flux packages, Flux tasks with UDFs. Rewrite as SQL + external orchestrator (Airflow, Prefect) if needed.

The DataFusion optimiser — the four pushdowns.

• Predicate pushdown. WHERE time >= now() - '1h' AND region = 'us-east' — the time predicate goes to the Parquet reader (row-group pruning by min/max stats); the tag predicate is evaluated at the row-group level using dictionary pages.
• Partition pruning. WHERE time >= '2026-07-01' — the catalog is consulted; Parquet files whose partition range doesn't overlap the predicate are never opened.
• Projection pushdown. SELECT time, usage_pct FROM cpu — only two columns are read from Parquet. A 50-column measurement reads 2/50 of the bytes.
• Aggregation pushdown (partial). SELECT COUNT(*) — pushed to the Parquet reader's row-count metadata; never scans rows. MIN / MAX on some columns can use Parquet's per-row-group stats. Complex aggregations are two-phase (partial per-node → final merge).

Time-series-native SQL functions.

• date_bin(interval, timestamp). Buckets a timestamp into fixed intervals. date_bin(INTERVAL '5 minutes', time) returns the bucket start. This replaces InfluxQL's time(5m) and Flux's aggregateWindow(every: 5m).
• date_bin_gapfill(interval, timestamp). Same as date_bin but with rows synthesised for buckets that had no data — critical for dashboards that need every bucket represented even during outages.
• first_value / last_value. Window functions returning the first or last value in a window ordered by time — replacements for InfluxQL's FIRST() / LAST().
• lag / lead. Standard SQL window functions for row-over-row deltas (rate calculations, gap detection).

Downstream BI integrations.

• Grafana. Native InfluxDB v3 datasource speaks Flight SQL. SQL, InfluxQL, and Flux (legacy) are all supported in the query editor.
• Superset. Uses the Flight SQL SQLAlchemy dialect; SQL only.
• Metabase. Same — SQL over Flight SQL.
• DBeaver / IntelliJ. JDBC Flight SQL driver; SQL only.
• Programmatic. Python via adbc_driver_flightsql, Rust via arrow-flight, Go via arrow-adbc.

Common interview probes on querying.

• "How does a SELECT hit Parquet in S3?" — querier → catalog lookup → download (or cache) → DataFusion plan → Parquet reader with pushdowns → Arrow batch → result.
• "What does DataFusion optimise?" — predicate + partition + projection + partial-aggregate pushdowns; two-phase aggregation; vectorised execution.
• "Why is Flight SQL faster than HTTP for BI?" — zero-copy Arrow batches; no CSV/JSON serialisation tax.
• "What replaces time(5m) from InfluxQL?" — date_bin(INTERVAL '5 minutes', time) in SQL.
• "Is Flux still supported?" — deprecated in Cloud, absent in Core, compat-only in Enterprise; new work uses SQL.

Worked example — a Grafana-style aggregation query with date_bin

Detailed explanation. The single most common IOx query in production is a time-bucketed aggregation for a Grafana dashboard: "give me AVG(cpu) in 5-minute buckets for the last hour, grouped by namespace." Walk through the SQL, the EXPLAIN plan, and the pushdowns that fire.

• Query. SELECT date_bin('5m', time), namespace, AVG(usage_pct) FROM cpu WHERE time > now() - '1h' GROUP BY 1, 2.
• Expected pushdowns. Partition pruning (only last hour's files), predicate pushdown (time filter into Parquet), projection pushdown (only time, namespace, usage_pct read).
• Expected shape. 12 buckets (60 min / 5 min) × N namespaces = tens to hundreds of rows.

Question. Write the query, the EXPLAIN plan, and interpret the pushdowns.

Input.

Parameter	Value
Measurement	cpu
Time range	last hour
Bucket	5 minutes
Group by	namespace
Metric	AVG(usage_pct)

Code.

-- The Grafana panel query
SELECT date_bin(INTERVAL '5 minutes', time) AS bucket,
       namespace,
       AVG(usage_pct)                        AS avg_cpu
FROM   cpu
WHERE  time >= now() - INTERVAL '1 hour'
GROUP  BY 1, 2
ORDER  BY 1, 2;

-- Same query with EXPLAIN to see the physical plan
EXPLAIN
SELECT date_bin(INTERVAL '5 minutes', time) AS bucket,
       namespace,
       AVG(usage_pct)                        AS avg_cpu
FROM   cpu
WHERE  time >= now() - INTERVAL '1 hour'
GROUP  BY 1, 2;

-- Sample output (abbreviated):
-- SortExec: expr=[bucket ASC, namespace ASC]
--   AggregateExec: mode=Final, gby=[bucket, namespace], aggr=[AVG(usage_pct)]
--     RepartitionExec: partitioning=Hash([bucket, namespace], 8)
--       AggregateExec: mode=Partial, gby=[bucket, namespace], aggr=[AVG(usage_pct)]
--         ProjectionExec: expr=[date_bin(...) AS bucket, namespace, usage_pct]
--           RepartitionExec: partitioning=RoundRobinBatch(8)
--             ParquetExec:
--               file_groups={40 files},
--               projection=[time, namespace, usage_pct],       -- projection pushdown
--               predicate=time >= <lit_expr>,                   -- predicate pushdown
--               pruning_predicate=time_max >= <lit_expr>        -- row-group pruning

Step-by-step explanation.

1. The SQL is standard DataFusion — date_bin is the time-bucketing function, AVG is the standard aggregate, GROUP BY 1, 2 refers to the first two SELECT expressions. INTERVAL '5 minutes' is the SQL interval literal.
2. The EXPLAIN plan reveals the pushdowns. ParquetExec shows projection=[time, namespace, usage_pct] — of cpu's N columns, only these three are read from disk. predicate=time >= <lit_expr> is evaluated during the Parquet scan; pruning_predicate=time_max >= <lit_expr> uses per-row-group min/max stats to skip whole row groups.
3. The file_groups={40 files} in the plan shows the catalog-side partition pruning result: 40 Parquet files are opened, out of potentially thousands. This is the biggest single latency win — files not opened cost nothing.
4. Aggregation is two-phase. AggregateExec: mode=Partial runs per-partition, producing partial sums + counts per (bucket, namespace). RepartitionExec: Hash([bucket, namespace], 8) shuffles by group key across 8 threads. AggregateExec: mode=Final merges the partials. This is the standard MPP shape — parallel enough for a single querier node, not distributed enough to need cross-node coordination.
5. The final SortExec orders results by bucket + namespace for Grafana's line chart. Sorting is done on the (small) final result, not on the raw Parquet scan.

Output.

bucket	namespace	avg_cpu
2026-07-27 08:00:00	prod	42.3
2026-07-27 08:00:00	staging	18.7
2026-07-27 08:05:00	prod	44.1
2026-07-27 08:05:00	staging	19.2
...	...	...

Rule of thumb. Always run EXPLAIN on a slow query. If projection doesn't match the SELECT columns, you have a projection-pushdown bug. If pruning_predicate is absent, add explicit time predicates on the partitioned column. If file_groups opens > 100 files, tighten the time range or check partition config.

Worked example — the InfluxQL compatibility shim for legacy dashboards

Detailed explanation. A team migrates from InfluxDB 2.x with 300 legacy Grafana dashboards written in InfluxQL. Rewriting each dashboard would take weeks. IOx's InfluxQL compatibility runs the legacy queries unchanged, translating them to DataFusion under the hood. Walk through the mapping.

• Legacy query. SELECT mean("usage_pct") FROM "cpu" WHERE time > now() - 1h GROUP BY time(5m), "namespace".
• DataFusion translation. SELECT date_bin('5m', time), namespace, AVG(usage_pct) FROM cpu WHERE time > now() - '1h' GROUP BY 1, 2.
• User-visible behaviour. Identical; same rows, same order, same dashboard.

Question. Show the InfluxQL query, the equivalent SQL, and what happens in the querier.

Input.

InfluxQL construct	SQL equivalent
SELECT mean("usage_pct")	SELECT AVG(usage_pct)
FROM "cpu"	FROM cpu
WHERE time > now() - 1h	WHERE time > now() - INTERVAL '1 hour'
GROUP BY time(5m), "namespace"	GROUP BY date_bin('5 minutes', time), namespace

Code.

-- 1. Legacy InfluxQL query (existing Grafana dashboard) — unchanged
SELECT mean("usage_pct") FROM "cpu"
WHERE time > now() - 1h
GROUP BY time(5m), "namespace"

-- 2. The equivalent SQL that IOx compiles it to
SELECT date_bin(INTERVAL '5 minutes', time) AS time,
       namespace,
       AVG(usage_pct)                        AS mean
FROM   cpu
WHERE  time > now() - INTERVAL '1 hour'
GROUP  BY 1, 2
ORDER  BY 1;

# Grafana datasource config for InfluxDB v3 — dialect can be per-query
apiVersion: 1
datasources:
  - name: InfluxDB-3
    type: influxdb
    url: https://influx3-flight.internal:8443
    jsonData:
      version: SQL             # default dialect; individual panels can override
      product: InfluxDB v3
      dbName: observability
    secureJsonData:
      token: ${INFLUX3_TOKEN}

# Programmatic InfluxQL via the HTTP API — for scripts
import requests

r = requests.post(
    "http://influx3.internal:8181/api/v3/query_influxql",
    params={"db": "observability"},
    json={
        "q": 'SELECT mean("usage_pct") FROM "cpu" '
             'WHERE time > now() - 1h GROUP BY time(5m), "namespace"'
    },
    headers={"Authorization": f"Bearer {TOKEN}"},
)
r.raise_for_status()
print(r.json())

Step-by-step explanation.

1. The Grafana panel sends the InfluxQL query verbatim to the IOx querier's InfluxQL endpoint (/api/v3/query_influxql for HTTP; Flight SQL has an equivalent). The querier parses the InfluxQL, translates each construct to its SQL equivalent, then hands the SQL to DataFusion.
2. mean(field) becomes AVG(field). GROUP BY time(5m) becomes GROUP BY date_bin(INTERVAL '5 minutes', time). The double-quoted identifiers ("cpu", "namespace") become unquoted SQL identifiers. Grafana macros ($__timeFilter) are expanded before the query even reaches IOx.
3. Under the hood, this is exactly the same DataFusion plan as the equivalent SQL from the previous example — same pushdowns, same performance, same result shape. The InfluxQL layer is a text-to-text translator.
4. Some InfluxQL features don't have a direct SQL equivalent — SHOW MEASUREMENTS, SHOW TAG KEYS, SHOW SERIES map to system-catalog queries (SELECT DISTINCT table_name FROM ...). Everything visible from Chronograf's schema panel works.
5. When migrating, keep the InfluxQL dashboards running as-is; write new dashboards in SQL. Over time, backfill the old dashboards to SQL only if there's a specific reason (advanced windowing, joins, CTEs — features InfluxQL doesn't have).

Output.

Aspect	InfluxQL path	SQL path
User writes	InfluxQL	SQL
Querier parses	InfluxQL parser	SQL parser
Intermediate plan	DataFusion logical plan	DataFusion logical plan
Physical execution	identical	identical
Result shape	identical	identical
Performance	identical	identical

Rule of thumb. Don't rewrite 300 legacy dashboards to SQL on day one of migration — the InfluxQL shim handles them. Write new dashboards in SQL to build muscle memory in the modern dialect; backfill only where SQL unlocks a feature (window functions, joins, CTEs) that InfluxQL can't express.

Worked example — Flight SQL from a Python analytics script

Detailed explanation. A data scientist wants to pull the last 24 hours of cpu data into a Polars DataFrame for feature engineering. The obvious approach — pull via HTTP + JSON — would serialise 10 GB of numbers as strings; the Flight SQL path is zero-copy Arrow. Walk through the Python client.

• Query. SELECT time, pod, namespace, usage_pct FROM cpu WHERE time > now() - '24h'.
• Client. Python with adbc_driver_flightsql returning PyArrow batches; convert to Polars via pl.from_arrow.
• Volume. ~10 M rows, ~250 MB Arrow-encoded.

Question. Write the Python analytics script and quantify the bandwidth / latency savings vs HTTP + JSON.

Input.

Aspect	HTTP + JSON	Flight SQL
Wire format	JSON text	Arrow IPC (binary)
Rows per second (10 M rows)	~50 K r/s (client parse-bound)	~5 M r/s
Bandwidth (10 M rows)	~2 GB	~250 MB
Client CPU	~4 vCPU on parse	~0.3 vCPU
Result: Polars DataFrame time	~200 s	~5 s

Code.

# High-throughput analytics pull via Flight SQL
import adbc_driver_flightsql.dbapi as flight_sql
import polars as pl
import pyarrow as pa

INFLUX3_FLIGHT_URI = "grpc+tls://influx3-flight.internal:8443"
TOKEN              = "..."
DB                 = "observability"

def pull_last_24h_cpu() -> pl.DataFrame:
    """Return the last 24 h of cpu measurements as a Polars DataFrame."""
    with flight_sql.connect(
        uri=INFLUX3_FLIGHT_URI,
        db_kwargs={
            "adbc.flight.sql.authorization_header": f"Bearer {TOKEN}",
            "adbc.flight.sql.rpc.call_header.database": DB,
        },
    ) as conn:
        with conn.cursor() as cur:
            cur.execute("""
                SELECT time, pod, namespace, usage_pct
                FROM   cpu
                WHERE  time >= now() - INTERVAL '24 hours'
                ORDER  BY time
            """)
            # fetch_arrow_table returns an Arrow table (streamed batches)
            table: pa.Table = cur.fetch_arrow_table()

    # Zero-copy Arrow → Polars conversion
    return pl.from_arrow(table)


# Use the DataFrame for feature engineering
df = pull_last_24h_cpu()
print(f"rows = {len(df)}, columns = {df.columns}")

# Sample transformation — rolling 5-minute mean per pod
features = (
    df.sort("time")
      .with_columns([
          pl.col("usage_pct").rolling_mean("5m", by="time").over("pod").alias("cpu_5m_avg"),
      ])
)

Step-by-step explanation.

1. adbc_driver_flightsql.dbapi.connect() opens a gRPC connection to the querier's Flight SQL endpoint. TLS + bearer-token auth via the connection headers; the database is set as a call header so all subsequent queries target it.
2. cur.execute(sql) sends the SQL over the Flight SQL protocol. The querier begins executing and returns an "endpoint" that the client can DoGet — this is the streaming result path, so large result sets don't materialise in memory server-side.
3. cur.fetch_arrow_table() streams the Arrow IPC batches back and assembles them into a PyArrow Table. Every batch is already in Arrow columnar layout — no parsing, no type inference, no format conversion.
4. pl.from_arrow(table) is a zero-copy handoff — Polars shares the Arrow buffers with PyArrow without copying. The DataFrame is ready for feature engineering in Polars' vectorised engine within milliseconds of the last batch arriving.
5. The performance delta comes from avoiding two full data conversions: server-side JSON encoding (Arrow → JSON strings) and client-side JSON parsing (JSON strings → Python objects → columnar array). With Flight SQL, the on-wire bytes are the in-memory bytes.

Output.

Aspect	HTTP + JSON	Flight SQL
Wall-clock pull time (10 M rows)	~200 s	~5 s
Peak memory (client)	~5 GB (Python objects)	~500 MB (Arrow)
Network bytes (10 M rows)	~2 GB	~250 MB
Client CPU during pull	~4 vCPU	~0.3 vCPU
Ready for Polars analysis	yes (after conversion)	yes (zero-copy)

Rule of thumb. For any programmatic analytics pull > 100 K rows, prefer Flight SQL over HTTP. The 5-50× speedup is not marginal; it's the difference between "sync overnight" and "sync while I get coffee."

Senior interview question on querying

A senior interviewer might ask: "You have 300 Grafana dashboards on InfluxQL, a new data-science team wants Polars-based feature engineering on 24-hour slices, and platform engineers want to run ad-hoc SQL against the last 90 days. Design the query-side architecture: dialect strategy per audience, Flight SQL vs HTTP endpoints, EXPLAIN discipline for slow queries, and how to keep Grafana snappy while a batch analytics query hammers the querier."

Solution Using dialect-per-audience, Flight SQL for programmatic, dedicated querier pools, and EXPLAIN-driven tuning

# 1. Split querier pool into "interactive" (dashboards) + "analytics" (batch)
querier:
  pools:
    - name: interactive           # Grafana + InfluxQL + fast SQL
      replicas: 6
      resources: { cpu: 8, memory: 32Gi }
      object_store_cache: 300Gi
      max_concurrent_queries: 50
      query_timeout_seconds: 30
      allowed_dialects: [sql, influxql]

    - name: analytics             # Polars + heavy ad-hoc SQL
      replicas: 3
      resources: { cpu: 16, memory: 128Gi }
      object_store_cache: 1Ti
      max_concurrent_queries: 10
      query_timeout_seconds: 900
      allowed_dialects: [sql]

  routing:
    # Route by client-declared dialect + query hint
    - match: { header: "X-IOx-Pool", value: "analytics" }
      pool: analytics
    - default: interactive

# 2. Grafana datasource — pinned to interactive pool
apiVersion: 1
datasources:
  - name: InfluxDB-3-Grafana
    type: influxdb
    url: https://influx3-flight.internal:8443
    jsonData:
      version: SQL
      product: InfluxDB v3
      dbName: observability
      httpHeaderName1: "X-IOx-Pool"
    secureJsonData:
      httpHeaderValue1: "interactive"
      token: ${INFLUX3_TOKEN}

# 3. Data-science client — pinned to analytics pool
import adbc_driver_flightsql.dbapi as flight_sql
import polars as pl

def analytics_conn():
    return flight_sql.connect(
        uri="grpc+tls://influx3-flight.internal:8443",
        db_kwargs={
            "adbc.flight.sql.authorization_header": f"Bearer {TOKEN}",
            "adbc.flight.sql.rpc.call_header.database": "observability",
            "adbc.flight.sql.rpc.call_header.x-iox-pool": "analytics",
        },
    )

-- 4. EXPLAIN discipline for slow queries — the on-call runbook step 1
EXPLAIN
SELECT date_bin('1m', time) AS bucket,
       pod,
       AVG(usage_pct)        AS avg
FROM   cpu
WHERE  time >= '2026-07-01' AND time < '2026-07-02'
  AND  namespace = 'prod'
GROUP  BY 1, 2;

-- Look for:
--   file_groups = <small number>           (partition pruning worked)
--   projection = [time, pod, namespace, usage_pct]  (only 4 cols)
--   predicate = time >= AND time <         (predicate pushdown)
--   pruning_predicate = ...                (row-group pruning)

Step-by-step trace.

Concern	Configuration	Reasoning
Dashboard dialect	InfluxQL (legacy) + SQL (new)	dual dialect via one datasource
Analytics dialect	SQL only	Polars integration
Programmatic protocol	Flight SQL	zero-copy for large pulls
Pool separation	interactive vs analytics	prevent OLAP from blocking dashboards
Concurrency cap	50 (int) vs 10 (an)	dashboards need many small queries
Timeout	30 s vs 900 s	analytics is patient; dashboards aren't
Cache size	300 GB vs 1 TB	analytics touches wider history
Slow-query discipline	mandatory EXPLAIN before tuning	pushdowns tell you the story

After deployment, Grafana panels render in < 200 ms because the interactive pool is never blocked by a heavy analytics query; the data-science team pulls 10 M rows in 5 s via Flight SQL; ad-hoc SQL from platform engineers goes to the analytics pool and doesn't degrade dashboards. InfluxQL dashboards keep working unchanged.

Output:

Metric	Value
Interactive P95 latency	~180 ms
Analytics 10 M-row pull	~5 s
Grafana concurrent users supported	~200
Analytics concurrent jobs supported	10
Slow-query root-cause time	< 5 min (EXPLAIN diagnoses)
Legacy dashboard breakage	0

Why this works — concept by concept:

• Dedicated querier pools — the analytics workload has fundamentally different resource curves (long queries, wide time ranges, large results) than the interactive workload (many short queries, narrow time ranges, small results). Sharing a pool means one bad analytics query stalls every dashboard. Separation isolates blast radius.
• Flight SQL for programmatic — zero-copy Arrow eliminates the serialisation tax that dominates HTTP-based pulls. For any Python / Polars / Spark / DuckDB integration, this is the default.
• EXPLAIN discipline — the four DataFusion pushdowns (predicate + partition + projection + partial-aggregate) are the difference between 150 ms and 15 s. Every slow query gets an EXPLAIN as step one; if the pushdowns aren't firing, fix the query (add predicates, project fewer columns) before tuning hardware.
• InfluxQL compat shim — legacy dashboards keep working without rewrite; the shim is a translation layer, not a separate engine. New dashboards use SQL; old ones migrate opportunistically.
• Cost — 6 interactive queriers + 3 analytics queriers vs a single undifferentiated pool. Roughly 30% more compute than a monolithic pool but qualitatively better user experience — dashboards stay snappy, analytics stays productive. O(1) per query on the querier side; caching handles the object-store hit for hot data.

Window Functions
Topic — window-functions
Window-function problems on time-series metrics

Practice →

SQL Topic — sql SQL problems on date_bin and time-bucketing

Practice →

---

5. Migration + when InfluxDB 3 wins + interview signals

Migrating from 2.x TSM/Flux to 3 IOx/SQL — the dual-write bridge, the Flux rewrite tax, and knowing when the switch is worth it

The mental model in one line: there is no in-place upgrade from InfluxDB 2.x to InfluxDB 3 — the TSM storage engine and the IOx storage engine share no on-disk format, so migration is always a side-by-side deployment, a dual-write bridge while dashboards are validated, a history backfill to move the retention window that pre-dates the cutover, and a cutover once query traffic is green — and this multi-quarter operational reality is often the deciding factor in whether a team should adopt IOx or wait for their next replatforming cycle. Every senior migration plan starts with the honest question: does the cardinality / retention / cost pain justify a quarter of migration work?

The four-phase migration playbook.

• Phase 1 — side-by-side deploy. Stand up an IOx cluster (Enterprise or Cloud) alongside the existing 2.x. Do not touch the 2.x cluster; it stays authoritative for now. Create databases + retention policies on IOx that mirror the 2.x buckets.
• Phase 2 — dual-write bridge. Update every LineProtocol writer (Telegraf, Fluent Bit, custom scripts) to POST both to 2.x (existing) and 3 (new). Continue this for at least one full retention window's worth of data so that when you cut over queries, IOx already has the history.
• Phase 3 — history backfill. For any pre-existing data older than the dual-write start, export from 2.x (via influx_inspect export or a Flux range() → to() job that writes to an intermediate CSV / Parquet), then bulk-load into IOx via Flight SQL or LineProtocol. This is optional if you can tolerate a shorter effective history on IOx.
• Phase 4 — cutover. Repoint Grafana / alert engines / consumers from 2.x to 3. Keep 2.x running for a rollback window (7-30 d). Delete 2.x after the rollback window closes.

The Flux rewrite tax — what actually has to change.

• Simple Flux → SQL. from(bucket) |> range(start: -1h) |> filter(fn: (r) => r._measurement == "cpu") |> aggregateWindow(every: 5m, fn: mean) becomes SELECT date_bin('5m', time), AVG(value) FROM cpu WHERE time > now() - '1h' GROUP BY 1. Tractable — scripted translation is possible for most cases.
• Medium Flux. Joins across measurements, pivot() transforms, custom map() — needs manual review. SQL can express all of it but the shape is different.
• Hard Flux. Custom Flux packages, Kapacitor TICKscripts, alert rules with side effects (http.post(), slack.message()) — these were doing things that don't belong in a query engine. Rewrite as external orchestration (Airflow, Prefect, a serverless function).
• The estimate. Budget ~1-3 engineer-days per non-trivial Flux script. A team with 200 Flux scripts is looking at ~4-12 engineer-weeks of rewrite work; this is often the dominant migration cost.

When InfluxDB 3 wins — the four scenarios.

• High cardinality. Any workload with > 4 M distinct series is a slam-dunk for IOx. This is where TSM's series-file cliff hits hardest.
• Unlimited retention with cost. Workloads that want 1+ year retention at high resolution — IoT industrial telemetry, compliance-driven audit trails — become tractable at object-storage prices.
• SQL-first stack. Teams that use dbt, Superset, Metabase, or Snowflake elsewhere want SQL over their telemetry too. IOx's SQL-first design plugs into the rest of the analytics stack.
• Cloud-native operations. Teams on Kubernetes with GitOps + Helm want stateless services on ephemeral storage backed by durable object storage. IOx's architecture matches; TSM's monolith doesn't.

When alternatives win.

• TimescaleDB — when the workload has heavy relational joins (correlating telemetry with a Postgres customer/user/asset master table). TimescaleDB's hypertables on Postgres give you the join engine for free.
• QuestDB — when the workload is ingest-first at extreme rates on a single node (financial tick data, HFT logging), and cardinality is bounded. QuestDB's per-column-file storage and SIMD ingest beat IOx on raw ingest throughput per node.
• ClickHouse — when the workload is general-purpose analytics (event data, product metrics, user telemetry) rather than time-series-native. ClickHouse's engine is more general; IOx's is more specialised.
• VictoriaMetrics — when the workload is Prometheus-remote-write metrics with well-bounded cardinality. VictoriaMetrics is optimised for that specific shape and often cheaper.
• Grafana Mimir / Cortex / Thanos — when the workload is purely Prometheus-federated observability and you already run Prometheus everywhere. Ecosystem gravity matters.

Senior interview signals — the six things they listen for.

• Names IOx separation of concerns. "Router / ingester / compactor / querier + catalog" — five words that signal you understand the architecture.
• Names DataFusion + Parquet role. "DataFusion is the query engine; Parquet is the storage format; both are Arrow-native" — three sentences that signal you've read the design docs.
• Names object-storage cost model. "Storage is S3 at ~$0.023/GB/month; queries pay per-GET on cache miss" — signals you've thought about the economics.
• Names cardinality reasoning correctly. "Tags are dictionary-encoded Parquet columns; no series file; cardinality is bounded by object storage cost" — signals you understand the structural change.
• Names migration reality honestly. "No in-place upgrade; dual-write bridge; Flux rewrite tax at ~1-3 days per non-trivial script" — signals you've done a migration, not just read about one.
• Names when NOT to use InfluxDB 3. "Relational-join-heavy → Timescale; ingest-first single-node → Quest; general analytics → ClickHouse" — signals architectural maturity.

Common architecture-review red flags.

• Deploying Core in production for multi-user workloads (Core is single-node OSS; Enterprise or Cloud is what production wants).
• Skipping the catalog Postgres for "simplicity" (SQLite works but only for Core; Enterprise needs Postgres).
• Sizing the querier without cache (cache-miss latency is 10-100× cache-hit).
• Retention set to INF without a cost budget (unlimited means unlimited bill).
• Continuing to build new dashboards on Flux (it's deprecated; use SQL).

Worked example — the dual-write bridge script

Detailed explanation. The canonical dual-write pattern for a 2.x → 3 migration: a small library sits between the existing writer and the InfluxDB endpoint, POSTs to both 2.x and 3, treats 2.x as authoritative (fail on error), and treats 3 as best-effort (log and continue on error). Walk through the wrapper.

• Contract. Existing writer code is unchanged; the library is a drop-in replacement for the InfluxDB HTTP client.
• 2.x authoritative. Any 2.x error fails the write; 3 errors are logged, never fail.
• Observability. Every dual-write increments Prometheus counters for both endpoints; drift is visible.

Question. Write the dual-write library and the Prometheus metrics.

Input.

Aspect	2.x endpoint	3 endpoint
Path	POST /api/v2/write	POST /api/v3/write_lp
Auth	Token	Bearer
Success code	204	204
Error behaviour	fail write	log + continue
Metrics	count success/error	count success/error/drift

Code.

# dual_write.py — drop-in replacement for the 2.x InfluxDB client
import logging
import requests
from prometheus_client import Counter

log = logging.getLogger("dual_write")

INFLUX2_URL = "http://influx2.internal:8086/api/v2/write"
INFLUX3_URL = "http://influx3.internal:8181/api/v3/write_lp"

INFLUX2_TOKEN = "..."
INFLUX3_TOKEN = "..."

c_ok_2   = Counter("dual_write_2x_ok_total",    "Successful 2.x writes")
c_err_2  = Counter("dual_write_2x_error_total", "Failed 2.x writes")
c_ok_3   = Counter("dual_write_3_ok_total",     "Successful 3 writes")
c_err_3  = Counter("dual_write_3_error_total",  "Failed 3 writes")
c_drift  = Counter("dual_write_drift_total",    "2.x ok but 3 failed")


def dual_write(org: str, bucket_or_db: str, lines: str, precision: str = "ns") -> None:
    """Send a LineProtocol batch to both InfluxDB 2.x and 3.

    2.x is authoritative — any error fails the call.
    3 is best-effort — errors are counted and logged, never re-raised.
    """
    payload = lines.encode("utf-8") if isinstance(lines, str) else lines

    # 1. Write to 2.x (authoritative)
    try:
        r2 = requests.post(
            INFLUX2_URL,
            params={"org": org, "bucket": bucket_or_db, "precision": precision},
            headers={"Authorization": f"Token {INFLUX2_TOKEN}",
                     "Content-Type": "text/plain; charset=utf-8"},
            data=payload,
            timeout=15,
        )
        r2.raise_for_status()
        c_ok_2.inc()
    except requests.RequestException:
        c_err_2.inc()
        raise                                     # 2.x error → fail the call

    # 2. Write to 3 (best-effort)
    try:
        r3 = requests.post(
            INFLUX3_URL,
            params={"db": bucket_or_db, "precision": precision},
            headers={"Authorization": f"Bearer {INFLUX3_TOKEN}",
                     "Content-Type": "text/plain; charset=utf-8"},
            data=payload,
            timeout=15,
        )
        r3.raise_for_status()
        c_ok_3.inc()
    except requests.RequestException as e:
        c_err_3.inc()
        c_drift.inc()
        log.warning("3 write failed while 2.x succeeded: %s", e)

# Prometheus alerts for drift
groups:
- name: influxdb_migration
  rules:
  - alert: DualWriteDriftHigh
    expr: rate(dual_write_drift_total[10m])
        / rate(dual_write_2x_ok_total[10m]) > 0.01
    for: 30m
    annotations:
      summary: "Dual-write drift > 1% over 30 min — 3 endpoint is dropping writes"
  - alert: DualWrite3Down
    expr: rate(dual_write_3_ok_total[5m]) == 0
        and rate(dual_write_2x_ok_total[5m]) > 0
    for: 5m
    annotations:
      summary: "3 endpoint is receiving no writes while 2.x is healthy"

Step-by-step explanation.

1. The wrapper is a drop-in — existing writers call dual_write(...) instead of the direct InfluxDB HTTP call. The signature matches the 2.x semantics (org, bucket, lines, precision) so no other code changes.
2. The 2.x write is authoritative. If it fails, the whole call raises — the caller's retry logic kicks in as before. This preserves the safety invariant: nothing regresses on the existing pipeline.
3. The 3 write is best-effort. If it fails (auth, timeout, backpressure), we increment counters and log a warning, but never re-raise. This means a 3 outage cannot break the existing production write path — a critical property during the multi-quarter dual-write window.
4. Every path increments a Prometheus counter. The c_drift counter fires specifically when 2.x succeeded but 3 failed — this is the migration correctness metric. Its rate should trend toward zero as the 3 deployment stabilises.
5. The DualWriteDriftHigh alert fires if drift exceeds 1% over 30 min — a meaningful sample. DualWrite3Down fires immediately if 3 stops receiving writes while 2.x is healthy — an obvious "the 3 endpoint is down" signal.

Output.

Scenario	2.x behaviour	3 behaviour	Call result
Both OK	204	204	success
2.x fails, 3 OK	5xx	204	fail (2.x is authoritative)
2.x OK, 3 fails	204	5xx	success (3 error logged)
Both fail	5xx	5xx	fail (2.x error propagated)
3 auth broken (start)	204	401	success (log flooded until fixed)

Rule of thumb. Run the dual-write bridge for at least one full retention window before considering cutover. Watch the drift metric — it should be < 0.1% before you cut over query traffic. Never make the new endpoint authoritative during the bridge phase; the whole point is that the migration is safe.

Worked example — the InfluxDB 3 vs TimescaleDB vs QuestDB vs ClickHouse comparison table

Detailed explanation. The senior "when do I pick each engine" question deserves a memorised comparison across the four axes interviewers care about: cardinality, cost, latency, SQL fit, ops burden. Build the table for a hypothetical greenfield telemetry workload.

• Workload. 200 K pts/sec, 15 M distinct series, 90 d retention, ~200 concurrent dashboard users, SQL-native BI stack.
• Question. Which engine wins?

Question. Build the four-column comparison and name the winner per axis.

Input.

Axis	InfluxDB 3	TimescaleDB	QuestDB	ClickHouse
Cardinality ceiling	unlimited (Parquet cols)	high (Postgres indexes)	medium (per-column files)	very high (MergeTree)
Storage tier	object storage (S3)	local NVMe (PG)	local NVMe	local NVMe or S3
Cost per TB / month	~$23 (S3)	~$80 (EBS)	~$80 (EBS)	~$40 (mixed)
Latency (p95 dashboard)	~200 ms	~150 ms	~100 ms	~300 ms
SQL fit	first-class (DataFusion)	first-class (Postgres)	ANSI SQL	ANSI SQL (extended)
Joins	limited	full (Postgres)	limited	full
Ops burden	4 services + catalog	Postgres + hypertable ext	single binary	multi-node cluster
Best fit	high card + object storage	joins + Postgres ecosystem	ingest-first single-node	general analytics

Code.

Engine-selection decision cheat sheet
=====================================

INPUT: cardinality, retention, latency, SQL fit, joins, ops budget

If cardinality > 4 M and retention > 90 d and joins are thin:
    → InfluxDB 3 IOx  (unlimited card + object storage retention)

If joins with a Postgres master table are load-bearing:
    → TimescaleDB     (Postgres hypertables; joins for free)

If ingest > 500 K pts/sec on a single node and cardinality bounded:
    → QuestDB         (per-column-file storage; SIMD ingest)

If general-purpose analytics (event / product / user data):
    → ClickHouse      (MergeTree; more general than time-series-only)

If Prometheus-remote-write and cardinality bounded:
    → VictoriaMetrics (specialised for Prom shape; cheaper)

If already-on-Prometheus and want federation:
    → Grafana Mimir / Thanos / Cortex (Prom ecosystem gravity)

-- Same query, different engines — the SQL is nearly identical
-- InfluxDB 3 (DataFusion):
SELECT date_bin('5m', time), AVG(usage_pct) FROM cpu
WHERE time > now() - INTERVAL '1 hour' GROUP BY 1;

-- TimescaleDB:
SELECT time_bucket('5m', time), AVG(usage_pct) FROM cpu
WHERE time > now() - INTERVAL '1 hour' GROUP BY 1;

-- QuestDB (SAMPLE BY is Quest-specific):
SELECT time, AVG(usage_pct) FROM cpu
WHERE time > dateadd('h', -1, now()) SAMPLE BY 5m;

-- ClickHouse (toStartOfInterval):
SELECT toStartOfInterval(time, INTERVAL 5 MINUTE), AVG(usage_pct)
FROM cpu
WHERE time > now() - INTERVAL 1 HOUR GROUP BY 1;

Step-by-step explanation.

1. The comparison table forces the tradeoffs into a single view. On our hypothetical 200 K pts/sec + 15 M series + 90 d workload, cardinality alone eliminates TimescaleDB (it can handle 15 M series but the Postgres index maintenance cost is meaningful) and QuestDB (its per-column file model degrades past a few million series).
2. Cost per TB per month strongly favours IOx because of the S3 storage tier. At 90 d × 200 K pts/sec × ~200 bytes/point after compression, we're looking at ~15 TB. S3 at $23/TB is ~$350/mo; local NVMe at $80/TB is ~$1200/mo. Compounding over multi-year retention becomes decisive.
3. Latency is a wash for most dashboard queries — all four engines return 5-minute-bucketed aggregations in a few hundred ms. QuestDB wins the tightest single-node latency race; IOx has the most consistent latency because DataFusion's optimiser handles skew well; ClickHouse is slowest on time-only workloads because MergeTree is general-purpose.
4. SQL fit is comparable across IOx / Timescale / QuestDB / ClickHouse — all four speak SQL. IOx's DataFusion has the most modern optimiser but the smallest dialect. Timescale has the most complete SQL (it's Postgres). QuestDB and ClickHouse have their own idiomatic time functions (SAMPLE BY, toStartOfInterval).
5. Ops burden separates the field. IOx needs four services + Postgres catalog + object storage — most complex. Timescale needs Postgres + hypertable extension — moderate. QuestDB is a single binary — simplest. ClickHouse needs a cluster for HA — most operationally intensive at scale.

Output.

Axis winner	Engine	Why
Cardinality > 4 M	InfluxDB 3	no series-file cliff
Storage cost	InfluxDB 3	S3 vs local NVMe
Joins with Postgres	TimescaleDB	Postgres native
Single-node ingest	QuestDB	SIMD + per-column files
General analytics	ClickHouse	MergeTree flexibility
Prometheus-remote-write	VictoriaMetrics	specialised
Kubernetes-native ops	InfluxDB 3	stateless services
Ecosystem gravity (Prom)	Mimir / Thanos / Cortex	native Prom

Rule of thumb. The engine choice is a function of (cardinality × retention × cost × SQL fit × joins × ops burden) — six axes. Print the comparison table; walk the axes in an interview; the winner falls out. Never pick based on "which is trendy."

Worked example — the senior interview-signals cheat sheet

Detailed explanation. The specific vocabulary senior interviewers listen for on IOx. Rehearsing these phrases makes the difference between a competent answer and a senior signal. Walk through the six phrases and when to deploy each.

• When asked about architecture. "IOx separates the write path (router → ingester → WAL → Parquet flush) from the read path (querier → catalog → Parquet → DataFusion), coordinated through a shared Postgres catalog."
• When asked about cardinality. "TSM had an in-memory series file that capped at ~4 M series; IOx replaces it with dictionary-encoded Parquet columns, so cardinality is bounded by object storage cost, not RAM."
• When asked why Rust. "Rust gives deterministic memory, zero GC pauses, and the borrow checker enables the service-per-role decomposition without concurrency footguns."
• When asked about DataFusion. "InfluxData adopted DataFusion rather than writing a new engine; DataFusion contributes the SQL planner, optimiser, and vectorised execution; IOx contributes the Parquet reader and the InfluxQL translator."
• When asked about migration. "There's no in-place upgrade; migration is a dual-write bridge, a history backfill, and a cutover — plus a Flux rewrite tax of ~1-3 engineer-days per non-trivial script."
• When asked when NOT to use IOx. "TimescaleDB wins on relational joins; QuestDB wins on single-node ingest; ClickHouse wins on general analytics; VictoriaMetrics wins on Prometheus federation."

Question. For each of the five common senior IOx interview probes, produce the one-sentence senior answer.

Input.

Probe	Weak answer	Senior answer
"Explain IOx."	"It's the new InfluxDB."	"Rust rewrite; Router/Ingester/Compactor/Querier + Postgres catalog; DataFusion + Parquet + object storage."
"How does cardinality work?"	"Better than before."	"Tags are dictionary-encoded Parquet columns; no series file; bounded by object storage cost."
"Why Rust?"	"Faster."	"Deterministic memory, zero GC pauses, borrow-checker-enabled service decomposition."
"What's DataFusion?"	"The query engine."	"The Arrow-native SQL planner IOx adopted; contributes optimiser + vectorised execution."
"How do I migrate?"	"In-place upgrade."	"No in-place upgrade; dual-write bridge + history backfill + cutover; Flux rewrite tax."

Code.

Senior IOx interview 60-second answer template
==============================================

Line 1 — architecture
  "IOx is InfluxData's Rust rewrite that decomposes the old monolithic
   engine into a router, ingester, compactor, and querier — all stateless,
   coordinated through a shared Postgres catalog, with Parquet on object
   storage as the durable base and DataFusion as the SQL engine."

Line 2 — cardinality
  "Cardinality is unlimited because tags are dictionary-encoded Parquet
   columns rather than entries in an in-memory series file — the old
   TSM engine capped at around 4 million series; IOx is bounded only
   by the object-storage bill."

Line 3 — dialects
  "SQL is the primary dialect via DataFusion; InfluxQL is preserved
   for legacy dashboards; Flight SQL is the Arrow-native binary lane
   for high-throughput programmatic access; Flux is deprecated."

Line 4 — migration
  "There's no in-place upgrade — TSM and IOx share no on-disk format —
   so migration is a side-by-side deploy, a dual-write bridge for at
   least one retention window, a history backfill, and a cutover."

Line 5 — when to pick it
  "IOx wins on high-cardinality plus long retention plus cloud-native
   ops; TimescaleDB wins on relational joins; QuestDB wins on
   single-node ingest; ClickHouse wins on general analytics."

Step-by-step explanation.

1. Line 1 delivers the architecture in one breath. Naming all four services + the catalog + DataFusion + Parquet + object storage in ~40 words signals command of the material. Weak candidates would take three sentences to say the same thing.
2. Line 2 addresses cardinality with the specific numerical anchor ("4 million series") — anchored answers are more credible than vague ones. Naming the mechanism ("dictionary-encoded Parquet columns") shows you understand why, not just that.
3. Line 3 covers all three dialects and their target audiences. The one-line dismissal of Flux ("deprecated") is important — dodging the question would be a red flag.
4. Line 4 is the honest migration paragraph. Saying "no in-place upgrade" up-front pre-empts the "so we can just upgrade?" trap. Naming the three phases (dual-write, backfill, cutover) shows you've done a migration.
5. Line 5 is the competitive-landscape summary. Naming three specific competitors + their strengths shows architectural maturity. Never claim IOx is universally best.

Output.

Section	Word count	Interview grade
Line 1 (architecture)	~40 words	senior signal
Line 2 (cardinality)	~40 words	mandatory
Line 3 (dialects)	~35 words	required
Line 4 (migration)	~35 words	senior signal
Line 5 (when to pick)	~30 words	senior signal
Total	~180 words	~60-second monologue

Rule of thumb. Rehearse the 5-line senior IOx template until it flows in one breath. Every subsequent interviewer question ("tell me more about DataFusion", "walk me through the write path") becomes an expansion of one of the lines — you've pre-loaded the whole conversation.

Senior interview question on migration + engine selection

A senior interviewer might ask: "You inherit an InfluxDB 2.x deployment that has hit max-series-per-database twice, runs 200 Grafana dashboards on InfluxQL and 50 Flux scripts, and costs $8K/month on a 20-node cluster with local NVMe. Design the migration to InfluxDB 3 Cloud Serverless: the dual-write bridge, the Flux rewrite plan, the dashboard compatibility strategy, the cutover criteria, and the rollback plan."

Solution Using a phased dual-write bridge, InfluxQL shim, prioritised Flux rewrite, and observable cutover

# 1. Dual-write library — installed as a drop-in replacement for the 2.x client
#    (see dual_write.py in the earlier example — same shape)

# Optional: gradual traffic ramp — 10% of writers on week 1, 50% on week 2,
# 100% on week 3. Lets the 3 cluster warm up gradually.

# 2. Grafana datasource — dual endpoint during migration
apiVersion: 1
datasources:
  - name: InfluxDB-2x    # existing
    type: influxdb
    url: http://influx2.internal:8086
    jsonData: { version: InfluxQL }
    isDefault: true       # dashboards keep using this initially

  - name: InfluxDB-3     # new
    type: influxdb
    url: https://influx3-cloud-eu.a.influxcloud.io
    jsonData: { version: SQL, product: InfluxDB v3 }
    isDefault: false      # opt-in per dashboard until cutover

# 3. Flux → SQL migration priority queue
#    Score each Flux script by (business criticality × complexity)
#    Rewrite the top decile first

FLUX_SCRIPTS = [
    {"name": "sla_dashboard",       "critical": 5, "complexity": 3},  # 15 → rewrite first
    {"name": "iot_downsampling",    "critical": 4, "complexity": 5},  # 20 → rewrite first
    {"name": "adhoc_ceo_report",    "critical": 3, "complexity": 2},  # 6
    {"name": "engineering_debug",   "critical": 1, "complexity": 4},  # 4 → last
]

for s in sorted(FLUX_SCRIPTS, key=lambda x: -x["critical"] * x["complexity"]):
    print(f"{s['name']}: score={s['critical'] * s['complexity']}")

# 4. Cutover criteria — do not proceed unless ALL are green for 7 days
cutover_criteria:
  - drift_ratio: "< 0.1%"           # dual_write_drift / dual_write_2x_ok
  - iox_p95_query_latency: "< 250 ms"
  - iox_write_error_rate: "< 0.01%"
  - iox_sustained_write_rate: "> 1.2x current 2.x rate"    # headroom check
  - dashboard_parity_sampled: "100% match for 20 sampled panels"
  - rewritten_flux_scripts: "> 80% of critical-tier scripts"

# 5. Rollback plan — always keep 2.x running for 14 days post-cutover
rollback:
  keep_2x_running_for_days: 14
  reverse_dashboard_datasource: "one commit; revert Grafana JSON"
  writer_config_toggle: "single env var flip stops 3 writes"
  data_replay_from_2x: "possible via export → 3 backfill if needed"

Step-by-step trace.

Phase	Duration	Success signal
Side-by-side deploy	1 week	3 cluster provisioned; auth works
Dual-write ramp (10 → 50 → 100%)	3 weeks	drift ratio < 0.1% at 100%
Flux rewrite (critical tier)	4-6 weeks	80% of critical scripts on SQL
Dashboard parity validation	ongoing	20 sampled panels match visually
Cutover	1 hour	Grafana datasource flip; alerts follow
Rollback window	14 days	2.x still writable; dual-write off
2.x decommission	after 14 d	one-way cutover complete

After the migration, the platform runs on IOx Cloud Serverless at ~$2K/month (down from $8K on 2.x self-hosted), cardinality is no longer a bottleneck, retention doubles at half the cost, 80% of Flux scripts are on SQL (the rest continue on the Flux compat shim), and Grafana dashboards keep working across the cutover with no downtime.

Output:

Metric	Before (2.x)	After (3 Cloud)
Monthly cost	~$8K	~$2K
Cardinality ceiling	~4 M	unlimited
Retention (raw)	30 d	90 d
Nodes to manage	20	0 (Cloud managed)
P95 dashboard latency	~300 ms	~200 ms
Flux script maintenance	ongoing	80% migrated to SQL
On-call burden	~10 pages/quarter	~1 page/quarter

Why this works — concept by concept:

• Dual-write bridge — the migration-safety primitive. 2.x stays authoritative until the moment of cutover; a 3 outage during migration cannot break production. The drift metric quantifies migration readiness in one number.
• InfluxQL compat shim — the dashboard-safety primitive. 200 Grafana dashboards keep working unchanged during and after migration; the InfluxQL layer is a translator, not a separate engine.
• Prioritised Flux rewrite — the operational-safety primitive. Business-critical scripts get rewritten first (by criticality × complexity score); low-priority scripts continue on the compat shim indefinitely.
• Cutover criteria + rollback window — the risk-reduction primitives. Six explicit signals must be green for 7 days before cutover; 2.x runs for 14 days after cutover for one-command rollback. Migration is reversible until the rollback window closes.
• Cost — Cloud Serverless at ~$2K/month vs $8K self-hosted 2.x. The migration takes ~2 engineer-months of work; ROI is under 6 months on cost savings alone; the qualitative win (no cardinality drops, no more on-call pages) is on top. O(1) per writer change (dual-write library), O(scripts) for Flux rewrites, O(dashboards) for opt-in datasource migration.

Design
Topic — design
Design problems on database migration + dual-write patterns

Practice →

SQL
Topic — sql
SQL problems on time-series migration and backfill

Practice →

---

Cheat sheet — InfluxDB 3 (IOx) recipes

• The one-line architecture. InfluxDB 3 (IOx) = Rust rewrite + Apache DataFusion (SQL engine) + Apache Parquet (columnar storage) + object storage (S3 / GCS / Azure Blob) durable base + separated router / ingester / compactor / querier services + Postgres catalog holding partition metadata. The four services are stateless; only the catalog and the object store hold durable state.
• Why Rust. Deterministic memory (no GC), zero-pause tail latency (no stop-the-world), borrow-checker guarantees that let InfluxData ship the service-per-role decomposition without concurrency footguns. "Faster" is the marketing answer; the structural answer is memory + concurrency.
• DataFusion role. The Arrow-native SQL/DataFrame engine IOx adopted rather than writing a new one. Contributes parsing → logical plan → physical plan → vectorised execution + four pushdowns (predicate, partition, projection, partial-aggregate). IOx contributes the Parquet reader and the InfluxQL-to-SQL translator on top.
• Parquet + object-storage cost model. ~$0.023/GB/month on S3 Standard vs $0.08+/GB/month for EBS gp3 — about 4× cheaper. Latency trade-off: ~30 ms per S3 GET vs ~0.1 ms per NVMe read, mitigated by aggressive querier-side caching (500 GB local NVMe cache per querier is a good default). 11-nines durability comes free.
• LineProtocol write recipe. POST /api/v3/write_lp?db=<db>&precision=ns with body = 1000-10000 lines of measurement,tag1=v1,tag2=v2 field1=x,field2=y timestamp. Authorization: Bearer <token>. Success = 204 No Content. Retry on 429 (rate-limited) and 503 (backpressure) with exponential backoff; fail hard on 400 (bad LineProtocol).
• Flight SQL ingest recipe. For Arrow-native producers (Spark, Polars, Pandas, DuckDB) or throughput > 100 K pts/sec/client — build a PyArrow RecordBatch, connect via adbc_driver_flightsql.dbapi.connect(uri="grpc+tls://..."), call cursor.adbc_ingest(table_name, data, mode="append"). Zero-copy Arrow IPC over gRPC; ~5-50× throughput vs LineProtocol.
• SQL query template with date_bin. SELECT date_bin(INTERVAL '5 minutes', time) AS bucket, tag, AVG(field) FROM measurement WHERE time >= now() - INTERVAL '1 hour' GROUP BY 1, 2 ORDER BY 1 — the canonical time-bucketed aggregation. Use date_bin_gapfill(...) when the dashboard needs empty buckets rendered as zeros. Wrap every slow query in EXPLAIN to verify pushdowns.
• Migration from 2.x — dual-write pattern. Wrap every InfluxDB HTTP client with a dual_write(...) shim that POSTs to both 2.x (authoritative — fail on error) and 3 (best-effort — log on error). Instrument with Prometheus counters for dual_write_2x_ok, dual_write_3_ok, dual_write_drift. Run for ≥ 1 full retention window before cutover; drift < 0.1% is the cutover signal.
• Pick InfluxDB 3 vs Timescale / Quest / ClickHouse. InfluxDB 3 wins on high cardinality (> 4 M series) + object-storage retention + cloud-native ops. TimescaleDB wins on Postgres joins with a customer/asset master table. QuestDB wins on single-node ingest > 500 K pts/sec with bounded cardinality. ClickHouse wins on general analytics (event data, product metrics) beyond time-series.
• Retention config with unlimited history. CREATE DATABASE <name> WITH RETENTION_PERIOD = '90d' sets 90 days; omit or use 'INF' for unlimited. Downsample via scheduled INSERT ... SELECT date_bin(...) FROM <raw> GROUP BY ... into a coarser-grained table with a longer retention. "Unlimited" means unlimited bill; always budget the storage cost.
• Compactor tuning. target_file_size: 128MB and max_files_per_partition: 500 are safe defaults. Increase target size for large stable workloads to reduce catalog rows; decrease for high-churn tables to reduce per-write amplification. Alert on iox_compactor_pending_files > 20000 — sustained backlog means the compactor is undersized.
• Grafana / Superset integration. Grafana native InfluxDB v3 datasource speaks Flight SQL; per-panel dialect selector supports SQL, InfluxQL, and (legacy) Flux. Superset uses the Flight SQL SQLAlchemy dialect (SQL only). Metabase, DBeaver, IntelliJ all use JDBC Flight SQL driver. dbt has a community adapter for IOx.
• Catalog role (Postgres). Holds databases, tables, schemas, retention policies, partition metadata, Parquet file locations, WAL positions, and compaction bookmarks. Does not hold time-series data — that's in Parquet in object storage. ACID transactions in Postgres serialise catalog mutations across the four services. Enterprise / Cloud need real Postgres; Core defaults to SQLite.
• Cardinality reasoning. Tags become dictionary-encoded Parquet columns; no series file; ingester memory is O(distinct-values-in-current-buffer), not O(total-cardinality). Query cost is O(row-groups-matching-predicate) — pruning stats keep this bounded even at 100 M series. The only real limits are (a) catalog row count if you partition per-value and (b) object storage bill.
• InfluxQL vs SQL vs Flux status. SQL (via DataFusion) is the first-class dialect for new work. InfluxQL is preserved for legacy 1.x / 2.x dashboards — the shim translates to DataFusion behind the scenes; no rewrite needed. Flight SQL is the Arrow-native binary protocol used by Grafana v3 datasource and programmatic clients. Flux is deprecated — absent in Core, compat-only in Enterprise, sunset in Cloud.
• Failure semantics summary. S3 slow → ingester WAL grows until max_wal_bytes_per_partition cap (default 5-10 GB) then backpressures writes; querier cache-hits stay fast, cache-misses stall; compactor backs off with exponential retry. S3 outage → same as slow, just longer. Ingester crash → WAL replay on restart re-creates buffer, no ACKed writes lost. Catalog outage → all four services degrade; treat catalog Postgres as production-critical.

Frequently asked questions

What is InfluxDB 3 in one sentence?

InfluxDB 3 is the ground-up Rust rewrite of the InfluxDB engine — codenamed IOx during development — that keeps the InfluxDB data model (measurements, tags, fields, timestamps) and the LineProtocol wire format intact while replacing the entire storage engine with Apache Parquet on object storage (S3 / GCS / Azure Blob), adopting Apache DataFusion as the SQL query engine, and decomposing the old monolith into four stateless services (router, ingester, compactor, querier) coordinated through a Postgres catalog — all of which combined removes the cardinality cliff that made 1.x / 2.x TSM engines untenable past a few million series and makes retention bounded only by the object-storage bill. The influxdb 3 marketing shorthand refers to this engine across the three editions (Core OSS, Enterprise self-hosted, Cloud Serverless managed); the underlying architecture is the same in all three.

Why did InfluxData rewrite InfluxDB in Rust?

Three structural reasons and one operational one. Structurally, Rust gives deterministic memory management with zero GC pauses — critical for the tail-latency SLOs modern observability platforms sign; the Go + TSM engine's stop-the-world GC pauses used to spike P99 query latency by 50-500 ms during compactions, which is unacceptable when Grafana panels refresh every 10 seconds. Concurrency-wise, Rust's borrow checker makes the service-per-role decomposition (router / ingester / compactor / querier) tractable — the same decomposition in Go would be a concurrency-bug minefield, and the monolithic 2.x process couldn't scale each role independently. Ecosystem-wise, Rust hosts Apache Arrow + Apache DataFusion + Apache Parquet as first-class libraries, which means InfluxData could adopt the columnar analytics stack (rather than reinvent it) and got SQL, window functions, joins, predicate pushdown, and vectorised execution "for free" from DataFusion. Operationally, the rewrite let InfluxData ship a truly cloud-native architecture — stateless services on ephemeral disk backed by durable object storage — that Kubernetes + GitOps + Helm teams can operate the same way they operate every other modern OLAP.

How does InfluxDB 3 achieve unlimited cardinality?

By eliminating the in-memory series file that was the 1.x / 2.x TSM engine's fundamental bottleneck. In TSM, every unique (measurement, tag-set) combination consumed a series-file entry (~50 bytes plus indexing overhead); past ~4 M series per shard the engine would OOM, slow down catastrophically, or refuse new writes via max-series-per-database. In IOx, tags are represented as dictionary-encoded Apache Parquet columns — a new tag value writes a single dictionary entry (~10 bytes) plus a small integer reference per row. Ingester memory pressure is O(distinct-values-in-current-buffer), which stays bounded because the buffer flushes every 60 seconds. Query cost is O(row-groups-matching-predicate) — the DataFusion optimiser uses per-row-group min/max statistics to prune row groups, so even at 100 M series a well-pruned query touches only a few thousand row groups. The catalog holds partition metadata (one row per Parquet file, roughly), which grows linearly with retention × distinct-partitions but is easily bounded by choosing a bucketed partition scheme (hash(sensor_id, 1024) × day(time)). The end result: cardinality is bounded only by the object-storage bill, which is roughly 4× cheaper than the local NVMe TSM used to require.

Can I upgrade in-place from InfluxDB 2.x to 3?

No. The 1.x / 2.x TSM storage engine and the IOx storage engine share no on-disk format — TSM uses proprietary shard / series / TSI files on local disk; IOx uses Apache Parquet on object storage plus a Postgres catalog. There is no in-place upgrade path, no influxd upgrade command, no format converter. Migration is always a side-by-side deployment — stand up a new IOx cluster (Enterprise or Cloud) alongside the existing 2.x, wire up a dual-write bridge in every LineProtocol writer so both endpoints receive writes for at least one full retention window (typical: 30-90 days), backfill any pre-existing history via export from 2.x (influx_inspect export or a Flux range() → to() job) plus bulk-load into IOx via Flight SQL or LineProtocol, then cut over query traffic (Grafana datasource repointing, alert-rule migration). Keep 2.x running for a rollback window of 7-30 days after cutover, then decommission. The migration itself is straightforward operationally; the Flux rewrite tax is often the dominant cost — budget 1-3 engineer-days per non-trivial Flux script, plus custom Kapacitor TICKscripts and Chronograf dashboards. Teams with 100+ Flux scripts should expect a multi-quarter migration; teams with clean SQL-first pipelines can migrate in weeks.

InfluxDB 3 vs TimescaleDB vs ClickHouse — when do I pick each?

Six axes decide: cardinality, retention cost, latency, SQL fit, joins, ops burden. InfluxDB 3 wins on high cardinality (> 4 M series) plus long retention (> 90 d) plus cloud-native ops (stateless services on object storage) — the classic profile is Kubernetes observability, IoT industrial telemetry, or a multi-tenant SaaS metrics service. TimescaleDB wins when the workload has heavy relational joins (correlating time-series with a Postgres customer / asset / user master table) — TimescaleDB is a Postgres extension, so joins are free and mature; if your telemetry query pattern is SELECT metric.time, metric.value, customer.tier FROM metric JOIN customer ON ..., Timescale is the answer. ClickHouse wins on general-purpose analytics (event data, product metrics, user telemetry) where the workload extends beyond pure time-series into arbitrary joins, arbitrary aggregations, and heavy analytical SQL — ClickHouse's MergeTree is more general than IOx's Parquet-on-object-storage. Not on the shortlist: QuestDB (single-node ingest-first workloads with bounded cardinality — HFT tick data, financial telemetry), VictoriaMetrics (Prometheus-remote-write with well-bounded cardinality), Grafana Mimir / Thanos / Cortex (Prometheus federation with strong ecosystem gravity). Never pick based on "which is trendy" or "which we already run" — pick on the six axes.

Is Flux dead in InfluxDB 3?

Effectively yes for new work. Deprecation status: InfluxDB 3 Core does not ship Flux at all — the OSS binary has no Flux runtime. InfluxDB 3 Enterprise ships a Flux compatibility layer for 2.x migration but does not add new Flux features; the compat layer's SLA is "best-effort translation." InfluxDB 3 Cloud has a documented Flux sunset timeline; new Cloud instances default to SQL-only. Migration reality: simple Flux scripts (single range → filter → aggregateWindow → yield chains) translate cleanly to SQL — SELECT date_bin('5m', time), AVG(value) FROM measurement WHERE time > now() - INTERVAL '1 hour' GROUP BY 1. Medium Flux (joins across measurements, pivot() transforms, custom map() functions) needs manual review. Hard Flux (custom packages, Kapacitor TICKscripts, alert rules with side effects like http.post() or slack.message()) needs external rewriting — the side-effect behaviour belongs in an orchestrator (Airflow, Prefect, a serverless function), not a query engine. The upshot: any new dashboard, alert rule, or ETL pipeline in InfluxDB 3 should be written in SQL, using date_bin, date_bin_gapfill, window functions, and DataFusion aggregates. Flux is preserved to keep the migration door open, not to invite new development.

Practice on PipeCode

• Drill the SQL practice library → for the time-series, date_bin, aggregation, and pushdown-aware SELECT problems senior interviewers love.
• Rehearse the aggregation practice library → for the time-bucketed AVG / MAX / percentile patterns that dominate InfluxDB 3 dashboards.
• Sharpen the window-functions practice library → for lag, lead, first_value, last_value, and rolling-window rate calculations that InfluxQL couldn't express.
• Practice the design practice library → for the distributed-time-series architecture, dual-write migration, and querier-pool separation scenarios that IOx interviews probe.
• Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the four-pillar (Rust + DataFusion + Parquet + object storage) decision matrix against real graded inputs.

Lock in influxdb 3 muscle memory

Docs explain features. PipeCode drills explain the decision — when IOx's unlimited cardinality actually matters, when DataFusion's pushdowns collapse a slow query to 150 ms, when object-storage retention wins the cost argument, when the dual-write bridge is safer than a big-bang migration, when TimescaleDB or ClickHouse is the honest answer instead. Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the production trade-offs senior data engineers actually face.

Practice SQL problems →
Practice design problems →