DEV Community: Vladimir Chemeris

ClickHouse "Too many parts": what to check before you run OPTIMIZE FINAL

Vladimir Chemeris — Tue, 11 Aug 2026 13:59:02 +0000

Disclosure: I build ProbeDeck, an iOS app for monitoring and operating ClickHouse clusters. The SQL
below works in any ClickHouse client. You do not need ProbeDeck to use it.

An insert fails with this message:

Too many parts (N). Merges are processing significantly slower than inserts

The tempting response is OPTIMIZE TABLE ... FINAL. Hold that command. ClickHouse raises this error
when a partition accumulates active parts faster than the server can merge them. A forced merge can
consume the same CPU, disk bandwidth, and free space that the background merge process needs.

Use this order during an incident:

Count active parts by partition and by table.
Read the live delay, throw, and total-part thresholds from the cluster.
Sample active merges twice to see whether the backlog shrinks.
Check disk headroom and, for replicated tables, replica health.
Reduce part creation at the writer or partitioning layer.
Consider a forced merge after you control incoming pressure.

Find the affected partition

parts_to_throw_insert applies to active parts in one partition. A table-wide count can hide the
shape of the problem. Run this first:

The system tables below report data from the node that serves your query. A load balancer can route
you to a healthy replica while another replica holds the backlog. Connect to each node, or list the
configured cluster names with SELECT DISTINCT cluster FROM system.clusters and replace a local
table such as system.parts with clusterAllReplicas('my_cluster', system.parts). Keep
hostName() in the result so you can see which node produced each row.

SELECT
    hostName() AS host,
    database,
    table,
    partition,
    count() AS active_parts,
    countIf(level = 0) AS level0_parts,
    max(level) AS highest_level,
    sum(rows) AS total_rows,
    formatReadableSize(sum(bytes_on_disk)) AS size_on_disk
FROM system.parts
WHERE active
  AND database NOT IN ('system', 'INFORMATION_SCHEMA', 'information_schema')
GROUP BY host, database, table, partition
ORDER BY active_parts DESC
LIMIT 20;

If one partition sits far above the rest, inspect the hot partition and its partitioning key. If you
see level0_parts grow between samples, inspect the writer for frequent small inserts. If the count
is high but spread across healthy partitions, compare the table total with max_parts_in_total
below.

Do not copy a threshold from a blog post. Read the settings on the cluster you are debugging:

SELECT
    hostName() AS host,
    name,
    value,
    changed
FROM system.merge_tree_settings
WHERE name IN (
    'parts_to_delay_insert',
    'parts_to_throw_insert',
    'max_parts_in_total',
    'max_avg_part_size_for_too_many_parts'
)
ORDER BY name;

system.merge_tree_settings gives you the server-level value. A table can override it in its own
SETTINGS clause. Check the table definition before you treat the result as the effective limit:

SHOW CREATE TABLE analytics.events;

The writer can also pass settings with the insert query. Run the checks through the same user and
connection path as the writer, then inspect its client configuration. Treat an app or dashboard
threshold as a warning heuristic unless it reads the effective setting.

ClickHouse's current guide to OPTIMIZE FINAL uses 150 active parts as an early-warning heuristic.
Your cluster limit comes from its effective settings. Investigate when two samples show growth or
level-0 parts dominate. A count near parts_to_delay_insert means ClickHouse may throttle inserts
soon; parts_to_throw_insert rejects them. max_avg_part_size_for_too_many_parts can disable the
delay and throw checks when the average part size in the affected partition exceeds its value. The
total table limit still applies.

max_parts_in_total protects the table across all partitions. Check the matching table total:

SELECT
    hostName() AS host,
    database,
    table,
    count() AS active_parts_total
FROM system.parts
WHERE active
  AND database NOT IN ('system', 'INFORMATION_SCHEMA', 'information_schema')
GROUP BY host, database, table
ORDER BY active_parts_total DESC
LIMIT 20;

Check whether merges are making progress

Next, inspect the active merge workload:

SELECT
    hostName() AS host,
    database,
    table,
    partition,
    elapsed,
    round(progress * 100, 1) AS progress_pct,
    num_parts,
    is_mutation,
    merge_type,
    formatReadableSize(total_size_bytes_compressed) AS source_size,
    formatReadableSize(memory_usage) AS memory
FROM system.merges
ORDER BY elapsed DESC;

system.merges shows work in progress, not a queue. An empty result does not prove that merges have
stopped. Read it together with two system.parts samples:

Parts are high and merge progress changes between samples: the cluster is working through a backlog. Reduce incoming pressure and keep measuring.
Parts keep rising while no merge appears for the affected table: inspect disk space, merge-pool pressure, disabled merges, replication state, and recent configuration changes.
Merges run, but new level-0 parts arrive faster: fix the writer. More merge threads will not cure a producer that creates parts without bound.

If your server writes system.part_log, estimate how completed merges changed the part count over
the same window:

SELECT
    hostName() AS host,
    countIf(event_type = 'NewPart') AS new_parts,
    countIf(event_type = 'MergeParts') AS merge_outputs,
    sumIf(length(merged_from), event_type = 'MergeParts') AS merge_inputs,
    new_parts + merge_outputs - merge_inputs AS estimated_net_change
FROM system.part_log
WHERE event_time >= now() - INTERVAL 10 MINUTE
  AND database = 'analytics'
  AND table = 'events'
  AND event_type IN ('NewPart', 'MergeParts')
GROUP BY host;

One MergeParts event can consume several source parts, so raw event counts use different units.
The estimate above ignores downloads, removals, and events that cross the time-window boundary.
Use repeated system.parts snapshots as the primary signal. Replace the example identifiers and
save two or three samples a few minutes apart.

Stalled merges also need disk and replica checks. On self-managed clusters, inspect every server
that stores the affected table:

SELECT
    hostName() AS host,
    name,
    path,
    formatReadableSize(free_space) AS free,
    formatReadableSize(unreserved_space) AS unreserved,
    formatReadableSize(total_space) AS total,
    formatReadableSize(keep_free_space) AS reserved
FROM system.disks
ORDER BY name;

No fixed free-space percentage fits every merge because candidate parts and storage policies differ.
Treat disk as the bottleneck when unreserved_space keeps falling while merges stall, or when the
server log reports space-reservation failures. Check disk saturation in your host monitoring. For
ReplicatedMergeTree tables, inspect replica state as well:

SELECT
    hostName() AS host,
    database,
    table,
    is_readonly,
    is_session_expired,
    queue_size,
    inserts_in_queue,
    merges_in_queue,
    queue_oldest_time,
    absolute_delay
FROM system.replicas
WHERE is_readonly OR is_session_expired OR queue_size > 0
ORDER BY queue_size DESC;

A nonzero queue can be normal. Sample it twice. A read-only replica, an expired session, a growing
queue, or rising absolute_delay points to replication trouble and adds pressure to ClickHouse
Keeper.

Check the writer before tuning the server

Each synchronous insert can create at least one part for every partition touched by its block.
ClickHouse recommends batching at least 1,000 rows per insert, with 10,000 to 100,000 rows as the
better range for many workloads. The right batch size still depends on row width, latency, memory,
and the number of partitions touched. For synchronous ingestion, start near one insert per second
and adjust from measurements.

Check these failure modes:

A producer sends one row or a tiny batch per request.
One insert touches many partitions.
Several writers use incompatible async-insert settings, so ClickHouse cannot combine their data into one buffer.
Materialized views multiply one source insert into parts in several target tables.
Mutations, TTL work, or disk pressure compete with regular merges.

Check partition granularity before you tune merge pools. Run SHOW CREATE TABLE and count the
active partitions:

SELECT
    hostName() AS host,
    database,
    table,
    uniqExact(partition) AS active_partitions,
    count() AS active_parts
FROM system.parts
WHERE active
  AND database NOT IN ('system', 'INFORMATION_SCHEMA', 'information_schema')
GROUP BY host, database, table
ORDER BY active_partitions DESC
LIMIT 20;

A large partition count alone does not prove a bad key. Look for a key that opens a partition per
ID, per tenant, or per timestamp truncated to an hour or less. ClickHouse cannot merge parts across
partition boundaries. Fixing that design requires a replacement table with a coarser partition key
and a controlled data migration.

ClickHouse 26.3 LTS enables async inserts by default. On older versions, enable them on the ingest
path when client-side batching is not practical. On 26.3 and later, verify that the client did not
override the setting. Check it using the same user and connection settings as the writer:

SELECT hostName() AS host, name, value, changed
FROM system.settings
WHERE name IN ('async_insert', 'wait_for_async_insert', 'compatibility')
ORDER BY name;

An older pinned compatibility value can preserve the previous async-insert default after a server
upgrade. Trust the async_insert value returned through the writer's connection instead of the
server version alone.

Keep wait_for_async_insert = 1 when the client needs acknowledgement after the buffer reaches
durable storage. With 0, the server can acknowledge data while it still lives in memory, and the
client will not receive flush errors in the same way.

`OPTIMIZE FINAL` does not fix ingestion

OPTIMIZE TABLE ... FINAL rewrites the parts that exist now. It does not change the rate at which
the writer creates new ones. When ingestion resumes, the exception can return. The forced merge can
also create a large part and consume resources on a cluster that already lacks merge capacity.

Reserve a forced merge for cases where you understand the affected table and partition, have enough
disk and I/O headroom, and have reduced the source of new parts. Raising parts_to_throw_insert has
the same problem: it moves the safety boundary without repairing ingestion.

The durable fixes are upstream:

Batch more rows per insert.
Reduce insert frequency.
Use async inserts where they fit the durability and latency requirements.
Revisit a high-cardinality partition key through a planned migration.
Tune merge resources after measurements show a server-side bottleneck.

During an incident, I run the same checks from the ProbeDeck guide to ClickHouse Too many parts.
ProbeDeck shows raw maximum active-part counts and current merges from an iPhone or iPad. Compare
those counts with the effective settings on your cluster. Monitoring is free. Credentials stay in
the iOS Keychain, and the app has no backend between the device and your cluster.

ClickHouse is a registered trademark of ClickHouse, Inc. ProbeDeck is not affiliated with,
endorsed by, or sponsored by ClickHouse, Inc.

Why iperf3 Says “Bad File Descriptor” and What to Check First

Vladimir Chemeris — Fri, 07 Aug 2026 13:58:34 +0000

Disclosure: I develop an iPerf3 app for Apple platforms and Android. The examples below use the upstream iperf3 CLI and apply to compatible clients and servers.

Suppose iperf3 prints this line:

iperf3: error - unable to write to the control socket: Bad file descriptor

Keep the whole line. Bad file descriptor comes from your operating system. It does not identify the operation that failed. The text before it names the iperf3 stage.

In this case, iperf3 tried to write to its control socket. The operating system rejected the descriptor passed to that write with EBADF.

One line, two sources

iperf3 builds many error strings from two parts:

iperf3 operation: unable to write to the control socket
OS errno: Bad file descriptor

The iperf3 message answers: What was the program doing?

The errno text answers: Why did that system call fail?

The implementation is visible in src/iperf_error.c. For IECTRLWRITE, iperf3 starts with unable to write to the control socket. It then appends strerror(errno) when the failing call supplied an errno.

Some messages have no appended errno. control socket has closed unexpectedly is one example. iperf3 uses it after a control read returns no state byte. On the client, an empty read points to end-of-file from the peer. The server can reach the same message after its timed read returns without data.

What `EBADF` tells you

Bad file descriptor is the text for EBADF. The process passed an invalid descriptor to a system call.

That establishes one local fact: the descriptor was invalid inside the process that printed the error. It does not tell you why the descriptor reached that state. Possible triggers include cleanup after another failure, a descriptor-lifetime bug, or code running after a connection teardown.

A firewall or the remote peer can start a failure sequence by closing a connection. The final EBADF still describes only the local system call that failed at the end of that sequence.

Similar messages point to different stages

Read the iperf3 part before interpreting the suffix:

Message	Failed stage	First check
`unable to write to the control socket: <errno>`	A write to the control socket failed.	Check the previous state exchange and the appended errno.
`unable to receive cookie at server: <errno>`	The server accepted a data-stream socket but could not read its cookie.	Compare the client log and the server errno.
`select failed: <errno>`	The kernel rejected something in a `select()` descriptor set.	Use `--debug` to find the last completed stage.
`control socket has closed unexpectedly`	A control read returned no state byte. No errno is appended.	Compare both logs at the same timestamp.

This table explains why searching only for bad file descriptor produces weak answers. The same suffix can appear after several iperf3 operations.

A debugging sequence that keeps the evidence

Start with the process that printed the line. Then collect the matching evidence from the other side.

Save the complete client and server commands.
Save the full output from both processes.
Record iperf3 --version on both machines. iperf2 and iperf3 do not interoperate.
Classify the failed stage from the iperf3 message.
Interpret errno only after you know that stage.
Repeat the test with debug logs:

iperf3 -s --debug >server.log 2>&1
iperf3 -c 192.168.1.10 --debug >client.log 2>&1

Match the last completed state on both sides. If one log stops first, check whether that host slept, the process exited, or the network path changed.

Run a short and a long test when the failure appears late:

iperf3 -c 192.168.1.10 -t 30
iperf3 -c 192.168.1.10 -t 300

A short run that passes and a long run that fails gives you a time-dependent lead. Check idle firewall state, device sleep, Wi-Fi roaming, and process restarts. Treat each as a testable candidate.

If the logs disagree about which side stopped first, capture port 5201 and compare packet timestamps with both logs:

sudo tcpdump -ni any tcp port 5201

Use a real interface name on systems without the any pseudo-interface.

Other suffixes use the same method

The same reading order works for Connection refused, Permission denied, and Address already in use.

For example, unable to connect to server: Connection refused identifies a failed connect attempt. The suffix says the destination rejected that TCP connection. Check the server process, address, and port before changing unrelated socket settings.

The full iperf3 error reference covers busy servers, listener failures, temporary-file permissions, socket buffers, mobile sleep, and long-test failures.

If you want to run the same tests from an iPhone, iPad, or Mac, the iPerf3 app for Apple platforms can act as a client or server.

Measuring real LAN speed from a phone with iperf3

Vladimir Chemeris — Wed, 05 Aug 2026 16:49:27 +0000

Disclosure before we start: I build the native iPerf3 apps for iPhone, iPad, Mac, and Android mentioned near the end. Everything in this post also works with the free iperf3 CLI on two computers. The phone enters the picture when one endpoint has to move around the building.

Speedtest answers one question: how fast is the path from your device to the ISP's nearest test server. Inside a home or office a different question comes up: how fast is the link between the phone and the NAS, between two mesh nodes, between the office switch and the upstairs access point. Traffic between two local devices never leaves the LAN, so an internet speed test cannot see it.

iperf3 measures that link. ESnet maintains it as a from-scratch redesign of the original iperf, and common Linux, BSD, and macOS package repositories carry it.

How iperf3 measures

iperf3 is a client-server tool. One device listens (iperf3 -s), the other connects to it, pushes traffic for a fixed duration, and counts what arrived. Because you control both endpoints, the result describes one specific link instead of a whole path across the internet.

The server side installs in one line on Linux and macOS:

# Debian/Ubuntu
sudo apt install iperf3 && iperf3 -s

# macOS
brew install iperf3 && iperf3 -s

ESnet does not publish or support native Windows binaries. If you need a Windows endpoint, check the upstream installation notes before choosing a third-party build.

Some router and NAS platforms make iperf3 available through their package systems. Availability depends on the model and firmware, so check the package source before treating the appliance as a permanent server.

A five-minute test

Start the server on a wired device and note its LAN IP, for example 192.168.1.10.
From the client, start with the default ten-second test. Use sixty seconds when you need to see changes caused by interference, roaming, or sustained load:

iperf3 -c 192.168.1.10 -t 60

Read the receiver line at the end. That number is the throughput the link sustained.

A healthy wired 1GbE TCP link often sustains about 930 to 950 Mbit/s after protocol overhead. Wi-Fi varies with channel width, spatial streams, interference, access-point configuration, and the phone's radio. Compare the same phone and settings across locations instead of judging a result from the Wi-Fi generation label alone. If a wired link reports far below its expected rate, check the negotiated link speed, cable, switch port, USB adapter, and CPU load on both endpoints.

Now walk. Run the same test from the far bedroom, the garage, the conference room. The room-to-room gap is the thing a signal-strength bar hides.

Which output fields to read

Throughput is the headline number: application data delivered per second, usually shown in Mbit/s or Gbit/s.

Retransmits (TCP) count segments the sender had to transmit again. A clean wired LAN should produce few retransmits. A rising count on repeated runs gives you a reason to isolate the cable, port, queueing, or wireless hop; the counter alone does not identify the cause.

Jitter and packet loss appear in UDP mode. Set the target below the link's measured TCP capacity, then raise it in steps. For example:

iperf3 -c 192.168.1.10 -u -b 100M -t 60

Packet loss and jitter affect calls and games before raw throughput becomes the obvious problem. Your acceptable limits depend on the application, codec, and traffic pattern.

Traps that produce wrong numbers

Both endpoints on Wi-Fi. You measure the weaker of two radio links plus their contention for airtime. Put the server on Ethernet and test the wireless hop you care about.
USB Ethernet adapters. USB 2.0 adapters often top out around 300 to 350 Mbit/s in practice. Check the adapter and negotiated link speed before blaming the network.
Short tests. Connection setup, TCP ramp-up, and brief interference can dominate a short result. Use longer runs when you care about stability under sustained load.
A sleeping phone. Some phones reduce background network activity or suspend the test when the screen locks. Keep the app in the foreground and the screen on during a run.

Running iperf3 on the phone itself

The phone is often the endpoint that matters: it is the device the complaint is about, and it is the device you carry from room to room. Options for a native endpoint:

iPhone / iPad / Mac: iPerf3 Client & Server on the App Store. One purchase ($6.99, one-time) covers iPhone, iPad, and Mac. Client and server modes, TCP/UDP, live charts, history, CSV/JSON export, Apple Shortcuts automation.
Android: iPerf3 TCP/UDP Client & Server on Google Play ($3.99, one-time). Same idea: both modes, advanced parameters, export.

Both apps use the iperf3 protocol, so a phone in server mode can accept tests from compatible iperf3 CLI versions and vice versa. If two endpoints fail to negotiate, compare their iperf3 versions before debugging the network.

Two phones make a portable end-to-end test kit: put one in server mode and use the other as the client. Because both endpoints share Wi-Fi airtime, this setup cannot isolate mesh backhaul performance. Use a wired server when you need to test one wireless hop or compare rooms.

Where to go deeper

iperf3 upstream project (source, releases, and official documentation)
How to test LAN speed, step by step
Why Speedtest and iperf3 answer different questions

Questions about a weird result are welcome in the comments; I read all of them.