DEV Community

Cover image for Why My ClickHouse Server Ran Out of Disk (and It Wasn't the Data I Expected)
Mohamed Hussain S
Mohamed Hussain S

Posted on

Why My ClickHouse Server Ran Out of Disk (and It Wasn't the Data I Expected)

#clickhouse #linux #devops #database

A deployment failed with an error that had nothing to do with ClickHouse on the surface.

usermod: /etc/passwd... No space left on device
Enter fullscreen mode Exit fullscreen mode

At first, it looked like a Linux problem. Maybe a Docker problem.

It wasn't either.

The root filesystem was completely full.

Checking Disk Usage

The first step was the obvious one.

df -h
Enter fullscreen mode Exit fullscreen mode 
/dev/sda1   96G   100% Used
Enter fullscreen mode Exit fullscreen mode

100% used. No surprise there, given the error. But that's a symptom, not a cause.

Ruling Out Inode Exhaustion

Before assuming it was actual data filling the disk, I checked whether it was inode exhaustion instead - a different problem that looks similar from the outside.

df -i
Enter fullscreen mode Exit fullscreen mode 
3% inode usage
Enter fullscreen mode Exit fullscreen mode

Not an inode problem. Something was genuinely consuming space.

Finding What Was Actually Consuming Disk

Instead of guessing, I started at the root and walked down.

sudo du -sh /* | sort -hr
Enter fullscreen mode Exit fullscreen mode

/var was consuming almost everything.

sudo du -sh /var/* | sort -hr
Enter fullscreen mode Exit fullscreen mode

/var/lib.

sudo du -sh /var/lib/* | sort -hr
Enter fullscreen mode Exit fullscreen mode

/var/lib/clickhouse.

sudo du -sh /var/lib/clickhouse/* | sort -hr
Enter fullscreen mode Exit fullscreen mode

store.

Five commands, one directory deeper each time. No assumptions, just narrowing the search space until there was nowhere left to hide.

Finding Which Table Actually Owned the Space

store is where ClickHouse keeps its actual data parts, so this didn't tell me what was large, only where. I connected to ClickHouse to ask it directly.

SELECT
    database,
    table,
    formatReadableSize(sum(bytes_on_disk)) AS size
FROM system.parts
GROUP BY database, table
ORDER BY sum(bytes_on_disk) DESC;
Enter fullscreen mode Exit fullscreen mode 
system.text_log   76 GiB
Enter fullscreen mode Exit fullscreen mode

Not application data. Not the tables I actually cared about.

system.text_log - ClickHouse's own internal server log, stored as a table - had quietly grown to 76 gigabytes.

Nobody had configured a retention policy on it. It had just been logging, forever, since the server started.

Deciding Not to Preserve It

This was a development server, so I didn't need to preserve anything. Stop the service, clear the data, move on.

sudo systemctl stop clickhouse-server
Enter fullscreen mode Exit fullscreen mode 
sudo rm -rf /var/lib/clickhouse/*
Enter fullscreen mode Exit fullscreen mode

Checked the disk.

df -h
Enter fullscreen mode Exit fullscreen mode

Still full.

sudo du -sh /var/lib/clickhouse
Enter fullscreen mode Exit fullscreen mode 
87G
Enter fullscreen mode Exit fullscreen mode

The wildcard delete hadn't actually cleared everything. I listed the directory directly instead of trusting the glob.

sudo ls -lah /var/lib/clickhouse
sudo bash -c 'du -sh /var/lib/clickhouse/* | sort -hr'
Enter fullscreen mode Exit fullscreen mode 
87G   store
Enter fullscreen mode Exit fullscreen mode

store was still there. I never fully pinned down why the first wildcard delete skipped it - possibly the server hadn't finished releasing file handles the moment systemctl stop returned. Either way, trusting the first command's exit code without re-checking the actual directory size cost me a step.

Removed it directly.

sudo rm -rf /var/lib/clickhouse/store
Enter fullscreen mode Exit fullscreen mode 
df -h
Enter fullscreen mode Exit fullscreen mode 
84G free
Enter fullscreen mode Exit fullscreen mode

Problem solved, for now. But "for now" wasn't good enough, because the same 96GB disk was still the only place ClickHouse could write to.

The Real Fix: Get ClickHouse Off the Root Disk Entirely

The server had a second, 256GB disk. Unmounted. Unused. The whole time.

lsblk -f
Enter fullscreen mode Exit fullscreen mode 
sda   root filesystem
sdb   ext4   UUID=b16098f7-bd6a-496d-a2f5-86ad19913c7c
Enter fullscreen mode Exit fullscreen mode

Mounted it temporarily to check what was on it.

sudo mkdir -p /mnt/chdisk
sudo mount /dev/sdb /mnt/chdisk
sudo ls -la /mnt/chdisk
Enter fullscreen mode Exit fullscreen mode 
lost+found
Enter fullscreen mode Exit fullscreen mode

Empty. Unmounted it again and made the arrangement permanent instead of temporary.

sudo umount /mnt/chdisk
Enter fullscreen mode Exit fullscreen mode

Configuring the Permanent Mount

Edited /etc/fstab:

UUID=b16098f7-bd6a-496d-a2f5-86ad19913c7c  /var/lib/clickhouse  ext4  defaults  0  2
Enter fullscreen mode Exit fullscreen mode

Moved the old (now-empty) directory out of the way, created a fresh mount point, and mounted it.

sudo mv /var/lib/clickhouse /var/lib/clickhouse.old
sudo mkdir /var/lib/clickhouse
sudo mount -a
sudo systemctl daemon-reload
Enter fullscreen mode Exit fullscreen mode

Verified the mount actually took:

findmnt /var/lib/clickhouse
Enter fullscreen mode Exit fullscreen mode 
TARGET                SOURCE
/var/lib/clickhouse   /dev/sdb
Enter fullscreen mode Exit fullscreen mode

Every read and write to /var/lib/clickhouse now transparently goes to /dev/sdb instead of the root disk. ClickHouse doesn't need to know or care - as far as its own config is concerned, it's still just writing to /var/lib/clickhouse.

Fixed ownership and permissions, since a fresh mount point doesn't inherit them:

sudo chown -R clickhouse:clickhouse /var/lib/clickhouse
sudo chmod 750 /var/lib/clickhouse
Enter fullscreen mode Exit fullscreen mode

Restarted and confirmed:

df -h
Enter fullscreen mode Exit fullscreen mode 
/dev/sda1   84G free
/dev/sdb    mounted on /var/lib/clickhouse
Enter fullscreen mode Exit fullscreen mode

What I'm Changing Going Forward

Clearing text_log fixed the immediate problem. It didn't fix the actual cause - nothing stops it from growing back to 76GB again.

The real fix is giving ClickHouse's system logs a retention policy, the same way you'd expect any log to expire eventually.

<clickhouse>
    <text_log>
        <ttl>event_date + INTERVAL 7 DAY DELETE</ttl>
    </text_log>

    <trace_log>
        <ttl>event_date + INTERVAL 7 DAY DELETE</ttl>
    </trace_log>

    <metric_log>
        <ttl>event_date + INTERVAL 7 DAY DELETE</ttl>
    </metric_log>

    <part_log>
        <ttl>event_date + INTERVAL 7 DAY DELETE</ttl>
    </part_log>

    <query_log>
        <ttl>event_date + INTERVAL 7 DAY DELETE</ttl>
    </query_log>

    <query_thread_log>
        <ttl>event_date + INTERVAL 7 DAY DELETE</ttl>
    </query_thread_log>

    <processors_profile_log>
        <ttl>event_date + INTERVAL 7 DAY DELETE</ttl>
    </processors_profile_log>

    <asynchronous_metric_log>
        <ttl>event_date + INTERVAL 7 DAY DELETE</ttl>
    </asynchronous_metric_log>
</clickhouse>
Enter fullscreen mode Exit fullscreen mode

Dropped in as /etc/clickhouse-server/config.d/system_log_retention.xml, this keeps every system log table capped at 7 days of history, deleted automatically, no manual cleanup required again.

Before and After

The Bigger Lesson

I went in assuming a full disk meant the actual data had grown too fast.

It hadn't. The application data was fine. What had grown out of control was ClickHouse's own internal logging - a system table nobody had put a limit on, quietly consuming more space than the data it was supposed to be observing.

And underneath that, a second problem had been sitting there the whole time: a 256GB disk that was never mounted, on a server that only ever had 96GB to work with.

Neither problem was visible from df -h alone. Getting to the real cause meant walking the filesystem one directory at a time, then asking ClickHouse directly which table actually owned the space.

Isolating database storage onto its own disk, separate from the OS, isn't just a performance habit. It's what stops a logging table from being able to take your entire root filesystem down with it.

Top comments (0)