DEV Community

Cover image for Day 88 - Upgrading to ClickHouse® 26.3 LTS: A Step-by-Step Safe Guide
Kanishga Subramani
Kanishga Subramani

Posted on

Day 88 - Upgrading to ClickHouse® 26.3 LTS: A Step-by-Step Safe Guide

#clickhouse #devops #analytics #database

Introduction

Upgrading a database is much more than installing a newer version. It requires careful planning to preserve data integrity, maintain application compatibility, minimize downtime, and ensure that production workloads continue to run smoothly.

ClickHouse® 26.3 is a Long-Term Support (LTS) release that introduces several significant improvements, including SQL standards compliance updates, query optimizer enhancements, performance improvements, and new features. Because LTS releases receive extended maintenance and bug fixes, they are generally the recommended versions for production deployments.

However, every major upgrade introduces behavioral changes that should be validated before moving to production. A structured upgrade process significantly reduces the risk of unexpected issues and provides a clear rollback path if necessary.

In this guide, we'll walk through a production-ready approach to upgrading a ClickHouse® cluster to version 26.3 LTS, covering preparation, installation, validation, rollback planning, and post-upgrade monitoring.

Why Upgrade to ClickHouse® 26.3 LTS?

ClickHouse® 26.3 LTS includes numerous improvements, including:

  • Long-term support and maintenance
  • Better SQL standards compliance
  • Query optimizer improvements
  • Performance enhancements
  • Stability improvements
  • Improved compatibility with modern SQL syntax
  • New features and bug fixes

Although upgrading is recommended, it's equally important to perform the upgrade safely.

Recommended Upgrade Strategy

A production upgrade should always be performed gradually.

Follow this sequence:

  1. Upgrade your staging environment.
  2. Validate applications and workloads.
  3. Upgrade one production replica.
  4. Verify cluster health.
  5. Continue upgrading remaining replicas one by one.
  6. Remove the old version only after complete validation.

Never upgrade every node simultaneously.

Rolling upgrades keep the cluster available while minimizing operational risk.

Pre-Upgrade Checklist

Before touching any production system, verify the current environment.

1. Check the Current Version 

SELECT version();
Enter fullscreen mode Exit fullscreen mode

Example:

version()
25.x.x.x

Record the current version for rollback planning.

2. Backup Configuration Files

Always create backups before upgrading.

sudo cp -r /etc/clickhouse-server \
/etc/clickhouse-server-backup-$(date +%Y%m%d)
Enter fullscreen mode Exit fullscreen mode

Also back up table metadata.

sudo cp -r /var/lib/clickhouse/metadata \
/var/lib/clickhouse/metadata-backup-$(date +%Y%m%d)
Enter fullscreen mode Exit fullscreen mode

3. Verify Replica Health

Before upgrading, every replica should be healthy.

SELECT
    database,
    table,
    replica_name,
    is_readonly,
    active_replicas,
    total_replicas,
    queue_size,
    last_exception
FROM system.replicas;
Enter fullscreen mode Exit fullscreen mode

A healthy cluster should report:

  • is_readonly = 0
  • queue_size = 0
  • active_replicas = total_replicas

If replication is unhealthy, fix those issues before upgrading.

4. Check Running Background Tasks

Avoid upgrading while merges or mutations are still running.

Running merges 

SELECT *
FROM system.merges;
Enter fullscreen mode Exit fullscreen mode

Running mutations 

SELECT *
FROM system.mutations
WHERE is_done = 0;
Enter fullscreen mode Exit fullscreen mode

Wait until all background mutations have completed.

5. Verify Available Disk Space

Ensure sufficient free disk space.

df -h /var/lib/clickhouse
Enter fullscreen mode Exit fullscreen mode

Package installation and temporary files require additional storage during upgrades.

6. Create Data Backups

Although package upgrades preserve data, backups provide an essential recovery point.

Possible options include:

  • BACKUP command
  • Filesystem snapshots
  • Cloud storage snapshots
  • Existing disaster recovery procedures

Never skip backups before upgrading production.

7. Review Async Insert Behavior

ClickHouse® 26.3 enables asynchronous inserts by default.

Applications expecting synchronous acknowledgements should explicitly set:

SET wait_for_async_insert = 1;
Enter fullscreen mode Exit fullscreen mode

Alternatively, disable asynchronous inserts.

SET async_insert = 0;
Enter fullscreen mode Exit fullscreen mode

Review your application's insert logic before upgrading.

8. Stop ClickHouse® Gracefully 

sudo systemctl stop clickhouse-server
Enter fullscreen mode Exit fullscreen mode

Verify the service has stopped.

sudo systemctl status clickhouse-server
Enter fullscreen mode Exit fullscreen mode

Rolling Upgrade Procedure

For replicated clusters, upgrade one node at a time.

For each replica:

  1. Stop ClickHouse®.
  2. Install version 26.3.
  3. Start ClickHouse®.
  4. Validate replica health.
  5. Continue with the next replica.

Example workflow:

Replica 1
Upgrade
Validate

↓

Replica 2
Upgrade
Validate

↓

Replica 3
Upgrade
Validate
Enter fullscreen mode Exit fullscreen mode

This approach maintains cluster availability throughout the upgrade.

Installing ClickHouse® 26.3

The following examples use Debian/Ubuntu packages.

Install the New Version 

sudo apt update

sudo apt install -y \
clickhouse-server=26.3.* \
clickhouse-client=26.3.*
Enter fullscreen mode Exit fullscreen mode

For a specific patch release:

sudo apt install -y \
clickhouse-server=26.3.x.x \
clickhouse-client=26.3.x.x \
clickhouse-common-static=26.3.x.x
Enter fullscreen mode Exit fullscreen mode

Start ClickHouse® 

sudo systemctl start clickhouse-server
Enter fullscreen mode Exit fullscreen mode

Verify the service.

sudo systemctl status clickhouse-server
Enter fullscreen mode Exit fullscreen mode

Confirm the Installed Version 

SELECT version();
Enter fullscreen mode Exit fullscreen mode

Expected output:

version()
26.3.x.x

Validate Cluster Health

After upgrading a node, confirm replication remains healthy.

SELECT
    replica_name,
    is_readonly,
    active_replicas,
    total_replicas,
    queue_size,
    last_exception
FROM system.replicas;
Enter fullscreen mode Exit fullscreen mode

Everything should remain healthy before upgrading the next node.

Run Validation Queries

Check that the database is operating normally.

Verify data access 

SELECT count()
FROM your_main_table;
Enter fullscreen mode Exit fullscreen mode

Verify Keeper connectivity 

SELECT *
FROM system.zookeeper_connection;
Enter fullscreen mode Exit fullscreen mode

Verify cluster configuration 

SELECT *
FROM system.clusters
WHERE cluster = 'your_cluster_name';
Enter fullscreen mode Exit fullscreen mode

Also confirm every node reports version 26.3.

Validate Breaking Changes

One of the most important behavioral changes in ClickHouse® 26.3 involves the NOT operator precedence.

Create a small test table.

CREATE TABLE test_not_operator
(
    id UInt32,
    value Nullable(Float64)
)
ENGINE = MergeTree
ORDER BY id;
Enter fullscreen mode Exit fullscreen mode

Insert test data.

INSERT INTO test_not_operator VALUES
(1, 10.0),
(2, NULL),
(3, 20.0);
Enter fullscreen mode Exit fullscreen mode

Run the query.

SELECT
    id,
    value
FROM test_not_operator
WHERE NOT value IS NULL;
Enter fullscreen mode Exit fullscreen mode

Expected output:

id value
1 10.0
3 20.0

If your applications contain similar queries, review them before upgrading.

Replacing

NOT value IS NULL
Enter fullscreen mode Exit fullscreen mode

with

value IS NOT NULL
Enter fullscreen mode Exit fullscreen mode

is the recommended approach.

Post-Upgrade Validation Checklist

After completing the upgrade, verify:

  • Server version
  • Replica health
  • Query performance
  • Distributed queries
  • Materialized Views
  • Background merges
  • Background mutations
  • Keeper connectivity
  • Disk usage
  • Error logs
  • Application connectivity

Everything should behave exactly as expected before considering the upgrade complete.

Rollback Plan

If an upgraded node experiences unexpected issues, rollback is straightforward.

Stop ClickHouse®.

sudo systemctl stop clickhouse-server
Enter fullscreen mode Exit fullscreen mode

Install the previous version.

sudo apt install -y \
clickhouse-server=25.x.x.x \
clickhouse-client=25.x.x.x \
clickhouse-common-static=25.x.x.x
Enter fullscreen mode Exit fullscreen mode

Restore configuration if necessary.

sudo cp -r \
/etc/clickhouse-server-backup-YYYYMMDD/* \
/etc/clickhouse-server/
Enter fullscreen mode Exit fullscreen mode

Restart the service.

sudo systemctl start clickhouse-server
Enter fullscreen mode Exit fullscreen mode

Always test rollback procedures on staging before relying on them in production.

Common Upgrade Mistakes

Mistake Impact
Skipping backups Risk of data loss
Upgrading every node simultaneously Cluster downtime
Ignoring release notes Unexpected breaking changes
Skipping staging validation Production failures
Not checking replica health Replication problems
Forgetting application compatibility Incorrect query behavior

Best Practices After Upgrading

Once the cluster is running successfully:

  • Monitor the environment for at least 24–48 hours.
  • Watch system.replicas for replication health.
  • Review system.query_log for unexpected query behavior.
  • Monitor application error rates.
  • Update SQL queries to use IS NOT NULL where appropriate.
  • Review asynchronous insert settings.
  • Remove temporary backups only after confirming stability.
  • Document the upgrade process, version numbers, and observations for future maintenance.

Quick Upgrade Checklist

Step Action
1 Verify cluster health
2 Resolve replication issues
3 Backup configuration and metadata
4 Upgrade staging environment
5 Validate applications
6 Upgrade production one node at a time
7 Validate each replica
8 Test breaking changes
9 Monitor production for 24–48 hours

Final Thoughts

ClickHouse® 26.3 LTS delivers important improvements in stability, standards compliance, and overall performance, making it an excellent choice for production deployments. However, even well-tested LTS releases can introduce subtle behavioral changes that require careful validation.

A successful upgrade isn't just about installing new packages—it's about confirming that applications, replication, queries, and operational workflows continue to behave exactly as expected. Taking the time to validate changes in a staging environment, upgrading replicas one at a time, and monitoring production after deployment dramatically reduces upgrade risk.

Two changes deserve particular attention in 26.3: the updated NOT operator precedence and the default asynchronous insert behavior. Neither change generates obvious errors, but both can affect application behavior if left unreviewed.

With thorough preparation, incremental deployment, and careful validation, upgrading to ClickHouse® 26.3 LTS can be completed smoothly while taking full advantage of its latest improvements.

Top comments (0)