Feeding the OLAP beast: ClickHouse massive bulk ingestion steps.

ClickHouse is a columnar beast, but treating it like MySQL by inserting row-by-row will bring your cluster to its knees. ClickHouseBulkStep packages optimal bulk buffering into a reusable step.

Here is how you use ClickHouseBulkStep & OLAP Pipelines in a production pipeline with wpipe-steps:

from wpipe import Pipeline
from wpipe_steps.database import ClickHouseBulkStep

pipeline = Pipeline(pipeline_name="telemetry_ingest")

pipeline.set_steps([
    ClickHouseBulkStep.as_step(
        name="insert_telemetry_batch",
        host="clickhouse.production.internal",
        database="analytics",
        table="device_telemetry",
        data_key="telemetry_batch",
        response_key="clickhouse_status"
    )
])

sample_batch = [
    {"sensor_id": f"S-{i}", "temp": 24.5 + i, "timestamp": "2026-09-19 12:00:00"}
    for i in range(10000)
]

result = pipeline.run({"telemetry_batch": sample_batch})
print(f"Inserted {result['clickhouse_status']['count']} records successfully!")

Why developers love wpipe-steps:

• 196 cataloged steps covering Redis, ClickHouse, MySQL, WAF, S3, Docker, and local HuggingFace AI.
• Lazy-loading imports for instant sub-100ms startup times.
• Clean .as_step() factory interface.

Check out the full repository on GitHub!