VK Data Platform combines shared Iceberg/S3 storage, workload-specific engines and isolated compute pools for analytics, data engineering and ML.
VK Tech has detailed the architecture of VK Data Platform, its enterprise lakehouse system for large-scale data processing, analytics and ML preparation. The platform is designed to provide a common storage and computing environment for BI, data engineering and AI workloads instead of requiring separate systems and data copies for each use case.
VK Object Storage forms the storage foundation, while Apache Iceberg provides the table layer and Apache Parquet the columnar file format. According to VK Tech, this combination allows multiple compute engines to work with the same Iceberg tables over S3 without duplicating data. Iceberg also supports ACID transactions, access to earlier versions, schema evolution and file pruning during reads.
Workloads are distributed among several managed engines. ClickHouse is intended primarily for interactive dashboards, frequently accessed data marts and direct Iceberg/S3 analysis. Trino covers complex SQL, multiple joins, federated access and SQL-based ETL or ELT, while Spark handles resource-intensive pipelines, batch and streaming processing, Iceberg maintenance and ML tasks. VK Tech positions CedrusData Engine as an accelerated Trino fork with key core elements rewritten in Rust; its claimed two-to-fivefold performance gain reflects the company’s production workloads, not a separately presented independent benchmark.
Managed Kafka supplies streaming ingestion, Managed Airflow orchestrates pipelines, and Managed JupyterHub provides notebook environments for analysts, data engineers and ML teams. CedrusData Catalog coordinates engine access to Iceberg tables. Data Platformer brings deployment, scaling, resource allocation, access policies, monitoring, auditing, backup and recovery under a common management layer.
Compute resources can be separated into isolated pools for BI reporting, ad hoc analysis, Spark ETL or ML. VK Tech says this reduces interference between workloads, so a heavy Spark calculation does not compete with dashboard generation on the same shared cluster. Components run in Kubernetes, but stateful services—including ClickHouse with local storage, Kafka and PostgreSQL—require more cautious scaling than stateless Trino and Spark services.
Security spans IAM for users and administrative access, role-based permissions within analytical services, and S3-level safeguards. The storage controls described by VK Tech include service accounts, encryption, Object Lock, versioning and operation controls. Availability measures combine replication, backups, Kubernetes recovery and components’ native high-availability mechanisms.
The architecture could provide a shared corporate data layer for reporting, RAG, training-data preparation and other ML or AI tasks while retaining different engines for different workload profiles. However, VK Tech stresses that the platform cannot repair poor-quality data, engineer features, validate models or replace governance and operating rules; those remain the adopting team’s responsibility. Managed, on-premises and hybrid delivery options are supported, but the source gives neither specific pricing nor minimum system requirements.
| Engine | Primary tasks | Limitation or characteristic |
|---|---|---|
| Managed ClickHouse | Interactive dashboards, frequently accessed data marts and aggregates | JOIN operations are supported but require optimization |
| Managed Trino | Complex SQL, multiple JOINs, federated access and ETL/ELT | Uses the shared Iceberg/S3 layer |
| Managed Spark | Heavy ETL/ELT, batch and streaming, Iceberg maintenance and ML | Designed for resource-intensive workloads |
| CedrusData Engine | SQL analytics, federated access and BI | VK Tech reports a two-to-fivefold speed gain over standard Trino |
Sources
Originally published by AI Feed: VK Tech Details Its Lakehouse Architecture for Analytics and ML.
Top comments (0)