YTsaurus Flow brings stateful stream processing and default exactly-once semantics to the YTsaurus ecosystem, with support for several languages.
Yandex Tech released the source code for YTsaurus Flow under the Apache 2.0 license on October 6. Built jointly by Yandex Infrastructure and Yandex Advertising, the framework targets high-load, real-time data processing, particularly the stateful pipelines used in recommendation systems. Yandex says exactly-once processing is enabled by default.
Flow is part of YTsaurus, the storage and data-processing platform that Yandex published on GitHub in March 2023. It uses YTsaurus queues, dynamic tables and the shared transaction mechanism to manage pipeline state, distribute work and recover processing after failures, leaving application developers to concentrate on business logic.
For supported synchronous sinks, Flow commits state, processing progress and output data in a single transaction called an epoch. An epoch is either committed in full or not committed at all. External asynchronous sinks use a transactional outbox: data is first stored in YTsaurus and then sent with deduplication identifiers and retries. Teams can instead select at-least-once or at-most-once processing to reduce resource use.
A Flow pipeline separates infrastructure from application logic. Controllers elect a leader, assign work, adjust partition counts and rebalance load; workers read messages, invoke companion applications and keep long-term state in YTsaurus dynamic tables. An administrative client validates and applies pipeline specifications and can be called from CI/CD. Deployment is supported in Linux environments with network access to YTsaurus or inside a cluster through Vanilla operations.
The core is written in C++, while pipeline logic can use C++, Python, Go, Kotlin or Java; simple pipelines can be defined in YQL. Processing graphs contain source, transform and sink nodes, as well as Swift nodes that reduce intermediate-data materialization in exchange for deterministic-computation requirements and restricted functionality. Yandex says some internal graphs contain hundreds of computation nodes and may include cycles.
For machine-learning teams, the immediate value is faster delivery of fresh training data rather than model training itself. Yandex reports that one advertising workflow cut data-delivery latency for model retraining from roughly 10–20 hours to about two hours after moving to Flow. The company provides no comparative benchmark against other stream-processing systems and recommends testing performance with each organization’s own workloads and infrastructure.
The exactly-once boundary covers processing performed through Flow, not arbitrary external effects triggered by user code. If a computation makes an HTTP request or calls another service, the pipeline developer remains responsible for correctness, such as by making the operation idempotent. For resilience to a complete data-center outage, Yandex recommends a multi-site configuration—typically three data centers—with Flow components, cross-site tables and spare capacity in each location.
| Mode | Property | Trade-off |
|---|---|---|
| exactly-once | Messages are neither lost nor counted twice within supported Flow processing | Requires additional computing resources |
| at-least-once | Messages are not lost | Duplicates are possible |
| at-most-once | Messages are not duplicated | Losses are possible |
| Tool | Intended use |
|---|---|
| C++ | Performance-critical workloads |
| Python | Prototyping |
| Go, Kotlin, Java | Application logic through similar APIs |
| YQL | Simple pipelines defined declaratively |
Sources
Originally published by AI Feed: Yandex Tech Open-Sources YTsaurus Flow Under Apache 2.0.
Top comments (0)