Part 1: Introduction to Workflow Orchestration & Kestra Fundamentals
What is Workflow Orchestration?
Think of a music orchestra with various instruments that need to work together. The conductor helps them play in harmony. Similarly, a workflow orchestrator coordinates multiple tools and platforms to work together.
A workflow orchestrator typically:
- Runs workflows containing predefined steps
- Monitors and logs errors with additional handling when they occur
- Automatically triggers workflows based on schedules or events
In data engineering, we often need to move data from one place to another with modifications. The orchestrator manages these steps while providing visibility into the process.
What is Kestra?
Kestra is an open-source, event-driven, infinitely-scalable orchestration platform. Key features include:
| Feature | Description |
|---|---|
| Flow Code (YAML) | Build workflows with code, no-code, or AI Copilot |
| 1000+ Plugins | Integrate with virtually any tool |
| Multi-language Support | Use Python, SQL, or any programming language |
| Flexible Triggers | Schedule-based or event-based execution |
Core Kestra Concepts
- Flow - A container for tasks and orchestration logic (defined in YAML)
- Tasks - Individual steps within a flow
- Inputs - Dynamic values passed at runtime
- Outputs - Data passed between tasks and flows
- Triggers - Mechanisms that automatically start flow execution
- Execution - A single run of a flow with a specific state
- Variables - Key-value pairs for reusable values across tasks
- Plugin Defaults - Default values applied to tasks of a given type
- Concurrency - Control how many executions can run simultaneously
Installing Kestra
Kestra runs via Docker Compose with two main services:
- Kestra server container
- PostgreSQL database container
cd 02-workflow-orchestration
docker compose up -d
Access the UI at: http://localhost:8080
Running Python Code in Kestra
Kestra can execute Python code either:
- From a dedicated file
- Written directly inside the workflow YAML
This allows you to pick the right tools for your pipelines without limitations.
Top comments (0)