Study Notes 2.2.1: Workflow Orchestration with Kestra

Introduction to Workflow Orchestration

Key Concepts

1. Orchestration Analogy:
   • Orchestration is compared to a musical orchestra.
   • Different instruments (like violins, trumpets, flutes) produce different sounds and need to be played at specific times.
   • A conductor ensures all instruments play in harmony, similar to how orchestration tools manage different pieces of code.
2. Kestra Overview:
   • Kestra is an all-in-one automation and orchestration platform.
   • Supports ETL processes, batch data pipelines, and workflow scheduling (routine or event-based).
   • Offers flexibility: no-code, low-code, and full-code options.
   • Allows writing workflows in various languages (Python, Rust, C).

1. Monitoring and Visualization:
   • Provides a topology view to visualize workflow stages.
   • Gantt view for monitoring task progress and logs.
   • Integrates with over 600 plugins including cloud platforms (AWS, GCP), Databricks, Snowflake, DBT.

Workflow Components

1. Workflow Definition:
   • ID: Unique identifier for the workflow.
   • Namespace: Organizational unit (like a folder) to store workflows.
2. User-Defined Parameters (UDPs):
   • Parameters passed at the start of workflow execution.
   • Example: brand and price.
3. Tasks:
   • Define the actions in the workflow.
   • Can include extracting data from APIs, running Python scripts, transforming data, etc.
   • Tasks can pass data between each other.

Example Workflow

1. Pipeline Description:
   • Simple pipeline with extract, transform, and query tasks.
2. Workflow Components:
   • ID: Unique name for the workflow.
   • Namespace: Example Zoom Camp.
   • UDPs: Parameters like brand and price.
3. Tasks:
   • Extract Task: Fetches data from dummyjson.com.
   • Transform Task: Python script to process and transform the data.
   • Query Task: SQL query to further process the data.
4. Execution and Monitoring:
   • Gantt View: Visualizes task execution order and status.
   • Logs: Provides insights into task execution.
   • Outputs: Displays generated data like JSON files.

Practical Application

1. Building ETL Pipelines:
   • Extract data (e.g., New York taxis data).
   • Load into PostgreSQL database.
   • Further load into Google Cloud Storage and BigQuery.
2. Dynamic Workflows:
   • Use parameters to control workflow execution without modifying the code.
3. Scheduling and Backfills:
   • Schedule workflows to run automatically.
   • Run backfills to handle missed executions.
4. Production Deployment:
   • Deploy Kestra on Google Cloud.
   • Use Git for version control and syncing workflows.

Additional Resources

• YouTube Video Series: Detailed tutorials on Kestra.
• Documentation: Comprehensive guides and examples.
• Slack Community: Support and discussion with peers.

Conclusion

• Kestra is a powerful tool for orchestrating complex workflows and pipelines.
• Its flexibility and integration capabilities make it suitable for various use cases.
• The upcoming videos will cover practical applications of Kestra in ETL and cloud operations.

Demo Overview

1. Pipeline Execution:
   • ID and Namespace: Organize and identify workflows.
   • UDPs: Parameters like brand and price.
   • Tasks: Extract, transform, and query tasks with data passing between them.
2. Execution Monitoring:
   • Gantt View: Visualize task progress.
   • Logs and Outputs: Inspect task logs and output data.
3. Practical Example:
   • Extract data, transform it, and query it to produce a useful output.