PyJaws enables declaring Databricks Jobs and Workflows as Python code, allowing for:
- Code Linting (e.g. with Flake or Ruff)
- Formatting (e.g. with Black)
- Parameter Validation
- Modularity and reusability
In addition to those, PyJaws also provides some nice features such as cycle detection out of the box.
Folks who have used Python-based orchestration tools such as Apache Airflow, Luigi and Mage will be familiar with the concepts and the API if PyJaws.
PyJaws leverages some existing libraries in order to allow for modularisation, reusability and validation, such as:
- Click - for providing a rich CLI functionality
- Pydantic - for efficient parameter validation
- NetworkX - for Graph and Cycle Detection features
- Jinja2 - for templating
Check it out: https://github.com/rafaelpierre/pyjaws
Top comments (0)