Distributed Workflow Runner

Deepesh Pathak — Tue, 30 Jun 2020 05:05:49 +0000

Xene

A distributed workflow runner with a focus on performance and simplicity.

Xene is a high performance, distributed workflow execution platform. It lets you create custom workflows in the form of a Directed Acyclic Graph, which can then be executed based on a trigger configured with the workflow.

Xene is built with Golang with a focus on performance, resiliency, and simplicity. Xene internally runs in a way similar to Kubernetes by getting triggered on a level and then reconciling the desired state with the current state. Currently, xene supports Badger as the storage engines, which is pluggable and can be replaced with a similar key-value store.

Features

Xene aims to tackle two main problems in any workflow execution system:

Low resource footprint
Flexibility and Simplicity

Xene exploits a lot of small enhancement to reduce the network and memory footprint of the executors and APIServer.
APIServer acts as the brains of the whole ecosystem by directing the decisions of scheduling workloads and managing coordination
between all the moving components. All the communication between APIServer and Agent is encrypted using mTLS and xene doesn't require
Agents to communicate with each other reducing network footprint.

Xene derives a lot of its design decisions from Kubernetes, like:

Like Kubernetes, xene is also level triggered and does the reconciliation of state described by the user with the actual state.
It also uses a KeyValue store to store all the states and objects. To reduce the memory footprint, xene supports the use of Badger as a key-value store.
Objects are reconciled using controllers configured to watch changes on keys in the KVStore.

Some of the main features of Xene are:

Locally running a workflow from provided specs.
Command-line tool for xene(xenectl) to interact with apiserver.
mTLS based authentication between agent and apiserver.
JWT based authentication for API server.
Google OAuth based authentication integration.
Beta UI based on React and Typescript
Workflow creation and running on top of Docker-based container runtime executor on the agent.
Secrets management for agents.
Workflow pipelines run live log streaming and status feed.
Google-based OAuth for APIServer with domain filtered authorization.

Contributing

You can start with setting up Xene on your system and try solving a few bugs listed here: https://github.com/fristonio/xene/issues

Documentation

License

Xene is licensed under MIT License.

Contact

If you have any queries regarding the project or just want to say hello, feel free to drop a mail at deepshpathak@gmail.com.