Linode + DEV Hackathon: Airflow running in Linode + daemon setup

#linodehackathon

What I built

I built a gist to install airflow (workflow + scheduler) in Linode

Category Submission:

Smooth Shifters

App Link

http://143.42.142.188:8080 (you won't be able to log in but you will see the airflow instance running)

Screenshots

Description

Airflow is running in linode. A daemon was set as well to enable accessibility to the tool.

Link to Source Code

https://gist.github.com/mamontesp/2d2fb5aa7ab880f257cbed25951da5f1

Permissive License

MIT License

Background

I need airflow in my work, but I have some security restrictions to create an airflow instance in the cloud provider the company uses. Linode offers a straightforward way to fast testing a tool like airflow

How I built it

I used the PyPI version of Airflow, created a virtual environment with python and in there, I did the airflow install. Then, I started the airflow webserver. All the instructions are in the gist file provided :)

I wanted to use my own user in Linode, but got several issues granting permissions, I figured some out, but at the end I just needed to test airflow, so I switched to root and complete the install. It would be good to have a tutorial to set new users.

Also when trying to use my own new user, shell command navigation wasn't working (pressing arrows wasn't navigating across my command history), I was able to solve that out running this.

sudo chsh -s /bin/bash <username>

Finally, I needed to create a daemon to keep up airflow, I also included the instructions that I followed to accomplished that.