Linode + DEV Hackathon: Airflow running in Linode + daemon setup

Andrea Montes — Mon, 20 Feb 2023 00:39:38 +0000

What I built

I built a gist to install airflow (workflow + scheduler) in Linode

Category Submission:

Smooth Shifters

App Link

http://143.42.142.188:8080 (you won't be able to log in but you will see the airflow instance running)

Screenshots

Description

Airflow is running in linode. A daemon was set as well to enable accessibility to the tool.

Link to Source Code

https://gist.github.com/mamontesp/2d2fb5aa7ab880f257cbed25951da5f1

Permissive License

MIT License

Background

I need airflow in my work, but I have some security restrictions to create an airflow instance in the cloud provider the company uses. Linode offers a straightforward way to fast testing a tool like airflow

How I built it

I used the PyPI version of Airflow, created a virtual environment with python and in there, I did the airflow install. Then, I started the airflow webserver. All the instructions are in the gist file provided :)

I wanted to use my own user in Linode, but got several issues granting permissions, I figured some out, but at the end I just needed to test airflow, so I switched to root and complete the install. It would be good to have a tutorial to set new users.

Also when trying to use my own new user, shell command navigation wasn't working (pressing arrows wasn't navigating across my command history), I was able to solve that out running this.

sudo chsh -s /bin/bash <username>

Finally, I needed to create a daemon to keep up airflow, I also included the instructions that I followed to accomplished that.

Additional Resources

Airflow: https://airflow.apache.org/docs/apache-airflow/stable/index.html
YouTube Course in Linux TV to get more confident with Linux general commands: https://www.youtube.com/watch?v=2733cRPudvI&list=PLT98CRl2KxKGj-VKtApD8-zCqSaN2mD4w
Daemon creation process: https://baykara.medium.com/how-to-daemonize-a-process-or-service-with-systemd-c34501e646c9

Let's connect

Mastodon: @mamontesp@fosstodon.org
LinkedIn: https://www.linkedin.com/in/andrea-montes-5a9a25b1/

DEV Community: Andrea Montes