DEV Community

Andrea Montes
Andrea Montes

Posted on

1

Linode + DEV Hackathon: Airflow running in Linode + daemon setup

What I built

I built a gist to install airflow (workflow + scheduler) in Linode

Category Submission:

Smooth Shifters

App Link

http://143.42.142.188:8080 (you won't be able to log in but you will see the airflow instance running)

Screenshots

First view Airflow running in Linode

Evidence of dags runs in Airflow - running in Linode

Description

Airflow is running in linode. A daemon was set as well to enable accessibility to the tool.

Link to Source Code

https://gist.github.com/mamontesp/2d2fb5aa7ab880f257cbed25951da5f1

Permissive License

MIT License

Background

I need airflow in my work, but I have some security restrictions to create an airflow instance in the cloud provider the company uses. Linode offers a straightforward way to fast testing a tool like airflow

How I built it

I used the PyPI version of Airflow, created a virtual environment with python and in there, I did the airflow install. Then, I started the airflow webserver. All the instructions are in the gist file provided :)

I wanted to use my own user in Linode, but got several issues granting permissions, I figured some out, but at the end I just needed to test airflow, so I switched to root and complete the install. It would be good to have a tutorial to set new users.

Also when trying to use my own new user, shell command navigation wasn't working (pressing arrows wasn't navigating across my command history), I was able to solve that out running this.

sudo chsh -s /bin/bash <username>

Finally, I needed to create a daemon to keep up airflow, I also included the instructions that I followed to accomplished that.

Additional Resources

Let's connect

Mastodon: @mamontesp@fosstodon.org
LinkedIn: https://www.linkedin.com/in/andrea-montes-5a9a25b1/

AWS Q Developer image

Your AI Code Assistant

Automate your code reviews. Catch bugs before your coworkers. Fix security issues in your code. Built to handle large projects, Amazon Q Developer works alongside you from idea to production code.

Get started free in your IDE

Top comments (0)

Billboard image

Create up to 10 Postgres Databases on Neon's free plan.

If you're starting a new project, Neon has got your databases covered. No credit cards. No trials. No getting in your way.

Try Neon for Free →

👋 Kindness is contagious

Please leave a ❤️ or a friendly comment on this post if you found it helpful!

Okay