Apache Airflow on EC2 with RDS

Mahmudul Haque — Thu, 17 Aug 2023 21:13:40 +0000

sudo apt update && sudo apt upgrade -y

sudo apt install python3-pip python3-psycopg2 default-libmysqlclient-dev

python3 -m pip install --upgrade pip

pip install --user apache-airflow[amazon] apache-airflow[postgres] apache-airflow[celery] psycopg2-binary apache-airflow-providers-mysql Redis

echo "export AIRFLOW_HOME=${HOME}/airflow" >> .zshrc

airflow db init

airflow.cfg

[core]
executor = CeleryExecutor
load_examples = False

[database]
sql_alchemy_conn = postgresql://myuser:mypassword@my-rds-instance.abc123.us-west-2.rds.amazonaws.com:5432/mydatabase`

[celery]
broker_url = redis://localhost:6379/0
result_backend = redis://localhost:6379/1

Celery Executor: https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/celery.html

airflow db init

airflow users create --username admin --firstname FirstName --lastname LastName --role Admin --email email@example.com

/etc/systemd/system

sudo systemctl enable airflow-webserver.service

sudo systemctl enable airflow-scheduler.service

sudo systemctl enable airflow-celery-flower.service

sudo systemctl enable airflow-celery-worker.service

sudo systemctl start airflow-webserver

sudo systemctl start airflow-scheduler

sudo systemctl start airflow-celery-flower

sudo systemctl start airflow-celery-worker

Global gitignore

Mahmudul Haque — Mon, 15 Nov 2021 10:23:02 +0000

The .gitignore file is a powerful tool in Git for excluding certain files and directories from being tracked. However, it's essential to use it correctly and avoid including environment-specific exclusions in project-specific .gitignore files. In this article, we will discuss the proper usage of .gitignore files, distinguishing between project-specific and global exclusions.

Project-specific .gitignore

The primary purpose of a project-specific .gitignore file is to exclude files and directories that are specific to the project itself. It helps in keeping the repository clean and prevents accidental commits of unnecessary or sensitive information.

A good practice is to include only project-related files and directories in the project-specific .gitignore. This includes:

Build artifacts and compiled binaries
Dependency directories (e.g., node_modules, venv, etc.)
Generated files (e.g., log files, cache files, etc.)
Sensitive information (e.g., credentials, API keys, etc.)

For example:

# Build artifacts
build/
dist/

# Dependency directories
node_modules/
vendor/

# Generated files
*.log
*.cache

# Sensitive information
credentials.txt
secrets/

By focusing on project-specific exclusions, you ensure that anyone cloning or working on the project doesn't inadvertently commit or track unnecessary files. It keeps the repository clean and focused on the project itself.

Global .gitignore

While project-specific .gitignore files handle exclusions specific to a particular project, there may be files and directories that are not project-specific but should still be ignored across all Git repositories. Examples include IDE-specific files, operating system artifacts, and common build artifacts.

To handle such global exclusions, Git provides the concept of a global .gitignore file. This file resides outside any project and defines exclusions that apply to all repositories on your local machine.

To create a global .gitignore file, follow these steps:

Create the .gitignore file in your home directory or any other preferred location:

touch ~/.gitignore

Configure Git to use the global .gitignore file:

git config --global core.excludesfile ~/.gitignore

Once the global .gitignore file is set up, you can include exclusion patterns that apply globally. One convenient way to obtain these patterns is by referencing the GitHub Gitignore repository, specifically the "Global" directory:

Global .gitignore templates: https://github.com/github/gitignore/tree/master/Global

In the global .gitignore file, copy and paste the necessary exclusion patterns from the repository, depending on your specific requirements.

By separating global exclusions from project-specific ones, you maintain a clean and focused .gitignore file for each project while ensuring that common exclusions are applied consistently across all repositories on your machine.

Improved by: ChatGPT

DEV Community: Mahmudul Haque

Apache Airflow on EC2 with RDS

Global gitignore