"Building a Data Pipeline: Apache Airflow and PostgreSQL Integration on WSL"

#dataengineering #linux #postgres #tutorial

Install WSL and Ubuntu
Open PowerShell as Administrator:

Press Win + X and select Windows PowerShell (Admin).

Install WSL:

Run the following command:

powershell
wsl --install

This command installs WSL, the latest Ubuntu distribution, and sets WSL 2 as your default version.

After installation, restart your computer.

Open Ubuntu from the Start menu and complete the initial setup by creating a user and password.

🐘 Step 2: Install PostgreSQL on Ubuntu (WSL)
Update Package Lists:

sudo apt update
Install PostgreSQL:

sudo apt install postgresql postgresql-contrib
Start PostgreSQL Service:

sudo service postgresql start

Access PostgreSQL:
sudo -u postgres psql

Create Airflow Database and User:

CREATE DATABASE airflow;
CREATE USER airflow WITH PASSWORD 'airflow';
GRANT ALL PRIVILEGES ON DATABASE airflow TO airflow;
Exit PostgreSQL:
\q
🐍 Step 3: Set Up Python Virtual Environment
Install Python 3 and venv:

sudo apt install python3 python3-venv python3-pip
Create Virtual Environment:

python3 -m venv airflow_env
Activate Virtual Environment:
source airflow_env/bin/activate

pip install --upgrade pip
📦 Step 4: Install Apache Airflow
Install Apache Airflow with PostgreSQL Support:

pip install apache-airflow[postgres]

Set Airflow Home Directory:

export AIRFLOW_HOME=~/airflow
Initialize Airflow Database:
airflow db init
Create Airflow Admin User:
airflow users create \
--username admin \
--password admin \
--firstname Admin \
--lastname User \
--role Admin \
--email admin@example.com

Start Airflow Web Server: