Most of my work is doing backend development and front end development , i had zero experience with ETL . Most of the the time is build an api , web app , debugging application error , google how to center div and etc etc .
Data engineering ? For me it was something that i think was reserved for a different kind of developer or specialist who lived in SQL, optimized queries for breakfast, and wrangled giant data clusters .
Since my current role is not too busy , one day i volunteered to take on a task to build our first data pipeline .
At first, I had no idea what I was getting myself into. I didn’t even know what tools people normally used. Do I supposed to write a lot of cron jobs? Build custom scripts? Install some massive framework like Airflow that looked intimidating just from the docs?
First when i see Airflow , i feel like is it i made the wrong choice to volunteer ? haha , considering the timeline of the project , i know there's a lot i need to know when using airflow especially this need to be on production within short timeline .
Mage AI
Of course, there were other tools like Luigi, Prefect, and Dagster. I gave each of them a quick spin, just a simple “hello world” test.My reasoning was simple , if even the hello world felt complicated, how could I possibly feel confident using that tool for a real project with a tight deadline? Not on that also i need to consider the team adaptability as no one in the team also familiar with ETL .
That’s where Mage AI stood out. From the very first try to run it , it feels really easy and straight forward .
If you want to try just run this
docker run -it -p 6789:6789 -v $(pwd):/home/src mageai/mageai /app/run_app.sh mage start my-first-etl
Once you run you can access the mage ai dashboard , and can start doing your first pipeline . It's really easy to start , play around especially for first timer like me to explore ETL tools .
Of course , this is only tool , i spend a lot of time also to understand the foundation related to ETL , data modeling , data loading strategy , transformation , orchestration & scheduling ( which one of Mage AI do ) , i enroll to Kotak Sakti Bootcamp . Enrolling in the Kotak Sakti Bootcamp had a big impact on me which help me to strong my understanding on what data engineer needs .
I will try to share more what i experience when working with Mage AI in future .
Top comments (0)