Starting 2025 on a high note, I signed up for the DE Zoomcamp program, and wow, what a week it's been! Here’s what I’ve learned and built during my first week.
Docker and Docker-Compose: Laying the Foundation
I’ve always been curious about Docker, and after this week, I can safely say that I’ve got a solid foundation in it. Docker allows you to create isolated environments for running applications, which is a huge deal when working with different tools in data engineering. docker-compose
was a game-changer, enabling me to spin up multiple services in one go. At first, I was a bit confused about how everything connects, but after some trial and error, it clicked!
Postgres and pgAdmin
Setting up Postgres locally with Docker was straightforward, and I quickly became familiar with using pgcli to connect to the database. However, I think my preferred way of interacting with Postgres was through pgAdmin — it just felt more intuitive with its GUI. I could easily navigate through tables and execute SQL queries without having to rely on terminal commands.
Here’s an example of a simple docker-compose.yml to run Postgres and pgAdmin:
Ingestion Script Optimization
One of my highlights this week was optimizing my ingestion script. Initially, the script was set up to load one table at a time. I refactored it to load both tables in parallel by breaking out the logic into separate functions. This not only made the process more efficient but also gave me better insight into how to structure reusable code in data pipelines.
Terraform and Google Cloud
A major highlight for me was Terraform. Before starting this course, my boss had mentioned Terraform for a project I was working on, but I found the concept pretty overwhelming. When I saw it was part of the curriculum, I was pumped! The tutorials were thorough and easy to follow, and now I’m confident with the basics of setting up Google Cloud Storage and BigQuery using Terraform. It’s definitely something I’ll be using in future projects.
Wrapping Up
Overall, week 1 has been an eye-opener! Docker, PostgreSQL, and Terraform — it’s a lot to digest, but it’s been a rewarding journey so far. The hands-on exercises, especially the Docker-Postgres setup and Terraform configuration, were incredibly valuable. I’m already looking forward to what week 2 has in store so stay tuned for more updates as I continue this journey.
And hey, you should check out my GitHub Repository to see my project!
Top comments (0)