DEV Community

Cover image for Diving into Data Engineering: My First Week at DataTalksClub's ZoomCamp 2025
Sunny ✨
Sunny ✨

Posted on

1 1 2

Diving into Data Engineering: My First Week at DataTalksClub's ZoomCamp 2025

This year, one of my key goals is to level up my professional skills, especially in areas that align with my career growth. Last week, I took a big step toward that goal by joining the DataTalk's Data Engineering ZoomCamp 2025 cohort, a free nine-week program that dives deep into the essentials and practical applications of data engineering.

DataTalks-DE-Course-Overview

Just a week in, and I've already learned so much! Here's a quick rundown of what we've covered so far:

1. Docker Basics: Containerizing Applications

We started with Docker, a powerful tool for creating, deploying, and running applications in containers. Containers are lightweight, isolated environments that package an application and its dependencies, making it easy to run consistently across different environments.

Key Concepts:

  • Images: Read-only templates used to create containers
  • Containers: Running instances of Docker images
  • Dockerfile: A script that defines how to build a Docker image
  • Volumes: Persistent storage for containers, ensuring data isn't lost when a container is deleted

Hands-On: We containerized a PostgreSQL database and ran it using Docker. This allowed us to set up a fully functional database environment in minutes!

2. Docker Compose: Managing Multi-Container Applications

Next, we explored Docker Compose, a tool for defining and running multi-container Docker applications. Using a docker-compose.yaml file, we configured services, networks, and volumes for our application.

Docker-Compose

Example Setup:

  • A PostgreSQL database container
  • A pgAdmin container for database management
  • Both containers connected via a custom Docker network

Commands:

  • docker-compose up -d: Start services in detached mode
  • docker-compose down: Stop and remove services

3. Terraform: Infrastructure as Code (IaC)

On the cloud side, we dove into Terraform, an open-source Infrastructure-as-Code (IaC) tool. Terraform allows you to define and provision infrastructure using a declarative configuration language.

Key Concepts:

  • State Management: Terraform tracks the state of your infrastructure in a .tfstate file
  • Providers: Plugins that interact with cloud APIs (e.g., GCP, AWS)
  • Resources: Components of your infrastructure (e.g., VMs, databases)

Hands-On: We used Terraform to automate the setup of cloud resources on Google Cloud Platform (GCP), including storage buckets and virtual machines.

4. Real-World Application: New York TLC Datasets

To tie everything together, we worked with the New York TLC datasets, a real-world dataset used for taxi and ride-sharing analysis. We applied the concepts we learned—Docker, PostgreSQL, and Terraform—to ingest, store, and analyze the data.

5. What's Next?

This is just the beginning! Over the next eight weeks, we'll dive deeper into data pipelines, workflow orchestration, and more. I'm excited to continue this journey and share my learnings along the way.

A big thank you to @alexeygrigorev and the entire DataTalksClub team for their guidance and support. This journey has been both challenging and rewarding, and I can't wait to see where it takes me next!

What about you? Are you working on up-skilling in data engineering or cloud technologies? Let me know in the comments!

DataEngineering #Docker #Terraform #GCP #PostgreSQL #DEZoomcamp

Billboard image

Use Playwright to test. Use Playwright to monitor.

Join Vercel, CrowdStrike, and thousands of other teams that run end-to-end monitors on Checkly's programmable monitoring platform.

Get started now!

Top comments (0)

Billboard image

Try REST API Generation for Snowflake

DevOps for Private APIs. Automate the building, securing, and documenting of internal/private REST APIs with built-in enterprise security on bare-metal, VMs, or containers.

  • Auto-generated live APIs mapped from Snowflake database schema
  • Interactive Swagger API documentation
  • Scripting engine to customize your API
  • Built-in role-based access control

Learn more

👋 Kindness is contagious

Explore a sea of insights with this enlightening post, highly esteemed within the nurturing DEV Community. Coders of all stripes are invited to participate and contribute to our shared knowledge.

Expressing gratitude with a simple "thank you" can make a big impact. Leave your thanks in the comments!

On DEV, exchanging ideas smooths our way and strengthens our community bonds. Found this useful? A quick note of thanks to the author can mean a lot.

Okay