I finished Source Systems, Data Ingestion, and Pipelines, the second course in the DeepLearning.AI + AWS Data Engineering Professional Certificate, on October 3, 2026. The labs were hands-on on AWS, and I cleared the final exam.
Course
Source Systems, Data Ingestion, and Pipelines, DeepLearning.AI and Amazon Web Services, on Coursera.
Verify certificate: https://coursera.org/verify/FWGU74AERQUM
What I did
- Explored source systems. I looked at how different source systems generate and update data, and how to connect to them and troubleshoot common connection issues.
- Compared batch and streaming ingestion. I identified the use cases and trade-offs of each pattern, and compared the ETL and ELT paradigms.
- Built a batch and a streaming ingestion pipeline on AWS. Both pulled data in from source systems, using AWS services for each pattern.
- Automated and orchestrated pipelines. The course covers infrastructure as code and pipelines as code, with Apache Airflow as the orchestration tool.
- Looked at monitoring and data quality. I explored AWS and open source tools for watching both the data systems and the quality of the data moving through them.
What I took from it
Batch versus streaming is a decision driven by the use case, not by which one sounds more modern. ETL versus ELT comes down to where the transformation happens: before the data is loaded, or after it lands in the destination.
Series: Data Engineering Log, part 2 of 3, from the DeepLearning.AI + AWS Data Engineering Professional Certificate.
More of my work is on GitHub.
Top comments (0)