DEV Community πŸ‘©β€πŸ’»πŸ‘¨β€πŸ’»

Tobias Macey
Tobias Macey

Posted on • Originally published at dataengineeringpodcast.com

Automating Your Production Dataflows On Spark - Episode 105

As data engineers the health of our pipelines is our highest priority. Unfortunately, there are countless ways that our dataflows can break or degrade that have nothing to do with the business logic or data transformations that we write and maintain. Sean Knapp founded Ascend to address the operational challenges of running a production grade and scalable Spark infrastructure, allowing data engineers to focus on the problems that power their business. In this episode he explains the technical implementation of the Ascend platform, the challenges that he has faced in the process, and how you can use it to simplify your dataflow automation. This is a great conversation to get an understanding of all of the incidental engineering that is necessary to make your data reliable.

Listen Now!

Top comments (0)

πŸ‘‹ New to DEV?

Head over to our Welcome Thread and tell us a bit about yourself!