Background in politics, commodity trading, and converted to being a data engineer in 2017. I worked with Django, Flask, Plotly, and Vue.JS, but now Airflow and PySpark for ETL pipelines.
You're all done with your pipeline until an edge case pops up you hadn't planned for and you spend a while looking through logs to figure out what exactly went wrong in the process :)
One of the things you missed is the difference in expectations. Stakeholders expect data engineers to process data and get it into a state others can use. Stakeholders expect data scientists to cure cancer in a single sprint. It's a lot harder to manage expectations when there is so much hype built up around a field. I just keep telling people Tesla promised full self driving cars "next year" 3 years ago and I still have to drive myself to work. It took Google several years and PhDs to get Google Assistant to "understand" a single sentence.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
You're all done with your pipeline until an edge case pops up you hadn't planned for and you spend a while looking through logs to figure out what exactly went wrong in the process :)
One of the things you missed is the difference in expectations. Stakeholders expect data engineers to process data and get it into a state others can use. Stakeholders expect data scientists to cure cancer in a single sprint. It's a lot harder to manage expectations when there is so much hype built up around a field. I just keep telling people Tesla promised full self driving cars "next year" 3 years ago and I still have to drive myself to work. It took Google several years and PhDs to get Google Assistant to "understand" a single sentence.