loading...

What Is The Difference Between A Data Engineer And A Data Scientist

SeattleDataGuy on September 25, 2019

By Ben Rogojan We recently did an AMA on Reddit. The most common question that came up was what is the difference between a data scientist and a d... [Read Full]
markdown guide
 

I have to do both at my job, but I'm primarily programming in Scala in data-engineer mode, then switch to Python for DS.

It's kind of weird, but I think I actually prefer the data engineering side more.

 

Sounds about write. Scala is another language that can be used as a DE.

I don't think it's weird. Personally, I enjoy the engineering style work often more than the analytical side. This is not to say I don't like the DS side. I just prefer one over the other.

 

Also props for mentioning Luigi. Have you looked into Kedro? McKinsey's new open sourced thing. It's kind of neat. I've been using it for personal projects recently.

Oh man! No. It's so hard to keep up with all the new frameworks. But I will take a look.

The tutorial explains the features pretty well, it makes a sample regression pipeline that pre-process data and puts them into raw, processed, etc. folders.

I'm trying to see if I can combine it with tox and then maybe MLFlow for parameter saving.

code of conduct - report abuse