Skip to content

re: What Is The Difference Between A Data Engineer And A Data Scientist VIEW POST

re: I have to do both at my job, but I'm primarily programming in Scala in data-engineer mode, then switch to Python for DS. It's kind of weird, but I...

Sounds about write. Scala is another language that can be used as a DE.

I don't think it's weird. Personally, I enjoy the engineering style work often more than the analytical side. This is not to say I don't like the DS side. I just prefer one over the other.


Also props for mentioning Luigi. Have you looked into Kedro? McKinsey's new open sourced thing. It's kind of neat. I've been using it for personal projects recently.

Oh man! No. It's so hard to keep up with all the new frameworks. But I will take a look.

The tutorial explains the features pretty well, it makes a sample regression pipeline that pre-process data and puts them into raw, processed, etc. folders.

I'm trying to see if I can combine it with tox and then maybe MLFlow for parameter saving.

code of conduct - report abuse