I have to do both at my job, but I'm primarily programming in Scala in data-engineer mode, then switch to Python for DS.
It's kind of weird, but I think I actually prefer the data engineering side more.
Sounds about write. Scala is another language that can be used as a DE.
I don't think it's weird. Personally, I enjoy the engineering style work often more than the analytical side. This is not to say I don't like the DS side. I just prefer one over the other.
Also props for mentioning Luigi. Have you looked into Kedro? McKinsey's new open sourced thing. It's kind of neat. I've been using it for personal projects recently.
Oh man! No. It's so hard to keep up with all the new frameworks. But I will take a look.
The tutorial explains the features pretty well, it makes a sample regression pipeline that pre-process data and puts them into raw, processed, etc. folders.
I'm trying to see if I can combine it with tox and then maybe MLFlow for parameter saving.
We're a place where coders share, stay up-to-date and grow their careers.
We strive for transparency and don't collect excess data.