I started my first job as a Data Engineer back in September. I wasn't exactly sure what the job entailed, but I was ready to shift my focus and lea...
For further actions, you may consider blocking this person and/or reporting abuse
This is an anonymous comment sent in by a member who does not want their name disclosed.
I'm not sure how valid these questions are or if I'm articulating them properly.
What is a data warehouse and when do you need it? This is what I've been assuming: a gigantor database that extracts data from other sources, transforms them into the format you want and consolidates them all in one place so you can query all you want. i.e. if you don't need data from a db to be combined with other data, you wouldn't need to include it your data warehouse. Is that in the ballpark?
Do apps pull data from the data warehouse? I would assume not because they'd pull directly from the app database ^ if my above notion is correct.
Do data warehouses typically get updated in real time?
Hi! I've been working in DWH some years, in my expierence i could say about question 1&2: a data warehouse is not only for querying data, it allows you to get information(next level of data) and explode them with anothers tools like PowerBI, Microstrategy, etc. I didn't know about an app the reads a data warehouse.
Datawarehouses get updated in batch, it's not common a streamming updating, this is an app database behaviour.
:)
Totally valid questions! I will definitely start working on an explainer post about data warehouses - what they are, what commercial products are available, and common use cases! Thank you for reaching out!
"how do you optimize this SQL query" -- Best statement ever hahaha, I can relate here u.u
I've been working in a Datalake team (data whatever position) for a while and now switching back to frontend :p
IMHO it is a mix of DevOps and Big Data processing...
Data Engineering on AWS is a challenging role. You should know which service should use for which job. Otherwise you will end up in so much cost. If you guys are interested in how to build Data Make on AWS, just check out the link: youtu.be/lRWkGVBb13o
Chicago Python community represent!
This would make a great panel discussion at a ChiPy Data event. We'll see what we can set up in early 2019.
Ooh, yes. I'm willing to bet AC would be willing to host 😄