Over the weekend I stumbled on a well-known ETL tool vender's documentation site. (I'm not naming names). It said many times you might want to ingest a new data file directly from an FTP site into the silver layer of your data lake. ¯\_(ツ)_/
My thoughts were:
If a file is ETLed successfully into a database you could assume it has been preboarded as well as loaded. But has it been, really? You can put milk in a bottle and sell it, but that doesn’t necessarily mean it has been pasteurized.
and
In many companies a 10+% firefighting load on every million dollars in data feed-tied revenue is seen as perfectly normal. It shouldn't be.
If you're thinking of BASE-jumping into silver because the preboarding elevator isn't fast enough, please take a minute: https://blog.csvpath.org/data-preboarding-ingestion-etl-and-onboarding
Top comments (0)