We are building a new version of our current big data lab and I'm interested in understanding how others structure their logical data architecture.
Do you have data warehouses separate from real-time streaming data, data science applications, shared data sets?
I think this is a nice topic! I'm also keen on learning from our fellow practitioners. For us, we are still on the infancy stage of our arch. We still have the typical ETL, DW and Data Viz stack. However, what changes now from then is that they are all cloud based. Less hassle maintenance wise.
I'm still seeing (and recommending) data warehouses to serve as the operational reporting and ad hoc analysis backbone. They don't necessarily need to do all of the data preparation and processing like the DW's of old. I think dimensional models work well for people who want to use data but don't want to spend time studying data modeling techniques.
Because of the proliferation of other complementary platforms and tools, I do find that strong metadata management has never been more important. Using a NoSQL platform makes it possible to create something highly-customized without reinventing the wheel. I should probably also note that I assume most organizations would do all of these things in the cloud. "Cloud data architecture" is just "data architecture" now.
Anthony J. Algmin
Founder and CEO
Algmin Data Leadership, LLC
[login to unmask email]
Make an Impact! Visit algmin.com today!
Just curious to know what you mean when you mention "... I think dimensional models work well for people who want to use data but don't want to spend time studying data modeling techniques....". What dimensional model structure are you referencing? We are currently still doing our data prep and ETL in SQL On-Prem with SQL Enterprise as our Warehouse. I'm looking at other options
