It wasn't too long ago that a data warehouse was an expensive, complex affair - housed on-premise and requiring armies of consultants to keep it running. But in the modern world, where companies are moving towards a limited on-premise footprint and faster movements of larger volumes of data, this strategy is no longer viable. To be successful in today's market, you need to adjust your strategy and focus on simplicity and speed. In this blog post, we'll discuss why a modern data warehouse is so important, and how you can make sure yours is up to par.
A data warehouse is a centralized store of information, potentially federated at a data mart level, but with the essential view of democratizing access to all. This means that anyone in the organization can access and use the data, without having to go through IT or other gatekeepers. This is a vast departure from the old model, where only a select few had the knowledge and understanding to navigate an unclear labyrinth of tables, hierarchies, and database objects. These few were the data heroes of old - the wise oracles (pun intended) who could get decision makers what they needed. But in today's world, where users are more sophisticated and demanding, this model is no longer feasible.
To be successful in the modern world, you need to focus on simplicity and speed. This means removing unnecessary hops, keeping transform logic (if it's even needed!) centralized and concise, and making sure that everyone in the organization has access to the data they need when they need it. With the advent of cloud data warehouses, cloud native databases, API focused services, and a variety of other public and private cloud features, we can do better.
ETL and traditional transform oriented tooling and processes need to be reconsidered for the modern data warehouse. Gone are the days when it was always necessary to Extract, Transform, and Load your data. In many cases newer sources have the ability to stream data to targets, and not require transformations due to the performance benefits of auto-scaling warehouses. Maintaining complex, custom stored procedures, or heavily decorated SSIS packages, may serve legacy use cases, but aren't needed in a future forward data warehouse. As technologies and platforms shift in terms of their paradigm, it requires a rethinking of how we approach traditional problems in light of newly viable solutions.
So what does this mean for you and your data warehouse? It means that you need to focus on simplicity and speed. Remove unnecessary hops and keep transform logic to a minimum. Make sure that everyone in the organization has access to the data they need, when they need it. And most importantly, don't be afraid to rethink traditional problems in a new way. With the right approach, you can have a data warehouse that is up to par with the best in the business.